INtime SDK Help
Distributed Applications
With INtime, you can distribute applications across different nodes on multiple hosts. The features that enable this include:
Note: INtime uses these definitions:
Host: A processor with shared memory & IO bus with one BIOS on one or more cores on one or more sockets.
Core: One of many cores in a multicore processor.
Node: One OS controlled entity - one instance of the INtime or Windows kernel each using one or more cores.
With the node architecture and Global Objects, inter-process communication permits multiple independent applications to run on the same node, over several nodes, or over several hosts at the same time while interacting with each other in a real-time, deterministic way. The Distributed Systems Manager (DSM) allows the whole system of connected nodes and hosts to have the following attributes:
Host partitioning on multicore processors hosts allow the division of shared resources between independent nodes running concurrently on that processor.
INtime architecture allows multiple nodes, complete instances of an INtime kernel and an INtime application, to run together on a multicore host, across several hosts or a combination thereof.
With processes on each node communicating in a managed way via GOBSnet, several applications running on different nodes and different hosts can work together as a system. For this to occur, nodes must be detectable by other nodes and identifiable via a unique name.
The node architecture and host partitioning has been described in the System Description section.
The following figure shows three hosts, Hosts 1 and 2 have a 4-core processor and the third one has a single core processor. Application processes running on individual nodes communicate with each other with the Global Objects technology.
Partitioning the host requires allocating a host’s common resources among INtime nodes running independent applications on each processor core. You can allocate the following resources:
• Memory: PC hosts share common memory. INtime is a 32-bit operating system that runs in the first 4GB of memory. In a multicore processor, you can manage portions of that memory to make it exclusively accessible to an application running on a particular node. (For more information on memory allocation, see Memory Management.)
• I/O devices: In INtime for WIndows an I/O device is initially "owned" by Windows. It may be assigned to an INtime node using the INtime Device Manager. This action causes a dummy Windows device driver to be installed on the device which prevents Windows from further access to the device, then signals the node on startup to indicate INtime now owns the resources associated with the device.
• Interrupts: All interrupts are owned by Windows until a device which owns an interrupt is assigned to an INtime node. At that point when the interrupt triggers, only the node which is handling the interrupt for the device is signaled, and it is not directed to the other nodes. The exception to this is if an interrupt is shared between two devices which are assigned to two different INtime nodes. In this case both nodes are signaled when the interrupt triggers. For further information see Interrupt Management.
Example:
This example shows four-core processor host partitioning, with the following instances of INtime nodes loaded on the host, one per core:
| Node | Application | Description |
|---|---|---|
| A | APP.1 | HMI (Human Machine Interface). |
| A | APP.2 | Activity logging and communication to central control system. |
| B | APP.3a | First part of robot control based on “setup” input via the control console application. |
| C | APP.3b | Second part of the robot control application. |
| D | APP.4 | Image capture and analysis application. |
Each node runs separate applications:
Separating I/Os enables applications to make better use of available resources and ensures a high degree of determinacy.
Processes use Global Objects technology to pass information between processes running on different nodes on the same host or on a separate host. Global Objects consists of these components:
The Global Objects manager is a process which exists on each INtime node. It is responsible for maintaining information about all the nodes in the local host, as well as about nodes on other hosts it has communicated with, including the state of each node. It maintains a database of object references and also interacts with the local DSM in order to consume and provide system events.
The Global Objects network ("GOBSnet") manager is a process which extends the functionality of the GOBS manager over the network to nodes on other hosts, completing the functional layer which extends the Global Objects functionality to all nodes on all hosts in an INtme system. Typically there is one instance of the GOBSnet manager per host, and it depends on a local TCP/IP networking stack. It keeps information about all the GOBSnet hosts it communicates with, and monitors the network links between them. If the link to another host becomes unavailable it will cause any requests directed over that link to fail.
The GOBSnet manager is also responsible for forwarding requests to and from all of the nodes on the local host. See the examples below for details of how this works in practice.
The key to the global objects technology is a special object called a reference object. When a process wants to access an object on another node it must obtain a reference to that object. This object contains information about the type and location of the object so that the GOBS software can locate it and perform operations on it. The handle for a reference object can be used in the system calls which would normally be used to manipulate that object type.
For example, if a process on NodeA is to access a mailbox on NodeB, it must first obtain a reference to that mailbox. A local reference object is created and the handle which refers to that reference may be used in the normal mailbox calls. If the process calls GetRtHandleType on the handle it will return the value MAILBOX_TYPE, as if it was really a mailbox handle. In the case that the caller needs to know that this is really a handle for a reference object, the system call GetRtHandleTypeEx returns a value REFERENCE_SUBTYPE which is OR-ed with the normal subtype for that object.
References may be obtained implicitly, for example by looking up a handle on another node. In this case the LookupRtHandle call determines that the process handle is itself a reference to a process on another node and implicitly creates a reference object for the handle being looked up. More often, references are explicitly created using the CreateRtReferenceObject system call with the actual value of the handle on the other node and the location handle for the node.
A reference object counts towards the object count for the local process. So there are two objects involved in the operation, the reference object and the target object. When the process is finished with a reference object it can delete it using the DeleteRtReferenceObject system call. Deleting the reference object does not automatically delete the target object. However if the process chooses to delete the target object then the reference object is also deleted implicitly.
A special class of object is created for the case where an object is created by a process but it is desired that the object continues to exist even when the process is deleted. For example, a process may be used to create a shared memory object which is used by processes on multiple nodes. Even if the original process terminates the other processes need to continue to use the shared memory.
In this case a special reference-counted version of the object may be created. This keeps a count of the number of reference objects which have been created for the object and is only deleted when that number drops to zero as the reference objects are deleted. In this way the object remains extant until the last user of the object is deleted. This special property is currently applicable to Semaphores, Mailboxes and Memory objects using the special calls CreateGlobalRtSemaphore, CreateGlobalRtMailbox, CreateGlobalRtMemoryObject (to create a memory object of a certain size) and CreateGlobalRtMemoryHandle (to create a memory handle which refers to a physical memory area). Once the object is created then the standard operations apply. Attempting to delete the object will only reduce the reference count and not actually delete the object unless the reference count falls to zero.
You can also use these calls to create an object on another node. If the call does not specify the REFCOUNT flag, then the mailbox (for example) is created in the GOBS manager process of the target node indicated by the location parameter but it is still considered to be owned by the calling process. Making the call creates a local reference object for the object and when the process is deleted both the reference object and the actual object will be deleted. This is different from the general case where a reference has been created for a "normal" object where only the reference object is deleted when the local process is deleted.
In this case, a process on NodeA creates and catalogs a mailbox. Another process on NodeB finds the node location then looks up the mailbox handle.
| Actions on NodeA | Actions on NodeB |
|---|---|
| Create a mailbox (CreateRtMailbox) | - |
| Catalog the mailbox in the root process (CatalogRtHandle) | - |
| - | Lookup NodeA and obtain a location handle (GetRtNodeLocationByName) |
| - | Lookup the node's root process (GetRemoteRootRtProcess) |
| - | Lookup the mailbox handle (LookupRtHandle) |
| - | Send a message (SendRtData) |
| Receive a message (ReceiveRtData) | - |
| Delete the mailbox (DeleteRtMailbox) | - |
| Delete the reference object (implicit)(DeleteRtReferenceObject) |
The DSM is a cooperative, multiple-process application that manages a distributed INtime multi-node system. The DSM tracks the system’s state, monitors the health of its components, and cleans up in the event of component termination or failure. An API allows applications to receive notifications about the system or other processes.
The DSM provides these service levels:
When a node starts, its management software does the following:
Each INtime node’s management software does the following:
In an INtime for Windows host, Windows provides the following system services to the local INtime nodes:
INtime nodes on an INtime for Windows host register for these services by default. A Windows host may dynamically be assigned as a host for an INtime Distributed RTOS host by use of the makeintimehost utility. Once a Windows host is connected as a host, applications on that node may use Windows services.