Chapter 8. Communications Components

Table of Contents
The Real-Time Ethernet Architecture
RT-CANopen architecture
Verification of Distrubuted Systems

The Real-Time Ethernet Architecture

Real-Time Publish-Subscribe (RTPS)

Real-time applications require more functionality then the one provided by traditional publish-subscribe architectures. RTPS adds publication and subscription timing parameters and properties so the developer can control different types of data flows and achieve their application's performance and reliability goals.

Figure 8-3 illustrates how RTPS uses the deadline, minimum separation, strength and persistence properties to provide network communication for real-time applications and simplify system design. Features shown on the figure are:

Subscription Issue data flow: If a new issue does not arrive by the deadline, the application is notified.

Redundant Issue hot-swap: During persistance period ORTE stack accepts issues from publiccations with equal or higher strength. After persistance, it accepts the first issue, regardless of the publication's strength.

ORTE architecture

The OCERA Real-Time Ethernet (ORTE) will be open source implementation of RTPS communication protocol. This protocol is being to submit to IETF as an informational RFC and has been adopted by the IDA group. Figure 8-4 shows the network stack layers. Non Real-Time applications, which are using standard protocols such as HTTP, FTP, DCOM etc., are running on top of standard TCP or UDP stack. ORTE is new application layer protocol, which is build on top of standard UDP stack. Since there are many TCP/IP stack implementations under many operating systems and ORTE protocol does not have any other special HW/SW requirements, it should be easily ported to many HW/SW target platforms. It doesn't use or require TCP, so it retains control of timing and reliability.

The ORTE is composed from three main components, as shown on Figure 8-5:

Database: stores parameters describing both local as well as remote node's objects.

Processes: perform message processing, serialization/deserialization and communication between objects.

API: application interface.

The RTPS protocol is implemented as a set of objects. Objects are of the following types:

  • Manager (M)

  • ManagedApplication (MA)

  • Services

  • Writers (Publication, CSTWriter)

  • Readers (Subscription, CSTReader)

A Manager is a special object that facilitates the automatic discovery of other Managers. There is one Manager on each participating network node. A ManagedApplication is an applciation that is managed by one or more Managers. The Publication is used to publish issues to matching Subscription. The CSTWriter and CSTReader are the equivalent of the Publication and Subscription, respectively, but are used solely for the state-synchronization protocol. Each object on network is characterized by GUID (Globally Unique Id).

The RTPS protocol uses five logical messages:

ISSUE: Contains the application's user data. ISSUES are sent by Publications to one or more Subscriptions.

VAR: Contain information about attributes of state of objects.

HEARTBEAT (HB): Describes the information which is available in a writer.

GAP: Describes information which is no more relevant to readers.

ACK: Provides information on the state of a reader to a writer.

Each of these logical messages are sent between specific readers and writers as follows:

Publication to subscription(s): ISSUE, HEARTBEAT

Subscription to publication: ACK


CSTReader to a CSTWriter: ACK

Database implementation

Status of all objects is stored in ORTE database. The database is set data structures containing parameters of all known objects. There are not only data describing local node's objects, but also description of all known remote nodes' objects. Since access to this database should be granted to both ORTE stack processes as well as to applications, there should be implemented some access control algorythm allowing concurrent access. Applications use an API call to access the database.

Database contains two types of information. The first type is description of all known objects without any specification of relationship among them. This information is stored using a binary tree structure with object's GUID used as the key. The second type of information describes relationship among objects. Such relationship is for example association between Manager and ManagedApplications or between publishers and subscribers. This information is stored as a (double) linked list of GUID of objects which belong to certain owner. As an example, such owner is a publisher, the linked list contains list of GUIDs of its subscribers. The communication objects can be represented as the branch of a tree. Each of the nodes in the branch and its descendants represents an element of the complex data-structure.

Example of content of a database is shown on Figure 8-6. There are two nodes Node 1 and Node 2. The figure shows content of database on both these nodes. The situation on Node 1 is as follow:

  • there is local manager Manager 1

  • Manager O2 is local copy of parameters describing Manager 2, which is located on remote node Node 2

  • there is local managed application ManagedApp 1

  • there is local copy of remote application ManagedApp O2

  • there are publishers Publisher 1 and Publisher 2

  • there is remote subscriber Subscriber O1 which is subscribed to data published by Publisher 1

  • there is currently no subscription to Publisher 2

Situation in node 2's database is similar.

Note: letter O, which is placed before numerical index of an object means, that this database object represents object which belongs to other node than local. For example node 1's object ManagedApp O2 is local representation of node 2's object ManagedApp 2.

ORTE processes implementation

The ORTE system spawns several processes during its lifetime as shown on Figure 8-7. Generaly there are two groups of processes. The first group represents processes involved in real-time application's data exchange between local and remote nodes. These processes are responsible for publishing of local application's data if there are any remote subscribers and also for receiving of remote application's data if there are any local subscribers. Another group are processes involved in network management.

Process Subscription which is responsible for processing of all local subscriptions, i.e. it receives all application data from all remote publishers to which local applications want to be subscribed. It performs whole processing of received data including storing of all data changes into database and, if requested by application, also calling an application's callback functions. This process can listen on more UDP ports where each port belongs to one local subscription, or on a single UDP port, which is common to all subscriptions. In any case, this process will never listen on default NDDS port 7400.

Processes Publication1, Publication2, ..., PublicationN are responsible for proper publication of local data to all its subscribers across the network. Each process is associated with queue Pqueue1, Pqueue2,... PqueueN. Data changes are stored (using proper API call) into certain queue. Process which is associated with this queue reads its content and sends ISSUE message to all subscribers and waits for their ACK. Each process in this group use its own UDP port, they will never use default NDDS port 7400.

Process NodeManagement is responsible for whole management traffic. It is the only process which uses default NDDS port. Since management traffic does not have real-time requirements, it is designed as single process which processes both incomming as well as outgoing requests. During evaluating phase it may be split to two separate processes, one for incomming requests and second for outgoing requests.