iLab Neuromorphic Robotics Toolkit  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Messages, Blackboard and Modules

Messages

Messages carry data from one module (which posts a message) to other modules (which may subscribe to that message type, or may asynchronously check for that message type).

In NRT, messages are just any C++ class, with as only requirement that the message should be serializable. That is, the message class should provide a method to convert itself into an array of bytes (on the sender/poster side, for transmission over the network), and to re-construct itself as a C++ class from such array of bytes (on the receiver/subscriber side).

Note
In NRT, as long as poster and subscriber/checker are on the same machine, no time or CPU are wasted in serializing/deserializing a message or even in making a copy of the message. Instead, NRT directly passes the message from the poster to the subscriber/checker using smart pointers. Of course, if poster and subscriber/checker are on different machines, serialization and transfer over the network are necessary.

In NRT, some messages are splittable. Those have a more restrictive definition. They are basically just a list of named element variables (fields). As the name suggests, splittable messages can be broken into their elementary members, and can be re-assembled from those members. This is particularly useful when some modules may be interested in only parts of complex messages posted by some other modules. Conversely, this allows one to aggregate a complex message from different parts possibly provided by different modules.

Note
NRT uses splittable messages and message composition rather than derived messages and inheritance. For now, just remember that in NRT we will not try to create base messages, derived messages, etc as instead we prefer composition over inheritance. Splittable messages are a crucial feature in avoiding the sprawling of modules which can only operate on complex data types and hence are not re-usable to operate on different data types. Splittable messages are further discussed below.

NRT finally implements a AnyMessage type, which is a wrapper for any message. Highly generic modules can be written that handle those generic messages (e.g., message queue, barrier, trigger, dispatcher, etc). This avoids having to re-write such modules for different message types. Generic messages are further discussed below.

Modules and ports

All computation in NRT is done inside of Modules, which represent self-contained processing units. Modules can communicate with each other by posting, subscribing to, or asynchronously checking for Messages. Posting, subscribing to or checking for messages is achived through Ports which are parts of modules. In the NRT Designer GUI, ports appear on the sides of modules, with color coding that represents message types.

module-ports.png

Between posters and subscribers, a transaction consists of an outgoing message (of type indicated by the Message type) posted by the poster, and a return message (of type indicated as the Return type) returned by the subscriber after it has received and processed the poster's message. The return message may contain, for example, results obtained after processing the received message. Not all transactions do return a message, as indeed the Return type can be void, and it often is. Even when the return is void, you should still think of post/subscribe transactions as round-trips, since any uncaught exception (runtime error in C++) thrown by a subscriber as it processes a received message will be propagated back to the poster.

To each poster port is associated a post member function that is accessible only from within the module to which the port is attached. Thus programmers of that module can post messages on any of the module's poster ports from any member function of the module at any time. Upon completion, post returns an object that contains a list of future results that will be returned by all the subscribers that have been triggered by the post. This is explained in more details in NRT Data Flow .

To each subscriber port is associated a callback or onMessage module member function. The onMessage member functions must be instantiated (programmed) by a module's programmer, one for each subscriber port. The onMessage function of a subscriber port is automatically called by the NRT core executive when a message is posted by some module which matches the requirements of a subscriber. When a poster is connected to several subscribers, the onMessage functions of all subscribers are called simultaneously in parallel execution threads. More details in NRT Data Flow .

Checker ports allow a module to asynchronously poll for previously posted messages. That is, when a poster is connected to a checker, the checker is not automatically triggered by each post (as was the case of subscribers). Instead, the module that has a checker port may issue, at any time, a check for messages on that port. The check may or may not return any messages (depending on whether or not some have already been posted). Checker ports are very useful to use from within onMessage functions of subscriber ports, to asynchronously gather the latest available auxiliary data that can be used to process the message just received by the subscriber port.

Checker ports do not have a return message because they are not triggered by posts (to whom would we return anything?)

In brief, when a module posts a message on a given Poster port, three things happen:

  1. The message is stored for later asynchronous use by possible Checkers for that message (each post on a given port overwrites the previous one on that port);
  2. All matching subscribers are called and passed the message. See below for a more detailed definition of matching subscribers;
  3. Results (if the return message type is not void), or possible exceptions, from all triggered subscribers are returned to the poster in an list, one entry per matching subscriber.

In actuality, things are a bit more complex, as we will learn in more details later.

Blackboard

When Modules post messages, these are not only dispatched to all modules which subscribe to those messages (standard publish/subscribe model), but they are also stored onto a local Blackboard, which acts as a repository of knowledge accessible to modules and also as a broker for messages. The Blackboard keeps the last posted message for each port of each module, overwriting it each time a new version is posted. This last posted version can be accessed asynchronously by modules, an action called Checking, which in essence is like polling the Blackboard for some data.

NRT implements a hybrid model between publish/subscribe (whereby each time a message is posted, callbacks are immediately triggered on the corresponding subscriber modules) and asynchronous or polling Blackboard (whereby the latest posted messages remain available for use, at any time, not necessarily dictated by the time at which they were posted). Typically, the publish/subscribe model will make data flow through streams of connected modules, which is useful, for example, in streaming video applications where each frame captured by a camera is subjected to a number of transformations. In addition, the asynchronous Blackboard mode allows any module to access data on demand, for example, a module may only need to display the latest video frame when the user clicks on a button, but should not be bothered and waste CPU receiving and processing each and every frame captured by a camera.

In distributed systems where NRT modules run over several computers on a network, one local Blackboard runs on each machine, and partial coherency is maintained over the federation of blackboards in the distributed system. This partial coherency is optimized to give the illusion of full coherency to Modules and to end users (i.e., any message on any computer on the network can be accessed by any module on any computer on the network) while minimizing the network cost of maintaining such full coherency (e.g., there is no need to transmit those messages stored on the blackboard of computer 1 to computer 2 if no module running on computer 2 might ever access those messages).

Distributed blackboard federations in NRT are built around a first blackboard, started on one computer, which assumes the role of master. As additional blackboards are started on additional computers, they will assume the roles of blackboard clients and will connect to the existing master.

Note
There is not much difference in terms of roles between master and client blackboards in NRT. The master is just the first one to start, and clients then connect to it as they start. While clients can die at any time without bringing down the entire system, in the current implementation the entire federation will be brought down if the master dies. Note that the master does not necessarily have to do any work, but rather it will help orchestrate the communications across the entire federation. For this reason, it may be wise to start the master on a reliable machine that will be subjected to low stress (e.g., same machine that also serves a networked filesystem), and then to start clients on high-performance compute nodes that will be subjected to maximum compute stress but would not bring down the entire system if they crash.

Topics and Topic Filters, Connections

Each poster port has a topic, which is a simple string that helps define which subscribers or checkers will be connected to that poster. Indeed, each subscriber and each checker port has a topic filter, which is a string-based regular expression. As a special case, a topic filter can be a plain string (which is a trivial regular expression), or it can contain wildcards.

While a module's port are defined and fixed at design (and compile) time, topics and topic filters are set at runtime to indicate, dynamically, which port is connected with which other port.

Matching (connection) between a poster and a subscriber in NRT requires three conditions to be satisfied:

  1. The outgoing message type of the poster matches that of the subscriber;
  2. The return message type of the poster matches that of the subscriber;
  3. The topic of the poster meshes with the topic filter of the subscriber.

Meshing between a topic (string) and a topic filter (regular expression) means that there is a regular expression match (for example, topic ImageLeft meshes with topic filter Image* but not with topic filter ImageRight).

Matching (connection) between a poster and a checker in NRT requires two conditions to be satisfied:

  1. The outgoing message type of the poster matches that of the checker;
  2. The topic of the poster meshes with the topic filter of the checker.
connections.png

In the above example, the top poster port of the Image Source posts an Image message and expects no return (void). The subscriber ports of both the Morphology Filter and the Image Sink modules have matching message and return types. What defines which connections exist, then, are the values of the topics and topic filters. In this example, the topics and filters were set as indicated by the blue text. Because RawImage is an exact match between the first pair of poster and subscriber, a connection is established. Because FiltImgMorpho is a regular expression match with FiltImg*, the second connection is established. Because RawImage is not a regular expression match for FiltImg*, no direct connection exists between the Image Source and the Image Sink.

In the NRT Designer GUI, topics are automatically set to matching unique names as the user drags a connection between two ports.

Note
Why do we need all three of a unique port name, message/return types, and topic? Isn't there some redundancy here? Actually, no there isn't. Consider the simple example of a module that converts an RGB image into the LAB color space, posting each of the L, a and b components on three separate ports. In this module, each of the three poster ports posts the same message type (an image), expecting nothing in return (void return type).
image-lab-example.png
The three output ports need distinct names, so that the module's programmer can post the L component onto the first port, the a component onto the second, and the b component onto the third. What about topics? Those are used at run time, in the example above, we have set the topic of the L (top) port of our module to match the subscriber of the first image sink (and this match is represented in the NRT Designer by a curve linking the two ports). Likewise, we have chosen a different topic to link the middle poster of our LAB module to the middle image sink, and similarly for the last port and sink.
So, in summary, unique port names are used by programmers at compile time to know onto which port they want to post their messages, and topics are used by users to connect ports ar runtime.

To sum up, each port is uniquely defined by the following:

  • The module to which it belongs.
  • A unique port identifier within the module (fixed at design-time / compile-time). In NRT, this is a unique C++ type, which makes it very hard for programmers to mistakenly post the wrong data to the wrong port.
  • The type of the outgoing message that gets posted (fixed). We often refer to this as the Message type.
  • For Posters and Subscribers, the type of message that is expected in return from subscribers triggered by a post, or void if nothing is expected in return (fixed). We often refer to this as the Return type.
  • A topic string (for posters) or topic filter (for checkers and subscribers) that is chosen at run-time and can be changed at any time during operation, to establish connections between ports.

Splittable Messages and Ports

One lethal problem with many message passing frameworks is that one often writes modules which post complex and specific message types, such that more elementary modules cannot easily be used jointly with the complex messages. A typical example is: consider a robot, with a GPS sensor. The GPS sensor may post a message about the GPS data each time that data is updated. This would typically result in the software designer creating a new GPSData message type, with fields for latitude, longitude, etc. The major problem with this is if one wants to access only part of the data. For example, let’s say we have already designed a temporal low-pass filter module that can filter scalar time series. If one wants to filter the longitude of the GPS data, unfortunately most message passing frameworks would not provide facilities that would allow using the existing filter on just one member of the GPS message. NRT resolves this problem by introducing splittable messages and ports:

split1.png

Figure above: Example of a BinaryOperation module that expects as input a pair of images to which some operation should be applied (users can choose sum, difference, etc operations), and which outputs an image. If one has two image sources, each one outputting an image, in most existing message-oriented frameworks one would not be able to directly connect them to the BinaryOperation (because Image and ImagePair are different message types).

split2.png

Figure above: In NRT, the user can right-click onto the ImagePair port and split that port. Note the two new sub-ports that now appear on the left side of the BinaryOperation module (here sub-ports have an additional black dot next to them). Each one is a single Image message input port. NRT automatically handles the re-assembling of the two separately received messages into the expected ImagePair message. Hence, the programmer of the BinaryOperation module did not have to worry about this and simply programmed the BinaryOperation module under the assumption that a pair of images would be received and some operation would be applied to it.

In NRT, we thus use composition and composite messages built out of simpler parts, rather than inheritance and messages derived from a simpler base.

Generic Messages

A module in NRT may be designed to handle any message type. In such case, the module is not able to access the contents of the message, but it can store and later repost received messages. Typical applications of this paradigm are queues, dispatchers, collectors, conditional routers, barriers, triggers, etc. In previous message-passing frameworks, one would typically have to write different modules for a FIFO queue for images vs for integers vs for sound waves, etc. In NRT, only one module is necessary.

genericmsg.png

Figure: Generic FIFO queue implemented in NRT. The FIFO queue module is programmed using AnyMessage input and output ports (in light blue). AnyMessage objects can be manipulated (e.g., stored in a queue) but are opaque (their internal data is not accessible to the module). Any other module can connect to AnyMessage ports: for example, here the Image Source posts an Image message, which can be received by the FIFO Queue (note the transition between the pink color of the Image message and the blue of the AnyMessage). Likewise, the AnyMessage output of the FIFO Queue can be connected to a Display Sink module that accepts Image messages.

Note
There is no computation overhead in this framework (e.g., no deep copy of a message’s data occurs as it is wrapped into an AnyMessage; all is done using smart pointers). Also note that it is possible to create mistyped connections (e.g., a Sound Source connects to a FIFO Queue which then connects to an Image Sink); those are detected at runtime and throw exceptions when encountered.