Authors: Alankrita Pathak and Dhananjay Sathe

The Current State of Multi-Robot Communication with ROS1

In the ROS1 world, all relevant ROS nodes form a fully connected communication graph with peer-to-peer data transfer mediated by a single point of discovery (ROS master). It is a reliable approach when dealing with communication on the same piece of hardware or to an extent within a local area network.

Unfortunately, ROS1 architecture (ROS master, params, protocol) does not react kindly to hardware failure, network failure, or transient loss which in our experience occurs quite often in most real-world scenarios. It also lacks necessary features such as QoS, encryption/security, compression required while building enterprise-grade applications.

Additionally, while working with multiple robots/multiple hardware nodes, it mandates complex configurations and namespacing to assign identities, and avoid inadvertent cross-talk (e.g. two robots publishing to the same /odom topic).

A combination of all these factors makes it hard for users to leverage ROS to build robust multi-site, multi-agent real-world applications.

Introducing rapyuta.io Routed Networks

While building rapyuta.io we wanted to address these sets of problems transparently without further burdening developers familiar with their favorite tools, constructs, and paradigms. We went back to the drawing board to look at what a robust communication system for cloud-connected robots would look like and the result was a feature we call “Routed Networks”.  

To be able to facilitate diverse workloads on diverse kinds of hardware at multiple remote locations (or even in the cloud) we needed a mechanism that would possess the following properties:

Simple, Declarative, Type Agnostic Information Routing

To minimize friction for developers, it is important to allow developers to bring their existing codebase and tools. To make this possible we adopted a whitelist-based configuration that only requires the developer to provide a list of ROS topics/services/actions that the application wishes to expose from one ROS environment (all nodes connected to a local ROS master) to any other ROS environment. 

The platform automatically detects arbitrary user-defined ROS message types, topic latching configurations, and remote peers whether local, at a remote site, or in the cloud. This works for all languages and does not require code changes, dependency management, complex catkin compile steps, and custom ROS launch files.

Fine-Grained Tunables and Enhancements

Topic/service/action whitelist ensures optimal use of precious bandwidth while preventing cross-talk. We have added support for features that enable the developer to set QoS on a per topic level to ensure certain critical data gets priority over the other. The developer can additionally choose to enable transparent compression by selecting a simple toggle. We have also introduced mechanisms to add timeouts to ROS services. 

First Class Multi-Robot Communication Semantics

We wanted to solve the problems associated with the configuration and management of multi-robot communication in a simple way. Routed networks intelligently manage robot identity, allowing for optimally routed data flow between multiple robots doing away with the need for complex topic remaps, namespaced launches, or a “broadcast-all” topology.

Resilience, Security, and Robustness

Building applications that transmit information over the public internet mandates communicating differently from locally networked applications. When using a routed network the platform does this transparently for the user at multiple levels. i.e

• Cloud brokered data flows require zero special configuration/VPNs on the network layer.
• Automatic setup of network layer for each deployment along with end-to-end 2048 bit ssl encryption and dynamic credentials based authentication. 
• Dead peer-detection, liveliness checks, and features like retries, connection-pooling, order reassembly mechanisms ensure deterministic performance even in case of transient network loss or hardware failure.

You can configure these options when creating a rapyuta.io package.

Additionally, the platform provides a way to deploy a  routed network on edge or device on-site so multi-agent applications can skip the round trip to the cloud and reap all these benefits without sacrificing latency and performance.

Leveraging Routed Networks In your Application 

Prerequisites

A package is a fundamental rapyuta.io resource that represents a declaration of your application. It is the smallest unit of deployment in rapyuta.io, and it can be deployed either on a device or the cloud or both. Packages marked as ROS packages offer several features that are ROS specific including the ability to define the set of topics, services, and actions which the nodes in the package provide to anyone in the world.

A deployment is a rapyuta.io resource that represents a unique instantiation of a rapyuta.io package. It holds information about the package deployed, the configuration used, interfaces exposed, and provides a mechanism to introspect its phase and state.

Creating A New Routed Network

To begin, the user must first create a new routed network. A routed network can be deployed to a cloud or on a local device using similar workflows.

Cloud Routed Network

When a user deploys a routed network to the cloud it is considered a cloud routed network. Any computing resources (CPU/memory) consumed by this routed network deployment count against your cloud deployment hours quota.

Package deployments in the cloud OR device can bind to a cloud routed network.

Device Routed Network

In certain cases where communication is latency-sensitive or has high throughput, the user can choose to deploy a routed network to a device. While avoiding a round trip of information to the cloud minimizes latency and allows for better throughput, keep in mind that only deployments on devices on the same local area network can bind to it.

Routed networks can be deployed to a device with the following parameters:

• Device: Any online device with docker runtime and AMD64 architecture
• Device IP Interface: network interface (i.e., an IP address) that will be used by other deployments for communication to this routed network.

Connecting deployments to the routed network

By default, each rapyuta.io ROS deployment gets its own isolated ROS master. All ROS nodes within the component use native ROS discovery to find the required topics/services/actions. This works fairly well for ROS nodes on the same physical robot, or robots or cloud instances in the same local network. To facilitate multiple deployments to communicate with each other on rapyuta.io developers are required to bind their deployment to one or more “Routed Networks”.

Binding a routed network resource to your deployment will enable other deployments on the same routed network to consume ROS topics/services/actions as defined in the package. Due to the intelligence built into the discovery mechanism, data flow occurs only when another package chooses to subscribe to a topic, call service, or call an action.

There are few restrictions while connecting applications to a routed network in local mode. For example, when using routed networks deployed to a rapyuta.io device, only the deployments in the same physical network as the routed network can communicate with it.

Decoupled Deployment Lifecycles

The routed network has its independent lifecycle. The dynamic announcement and discovery allow deployments connected to it to be independently started, stopped, upgraded, and destroyed without affecting any other deployments. This facilitates in-place upgrades, self-healing characteristics to your distributed application. 

Identity 

When using advanced multi-robot features, the routed network ensures the uniqueness of deployment alias/identity to prevent inadvertent duplication.

Build your solutions leveraging routed networks

This feature is now available for you to build robust multi-site, multi-agent real-world applications. If you haven’t yet, please do sign up on rapyuta.io and check it out.  We built the pick assist AMR  solution which is now being used in real-world applications in Japan.

Please refer to Routed Networks documentation in case you want to know more. We would love to hear about your use cases and the applications you have in mind. Please write to us at support@rapyuta.io for any feedback, suggestions, or queries.