rapyuta.io, our cloud robotics simulation software, is currently based on Gazebo and ROS (which we’ll call ROS1 hereafter, as ROS can also refer to both versions).

This architecture presents a series of issues when trying to scale to multiple robots:

• ROS1 was originally designed for a single robot. Some hacks are required to run multiple robot simulation
• Gazebo’s performance/scalability issue to simulate multiple robots
• We needed to create cloud distributed environment to scale the simulation. /clock topic over the internet and to sync multiple ros master environment is not very optimal.
• Creating/editing URDF/SDF is generally not trivial for non-roboticists.

To scale our simulation on the cloud and provide a more user-friendly GUI for non-roboticist users, we decided to use Unreal Engine 4, an open source game engine that provides high quality rendering and real-time physics simulation in a potentially distributed environment. 

To reach this objective, we created rclUE, which bridges UE4 and ROS2, opening UE4’s tools to ROS developers through a C++ interface. Furthermore non-ROS engineers should be able to quickly set up robot simulations through UE4’s blueprint visual scripting language by using the instruments that ROS developers and the plugin exposed. 

A demo of our tool can be found in our port of the  turtlebot examples to UE4. The project relies on our rclUE plugin for ROS2 as well as the RapyutaSimulationPlugins for additional support.

Why Unreal Engine as a robotics simulator?

Unreal Engine is an open source game engine developed by Epic Games that provides high quality rendering (including photorealistic), agent behavior AI (through behavior trees), a scalable network architecture and real-time physics simulations (with Nvidia’s PhysX and Epic Games’s Chaos) through a C++ and a visual scripting interfaces.

In the context of robotics, we can take advantage of this framework in the following ways:

• simulations with photorealistic rendering can be used for marketing tools and machine learning
• Unreal Editor and Blueprint visual scripting can be used to quickly and easily setup simulation environments and robots
• behaviour trees (BT) can be used to intuitively create AI for different agents (including non-robotics elements)
• Unreal’s built-in network architecture and replication enables consistent simulations over the internet with tens of clients
• PhysX and Chaos physics engines can unlock faster-than-real time simulations for multi-robot systems

rclUE Design

*skip to the next section to see some examples

Unreal Architecture 

One of the core features of Unreal Engine is the ability to create new objects and expose properties and functions to the editor by way of panel integration (i.e. for changing data properties) or Blueprint, for scripting logic.

Unreal’s type system favors inheritance over composition, and requires that (ultimately) all classes that work with these features derive from a base UObject class. These UObjects are managed memory, similar to Java objects, and Unreal has a built-in garbage collector to service them.

To further empower the system, data structures such as arrays, maps, and many functionalities which exist inside the C++ STL have a custom implementation to also support such serialization. While it is possible to use STL, the preference in Unreal is to use the built-in equivalent functionality.

Previous works for UE4-ROS integration

ROSIntegration is a UE4’s plugin to enable communication between ROS1 and UE4 through rosbridge. CARLA, a simulator for autonomous driving research, is based on a fork of UE4 and has its own ROS bridge to enable communication between ROS (including ROS2) and itself.

AirSim is a simulation software designed for drones and wheeled vehicles, and integrates with ROS through a wrapper composed of 2 nodes: one for the C++ multirotor client library and one for the PD position controller.

Both CARLA Sim and AirSim take advantage of Unreal Engine’s rendering capabilities to perform machine learning tasks related to autonomous driving. 

In contrast to the integrations done in AirSim and CARLA, rclUE is designed to be simple and portable. For this, we implement the required ROS2 functionalities in a UE4 plugin (similarly to ROSIntegration), so that it can be easily included in different UE4 projects and is therefore more accessible for the community.

Additionally, our tool integrates with UE4’s architecture, mapping ROS2 elements to UE4 types (e.g. Nodes as Actors, Publishers as ActorComponents) and the editing of parameters from UE4’s interface and extension through Blueprint visual scripting language.

It is our hope that the rclUE plugin will also allow non-ROS developers to leverage ROS2 software in UE4 with minimal effort, thus inviting contributions from specialists outside of robotics fields. 

rclUE Architecture

Because of Unreal’s opinionated architecture, a clean implementation must follow the general practices of the engine, and cannot itself bring design opinions. Therefore it makes sense to choose a ROS2 client library written in C (rcl, rclc), rather than C++ (rclcpp). In particular, rclcpp is built on fundamentals of inheritance which can directly conflict with UE4’s version of inheritance necessary to work with serialization and garbage collection.

The picture below shows how our rclUE layer fits in the overall architecture.

Hereafter we discuss some implementation details for ROS2 nodes, publishers and subscribers, with other components (services, actions) following the same principles.

ROS2 Node

ROS2Node is implemented as UE4 Actor, the base class for objects that can be placed in a level (i.e. the simulation world). The ROS2Node tracks and manages publishers, subscribers, service clients and services (and thus contains a simplified equivalent of rclcpp’s executor).

ROS2 Publisher

ROS2Pulisher is implemented as an ActorComponent, the base class for components that define reusable behavior that can be added to different types of Actors. As an ActorComponent, it can be attached to any actor that requires a publisher.

ROS2 Subscriber

Topic subscription is implemented as a method of ROS2Node and is bound to a callback method through a UE4 delegate. An Actor that wants to subscribe to a topic therefore registers its callback function to a ROS2Node.

Examples

PubSub Example (Unreal C++) 

*Currently rclUE supports Ubuntu 20.04 and ROS2 foxy. Please setup UE4 and ROS2 by following tutorials

• https://docs.unrealengine.com/4.26/en-US/SharingAndReleasing/Linux/BeginnerLinuxDeveloper/SettingUpAnUnrealWorkflow/
• https://docs.ros.org/en/foxy/Installation.html

The content presented here can be found in the turtlebot3-UE sample code.

On a ROS2Node Actor, after initializing it, we initialize the ActorComponent StringPublisher with NewObject, set the parameters and bind the delegate.

Subsequently we add the publisher and initialize it.

Note that we set up the publisher in BeginPlay, a method for Actors that gets called when we start the simulation, instead of doing it in the constructor, which only gets called when the actor is added to the scene or when the executable is compiled. An important distinction is that by initializing things in BeginPlay, variables (such as publication frequency) can be set in the editor and their change will be reflected when running the simulation. If, in contrast, we initialize things in the constructor,, variables changed in the editor would not reflect in the simulation, unless we restart the editor.

To add a subscription, after the node is initialized, we bind a callback function to the object of type FSubscriptionCallback and call AddSubscription.

In this example, we used a publisher and a subscriber node to keep it simple, but in practical use cases, the elements presented should be integrated in the actors that need these functionalities.

PubSub Example (Blueprint) 

Setup of a ROS2 Node with a string publisher: initialize the Actor ROS2Node actor, then add the ActorComponent Publisher and initialize it on the node that will publish the message. The node and publisher parameters (node name, message type, topic name and publication frequency) are set in the Details panel (note that the red line connects to the callback function shown at the bottom).

Setup of a ROS2 Node with a string subscriber: initialize the Actor ROS2Node actor, then add the subscription and bind callback function which prints the logs on the window (note that the red line connects to the callback function shown at the bottom).

Note that this is only one of the many ways in which a pubsub can be set up.

In the end

We introduced rclUE. You can control the robot inside UE4 by applying the contents above.

You can try turtlebot3 simulation with UE4 by following the instructions on the README at https://github.com/rapyuta-robotics/turtlebot3-UE 

This work is still in its early stages and you will find some bugs and missing features. We’d appreciate bug reports and feedback and we look forward to collaborating with the community to make rclUE more useful.

This project is supported by Tokyo metropolitan government subsidy