Incident Report: Breach of rapyuta.io devices due to a vulnerability in SaltStack

Author

Date

A new exploit on SaltStack was actively exploited, resulting in the installation of malware on some rapyuta.io devices that were online during the time of the attack. Fortunately, the malware was a more uncomplicated variety, and we were able to remove it successfully from all affected devices.

We take data security and privacy very seriously on the rapyuta.io, and we are re-evaluating all internal components for security.

Technical Details

On April 30, F-secure discovered a new vulnerability found in SaltStack(CVE-2020-11651), which has affected other major companies as well. SaltStack is a popular tool used by major organizations. rapyuta.io uses it for managing devices on the platform. The vulnerability on SaltStack allowed for remote code execution on online devices on the platform.

On May 3, the exploit was actively exploited by a malicious party on rapyuta.io as well as several thousand other servers resulting in the installation of malware on rapyuta.io devices. We noticed the presence of the malware internally on May 4. The malware turned out to be a crypto-miner that starts using up all the CPU cycles of the device. It also adds a crontab entry to have persistence. We removed the malware from all active devices and also released the script we used to remove it as well.

Unfortunately, the patch on 29 April was missed by us, resulting in the vulnerability being exploitable for four days. The long weekend with engineers being away contributed to the delay as well.

Timeline

  • 29-04-2020 – Public patches for the attack are pushed into SaltStack upstream.
  • 03-05-2020 06:45:00 UTC – rapyuta.io devices are hit with the malware.
  • 04-05-2020 04:40:00 UTC – Confirmed report of malware on rapyuta.io devices.
  • 04-05-2020 06:40:00 UTC – Device Management is shutdown on rapyuta.io to prevent any additional payloads from being executed.
  • 04-05-2020 13:30:00 UTC – All devices that were affected are successfully cleaned.
  • 04-05-2020 15:04:00 UTC – Patched version is deployed, and device management is back online.

How it affects you as the user of rapyuta.io

We have reached out to all the users who were directly affected by this malware. If you did not get an email, you were not part of the affected group.

As a security measure, we have refreshed security and authentication tokens for all devices on boarded by users on our platform. As a result, all your devices will have to be re-authenticated with the rapyuta.io platform before you can use them again. We have preserved all the labels and metadata associated with a device.

To re-authenticate your devices, follow the steps below:

  1. Please select the device.
  2. Click on Token button.
  3. Copy the curl command
  4. Execute it on the physical device.

This only applies for older users. New users can follow the onboarding tutorial.

Measures taken by rapyuta.io for future security

We are making sure that all our internet-facing components are secured against generic vulnerabilities in the future. To prevent us from missing essential patches, we are making sure that we read and acknowledge patch notes/emails for all vital services used by rapyuta.io.

Security has always been an important priority at rapyuta.io. We have timely external third-party security testing of our platform done and also continue to work on improving the general security of the components.

Feel free to reach out to us at support@rapyuta.io for any questions or clarifications.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Take advantage of our thousands of hours of research and domain expertise. Talk to us for your robotics transformations solutions!

Recent posts

What we develop here at Rapyuta Robotics, is beneficial for society as a whole – Michael Orr

Interview with Michael Orr, who is an engineering lead at Rapyuta. He talks about careers in robotics, his view on engineering and his motivations

Cloud Robotics: Our Perspective

This blog post details our perspective of Cloud Robotics presented at the Robotics: Science and Systems (RSS) 2019 Workshop, Messe Freiburg, Germany...

A Safari into the KubeVerse — Talk by Dhananjay Sathe at Kubernetes Forum

Dhanajay Sathe giving his talk at Kubernetes Forum Bangalore As part of our knowledge-sharing initiatives, our Staff Engineer Dhananjay Sathe gave a talk at Kubernetes...

CloudRobotics #101 — Deploying, Managing, and Operating TurtleBot3

Introduction TurtleBot3 is a small, programmable, popular ROS-based mobile robot for use in education, research, hobby, and product prototyping. The goal of TurtleBot3 is to...

Event report — iREX 2019

We exhibited for the first time at the “2019 International Robot Exhibition (iREX 2019)” held at Tokyo Big Sight from December 18 (Wednesday) to...

Rapyuta Robotics launches drone solution powered by rapyuta.io

Press release: November 15, 2017 Rapyuta Robotics (Chuo, Tokyo. CEO: Gajan Mohanarajah) announced the beta launch of its new...