Marten Lohstroh

Jacky Kwok, Marten Lohstroh, Edward A. Lee

Parallel Reinforcement Learning (RL) frameworks are essential for mapping RL workloads to multiple computational resources, allowing for faster generation of samples, estimation of values, and policy improvement. These computational paradigms require a seamless integration of training, serving, and simulation workloads. Existing frameworks, such as Ray, are not managing this orchestration efficiently, especially in RL tasks that demand intensive input/output and synchronization between actors on a single node. In this study, we have proposed a solution implementing the reactor model, which enforces a set of actors to have a fixed communication pattern. This allows the scheduler to eliminate work needed for synchronization, such as acquiring and releasing locks for each actor or sending and processing coordination-related messages. Our framework, Lingua Franca (LF), a coordination language based on the reactor model, also supports true parallelism in Python and provides a unified interface that allows users to automatically generate dataflow graphs for RL tasks. In comparison to Ray on a single-node multi-core compute platform, LF achieves 1.21x and 11.62x higher simulation throughput in OpenAI Gym and Atari environments, reduces the average training time of synchronized parallel Q-learning by 31.2%, and accelerates multi-agent RL inference by 5.12x.

Jacky Kwok, Shulu Li, Marten Lohstroh et al.

The rise of intelligent autonomous systems, especially in robotics and autonomous agents, has created a critical need for robust communication middleware that can ensure real-time processing of extensive sensor data. Current robotics middleware like Robot Operating System (ROS) 2 faces challenges with nondeterminism and high communication latency when dealing with large data across multiple subscribers on a multi-core compute platform. To address these issues, we present High-Performance Robotic Middleware (HPRM), built on top of the deterministic coordination language Lingua Franca (LF). HPRM employs optimizations including an in-memory object store for efficient zero-copy transfer of large payloads, adaptive serialization to minimize serialization overhead, and an eager protocol with real-time sockets to reduce handshake latency. Benchmarks show HPRM achieves up to 173x lower latency than ROS2 when broadcasting large messages to multiple nodes. We then demonstrate the benefits of HPRM by integrating it with the CARLA simulator and running reinforcement learning agents along with object detection workloads. In the CARLA autonomous driving application, HPRM attains 91.1% lower latency than ROS2. The deterministic coordination semantics of HPRM, combined with its optimized IPC mechanisms, enable efficient and predictable real-time communication for intelligent autonomous systems.

Marten Lohstroh

Data security, which is concerned with the prevention of unauthorized access to computers, databases, and websites, helps protect digital privacy and ensure data integrity. It is extremely difficult, however, to make security watertight, and security breaches are not uncommon. The consequences of stolen credentials go well beyond the leakage of other types of information because they can further compromise other systems. This paper criticizes the practice of using clear-text identity attributes, such as Social Security or driver's license numbers -- which are in principle not even secret -- as acceptable authentication tokens or assertions of ownership, and proposes a simple protocol that straightforwardly applies public-key cryptography to make identity claims verifiable, even when they are issued remotely via the Internet. This protocol has the potential of elevating the business practices of credit providers, rental agencies, and other service companies that have hitherto exposed consumers to the risk of identity theft, to where identity theft becomes virtually impossible.