Robust Reinforcement Learning over Wireless Networks with Homomorphic State Representations
This work addresses the challenge of deploying RL in real-world wireless systems, offering a more robust and efficient solution for applications like robotics or IoT, though it is incremental by building on existing RL frameworks.
The paper tackles the problem of training reinforcement learning agents over lossy or delayed wireless networks by proposing HR3L, a novel architecture that uses homomorphic state representations to enable efficient remote training without gradient exchange, resulting in significant improvements in sample efficiency and adaptability to various communication issues.
In this work, we address the problem of training Reinforcement Learning (RL) agents over communication networks. The RL paradigm requires the agent to instantaneously perceive the state evolution to infer the effects of its actions on the environment. This is impossible if the agent receives state updates over lossy or delayed wireless systems and thus operates with partial and intermittent information. In recent years, numerous frameworks have been proposed to manage RL with imperfect feedback; however, they often offer specific solutions with a substantial computational burden. To address these limits, we propose a novel architecture, named Homomorphic Robust Remote Reinforcement Learning (HR3L), that enables the training of remote RL agents exchanging observations across a non-ideal wireless channel. HR3L considers two units: the transmitter, which encodes meaningful representations of the environment, and the receiver, which decodes these messages and performs actions to maximize a reward signal. Importantly, HR3L does not require the exchange of gradient information across the wireless channel, allowing for quicker training and a lower communication overhead than state-of-the-art solutions. Experimental results demonstrate that HR3L significantly outperforms baseline methods in terms of sample efficiency and adapts to different communication scenarios, including packet losses, delayed transmissions, and capacity limitations.