When Sensors Fail: Temporal Sequence Models for Robust PPO under Sensor Drift
This work provides a practical mechanism for improving the reliability of reinforcement learning agents operating in real-world environments with unreliable sensors, which is a common challenge for robotics and autonomous systems.
This paper addresses the problem of sensor drift in reinforcement learning systems, specifically with Proximal Policy Optimization (PPO). The authors augment PPO with temporal sequence models, such as Transformers, to infer missing information from historical data. Empirically, Transformer-based policies significantly outperform MLP, RNN, and SSM baselines on MuJoCo continuous-control benchmarks, maintaining high returns even with substantial sensor dropout.
Real-world reinforcement learning systems must operate under distributional drift in their observation streams, yet most policy architectures implicitly assume fully observed and noise-free states. We study robustness of Proximal Policy Optimization (PPO) under temporally persistent sensor failures that induce partial observability and representation shift. To respond to this drift, we augment PPO with temporal sequence models, including Transformers and State Space Models (SSMs), to enable policies to infer missing information from history and maintain performance. Under a stochastic sensor failure process, we prove a high-probability bound on infinite-horizon reward degradation that quantifies how robustness depends on policy smoothness and failure persistence. Empirically, on MuJoCo continuous-control benchmarks with severe sensor dropout, we show Transformer-based sequence policies substantially outperform MLP, RNN, and SSM baselines in robustness, maintaining high returns even when large fractions of sensors are unavailable. These results demonstrate that temporal sequence reasoning provides a principled and practical mechanism for reliable operation under observation drift caused by sensor unreliability.