Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedback
This work addresses the need for adaptive robotic systems that can self-improve during operation, offering a step toward more autonomous and resilient agents, though it appears incremental as it builds on existing model-based reinforcement learning methods.
The paper tackles the problem of robotic controllers being unable to adapt to unforeseen changes during deployment by introducing an online continual reinforcement learning framework that uses world model feedback to detect out-of-distribution events and trigger finetuning, validated on continuous control tasks including a quadruped robot in simulation and a real-world model vehicle.
As learning-based robotic controllers are typically trained offline and deployed with fixed parameters, their ability to cope with unforeseen changes during operation is limited. Biologically inspired, this work presents a framework for online Continual Reinforcement Learning that enables automated adaptation during deployment. Building on DreamerV3, a model-based Reinforcement Learning algorithm, the proposed method leverages world model prediction residuals to detect out-of-distribution events and automatically trigger finetuning. Adaptation progress is monitored using both task-level performance signals and internal training metrics, allowing convergence to be assessed without external supervision and domain knowledge. The approach is validated on a variety of contemporary continuous control problems, including a quadruped robot in high-fidelity simulation, and a real-world model vehicle. Relevant metrics and their interpretation are presented and discussed, as well as resulting trade-offs described. The results sketch out how autonomous robotic agents could once move beyond static training regimes toward adaptive systems capable of self-reflection and -improvement during operation, just like their biological counterparts.