SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
This addresses a critical issue for robotic agents in real-world environments where OOD states are common, though it appears incremental as it builds on existing recovery methods.
The paper tackles the problem of robotic agents taking unreliable actions in out-of-distribution (OOD) states by proposing a self-supervised reinforcement learning method for recovery, which substantially improves sample efficiency and restores performance for original tasks.
Robotic agents trained using reinforcement learning have the problem of taking unreliable actions in an out-of-distribution (OOD) state. Agents can easily become OOD in real-world environments because it is almost impossible for them to visit and learn the entire state space during training. Unfortunately, unreliable actions do not ensure that agents perform their original tasks successfully. Therefore, agents should be able to recognize whether they are in OOD states and learn how to return to the learned state distribution rather than continue to take unreliable actions. In this study, we propose a novel method for retraining agents to recover from OOD situations in a self-supervised manner when they fall into OOD states. Our in-depth experimental results demonstrate that our method substantially improves the agent's ability to recover from OOD situations in terms of sample efficiency and restoration of the performance for the original tasks. Moreover, we show that our method can retrain the agent to recover from OOD situations even when in-distribution states are difficult to visit through exploration.