DiSA-IQL: Offline Reinforcement Learning for Robust Soft Robot Control under Distribution Shifts
This addresses robust control for soft snake robots in unseen scenarios, but it is incremental as it extends an existing method (IQL) with a specific modification.
The paper tackled the problem of offline reinforcement learning for soft robot control under distribution shifts by proposing DiSA-IQL, which penalizes unreliable state-action pairs, resulting in higher success rates and improved robustness compared to baselines like BC, CQL, and vanilla IQL.
Soft snake robots offer remarkable flexibility and adaptability in complex environments, yet their control remains challenging due to highly nonlinear dynamics. Existing model-based and bio-inspired controllers rely on simplified assumptions that limit performance. Deep reinforcement learning (DRL) has recently emerged as a promising alternative, but online training is often impractical because of costly and potentially damaging real-world interactions. Offline RL provides a safer option by leveraging pre-collected datasets, but it suffers from distribution shift, which degrades generalization to unseen scenarios. To overcome this challenge, we propose DiSA-IQL (Distribution-Shift-Aware Implicit Q-Learning), an extension of IQL that incorporates robustness modulation by penalizing unreliable state-action pairs to mitigate distribution shift. We evaluate DiSA-IQL on goal-reaching tasks across two settings: in-distribution and out-of-distribution evaluation. Simulation results show that DiSA-IQL consistently outperforms baseline models, including Behavior Cloning (BC), Conservative Q-Learning (CQL), and vanilla IQL, achieving higher success rates, smoother trajectories, and improved robustness. The codes are open-sourced to support reproducibility and to facilitate further research in offline RL for soft robot control.