SPLASH! Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations
This addresses a critical shortcoming in inverse reinforcement learning for producing field-ready robotic agents in challenging scenarios like long-horizon and adversarial tasks, though it appears incremental by extending existing methods to new settings.
The paper tackles the problem of learning complex robotic tasks from suboptimal demonstrations in long-horizon and adversarial settings, where existing inverse reinforcement learning methods fall short, and shows that SPLASH significantly outperforms state-of-the-art methods in reward learning from such demonstrations, with empirical validation on a maritime capture-the-flag task and sim-to-real experiments.
Inverse Reinforcement Learning (IRL) presents a powerful paradigm for learning complex robotic tasks from human demonstrations. However, most approaches make the assumption that expert demonstrations are available, which is often not the case. Those that allow for suboptimality in the demonstrations are not designed for long-horizon goals or adversarial tasks. Many desirable robot capabilities fall into one or both of these categories, thus highlighting a critical shortcoming in the ability of IRL to produce field-ready robotic agents. We introduce Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations (SPLASH), which advances the state-of-the-art in learning from suboptimal demonstrations to long-horizon and adversarial settings. We empirically validate SPLASH on a maritime capture-the-flag task in simulation, and demonstrate real-world applicability with sim-to-real translation experiments on autonomous unmanned surface vehicles. We show that our proposed methods allow SPLASH to significantly outperform the state-of-the-art in reward learning from suboptimal demonstrations.