Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examples
This work addresses the challenge of learning from task examples without demonstrations, which is incremental but could benefit robotics and AI by reducing reliance on expert data.
The paper tackles the problem of sample inefficiency in example-based control for reinforcement learning by introducing VPACE, which improves exploration using auxiliary tasks and a value penalty. The approach shows substantial efficiency gains in simulated and real robotic environments, with preliminary results suggesting it may outperform methods using full trajectories or sparse rewards.
Common approaches to providing feedback in reinforcement learning are the use of hand-crafted rewards or full-trajectory expert demonstrations. Alternatively, one can use examples of completed tasks, but such an approach can be extremely sample inefficient. We introduce value-penalized auxiliary control from examples (VPACE), an algorithm that significantly improves exploration in example-based control by adding examples of simple auxiliary tasks and an above-success-level value penalty. Across both simulated and real robotic environments, we show that our approach substantially improves learning efficiency for challenging tasks, while maintaining bounded value estimates. Preliminary results also suggest that VPACE may learn more efficiently than the more common approaches of using full trajectories or true sparse rewards. Project site: https://papers.starslab.ca/vpace/.