Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
This addresses the challenge of efficient learning from few demonstrations in robotics, though it is incremental as it builds on existing methods like Q-filter.
The paper tackled the problem of when to imitate demonstrations versus following an agent's own policy in reinforcement learning with sparse rewards, proposing SPReD, which uses ensemble methods to model Q-value distributions and apply continuous, uncertainty-proportional regularisation, achieving up to a factor of 14 improvement over existing approaches in complex robotics tasks.
In reinforcement learning with sparse rewards, demonstrations can accelerate learning, but determining when to imitate them remains challenging. We propose Smooth Policy Regularisation from Demonstrations (SPReD), a framework that addresses the fundamental question: when should an agent imitate a demonstration versus follow its own policy? SPReD uses ensemble methods to explicitly model Q-value distributions for both demonstration and policy actions, quantifying uncertainty for comparisons. We develop two complementary uncertainty-aware methods: a probabilistic approach estimating the likelihood of demonstration superiority, and an advantage-based approach scaling imitation by statistical significance. Unlike prevailing methods (e.g. Q-filter) that make binary imitation decisions, SPReD applies continuous, uncertainty-proportional regularisation weights, reducing gradient variance during training. Despite its computational simplicity, SPReD achieves remarkable gains in experiments across eight robotics tasks, outperforming existing approaches by up to a factor of 14 in complex tasks while maintaining robustness to demonstration quality and quantity. Our code is available at https://github.com/YujieZhu7/SPReD.