TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations
This addresses the challenge of noisy human or VLM preference labels for reinforcement learning in robotics, representing a strong specific gain.
The paper tackles the problem of noisy preference feedback in preference-based reinforcement learning by proposing TREND, a framework that integrates few-shot expert demonstrations with a tri-teaching strategy for noise mitigation, achieving up to 90% success rates on robotic manipulation tasks with noise levels as high as 40%.
Preference feedback collected by human or VLM annotators is often noisy, presenting a significant challenge for preference-based reinforcement learning that relies on accurate preference labels. To address this challenge, we propose TREND, a novel framework that integrates few-shot expert demonstrations with a tri-teaching strategy for effective noise mitigation. Our method trains three reward models simultaneously, where each model views its small-loss preference pairs as useful knowledge and teaches such useful pairs to its peer network for updating the parameters. Remarkably, our approach requires as few as one to three expert demonstrations to achieve high performance. We evaluate TREND on various robotic manipulation tasks, achieving up to 90% success rates even with noise levels as high as 40%, highlighting its effective robustness in handling noisy preference feedback. Project page: https://shuaiyihuang.github.io/publications/TREND.