Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback
This addresses the problem of high cognitive load and unnatural interaction in human-robot alignment for robotics researchers, though it is incremental over existing RLHF methods.
The paper tackles the challenge of sparse rewards in reinforcement learning by proposing a framework that uses EEG signals (error-related potentials) to provide continuous, implicit human feedback, eliminating the need for explicit user input. In a simulated pick-and-place task with a robotic arm, agents trained with this EEG feedback achieved performance comparable to those trained with dense, manually designed rewards.
Conventional reinforcement learning (RL) ap proaches often struggle to learn effective policies under sparse reward conditions, necessitating the manual design of complex, task-specific reward functions. To address this limitation, rein forcement learning from human feedback (RLHF) has emerged as a promising strategy that complements hand-crafted rewards with human-derived evaluation signals. However, most existing RLHF methods depend on explicit feedback mechanisms such as button presses or preference labels, which disrupt the natural interaction process and impose a substantial cognitive load on the user. We propose a novel reinforcement learning from implicit human feedback (RLIHF) framework that utilizes non-invasive electroencephalography (EEG) signals, specifically error-related potentials (ErrPs), to provide continuous, implicit feedback without requiring explicit user intervention. The proposed method adopts a pre-trained decoder to transform raw EEG signals into probabilistic reward components, en abling effective policy learning even in the presence of sparse external rewards. We evaluate our approach in a simulation environment built on the MuJoCo physics engine, using a Kinova Gen2 robotic arm to perform a complex pick-and-place task that requires avoiding obstacles while manipulating target objects. The results show that agents trained with decoded EEG feedback achieve performance comparable to those trained with dense, manually designed rewards. These findings validate the potential of using implicit neural feedback for scalable and human-aligned reinforcement learning in interactive robotics.