Bootstrapping Adaptive Human-Machine Interfaces with Offline Reinforcement Learning
This addresses the challenge of limited data collection from individual users in adaptive interfaces, which is a problem for users relying on noisy input channels like eye gaze control, though it appears incremental as it builds on existing human-in-the-loop and reinforcement learning approaches.
The paper tackles the problem of training adaptive human-machine interfaces for sequential decision-making tasks with noisy, high-dimensional command signals, such as from brain-computer interfaces, by proposing a reinforcement learning algorithm that combines offline pre-training and online fine-tuning. The result shows that their method enables successful goal navigation more often than a baseline directional interface in a user study with 12 participants, and it also improves performance in simulated tasks like Sawyer pushing and Lunar Lander.
Adaptive interfaces can help users perform sequential decision-making tasks like robotic teleoperation given noisy, high-dimensional command signals (e.g., from a brain-computer interface). Recent advances in human-in-the-loop machine learning enable such systems to improve by interacting with users, but tend to be limited by the amount of data that they can collect from individual users in practice. In this paper, we propose a reinforcement learning algorithm to address this by training an interface to map raw command signals to actions using a combination of offline pre-training and online fine-tuning. To address the challenges posed by noisy command signals and sparse rewards, we develop a novel method for representing and inferring the user's long-term intent for a given trajectory. We primarily evaluate our method's ability to assist users who can only communicate through noisy, high-dimensional input channels through a user study in which 12 participants performed a simulated navigation task by using their eye gaze to modulate a 128-dimensional command signal from their webcam. The results show that our method enables successful goal navigation more often than a baseline directional interface, by learning to denoise user commands signals and provide shared autonomy assistance. We further evaluate on a simulated Sawyer pushing task with eye gaze control, and the Lunar Lander game with simulated user commands, and find that our method improves over baseline interfaces in these domains as well. Extensive ablation experiments with simulated user commands empirically motivate each component of our method.