DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback
This addresses the problem of slow learning in RL for robotics by incorporating human feedback, though it appears incremental as it builds on existing methods.
The study tackled the challenge of exploration in reinforcement learning by introducing a human-in-the-loop scheme with real-time feedback, resulting in DQN-TAMER agents outperforming baselines in simulated Maze and Taxi environments.
Exploration has been one of the greatest challenges in reinforcement learning (RL), which is a large obstacle in the application of RL to robotics. Even with state-of-the-art RL algorithms, building a well-learned agent often requires too many trials, mainly due to the difficulty of matching its actions with rewards in the distant future. A remedy for this is to train an agent with real-time feedback from a human observer who immediately gives rewards for some actions. This study tackles a series of challenges for introducing such a human-in-the-loop RL scheme. The first contribution of this work is our experiments with a precisely modeled human observer: binary, delay, stochasticity, unsustainability, and natural reaction. We also propose an RL method called DQN-TAMER, which efficiently uses both human feedback and distant rewards. We find that DQN-TAMER agents outperform their baselines in Maze and Taxi simulated environments. Furthermore, we demonstrate a real-world human-in-the-loop RL application where a camera automatically recognizes a user's facial expressions as feedback to the agent while the agent explores a maze.