BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
This addresses the challenge of efficient policy learning from offline data for reinforcement learning practitioners, though it is incremental as it builds on existing batch DRL and imitation learning techniques.
The paper tackles the problem of learning high-performing policies from fixed datasets without environment interaction in batch deep reinforcement learning, proposing BAIL which achieves significantly higher performance and faster computation compared to existing methods on the MuJoCo benchmark.
There has recently been a surge in research in batch Deep Reinforcement Learning (DRL), which aims for learning a high-performing policy from a given dataset without additional interactions with the environment. We propose a new algorithm, Best-Action Imitation Learning (BAIL), which strives for both simplicity and performance. BAIL learns a V function, uses the V function to select actions it believes to be high-performing, and then uses those actions to train a policy network using imitation learning. For the MuJoCo benchmark, we provide a comprehensive experimental study of BAIL, comparing its performance to four other batch Q-learning and imitation-learning schemes for a large variety of batch datasets. Our experiments show that BAIL's performance is much higher than the other schemes, and is also computationally much faster than the batch Q-learning schemes.