Offline Reinforcement Learning with Behavioral Supervisor Tuning
This addresses a key bottleneck for practitioners in offline RL by eliminating the need for per-dataset tuning, though it is incremental as it builds on existing methods like TD3.
The paper tackles the problem of offline reinforcement learning requiring extensive per-dataset hyperparameter tuning, which is cumbersome and limits practical adoption, by introducing TD3-BST, an algorithm that uses an uncertainty model to guide policy actions within dataset support, achieving the best performance across challenging benchmarks without such tuning.
Offline reinforcement learning (RL) algorithms are applied to learn performant, well-generalizing policies when provided with a static dataset of interactions. Many recent approaches to offline RL have seen substantial success, but with one key caveat: they demand substantial per-dataset hyperparameter tuning to achieve reported performance, which requires policy rollouts in the environment to evaluate; this can rapidly become cumbersome. Furthermore, substantial tuning requirements can hamper the adoption of these algorithms in practical domains. In this paper, we present TD3 with Behavioral Supervisor Tuning (TD3-BST), an algorithm that trains an uncertainty model and uses it to guide the policy to select actions within the dataset support. TD3-BST can learn more effective policies from offline datasets compared to previous methods and achieves the best performance across challenging benchmarks without requiring per-dataset tuning.