Automating Control of Overestimation Bias for Reinforcement Learning
This work addresses the need for environment-specific tuning in reinforcement learning algorithms, offering an incremental improvement by automating hyperparameter selection.
The paper tackled the problem of automating hyperparameter selection for overestimation bias control in off-policy reinforcement learning, resulting in a significant reduction in the number of interactions needed while maintaining performance.
Overestimation bias control techniques are used by the majority of high-performing off-policy reinforcement learning algorithms. However, most of these techniques rely on pre-defined bias correction policies that are either not flexible enough or require environment-specific tuning of hyperparameters. In this work, we present a general data-driven approach for the automatic selection of bias control hyperparameters. We demonstrate its effectiveness on three algorithms: Truncated Quantile Critics, Weighted Delayed DDPG, and Maxmin Q-learning. The proposed technique eliminates the need for an extensive hyperparameter search. We show that it leads to a significant reduction of the actual number of interactions while preserving the performance.