Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting
This addresses the challenge of balancing under- and overfitting in model-based RL, eliminating manual hyperparameter setting for practitioners, though it is incremental as it builds on existing methods like DreamerV2.
The paper tackles the problem of overfitting in world model learning for reinforcement learning by proposing a method to dynamically adjust the update-to-data ratio based on under- and overfitting detection, achieving competitive performance with reduced hyperparameter tuning on benchmarks like DeepMind Control Suite and Atari 100k.
Early stopping based on the validation set performance is a popular approach to find the right balance between under- and overfitting in the context of supervised learning. However, in reinforcement learning, even for supervised sub-problems such as world model learning, early stopping is not applicable as the dataset is continually evolving. As a solution, we propose a new general method that dynamically adjusts the update to data (UTD) ratio during training based on under- and overfitting detection on a small subset of the continuously collected experience not used for training. We apply our method to DreamerV2, a state-of-the-art model-based reinforcement learning algorithm, and evaluate it on the DeepMind Control Suite and the Atari $100$k benchmark. The results demonstrate that one can better balance under- and overestimation by adjusting the UTD ratio with our approach compared to the default setting in DreamerV2 and that it is competitive with an extensive hyperparameter search which is not feasible for many applications. Our method eliminates the need to set the UTD hyperparameter by hand and even leads to a higher robustness with regard to other learning-related hyperparameters further reducing the amount of necessary tuning.