Pretraining & Reinforcement Learning: Sharpening the Axe Before Cutting the Tree
This work addresses the problem of optimizing pretraining strategies for RL practitioners, but it is incremental as it builds on existing pretraining techniques.
The study evaluated pretraining's effectiveness in deep reinforcement learning, finding that pretraining with relevant datasets reduces training time and improves performance after 80k steps, while irrelevant datasets render it ineffective.
Pretraining is a common technique in deep learning for increasing performance and reducing training time, with promising experimental results in deep reinforcement learning (RL). However, pretraining requires a relevant dataset for training. In this work, we evaluate the effectiveness of pretraining for RL tasks, with and without distracting backgrounds, using both large, publicly available datasets with minimal relevance, as well as case-by-case generated datasets labeled via self-supervision. Results suggest filters learned during training on less relevant datasets render pretraining ineffective, while filters learned during training on the in-distribution datasets reliably reduce RL training time and improve performance after 80k RL training steps. We further investigate, given a limited number of environment steps, how to optimally divide the available steps into pretraining and RL training to maximize RL performance. Our code is available on GitHub