Simplifying Deep Reinforcement Learning via Self-Supervision
This work addresses the problem of instability in deep reinforcement learning for researchers and practitioners by offering a simpler, supervised learning-based approach.
The paper tackles the challenge of training deep reinforcement learning agents stably by proposing Self-Supervised Reinforcement Learning (SSRL), which uses supervised regression without policy gradients or value estimation, resulting in competitive performance with more stable and faster execution compared to contemporary algorithms.
Supervised regression to demonstrations has been demonstrated to be a stable way to train deep policy networks. We are motivated to study how we can take full advantage of supervised loss functions for stably training deep reinforcement learning agents. This is a challenging task because it is unclear how the training data could be collected to enable policy improvement. In this work, we propose Self-Supervised Reinforcement Learning (SSRL), a simple algorithm that optimizes policies with purely supervised losses. We demonstrate that, without policy gradient or value estimation, an iterative procedure of ``labeling" data and supervised regression is sufficient to drive stable policy improvement. By selecting and imitating trajectories with high episodic rewards, SSRL is surprisingly competitive to contemporary algorithms with more stable performance and less running time, showing the potential of solving reinforcement learning with supervised learning techniques. The code is available at https://github.com/daochenzha/SSRL