1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
This work addresses the scalability problem in self-supervised RL for researchers and practitioners, showing that deep architectures can enable new capabilities in goal-reaching without demonstrations or rewards, though it is incremental in focusing on depth scaling.
The paper tackles the challenge of scaling self-supervised reinforcement learning by demonstrating that increasing network depth up to 1024 layers significantly boosts performance in unsupervised goal-conditioned tasks, achieving improvements of 2× to 50× in success rates on simulated locomotion and manipulation tasks.
Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance by $2\times$ - $50\times$. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned.