Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
This work addresses scalability issues in deep reinforcement learning for researchers and practitioners, offering a practical framework to enhance performance without modifying algorithms, though it is incremental as it builds on existing dynamic training methods.
The paper tackled the problem of scaling neural networks in deep reinforcement learning, where larger models often degrade performance, and found that dynamic sparse training strategies provide module-specific benefits that complement architectural improvements, leading to substantial scalability gains across diverse RL algorithms without algorithmic modifications.
Scaling neural networks has driven breakthrough advances in machine learning, yet this paradigm fails in deep reinforcement learning (DRL), where larger models often degrade performance due to unique optimization pathologies such as plasticity loss. While recent works show that dynamically adapting network topology during training can mitigate these issues, existing studies have three critical limitations: (1) applying uniform dynamic training strategies across all modules despite encoder, critic, and actor following distinct learning paradigms, (2) focusing evaluation on basic architectures without clarifying the relative importance and interaction between dynamic training and architectural improvements, and (3) lacking systematic comparison between different dynamic approaches including sparse-to-sparse, dense-to-sparse, and sparse-to-dense. Through comprehensive investigation across modules and architectures, we reveal that dynamic sparse training strategies provide module-specific benefits that complement the primary scalability foundation established by architectural improvements. We finally distill these insights into Module-Specific Training (MST), a practical framework that further exploits the benefits of architectural improvements and demonstrates substantial scalability gains across diverse RL algorithms without algorithmic modifications.