Eau De $Q$-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning
This work addresses cost-sensitive or hardware-limited reinforcement learning applications by enabling adaptive sparsity without manual tuning, though it is incremental as it builds on existing dense-to-sparse methods.
The paper tackled the problem of dense-to-sparse methods in deep reinforcement learning relying on hand-designed sparsity schedules and hyperparameter tuning, by proposing Eau De Q-Network (EauDeQN), which adaptively increases sparsity at the agent's learning pace and achieves high sparsity levels while maintaining performance on Atari 2600 and MuJoCo benchmarks.
Recent works have successfully demonstrated that sparse deep reinforcement learning agents can be competitive against their dense counterparts. This opens up opportunities for reinforcement learning applications in fields where inference time and memory requirements are cost-sensitive or limited by hardware. Until now, dense-to-sparse methods have relied on hand-designed sparsity schedules that are not synchronized with the agent's learning pace. Crucially, the final sparsity level is chosen as a hyperparameter, which requires careful tuning as setting it too high might lead to poor performances. In this work, we address these shortcomings by crafting a dense-to-sparse algorithm that we name Eau De $Q$-Network (EauDeQN). To increase sparsity at the agent's learning pace, we consider multiple online networks with different sparsity levels, where each online network is trained from a shared target network. At each target update, the online network with the smallest loss is chosen as the next target network, while the other networks are replaced by a pruned version of the chosen network. We evaluate the proposed approach on the Atari $2600$ benchmark and the MuJoCo physics simulator, showing that EauDeQN reaches high sparsity levels while keeping performances high.