Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
This addresses the issue of neuron inactivity in deep RL for improved learning adaptability, representing an incremental advance in method.
The paper tackled the problem of neuronal activity loss in deep reinforcement learning agents by shifting from activation-based to gradient-based metrics, introducing GraMa to quantify neuron-level learning capacity, and showing that resetting neurons guided by GraMa improves performance across multiple algorithms and benchmarks.
Deep reinforcement learning (RL) agents frequently suffer from neuronal activity loss, which impairs their ability to adapt to new data and learn continually. A common method to quantify and address this issue is the tau-dormant neuron ratio, which uses activation statistics to measure the expressive ability of neurons. While effective for simple MLP-based agents, this approach loses statistical power in more complex architectures. To address this, we argue that in advanced RL agents, maintaining a neuron's learning capacity, its ability to adapt via gradient updates, is more critical than preserving its expressive ability. Based on this insight, we shift the statistical objective from activations to gradients, and introduce GraMa (Gradient Magnitude Neural Activity Metric), a lightweight, architecture-agnostic metric for quantifying neuron-level learning capacity. We show that GraMa effectively reveals persistent neuron inactivity across diverse architectures, including residual networks, diffusion models, and agents with varied activation functions. Moreover, resetting neurons guided by GraMa (ReGraMa) consistently improves learning performance across multiple deep RL algorithms and benchmarks, such as MuJoCo and the DeepMind Control Suite.