Budgeted Broadcast: An Activity-Dependent Pruning Rule for Neural Network Efficiency
This work addresses the need for more efficient neural networks by introducing a novel pruning rule that enhances representation diversity and efficiency, though it appears incremental as it builds on existing pruning methods.
The paper tackles the problem of neural network pruning by proposing Budgeted Broadcast (BB), a method that uses local traffic budgets based on unit activity and fan-out to prune connections, resulting in improved accuracy at matched sparsity across various architectures and achieving state-of-the-art performance on electron microscopy images with specific metrics like F1 and PR-AUC.
Most pruning methods remove parameters ranked by impact on loss (e.g., magnitude or gradient). We propose Budgeted Broadcast (BB), which gives each unit a local traffic budget (the product of its long-term on-rate $a_i$ and fan-out $k_i$). A constrained-entropy analysis shows that maximizing coding entropy under a global traffic budget yields a selectivity-audience balance, $\log\frac{1-a_i}{a_i}=βk_i$. BB enforces this balance with simple local actuators that prune either fan-in (to lower activity) or fan-out (to reduce broadcast). In practice, BB increases coding entropy and decorrelation and improves accuracy at matched sparsity across Transformers for ASR, ResNets for face identification, and 3D U-Nets for synapse prediction, sometimes exceeding dense baselines. On electron microscopy images, it attains state-of-the-art F1 and PR-AUC under our evaluation protocol. BB is easy to integrate and suggests a path toward learning more diverse and efficient representations.