Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm
This work addresses fairness and performance issues in neural network pruning for underrepresented classes, offering an incremental improvement to mitigate recall distortions.
The authors studied how neural network pruning disproportionately affects recall for underrepresented classes, hypothesizing an intensification effect where pruning worsens recall for classes with recall below accuracy and improves it for those above. They proposed a new pruning algorithm that reduces this effect, observing that intensification is less severe with their method but more pronounced with harder tasks, simpler models, and higher pruning ratios, while moderate pruning can have a corrective effect.
Pruning techniques have been successfully used in neural networks to trade accuracy for sparsity. However, the impact of network pruning is not uniform: prior work has shown that the recall for underrepresented classes in a dataset may be more negatively affected. In this work, we study such relative distortions in recall by hypothesizing an intensification effect that is inherent to the model. Namely, that pruning makes recall relatively worse for a class with recall below accuracy and, conversely, that it makes recall relatively better for a class with recall above accuracy. In addition, we propose a new pruning algorithm aimed at attenuating such effect. Through statistical analysis, we have observed that intensification is less severe with our algorithm but nevertheless more pronounced with relatively more difficult tasks, less complex models, and higher pruning ratios. More surprisingly, we conversely observe a de-intensification effect with lower pruning ratios, which indicates that moderate pruning may have a corrective effect to such distortions.