Pruning has a disparate impact on model accuracy
This addresses fairness issues in machine learning for practitioners using pruning, highlighting a critical but often overlooked problem in model optimization.
The paper demonstrates that network pruning, a common model compression technique, can create or worsen disparate impacts on model accuracy across different groups, attributing this to differences in gradient norms and distance to decision boundary, and proposes a solution to mitigate these effects.
Network pruning is a widely-used compression technique that is able to significantly scale down overparameterized models with minimal loss of accuracy. This paper shows that pruning may create or exacerbate disparate impacts. The paper sheds light on the factors to cause such disparities, suggesting differences in gradient norms and distance to decision boundary across groups to be responsible for this critical issue. It analyzes these factors in detail, providing both theoretical and empirical support, and proposes a simple, yet effective, solution that mitigates the disparate impacts caused by pruning.