Quantifying lottery tickets under label noise: accuracy, calibration, and complexity
This work addresses the need for better understanding of pruned models' properties in machine learning, offering insights into their uncertainty representation, but it is incremental as it builds on existing pruning methods.
The paper tackles the problem of characterizing pruned neural networks beyond accuracy, showing that iterative magnitude pruning yields models with comparable sizes that better capture label uncertainty and are less overconfident, with pruned models achieving improved calibration over full networks.
Pruning deep neural networks is a widely used strategy to alleviate the computational burden in machine learning. Overwhelming empirical evidence suggests that pruned models retain very high accuracy even with a tiny fraction of parameters. However, relatively little work has gone into characterising the small pruned networks obtained, beyond a measure of their accuracy. In this paper, we use the sparse double descent approach to identify univocally and characterise pruned models associated with classification tasks. We observe empirically that, for a given task, iterative magnitude pruning (IMP) tends to converge to networks of comparable sizes even when starting from full networks with sizes ranging over orders of magnitude. We analyse the best pruned models in a controlled experimental setup and show that their number of parameters reflects task difficulty and that they are much better than full networks at capturing the true conditional probability distribution of the labels. On real data, we similarly observe that pruned models are less prone to overconfident predictions. Our results suggest that pruned models obtained via IMP not only have advantageous computational properties but also provide a better representation of uncertainty in learning.