Are Compressed Language Models Less Subgroup Robust?
This addresses the problem of ensuring fairness and robustness in compressed language models for AI practitioners, though it is incremental as it builds on existing compression and robustness research.
The paper investigates how 18 different compression methods affect the subgroup robustness of BERT language models, finding that worst-group performance depends on both model size and compression method, and compression does not always worsen minority subgroup performance.
To reduce the inference cost of large language models, model compression is increasingly used to create smaller scalable models. However, little is known about their robustness to minority subgroups defined by the labels and attributes of a dataset. In this paper, we investigate the effects of 18 different compression methods and settings on the subgroup robustness of BERT language models. We show that worst-group performance does not depend on model size alone, but also on the compression method used. Additionally, we find that model compression does not always worsen the performance on minority subgroups. Altogether, our analysis serves to further research into the subgroup robustness of model compression.