What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models
This addresses the low-resource double-bind problem for researchers and practitioners working with small-data language models, but it is incremental as it applies known compression methods to a new context.
The paper investigated the impact of pruning, knowledge distillation, and quantization on AfriBERTa, a low-resource language model, finding that compression techniques significantly improve efficiency and effectiveness, confirming beliefs from large models hold for small-data models.
Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data language models, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models.