COPAL: Continual Pruning in Large Language Generative Models
This addresses the challenge of computational demands and lack of continual adaptation in large language models for NLP applications, though it is incremental as it builds on existing pruning techniques.
The paper tackles the problem of adapting large language models to new domains without high computational costs, proposing COPAL, a continual pruning algorithm that uses sensitivity analysis to identify weights relevant across datasets, resulting in improved resource efficiency and adaptability compared to baselines.
Adapting pre-trained large language models to different domains in natural language processing requires two key considerations: high computational demands and model's inability to continual adaptation. To simultaneously address both issues, this paper presents COPAL (COntinual Pruning in Adaptive Language settings), an algorithm developed for pruning large language generative models under a continual model adaptation setting. While avoiding resource-heavy finetuning or retraining, our pruning process is guided by the proposed sensitivity analysis. The sensitivity effectively measures model's ability to withstand perturbations introduced by the new dataset and finds model's weights that are relevant for all encountered datasets. As a result, COPAL allows seamless model adaptation to new domains while enhancing the resource efficiency. Our empirical evaluation on a various size of LLMs show that COPAL outperforms baseline models, demonstrating its efficacy in efficiency and adaptability.