Task-oriented Memory-efficient Pruning-Adapter
This work addresses efficiency challenges for users of large language models by combining pruning and adapters, though it appears incremental as it builds on existing approaches.
The paper tackles the trade-off between training and inference efficiency in parameter-efficient learning for large language models by proposing a task-oriented Pruning-Adapter method, which achieves high memory efficiency and speeds up training while maintaining accuracy on GLUE tasks.
The Outstanding performance and growing size of Large Language Models has led to increased attention in parameter efficient learning. The two predominant approaches are Adapters and Pruning. Adapters are to freeze the model and give it a new weight matrix on the side, which can significantly reduce the time and memory of training, but the cost is that the evaluation and testing will increase the time and memory consumption. Pruning is to cut off some weight and re-distribute the remaining weight, which sacrifices the complexity of training at the cost of extremely high memory and training time, making the cost of evaluation and testing relatively low. So efficiency of training and inference can't be obtained in the same time. In this work, we propose a task-oriented Pruning-Adapter method that achieve a high memory efficiency of training and memory, and speeds up training time and ensures no significant decrease in accuracy in GLUE tasks, achieving training and inference efficiency at the same time.