Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
This addresses the inefficiency of deploying multiple large models for different tasks, particularly beneficial for resource-constrained environments like mobile devices, though it is incremental as it builds on existing parameter-efficient transfer learning methods.
The paper tackles the problem of fine-tuning large generative language models for multiple downstream tasks without requiring separate large models for each, which is costly in low-memory scenarios, and shows that using only 2-3% additional parameters per task can maintain or improve performance compared to full fine-tuning.
Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results. However, this comes with the cost of having a single, large model for each task, which is not ideal in low-memory/power scenarios (e.g., mobile). In this paper, we propose an effective way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pre-trained model. The experiments on five diverse language generation tasks show that by just using an additional 2-3% parameters for each task, our model can maintain or even improve the performance of fine-tuning the whole model.