Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation
This addresses the low-resource adaptation problem for language generation tasks, though it is incremental as it builds on existing LoRA merge techniques.
The study tackled the problem of adapting large language models to low-resource language generation tasks by proposing a LoRA merge method that updates and prunes parameters during fine-tuning, achieving significant and consistent improvements in task adaptability over previous methods.
This study proposes a simple yet effective LoRA merge method to achieve LLM adaptation for low-resource language generation tasks. The LoRA merge technique, which integrates multiple LoRA modules trained on different tasks, has gained attention as an effective and efficient approach for adapting LLMs to target tasks. However, previous methods are limited in adaptability as they keep the LoRA parameters frozen. Additionally, the low-resource problem has been out of their scope. We propose a LoRA merge method that updates and prunes LoRA parameters through fine-tuning with minimal target task data, which allows finer-grained adjustments of LoRA parameters and enhancement of task adaptability. Extensive experiments have been conducted taking summarization as a benchmark task. Our datasets cover various domains and multiple languages of English and Japanese. The results confirm that the proposed method achieves significant and consistent improvements in task adaptability over the previous methods.