Empirical Studies of Parameter Efficient Methods for Large Language Models of Code and Knowledge Transfer to R
This work addresses the challenge of efficiently fine-tuning large language models for code tasks in unseen, low-resource languages like R, which is incremental as it applies existing PEFT methods to a new domain.
The study evaluated Parameter Efficient Fine-Tuning (PEFT) methods like LoRA, Compacter, and IA^3 for large language models on code summarization and generation, focusing on knowledge transfer to the low-resource programming language R, finding that LoRA consistently outperformed others while Compacter offered resource efficiency with minimal trade-offs.
Parameter Efficient Fine-Tuning (PEFT) methods are proposed as an alternative fine-tuning approach for Large Language Models (LLM) to minimize high training costs. While prior research demonstrates the effectiveness of PEFT methods in knowledge transfer using smaller language models, their application to larger LLMs, particularly in low-resource and unseen programming languages such as R, remains under-explored. In this work, we evaluate PEFT methods, LoRA, Compacter, and IA^3 on LLMs for code summarization and generation, with a particular emphasis on knowledge transfer to R as an unseen under-explored target language. Our experiments reveal that LoRA consistently outperforms Compacter and IA^3 in all settings, while Compacter offers significant resource efficiency with minimal performance trade-offs. Additionally, we find that the number of trainable parameters has a greater influence on the functional accuracy of the generated code than PEFT architecture. Our study can direct future research in developing code intelligent tasks for unseen languages including R, as well as the choice of PEFT methods for knowledge transfer, especially when balancing the computational cost and performance.