Parameter-Efficient Finetuning of Transformers for Source Code
This work addresses the deployment challenge of large pretrained Transformers for software development tools, though it is incremental as it applies existing methods to a new domain.
The study tested parameter-efficient fine-tuning methods (adapters and LoRA) on four code-processing tasks, finding they perform comparably or better than full fine-tuning in understanding tasks but underperform in generative tasks.
Pretrained Transformers achieve state-of-the-art performance in various code-processing tasks but may be too large to be deployed. As software development tools often incorporate modules for various purposes which may potentially use a single instance of the pretrained model, it appears relevant to utilize parameter-efficient fine-tuning for the pretrained models of code. In this work, we test two widely used approaches, adapters and LoRA, which were initially tested on NLP tasks, on four code-processing tasks. We find that though the efficient fine-tuning approaches may achieve comparable or higher performance than the standard, full, fine-tuning in code understanding tasks, they underperform full fine-tuning in code-generative tasks. These results underline the importance of testing efficient fine-tuning approaches on other domains than NLP and motivate future research in efficient fine-tuning for source code.