One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization
This addresses catastrophic forgetting in multilingual models for code intelligence, offering a parameter-efficient solution for developers and researchers, though it is incremental as it builds on existing adapter methods.
The paper tackles the problem of performance degradation in multilingual fine-tuning for code intelligence tasks by using adapter tuning, which updates only 0.6% of parameters and achieves state-of-the-art results on code search and summarization, with multilingual fine-tuning using 200 samples per language approaching full-dataset performance.
As pre-trained models automate many code intelligence tasks, a widely used paradigm is to fine-tune a model on the task dataset for each programming language. A recent study reported that multilingual fine-tuning benefits a range of tasks and models. However, we find that multilingual fine-tuning leads to performance degradation on recent models UniXcoder and CodeT5. To alleviate the potentially catastrophic forgetting issue in multilingual models, we fix all pre-trained model parameters, insert the parameter-efficient structure adapter, and fine-tune it. Updating only 0.6\% of the overall parameters compared to full-model fine-tuning for each programming language, adapter tuning yields consistent improvements on code search and summarization tasks, achieving state-of-the-art results. In addition, we experimentally show its effectiveness in cross-lingual and low-resource scenarios. Multilingual fine-tuning with 200 samples per programming language approaches the results fine-tuned with the entire dataset on code summarization. Our experiments on three probing tasks show that adapter tuning significantly outperforms full-model fine-tuning and effectively overcomes catastrophic forgetting.