OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models
This provides a practical tool for researchers and practitioners to efficiently adapt large pre-trained models, though it is incremental as it builds on existing delta tuning methods.
The paper tackles the high optimization and storage costs of adapting large pre-trained models by introducing OpenDelta, a plug-and-play library that enables parameter-efficient tuning without modifying backbone model code, achieving compatibility with various models and methods.
The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning", which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.