Ahead-of-Time P-Tuning
This provides a practical solution for real-world applications by enabling multi-task inference with a single backbone model, though it is incremental as it builds on existing parameter-efficient fine-tuning methods.
The paper tackles efficient fine-tuning of pre-trained language models by proposing Ahead-of-Time P-Tuning, which adds input-dependent bias before Transformer layers, and shows it outperforms BitFit and is comparable or better than other baselines on GLUE and SuperGLUE benchmarks with RoBERTa and DeBERTa models, while introducing negligible inference overhead.
In this paper, we propose Ahead-of-Time (AoT) P-Tuning, a novel parameter-efficient fine-tuning method for pre-trained Language Models (LMs) that adds input-dependent bias before each Transformer layer. We evaluate AoT P-Tuning on GLUE and SuperGLUE benchmarking datasets using RoBERTa and DeBERTa models, showing that it outperforms BitFit and is comparable or better than other baseline methods for efficient fine-tuning. Additionally, we assess the inference overhead of AoT P-Tuning and demonstrate that it introduces negligible overhead compared to established baseline methods. Our method enables multi-task inference with a single backbone LM, making it a practical solution for real-world applications.