RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models
This addresses a bottleneck in fine-tuning efficiency for encoder-decoder PLMs, offering a plug-and-play solution with broad applicability, though it is incremental as it builds on existing fine-tuning paradigms.
The paper tackles the discrepancy between encoder representations and decoder inputs in fine-tuned pre-trained language models by introducing RepCali, a representation calibration method in latent space, which improves performance across 25 models and 8 tasks.
Fine-tuning pre-trained language models (PLMs) has become a dominant paradigm in applying PLMs to downstream tasks. However, with limited fine-tuning, PLMs still struggle with the discrepancies between the representation obtained from the PLMs' encoder and the optimal input to the PLMs' decoder. This paper tackles this challenge by learning to calibrate the representation of PLMs in the latent space. In the proposed representation calibration method (RepCali), we integrate a specific calibration block to the latent space after the encoder and use the calibrated output as the decoder input. The merits of the proposed RepCali include its universality to all PLMs with encoder-decoder architectures, its plug-and-play nature, and ease of implementation. Extensive experiments on 25 PLM-based models across 8 tasks (including both English and Chinese datasets) demonstrate that the proposed RepCali offers desirable enhancements to PLMs (including LLMs) and significantly improves the performance of downstream tasks. Comparison experiments across 4 benchmark tasks indicate that RepCali is superior to the representative fine-tuning baselines.