NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning
This addresses calibration issues in ICL for language models, but it is incremental as it builds on existing perturbation methods.
The paper tackles the problem of unsatisfactory performance and under-calibration in In-Context Learning (ICL) by proposing NoisyICL, which perturbs model parameters with random noise to improve accuracy and calibration, as shown in experiments on two models and 12 datasets.
In-Context Learning (ICL) is suffering from unsatisfactory performance and under-calibration due to high prior bias and unfaithful confidence. Some previous works fine-tuned language models for better ICL performance with enormous datasets and computing costs. In this paper, we propose NoisyICL, simply perturbing the model parameters by random noises to strive for better performance and calibration. Our experiments on two models and 12 downstream datasets show that NoisyICL can help ICL produce more accurate predictions. Our further analysis indicates that NoisyICL enables the model to provide more fair predictions, and also with more faithful confidence. Therefore, we believe that NoisyICL is an effective calibration of ICL. Our experimental code is uploaded to Github.