Prototypical Calibration for Few-shot Learning of Language Models
This addresses the problem of unreliable few-shot learning in language models for AI practitioners, offering an incremental improvement over existing methods.
The paper tackles the fragility of in-context learning in GPT-like models by proposing prototypical calibration, which adaptively learns a robust decision boundary for zero- and few-shot classification, resulting in substantial improvements across diverse tasks and enhanced robustness to templates, permutations, and class imbalance.
In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations. In this work, we propose prototypical calibration to adaptively learn a more robust decision boundary for zero- and few-shot classification, instead of greedy decoding. Concretely, our method first adopts Gaussian mixture distribution to estimate the prototypical clusters for all categories. Then we assign each cluster to the corresponding label by solving a weighted bipartite matching problem. Given an example, its prediction is calibrated by the likelihood of prototypical clusters. Experimental results show that prototypical calibration yields a substantial improvement on a diverse set of tasks. Extensive analysis across different scales also indicates that our method calibrates the decision boundary as expected, greatly improving the robustness of GPT to templates, permutations, and class imbalance.