Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizes
This work addresses the challenge of robust performance under varying data sizes for machine learning practitioners, particularly in low-resource settings, representing an incremental advancement by integrating existing methods.
The paper tackles the problem of improving predictive performance for varying data sizes, especially in low-resource settings, by proposing prototypical fine-tuning, which combines large parametric models with non-parametric prototypical networks to automatically adjust model capacity, resulting in significant performance improvements in low-resource scenarios and comparable or better performances in high-resource ones.
In this paper, we move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the number of data points and the model's inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the optimal solution. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.