Applying SoftTriple Loss for Supervised Language Model Fine Tuning
This is an incremental improvement for fine-tuning language models, particularly beneficial for small datasets.
The paper tackled improving classification performance in fine-tuning pre-trained language models by introducing TripleEntropy, a new loss function based on cross-entropy and SoftTriple loss, resulting in gains of 0.02% to 2.29% over a robust RoBERTa baseline, with higher gains for smaller datasets (e.g., 0.78% for small datasets).
We introduce a new loss function TripleEntropy, to improve classification performance for fine-tuning general knowledge pre-trained language models based on cross-entropy and SoftTriple loss. This loss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss by about (0.02% - 2.29%). Thorough tests on popular datasets indicate a steady gain. The fewer samples in the training dataset, the higher gain -- thus, for small-sized dataset it is 0.78%, for medium-sized -- 0.86% for large -- 0.20% and for extra-large 0.04%.