Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning
This work addresses the need for emotion-aware text representations in NLP applications, offering an incremental improvement over existing methods.
The authors tackled the problem of inducing emotion awareness into pre-trained language models like BERT and RoBERTa using supervised contrastive learning, resulting in models (BERTEmo and RoBERTaEmo) that improved F1-scores by about 1% in sentiment analysis and sarcasm detection tasks and showed significant gains in few-shot learning.
We present a novel retrofitting method to induce emotion aspects into pre-trained language models (PLMs) such as BERT and RoBERTa. Our method updates pre-trained network weights using contrastive learning so that the text fragments exhibiting similar emotions are encoded nearby in the representation space, and the fragments with different emotion content are pushed apart. While doing so, it also ensures that the linguistic knowledge already present in PLMs is not inadvertently perturbed. The language models retrofitted by our method, i.e., BERTEmo and RoBERTaEmo, produce emotion-aware text representations, as evaluated through different clustering and retrieval metrics. For the downstream tasks on sentiment analysis and sarcasm detection, they perform better than their pre-trained counterparts (about 1% improvement in F1-score) and other existing approaches. Additionally, a more significant boost in performance is observed for the retrofitted models over pre-trained ones in few-shot learning setting.