Comparative Study of Pre-Trained BERT Models for Code-Mixed Hindi-English Data
This work addresses the challenge of natural language processing for code-mixed Hindi-English data, which is common on social media but often poorly handled by existing models, though it is incremental as it focuses on comparative analysis rather than introducing a new method.
The study tackled the problem of processing low-resource Hindi-English code-mixed text by comparing pre-trained BERT models, achieving state-of-the-art results with HingBERT-based models that significantly outperformed vanilla BERT models on tasks like sentiment analysis and hate speech identification.
The term "Code Mixed" refers to the use of more than one language in the same text. This phenomenon is predominantly observed on social media platforms, with an increasing amount of adaptation as time goes on. It is critical to detect foreign elements in a language and process them correctly, as a considerable number of individuals are using code-mixed languages that could not be comprehended by understanding one of those languages. In this work, we focus on low-resource Hindi-English code-mixed language and enhancing the performance of different code-mixed natural language processing tasks such as sentiment analysis, emotion recognition, and hate speech identification. We perform a comparative analysis of different Transformer-based language Models pre-trained using unsupervised approaches. We have included the code-mixed models like HingBERT, HingRoBERTa, HingRoBERTa-Mixed, mBERT, and non-code-mixed models like AlBERT, BERT, and RoBERTa for comparative analysis of code-mixed Hindi-English downstream tasks. We report state-of-the-art results on respective datasets using HingBERT-based models which are specifically pre-trained on real code-mixed text. Our HingBERT-based models provide significant improvements thus highlighting the poor performance of vanilla BERT models on code-mixed text.