From Universal Language Model to Downstream Task: Improving RoBERTa-Based Vietnamese Hate Speech Detection
This work addresses the challenge of fine-tuning language models for hate speech detection in Vietnamese, which is an incremental improvement for NLP applications in low-resource languages.
The paper tackled the problem of adapting a pre-trained language model to a specific downstream task, Vietnamese hate speech detection, by proposing a pipeline that includes re-training on masked language modeling and employing training techniques like layer freezing, achieving a new state-of-the-art F1 score of 0.7221.
Natural language processing is a fast-growing field of artificial intelligence. Since the Transformer was introduced by Google in 2017, a large number of language models such as BERT, GPT, and ELMo have been inspired by this architecture. These models were trained on huge datasets and achieved state-of-the-art results on natural language understanding. However, fine-tuning a pre-trained language model on much smaller datasets for downstream tasks requires a carefully-designed pipeline to mitigate problems of the datasets such as lack of training data and imbalanced data. In this paper, we propose a pipeline to adapt the general-purpose RoBERTa language model to a specific text classification task: Vietnamese Hate Speech Detection. We first tune the PhoBERT on our dataset by re-training the model on the Masked Language Model task; then, we employ its encoder for text classification. In order to preserve pre-trained weights while learning new feature representations, we further utilize different training techniques: layer freezing, block-wise learning rate, and label smoothing. Our experiments proved that our proposed pipeline boosts the performance significantly, achieving a new state-of-the-art on Vietnamese Hate Speech Detection campaign with 0.7221 F1 score.