Mono vs Multilingual Transformer-based Models: a Comparison across Several Language Tasks
This work provides insights for researchers and practitioners in natural language processing, particularly for Portuguese, but is incremental as it applies existing methods to new data.
The study compared monolingual and multilingual transformer-based models (BERT and ALBERT) across multiple Portuguese language tasks, finding that both achieved state-of-the-art results with minimal advantage for monolingual training.
BERT (Bidirectional Encoder Representations from Transformers) and ALBERT (A Lite BERT) are methods for pre-training language models which can later be fine-tuned for a variety of Natural Language Understanding tasks. These methods have been applied to a number of such tasks (mostly in English), achieving results that outperform the state-of-the-art. In this paper, our contribution is twofold. First, we make available our trained BERT and Albert model for Portuguese. Second, we compare our monolingual and the standard multilingual models using experiments in semantic textual similarity, recognizing textual entailment, textual category classification, sentiment analysis, offensive comment detection, and fake news detection, to assess the effectiveness of the generated language representations. The results suggest that both monolingual and multilingual models are able to achieve state-of-the-art and the advantage of training a single language model, if any, is small.