mALBERT: Is a Compact Multilingual BERT Model Still Worth It?
This provides a more sustainable and accessible multilingual language model for NLP practitioners, though it is incremental as it adapts an existing compact architecture to a multilingual setting.
The authors tackled the lack of a compact multilingual ALBERT model by releasing a free, ecologically friendly version pre-trained on Wikipedia data, achieving competitive performance in classical NLP tasks compared to larger multilingual PLMs.
Within the current trend of Pretained Language Models (PLM), emerge more and more criticisms about the ethical andecological impact of such models. In this article, considering these critical remarks, we propose to focus on smallermodels, such as compact models like ALBERT, which are more ecologically virtuous than these PLM. However,PLMs enable huge breakthroughs in Natural Language Processing tasks, such as Spoken and Natural LanguageUnderstanding, classification, Question--Answering tasks. PLMs also have the advantage of being multilingual, and,as far as we know, a multilingual version of compact ALBERT models does not exist. Considering these facts, wepropose the free release of the first version of a multilingual compact ALBERT model, pre-trained using Wikipediadata, which complies with the ethical aspect of such a language model. We also evaluate the model against classicalmultilingual PLMs in classical NLP tasks. Finally, this paper proposes a rare study on the subword tokenizationimpact on language performances.