Breaking Character: Are Subwords Good Enough for MRLs After All?
This work addresses the problem of tokenization effectiveness for morphologically-rich languages, providing evidence that subword tokenization remains a viable approach, though it is incremental in refining prior claims.
The study revisited the claim that subword tokenization is inadequate for morphologically-rich languages (MRLs) by pretraining a character-based BERT model (TavBERT) and comparing it to subword-based models on Hebrew, Turkish, and Arabic tasks. Results showed TavBERT had mild improvements on morphological tasks like POS tagging, but subword-based models achieved significantly higher performance on semantic tasks such as named entity recognition and question answering.
Large pretrained language models (PLMs) typically tokenize the input string into contiguous subwords before any pretraining or inference. However, previous studies have claimed that this form of subword tokenization is inadequate for processing morphologically-rich languages (MRLs). We revisit this hypothesis by pretraining a BERT-style masked language model over character sequences instead of word-pieces. We compare the resulting model, dubbed TavBERT, against contemporary PLMs based on subwords for three highly complex and ambiguous MRLs (Hebrew, Turkish, and Arabic), testing them on both morphological and semantic tasks. Our results show, for all tested languages, that while TavBERT obtains mild improvements on surface-level tasks à la POS tagging and full morphological disambiguation, subword-based PLMs achieve significantly higher performance on semantic tasks, such as named entity recognition and extractive question answering. These results showcase and (re)confirm the potential of subword tokenization as a reasonable modeling assumption for many languages, including MRLs.