Egalitarian Language Representation in Language Models: It All Begins with Tokenizers
This addresses representation issues for non-English languages in AI, but it is incremental as it builds on existing tokenization methods.
The paper tackled the problem of unfair tokenization for complex script languages like Tamil, Sinhala, and Hindi in language models, showing that pre-tokenization methods are more critical than tokenization algorithms and introducing Grapheme Pair Encoding (GPE) which outperforms byte-level tokenizers in experiments.
Tokenizers act as a bridge between human language and the latent space of language models, influencing how language is represented in these models. Due to the immense popularity of English-Centric Large Language Models (LLMs), efforts are being made to adapt them for other languages. However, we demonstrate that, from a tokenization standpoint, not all tokenizers offer fair representation for complex script languages such as Tamil, Sinhala, and Hindi, primarily due to the choice of pre-tokenization methods. We go further to show that pre-tokenization plays a more critical role than the tokenization algorithm itself in achieving an egalitarian representation of these complex script languages. To address this, we introduce an improvement to the Byte Pair Encoding (BPE) algorithm by incorporating graphemes, which we term Grapheme Pair Encoding (GPE). Our experiments show that grapheme-based character extraction outperforms byte-level tokenizers for complex scripts. We validate this approach through experiments on Tamil, Sinhala, and Hindi.