Training and Evaluation of a Multilingual Tokenizer for GPT-SW3
This work addresses the need for effective multilingual tokenization in language models, but it is incremental as it applies existing methods to a specific dataset.
The paper tackles the development and evaluation of a multilingual tokenizer for GPT-SW3, trained on the Nordic Pile using SentencePiece and BPE, resulting in a detailed analysis of its vocabulary and performance across languages.
This paper provides a detailed discussion of the multilingual tokenizer used for GPT-SW3. It was trained on the Nordic Pile using the SentencePiece library and the BPE algorithm. We outline the tokenizer's most important features and share details on its learned vocabulary. In addition, we systematically analyze the properties and evaluate the performance of the tokenizer with regard to the different languages present in the data.