CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models
This work addresses the decompounding problem for NLP researchers and practitioners working with diverse languages, offering a novel dataset and method, though it is incremental in improving existing techniques.
The authors tackled the lack of public datasets for decompounding across many languages by introducing a dataset of 255k words in 56 languages, and found that LLMs perform poorly on this task, especially due to subword tokenization issues. They developed a two-stage method with self-supervised and supervised learning, achieving a 13.9% accuracy improvement over prior unsupervised models and outperforming all prior language-specific tools.
While many languages possess processes of joining two or more words to create compound words, previous studies have been typically limited only to languages with excessively productive compound formation (e.g., German, Dutch) and there is no public dataset containing compound and non-compound words across a large number of languages. In this work, we systematically study decompounding, the task of splitting compound words into their constituents, at a wide scale. We first address the data gap by introducing a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We then use this dataset to evaluate an array of Large Language Models (LLMs) on the decompounding task. We find that LLMs perform poorly, especially on words which are tokenized unfavorably by subword tokenization. We thus introduce a novel methodology to train dedicated models for decompounding. The proposed two-stage procedure relies on a fully self-supervised objective in the first stage, while the second, supervised learning stage optionally fine-tunes the model on the annotated Wiktionary data. Our self-supervised models outperform the prior best unsupervised decompounding models by 13.9% accuracy on average. Our fine-tuned models outperform all prior (language-specific) decompounding tools. Furthermore, we use our models to leverage decompounding during the creation of a subword tokenizer, which we refer to as CompoundPiece. CompoundPiece tokenizes compound words more favorably on average, leading to improved performance on decompounding over an otherwise equivalent model using SentencePiece tokenization.