Better Character Language Modeling Through Morphology
This work addresses language modeling efficiency for low-resource and morphologically rich languages, though it is incremental as it builds on existing multitasking approaches.
The paper tackles the problem of improving character language modeling by incorporating morphological supervision through multitasking, resulting in better bits-per-character performance across 24 languages, with inflected words showing more benefit and improvements persisting even with more data.
We incorporate morphological supervision into character language models (CLMs) via multitasking and show that this addition improves bits-per-character (BPC) performance across 24 languages, even when the morphology data and language modeling data are disjoint. Analyzing the CLMs shows that inflected words benefit more from explicitly modeling morphology than uninflected words, and that morphological supervision improves performance even as the amount of language modeling data grows. We then transfer morphological supervision across languages to improve language modeling performance in the low-resource setting.