The Importance of Context in Very Low Resource Language Modeling
This addresses the challenge of building effective language models for low-resource languages, though it is incremental as it adapts existing neural methods rather than proposing a new paradigm.
The paper tackles the problem of language model pretraining with very low resources (under 100k sentences), finding that n-gram models outperform neural models due to their local context focus. It introduces methods to improve neural models, with limited self-attention boosting performance by up to 5% on tasks like NLI and POS tagging for English, Hindi, and Turkish.
This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model's performance in the low-resource setting, finding that limiting the model's self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.