Augmenting Statistical Machine Translation with Subword Translation of Out-of-Vocabulary Words
This addresses the issue of handling unseen words in machine translation, particularly beneficial for low-resource scenarios, though it is incremental as it builds on existing statistical methods.
The paper tackled the problem of translating out-of-vocabulary words in statistical machine translation by using subword information, resulting in consistent BLEU gains averaging 0.5 points and up to 2.0 points across fourteen languages.
Most statistical machine translation systems cannot translate words that are unseen in the training data. However, humans can translate many classes of out-of-vocabulary (OOV) words (e.g., novel morphological variants, misspellings, and compounds) without context by using orthographic clues. Following this observation, we describe and evaluate several general methods for OOV translation that use only subword information. We pose the OOV translation problem as a standalone task and intrinsically evaluate our approaches on fourteen typologically diverse languages across varying resource levels. Adding OOV translators to a statistical machine translation system yields consistent BLEU gains (0.5 points on average, and up to 2.0) for all fourteen languages, especially in low-resource scenarios.