MALT: Mechanistic Ablation of Lossy Translation in LLMs for a Low-Resource Language: Urdu
This addresses performance issues for users of low-resource languages in LLM applications, though it is incremental as it builds on known translation bottlenecks.
The study tackled the problem of LLMs performing poorly on low-resource languages like Urdu due to lossy translation features in their final layers, and found that removing these features and using a separate translation model improved performance significantly while preserving cultural nuances.
LLMs are predominantly trained on English data, which leads to a significant drop in performance on low-resource languages. Understanding how LLMs handle these languages is crucial for improving their effectiveness. This study focuses on Urdu as a use case for exploring the challenges faced by LLMs in processing low-resource languages. LLMs primarily reason in English when prompted in another language, with the final layers acting as translators to convert the English response into the target language. This study finds that even for low-resource languages, the internal latent response of LLMs in English is quite coherent; however, the translation features are lossy and result in poor translations, leading to reduced performance. By mechanistically removing these translation features and using a separate translation model to translate the internal latent response of LLM, the performance of LLMs improves significantly while also preserving the cultural nuances of the input in low-resource languages.