An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analyses
This work addresses the need for accurate and culturally sensitive translation systems for low-resource Indian languages, though it is incremental as it builds on existing evaluation methods.
The study evaluated LLMs and Google Translate for translating Indian languages like Sanskrit, Telugu, and Hindi, using sentiment and semantic analyses on texts such as the Bhagavad Gita, finding that GPT models outperformed Google Translate in preserving sentiment and semantics, with GPT better at sentiment polarity than expert translations.
Large Language models (LLMs) have been prominent for language translation, including low-resource languages. There has been limited study on the assessment of the quality of translations generated by LLMs, including Gemini, GPT, and Google Translate. This study addresses this limitation by using semantic and sentiment analysis of selected LLMs for Indian languages, including Sanskrit, Telugu and Hindi. We select prominent texts (Bhagavad Gita, Tamas and Maha Prasthanam ) that have been well translated by experts and use LLMs to generate their translations into English, and provide a comparison with selected expert (human) translations. Our investigation revealed that while LLMs have made significant progress in translation accuracy, challenges remain in preserving sentiment and semantic integrity, especially in metaphorical and philosophical contexts for texts such as the Bhagavad Gita. The sentiment analysis revealed that GPT models are better at preserving the sentiment polarity for the given texts when compared to human (expert) translation. The results revealed that GPT models are generally better at maintaining the sentiment and semantics when compared to Google Translate. This study could help in the development of accurate and culturally sensitive translation systems for large language models.