Comparative Analysis of Large Language Models in Generating Telugu Responses for Maternal Health Queries
This work addresses the need for reliable LLM assistance in maternal healthcare for low-resource languages like Telugu, though it is incremental as it compares existing models without introducing new methods.
The study evaluated ChatGPT-4o, GeminiAI, and Perplexity AI in generating Telugu responses for maternal health queries, finding that Gemini performed best in accuracy and coherence, while Perplexity excelled with Telugu prompts, highlighting the importance of model and language selection.
Large Language Models (LLMs) have been progressively exhibiting there capabilities in various areas of research. The performance of the LLMs in acute maternal healthcare area, predominantly in low resource languages like Telugu, Hindi, Tamil, Urdu etc are still unstudied. This study presents how ChatGPT-4o, GeminiAI, and Perplexity AI respond to pregnancy related questions asked in different languages. A bilingual dataset is used to obtain results by applying the semantic similarity metrics (BERT Score) and expert assessments from expertise gynecologists. Multiple parameters like accuracy, fluency, relevance, coherence and completeness are taken into consideration by the gynecologists to rate the responses generated by the LLMs. Gemini excels in other LLMs in terms of producing accurate and coherent pregnancy relevant responses in Telugu, while Perplexity demonstrated well when the prompts were in Telugu. ChatGPT's performance can be improved. The results states that both selecting an LLM and prompting language plays a crucial role in retrieving the information. Altogether, we emphasize for the improvement of LLMs assistance in regional languages for healthcare purposes.