HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settings
It addresses the problem of evaluating LLMs for health chatbots in real-world multilingual contexts, particularly for Indian languages, but is incremental as it applies existing methods to new data.
This study assessed 24 large language models (LLMs) in a real-world multilingual health chatbot setting using data from Indian patients in Indian English and four Indic languages, finding that models varied significantly in performance, with factual correctness lower for Indic queries and challenges posed by code-mixed and culturally relevant queries.
Assessing the capabilities and limitations of large language models (LLMs) has garnered significant interest, yet the evaluation of multiple models in real-world scenarios remains rare. Multilingual evaluation often relies on translated benchmarks, which typically do not capture linguistic and cultural nuances present in the source language. This study provides an extensive assessment of 24 LLMs on real world data collected from Indian patients interacting with a medical chatbot in Indian English and 4 other Indic languages. We employ a uniform Retrieval Augmented Generation framework to generate responses, which are evaluated using both automated techniques and human evaluators on four specific metrics relevant to our application. We find that models vary significantly in their performance and that instruction tuned Indic models do not always perform well on Indic language queries. Further, we empirically show that factual correctness is generally lower for responses to Indic queries compared to English queries. Finally, our qualitative work shows that code-mixed and culturally relevant queries in our dataset pose challenges to evaluated models.