Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settings
This work addresses the need for more grounded and culturally aware LLM evaluations in critical domains like healthcare, particularly for communities in India, though it appears incremental as it builds on existing benchmarking methods by incorporating community feedback.
The paper tackles the problem of evaluating LLMs in healthcare by proposing Samiksha, a community-driven evaluation pipeline co-created with civil-society organizations and community members in India, which highlights how current multilingual LLMs address nuanced community health queries and offers a scalable pathway for inclusive evaluation.
Large Language Models (LLMs) are typically evaluated through general or domain-specific benchmarks testing capabilities that often lack grounding in the lived realities of end users. Critical domains such as healthcare require evaluations that extend beyond artificial or simulated tasks to reflect the everyday needs, cultural practices, and nuanced contexts of communities. We propose Samiksha, a community-driven evaluation pipeline co-created with civil-society organizations (CSOs) and community members. Our approach enables scalable, automated benchmarking through a culturally aware, community-driven pipeline in which community feedback informs what to evaluate, how the benchmark is built, and how outputs are scored. We demonstrate this approach in the health domain in India. Our analysis highlights how current multilingual LLMs address nuanced community health queries, while also offering a scalable pathway for contextually grounded and inclusive LLM evaluation.