IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
This provides a practice-oriented evaluation framework for LLMs in multilingual educational settings, though it is incremental as it adapts existing prompting methods to a new bilingual benchmark.
The authors tackled the problem of evaluating large language models (LLMs) in real-world multilingual educational contexts by introducing IndicEval, a benchmarking platform using authentic high-stakes exam questions in English and Hindi. Results showed that Chain-of-Thought prompting improved reasoning accuracy, but significant performance disparities and multilingual degradation (e.g., accuracy drops in Hindi) persisted across models like Gemini 2.0 Flash and GPT-4.
The rapid advancement of large language models (LLMs) necessitates evaluation frameworks that reflect real-world academic rigor and multilingual complexity. This paper introduces IndicEval, a scalable benchmarking platform designed to assess LLM performance using authentic high-stakes examination questions from UPSC, JEE, and NEET across STEM and humanities domains in both English and Hindi. Unlike synthetic benchmarks, IndicEval grounds evaluation in real examination standards, enabling realistic measurement of reasoning, domain knowledge, and bilingual adaptability. The framework automates assessment using Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting strategies and supports modular integration of new models and languages. Experiments conducted on Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B reveal three major findings. First, CoT prompting consistently improves reasoning accuracy, with substantial gains across subjects and languages. Second, significant cross-model performance disparities persist, particularly in high-complexity examinations. Third, multilingual degradation remains a critical challenge, with marked accuracy drops in Hindi compared to English, especially under Zero-Shot conditions. These results highlight persistent gaps in bilingual reasoning and domain transfer. Overall, IndicEval provides a practice-oriented, extensible foundation for rigorous, equitable evaluation of LLMs in multilingual educational settings and offers actionable insights for improving reasoning robustness and language adaptability.