ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjects
This addresses a gap in assessing LLMs for deeper disciplinary understanding in the Indian setting, though it is incremental as it builds on existing benchmarking efforts.
The paper tackles the problem of evaluating large language models on graduate-level, culturally grounded questions in the Indian context, presenting ParamBench with over 17K Hindi questions from 21 subjects, and finds that the best-performing model, Gemma3-27B, achieves an overall accuracy of 56.4%.
Large language models have been widely evaluated on tasks such as comprehension, summarization, code generation, etc. However, their performance on graduate-level, culturally grounded questions in the Indian context remains largely unexplored. Existing Indian benchmarks emphasise basic fact-orientated queries that offer limited assessment of a deeper disciplinary understanding tailored to the Indian setting. In this paper, we present ParamBench, consisting of more than 17K questions in the Hindi language, comprising questionnaires from 21 diverse subjects. These questions are primarily derived from a nationwide graduate-level entrance examination covering topics such as history, music, instruments, yoga, literature, philosophy, law, etc.~ specifically for the Indian context. Additionally, we assess the ability of LLMs to handle diverse question formats - such as list-based matching, assertion-reason pairs, and sequence ordering - alongside conventional multiple-choice questions. We evaluated the performance of more than 16 open source LLMs on this benchmark, observing that Gemma3-27B attains the highest overall accuracy of 56.4\%. Furthermore, subject-wise analysis indicates that even for the best-performing LLMs, performance remains weak on topics such as music, classical instruments, and law, underscoring persistent challenges in culturally grounded reasoning. The dataset and source code is present at https://github.com/ayushbits/ParamBench.