IndicGEC: Powerful Models, or a Measurement Mirage?
This work addresses grammatical error correction for Indian languages, but is incremental as it applies existing prompting methods to new data.
The authors tackled grammatical error correction for Indian languages using zero/few-shot prompting of language models, achieving GLEU scores of 83.78 in Telugu and 84.31 in Hindi. They extended experiments to three more languages and raised concerns about data quality and evaluation metrics.
In this paper, we report the results of the TeamNRC's participation in the BHASHA-Task 1 Grammatical Error Correction shared task https://github.com/BHASHA-Workshop/IndicGEC2025/ for 5 Indian languages. Our approach, focusing on zero/few-shot prompting of language models of varying sizes (4B to large proprietary models) achieved a Rank 4 in Telugu and Rank 2 in Hindi with GLEU scores of 83.78 and 84.31 respectively. In this paper, we extend the experiments to the other three languages of the shared task - Tamil, Malayalam and Bangla, and take a closer look at the data quality and evaluation metric used. Our results primarily highlight the potential of small language models, and summarize the concerns related to creating good quality datasets and appropriate metrics for this task that are suitable for Indian language scripts.