Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
This highlights a critical reliability issue in LLM evaluation for researchers and practitioners, as it shows models are sensitive to shallow input variations, though it is incremental in addressing known concerns about benchmark robustness.
The paper investigated how lexical and syntactic perturbations affect the performance and rankings of 23 LLMs on benchmarks like MMLU, SQuAD, and AMEGA, finding that lexical changes consistently degrade performance and destabilize leaderboards, while syntactic changes have mixed effects.
The rapid advancement of Large Language Models (LLMs) has established standardized evaluation benchmarks as the primary instrument for model comparison. Yet, their reliability is increasingly questioned due to sensitivity to shallow variations in input prompts. This paper examines how controlled, truth-conditionally equivalent lexical and syntactic perturbations affect the absolute performance and relative ranking of 23 contemporary LLMs across three benchmarks: MMLU, SQuAD, and AMEGA. We employ two linguistically principled pipelines to generate meaning-preserving variations: one performing synonym substitution for lexical changes, and another using dependency parsing to determine applicable syntactic transformations. Results show that lexical perturbations consistently induce substantial, statistically significant performance degradation across nearly all models and tasks, while syntactic perturbations have more heterogeneous effects, occasionally improving results. Both perturbation types destabilize model leaderboards on complex tasks. Furthermore, model robustness did not consistently scale with model size, revealing strong task dependence. Overall, the findings suggest that LLMs rely more on surface-level lexical patterns than on abstract linguistic competence, underscoring the need for robustness testing as a standard component of LLM evaluation.