Language Complexity Measurement as a Noisy Zero-Shot Proxy for Evaluating LLM Performance
This provides a practical, incremental method for assessing general LLM capabilities without extensive datasets, benefiting researchers and practitioners in AI evaluation.
The paper tackled the problem of evaluating LLM performance by testing state-of-the-art models on language complexity measurement tasks, such as computing LIX readability scores and dependency parsing on Swedish essays, and found that ChatGPT-o1-mini performed best with a strong correlation of -0.875 to MMLU benchmark scores.
Large Language Models (LLMs) have made significant strides in natural language generation but often face challenges in tasks requiring precise calculations and structural analysis. This paper investigates the performance of state-of-the-art LLMs on language complexity measurement tasks, through the computation of the LIX readability metric and Average Dependency Distance (ADD). Using Swedish high school and university-level essays, we evaluate the models' abilities to compute LIX scores and perform dependency parsing, comparing their results to established ground truths. Our findings reveal that while all models demonstrate some capacity for these tasks, ChatGPT-o1-mini performs most consistently, achieving the highest accuracy in both LIX computation and dependency parsing. Additionally, we observe a strong significant correlation -0.875 p 0.026 (N=6) between the models' accuracy in computing LIX and their overall performance on the Massive Multitask Language Understanding (MMLU) benchmark. These results suggest that language complexity measurement abilities can serve as a noisy zero-shot proxies for assessing the general capabilities of LLMs, providing a practical method for model evaluation without the need for extensive benchmarking datasets.