Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned
This work addresses the misalignment between language models and human language processing in a specific evaluation domain, highlighting limitations for NLP applications.
The study compared language models to human performance on the cloze task, finding that models underestimate human response probabilities, misrank responses, and produce distinct semantic spaces, demonstrating they cannot replace or model the cloze task effectively.
In this work we compare the generative behavior at the next token prediction level in several language models by comparing them to human productions in the cloze task. We find that while large models trained for longer are typically better estimators of human productions, but they reliably under-estimate the probabilities of human responses, over-rank rare responses, under-rank top responses, and produce highly distinct semantic spaces. Altogether, this work demonstrates in a tractable, interpretable domain that LM generations can not be used as replacements of or models of the cloze task.