From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors
This work addresses a specific issue in natural language processing for improving model reliability, but it is incremental as it builds on existing concepts like surprisal and Uniform Information Density.
The paper tackles the problem of language models misinterpreting idiomatic or context-sensitive inputs by proposing an input-only method to predict failures using token-level likelihood features, achieving improved error detection across five challenging datasets.
Language models often struggle with idiomatic, figurative, or context-sensitive inputs, not because they produce flawed outputs, but because they misinterpret the input from the outset. We propose an input-only method for anticipating such failures using token-level likelihood features inspired by surprisal and the Uniform Information Density hypothesis. These features capture localized uncertainty in input comprehension and outperform standard baselines across five linguistically challenging datasets. We show that span-localized features improve error detection for larger models, while smaller models benefit from global patterns. Our method requires no access to outputs or hidden activations, offering a lightweight and generalizable approach to pre-generation error prediction.