No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes
This work provides incremental insights into LLM internals by identifying a self-assessment mechanism, aiding researchers in understanding model confidence and potential applications in reliability assessment.
The researchers investigated whether large language models (LLMs) can predict their own answer accuracy before generating responses, using linear probes on activations to forecast correctness, achieving predictive performance across diverse knowledge datasets and outperforming baselines, with saturation in intermediate layers and limitations on mathematical reasoning.
Do large language models (LLMs) anticipate when they will answer correctly? To study this, we extract activations after a question is read but before any tokens are generated, and train linear probes to predict whether the model's forthcoming answer will be correct. Across three open-source model families ranging from 7 to 70 billion parameters, projections on this "in-advance correctness direction" trained on generic trivia questions predict success in distribution and on diverse out-of-distribution knowledge datasets, outperforming black-box baselines and verbalised predicted confidence. Predictive power saturates in intermediate layers, suggesting that self-assessment emerges mid-computation. Notably, generalisation falters on questions requiring mathematical reasoning. Moreover, for models responding "I don't know", doing so strongly correlates with the probe score, indicating that the same direction also captures confidence. By complementing previous results on truthfulness and other behaviours obtained with probes and sparse auto-encoders, our work contributes essential findings to elucidate LLM internals.