Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models
Provides a lightweight, parameter-free abstention method for small language models, addressing the critical need for reliable uncertainty detection in resource-constrained autonomous systems.
Second Guess detects uncertainty in small language models by adding an 'I don't know' option and measuring answer stability, achieving a 10.81% composite risk improvement on multiple-choice QA benchmarks.
Large language models often generate confident but incorrect answers rather than abstaining when uncertain. This problem is particularly acute for small language models (SLMs), where computational constraints and autonomous operation amplify the need for reliable uncertainty detection. We propose _Second Guess_, a lightweight, parameter-free prompting technique for abstention in multiple-choice question answering (MCQA) that is well-suited for SLMs. Our key empirical insight is that models which truly know an answer will select it consistently, while uncertain models exhibit unstable behavior when an ``I don't know'' option is added. Evaluated on four open models (2B-8B parameters) and four benchmarks, Second Guess achieves the highest composite risk improvement of 10.81\%. Notably, it maintains an 8\% composite risk improvement on fine-tuned models where entropy-based methods degrade, and improves most for lower-performing models. All code and results required to reproduce this work is available in https://github.com/Mystic-Slice/second-guess