IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluation
This addresses the problem of efficient and accurate quality prediction for LLM users, offering a novel method that improves over existing external classifiers.
The paper tackles the challenge of predicting large language model output quality for specific queries by proposing IntroLM, which enables models to self-evaluate during prefilling without external classifiers, achieving a 90% ROC AUC on question answering benchmarks and reducing latency by up to 33% and large model usage by up to 50% in routing systems.
A major challenge for the operation of large language models (LLMs) is how to predict whether a specific LLM will produce sufficiently high-quality output for a given query. Existing approaches rely on external classifiers, most commonly BERT based models, which suffer from limited context windows, constrained representational capacity, and additional computational overhead. We propose IntroLM, a method that enables causal language models to predict their own output quality during the prefilling phase without affecting generation using introspective tokens. By introducing token conditional LoRA that activates only for the introspective token, the model learns to predict the output quality for a given query while preserving the original backbone behavior and avoiding external evaluators. On question answering benchmarks, IntroLM applied to Qwen3 8B achieves a ROC AUC of 90 precent for success prediction, outperforming a DeBERTa classifier by 14 precent. When integrated into multi model routing systems, IntroLM achieves superior cost performance tradeoffs, reducing latency by up to 33 precent and large model usage by up to 50 precent at matched reliability.