AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models
This addresses the need for reliable cross-domain knowledge in language models for practical applications, though it is incremental as it builds on existing evaluation methods by introducing a new benchmark.
The authors tackled the problem of evaluating factual accuracy and knowledge calibration in large language models across multiple domains, introducing the AA-Omniscience benchmark with 6,000 questions and finding that Claude 4.1 Opus achieved the highest score of 4.8, with only three models scoring above zero, revealing persistent weaknesses in factuality and calibration.
Existing language model evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.