AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems
This work provides a new methodology for understanding and improving benchmark reliability in AI, addressing the problem of measurement noise in leaderboard rankings for the AI community.
The authors introduced a framework using Confirmatory Factor Analysis and Generalizability Theory to analyze the latent structure of AI benchmark ecosystems, applied to 4,000+ models from the Open LLM Leaderboard. They found that current reporting underestimates benchmark relationships, local dependence undermines measurement, contributor metadata explains ~9% of variance, and the latent general-factor size slope is highly stable (R_g=0.97) while manifest-score scaling law slope has low reliability (R_β=0.53).
While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis (CFA) and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance and establish: (1) structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; (2) evidence of local dependence among leaderboard items, undermining uses of benchmarks as measurement instruments under current scoring systems; (3) contributor metadata explains more rank-relevant variance ($\approx9\%$) than architecture or deployment categories in this context; (4) a manifest-score "scaling law" slope has low reliability ($R_β=0.53$); by contrast, the latent general-factor size slope is highly stable across ecosystem controls ($R_g=0.97$). We are able to provide unique insights into benchmark dynamics, such as which benchmarks are a function of LLM size and which can be oppositely impacted by post-training practices. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.