Eval Factsheets: A Structured Framework for Documenting AI Evaluations
This addresses the lack of systematic documentation standards for AI evaluations, which is a problem for researchers and practitioners dealing with reproducibility and transparency issues in benchmarking.
The authors tackled the problem of inconsistent documentation for AI evaluation methodologies by introducing Eval Factsheets, a structured framework with a taxonomy and questionnaire that organizes evaluation characteristics across five dimensions, and demonstrated its effectiveness through case studies on multiple benchmarks.
The rapid proliferation of benchmarks has created significant challenges in reproducibility, transparency, and informed decision-making. However, unlike datasets and models -- which benefit from structured documentation frameworks like Datasheets and Model Cards -- evaluation methodologies lack systematic documentation standards. We introduce Eval Factsheets, a structured, descriptive framework for documenting AI system evaluations through a comprehensive taxonomy and questionnaire-based approach. Our framework organizes evaluation characteristics across five fundamental dimensions: Context (Who made the evaluation and when?), Scope (What does it evaluate?), Structure (With what the evaluation is built?), Method (How does it work?) and Alignment (In what ways is it reliable/valid/robust?). We implement this taxonomy as a practical questionnaire spanning five sections with mandatory and recommended documentation elements. Through case studies on multiple benchmarks, we demonstrate that Eval Factsheets effectively captures diverse evaluation paradigms -- from traditional benchmarks to LLM-as-judge methodologies -- while maintaining consistency and comparability. We hope Eval Factsheets are incorporated into both existing and newly released evaluation frameworks and lead to more transparency and reproducibility.