Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation
This addresses the issue of benchmark interpretability and comparison in AI, though it appears incremental as it builds on existing tools and methods for documentation automation.
The paper tackled the problem of incomplete and inconsistent AI benchmark documentation by developing Auto-BenchmarkCard, a workflow that automates the synthesis of validated benchmark descriptions, resulting in improved transparency and comparability for researchers and practitioners.
We present Auto-BenchmarkCard, a workflow for generating validated descriptions of AI benchmarks. Benchmark documentation is often incomplete or inconsistent, making it difficult to interpret and compare benchmarks across tasks or domains. Auto-BenchmarkCard addresses this gap by combining multi-agent data extraction from heterogeneous sources (e.g., Hugging Face, Unitxt, academic papers) with LLM-driven synthesis. A validation phase evaluates factual accuracy through atomic entailment scoring using the FactReasoner tool. This workflow has the potential to promote transparency, comparability, and reusability in AI benchmark reporting, enabling researchers and practitioners to better navigate and evaluate benchmark choices.