Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks
This work addresses the need for more realistic performance understanding in ML evaluation, particularly for researchers and practitioners using benchmarks, though it is incremental as it applies existing statistical methods to a new context.
The authors tackled the problem of quantifying statistical uncertainty in aggregate performance metrics for machine learning benchmarks, demonstrating methods like bootstrapping and Bayesian hierarchical modeling on the Visual Task Adaptation Benchmark (VTAB) to reveal insights such as model dominance in specific tasks despite overall poor performance.
Modern artificial intelligence is supported by machine learning models (e.g., foundation models) that are pretrained on a massive data corpus and then adapted to solve a variety of downstream tasks. To summarize performance across multiple tasks, evaluation metrics are often aggregated into a summary metric, e.g., average accuracy across 10 question-answering tasks. When aggregating evaluation metrics, it is useful to incorporate uncertainty in the aggregate metric in order to gain a more realistic understanding of model performance. Our objective in this work is to demonstrate how statistical methodology can be used for quantifying uncertainty in metrics that have been aggregated across multiple tasks. The methods we emphasize are bootstrapping, Bayesian hierarchical (i.e., multilevel) modeling, and the visualization of task weightings that consider standard errors. These techniques reveal insights such as the dominance of a specific model for certain types of tasks despite an overall poor performance. We use a popular ML benchmark, the Visual Task Adaptation Benchmark (VTAB), to demonstrate the usefulness of our approaches.