Personalized Benchmarking with the Ludwig Benchmarking Toolkit
This addresses the need for fair and customizable benchmarking for industry practitioners and other communities, though it is incremental as it builds on existing benchmarking concepts with new toolkit features.
The authors tackled the problem of personalized benchmarking for machine learning models by introducing the Ludwig Benchmarking Toolkit (LBT), which enables end-to-end benchmark studies with configurable training and multi-objective evaluation, demonstrated through a large-scale analysis of 7 models and 9 datasets for text classification.
The rapid proliferation of machine learning models across domains and deployment settings has given rise to various communities (e.g. industry practitioners) which seek to benchmark models across tasks and objectives of personal value. Unfortunately, these users cannot use standard benchmark results to perform such value-driven comparisons as traditional benchmarks evaluate models on a single objective (e.g. average accuracy) and fail to facilitate a standardized training framework that controls for confounding variables (e.g. computational budget), making fair comparisons difficult. To address these challenges, we introduce the open-source Ludwig Benchmarking Toolkit (LBT), a personalized benchmarking toolkit for running end-to-end benchmark studies (from hyperparameter optimization to evaluation) across an easily extensible set of tasks, deep learning models, datasets and evaluation metrics. LBT provides a configurable interface for controlling training and customizing evaluation, a standardized training framework for eliminating confounding variables, and support for multi-objective evaluation. We demonstrate how LBT can be used to create personalized benchmark studies with a large-scale comparative analysis for text classification across 7 models and 9 datasets. We explore the trade-offs between inference latency and performance, relationships between dataset attributes and performance, and the effects of pretraining on convergence and robustness, showing how LBT can be used to satisfy various benchmarking objectives.