Beyond Accuracy: What Matters in Designing Well-Behaved Models?
This work addresses the need for more comprehensive model evaluation in computer vision, providing insights and a metric to guide practitioners in selecting models based on multiple quality criteria, though it is incremental in extending existing studies to more dimensions.
The paper tackles the problem of deep neural networks lacking in quality dimensions beyond accuracy, such as robustness and fairness, by conducting a large-scale study analyzing 326 backbone models across nine dimensions for image classification, revealing insights like vision-language models' high fairness and self-supervised learning's effectiveness, and introducing the QUBA score for model ranking.
Deep learning has become an essential part of computer vision, with deep neural networks (DNNs) excelling in predictive performance. However, they often fall short in other critical quality dimensions, such as robustness, calibration, or fairness. While existing studies have focused on a subset of these quality dimensions, none have explored a more general form of "well-behavedness" of DNNs. With this work, we address this gap by simultaneously studying nine different quality dimensions for image classification. Through a large-scale study, we provide a bird's-eye view by analyzing 326 backbone models and how different training paradigms and model architectures affect the quality dimensions. We reveal various new insights such that (i) vision-language models exhibit high fairness on ImageNet-1k classification and strong robustness against domain changes; (ii) self-supervised learning is an effective training paradigm to improve almost all considered quality dimensions; and (iii) the training dataset size is a major driver for most of the quality dimensions. We conclude our study by introducing the QUBA score (Quality Understanding Beyond Accuracy), a novel metric that ranks models across multiple dimensions of quality, enabling tailored recommendations based on specific user needs.