A Meta-Analysis of Distributionally-Robust Models
This work addresses robustness issues in image classification for AI practitioners, but it is incremental as it synthesizes existing models without introducing new methods.
The paper tackled the vulnerability of state-of-the-art image classifiers to distribution shifts by conducting a meta-analysis of recent models, identifying four commonalities in the best-performing out-of-distribution robust models that highlight the promise of vision-language pre-training.
State-of-the-art image classifiers trained on massive datasets (such as ImageNet) have been shown to be vulnerable to a range of both intentional and incidental distribution shifts. On the other hand, several recent classifiers with favorable out-of-distribution (OOD) robustness properties have emerged, achieving high accuracy on their target tasks while maintaining their in-distribution accuracy on challenging benchmarks. We present a meta-analysis on a wide range of publicly released models, most of which have been published over the last twelve months. Through this meta-analysis, we empirically identify four main commonalities for all the best-performing OOD-robust models, all of which illuminate the considerable promise of vision-language pre-training.