On Pitfalls of Test-Time Adaptation
This work addresses the robustness challenge in machine learning under distribution shifts for researchers, but it is incremental as it focuses on benchmarking rather than proposing new methods.
The paper tackles the lack of consistent evaluation in test-time adaptation (TTA) by introducing TTAB, a benchmark that reveals three common pitfalls, such as hyper-parameter sensitivity and inability to handle all distribution shifts, based on experiments with ten algorithms.
Test-Time Adaptation (TTA) has recently emerged as a promising approach for tackling the robustness challenge under distribution shifts. However, the lack of consistent settings and systematic studies in prior literature hinders thorough assessments of existing methods. To address this issue, we present TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art algorithms, a diverse array of distribution shifts, and two evaluation protocols. Through extensive experiments, our benchmark reveals three common pitfalls in prior efforts. First, selecting appropriate hyper-parameters, especially for model selection, is exceedingly difficult due to online batch dependency. Second, the effectiveness of TTA varies greatly depending on the quality and properties of the model being adapted. Third, even under optimal algorithmic conditions, none of the existing methods are capable of addressing all common types of distribution shifts. Our findings underscore the need for future research in the field to conduct rigorous evaluations on a broader set of models and shifts, and to re-examine the assumptions behind the empirical success of TTA. Our code is available at \url{https://github.com/lins-lab/ttab}.