Label Shift Adapter for Test-Time Adaptation under Covariate and Label Shifts
This addresses a practical issue for real-world machine learning deployments where label imbalances shift across domains, though it is incremental as it builds on existing TTA methods.
The paper tackles the problem of test-time adaptation (TTA) under both covariate and label shifts, where existing methods often fail, by proposing a label shift adapter that estimates target label distributions and adapts model parameters efficiently. The result shows substantial performance improvements when integrated with TTA approaches, as demonstrated through extensive experiments.
Test-time adaptation (TTA) aims to adapt a pre-trained model to the target domain in a batch-by-batch manner during inference. While label distributions often exhibit imbalances in real-world scenarios, most previous TTA approaches typically assume that both source and target domain datasets have balanced label distribution. Due to the fact that certain classes appear more frequently in certain domains (e.g., buildings in cities, trees in forests), it is natural that the label distribution shifts as the domain changes. However, we discover that the majority of existing TTA methods fail to address the coexistence of covariate and label shifts. To tackle this challenge, we propose a novel label shift adapter that can be incorporated into existing TTA approaches to deal with label shifts during the TTA process effectively. Specifically, we estimate the label distribution of the target domain to feed it into the label shift adapter. Subsequently, the label shift adapter produces optimal parameters for the target label distribution. By predicting only the parameters for a part of the pre-trained source model, our approach is computationally efficient and can be easily applied, regardless of the model architectures. Through extensive experiments, we demonstrate that integrating our strategy with TTA approaches leads to substantial performance improvements under the joint presence of label and covariate shifts.