Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation
This addresses domain adaptation challenges for machine learning models in unpredictable target environments, though it is incremental as it builds on existing FTTA frameworks.
The paper tackles the problem of domain shifts in Fully Test-Time Adaptation (FTTA) without access to source data, proposing an uncover-and-unlearn approach that simulates and regularizes nuisances to improve generalization, and it consistently outperforms existing methods in experiments on corruption and style shifts.
Fully Test-Time Adaptation (FTTA) addresses domain shifts without access to source data and training protocols of the pre-trained models. Traditional strategies that align source and target feature distributions are infeasible in FTTA due to the absence of training data and unpredictable target domains. In this work, we exploit a dual perspective on FTTA, and propose Agnostic FTTA (AFTTA) as a novel formulation that enables the usage of off-the-shelf domain transformations during test-time to enable direct generalization to unforeseeable target data. To address this, we develop an uncover-and-unlearn approach. First, we uncover potential unwanted shifts between source and target domains by simulating them through predefined mappings and consider them as nuisances. Then, during test-time prediction, the model is enforced to unlearn these nuisances by regularizing the consequent shifts in latent representations and label predictions. Specifically, a mutual information-based criterion is devised and applied to guide nuisances unlearning in the feature space and encourage confident and consistent prediction in label space. Our proposed approach explicitly addresses agnostic domain shifts, enabling superior model generalization under FTTA constraints. Extensive experiments on various tasks, involving corruption and style shifts, demonstrate that our method consistently outperforms existing approaches.