GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation
For researchers and practitioners deploying TTA in real-world scenarios with limited memory and non-i.i.d. data, this work identifies memory management as a first-class component and establishes diversity as a key principle.
The paper systematically benchmarks memory policies for test-time adaptation (TTA) and finds that intra-class diversity in memory buffers is crucial for robust adaptation under non-i.i.d. and label-skewed streams. The proposed GOTTA memory policies, which combine class-balanced allocation with feature-space diversity, improve adaptation performance under constrained memory budgets and challenging streams, while remaining competitive as memory increases.
Test-time adaptation (TTA) enables a pre-trained model to adapt online to an unlabeled test stream under distribution shift. While most TTA research focuses on the adaptation objective, practical streams also depend critically on the memory used to select which test samples drive adaptation. Existing memory mechanisms are usually evaluated as components of specific TTA algorithms, making it difficult to isolate which memory design choices matter and when they matter. In this work, we provide a systematic benchmark that decouples memory from the adaptation algorithm and evaluates memory policies under unified conditions across i.i.d., non-i.i.d., continual, and practical test streams. Our study shows that effective memory management requires more than retaining recent or class-balanced samples. In particular, intra-class diversity is a key factor for avoiding redundant buffers and maintaining representative adaptation signals under temporally correlated and label-skewed streams. Motivated by this finding, we introduce Guided Observational Test-Time Adaptation (GOTTA), a family of diversity-aware memory policies that combine class-balanced allocation with feature-space diversity. GOTTA memories act as drop-in replacements for existing buffers and can be paired with different TTA objectives. Across corruption benchmarks and video-stream settings, diversity-aware memory improves adaptation most clearly under constrained memory budgets and challenging non-i.i.d. streams, while remaining competitive as memory capacity increases. These results highlight memory management as a first-class component of robust test-time adaptation and identify diversity as a central principle for practical TTA.