LookSharp: Attention Entropy Minimization for Test-Time Adaptation
This work addresses robustness for vision models under distribution shifts, but it is incremental as it builds on existing entropy minimization techniques.
The paper tackled the problem of model robustness under distribution shifts by proposing LookSharp, a test-time adaptation method that minimizes attention entropy in transformers, which improved performance on ImageNet-C while maintaining clean data accuracy.
Test-time adaptation (TTA) updates models during inference to reduce error on distribution shifts. While entropy minimization over the output distribution has proven effective as a TTA loss, we study using the intermediate distributions computed by transformers in the attention mechanism. We propose LookSharp, which minimizes the entropy of CLS-to-patch attention in the final layer as a novel TTA objective, encouraging the model to maintain focused attention on shifted data. We demonstrate that attention entropy minimization improves robustness on ImageNet-C. We also show that it is complementary to output entropy minimization and maintains performance on clean data.