Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback
This work addresses the challenge of enhancing discriminative model performance in real-time or online settings, though it is incremental as it builds on existing test-time adaptation methods.
The paper tackles the problem of using generative models for discriminative tasks by introducing Diffusion-TTA, a method that adapts pre-trained discriminative models at test time using generative feedback from diffusion models, resulting in significant accuracy improvements for models like ImageNet classifiers and CLIP models.
The advancements in generative modeling, particularly the advent of diffusion models, have sparked a fundamental question: how can these models be effectively used for discriminative tasks? In this work, we find that generative models can be great test-time adapters for discriminative models. Our method, Diffusion-TTA, adapts pre-trained discriminative models such as image classifiers, segmenters and depth predictors, to each unlabelled example in the test set using generative feedback from a diffusion model. We achieve this by modulating the conditioning of the diffusion model using the output of the discriminative model. We then maximize the image likelihood objective by backpropagating the gradients to discriminative model's parameters. We show Diffusion-TTA significantly enhances the accuracy of various large-scale pre-trained discriminative models, such as, ImageNet classifiers, CLIP models, image pixel labellers and image depth predictors. Diffusion-TTA outperforms existing test-time adaptation methods, including TTT-MAE and TENT, and particularly shines in online adaptation setups, where the discriminative model is continually adapted to each example in the test set. We provide access to code, results, and visualizations on our website: https://diffusion-tta.github.io/.