Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams
This work addresses the problem of adapting models to evolving, non-stationary streaming data for applications like action recognition and semantic segmentation, offering an incremental approach by exploiting forgetting rather than avoiding it.
This paper proposes a domain incremental learning approach that intentionally allows catastrophic forgetting. It combines a main task head with a self-supervised masked autoencoder (MAE) head, learning domain-specific LoRA adapters during incremental training. At inference, online test-time training on the MAE head identifies the best-matching LoRA for the current input, enabling the model to 'remember' the domain.
In this work we introduce a novel approach to domain incremental learning, adapting models over time to evolving, non-stationary data. In contrast to other works, we do not attempt to avoid catastrophic forgetting, but rather allow it and exploit it. Our model combines a main task head with a self-supervised masked autoencoder (MAE) head. We then learn domain-specific LoRA adapters during incremental training. Each adapter specializes to its domain, naturally inducing forgetting on other domains in both heads. At inference, we perform online test-time training on the self-supervised MAE head to identify which LoRAs best matches the current input, so the model can `remember' the domain again. Our scheme is especially well-suited to real-world streaming data, such as video, where consecutive samples are highly correlated and domain shifts are gradual. We demonstrate our method on domain-incremental action recognition and semantic segmentation tasks.