Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition
This work addresses the problem of adapting pre-trained models to novel data with limited samples for researchers in video analysis, though it appears incremental as it builds on existing causal representation learning methods.
The paper tackles few-shot action recognition under distribution shift by proposing CDTD, a method that identifies and leverages transferable temporal dynamics, achieving superior accuracy over leading alternatives on standard datasets.
Few-shot action recognition aims at quickly adapting a pre-trained model to the novel data with a distribution shift using only a limited number of samples. Key challenges include how to identify and leverage the transferable knowledge learned by the pre-trained model. We therefore propose CDTD, or Causal Domain-Invariant Temporal Dynamics for knowledge transfer. To identify the temporally invariant and variant representations, we employ the causal representation learning methods for unsupervised pertaining, and then tune the classifier with supervisions in next stage. Specifically, we assume the domain information can be well estimated and the pre-trained image decoder and transition models can be well transferred. During adaptation, we fix the transferable temporal dynamics and update the image encoder and domain estimator. The efficacy of our approach is revealed by the superior accuracy of CDTD over leading alternatives across standard few-shot action recognition datasets.