Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization
This addresses the problem of adapting AQA models to changing real-world scenarios for applications like sports scoring and rehabilitation, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of Action Quality Assessment (AQA) in videos, which suffers from non-stationary quality distributions, by introducing Continual AQA (CAQA) with a method called Adaptive Manifold-Aligned Graph Regularization (MAGR++) to handle evolving distributions while mitigating catastrophic forgetting. The result is state-of-the-art performance with average correlation gains of 3.6% offline and 12.2% online over the strongest baseline.
Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation. A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios, which limits the generalization ability of conventional methods. We introduce Continual AQA (CAQA), which equips AQA with Continual Learning (CL) capabilities to handle evolving distributions while mitigating catastrophic forgetting. Although parameter-efficient fine-tuning of pretrained models has shown promise in CL for image classification, we find it insufficient for CAQA. Our empirical and theoretical analyses reveal two insights: (i) Full-Parameter Fine-Tuning (FPFT) is necessary for effective representation learning; yet (ii) uncontrolled FPFT induces overfitting and feature manifold shift, thereby aggravating forgetting. To address this, we propose Adaptive Manifold-Aligned Graph Regularization (MAGR++), which couples backbone fine-tuning that stabilizes shallow layers while adapting deeper ones with a two-step feature rectification pipeline: a manifold projector to translate deviated historical features into the current representation space, and a graph regularizer to align local and global distributions. We construct four CAQA benchmarks from three datasets with tailored evaluation protocols and strong baselines, enabling systematic cross-dataset comparison. Extensive experiments show that MAGR++ achieves state-of-the-art performance, with average correlation gains of 3.6% offline and 12.2% online over the strongest baseline, confirming its robustness and effectiveness. Our code is available at https://github.com/ZhouKanglei/MAGRPP.