Connectionist Temporal Modeling for Weakly Supervised Action Labeling
This addresses the challenge of reducing annotation costs for video action recognition, making it more scalable for real-world applications, though it is incremental as it builds on existing weakly supervised methods.
The paper tackles the problem of action labeling in video with weak supervision, where only the order of actions is known during training, by proposing the Extended Connectionist Temporal Classification (ECTC) framework to handle unknown per-frame alignments and enforce consistency with visual similarities, achieving comparable performance to fully supervised approaches with less than 1% labeled frames per video.
We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time. The key challenge is that the per-frame alignments between the input (video) and label (action) sequences are unknown during training. We address this by introducing the Extended Connectionist Temporal Classification (ECTC) framework to efficiently evaluate all possible alignments via dynamic programming and explicitly enforce their consistency with frame-to-frame visual similarities. This protects the model from distractions of visually inconsistent or degenerated alignments without the need of temporal supervision. We further extend our framework to the semi-supervised case when a few frames are sparsely annotated in a video. With less than 1% of labeled frames per video, our method is able to outperform existing semi-supervised approaches and achieve comparable performance to that of fully supervised approaches.