Weakly supervised learning of actions from transcripts
This work addresses the challenge of reducing annotation costs for video action recognition, offering a weakly supervised method that is incremental but improves alignment accuracy.
The paper tackles the problem of learning human actions from video transcriptions without frame-level annotations by splitting video sequences uniformly and learning action models that maximize the probability of generating training sequences given the transcript order. The approach achieves competitive performance in localizing and classifying actions compared to fully supervised models and outperforms state-of-the-art methods for aligning transcripts with video data on four datasets.
We present an approach for weakly supervised learning of human actions from video transcriptions. Our system is based on the idea that, given a sequence of input data and a transcript, i.e. a list of the order the actions occur in the video, it is possible to infer the actions within the video stream, and thus, learn the related action models without the need for any frame-based annotation. Starting from the transcript information at hand, we split the given data sequences uniformly based on the number of expected actions. We then learn action models for each class by maximizing the probability that the training video sequences are generated by the action models given the sequence order as defined by the transcripts. The learned model can be used to temporally segment an unseen video with or without transcript. We evaluate our approach on four distinct activity datasets, namely Hollywood Extended, MPII Cooking, Breakfast and CRIM13. We show that our system is able to align the scripted actions with the video data and that the learned models localize and classify actions competitively in comparison to models trained with full supervision, i.e. with frame level annotations, and that they outperform any current state-of-the-art approach for aligning transcripts with video data.