Unsupervised learning of action classes with continuous temporal embedding
This addresses the need for efficient action segmentation in videos for computer vision applications, but it is incremental as it builds on existing unsupervised and embedding techniques.
The paper tackles the problem of unsupervised learning of action classes from untrimmed videos to avoid costly manual annotation, proposing a method using continuous temporal embeddings and clustering, and shows it works on datasets like Breakfast, YouTube Instructions, and 50Salads, including in general settings with unknown video content.
The task of temporally detecting and segmenting actions in untrimmed videos has seen an increased attention recently. One problem in this context arises from the need to define and label action boundaries to create annotations for training which is very time and cost intensive. To address this issue, we propose an unsupervised approach for learning action classes from untrimmed video sequences. To this end, we use a continuous temporal embedding of framewise features to benefit from the sequential nature of activities. Based on the latent space created by the embedding, we identify clusters of temporal segments across all videos that correspond to semantic meaningful action classes. The approach is evaluated on three challenging datasets, namely the Breakfast dataset, YouTube Instructions, and the 50Salads dataset. While previous works assumed that the videos contain the same high level activity, we furthermore show that the proposed approach can also be applied to a more general setting where the content of the videos is unknown.