Multi-View Video-Based Learning: Leveraging Weak Labels for Frame-Level Perception
This work addresses the problem of reducing annotation effort for video-based action recognition in multi-view settings, offering an incremental improvement by leveraging weak labels for frame-level tasks.
The paper tackles the challenge of training multi-view video action recognition models with only sequence-level weak labels for frame-level perception tasks, proposing a novel learning framework that uses weak labels to train a base model with a latent loss function, which improves downstream frame-level action recognition and detection accuracy on the MM Office dataset.
For training a video-based action recognition model that accepts multi-view video, annotating frame-level labels is tedious and difficult. However, it is relatively easy to annotate sequence-level labels. This kind of coarse annotations are called as weak labels. However, training a multi-view video-based action recognition model with weak labels for frame-level perception is challenging. In this paper, we propose a novel learning framework, where the weak labels are first used to train a multi-view video-based base model, which is subsequently used for downstream frame-level perception tasks. The base model is trained to obtain individual latent embeddings for each view in the multi-view input. For training the model using the weak labels, we propose a novel latent loss function. We also propose a model that uses the view-specific latent embeddings for downstream frame-level action recognition and detection tasks. The proposed framework is evaluated using the MM Office dataset by comparing several baseline algorithms. The results show that the proposed base model is effectively trained using weak labels and the latent embeddings help the downstream models improve accuracy.