Speech2Action: Cross-modal Supervision for Action Recognition
This work addresses action recognition for video analysis by enabling unsupervised learning from large-scale movie data, though it is incremental as it builds on existing cross-modal methods.
The authors tackled the problem of action recognition in videos by leveraging the correlation between spoken words and actions in movies, using screenplays to train a BERT-based classifier that predicts actions from speech without manual labels. They achieved superior performance on standard benchmarks, generating weak labels for over 800K video clips from 188M speech segments.
Is it possible to guess human action from dialogue alone? In this work we investigate the link between spoken words and actions in movies. We note that movie screenplays describe actions, as well as contain the speech of characters and hence can be used to learn this correlation with no additional supervision. We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments. We then apply this model to the speech segments of a large unlabelled movie corpus (188M speech segments from 288K movies). Using the predictions of this model, we obtain weak action labels for over 800K video clips. By training on these video clips, we demonstrate superior action recognition performance on standard action recognition benchmarks, without using a single manually labelled action example.