Video-aided Unsupervised Grammar Induction
This work addresses the problem of improving unsupervised grammar induction for natural language processing by leveraging richer video information, representing an incremental advance over existing multi-modal methods.
The paper tackles unsupervised grammar induction by learning a constituency parser from unlabeled text and corresponding videos, proposing a Multi-Modal Compound PCFG model that outperforms previous state-of-the-art systems on three benchmarks.
We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction.