Audio-Visual Neural Syntax Acquisition
This addresses unsupervised language acquisition for AI systems by bridging speech and visual grounding, though it builds incrementally on prior work.
The paper tackles the problem of inducing phrase structure from visually-grounded speech without text supervision, resulting in AV-NSL achieving phrase structures comparable to text-based parsers for English and German.
We study phrase structure induction from visually-grounded speech. The core idea is to first segment the speech waveform into sequences of word segments, and subsequently induce phrase structure using the inferred segment-level continuous representations. We present the Audio-Visual Neural Syntax Learner (AV-NSL) that learns phrase structure by listening to audio and looking at images, without ever being exposed to text. By training on paired images and spoken captions, AV-NSL exhibits the capability to infer meaningful phrase structures that are comparable to those derived by naturally-supervised text parsers, for both English and German. Our findings extend prior work in unsupervised language acquisition from speech and grounded grammar induction, and present one approach to bridge the gap between the two topics.