Towards visually prompted keyword localisation for zero-resource spoken languages
This addresses the challenge of keyword search in zero-resource spoken languages, enabling access to speech data without textual annotations, though it is incremental as it builds on existing speech-vision models.
The paper tackles the problem of detecting and localizing keywords in spoken utterances using only visual prompts, without relying on text resources, and achieves a 16% relative improvement in localization F1 score over a baseline model.
Imagine being able to show a system a visual depiction of a keyword and finding spoken utterances that contain this keyword from a zero-resource speech corpus. We formalise this task and call it visually prompted keyword localisation (VPKL): given an image of a keyword, detect and predict where in an utterance the keyword occurs. To do VPKL, we propose a speech-vision model with a novel localising attention mechanism which we train with a new keyword sampling scheme. We show that these innovations give improvements in VPKL over an existing speech-vision model. We also compare to a visual bag-of-words (BoW) model where images are automatically tagged with visual labels and paired with unlabelled speech. Although this visual BoW can be queried directly with a written keyword (while our's takes image queries), our new model still outperforms the visual BoW in both detection and localisation, giving a 16% relative improvement in localisation F1.