Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings
This work addresses keyword localisation for low-resource languages like Yoruba, where transcriptions are unavailable, but it is incremental as it builds on prior methods with automated pair mining.
The paper tackles the problem of visually prompted keyword localisation in low-resource languages by introducing a few-shot learning scheme to automatically mine pairs without transcriptions, resulting in a small performance drop on English and reasonable but lower scores on Yoruba.
Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.