Visually Grounded Keyword Detection and Localisation for Low-Resource Languages
It addresses keyword localization for low-resource languages, but is incremental as it builds on existing VGS models with new methods and datasets.
This study tackled keyword localization in speech using Visually Grounded Speech models, achieving 57% accuracy on an English dataset and 16% precision in cross-lingual settings with Yoruba, with performance improving via English pretraining.
This study investigates the use of Visually Grounded Speech (VGS) models for keyword localisation in speech. The study focusses on two main research questions: (1) Is keyword localisation possible with VGS models and (2) Can keyword localisation be done cross-lingually in a real low-resource setting? Four methods for localisation are proposed and evaluated on an English dataset, with the best-performing method achieving an accuracy of 57%. A new dataset containing spoken captions in Yoruba language is also collected and released for cross-lingual keyword localisation. The cross-lingual model obtains a precision of 16% in actual keyword localisation and this performance can be improved by initialising from a model pretrained on English data. The study presents a detailed analysis of the model's success and failure modes and highlights the challenges of using VGS models for keyword localisation in low-resource settings.