Visually grounded few-shot word learning in low-resource settings
This work addresses the challenge of multimodal few-shot learning for low-resource languages, offering a method to improve word acquisition with limited data, though it is incremental by building on prior visually grounded models.
The paper tackles the problem of learning new words from few word-image pairs by proposing a visually grounded speech model that mines unsupervised training pairs and uses word-to-image attention, achieving better performance with fewer shots on an English benchmark and demonstrating applicability to Yorùbá, a low-resource language.
We propose a visually grounded speech model that learns new words and their visual depictions from just a few word-image example pairs. Given a set of test images and a spoken query, we ask the model which image depicts the query word. Previous work has simplified this few-shot learning problem by either using an artificial setting with digit word-image pairs or by using a large number of examples per class. Moreover, all previous studies were performed using English speech-image data. We propose an approach that can work on natural word-image pairs but with less examples, i.e. fewer shots, and then illustrate how this approach can be applied for multimodal few-shot learning in a real low-resource language, Yorùbá. Our approach involves using the given word-image example pairs to mine new unsupervised word-image training pairs from large collections of unlabelled speech and images. Additionally, we use a word-to-image attention mechanism to determine word-image similarity. With this new model, we achieve better performance with fewer shots than previous approaches on an existing English benchmark. Many of the model's mistakes are due to confusion between visual concepts co-occurring in similar contexts. The experiments on Yorùbá show the benefit of transferring knowledge from a multimodal model trained on a larger set of English speech-image data.