A Computational Acquisition Model for Multimodal Word Categorization
This addresses the challenge of creating more faithful models of child language acquisition for researchers in computational linguistics and developmental psychology, though it is incremental as it builds on existing self-supervised methods.
The paper tackles the problem of developing a computational model for child language acquisition that avoids reliance on pre-defined object categories in vision models, by proposing a cognitively-inspired, multimodal model trained on naturalistic image-caption pairs using cross-modal self-supervision. The result is that the model learns word categories and object recognition abilities, showing trends similar to those in developmental studies.
Recent advances in self-supervised modeling of text and images open new opportunities for computational models of child language acquisition, which is believed to rely heavily on cross-modal signals. However, prior studies have been limited by their reliance on vision models trained on large image datasets annotated with a pre-defined set of depicted object categories. This is (a) not faithful to the information children receive and (b) prohibits the evaluation of such models with respect to category learning tasks, due to the pre-imposed category structure. We address this gap, and present a cognitively-inspired, multimodal acquisition model, trained from image-caption pairs on naturalistic data using cross-modal self-supervision. We show that the model learns word categories and object recognition abilities, and presents trends reminiscent of those reported in the developmental literature. We make our code and trained models public for future reference and use.