Do Blind Spots Matter for Word-Referent Mapping? A Computational Study with Infant Egocentric Video
This addresses the challenge of early language acquisition for developmental psychology and AI, but it is incremental as it modifies an existing method with a biologically inspired twist.
The study tackled the problem of how infants learn word-referent mappings by proposing a biologically plausible masking strategy based on human eye blind spots, using a masked autoencoder and contrastive learning on infant egocentric video data, and found it to be at least as effective as random masking.
Typically, children start to learn their first words between 6 and 9 months, linking spoken utterances to their visual referents. Without prior knowledge, a word encountered for the first time can be interpreted in countless ways; it might refer to any of the objects in the environment, their components, or attributes. Using longitudinal, egocentric, and ecologically valid data from the experience of one child, in this work, we propose a self-supervised and biologically plausible strategy to learn strong visual representations. Our masked autoencoder-based visual backbone incorporates knowledge about the blind spot in human eyes to define a novel masking strategy. This mask and reconstruct approach attempts to mimic the way the human brain fills the gaps in the eyes' field of view. This represents a significant shift from standard random masking strategies, which are difficult to justify from a biological perspective. The pretrained encoder is utilized in a contrastive learning-based video-text model capable of acquiring word-referent mappings. Extensive evaluation suggests that the proposed biologically plausible masking strategy is at least as effective as random masking for learning word-referent mappings from cross-situational and temporally extended episodes.