Cross-Modal Alignment Learning of Vision-Language Conceptual Systems
This work addresses the challenge of cross-modal alignment for AI systems, offering a novel approach to self-supervised learning, though it appears incremental in its method development.
The paper tackles the problem of learning aligned vision-language conceptual systems without explicit supervision, inspired by infant word learning, and shows that the proposed model significantly outperforms baselines in tasks like object-to-word mapping and zero-shot learning, with topological alignment between modalities.
Human infants learn the names of objects and develop their own conceptual systems without explicit supervision. In this study, we propose methods for learning aligned vision-language conceptual systems inspired by infants' word learning mechanisms. The proposed model learns the associations of visual objects and words online and gradually constructs cross-modal relational graph networks. Additionally, we also propose an aligned cross-modal representation learning method that learns semantic representations of visual objects and words in a self-supervised manner based on the cross-modal relational graph networks. It allows entities of different modalities with conceptually the same meaning to have similar semantic representation vectors. We quantitatively and qualitatively evaluate our method, including object-to-word mapping and zero-shot learning tasks, showing that the proposed model significantly outperforms the baselines and that each conceptual system is topologically aligned.