Kiki or Bouba? Sound Symbolism in Vision-and-Language Models
This provides a novel computational method to analyze sound symbolism, which is incremental in applying existing models to a new cognitive phenomenon.
The study investigated whether vision-and-language models like CLIP and Stable Diffusion exhibit sound symbolism, similar to the kiki-bouba effect in humans, and found strong evidence that they do.
Although the mapping between sound and meaning in human language is assumed to be largely arbitrary, research in cognitive science has shown that there are non-trivial correlations between particular sounds and meanings across languages and demographic groups, a phenomenon known as sound symbolism. Among the many dimensions of meaning, sound symbolism is particularly salient and well-demonstrated with regards to cross-modal associations between language and the visual domain. In this work, we address the question of whether sound symbolism is reflected in vision-and-language models such as CLIP and Stable Diffusion. Using zero-shot knowledge probing to investigate the inherent knowledge of these models, we find strong evidence that they do show this pattern, paralleling the well-known kiki-bouba effect in psycholinguistics. Our work provides a novel method for demonstrating sound symbolism and understanding its nature using computational tools. Our code will be made publicly available.