Using Multimodal Deep Neural Networks to Disentangle Language from Visual Aesthetics
This addresses a fundamental question in cognitive science and AI about the role of perception versus language in aesthetic judgments, though it is incremental in its methodological approach.
The study tackled the problem of disentangling perceptual and linguistic contributions to visual aesthetic experiences by using deep neural network models to predict human beauty ratings of images. The results showed that unimodal vision models accounted for most of the explainable variance, with language-aligned models providing only small gains.
When we experience a visual stimulus as beautiful, how much of that experience derives from perceptual computations we cannot describe versus conceptual knowledge we can readily translate into natural language? Disentangling perception from language in visually-evoked affective and aesthetic experiences through behavioral paradigms or neuroimaging is often empirically intractable. Here, we circumnavigate this challenge by using linear decoding over the learned representations of unimodal vision, unimodal language, and multimodal (language-aligned) deep neural network (DNN) models to predict human beauty ratings of naturalistic images. We show that unimodal vision models (e.g. SimCLR) account for the vast majority of explainable variance in these ratings. Language-aligned vision models (e.g. SLIP) yield small gains relative to unimodal vision. Unimodal language models (e.g. GPT2) conditioned on visual embeddings to generate captions (via CLIPCap) yield no further gains. Caption embeddings alone yield less accurate predictions than image and caption embeddings combined (concatenated). Taken together, these results suggest that whatever words we may eventually find to describe our experience of beauty, the ineffable computations of feedforward perception may provide sufficient foundation for that experience.