Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era
This provides insights into unimodal learning and modality complementarity for AI researchers, though it is incremental as it revisits existing methods with new data.
The paper investigated how well large-scale models represent semantic feature norms of concrete object concepts, finding that multimodal image encoders slightly outperform language-only models, and image-only encoders perform comparably even on non-visual attributes.
Human learning and conceptual representation is grounded in sensorimotor experience, in contrast to state-of-the-art foundation models. In this paper, we investigate how well such large-scale models, trained on vast quantities of data, represent the semantic feature norms of concrete object concepts, e.g. a ROSE is red, smells sweet, and is a flower. More specifically, we use probing tasks to test which properties of objects these models are aware of. We evaluate image encoders trained on image data alone, as well as multimodally-trained image encoders and language-only models, on predicting an extended denser version of the classic McRae norms and the newer Binder dataset of attribute ratings. We find that multimodal image encoders slightly outperform language-only approaches, and that image-only encoders perform comparably to the language models, even on non-visual attributes that are classified as "encyclopedic" or "function". These results offer new insights into what can be learned from pure unimodal learning, and the complementarity of the modalities.