Describe me an Aucklet: Generating Grounded Perceptual Category Descriptions
This addresses the under-investigated issue of learning abstract perceptual concepts in language-and-vision, relevant for multi-modal NLP representation learning, but it is incremental as it builds on existing methods for testing grounding.
The paper tackles the problem of generating and interpreting descriptions of visual categories in multi-modal language models, introducing a framework to test category-level perceptual grounding and showing that communicative success reveals performance issues in generation models not captured by traditional metrics.
Human speakers can generate descriptions of perceptual concepts, abstracted from the instance-level. Moreover, such descriptions can be used by other speakers to learn provisional representations of those concepts. Learning and using abstract perceptual concepts is under-investigated in the language-and-vision field. The problem is also highly relevant to the field of representation learning in multi-modal NLP. In this paper, we introduce a framework for testing category-level perceptual grounding in multi-modal language models. In particular, we train separate neural networks to generate and interpret descriptions of visual categories. We measure the communicative success of the two models with the zero-shot classification performance of the interpretation model, which we argue is an indicator of perceptual grounding. Using this framework, we compare the performance of prototype- and exemplar-based representations. Finally, we show that communicative success exposes performance issues in the generation model, not captured by traditional intrinsic NLG evaluation metrics, and argue that these issues stem from a failure to properly ground language in vision at the category level.