Identifying concept libraries from language about object structure
This work addresses how humans conceptualize and communicate object parts, which is incremental in advancing understanding of visual cognition and language.
The researchers tackled the problem of identifying the part concepts people use to describe object structure by analyzing natural language descriptions of 2,000 procedurally generated objects, discovering that people favor a lexicon that balances concise object descriptions with minimal lexicon size.
Our understanding of the visual world goes beyond naming objects, encompassing our ability to parse objects into meaningful parts, attributes, and relations. In this work, we leverage natural language descriptions for a diverse set of 2K procedurally generated objects to identify the parts people use and the principles leading these parts to be favored over others. We formalize our problem as search over a space of program libraries that contain different part concepts, using tools from machine translation to evaluate how well programs expressed in each library align to human language. By combining naturalistic language at scale with structured program representations, we discover a fundamental information-theoretic tradeoff governing the part concepts people name: people favor a lexicon that allows concise descriptions of each object, while also minimizing the size of the lexicon itself.