Attributes as Operators: Factorizing Unseen Attribute-Object Compositions
This addresses the limitation in computer vision where models fail to generalize to new combinations of attributes and objects, which is crucial for real-world applications like robotics and image understanding.
The paper tackles the problem of recognizing unseen attribute-object compositions in visual recognition by modeling attributes as operators that factor out attributes from objects, achieving significant improvements over state-of-the-art methods on challenging datasets.
We present a new approach to modeling visual attributes. Prior work casts attributes in a similar role as objects, learning a latent representation where properties (e.g., sliced) are recognized by classifiers much in the way objects (e.g., apple) are. However, this common approach fails to separate the attributes observed during training from the objects with which they are composed, making it ineffectual when encountering new attribute-object compositions. Instead, we propose to model attributes as operators. Our approach learns a semantic embedding that explicitly factors out attributes from their accompanying objects, and also benefits from novel regularizers expressing attribute operators' effects (e.g., blunt should undo the effects of sharp). Not only does our approach align conceptually with the linguistic role of attributes as modifiers, but it also generalizes to recognize unseen compositions of objects and attributes. We validate our approach on two challenging datasets and demonstrate significant improvements over the state-of-the-art. In addition, we show that not only can our model recognize unseen compositions robustly in an open-world setting, it can also generalize to compositions where objects themselves were unseen during training.