The Missing Link: Finding label relations across datasets
This addresses the challenge of dataset interoperability in computer vision, which is incremental as it builds on existing methods for label analysis.
The paper tackles the problem of automatically discovering visual-semantic relations between class labels across different computer vision datasets, such as identity or parent/child links, and shows that their method can effectively identify these relations and their types. It applies this to applications like understanding label relations and predicting transfer learning gains.
Computer vision is driven by the many datasets available for training or evaluating novel methods. However, each dataset has a different set of class labels, visual definition of classes, images following a specific distribution, annotation protocols, etc. In this paper we explore the automatic discovery of visual-semantic relations between labels across datasets. We aim to understand how instances of a certain class in a dataset relate to the instances of another class in another dataset. Are they in an identity, parent/child, overlap relation? Or is there no link between them at all? To find relations between labels across datasets, we propose methods based on language, on vision, and on their combination. We show that we can effectively discover label relations across datasets, as well as their type. We apply our method to four applications: understand label relations, identify missing aspects, increase label specificity, and predict transfer learning gains. We conclude that label relations cannot be established by looking at the names of classes alone, as they depend strongly on how each of the datasets was constructed.