Learning to hash with semantic similarity metrics and empirical KL divergence
This work addresses efficiency and accuracy issues in nearest neighbor search for massive image databases, but it is incremental as it builds on existing learning-to-hash methods with specific improvements.
The paper tackled the problem of learning binary hash codes for image retrieval by addressing three drawbacks: lack of semantic similarity capture, information loss from rounding, and trivial solutions from class-wise similarity. The result was improved retrieval performance, demonstrated on CIFAR-100, ImageNet, and Conceptual Captions datasets using semantic metrics.
Learning to hash is an efficient paradigm for exact and approximate nearest neighbor search from massive databases. Binary hash codes are typically extracted from an image by rounding output features from a CNN, which is trained on a supervised binary similar/ dissimilar task. Drawbacks of this approach are: (i) resulting codes do not necessarily capture semantic similarity of the input data (ii) rounding results in information loss, manifesting as decreased retrieval performance and (iii) Using only class-wise similarity as a target can lead to trivial solutions, simply encoding classifier outputs rather than learning more intricate relations, which is not detected by most performance metrics. We overcome (i) via a novel loss function encouraging the relative hash code distances of learned features to match those derived from their targets. We address (ii) via a differentiable estimate of the KL divergence between network outputs and a binary target distribution, resulting in minimal information loss when the features are rounded to binary. Finally, we resolve (iii) by focusing on a hierarchical precision metric. Efficiency of the methods is demonstrated with semantic image retrieval on the CIFAR-100, ImageNet and Conceptual Captions datasets, using similarities inferred from the WordNet label hierarchy or sentence embeddings.