Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery
This addresses a challenging open-set recognition problem for computer vision and machine learning applications, but it is incremental as it builds on existing multi-modal methods.
The paper tackles the problem of Generalized Category Discovery (GCD) by proposing a multi-modal representation learning framework that emphasizes intra-modality alignment to improve cross-modality representations, achieving superior performance on benchmark datasets.
Generalized Category Discovery (GCD) aims to identify both known and unknown categories, with only partial labels given for the known categories, posing a challenging open-set recognition problem. State-of-the-art approaches for GCD task are usually built on multi-modality representation learning, which is heavily dependent upon inter-modality alignment. However, few of them cast a proper intra-modality alignment to generate a desired underlying structure of representation distributions. In this paper, we propose a novel and effective multi-modal representation learning framework for GCD via Semi-Supervised Rate Reduction, called SSR$^2$-GCD, to learn cross-modality representations with desired structural properties based on emphasizing to properly align intra-modality relationships. Moreover, to boost knowledge transfer, we integrate prompt candidates by leveraging the inter-modal alignment offered by Vision Language Models. We conduct extensive experiments on generic and fine-grained benchmark datasets demonstrating superior performance of our approach.