Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery
For researchers in multi-modal learning and open-world recognition, this work provides a plug-and-play enhancement that improves GCD performance by addressing cross-modal synergy limitations.
The paper tackles the problem of generalized category discovery (GCD) in multi-modal settings, where existing dual-branch methods suffer from coarse cross-modal synergy. The proposed SDBA framework achieves state-of-the-art performance on six benchmarks, with consistent improvements over baselines like GET and TextGCD.
Generalized Category Discovery (GCD) aims to classify old categories and discover new ones from unlabeled data. Recent multi-modal approaches introduce retrieved or synthesized texts into a dual-branch architecture to provide semantic cues complementary to visual features. However, the cross-modal synergy in existing dual-branch methods remains coarse and incomplete: the two modalities are encoded independently with the bias and noise in the derived text left unaddressed during encoding, and existing mutual learning strategies operate only on global class-level anchors, lacking fine-grained relational supervision. To address these limitations, we propose the Synergistic Dual-Branch Adaptation (SDBA) framework, which serves as a plug-and-play enhancement compatible with existing dual-branch methods such as GET and TextGCD. SDBA comprises two components: the cross-modal synergistic adapter inserts lightweight adapters into both branches and further injects visual information into the text adapter at each encoder layer to enhance text feature learning during encoding; the neighborhood mutual learning module enforces consistent local neighborhood distributions between the two branches via bidirectional KL divergence, providing fine-grained relational supervision for both old and new classes. Extensive experiments on six benchmarks demonstrate state-of-the-art performance, and consistent improvements on different baselines validate the broad scalability of the proposed framework.