Towards Effective Collaborative Learning in Long-Tailed Recognition
This work addresses class imbalance in real-world data for machine learning applications, offering incremental improvements in collaborative learning methods.
The paper tackles the problem of imbalanced knowledge transfer in collaborative learning for long-tailed recognition, proposing an Effective Collaborative Learning (ECL) framework that integrates re-weighted distillation loss and contrastive proxy tasks, achieving state-of-the-art performance on four standard datasets.
Real-world data usually suffers from severe class imbalance and long-tailed distributions, where minority classes are significantly underrepresented compared to the majority ones. Recent research prefers to utilize multi-expert architectures to mitigate the model uncertainty on the minority, where collaborative learning is employed to aggregate the knowledge of experts, i.e., online distillation. In this paper, we observe that the knowledge transfer between experts is imbalanced in terms of class distribution, which results in limited performance improvement of the minority classes. To address it, we propose a re-weighted distillation loss by comparing two classifiers' predictions, which are supervised by online distillation and label annotations, respectively. We also emphasize that feature-level distillation will significantly improve model performance and increase feature robustness. Finally, we propose an Effective Collaborative Learning (ECL) framework that integrates a contrastive proxy task branch to further improve feature quality. Quantitative and qualitative experiments on four standard datasets demonstrate that ECL achieves state-of-the-art performance and the detailed ablation studies manifest the effectiveness of each component in ECL.