CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning
This work addresses representation learning from unlabeled data streams for continual learning, offering a novel memory mechanism that improves performance over existing methods.
CLIMB introduces a hierarchical centroid-based memory for online continual self-supervised learning, combining replay and regularization to outperform state-of-the-art methods on Split CIFAR-100 and Split ImageNet-100 benchmarks.
Online Continual Self-Supervised Learning (OCSSL) aims to learn representations from a continuous stream of unlabeled data, without knowledge of task boundaries and under memory constraints. Existing methods rely either on replay buffers that exploit latent space structure, or on regularization alone. We present CLIMB (Continual Learning with Intelligent Memory Bank), which combines both simultaneously. Our method introduces a hierarchical centroid-based memory, bounded in total number of stored images, combined with knowledge distillation on replayed examples to limit representation drift. The memory groups similar images into centroids, providing hard-to-discriminate examples for contrastive learning while covering the diversity of observed distributions. Experiments on Split CIFAR-100 and Split ImageNet-100, on standard benchmarks from the state-of-the-art as well as a new protocol with irregular task distributions show that CLIMB outperforms state-of-the-art OCSSL methods.