Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers
This work addresses the problem of efficient replay selection for researchers and practitioners in continual learning, offering a practical, drop-in enhancement for existing methods, though it is incremental as it builds on prior embedding-based approaches.
The paper tackled catastrophic forgetting in continual learning with small replay buffers by proposing a graph-based method that integrates supervised and self-supervised embeddings for sample selection, resulting in consistent improvements over state-of-the-art strategies, particularly in low-memory regimes, such as outperforming baselines on CIFAR-100 and TinyImageNet without adding parameters or increasing replay volume.
Catastrophic forgetting remains a key challenge in Continual Learning (CL). In replay-based CL with severe memory constraints, performance critically depends on the sample selection strategy for the replay buffer. Most existing approaches construct memory buffers using embeddings learned under supervised objectives. However, class-agnostic, self-supervised representations often encode rich, class-relevant semantics that are overlooked. We propose a new method, Multiple Embedding Replay Selection, MERS, which replaces the buffer selection module with a graph-based approach that integrates both supervised and self-supervised embeddings. Empirical results show consistent improvements over SOTA selection strategies across a range of continual learning algorithms, with particularly strong gains in low-memory regimes. On CIFAR-100 and TinyImageNet, MERS outperforms single-embedding baselines without adding model parameters or increasing replay volume, making it a practical, drop-in enhancement for replay-based continual learning.