Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR
For researchers in continual self-supervised learning, this work addresses a critical stability-plasticity trade-off that causes performance drops, providing both diagnostic metrics and a practical solution.
This paper identifies and explains a collapse phenomenon in Online Continual Self-Supervised Learning (OCSSL) called Latent Rehearsal Decay, where excessive stability from replay buffers degrades latent representations. The proposed method SOLAR uses online proxies to manage buffer plasticity and achieves state-of-the-art performance on OCSSL benchmarks, balancing convergence speed and final accuracy.
This paper explores Online Continual Self-Supervised Learning (OCSSL), a scenario in which models learn from continuous streams of unlabeled, non-stationary data, where methods typically employ replay and fast convergence is a central desideratum. We find that OCSSL requires particular attention to the stability-plasticity trade-off: stable methods (e.g. replay with Reservoir sampling) are able to converge faster compared to plastic ones (e.g. FIFO buffer), but incur in performance drops under certain conditions. We explain this collapse phenomenon with the Latent Rehearsal Decay hypothesis, which attributes it to latent space degradation under excessive stability of replay. We introduce two metrics (Overlap and Deviation) that diagnose latent degradation and correlate with accuracy declines. Building on these insights, we propose SOLAR, which leverages efficient online proxies of Deviation to guide buffer management and incorporates an explicit Overlap loss, allowing SOLAR to adaptively managing plasticity. Experiments demonstrate that SOLAR achieves state-of-the-art performance on OCSSL vision benchmarks, with both high convergence speed and final performance.