DUEL: Adaptive Duplicate Elimination on Working Memory for Self-Supervised Learning
It addresses a specific issue in SSL for real-world data like crawled or robot-gathered observations, which is incremental as it builds on known collision problems.
The paper tackles the problem of performance degradation in Self-Supervised Learning (SSL) caused by class collisions from duplicate data, and introduces the DUEL framework with adaptive duplicate elimination to stabilize the model and prevent downstream task degradation.
In Self-Supervised Learning (SSL), it is known that frequent occurrences of the collision in which target data and its negative samples share the same class can decrease performance. Especially in real-world data such as crawled data or robot-gathered observations, collisions may occur more often due to the duplicates in the data. To deal with this problem, we claim that sampling negative samples from the adaptively debiased distribution in the memory makes the model more stable than sampling from a biased dataset directly. In this paper, we introduce a novel SSL framework with adaptive Duplicate Elimination (DUEL) inspired by the human working memory. The proposed framework successfully prevents the downstream task performance from degradation due to a dramatic inter-class imbalance.