i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
This addresses the need for more generalizable contrastive learning methods across diverse domains, though it appears incremental as it builds on existing contrastive frameworks.
The paper tackles the problem of contrastive representation learning being dependent on domain-specific data augmentations by proposing i-Mix, a domain-agnostic regularization strategy that mixes data in input and virtual label spaces, resulting in consistent improvements across image, speech, and tabular data domains.
Contrastive representation learning has shown to be effective to learn representations from unlabeled data. However, much progress has been made in vision domains relying on data augmentations carefully designed using domain knowledge. In this work, we propose i-Mix, a simple yet effective domain-agnostic regularization strategy for improving contrastive representation learning. We cast contrastive learning as training a non-parametric classifier by assigning a unique virtual class to each data in a batch. Then, data instances are mixed in both the input and virtual label spaces, providing more augmented data during training. In experiments, we demonstrate that i-Mix consistently improves the quality of learned representations across domains, including image, speech, and tabular data. Furthermore, we confirm its regularization effect via extensive ablation studies across model and dataset sizes. The code is available at https://github.com/kibok90/imix.