Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settings
It addresses a theoretical gap for researchers in machine learning, providing more realistic analysis for practical scenarios where data recycling is common, but it is incremental as it extends existing work to non-i.i.d. cases.
The paper tackles the lack of theoretical understanding of generalization in contrastive representation learning under non-i.i.d. settings, deriving bounds that show the required samples per class scale logarithmically with the covering number of feature representations.
Contrastive Representation Learning (CRL) has achieved impressive success in various domains in recent years. Nevertheless, the theoretical understanding of the generalization behavior of CRL has remained limited. Moreover, to the best of our knowledge, the current literature only analyzes generalization bounds under the assumption that the data tuples used for contrastive learning are independently and identically distributed. However, in practice, we are often limited to a fixed pool of reusable labeled data points, making it inevitable to recycle data across tuples to create sufficiently large datasets. Therefore, the tuple-wise independence condition imposed by previous works is invalidated. In this paper, we provide a generalization analysis for the CRL framework under non-$i.i.d.$ settings that adheres to practice more realistically. Drawing inspiration from the literature on U-statistics, we derive generalization bounds which indicate that the required number of samples in each class scales as the logarithm of the covering number of the class of learnable feature representations associated to that class. Next, we apply our main results to derive excess risk bounds for common function classes such as linear maps and neural networks.