Can Contrastive Learning Refine Embeddings
This work addresses the need for better embedding refinement in machine learning, but it is incremental as it builds on existing contrastive learning methods.
The paper tackles the problem of refining input embeddings for downstream tasks by introducing SIMSKIP, a contrastive learning framework that uses output embeddings from encoder models as input, and shows it improves performance on various datasets.
Recent advancements in contrastive learning have revolutionized self-supervised representation learning and achieved state-of-the-art performance on benchmark tasks. While most existing methods focus on applying contrastive learning to input data modalities such as images, natural language sentences, or networks, they overlook the potential of utilizing outputs from previously trained encoders. In this paper, we introduce SIMSKIP, a novel contrastive learning framework that specifically refines input embeddings for downstream tasks. Unlike traditional unsupervised learning approaches, SIMSKIP takes advantage of the output embeddings of encoder models as its input. Through theoretical analysis, we provide evidence that applying SIMSKIP does not result in larger upper bounds on downstream task errors than those of the original embeddings, which serve as SIMSKIP's input. Experimental results on various open datasets demonstrate that the embeddings produced by SIMSKIP improve performance on downstream tasks.