Non-Parametric Representation Learning with Kernels
This work addresses the lack of non-neural methods in representation learning, offering an alternative for researchers and practitioners, though it is incremental as it adapts existing kernel techniques to a new context.
The paper tackles the problem of representation learning by introducing kernel-based methods for self-supervised and autoencoder models, showing that these approaches can be expressed via kernel matrices and achieve competitive performance in small data regimes compared to neural networks.
Unsupervised and self-supervised representation learning has become popular in recent years for learning useful features from unlabelled data. Representation learning has been mostly developed in the neural network literature, and other models for representation learning are surprisingly unexplored. In this work, we introduce and analyze several kernel-based representation learning approaches: Firstly, we define two kernel Self-Supervised Learning (SSL) models using contrastive loss functions and secondly, a Kernel Autoencoder (AE) model based on the idea of embedding and reconstructing data. We argue that the classical representer theorems for supervised kernel machines are not always applicable for (self-supervised) representation learning, and present new representer theorems, which show that the representations learned by our kernel models can be expressed in terms of kernel matrices. We further derive generalisation error bounds for representation learning with kernel SSL and AE, and empirically evaluate the performance of these methods in both small data regimes as well as in comparison with neural network based models.