Learning finite-dimensional coding schemes with nonlinear reconstruction maps
This work addresses unsupervised representation learning problems by extending theoretical frameworks to nonlinear settings, offering a foundation for analyzing and learning coding schemes in machine learning.
The paper generalizes finite-dimensional lossy coding schemes to include nonlinear reconstruction maps, connecting them to approximate generative modeling with optimal transport, and provides generalization bounds for learning such schemes from finite samples, illustrated with deep neural networks.
This paper generalizes the Maurer--Pontil framework of finite-dimensional lossy coding schemes to the setting where a high-dimensional random vector is mapped to an element of a compact set of latent representations in a lower-dimensional Euclidean space, and the reconstruction map belongs to a given class of nonlinear maps. Under this setup, which encompasses a broad class of unsupervised representation learning problems, we establish a connection to approximate generative modeling under structural constraints using the tools from the theory of optimal transportation. Next, we consider problem of learning a coding scheme on the basis of a finite collection of training samples and present generalization bounds that hold with high probability. We then illustrate the general theory in the setting where the reconstruction maps are implemented by deep neural nets.