A Free-Energy Principle for Representation Learning
This provides a theoretical framework for representation learning in transfer learning, though it appears incremental as it builds on existing thermodynamic connections.
The paper tackles the problem of characterizing representation quality for transfer learning by connecting machine learning with thermodynamics, showing that information-theoretic functionals lie on a convex equilibrium surface. It demonstrates how dynamical processes on this surface can transfer representations between datasets while keeping classification loss constant, with experimental validation on standard image-classification datasets.
This paper employs a formal connection of machine learning with thermodynamics to characterize the quality of learnt representations for transfer learning. We discuss how information-theoretic functional such as rate, distortion and classification loss of a model lie on a convex, so-called equilibrium surface.We prescribe dynamical processes to traverse this surface under constraints, e.g., an iso-classification process that trades off rate and distortion to keep the classification loss unchanged. We demonstrate how this process can be used for transferring representations from a source dataset to a target dataset while keeping the classification loss constant. Experimental validation of the theoretical results is provided on standard image-classification datasets.