Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generation
This work addresses the challenge of producing Gaussian codes in autoencoders for applications like inverse problems, representing an incremental improvement through a novel regularization method.
The authors tackled the problem of generating Gaussian-like codes in autoencoders by introducing a novel regularization scheme based on matricial free energy, which minimizes a loss function derived from random matrix theory to sculpt the singular value distribution of the code matrix. Empirical results show this approach yields Gaussian codes that generalize across training and test sets, with applications demonstrated in underdetermined inverse problems.
We introduce a novel regularization scheme for autoencoders based on matricial free energy. Our approach defines a differentiable loss function in terms of the singular values of the code matrix (code dimension x batch size). From the standpoint of free probability an d random matrix theory, this loss achieves its minimum when the singular value distribution of the code matrix coincides with that of an appropriately sculpted random metric with i.i.d. Gaussian entries. Empirical simulations demonstrate that minimizing the negative matricial free energy through standard stochastic gradient-based training yields Gaussian-like codes that generalize across training and test sets. Building on this foundation, we propose a matricidal free energy maximizing autoencoder that reliably produces Gaussian codes and show its application to underdetermined inverse problems.