Eigenvalues of Autoencoders in Training and at Initialization
This work provides insights into the training dynamics of autoencoders, which is relevant for researchers in deep learning and neural network theory, though it is incremental as it builds on existing theoretical work.
The paper investigates how the eigenvalue distributions of autoencoders' Jacobian matrices evolve during early training on MNIST, finding that untrained autoencoders have qualitatively different distributions from fully trained ones, but these distributions rapidly converge to resemble fully trained ones even after a few epochs.
In this paper, we investigate the evolution of autoencoders near their initialization. In particular, we study the distribution of the eigenvalues of the Jacobian matrices of autoencoders early in the training process, training on the MNIST data set. We find that autoencoders that have not been trained have eigenvalue distributions that are qualitatively different from those which have been trained for a long time ($>$100 epochs). Additionally, we find that even at early epochs, these eigenvalue distributions rapidly become qualitatively similar to those of the fully trained autoencoders. We also compare the eigenvalues at initialization to pertinent theoretical work on the eigenvalues of random matrices and the products of such matrices.