Beyond Linear and Overcomplete Regimes: A Mean-Field Analysis of Bottleneck Autoencoders
It offers a theoretical foundation for understanding nonlinear autoencoders with bottlenecks, addressing a gap in the literature that was limited to linear models or bottleneck-free settings.
This work provides a theoretical analysis of nonlinear autoencoders with a finite bottleneck in the mean-field regime, deriving explicit learning dynamics and showing that finite-width networks closely track the mean-field risk trajectory and converge to the optimal solution.
Autoencoders (AEs) learn low-dimensional representations by mapping data into a latent space while minimizing reconstruction error. Despite their empirical success, theoretical understanding remains limited and largely restricted to linear models or settings without a bottleneck. In this work, we study nonlinear AEs with a fixed finite-dimensional bottleneck in the mean-field (MF) regime. We derive explicit MF learning dynamics for both encoder and decoder, providing a tractable characterization of training in the nonlinear setting. We show that, over finite time horizons, the empirical risk of finite-width networks trained with stochastic gradient descent closely tracks the MF risk trajectory with high probability. At optimality, we further establish that the finite-width risk converges to the MF optimum, demonstrating that finite networks are sufficiently expressive to approximate the infinite-width solution.