Principal Bit Analysis: Autoencoding with Schur-Concave Loss
This work addresses compression and representation learning for data processing, offering a theoretical foundation and improved performance, though it appears incremental as it builds on existing autoencoder frameworks.
The paper tackles the problem of designing optimal linear autoencoders with quantized or noisy latent variables under Schur-concave constraints, proving that decomposing the source into principal components is optimal. It introduces Principal Bit Analysis (PBA) as a practical fixed-rate compressor that outperforms existing algorithms.
We consider a linear autoencoder in which the latent variables are quantized, or corrupted by noise, and the constraint is Schur-concave in the set of latent variances. Although finding the optimal encoder/decoder pair for this setup is a nonconvex optimization problem, we show that decomposing the source into its principal components is optimal. If the constraint is strictly Schur-concave and the empirical covariance matrix has only simple eigenvalues, then any optimal encoder/decoder must decompose the source in this way. As one application, we consider a strictly Schur-concave constraint that estimates the number of bits needed to represent the latent variables under fixed-rate encoding, a setup that we call \emph{Principal Bit Analysis (PBA)}. This yields a practical, general-purpose, fixed-rate compressor that outperforms existing algorithms. As a second application, we show that a prototypical autoencoder-based variable-rate compressor is guaranteed to decompose the source into its principal components.