Sparse Coding with Multi-Layer Decoders using Variance Regularization
This addresses a technical bottleneck in sparse coding for computer vision applications, though it appears incremental as it modifies regularization rather than introducing a new paradigm.
The paper tackles the problem of dictionary collapse in sparse coding by proposing a variance regularization method that directly regularizes latent codes instead of the decoder, and shows this approach works with multi-layer decoders for better reconstructions. Results include higher quality reconstructions with sparser representations on MNIST and natural image patches, and improved performance in downstream denoising and classification tasks.
Sparse representations of images are useful in many computer vision applications. Sparse coding with an $l_1$ penalty and a learned linear dictionary requires regularization of the dictionary to prevent a collapse in the $l_1$ norms of the codes. Typically, this regularization entails bounding the Euclidean norms of the dictionary's elements. In this work, we propose a novel sparse coding protocol which prevents a collapse in the codes without the need to regularize the decoder. Our method regularizes the codes directly so that each latent code component has variance greater than a fixed threshold over a set of sparse representations for a given set of inputs. Furthermore, we explore ways to effectively train sparse coding systems with multi-layer decoders since they can model more complex relationships than linear dictionaries. In our experiments with MNIST and natural image patches, we show that decoders learned with our approach have interpretable features both in the linear and multi-layer case. Moreover, we show that sparse autoencoders with multi-layer decoders trained using our variance regularization method produce higher quality reconstructions with sparser representations when compared to autoencoders with linear dictionaries. Additionally, sparse representations obtained with our variance regularization approach are useful in the downstream tasks of denoising and classification in the low-data regime.