Target-Embedding Autoencoders for Supervised Representation Learning
This work addresses generalization challenges in supervised learning for high-dimensional target spaces, extending autoencoder methods to new applications, but it is incremental as it builds on existing autoencoder frameworks.
The paper tackles improving generalization in supervised learning with high-dimensional targets by proposing target-embedding autoencoders (TEA), which learn latent representations optimized for predictability from features and prediction of targets, and demonstrates theoretical generalization guarantees and empirical advantages in sequence forecasting.
Autoencoder-based learning has emerged as a staple for disciplining representations in unsupervised and semi-supervised settings. This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of target-embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets---encoding the prior that variations in targets are driven by a compact set of underlying factors. As our theoretical contribution, we provide a guarantee of generalization for linear TEAs by demonstrating uniform stability, interpreting the benefit of the auxiliary reconstruction task as a form of regularization. As our empirical contribution, we extend validation of this approach beyond existing static classification applications to multivariate sequence forecasting, verifying their advantage on both linear and nonlinear recurrent architectures---thereby underscoring the further generality of this framework beyond feedforward instantiations.