Modular Autoencoders for Ensemble Feature Extraction
This work addresses representation learning for machine learning practitioners, offering an incremental improvement in ensemble feature extraction.
The paper tackles the problem of learning diverse and complementary representations from unlabeled data for supervised tasks by introducing Modular Autoencoders (MAEs), showing that an optimal trade-off between independent small autoencoders and a single monolithic encoding outperforms baselines on six benchmark datasets.
We introduce the concept of a Modular Autoencoder (MAE), capable of learning a set of diverse but complementary representations from unlabelled data, that can later be used for supervised tasks. The learning of the representations is controlled by a trade off parameter, and we show on six benchmark datasets the optimum lies between two extremes: a set of smaller, independent autoencoders each with low capacity, versus a single monolithic encoding, outperforming an appropriate baseline. In the present paper we explore the special case of linear MAE, and derive an SVD-based algorithm which converges several orders of magnitude faster than gradient descent.