Joint Optimization of an Autoencoder for Clustering and Embedding
This work provides an incremental improvement for researchers and practitioners in unsupervised learning by enabling simultaneous optimization of clustering and embedding in deep autoencoders.
This paper proposes a method for jointly optimizing an autoencoder for clustering and embedding, addressing the limitation of alternating optimization in deep embedded clustering. By rephrasing the objective function of a class of Gaussian mixture models as an autoencoder loss, they integrate a 'clustering module' into a deep autoencoder, leading to improved performance over related baselines on several datasets.
Deep embedded clustering has become a dominating approach to unsupervised categorization of objects with deep neural networks. The optimization of the most popular methods alternates between the training of a deep autoencoder and a k-means clustering of the autoencoder's embedding. The diachronic setting, however, prevents the former to benefit from valuable information acquired by the latter. In this paper, we present an alternative where the autoencoder and the clustering are learned simultaneously. This is achieved by providing novel theoretical insight, where we show that the objective function of a certain class of Gaussian mixture models (GMMs) can naturally be rephrased as the loss function of a one-hidden layer autoencoder thus inheriting the built-in clustering capabilities of the GMM. That simple neural network, referred to as the clustering module, can be integrated into a deep autoencoder resulting in a deep clustering model able to jointly learn a clustering and an embedding. Experiments confirm the equivalence between the clustering module and Gaussian mixture models. Further evaluations affirm the empirical relevance of our deep architecture as it outperforms related baselines on several data sets.