Encoding Binary Concepts in the Latent Space of Generative Models for Enhancing Data Representation
This work addresses the challenge of learning transferable representations for model generalization in machine learning, though it appears incremental as it builds on existing VAE variants.
The paper tackles the problem of improving data representation in generative models by encoding binary concepts in the latent space, proposing a binarized regularization method that enhances reconstruction quality and prevents posterior collapse in variational autoencoders without computational overhead.
Binary concepts are empirically used by humans to generalize efficiently. And they are based on Bernoulli distribution which is the building block of information. These concepts span both low-level and high-level features such as "large vs small" and "a neuron is active or inactive". Binary concepts are ubiquitous features and can be used to transfer knowledge to improve model generalization. We propose a novel binarized regularization to facilitate learning of binary concepts to improve the quality of data generation in autoencoders. We introduce a binarizing hyperparameter $r$ in data generation process to disentangle the latent space symmetrically. We demonstrate that this method can be applied easily to existing variational autoencoder (VAE) variants to encourage symmetric disentanglement, improve reconstruction quality, and prevent posterior collapse without computation overhead. We also demonstrate that this method can boost existing models to learn more transferable representations and generate more representative samples for the input distribution which can alleviate catastrophic forgetting using generative replay under continual learning settings.