Multi-type Disentanglement without Adversarial Training
This work addresses the challenge of disentangling multiple style types in natural language generation for researchers and practitioners seeking more interpretable and controllable text generation models, offering an incremental improvement by avoiding adversarial training.
This paper proposes a unified distribution-controlling method for multi-type disentanglement in natural language, aiming to control sentence style by tuning style representations without affecting other features. It avoids adversarial training, which is typically difficult, especially with multiple features, and introduces loss functions to achieve style-content disentanglement and disentanglement among multiple style types, including a method to alleviate 'training bias' where co-occurring style values affect each other. Experiments on Yelp and Amazon review datasets for sentiment and tense demonstrate the model's effectiveness.
Controlling the style of natural language by disentangling the latent space is an important step towards interpretable machine learning. After the latent space is disentangled, the style of a sentence can be transformed by tuning the style representation without affecting other features of the sentence. Previous works usually use adversarial training to guarantee that disentangled vectors do not affect each other. However, adversarial methods are difficult to train. Especially when there are multiple features (e.g., sentiment, or tense, which we call style types in this paper), each feature requires a separate discriminator for extracting a disentangled style vector corresponding to that feature. In this paper, we propose a unified distribution-controlling method, which provides each specific style value (the value of style types, e.g., positive sentiment, or past tense) with a unique representation. This method contributes a solid theoretical basis to avoid adversarial training in multi-type disentanglement. We also propose multiple loss functions to achieve a style-content disentanglement as well as a disentanglement among multiple style types. In addition, we observe that if two different style types always have some specific style values that occur together in the dataset, they will affect each other when transferring the style values. We call this phenomenon training bias, and we propose a loss function to alleviate such training bias while disentangling multiple types. We conduct experiments on two datasets (Yelp service reviews and Amazon product reviews) to evaluate the style-disentangling effect and the unsupervised style transfer performance on two style types: sentiment and tense. The experimental results show the effectiveness of our model.