Regularizing Transformers With Deep Probabilistic Layers
This work addresses the need for more robust and versatile language models, particularly for tasks like text imputation, but it appears incremental as it builds on existing BERT and seq2seq architectures.
The authors tackled the problem of improving language models by incorporating deep generative models as regularizers, resulting in enhanced versatility for imputing missing or noisy words and an improved BLEU score.
Language models (LM) have grown with non-stop in the last decade, from sequence-to-sequence architectures to the state-of-the-art and utter attention-based Transformers. In this work, we demonstrate how the inclusion of deep generative models within BERT can bring more versatile models, able to impute missing/noisy words with richer text or even improve BLEU score. More precisely, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer and prove its effectiveness not only in Transformers but also in the most relevant encoder-decoder based LM, seq2seq with and without attention.