BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
This provides a theoretical insight for NLP researchers, but it is incremental as it builds on existing BERT analysis without major practical breakthroughs.
The paper tackled the problem of understanding BERT's generative capabilities by showing it is a Markov random field language model, enabling sentence sampling that produces high-quality but slightly worse generations compared to left-to-right models, with more diversity.
We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.