Promises, Outlooks and Challenges of Diffusion Language Modeling
This work addresses limitations like slow generation and exposure bias in autoregressive LLMs, offering a promising but incremental improvement for NLP applications.
The paper evaluates Score Entropy Discrete Diffusion (SEDD) as an alternative to autoregressive language models, showing it matches them in perplexity and benchmarks like HellaSwag while being up to 4.5x more efficient in inference latency, but it has weaknesses in conditional generation with short prompts.
The modern autoregressive Large Language Models (LLMs) have achieved outstanding performance on NLP benchmarks, and they are deployed in the real world. However, they still suffer from limitations of the autoregressive training paradigm. For example, autoregressive token generation is notably slow and can be prone to \textit{exposure bias}. The diffusion-based language models were proposed as an alternative to autoregressive generation to address some of these limitations. We evaluate the recently proposed Score Entropy Discrete Diffusion (SEDD) approach and show it is a promising alternative to autoregressive generation but it has some short-comings too. We empirically demonstrate the advantages and challenges of SEDD, and observe that SEDD generally matches autoregressive models in perplexity and on benchmarks such as HellaSwag, Arc or WinoGrande. Additionally, we show that in terms of inference latency, SEDD can be up to 4.5$\times$ more efficient than GPT-2. While SEDD allows conditioning on tokens at abitrary positions, SEDD appears slightly weaker than GPT-2 for conditional generation given short prompts. Finally, we reproduced the main results from the original SEDD paper.