State Fourier Diffusion Language Model (SFDLM): A Scalable, Novel Iterative Approach to Language Modeling
This addresses the problem of computational efficiency in language modeling for NLP researchers and practitioners, presenting a novel hybrid approach rather than an incremental improvement.
The paper tackles the computational cost of transformer-based text generation by introducing a fully diffusion-driven discrete text generation model that replaces transformers with structured state space dynamics and a Complex Fourier Multi Layer Perceptron. The result is a model that captures both short and long-range dependencies without high computational overhead.
In recent years, diffusion based methods have emerged as a powerful paradigm for generative modeling. Although discrete diffusion for natural language processing has been explored to a lesser extent, it shows promise for tasks requiring iterative denoising of token based data. In standard approaches to text generation, transformers dominate, but their reliance on self attention often incurs high computational costs. This paper introduces a fully diffusion driven discrete text generation model built without any transformer or large convolution modules. Instead, the model integrates structured state space dynamics in the time domain with a novel Complex Fourier Multi Layer Perceptron module that operates in the frequency domain. The forward noising process randomly samples the vocabulary to replace tokens with a controlled probability, while the learned reverse model systematically reverts corrupted sequences toward their original states. By composing local state space updates with global Fourier based mixing, the approach effectively captures both short and long range dependencies.