InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation
This addresses a specific problem in text generation for NLP researchers, offering an incremental improvement by refining diffusion model strategies.
The paper tackles the mismatch between diffusion models' 'easy-first' text generation and human 'keyword-first' processes by proposing InfoDiffusion, a non-autoregressive model with a 'keyinfo-first' strategy and information-aware noise schedule, resulting in improved generation quality, diversity, and sampling efficiency over baselines.
Diffusion models have garnered considerable interest in the field of text generation. Several studies have explored text diffusion models with different structures and applied them to various tasks, including named entity recognition and summarization. However, there exists a notable disparity between the "easy-first" text generation process of current diffusion models and the "keyword-first" natural text generation process of humans, which has received limited attention. To bridge this gap, we propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a "keyinfo-first" generation strategy and incorporates a noise schedule based on the amount of text information. In addition, InfoDiffusion combines self-conditioning with a newly proposed partially noising model structure. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity, as well as exhibiting higher sampling efficiency.