Unveiling the Potential of Diffusion Large Language Model in Controllable Generation
This work addresses a fundamental issue in NLP for applications requiring controllable generation, such as function calling and agentic communication, by introducing a new method that enhances reliability in structured output tasks.
The paper tackles the problem of unreliable structured output generation in autoregressive Large Language Models by proposing a novel framework, Self-adaptive Schema Scaffolding ($S^3$), which leverages diffusion-based LLMs to stably generate reliable structured outputs like JSON, with experiments showing substantial improvements in structure adherence, content fidelity, and faithfulness.
Controllable generation is a fundamental task in NLP with many applications, providing a basis for function calling to agentic communication. However, even state-of-the-art autoregressive Large Language Models (LLMs) today exhibit unreliability when required to generate structured output. Inspired by the current new diffusion-based large language models (dLLM), we realize that the architectural difference, especially the global information-sharing mechanism for language modeling, may be the key to unlock next-level controllable generation. To explore the possibility, we propose Self-adaptive Schema Scaffolding ($S^3$), a novel framework that enables dLLM to stably generate reliable structured outputs (e.g., JSON) by utilizing its innate reverse reasoning capability and global context awareness. $S^3$ initiates a schematic template directly in the output context as a starting state for dLLM, offering a more robust and general method than intricate prompt optimization. Experiments demonstrate that our method substantially unlocks the dLLM's potential in controllable generation in terms of structure adherence, content fidelity, and faithfulness. These results establish new perspectives and practical pathways for deploying language models in controllable generation tasks.