Align to Structure: Aligning Large Language Models with Structural Information
This addresses the problem of incoherent long-form text generation for users of LLMs, representing a novel method for a known bottleneck.
The paper tackles the challenge of generating long, coherent text in large language models by introducing Structural Alignment, a method that aligns models with human-like discourse structures using reinforcement learning, resulting in improved performance in tasks like essay generation and long-document summarization compared to standard and RLHF-enhanced models.
Generating long, coherent text remains a challenge for large language models (LLMs), as they lack hierarchical planning and structured organization in discourse generation. We introduce Structural Alignment, a novel method that aligns LLMs with human-like discourse structures to enhance long-form text generation. By integrating linguistically grounded discourse frameworks into reinforcement learning, our approach guides models to produce coherent and well-organized outputs. We employ a dense reward scheme within a Proximal Policy Optimization framework, assigning fine-grained, token-level rewards based on the discourse distinctiveness relative to human writing. Two complementary reward models are evaluated: the first improves readability by scoring surface-level textual features to provide explicit structuring, while the second reinforces deeper coherence and rhetorical sophistication by analyzing global discourse patterns through hierarchical discourse motifs, outperforming both standard and RLHF-enhanced models in tasks such as essay generation and long-document summarization. All training data and code will be publicly shared at https://github.com/minnesotanlp/struct_align.