Post-training Large Language Models for Diverse High-Quality Responses
This addresses a key limitation in post-training LLMs for users needing varied and high-quality responses, though it is an incremental improvement over existing RL methods.
The paper tackles the problem of reduced output diversity in large language models after reinforcement learning post-training, proposing DQO to jointly optimize for quality and semantic diversity, which substantially improves diversity without sacrificing quality across multiple tasks.
Reinforcement learning (RL) has emerged as a popular method for post-training large language models (LLMs). While improving the model's performance on downstream tasks, it often reduces the model's output diversity, leading to narrow, canonical responses. Existing methods to enhance diversity are limited, either by operating at inference time or by focusing on surface-level differences. We propose a novel training method named DQO (Diversity Quality Optimization) based on determinantal point processes (DPPs) to jointly optimize LLMs for quality and semantic diversity. Our approach samples and embeds a group of responses for each prompt, then uses the determinant of a kernel-based similarity matrix to measure diversity as the volume spanned by the embeddings of these responses. DQO is flexible and can be applied on top of existing RL algorithms. Experiments across instruction-following, summarization, story generation, and reasoning tasks demonstrate that our method substantially improves semantic diversity without sacrificing model quality.