Semantic-guided Diverse Decoding for Large Language Model
This addresses a key limitation in diverse decoding for LLMs, enabling more effective applications in reinforcement learning and data synthesis, though it is an incremental improvement over existing methods.
The paper tackles the problem of generating semantically diverse responses from large language models, which is crucial for applications like Best-of-N strategies and data synthesis, by introducing Semantic-guided Diverse Decoding (SemDiD) that improves Best-of-N coverage by 1.4-5.2% and accelerates RLHF training convergence by 15% with up to 2.1% accuracy gains.
Diverse decoding of large language models is crucial for applications requiring multiple semantically distinct responses, yet existing methods primarily achieve lexical rather than semantic diversity. This limitation significantly constrains Best-of-N strategies, group-based reinforcement learning, and data synthesis. While temperature sampling and diverse beam search modify token distributions or apply n-gram penalties, they fail to ensure meaningful semantic differentiation. We introduce Semantic-guided Diverse Decoding (SemDiD), operating directly in embedding space that balances quality with diversity through three complementary mechanisms: orthogonal directional guidance, dynamic inter-group repulsion, and position-debiased probability assessment. SemDiD harmonizes these competing objectives using adaptive gain functions and constraint optimization, ensuring both quality thresholds and maximal semantic differentiation. Experiments show SemDiD consistently outperforms existing methods, improving Best-of-N coverage by 1.4-5.2% across diverse tasks and accelerating RLHF training convergence by 15% while increasing accuracy by up to 2.1%.