DIVE: Diversified Iterative Self-Improvement
This work addresses a critical limitation in iterative self-improvement techniques for large language models, which is important for researchers and practitioners in AI and machine learning, though it is incremental as it builds on existing ISI methods.
The paper tackles the problem of reduced output diversity in iterative self-improvement (ISI) for large language models, particularly in reasoning tasks, and presents DIVE, a framework that increases output diversity by 10% to 45% relative to vanilla ISI while maintaining performance quality.
Recent advances in large language models (LLMs) have demonstrated the effectiveness of Iterative Self-Improvement (ISI) techniques. However, continuous training on self-generated data leads to reduced output diversity, a limitation particularly critical in reasoning tasks where diverse solution paths are essential. We present DIVE (Diversified Iterative Self-Improvement), a novel framework that addresses this challenge through two key components: Sample Pool Expansion for broader solution exploration, and Data Selection for balancing diversity and quality in preference pairs. Experiments on MATH and GSM8k datasets show that DIVE achieves a 10% to 45% relative increase in output diversity metrics while maintaining performance quality compared to vanilla ISI. Our ablation studies confirm both components' significance in achieving these improvements. Code is available at https://github.com/qinyiwei/DIVE.