SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models
This addresses the high computational cost of fine-tuning for smaller models, offering a more scalable approach, though it is incremental as it builds on existing curriculum learning methods.
The paper tackles the problem of inefficient reinforcement fine-tuning for large language models by proposing SPaRFT, a self-paced learning framework that uses cluster-based data reduction and multi-armed bandit selection, achieving comparable or better accuracy with up to 100x fewer samples.
Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL). However, such methods require extensive data and compute, making them impractical for smaller models. Current approaches to curriculum learning or data selection are largely heuristic-driven or demand extensive computational resources, limiting their scalability and generalizability. We propose \textbf{SPaRFT}, a self-paced learning framework that enables efficient learning based on the capability of the model being trained through optimizing which data to use and when. First, we apply \emph{cluster-based data reduction} to partition training data by semantics and difficulty, extracting a compact yet diverse subset that reduces redundancy. Then, a \emph{multi-armed bandit} treats data clusters as arms, optimized to allocate training samples based on model current performance. Experiments across multiple reasoning benchmarks show that SPaRFT achieves comparable or better accuracy than state-of-the-art baselines while using up to \(100\times\) fewer samples. Ablation studies and analyses further highlight the importance of both data clustering and adaptive selection. Our results demonstrate that carefully curated, performance-driven training curricula can unlock strong reasoning abilities in LLMs with minimal resources.