FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion
This addresses the challenge of improving LLM performance through better model fusion, offering a novel method for researchers and practitioners, though it is incremental as it builds on existing preference alignment techniques.
The paper tackles the problem of underutilizing knowledge in heterogeneous model fusion for LLMs by proposing FuseRL, a two-stage framework that integrates source models through weighted fine-tuning and preference optimization, achieving state-of-the-art performance among 8B LLMs on benchmarks like AlpacaEval-2 and Arena-Hard.
Heterogeneous model fusion enhances the performance of LLMs by integrating the knowledge and capabilities of multiple structurally diverse models. However, existing approaches often rely solely on selecting the best output for each prompt from source models, which underutilizes their full potential due to limited source knowledge and results in sparse optimization signals. To address this limitation, we propose FuseRL, a novel two-stage framework comprising FuseSFT and FusePO to maximize the utilization of source LLMs. FuseSFT establishes a robust initialization by integrating the strengths of heterogeneous source models through weighted supervised fine-tuning (SFT) on diverse outputs for each prompt. FusePO optimizes weighted preferences based on the outputs of multiple source models to enable superior alignment performance. Extensive experiments demonstrate the effectiveness of our framework across various preference alignment methods, including RLOO, DPO, and SimPO. Using Llama-3.1-8B-Instruct as the target model, our approach achieves state-of-the-art performance among 8B LLMs on the AlpacaEval-2 and Arena-Hard benchmarks. Further analysis suggests that FuseSFT regularizes the training process to reduce overfitting, while FusePO introduces dense and diverse signals for preference optimization.