Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuning
This is an incremental improvement for fine-tuning large language models using reinforcement learning, addressing efficiency in high-dimensional state spaces.
The paper tackles the exploration-exploitation tradeoff in reinforcement learning-based fine-tuning of large language models by introducing Entropy-Guided Sequence Weighting (EGSW), which dynamically weights generated outputs based on advantage and entropy, resulting in improved sample efficiency for Group Relative Policy Optimization.
We introduce Entropy-Guided Sequence Weighting (EGSW), a novel approach that enhances the exploration-exploitation tradeoff by dynamically assigning weights to generated outputs based on their advantage and entropy for Reinforcement Learning-based Large Language Model fine-tuning. EGSW integrates entropy regularization with advantage-based weighting to balance policy updates, enabling efficient exploration in high-dimensional state spaces. By employing temperature-scaled softmax weighting over sequences, EGSW prioritizing high-reward, high-uncertainty steps while maintaining training stability. Although originally developed to improve Group Relative Policy Optimization (GRPO) during large language model (LLM) fine-tuning, EGSW is generalizable to other reinforcement learning (RL) algorithms and can be implemented in both step-wise and trajectory-wise settings. Empirical evaluations demonstrate that EGSW enhances GRPO reasoning ability, yielding improvements in sample efficiency. Future work will explore the application of EGSW to advanced RL methodologies.