TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs
This addresses sparse-reward credit assignment for multi-turn LLM reasoning, offering a general solution that improves training stability and performance in QA tasks.
The paper tackles the challenge of unstable training in search-augmented LLMs for open-domain QA by introducing TIPS, a framework that assigns dense, turn-level rewards based on increased answer likelihood under a teacher model, resulting in improved average Exact Match by 11.8% and F1 by 13.6% relative to PPO on benchmarks.
Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remains a significant challenge. The optimization is often unstable due to sparse rewards and difficult credit assignments across reasoning and tool calls. To address this, we introduce Turn-Level Information Potential Reward Shaping (TIPS), a simple framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model. By leveraging the potential-based reward shaping, TIPS offers fine-grained and policy-invariant guidance that overcomes the limitations of outcome-only optimization. Evaluated on seven QA benchmarks, TIPS consistently outperforms GRPO/PPO baselines and substantially improves training stability. For instance, with a Qwen-2.5 7B Instruct model, TIPS improves the average Exact Match score by 11.8% and F1 by 13.6% relative to PPO. Our results demonstrate that turn-level information-potential reward shaping provides an effective and general solution to sparse-reward credit assignment for multi-turn LLM reasoning.