$\mathbf{T^3}$: Reducing Belief Deviation in Reinforcement Learning for Active Reasoning
This addresses a key bottleneck in reinforcement learning for active reasoning with LLMs, offering incremental improvements in training stability and efficiency.
The paper tackles the problem of belief deviation in LLM-based active reasoning agents, which causes errors and training failures, by proposing T^3, a method that truncates trajectories to remove uninformative parts, resulting in up to 30% performance gains and 25% reduction in rollout tokens across 5 tasks.
Active reasoning requires large language models (LLMs) to interact with external sources and strategically gather information to solve problems. Central to this process is belief tracking: maintaining a coherent understanding of the problem state and the missing information toward the solution. However, due to limited reasoning capabilities, LLM-based agents often suffer from belief deviation: they struggle to correctly model beliefs, lose track of problem states, and fall into uninformative or repetitive actions. Once this happens, errors compound and reinforcement learning (RL) training fails to properly credit the crucial exploratory steps. To address this issue, we propose to track the deviation of model beliefs and develop $\mathbf{T^3}$, a simple yet effective method that detects excessive belief deviation and truncates trajectories during training to remove uninformative tails. By preserving credit for informative prefixes, $\mathbf{T^3}$ systematically improves policy optimization. Across 5 challenging tasks, $\mathbf{T^3}$ consistently enhances training stability, token efficiency, and final performance, achieving up to 30% gains while cutting rollout tokens by roughly 25%. These results highlight belief control as a key principle for developing robust and generalizable LLM-based active reasoners.