Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction
This addresses the challenge of rigid reasoning processes in LLM agents for planning in complex environments, representing an incremental improvement over existing multi-turn interaction methods.
The paper tackled the problem of inefficient world model reasoning in LLM agents by introducing WMAct, which uses reward rescaling and interaction frequency annealing to enable tasks like Sokoban to be resolved in a single turn instead of multiple interactions.
Developing robust world model reasoning is crucial for large language model (LLM) agents to plan and interact in complex environments. While multi-turn interaction offers a superior understanding of environmental dynamics via authentic feedback, current approaches often impose a rigid reasoning process, which constrains the model's active learning, ultimately hindering efficient world model reasoning. To address these issues, we explore world-model internalization through efficient interaction and active reasoning (WMAct), which liberates the model from structured reasoning, allowing the model to shape thinking directly through its doing, and achieves effective and efficient world model reasoning with two key mechanisms: (1) a reward rescaling mechanism adjusting outcome reward based on action efficacy to incentivize redundancy reduction and purposeful interaction; (2) an interaction frequency annealing strategy to progressively reduce the maximum allowed interaction turns, which compels the model to condense its learning and internalize environmental dynamics rather than over-relying on environmental cues. Our experiments on Sokoban, Maze, and Taxi show that WMAct yields effective world model reasoning capable of resolving tasks in a single turn that previously required multiple interactions and fosters strong transferability to complex environments, improving performance on a suite of reasoning benchmarks.