Where Do CoT Training Gains Land in LLM based Agents?
For researchers and practitioners building LLM-based agents, this work clarifies the mechanism of CoT training gains, showing it mainly boosts direct action prediction rather than reasoning, which has implications for training strategies.
The paper investigates whether chain-of-thought (CoT) training in LLM-based agents improves reasoning or directly improves action prediction from the prompt. It finds that CoT training primarily enhances prompt-action quality rather than the relative advantage of CoT reasoning, and that selectively masking action-token supervision improves out-of-domain generalization.
Chain-of-thought (CoT) reasoning is widely used in language-model agents, but prior work has shown that verbalized CoT is not always faithful and may instead reflect post-hoc reasoning, which means the model already knows the answer before reasoning. We therefore ask what CoT training is actually improving: is the model getting better at changing its action through generated reasoning, or is it getting better at predicting the action directly from the prompt? We study this question by comparing \emph{prompt actions} (predicting action without CoT) with CoT actions (predicting action with CoT). Across checkpoints, prompt-action quality improves substantially. While interacting with the environment, the relative advantage of CoT actions over prompt actions remains similar, showing that CoT training does not widen the advantage of CoT reasoning, and it helps to improve the quality of prompt actions. We further find that later checkpoints are less likely to revise the action in response to CoT, suggesting greater reliance on the prompt. Motivated by these patterns, we selectively mask action-token supervision on a fraction of training examples. This intervention improves out-of-domain generalization.