Latent Thought Credit: Multi-Answer Credit Assignment for Latent Reasoning
This work provides a method for more accurately assigning credit to internal reasoning steps in language models, which is a significant problem for researchers and developers working on improving the interpretability and performance of LLMs.
This paper addresses the challenge of assigning credit to latent reasoning in language models when only final answer rewards are available. The proposed Latent Thought Credit (LTC) framework samples multiple latent thoughts and estimates thought-level expected reward by averaging rewards over multiple answers generated from each thought, leading to the best average accuracy among compared methods on mathematical reasoning and STEM multiple-choice tasks.
Latent reasoning allows language models to carry out intermediate reasoning in continuous latent representations rather than fully externalizing it as discrete chains of thought. However, assigning credit to such latent thoughts from answer-only rewards is difficult: a single final answer mixes thought quality with answer-sampling noise. We propose \textbf{Latent Thought Credit (LTC)}, a hierarchical credit-assignment framework for latent reasoning. For each prompt, LTC samples multiple latent thoughts, fixes the context after each thought, and estimates thought-level expected reward by averaging rewards over multiple answers generated from that fixed context. LTC uses thought-level advantages to optimize the latent-thought phase, answer-level advantages to optimize the answer phase, and an advantage-weighted thought-matching objective that helps the policy reproduce high-credit latent thoughts. We instantiate LTC in a GRPO-style on-policy training framework and evaluate it across mathematical reasoning and STEM multiple-choice tasks. LTC achieves the best average accuracy among the compared methods, while ablations and fixed-context diagnostics show that multi-answer estimation reduces reward-estimation error and mitigates ambiguous or incorrect thought-level credit.