GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization
This work addresses the problem of sparse and domain-specific reward signals in reinforcement learning for language model reasoning, offering a more general and fine-grained approach.
GeneralThinker introduces a reinforcement learning framework that uses likelihood-guided answer-conditioned optimization to enable dense, domain-general reasoning supervision without domain-specific verifiers, achieving the best average performance across 11 benchmarks in math, STEM, and general reasoning.
Reinforcement learning with verifiable rewards improves language model reasoning, but its reliance on domain-specific verifiers, sparse outcome rewards, and coarse-grained credit assignment limits its applicability. We introduce GeneralThinker, an on-policy framework that reformulates reasoning supervision as dense answer-conditioned optimization, enabling response-level evaluation and token-level credit assignment without domain-specific verifiers. GeneralThinker evaluates generated reasoning trajectories using the likelihood of the ground-truth answer and derives token-wise compatibility signals for fine-grained credit assignment. To stabilize optimization, it constrains token-level updates through clipping and direction-preserving modulation. Across 11 benchmarks spanning mathematics, STEM, and general reasoning, GeneralThinker achieves the best average performance. Further analyses show that uncontrolled token-level modulation can destabilize training, whereas controlled modulation makes fine-grained credit assignment consistently effective.