Shorter Thoughts, Same Answers: Difficulty-Scaled Segment-Wise RL for CoT Compression
This work is significant for researchers and practitioners using CoT, as it offers a method to reduce computational costs while maintaining reasoning performance, addressing the challenge of balancing conciseness and accuracy.
This paper addresses the problem of compressing Chain-of-Thought (CoT) reasoning traces to reduce token cost without sacrificing answer quality. The authors propose Difficulty-Scaled Segment-Wise GRPO (DSS-GRPO), which uses a decomposed learning signal and hard token masks to ensure compression only affects the reasoning trace, not the final answer.
Chain-of-thought (CoT) improves reasoning reliability but increases token cost, motivating post-training compression of explicit reasoning traces. However, the shortest sufficient reasoning is not universal: it depends on difficulty, model capacity, and training state, making fixed length targets brittle. In practice, naive RL-based compression can also undesirably shorten the user-facing answer, because a single completion-level learning signal leaks across the think/answer boundary. We propose Difficulty-Scaled Segment-Wise GRPO (DSS-GRPO), which decomposes returns into think and answer components, computes group-relative advantages per segment, and routes them with hard token masks so compression updates act only on think while answer alignment acts only on answer. DSS-GRPO uses prompt-wise within-group shaping and difficulty-aware scaling to encourage concise reasoning without collapsing answer behavior.