When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation
Provides condition-aware guidelines for practitioners choosing CoT compression strategies for different domains and regimes.
The paper systematically evaluates chain-of-thought compression methods across dimensions of importance criterion, restructuring level, and compression budget, finding that step-level criteria converge on a shared reasoning backbone while token-level pruning needs symbol-aware signals, and that restructuring level inverts across domains (Math degrades with disruption, General benefits from rewriting). Training-time compression does not guarantee inference savings due to student verbosity.
Chain-of-Thought (CoT) distillation transfers multi-step reasoning from large reasoning models to smaller students, but verbose teacher traces inflate both training and inference cost. Existing CoT compression methods fall into two families, selective pruning and generative rewriting, yet prior studies have left key factors entangled: granularity is confounded with importance criteria in pruning, restructuring level is rarely isolated in rewriting, and compression budgets are not systematically evaluated across domains or regimes. We recast CoT compression along three dimensions: importance criterion, restructuring level, and compression budget. Sweeping these across two model families, Math and General domains, and Long-/Short-CoT regimes, we find that (i) importance criterion utility is strictly governed by granularity: step-level criteria converge on a shared reasoning backbone, while token-level pruning requires symbol-aware signals to preserve the logical core; (ii) restructuring level inverts across domains: Math degrades monotonically with structural disruption, while aggressive rewriting acts as a denoiser on General tasks; (iii) training-time compression does not necessarily translate to inference-time savings: Long-CoT students retain verbose habits despite concise supervision, making the training ratio an optimistic lower bound on deployment cost. These findings yield condition-aware guidelines for matching compression to deployment context.