LoRi: Low-Rank Distillation for Implicit Reasoning
For practitioners of large language models, this work provides a method to improve implicit reasoning without the overhead of explicit CoT, though improvements are incremental over prior distillation methods.
Implicit chain-of-thought methods often underperform explicit CoT prompting. The authors propose a low-rank distillation framework that aligns teacher and student reasoning trajectories in a low-rank subspace, improving performance on mathematical reasoning benchmarks and approaching explicit CoT accuracy.
Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.