Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces
For practitioners deploying reasoning models on multi-label tasks with huge label spaces, this provides a principled and effective distillation method.
The paper characterizes reasoning in large-output-space tasks as a two-phase process (shortlisting then fine-grained reasoning) and shows that a distillation strategy based on this characterization consistently outperforms standard distillation.
Modern reasoning models offer surprisingly strong zero-shot performance on challenging multi-label tasks that require selecting a small set of relevant options from hundreds of thousands to millions of candidate labels. We investigate how they achieve this mechanistically. We characterize reasoning as a two-phase process: A broad "shortlisting" of candidates followed by fine-grained reasoning over the resulting set. We provide evidence across a range of datasets that these steps can be isolated and are complementary. Using this characterization, we develop a mechanistic distillation strategy that consistently outperforms standard distillation.