Distilling System 2 into System 1
This addresses the challenge of computational efficiency for AI systems, enabling them to focus System 2 capabilities on more difficult reasoning tasks, though it is incremental as it builds on existing System 2 methods.
The paper tackles the problem of reducing inference costs in large language models by distilling high-quality outputs from System 2 techniques, which use intermediate reasoning, back into System 1 generations without such reasoning sequences, resulting in improved performance compared to original System 1 and lower inference cost than System 2.
Large language models (LLMs) can spend extra compute during inference to generate intermediate thoughts, which helps to produce better final responses. Since Chain-of-Thought (Wei et al., 2022), many such System 2 techniques have been proposed such as Rephrase and Respond (Deng et al., 2023a), System 2 Attention (Weston and Sukhbaatar, 2023) and Branch-Solve-Merge (Saha et al., 2023). In this work we investigate self-supervised methods to ``compile'' (distill) higher quality outputs from System 2 techniques back into LLM generations without intermediate reasoning token sequences, as this reasoning has been distilled into System 1. We show that several such techniques can be successfully distilled, resulting in improved results compared to the original System 1 performance, and with less inference cost than System 2. We posit that such System 2 distillation will be an important feature of future continually learning AI systems, enabling them to focus System 2 capabilities on the reasoning tasks that they cannot yet do well.