Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
This enables more efficient reasoning in smaller models for applications like AI assistants, though it is incremental as it builds on existing distillation and prompting techniques.
The paper tackles the problem that chain-of-thought prompting benefits only large models, showing that smaller models (125M-1.3B parameters) can also benefit via Symbolic Chain-of-Thought Distillation, which enhances performance on commonsense benchmarks and yields human-comparable rationalizations.
Chain-of-thought prompting (e.g., "Let's think step-by-step") primes large language models to verbalize rationalization for their predictions. While chain-of-thought can lead to dramatic performance gains, benefits appear to emerge only for sufficiently large models (beyond 50B parameters). We show that orders-of-magnitude smaller models (125M -- 1.3B parameters) can still benefit from chain-of-thought prompting. To achieve this, we introduce Symbolic Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model. Experiments across several commonsense benchmarks show that: 1) SCoTD enhances the performance of the student model in both supervised and few-shot settings, and especially for challenge sets; 2) sampling many reasoning chains per instance from the teacher is paramount; and 3) after distillation, student chain-of-thoughts are judged by humans as comparable to the teacher, despite orders of magnitude fewer parameters. We test several hypotheses regarding what properties of chain-of-thought samples are important, e.g., diversity vs. teacher likelihood vs. open-endedness. We release our corpus of chain-of-thought samples and code.