Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition
This addresses the challenge of computational demands in LLM distillation for open-source, low-resource models, though it appears incremental as it builds on existing prompt-based and decomposition methods.
The paper tackles the problem of making knowledge distillation for Large Language Models more accessible by introducing Trace-of-Thought Prompting, which distills reasoning capabilities from high-resource teacher models to low-resource student models, resulting in accuracy gains of up to 113% on GSM8K and 21% on MATH datasets.
Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models (over 8 billion parameters) to low-resource student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 21% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers, potentially reducing our reliance on high-resource, proprietary models.