Enhancing Reasoning Capabilities of Small Language Models with Blueprints and Prompt Template Search
This work addresses the challenge of enhancing reasoning in SLMs for resource-constrained environments, offering a lightweight solution, though it is incremental as it builds on existing methods like prompting and blueprint generation.
The paper tackled the problem of limited reasoning capabilities and prompt sensitivity in small language models (SLMs) by proposing a framework using LLM-generated blueprints and prompt template search, resulting in improved performance on tasks like GSM8K, MBPP, and BBH without increasing model size or training.
Small language models (SLMs) offer promising and efficient alternatives to large language models (LLMs). However, SLMs' limited capacity restricts their reasoning capabilities and makes them sensitive to prompt variations. To address these challenges, we propose a novel framework that enhances SLM reasoning capabilities through LLM generated blueprints. The blueprints provide structured, high-level reasoning guides that help SLMs systematically tackle related problems. Furthermore, our framework integrates a prompt template search mechanism to mitigate the SLMs' sensitivity to prompt variations. Our framework demonstrates improved SLM performance across various tasks, including math (GSM8K), coding (MBPP), and logic reasoning (BBH). Our approach improves the reasoning capabilities of SLMs without increasing model size or requiring additional training, offering a lightweight and deployment-friendly solution for on-device or resource-constrained environments.