An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
This addresses the problem of generalizability for researchers and practitioners using LLMs, but it is incremental as it builds on existing CoT methods.
The study investigated whether chain-of-thought reasoning strategies generalize across different large language models and datasets, finding that gains from these strategies remain robust, with GPT-4 showing the most benefit and best performance using an automatically discovered prompt.
Emergent chain-of-thought (CoT) reasoning capabilities promise to improve performance and explainability of large language models (LLMs). However, uncertainties remain about how reasoning strategies formulated for previous model generations generalize to new model generations and different datasets. In this small-scale study, we compare different reasoning strategies induced by zero-shot prompting across six recently released LLMs (davinci-002, davinci-003, GPT-3.5-turbo, GPT-4, Flan-T5-xxl and Cohere command-xlarge) on a mixture of six question-answering datasets, including datasets from scientific and medical domains. Our findings demonstrate that while some variations in effectiveness occur, gains from CoT reasoning strategies remain robust across different models and datasets. GPT-4 has the most benefit from current state-of-the-art reasoning strategies and exhibits the best performance by applying a prompt previously discovered through automated discovery.