Code Prompting: a Neural Symbolic Method for Complex Reasoning in Large Language Models
This addresses the limitation of natural language intermediate steps in prompting methods for complex reasoning tasks in large language models, offering a novel approach with potential broad applicability.
The paper tackles the problem of imperfect task reduction and confusion in large language models by introducing code prompting, a neural symbolic method that uses code as intermediate steps for complex reasoning. The result shows that code prompting generally outperforms chain-of-thought prompting on 7 benchmarks involving symbolic and arithmetic reasoning.
Large language models (LLMs) have scaled up to unlock a wide range of complex reasoning tasks with the aid of various prompting methods. However, current prompting methods generate natural language intermediate steps to help reasoning, which can cause imperfect task reduction and confusion. To mitigate such limitations, we explore code prompting, a neural symbolic prompting method with both zero-shot and few-shot versions which triggers code as intermediate steps. We conduct experiments on 7 widely-used benchmarks involving symbolic reasoning and arithmetic reasoning. Code prompting generally outperforms chain-of-thought (CoT) prompting. To further understand the performance and limitations of code prompting, we perform extensive ablation studies and error analyses, and identify several exclusive advantages of using symbolic promptings compared to natural language. We also consider the ensemble of code prompting and CoT prompting to combine the strengths of both. Finally, we show through experiments how code annotations and their locations affect code prompting.