Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling
This addresses the efficiency-accuracy trade-off in LLM reasoning tasks for resource-constrained environments, representing a novel method for a known bottleneck.
The paper tackles the problem of inefficient sampling and unfaithful rationales in Self-Consistency for LLMs by introducing Reasoning-Aware Self-Consistency (RASC), which dynamically evaluates reasoning paths to guide early stopping and rationale selection, reducing sample usage by about 70% while maintaining accuracy.
Self-Consistency mitigates hallucinations in Large Language Models (LLMs) by sampling multiple reasoning paths,but it lacks a systematic approach to determine the optimal number of samples or select the most faithful rationale. To address this limitation, we introduce Reasoning-Aware Self-Consistency (RASC), a novel framework that enhances sampling efficiency and reasoning faithfulness by dynamically evaluating both outputs and rationales. RASC assesses the quality of reasoning and the consistency of answers for each generated sample, using these assessments to guide early stopping decisions and rationale selection. The framework employs criteria-based stopping and weighted majority voting, enabling more informed choices on when to halt sampling and which rationale to select. Our comprehensive experiments across diverse question-answering datasets demonstrate that RASC outperforms existing methods, reducing sample usage by approximately 70% while maintaining accuracy. Moreover, RASC facilitates the selection of high-fidelity rationales, thereby improving the faithfulness of LLM outputs. Our approach effectively addresses the efficiency-accuracy trade-off in LLM reasoning tasks, offering a new perspective for more nuanced, faithful, and effective utilization of LLMs in resource-constrained environments.