Confidence-Weighted Token Set Cover for Early Hypothesis Pruning in Self-Consistency
This work addresses token efficiency for users of self-consistency in reasoning tasks, though it is incremental as it builds on existing methods with lightweight indicators.
The paper tackled the high token expenditure of self-consistency in long chain-of-thought reasoning by introducing early hypothesis pruning based on model confidence and lexical coverage, improving token efficiency by 10-35% across five LLMs on three math benchmarks.
Despite its simplicity and efficacy, the high token expenditure of self-consistency can limit its practical utility. Here we investigate if self-consistency can be made more token-efficient for long chain-of-thought reasoning tasks, while preserving its parallelism, through early hypothesis pruning. Concretely, we generate all solutions in parallel, but periodically prune intermediate hypotheses that are deemed unnecessary based on two lightweight indicators: (a) the model's own confidence in individual hypotheses, and (b) lexical coverage of all current hypotheses by candidate subsets that are under consideration for continued retention. We design a fast weighted set cover algorithm that utilizes the two indicators; our evaluation of five LLMs on three math benchmarks shows that this method can improve token efficiency for all models, by 10-35% in many cases.