Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively
This addresses cost-effectiveness for LLM applications in complex decision-making tasks, representing an incremental improvement over existing search strategies.
The paper tackles the problem of balancing computational cost with performance in LLM decision-making by proposing the Speculative Reward Model (SRM), which reduces costs to 1/10 of original search frameworks on average while maintaining effectiveness.
Effective decision-making in Large Language Models (LLMs) is essential for handling intricate tasks. However, existing approaches prioritize performance but often overlook the balance between effectiveness and computational cost. To address this, we first introduce the 3E Criteria to systematically assess the cost-effectiveness of search strategies, revealing that existing methods often trade significant efficiency for marginal performance gains. To improve LLM decision-making while maintaining efficiency, we propose the Speculative Reward Model (SRM), a plug-and-play framework that seamlessly integrates with existing search strategies. Specifically, SRM employs an external reward assigner to predict optimal actions, reducing reliance on LLMs' internal self-evaluation. And a speculative verification mechanism is used to prune suboptimal choices and guide the search toward more promising steps. We evaluate SRM on several complex decision-making tasks including mathematical reasoning, planning and numerical reasoning in specialized domains. Experimental results show that SRM reduces costs to 1/10 of the original search framework on average while maintaining effectiveness.