KiC: Keyword-inspired Cascade for Cost-Efficient Text Generation with LLMs
This addresses cost efficiency for users of LLM APIs in text generation tasks, offering a novel approach to cascade methods.
The paper tackles the problem of high inference costs for large language models (LLMs) in free-form text generation by proposing KiC, a cascade framework that uses keyword-inspired semantic alignment to decide when to escalate to stronger models, achieving 97.53% of GPT-4's accuracy with a 28.81% average cost reduction.
Large language models (LLMs) have demonstrated state-of-the-art performance across a wide range of natural language processing tasks. However, high-performing models are typically accessible only via APIs, incurring substantial inference costs. Cascade methods address this by initially employing a cheaper model and escalating to a stronger one only when necessary. Nevertheless, existing cascade approaches struggle to select a reliable representative response and assess the overall reliability of free-form outputs, as they rely on exact text matching. To overcome these limitations, we propose Keyword-inspired Cascade (KiC), a novel framework for cost-efficient free-form text generation. KiC identifies the most representative answer among multiple outputs from a weaker model and evaluates the semantic alignment of other responses with it. Based on the degree of alignment, KiC determines whether to accept the weaker model's output or escalate to a stronger model. Experiments on three free-form text generation benchmarks show that KiC achieves 97.53 percent of GPT-4's accuracy while reducing API costs by 28.81 percent on average, and even outperforms GPT-4 in a specific benchmark.