Conformal Constrained Policy Optimization for Cost-Effective LLM Agents
This addresses the cost-effectiveness problem for deploying LLM agents, offering a principled framework that is incremental in combining existing techniques.
The paper tackles the problem of high computational and API costs of large language model (LLM) agents by proposing a strategy that combines multiple LLM models with varying cost/accuracy tradeoffs, using conformal prediction to ensure reliability, resulting in up to a 30% cost reduction on multi-hop question answering benchmarks without compromising reliability.
While large language models (LLMs) have recently made tremendous progress towards solving challenging AI problems, they have done so at increasingly steep computational and API costs. We propose a novel strategy where we combine multiple LLM models with varying cost/accuracy tradeoffs in an agentic manner, where models and tools are run in sequence as determined by an orchestration model to minimize cost subject to a user-specified level of reliability; this constraint is formalized using conformal prediction to provide guarantees. To solve this problem, we propose Conformal Constrained Policy Optimization (CCPO), a training paradigm that integrates constrained policy optimization with off-policy reinforcement learning and recent advances in online conformal prediction. CCPO jointly optimizes a cost-aware policy (score function) and an adaptive threshold. Across two multi-hop question answering benchmarks, CCPO achieves up to a 30% cost reduction compared to other cost-aware baselines and LLM-guided methods without compromising reliability. Our approach provides a principled and practical framework for deploying LLM agents that are significantly more cost-effective while maintaining reliability.