Latency and Token-Aware Test-Time Compute
This addresses the practical deployment challenge of balancing performance and efficiency in LLM inference, particularly for agentic workflows, though it is incremental over existing test-time compute methods.
The paper tackles the problem of dynamic compute allocation for inference-time scaling in large language models, which typically overlooks latency and incremental decoding methods. Their framework explicitly incorporates both token cost and latency, achieving favorable accuracy-cost trade-offs on reasoning benchmarks.
Inference-time scaling has emerged as a powerful way to improve large language model (LLM) performance by generating multiple candidate responses and selecting among them. However, existing work on dynamic allocation for test-time compute typically considers only parallel generation methods such as best-of-N, overlooking incremental decoding methods like beam search, and has largely ignored latency, focusing only on token usage. We formulate inference-time scaling as a problem of dynamic compute allocation and method selection, where the system must decide which strategy to apply and how much compute to allocate on a per-query basis. Our framework explicitly incorporates both token cost and wall-clock latency, the latter being critical for user experience and particularly for agentic workflows where models must issue multiple queries efficiently. Experiments on reasoning benchmarks show that our approach consistently outperforms static strategies, achieving favorable accuracy-cost trade-offs while remaining practical for deployment.