MiniCache: Reusable Program Caching with Small Model Interfaces for Efficient LLM Inference
For practitioners deploying LLMs on program-aided reasoning tasks, MiniCache offers a practical caching solution that significantly improves latency and throughput.
MiniCache reduces LLM inference cost by caching reusable program structures from Program-of-Thought, achieving up to 3.1x lower latency and 2.8x higher throughput on structured tasks without sacrificing quality.
Large language models (LLMs) are increasingly used for program-aided reasoning, agentic decision making, and structured task execution, but these applications often incur high inference cost. We present MiniCache, a reusable program caching framework that transforms Program-of-Thought (PoT) programs into parameterized cache objects, enabling reusable computation across structurally similar requests. MiniCache reuses the same small model for semantic variable extraction on cache-hit requests and speculative drafting during target-LLM generation, reducing expensive target-LLM invocations while preserving task quality. Experiments on shopping-style request datasets, WebShop, Formula, and CodeTAT-QA demonstrate that MiniCache improves the trade-off between inference latency, cache reuse, and accuracy, achieving up to 3.1x lower latency and 2.8x higher throughput under parallel serving. These results show that small models are most effective not as replacements for large models, but as lightweight interface models that enable reliable and efficient reusable program caching.