TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses
This addresses the challenge of preserving relevance in cached responses for high-volume LLM deployments, offering a scalable solution without compromising user experience.
The paper tackles the problem of efficiently caching responses for large language models (LLMs) to reduce cost and latency, and demonstrates that TweakLLM maintains response quality comparable to frontier models while significantly improving cache effectiveness.
Large Language Models (LLMs) process millions of queries daily, making efficient response caching a compelling optimization for reducing cost and latency. However, preserving relevance to user queries using this approach proves difficult due to the personalized nature of chatbot interactions and the limited accuracy of semantic similarity search. To address this, we present TweakLLM, a novel routing architecture that employs a lightweight LLM to dynamically adapt cached responses to incoming prompts. Through comprehensive evaluation, including user studies with side-by-side comparisons, satisfaction voting, as well as multi-agent LLM debates, we demonstrate that TweakLLM maintains response quality comparable to frontier models while significantly improving cache effectiveness. Our results across real-world datasets highlight TweakLLM as a scalable, resource-efficient caching solution for high-volume LLM deployments without compromising user experience.