ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling
This addresses latency issues for users of LLM-based Function Calling in production environments, though it appears incremental as it builds on existing distillation techniques.
The paper tackles the high latency problem in LLM-based Function Calling by proposing ODIA, which uses online user interaction data to distill knowledge from larger to smaller models, reducing response latency by 45% (expected) and 78% (median) while maintaining accuracy.
Function Calling is a crucial technique that enables Large Language Models (LLMs) to interact with external systems through APIs. However, the high latency associated with LLM-based Function Calling significantly impacts user experience. This paper presents a novel approach called Oriented Distillation for Inline Acceleration (ODIA) that leverages online user interaction data to accelerate Function Calling. By automatically identifying "simple queries" from production traffic and distilling knowledge from larger models to smaller ones, our method reduces response latency by 45% (expected) and 78% (median) while maintaining accuracy. We demonstrate the effectiveness of our approach through real-world deployment in a music application, where the smaller model successfully handles 60% of traffic with negligible accuracy loss. Our method requires minimal human intervention and continuously improves through automated data collection and model updating, making it a practical solution for production environments.