GeckOpt: LLM System Efficiency via Intent-Based Tool Selection
This addresses system efficiency and cost reduction for LLM-based platforms, though it appears incremental as it builds on existing intent-based methods.
The paper tackles the problem of inefficient tool selection in LLM systems by using a GPT-driven intent-based reasoning approach to narrow down required API tools at runtime, achieving up to 24.6% reduction in token consumption on a real-world Copilot platform with over 100 GPT-4-Turbo nodes.
In this preliminary study, we investigate a GPT-driven intent-based reasoning approach to streamline tool selection for large language models (LLMs) aimed at system efficiency. By identifying the intent behind user prompts at runtime, we narrow down the API toolset required for task execution, reducing token consumption by up to 24.6\%. Early results on a real-world, massively parallel Copilot platform with over 100 GPT-4-Turbo nodes show cost reductions and potential towards improving LLM-based system efficiency.