CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
This work addresses the problem of enhancing API-driven chatbot effectiveness for software engineers, though it is incremental as it builds on existing methods with new data and hybrid techniques.
The paper tackled the challenge of accurately generating and executing API calls in chatbot systems, particularly for multi-step interactions with complex dependencies, by introducing a novel dataset, empirically evaluating language models, and proposing a hybrid routing approach, which improved API execution in software engineering applications.
API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.