SEAL: Suite for Evaluating API-use of LLMs
This work addresses the need for stable and comprehensive evaluation of LLMs' API-use capabilities for researchers and developers, though it is incremental as it builds on existing benchmarks.
The paper tackles the problem of evaluating large language models (LLMs) in real-world API usage by introducing SEAL, a testbed that standardizes benchmarks and uses a GPT-4-powered simulator for deterministic evaluations, resulting in a reliable framework for structured performance comparison.
Large language models (LLMs) have limitations in handling tasks that require real-time access to external APIs. While several benchmarks like ToolBench and APIGen have been developed to assess LLMs' API-use capabilities, they often suffer from issues such as lack of generalizability, limited multi-step reasoning coverage, and instability due to real-time API fluctuations. In this paper, we introduce SEAL, an end-to-end testbed designed to evaluate LLMs in real-world API usage. SEAL standardizes existing benchmarks, integrates an agent system for testing API retrieval and planning, and addresses the instability of real-time APIs by introducing a GPT-4-powered API simulator with caching for deterministic evaluations. Our testbed provides a comprehensive evaluation pipeline that covers API retrieval, API calls, and final responses, offering a reliable framework for structured performance comparison in diverse real-world scenarios. SEAL is publicly available, with ongoing updates for new benchmarks.