DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues
This work addresses the gap in evaluating LLMs' tool-use capabilities in realistic, complex dialogue scenarios, which is incremental as it builds on existing benchmarks by adding multi-round and multi-party aspects.
The authors tackled the problem that existing function-calling benchmarks for large language models (LLMs) overlook multi-round, multi-party dialogues, and they introduced DICE-BENCH, a framework that constructs a dataset of 1,607 instances to evaluate LLMs, showing that significant advances are still needed for real-world deployment.
Existing function-calling benchmarks focus on single-turn interactions. However, they overlook the complexity of real-world scenarios. To quantify how existing benchmarks address practical applications, we introduce DICE-SCORE, a metric that evaluates the dispersion of tool-related information such as function name and parameter values throughout the dialogue. Analyzing existing benchmarks through DICE-SCORE reveals notably low scores, highlighting the need for more realistic scenarios. To address this gap, we present DICE-BENCH, a framework that constructs practical function-calling datasets by synthesizing conversations through a tool graph that maintains dependencies across rounds and a multi-agent system with distinct personas to enhance dialogue naturalness. The final dataset comprises 1,607 high-DICE-SCORE instances. Our experiments on 19 LLMs with DICE-BENCH show that significant advances are still required before such models can be deployed effectively in real-world settings. Our code and data are all publicly available: https://snuhcc.github.io/DICE-Bench/.