ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models
This addresses the need for more realistic evaluation benchmarks for TALMs, though it is incremental as it focuses on dataset creation rather than model advancement.
The authors tackled the problem of simplistic benchmark datasets for Tool-Augmented Language Models (TALMs) by constructing ToolDial, a dataset of 11,111 multi-turn dialogues with an average of 8.95 turns, which includes 16 user/system actions and simulates realistic scenarios like clarifying questions and proactive API calls. They evaluated language models on this dataset, finding accuracy scores below 70%, showing significant room for improvement.
Tool-Augmented Language Models (TALMs) leverage external APIs to answer user queries across various domains. However, existing benchmark datasets for TALM research often feature simplistic dialogues that do not reflect real-world scenarios, such as the need for models to ask clarifying questions or proactively call additional APIs when essential information is missing. To address these limitations, we construct and release ToolDial, a dataset comprising 11,111 multi-turn dialogues, with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial has two key characteristics. First, the dialogues incorporate 16 user and system actions (e.g., "Request", "Clarify", "Fail inform") to capture the rich dynamics of real-world interactions. Second, we simulate dialogues where the system requests necessary information from the user based on API documentation and seeks additional APIs if the user fails to provide the required information. To facilitate this process, we introduce a method for generating an API graph that represents input and output compatibility between APIs. Using ToolDial, we evaluate a suite of language models on their ability to predict correct actions and extract input parameter values for API calls from the dialogue history. Modern language models achieve accuracy scores below 70%, indicating substantial room for improvement. We release our dataset and code at https://github.com/holi-lab/ToolDial.