Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents
This addresses the problem of benchmarking and enhancing LLM agents for interactive data analysis, which is incremental as it builds on existing LLM agent frameworks with a new benchmark and method.
The paper tackles the challenge of evaluating LLM agents in interactive data analysis by introducing Tapilot-Crossing, a benchmark with 1024 interactions across four scenarios, and proposes Adaptive Interaction Reflection (AIR), which improves LLM performance by up to 44.5%.
Interactive Data Analysis, the collaboration between humans and LLM agents, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic interactive logs for data analysis hinder the quantitative evaluation of Large Language Model (LLM) agents in this task. To mitigate this issue, we introduce Tapilot-Crossing, a new benchmark to evaluate LLM agents on interactive data analysis. Tapilot-Crossing contains 1024 interactions, covering 4 practical scenarios: Normal, Action, Private, and Private Action. Notably, Tapilot-Crossing is constructed by an economical multi-agent environment, Decision Company, with few human efforts. We evaluate popular and advanced LLM agents in Tapilot-Crossing, which underscores the challenges of interactive data analysis. Furthermore, we propose Adaptive Interaction Reflection (AIR), a self-generated reflection strategy that guides LLM agents to learn from successful history. Experiments demonstrate that Air can evolve LLMs into effective interactive data analysis agents, achieving a relative performance improvement of up to 44.5%.