ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution
This work addresses the problem of robust evaluation for LLM-powered digital assistants in complex task execution, though it is incremental as it focuses on benchmarking rather than new methods.
The authors tackled the challenge of evaluating large language models (LLMs) for complex action execution in digital assistants by developing ASPERA, a framework that includes a simulation and data generation engine, and released Asper-Bench, a dataset of 250 tasks, showing that program generation with custom libraries is significantly harder for LLMs than dependency-free code generation.
This work evaluates the potential of large language models (LLMs) to power digital assistants capable of complex action execution. These assistants rely on pre-trained programming knowledge to execute multi-step goals by composing objects and functions defined in assistant libraries into action execution programs. To achieve this, we develop ASPERA, a framework comprising an assistant library simulation and a human-assisted LLM data generation engine. Our engine allows developers to guide LLM generation of high-quality tasks consisting of complex user queries, simulation state and corresponding validation programs, tackling data availability and evaluation robustness challenges. Alongside the framework we release Asper-Bench, an evaluation dataset of 250 challenging tasks generated using ASPERA, which we use to show that program generation grounded in custom assistant libraries is a significant challenge to LLMs compared to dependency-free code generation.