BLAST: Benchmarking LLMs with ASP-based Structured Testing
This work provides a structured evaluation framework for assessing LLMs' ability to generate declarative code, addressing a gap in benchmarking for ASP, though it is domain-specific and incremental.
BLAST introduces the first benchmark for evaluating LLMs on Answer Set Programming (ASP) code generation, using two novel semantic metrics. Evaluation on ten graph problems and eight LLMs shows that while some models achieve high accuracy on simple tasks, performance degrades on complex problems, with the best model achieving 85% accuracy on the simplest task.
Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dialogue systems, and code generation. Despite evident progress, less attention has been paid to their effectiveness in handling declarative paradigms such as Answer Set Programming (ASP), to date. In this paper we introduce BLAST: The first dedicated benchmarking methodology and associated dataset for evaluating the accuracy of LLMs in generating ASP code. BLAST provides a structured evaluation framework featuring two novel semantic metrics tailored to ASP code generation. The paper presents the results of an empirical evaluation involving ten well-established graph-related problems from the ASP literature and a diverse set of eight state-of-the-art LLMs.