Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version)
This work addresses the lack of robust benchmarks for assessing LLM capabilities in non-monotonic reasoning, which is incremental as it builds on existing evaluations but provides more detailed insights into limitations.
The paper tackles the problem of evaluating large language models (LLMs) on Answer Set Programming (ASP) tasks by introducing ASPBench, a comprehensive benchmark with three ASP-specific tasks, and finds that while LLMs perform well on simpler tasks like entailment and verification, they struggle with core answer set computation.
Answer Set Programming (ASP) is a powerful paradigm for non-monotonic reasoning. Recently, large language models (LLMs) have demonstrated promising capabilities in logical reasoning. Despite this potential, current evaluations of LLM capabilities in ASP are often limited. Existing works normally employ overly simplified ASP programs, do not support negation, disjunction, or multiple answer sets. Furthermore, there is a lack of benchmarks that introduce tasks specifically designed for ASP solving. To bridge this gap, we introduce ASPBench, a comprehensive ASP benchmark, including three ASP specific tasks: ASP entailment, answer set verification, and answer set computation. Our extensive evaluations on ASPBench reveal that while 14 state-of-the-art LLMs, including \emph{deepseek-r1}, \emph{o4-mini}, and \emph{gemini-2.5-flash-thinking}, perform relatively well on the first two simpler tasks, they struggle with answer set computation, which is the core of ASP solving. These findings offer insights into the current limitations of LLMs in ASP solving. This highlights the need for new approaches that integrate symbolic reasoning capabilities more effectively. The code and dataset are available at https://github.com/HomuraT/ASPBench.