ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints
This work addresses a gap in assessing LLMs for AI systems that operate in dynamic environments, but it is incremental as it focuses on benchmarking rather than solving the underlying reasoning challenges.
The paper tackles the problem of evaluating Large Language Models (LLMs) in Reasoning about Actions and Change (RAC), a foundational AI area, by introducing ActionReasoningBench, a diagnostic benchmark with 8 domains and up to 19 action sequences. The result shows LLMs achieve average accuracy rates of 73.55% to 62.38% on four key dimensions but drop to 33.16% and 51.19% on more complex dimensions, with GPT-4o failing on ramification constraints and o1-preview scoring only 18.4%.
Reasoning about Actions and Change (RAC) has historically played a pivotal role in solving foundational AI problems, such as the frame problem. It has driven advancements in AI fields, such as non-monotonic and commonsense reasoning. RAC remains crucial for AI systems that operate in dynamic environments, engage in interactive scenarios, or rely on commonsense reasoning. Despite substantial advances made by Large Language Models (LLMs) in various AI domains, their performance in RAC remains underexplored. To address this gap, we introduce a new diagnostic benchmark, ActionReasoningBench, which encompasses 8 domains and includes questions for up to 19 action sequences. This benchmark rigorously evaluates LLMs across six key RAC dimensions: Fluent Tracking, State Tracking, Action Executability, Effects of Actions, Numerical RAC, and Composite Questions. LLMs demonstrate average accuracy rates of 73.55%, 65.63%, 58.73%, and 62.38% on the former four dimensions, which are frequently discussed in RAC literature. However, the performance on the latter two dimensions, which introduce complex and novel reasoning questions, the average performance of LLMs is lowered to 33.16% and 51.19%, respectively, reflecting a 17.9% performance decline. We also introduce new ramification constraints to capture the indirect effects of actions, providing deeper insights into RAC challenges. Our evaluation of state-of-the-art LLMs, including both open-source and commercial models, reveals challenges across all RAC dimensions, particularly in handling ramifications, with GPT-4o failing to solve any question and o1-preview achieving a score of only 18.4%.