Robustness of Neurosymbolic Reasoners on First-Order Logic Problems
This addresses the issue of logical consistency in AI reasoning for NLP applications, but it is incremental as it builds on existing methods without achieving SOTA.
The paper tackled the problem of improving the robustness of reasoning systems on first-order logic problems under counterfactual variations, finding that neurosymbolic methods are more robust but perform worse overall than purely neural methods, and that combining neurosymbolic with Chain-of-Thought prompting improves performance but still lags behind standard CoT.
Recent trends in NLP aim to improve reasoning capabilities in Large Language Models (LLMs), with key focus on generalization and robustness to variations in tasks. Counterfactual task variants introduce minimal but semantically meaningful changes to otherwise valid first-order logic (FOL) problem instances altering a single predicate or swapping roles of constants to probe whether a reasoning system can maintain logical consistency under perturbation. Previous studies showed that LLMs becomes brittle on counterfactual variations, suggesting that they often rely on spurious surface patterns to generate responses. In this work, we explore if a neurosymbolic (NS) approach that integrates an LLM and a symbolic logical solver could mitigate this problem. Experiments across LLMs of varying sizes show that NS methods are more robust but perform worse overall that purely neural methods. We then propose NSCoT that combines an NS method and Chain-of-Thought (CoT) prompting and demonstrate that while it improves performance, NSCoT still lags behind standard CoT. Our analysis opens research directions for future work.