Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies
For researchers and developers of LLMs, this benchmark addresses the underexplored problem of robustness against fallacious persuasion, providing a systematic evaluation tool.
The paper introduces LoFa, a benchmark to evaluate LLM robustness against logical fallacies, and proposes LFR@k metric. Experiments reveal varying vulnerability profiles across models and fallacy types.
Large Language Models (LLMs) exhibit strong semantic capabilities, yet their resilience to manipulative linguistic patterns such as logical fallacies remains underexplored. Prior work has primarily examined whether LLMs can identify or classify fallacies, leaving their robustness against fallacious persuasion insufficiently studied. To address this gap, we introduce LoFa (Logical Fallacy), a comprehensive benchmark for evaluating LLM robustness against fallacies. LoFa is constructed through a multi-agent pipeline that pairs factual questions with fallacious arguments, and is accompanied by a multi-round debate framework for assessing model resilience under sustained adversarial persuasion. To disentangle fallacy robustness from a model's inherent knowledge limitations, we further propose Logical Fallacy Resistance at k (LFR@k), a metric that quantifies resistance to fallacious attacks. Experiments show that LLMs exhibit varying levels of robustness across different fallacy types, revealing distinct vulnerability profiles among models.