AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language Models
This addresses a knowledge gap for LLM users by evaluating model robustness to contradictory assertions, though it is incremental as it builds on existing fact verification datasets.
The paper tackles the problem of how directional framing of factually true statements influences agreement in Large Language Models (LLMs), and introduces AssertBench, a benchmark that measures an LLM's ability to maintain consistent truth evaluation across different user framings, with results showing variability in model self-assertion.
Recent benchmarks have probed factual consistency and rhetorical robustness in Large Language Models (LLMs). However, a knowledge gap exists regarding how directional framing of factually true statements influences model agreement, a common scenario for LLM users. AssertBench addresses this by sampling evidence-supported facts from FEVEROUS, a fact verification dataset. For each (evidence-backed) fact, we construct two framing prompts: one where the user claims the statement is factually correct, and another where the user claims it is incorrect. We then record the model's agreement and reasoning. The desired outcome is that the model asserts itself, maintaining consistent truth evaluation across both framings, rather than switching its evaluation to agree with the user. AssertBench isolates framing-induced variability from the model's underlying factual knowledge by stratifying results based on the model's accuracy on the same claims when presented neutrally. In doing so, this benchmark aims to measure an LLM's ability to "stick to its guns" when presented with contradictory user assertions about the same fact. The complete source code is available at https://github.com/achowd32/assert-bench.