Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?
This work provides a benchmark and analysis for evaluating and improving language models' ability to follow formal modal logic semantics, which is important for reliable reasoning in AI systems.
The paper tests whether language models follow modal logic specifications by constructing paired problems with identical premises but different frame/domain conditions, verified by automated reasoning. Results show four of five models perform below baseline under direct prompting, but DeepSeek V4 Flash improves from 4.4% to 88.1% with reasoning mode, indicating inference mode and model identity strongly affect adherence to stipulated semantics.
Reasoning about necessity and possibility depends on assumptions about accessibility between worlds and about which objects exist at each one. The same inference may therefore hold under one modal system and fail under another. Evaluating language models on such problems requires testing whether their judgments follow the stated semantics rather than a familiar logic. We construct paired modal problems with identical premises and conjecture but different frame or domain conditions; automated reasoning verifies opposite labels. A balanced core prevents the semantic condition alone from revealing the answer. On this core, four of five recent models perform below the condition-only baseline under direct prompting. Yet enabling reasoning mode raises DeepSeek V4 Flash from 4.4% to 88.1% on unchanged prompts. Following stipulated modal semantics thus depends strongly on inference mode as well as model identity. When frame conditions are omitted, models often agree but fit different familiar logics best. We release the formulas, oracle artifacts, countermodels, and responses.