CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models
This provides a crucial benchmark for developing and evaluating causal reasoning in language models, which is essential for applications requiring accurate causal assessment.
The authors tackled the problem of evaluating whether language models can distinguish genuine causal explanatory relationships from semantic relatedness, and found that models frequently confuse the two, with performance plateauing at a Matthews Correlation Coefficient of 0.55 even for the best models.
We introduce CLEAR-3K, a dataset of 3,000 assertion-reasoning questions designed to evaluate whether language models can determine if one statement causally explains another. Each question present an assertion-reason pair and challenge language models to distinguish between semantic relatedness and genuine causal explanatory relationships. Through comprehensive evaluation of 21 state-of-the-art language models (ranging from 0.5B to 72B parameters), we identify two fundamental findings. First, language models frequently confuse semantic similarity with causality, relying on lexical and semantic overlap instead of inferring actual causal explanatory relationships. Second, as parameter size increases, models tend to shift from being overly skeptical about causal relationships to being excessively permissive in accepting them. Despite this shift, performance measured by the Matthews Correlation Coefficient plateaus at just 0.55, even for the best-performing models.Hence, CLEAR-3K provides a crucial benchmark for developing and evaluating genuine causal reasoning in language models, which is an essential capability for applications that require accurate assessment of causal relationships.