Consistency evaluation of benchmarks used for causal discovery
For researchers evaluating causal discovery methods, this work highlights the problem of mis-aligned knowledge in benchmarks, especially affecting LLM-based methods.
This work systematically evaluates the quality of benchmark causal graphs used for causal discovery, finding that popular benchmarks vary significantly in their consistency with domain research, which has clear implications for causal discovery research.
In graphical causal model, causal discovery aims to construct a causal graph based on numerical data and domain knowledge in plain text. However, the evaluation of causal discovery methods remains a challenge in the area as the progress of domain researches often makes benchmark causal graphs contain mis-aligned knowledge. This problem especially affects the evaluation of large language model (LLM) based causal discovery methods as they are sensitive to the new discoveries in the literature. This work is the first to systematically study the quality of benchmark causal graphs. Specifically, we design a pipeline that automatically retrieves relevant research papers from scientific databases, and prompts LLMs to check the consistency between the benchmark causal graphs and domain research papers. We evaluate 11 popular real-world benchmarks, for which our pipeline in total proceeds 38,081 domain papers. Our results show that popular benchmarks vary significantly in their consistency with domain research, with clear implications for causal discovery research.