Dynamic Expert-Guided Model Averaging for Causal Discovery
This work addresses the problem of algorithm selection in causal discovery for healthcare practitioners, offering an incremental improvement by integrating expert knowledge into ensembling.
The paper tackles the challenge of selecting among many causal discovery algorithms by introducing a model averaging method that dynamically incorporates expert knowledge, including from LLMs, to ensemble diverse algorithms. Experiments show the method's efficacy with imperfect experts on both clean and noisy data, analyzing expert correctness and LLM capabilities for clinical applications.
Understanding causal relationships is critical for healthcare. Accurate causal models provide a means to enhance the interpretability of predictive models, and furthermore a basis for counterfactual and interventional reasoning and the estimation of treatment effects. However, would-be practitioners of causal discovery face a dizzying array of algorithms without a clear best choice. This abundance of competitive algorithms makes ensembling a natural choice for practical applications. At the same time, real-world use cases frequently face challenges that violate the assumptions of common causal discovery algorithms, forcing heavy reliance on expert knowledge. Inspired by recent work on dynamically requested expert knowledge and LLMs as experts, we present a flexible model averaging method leveraging dynamically requested expert knowledge to ensemble a diverse array of causal discovery algorithms. Experiments demonstrate the efficacy of our method with imperfect experts such as LLMs on both clean and noisy data. We also analyze the impact of different degrees of expert correctness and assess the capabilities of LLMs for clinical causal discovery, providing valuable insights for practitioners.