Predicting Causal Effects from Natural Language Queries using Structured Representations
For researchers needing causal effect estimates from text, this work provides a benchmark and method that significantly outperforms direct LLM prompting, though it is incremental in combining existing ideas.
The authors introduce Query2Effect, a benchmark of 72,000+ natural language queries aligned with experiment descriptions, and propose a two-step framework that generates structured representations before predicting causal effect sizes. Their method reduces absolute error by 27-71% compared to prompted LLMs.
Randomized controlled trials are a cornerstone of medicine and the social sciences as they enable reliable estimates of causal effects. However, they are costly and time-consuming to conduct, motivating interest in predicting causal effects from existing experimental evidence. Recent advances in large language models (LLMs) have demonstrated strong performance on knowledge-intensive tasks, raising the question of whether these models can be used for forecasting causal effect sizes. To investigate this, we introduce Query2Effect, a new large-scale benchmark consisting of more than 72,000 natural language questions aligned with experiment descriptions, created to simulate realistic information-seeking scenarios by varying query specificity along dimensions of implicitness, abstraction, and ambiguity. We then propose a two-step framework that first generates a synthetic structured representation of a query before predicting effect size using a supervised encoder model. Experiments show that finetuning plays a crucial role in improving prediction performance, with absolute error reducing by -27% up to -71% compared to prompted out-of-the-box LLMs, and that our two-step framework is beneficial for out-of-domain generalization, highlighting the benefits of separating semantic interpretation from numerical effect estimation.