CausalARC: Abstract Reasoning with Causal World Models
This work addresses the challenge of robust reasoning for AI systems in low-data and out-of-distribution scenarios, but it is incremental as it builds on existing frameworks like the Abstraction and Reasoning Corpus.
The authors tackled the problem of AI reasoning under limited data and distribution shift by introducing CausalARC, a testbed based on causal world models, and found that language model performance varied heavily across tasks, indicating significant room for improvement.
On-the-fly reasoning often requires adaptation to novel problems under limited data and distribution shift. This work introduces CausalARC: an experimental testbed for AI reasoning in low-data and out-of-distribution regimes, modeled after the Abstraction and Reasoning Corpus (ARC). Each CausalARC reasoning task is sampled from a fully specified causal world model, formally expressed as a structural causal model. Principled data augmentations provide observational, interventional, and counterfactual feedback about the world model in the form of few-shot, in-context learning demonstrations. As a proof-of-concept, we illustrate the use of CausalARC for four language model evaluation settings: (1) abstract reasoning with test-time training, (2) counterfactual reasoning with in-context learning, (3) program synthesis, and (4) causal discovery with logical reasoning. Within- and between-model performance varied heavily across tasks, indicating room for significant improvement in language model reasoning.