On the Generalization and Adaption Performance of Causal Models
This work addresses the problem of improving model robustness and adaptation for machine learning practitioners, but it is incremental as it builds on existing differentiable causal discovery methods.
The paper tackles the challenge of robust out-of-distribution generalization and fast adaptation in machine learning by studying modular neural causal models, which factorize data into causal modules. The results show that these models outperform monolithic and structured models in zero and few-shot adaptation in low data regimes, with more significant effects for sparser graphs.
Learning models that offer robust out-of-distribution generalization and fast adaptation is a key challenge in modern machine learning. Modelling causal structure into neural networks holds the promise to accomplish robust zero and few-shot adaptation. Recent advances in differentiable causal discovery have proposed to factorize the data generating process into a set of modules, i.e. one module for the conditional distribution of every variable where only causal parents are used as predictors. Such a modular decomposition of knowledge enables adaptation to distributions shifts by only updating a subset of parameters. In this work, we systematically study the generalization and adaption performance of such modular neural causal models by comparing it to monolithic models and structured models where the set of predictors is not constrained to causal parents. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes and offer robust generalization. We also found that the effects are more significant for sparser graphs as compared to denser graphs.