Learning Local Causal World Models with State Space Models and Attention
This work addresses the need for causal understanding in AI agents for complex tasks, but it is incremental as it builds on existing research at the intersection of causality and neural world modelling.
The paper tackled the problem of learning causal representations in world modelling by comparing State Space Models (SSMs) to Transformers, showing that SSMs can model dynamics and learn causal models with equivalent or better performance in a simple environment.
World modelling, i.e. building a representation of the rules that govern the world so as to predict its evolution, is an essential ability for any agent interacting with the physical world. Despite their impressive performance, many solutions fail to learn a causal representation of the environment they are trying to model, which would be necessary to gain a deep enough understanding of the world to perform complex tasks. With this work, we aim to broaden the research in the intersection of causality theory and neural world modelling by assessing the potential for causal discovery of the State Space Model (SSM) architecture, which has been shown to have several advantages over the widespread Transformer. We show empirically that, compared to an equivalent Transformer, a SSM can model the dynamics of a simple environment and learn a causal model at the same time with equivalent or better performance, thus paving the way for further experiments that lean into the strength of SSMs and further enhance them with causal awareness.