Shaking the foundations: delusions in sequence models for interaction and control
This addresses a foundational issue in AI for improving adaptive behavior in sequence models, though it appears incremental as it builds on existing causal intervention concepts.
The paper tackles the problem of sequence models lacking understanding of cause and effect, leading to incorrect inferences, and resolves it by treating actions as causal interventions, enabling training with factual and counterfactual error signals.
The recent phenomenal success of language models has reinvigorated machine learning research, and large sequence models such as transformers are being applied to a variety of domains. One important problem class that has remained relatively elusive however is purposeful adaptive behavior. Currently there is a common perception that sequence models "lack the understanding of the cause and effect of their actions" leading them to draw incorrect inferences due to auto-suggestive delusions. In this report we explain where this mismatch originates, and show that it can be resolved by treating actions as causal interventions. Finally, we show that in supervised learning, one can teach a system to condition or intervene on data by training with factual and counterfactual error signals respectively.