Reversal Invariance in Autoregressive Language Models
This identifies a fundamental limitation in current language modeling objectives that could affect all autoregressive models, though the analysis is theoretical without empirical validation.
The paper formalizes reversal invariance in autoregressive language models, showing that standard pretraining assigns identical likelihood to text and its reversal, which explains why models trained on reversed text perform comparably despite language's inherent time-asymmetry. It argues this invariance is a limitation that may fail to capture directional dependencies in language.
We formalize a structural property of the causal (autoregressive) language modeling (CLM) objective: reversal invariance. Formally, the next-token prediction loss assigns identical likelihood to a corpus and its reversal, implying that standard CLM pretraining is direction-blind. This symmetry explains why models trained on reversed text can achieve comparable performance to those trained on forward text, despite the inherently time-asymmetric nature of human language and reasoning. We argue that this invariance represents a limitation of current pretraining objectives rather than a benign artifact. If natural language encodes directional dependencies - phonological, morphological, or causal - a symmetric objective may fail to capture them. We therefore propose viewing pretraining through the lens of temporal asymmetry, motivating future work on loss functions and architectures that explicitly model the arrow of language while retaining standard language modeling capacity.