Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
For practitioners of recurrent memory models in language modeling, EDA provides a simple yet effective improvement that enhances memory management without sacrificing the corrective behavior of delta-rule updates.
Erase-then-Delta Attention (EDA) decouples erase and write addresses in delta-rule linear attention, enabling selective removal of outdated memory before writing new content. In language-model pretraining experiments with dense 2.5B and MoE 25B-A2.8B models, EDA achieves best performance, with gains persisting after 80B-token long-context midtraining and across 4k to 128k context lengths.
Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active correction is still anchored to that same write address. As a result, stale information stored at a different address cannot be actively removed before new content is written elsewhere. We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write. The key insight is that recurrent memory models should not only correct the current write, but also selectively suppress outdated memory at an independently chosen address. Concretely, our method first applies a targeted erase step along a learned erase direction, and then performs the standard delta-style corrective write along the current write direction. This preserves the corrective behavior of delta-rule updates while expanding their memory-management capacity. Language-model pretraining experiments across dense 2.5B and MoE 25B-A2.8B model families show that EDA performs best in both settings. The gain persists after 80B-token long-context midtraining of the MoE models, where EDA also performs best in long-context evaluations from 4k to 128k contexts. A compact update analysis and memory-state probes suggest why: EDA keeps the delta-rule corrective write intact while allocating an additional cleanup path most strongly when passive decay is weak. These results suggest that recurrent memory models should decide not only what to write, but also what stale information to erase and where.