Memorization Sinks: Isolating Memorization during LLM Training
This addresses privacy and copyright issues for users of large language models by enabling easier removal of memorized content without harming general language abilities, representing a novel approach rather than an incremental improvement.
The paper tackles the problem of large language models memorizing repeated sequences, which raises privacy and copyright concerns, by introducing MemSinks, a new paradigm that isolates memorization during training, and demonstrates effective isolation and strong generalization at billion-parameter and billion-token scales.
Large language models are susceptible to memorizing repeated sequences, posing privacy and copyright concerns. A popular mitigation strategy is to remove memorized information from specific neurons post-hoc. However, such approaches have shown limited success so far. In a controlled setting, we show that the memorization of natural sequences (those that resemble linguistically plausible text) become mechanistically entangled with general language abilities, thereby becoming challenging to remove post-hoc. In this work, we put forward a new paradigm of MemSinks that promotes isolation of memorization by design. We leverage a sequence identifier that activates a unique set of memorization neurons for each sequence across repetitions. By analyzing the dynamics of learning and forgetting, we argue that MemSinks facilitates isolation of memorized content, making it easier to remove without compromising general language capabilities. We implement MemSinks at the billion-parameter and billion-token scale, and observe both effective isolation and strong generalization. To our knowledge, this is the first proof-of-concept on real data demonstrating that simultaneous generalization and isolation is achievable. We open-source our code at http://github.com/grghosal/MemSinks.