Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs
This addresses the challenge of efficient knowledge injection in LLMs, though it is incremental in exploring memorization techniques.
The paper tackles the problem of enabling LLMs to generalize from rote memorized data, showing that models can reinterpret memorized facts through semantically meaningful prompts, with experiments across 8 LLMs demonstrating structured latent representations.
Rote learning is a memorization technique based on repetition. It is commonly believed to hinder generalization by encouraging verbatim memorization rather than deeper understanding. This insight holds for even learning factual knowledge that inevitably requires a certain degree of memorization. In this work, we demonstrate that LLMs can be trained to generalize from rote memorized data. We introduce a two-phase memorize-then-generalize framework, where the model first rote memorizes factual subject-object associations using a semantically meaningless token and then learns to generalize by fine-tuning on a small set of semantically meaningful prompts. Extensive experiments over 8 LLMs show that the models can reinterpret rote memorized data through the semantically meaningful prompts, as evidenced by the emergence of structured, semantically aligned latent representations between the two. This surprising finding opens the door to both effective and efficient knowledge injection and possible risks of repurposing the memorized data for malicious usage.