Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities
This addresses the issue of knowledge obsolescence in LLMs for users needing up-to-date information without losing post-training skills, though it is incremental as it builds on existing distillation techniques.
The paper tackles the problem of continual knowledge adaptation in post-trained LLMs, which suffer from forgetting earlier capabilities when learning new knowledge, and introduces DiSC, a context-distillation method that achieves the best trade-off between learning new knowledge and mitigating forgetting across four models and two domains.
Post-training endows pretrained LLMs with a variety of desirable skills, including instruction-following, reasoning, and others. However, these post-trained LLMs only encode knowledge up to a cut-off date, necessitating continual adaptation. Unfortunately, existing solutions cannot simultaneously learn new knowledge from an adaptation document corpora and mitigate the forgetting of earlier learned capabilities. To address this, we introduce Distillation via Split Contexts (DiSC), a simple context-distillation based approach for continual knowledge adaptation. \methodname~derives student and teacher distributions by conditioning on distinct segments of the training example and minimizes the KL divergence between the shared tokens. This allows us to efficiently apply context-distillation without requiring explicit generation steps during training. We run experiments on four post-trained models and two adaptation domains. Compared to prior finetuning and distillation methods for continual adaptation, DiSC consistently reports the best trade-off between learning new knowledge and mitigating forgetting of previously learned skills like instruction-following, reasoning, and factual knowledge.