IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation
This addresses the challenge of domain adaptation for LLMs in settings where base model weights or external resources are unavailable, offering a practical solution for real-world applications.
The paper tackles the problem of adapting large language models to new domains via continual pretraining without degrading instruction-following capabilities, proposing IKnow, a framework that uses self-supervised objectives in a dialogue format to encode domain knowledge from the text itself, achieving effective domain adaptation.
Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database - both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation (IKnow), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, IKnow leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level.