Conditional Language Learning with Context
This addresses the issue of selective learning in domain-specific finetuning for language models, though it appears incremental as a modification to existing methods.
The paper tackles the problem of language models learning useless corpus statistics and biases during finetuning by proposing conditional finetuning, which conditions language modeling on context to selectively learn useful knowledge, resulting in less forgetting and better stability-plasticity tradeoff.
Language models can learn sophisticated language understanding skills from fitting raw text. They also unselectively learn useless corpus statistics and biases, especially during finetuning on domain-specific corpora. In this paper, we propose a simple modification to causal language modeling called conditional finetuning, which performs language modeling conditioned on a context. We show that a context can "explain away" certain corpus statistics and make the model avoid learning them. In this fashion, conditional finetuning achieves selective learning from a corpus, learning knowledge useful for downstream tasks while avoiding learning useless corpus statistics like topic biases. This selective learning effect leads to less forgetting and better stability-plasticity tradeoff in domain finetuning, potentially benefitting lifelong learning with language models.