Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
This addresses the challenge for AI developers of adapting language models to new skills cost-effectively, though it is incremental as it builds on existing model merging techniques.
The paper tackles the problem of efficiently adding new skills to language models without expensive retraining or forgetting old skills, finding that training on new skills in isolation and then merging with the general model is often comparably effective and cheaper.
Adapting general-purpose language models to new skills is currently an expensive process that must be repeated as new instruction datasets targeting new skills are created, or can cause the models to forget older skills. In this work, we investigate the effectiveness of adding new skills to preexisting models by training on the new skills in isolation and later merging with the general model (e.g. using task vectors). In experiments focusing on scientific literature understanding, safety, and coding, we find that the parallel-train-then-merge procedure, which is significantly cheaper than retraining the models on updated data mixtures, is often comparably effective. Our experiments also show that parallel training is especially well-suited for enabling safety features in LMs relative to continued finetuning and retraining, as it dramatically improves model compliance with safe prompts while preserving its ability to refuse dangerous or harmful prompts.