Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
This work addresses the problem of improving instruction-tuning data quality for LLMs, offering a novel approach that consistently boosts performance over current methods.
The authors propose CoDIT, a method using contrastive decoding between post-trained and pre-trained models to generate instruction-tuning datasets that better separate instruction-following from world knowledge. Models trained on CoDIT datasets outperform those trained on directly generated responses and existing public datasets across multiple benchmarks.
Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.