IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning
This work addresses the challenge of model adaptation for AI practitioners by offering a practical improvement over standard SFT, though it is incremental as it builds on existing methods.
The paper tackled the problem of improving Supervised Fine-Tuning (SFT) by leveraging In-Context Learning (ICL) activations, resulting in significantly enhanced accuracy and calibration of model outputs across 12 benchmarks and 2 model families.
Supervised Fine-Tuning (SFT) is used to specialize model behavior by training weights to produce intended target responses for queries. In contrast, In-Context Learning (ICL) adapts models during inference with instructions or demonstrations in the prompt. ICL can offer better generalizability and more calibrated responses compared to SFT in data scarce settings, at the cost of more inference compute. In this work, we ask the question: Can ICL's internal computations be used to improve the qualities of SFT? We first show that ICL and SFT produce distinct activation patterns, indicating that the two methods achieve adaptation through different functional mechanisms. Motivated by this observation and to use ICL's rich functionality, we introduce ICL Activation Alignment (IA2), a self-distillation technique which aims to replicate ICL's activation patterns in SFT models and incentivizes ICL-like internal reasoning. Performing IA2 as a priming step before SFT significantly improves the accuracy and calibration of model outputs, as shown by our extensive empirical results on 12 popular benchmarks and 2 model families. This finding is not only practically useful, but also offers a conceptual window into the inner mechanics of model adaptation.