Exploring the Relationship between In-Context Learning and Instruction Tuning
This work provides insights into LLM behaviors by connecting two key adaptation paradigms, though it is incremental as it builds on existing research without broad practical impact.
The paper investigates the relationship between In-Context Learning (ICL) and Instruction Tuning (IT) in Large Language Models, finding through experiments with LLaMA-2 that ICL changes hidden states similarly to IT, indicating ICL is implicit IT, with convergence depending on demonstration factors.
In-Context Learning (ICL) and Instruction Tuning (IT) are two primary paradigms of adopting Large Language Models (LLMs) to downstream applications. However, they are significantly different. In ICL, a set of demonstrations are provided at inference time but the LLM's parameters are not updated. In IT, a set of demonstrations are used to tune LLM's parameters in training time but no demonstrations are used at inference time. Although a growing body of literature has explored ICL and IT, studies on these topics have largely been conducted in isolation, leading to a disconnect between these two paradigms. In this work, we explore the relationship between ICL and IT by examining how the hidden states of LLMs change in these two paradigms. Through carefully designed experiments conducted with LLaMA-2 (7B and 13B), we find that ICL is implicit IT. In other words, ICL changes an LLM's hidden states as if the demonstrations were used to instructionally tune the model. Furthermore, the convergence between ICL and IT is largely contingent upon several factors related to the provided demonstrations. Overall, this work offers a unique perspective to explore the connection between ICL and IT and sheds light on understanding the behaviors of LLM.