Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models
This work addresses a fundamental research question in AI about how LLMs learn in-context, which is incremental as it builds on existing debates but provides a new evaluation framework.
The paper tackles the problem of understanding the mechanisms behind in-context learning in large language models by proposing a framework to evaluate how models combine retrieval of internal knowledge and learning from in-context examples in regression tasks, showing that LLMs can solve real-world regression problems and analyzing factors influencing these mechanisms.
Generative Large Language Models (LLMs) are capable of being in-context learners. However, the underlying mechanism of in-context learning (ICL) is still a major research question, and experimental research results about how models exploit ICL are not always consistent. In this work, we propose a framework for evaluating in-context learning mechanisms, which we claim are a combination of retrieving internal knowledge and learning from in-context examples by focusing on regression tasks. First, we show that LLMs can solve real-world regression problems and then design experiments to measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. We argue that this process lies on a spectrum between these two extremes. We provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on various factors, such as prior knowledge about the tasks and the type and richness of the information provided by the in-context examples. We employ three LLMs and utilize multiple datasets to corroborate the robustness of our findings. Our results shed light on how to engineer prompts to leverage meta-learning from in-context examples and foster knowledge retrieval depending on the problem being addressed.