In-Context Learning with Hypothesis-Class Guidance
This work addresses the gap in ICL research by incorporating task instructions, which is incremental but important for improving practical applications in machine learning.
The paper tackles the problem of in-context learning (ICL) by proposing a synthetic data model called ICL-HCG that includes hypothesis-class guidance alongside labeled examples, showing that Transformers can learn this model and generalize to unseen hypotheses and classes, with ICL-HCG achieving significantly higher accuracy than ICL without instructions.
Recent research has investigated the underlying mechanisms of in-context learning (ICL) both theoretically and empirically, often using data generated from simple function classes. However, the existing work often focuses on the sequence consisting solely of labeled examples, while in practice, labeled examples are typically accompanied by an instruction, providing some side information about the task. In this work, we propose ICL with hypothesis-class guidance (ICL-HCG), a novel synthetic data model for ICL where the input context consists of the literal description of a (finite) hypothesis class H and $(x,y)$ pairs from a hypothesis chosen from H. Under our framework ICL-HCG, we conduct extensive experiments to explore: (i) a variety of generalization abilities to new hypothesis classes; (ii) different model architectures; (iii) sample complexity; (iv) in-context data imbalance; (v) the role of instruction; and (vi) the effect of pretraining hypothesis diversity. As a result, we show that (a) Transformers can successfully learn ICL-HCG and generalize to unseen hypotheses and unseen hypothesis classes, and (b) compared with ICL without instruction, ICL-HCG achieves significantly higher accuracy, demonstrating the role of instructions.