Assessing the Limits of In-Context Learning beyond Functions using Partially Ordered Relation
This work addresses a fundamental limitation in ICL for structured tasks, which is incremental but important for understanding model capabilities in AI research.
The paper investigates the limits of in-context learning (ICL) in large language models when applied to partially ordered relations, showing that performance saturates and effectiveness remains constrained as prompt complexity increases, even with sufficient examples.
Generating rational and generally accurate responses to tasks, often accompanied by example demonstrations, highlights Large Language Model's (LLM's) remarkable In-Context Learning (ICL) capabilities without requiring updates to the model's parameter space. Despite having an ongoing exploration focused on the inference from a document-level concept, its behavior in learning well-defined functions or relations in context needs a careful investigation. In this article, we present the performance of ICL on partially ordered relation by introducing the notion of inductively increasing complexity in prompts. In most cases, the saturated performance of the chosen metric indicates that while ICL offers some benefits, its effectiveness remains constrained as we increase the complexity in the prompts even in presence of sufficient demonstrative examples. The behavior is evident from our empirical findings and has further been theoretically justified in term of its implicit optimization process. The code is available \href{https://anonymous.4open.science/r/ICLonPartiallyOrderSet}{here}.