Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data
For practitioners using LLMs for structured data generation, the paper reveals a fundamental limitation of in-context learning and a trade-off with fine-tuning regarding adaptability and privacy.
The paper identifies a failure mode called categorical prior lock-in, where in-context learning cannot update LLMs' prior over token distributions for structured data generation, leading to inability to reproduce rare classes. Parameter-efficient fine-tuning (LoRA) overcomes this but introduces memorization risk and destabilization.
Large language models (LLMs) are increasingly used as conditional generators for structured data, relying on in-context learning (ICL) to adapt to new distributions without parameter updates. We investigate the limits of ICL for structured generation under distribution mismatch, using high-cardinality tabular data as a controlled test case, and identify a structural failure mode we term \textit{categorical prior lock-in}: the inability of ICL to update the model's prior over token distributions inherited from pre-training. Across two 7B-parameter open-weight models, ICL improves numerical fidelity with additional examples but exhibits a sharp ceiling on categorical distributions, failing to reproduce rare classes entirely. Parameter-efficient fine-tuning (LoRA) overcomes these limitations but introduces measurable memorization risk and, in some cases, destabilizes structured output generation, highlighting a fundamental trade-off between adaptability and privacy.