On the Power of Context-Enhanced Learning in LLMs
This addresses data security and copyright concerns for LLM training, though it is incremental as it builds on existing in-context learning concepts.
The authors formalized context-enhanced learning for LLMs, proving it can be exponentially more sample-efficient than standard learning in a simplified multi-step reasoning task, and experimentally showed it is hard to detect training materials used in the context.
We formalize a new concept for LLMs, context-enhanced learning. It involves standard gradient-based learning on text except that the context is enhanced with additional data on which no auto-regressive gradients are computed. This setting is a gradient-based analog of usual in-context learning (ICL) and appears in some recent works. Using a multi-step reasoning task, we prove in a simplified setting that context-enhanced learning can be exponentially more sample-efficient than standard learning when the model is capable of ICL. At a mechanistic level, we find that the benefit of context-enhancement arises from a more accurate gradient learning signal. We also experimentally demonstrate that it appears hard to detect or recover learning materials that were used in the context during training. This may have implications for data security as well as copyright.