In-Context Learning can Perform Continual Learning Like Humans
This work addresses the challenge of catastrophic forgetting and the stability-plasticity dilemma in continual learning for AI systems, offering a cognitively plausible, inference-only paradigm, though it is incremental as it builds on existing ICL methods.
The paper tackles the problem of whether in-context learning (ICL) in large language models can achieve long-term retention and knowledge accumulation across sequential tasks, extending it to in-context continual learning (ICCL) with task scheduling and prompt rearrangement. Experiments show that ICCL benefits from distributed practice, revealing a spacing 'sweet spot' for retention, and linear-attention models like MAMBA and RWKV exhibit human-like retention patterns, though their performance lags behind Transformer-based LLMs.
Large language models (LLMs) can adapt to new tasks via in-context learning (ICL) without parameter updates, making them powerful learning engines for fast adaptation. While extensive research has examined ICL as a few-shot learner, whether it can achieve long-term retention and cross-task knowledge accumulation when multitasks arrive sequentially remains underexplored. Motivated by human memory studies, we investigate the retention characteristics of ICL in multitask settings and extend it to in-context continual learning (ICCL), where continual learning ability emerges through task scheduling and prompt rearrangement. Experiments on Markov-Chain benchmarks demonstrate that, for specific large-language models, ICCL benefits from distributed practice (DP) in a manner analogous to humans, consistently revealing a spacing "sweet spot" for retention. Beyond retention performance, we propose a human-retention similarity metric to quantify how closely a continual-learning (CL) method aligns with human retention dynamics. Using this metric, we show that linear-attention models such as MAMBA and RWKV exhibit particularly human-like retention patterns, despite their retention performance lagging behind that of Transformer-based LLMs. Overall, our results establish ICCL as both cognitively plausible and practically effective, providing an inference-only CL paradigm that mitigates catastrophic forgetting and addresses the stability-plasticity dilemma in conventional CL methods.