In-Context Learning with Many Demonstration Examples
This addresses the memory and computational bottlenecks in large pre-trained language models for researchers and practitioners, enabling more efficient scaling of in-context learning and instruction tuning, though it is incremental as it builds on existing transformer architectures.
The paper tackles the problem of scaling in-context learning with many demonstration examples by proposing EVALM, a long-range language model based on an efficient transformer that handles contexts up to 256k tokens, 128 times more than existing models like GPT-3, resulting in a 4.1% higher average accuracy on diverse tasks.
Large pre-training language models (PLMs) have shown promising in-context learning abilities. However, due to the backbone transformer architecture, existing PLMs are bottlenecked by the memory and computational cost when scaling up to a large context size, leaving instruction tuning and in-context learning of many demonstration examples, as well as long-range language modeling under-explored. In this study, we propose a long-range language model EVALM based on an efficient transformer mechanism. EVALM is trained with 8k tokens per batch line and can test up to 256k-lengthed contexts with extrapolation, 128 times to the limit of existing PLMs (e.g. GPT3). Based on EVALM, we scale up the size of examples efficiently in both instruction tuning and in-context learning to explore the boundary of the benefits from more annotated data. Experimental results on a diverse set of tasks show that EVALM achieves 4.1% higher accuracy on average, and the average length of achieving the best accuracy score over tasks is around 12k. We find that in-context learning can achieve higher performance with more demonstrations under many-shot instruction tuning (8k), and further extending the length of instructions (16k) can further improve the upper bound of scaling in-context learning.