Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention
This work addresses a practical deployment bottleneck for many-shot in-context learning, enabling scalable use with reduced computational overhead.
The paper tackles the high inference-time computational cost of many-shot in-context learning by introducing Dynamic Block-Sparse Attention, a training-free framework that achieves comparable latency to finetuning while maintaining over 95% of the best method's accuracy on average.
Many-shot in-context learning has recently shown promise as an alternative to finetuning, with the major advantage that the same model can be served for multiple tasks. However, this shifts the computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. This cost is further increased if a custom demonstration set is retrieved for each inference example. We present Dynamic Block-Sparse Attention, a training-free framework for retrieval-based many-shot in-context learning. By combining carefully designed block-sparse attention and retrieval of cached groups of demonstrations, we achieve comparable per-example latency to finetuning while maintaining on average >95% of the best method's accuracy across strong ICL and finetuning baselines. We hope that this will further enable the deployment of many-shot ICL at scale.