Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning
This work addresses efficiency and performance issues in large multimodal models for researchers and practitioners, though it is incremental as it builds on existing task-vector-based approaches.
The paper tackled the challenge of scaling many-shot multimodal in-context learning by proposing a sensitivity-aware task vector insertion framework that identifies where and what to insert based on activation patterns, resulting in consistent improvements over previous methods across models and tasks.
Large Multimodal Models (LMMs) have shown promising in-context learning (ICL) capabilities, but scaling to many-shot settings remains difficult due to limited context length and high inference cost. To address these challenges, task-vector-based methods have been explored by inserting compact representations of many-shot in-context demonstrations into model activations. However, existing task-vector-based methods either overlook the importance of where to insert task vectors or struggle to determine suitable values for each location. To this end, we propose a novel Sensitivity-aware Task Vector insertion framework (STV) to figure out where and what to insert. Our key insight is that activation deltas across query-context pairs exhibit consistent structural patterns, providing a reliable cue for insertion. Based on the identified sensitive-aware locations, we construct a pre-clustered activation bank for each location by clustering the activation values, and then apply reinforcement learning to choose the most suitable one to insert. We evaluate STV across a range of multimodal models (e.g., Qwen-VL, Idefics-2) and tasks (e.g., VizWiz, OK-VQA), demonstrating its effectiveness and showing consistent improvements over previous task-vector-based methods with strong generalization.