HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks
This work addresses a bottleneck in adapting multimodal models to new tasks, offering a more efficient and stable approach, though it appears incremental as it builds on existing approximation methods.
The paper tackled the sensitivity and computational expense of In-Context Learning (ICL) in Large Multimodal Models by introducing HIFICL, which uses virtual key-value pairs and low-rank factorization to more faithfully model the ICL mechanism, resulting in consistent outperformance over existing methods on several multimodal benchmarks.
In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.