Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning
This addresses the challenge of prompt selection in VICL for vision tasks, offering a competitive solution that is incremental by building on existing prompt-based methods.
The paper tackles the problem of selecting a single ideal prompt for Visual In-Context Learning (VICL) by proposing prompt condensation, where multiple prompts collaborate to integrate context, resulting in outperforming state-of-the-art methods across benchmark tasks with superior context compression and computational efficiency.
Visual In-Context Learning (VICL) enables adaptively solving vision tasks by leveraging pixel demonstrations, mimicking human-like task completion through analogy. Prompt selection is critical in VICL, but current methods assume the existence of a single "ideal" prompt in a pool of candidates, which in practice may not hold true. Multiple suitable prompts may exist, but individually they often fall short, leading to difficulties in selection and the exclusion of useful context. To address this, we propose a new perspective: prompt condensation. Rather than relying on a single prompt, candidate prompts collaborate to efficiently integrate informative contexts without sacrificing resolution. We devise Condenser, a lightweight external plugin that compresses relevant fine-grained context across multiple prompts. Optimized end-to-end with the backbone, Condenser ensures accurate integration of contextual cues. Experiments demonstrate Condenser outperforms state-of-the-arts across benchmark tasks, showing superior context compression, scalability with more prompts, and enhanced computational efficiency compared to ensemble methods, positioning it as a highly competitive solution for VICL. Code is open-sourced at https://github.com/gimpong/CVPR25-Condenser.