Enhancing Target-unspecific Tasks through a Features Matrix
This addresses a key limitation in VLMs for researchers and practitioners working on generalizable tasks, though it appears incremental as it builds on existing frameworks.
The paper tackles the problem of target-unspecific tasks in Vision-Language Models, where existing prompting methods often overfit and lose general knowledge, by proposing a Features Matrix approach that extracts and leverages general knowledge to enhance performance, achieving state-of-the-art results in tasks like base-to-novel generalization.
Recent developments in prompt learning of large Vision-Language Models (VLMs) have significantly improved performance in target-specific tasks. However, these prompting methods often struggle to tackle the target-unspecific or generalizable tasks effectively. It may be attributed to the fact that overfitting training causes the model to forget its general knowledge. The general knowledge has a strong promotion on target-unspecific tasks. To alleviate this issue, we propose a novel Features Matrix (FM) approach designed to enhance these models on target-unspecific tasks. Our method extracts and leverages general knowledge, shaping a Features Matrix (FM). Specifically, the FM captures the semantics of diverse inputs from a deep and fine perspective, preserving essential general knowledge, which mitigates the risk of overfitting. Representative evaluations demonstrate that: 1) the FM is compatible with existing frameworks as a generic and flexible module, and 2) the FM significantly showcases its effectiveness in enhancing target-unspecific tasks (base-to-novel generalization, domain generalization, and cross-dataset generalization), achieving state-of-the-art performance.