When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning
For researchers in parameter-efficient learning, this work clarifies when adaptive gating is ineffective, cautioning against unnecessary architectural complexity.
The paper identifies that adaptive gating mechanisms in few-shot prompt learning for vision-language models often collapse, producing constant outputs and failing to outperform fixed prompts. Through controlled experiments, they pinpoint gradient magnitude imbalance and gate degradation as key failure modes.
Adaptive prompting mechanisms have been proposed to enhance vision-language models by dynamically tailoring prompts to inputs. However, in frozen few-shot prompt learning with CLIP-style backbones, we systematically observe that adaptive gates and prompt-selection modules often collapse: they produce nearly constant outputs, contribute negligible gradient signals, and frequently fail to outperform fixed prompts. To further explore this issue, we present a systematic diagnostic study to uncover the underlying causes and conditions of adaptation failure. Through controlled experiments across datasets and multiple prompt learning architectures, we identify two recurring failure modes: gradient magnitude imbalance and gate degradation. Our findings invite a re-examination of indiscriminately adding architectural complexity in parameter-efficient learning and clarify when prompt-level adaptive gating is, and is not, effective in this regime.