EPLKG: Efficient Prompt Learning with Knowledge Graph
This work addresses efficiency and interpretability issues in prompt learning for model adaptation, offering incremental improvements for researchers and practitioners in low-resource or few-shot scenarios.
The paper tackled the problem of computationally costly adaptation of large-scale pre-trained models like CLIP to new datasets, especially in low-resource settings, by introducing EPLKG, which reduced per-image training time by up to 45% and peak GPU memory by 30-40% while maintaining competitive accuracy within 2 percentage points across 11 benchmarks.
Large-scale pre-trained models such as CLIP excel in transferability and robust generalization across diverse datasets. However, adapting these models to new datasets or domains is computationally costly, especially in low-resource or few-shot settings, and existing prompt-learning methods often lack interpretability. We introduce Efficient Prompt Learning with Knowledge Graph (EPLKG), which uses a knowledge graph to curate diverse, interpretable prompts and, where KG coverage is limited, augments this bank with LLM-generated human-readable visual descriptions. EPLKG operates entirely on cached CLIP image and text embeddings and employs a lightweight Gumbel-Softmax module to select a single prompt per image-class pair, enabling low-memory, fast training. Across 11 benchmarks, EPLKG reduces per-image training time by up to 45 percent and peak GPU memory by around 30 to 40 percent compared to strong prompt-learning baselines, while keeping the average base-new harmonic-mean accuracy within 2 percentage points, thereby improving the efficiency of model adaptation without sacrificing competitive performance or interpretability.