Tree of Attributes Prompt Learning for Vision-Language Models
This work addresses a bottleneck in adapting vision-language models for downstream tasks, offering a structured approach to improve generalization, though it is incremental in enhancing existing prompt learning methods.
The paper tackles the problem of underutilizing rich contextual information in category names for vision-language model prompt learning by proposing Tree of Attributes Prompt learning (TAP), which generates structured attribute trees and learns them with prompts, resulting in outperforming state-of-the-art methods on zero-shot, cross-dataset, and few-shot tasks across 11 datasets.
Prompt learning has proven effective in adapting vision language models for downstream tasks. However, existing methods usually append learnable prompt tokens solely with the category names to obtain textual features, which fails to fully leverage the rich context indicated in the category name. To address this issue, we propose the Tree of Attributes Prompt learning (TAP), which first instructs LLMs to generate a tree of attributes with a "concept - attribute - description" structure for each category, and then learn the hierarchy with vision and text prompt tokens. Unlike existing methods that merely augment category names with a set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Furthermore, our approach introduces text and vision prompts designed to explicitly learn the corresponding visual attributes, effectively serving as domain experts. Additionally, the general and diverse descriptions generated based on the class names may be wrong or absent in the specific given images. To address this misalignment, we further introduce a vision-conditional pooling module to extract instance-specific text features. Extensive experimental results demonstrate that our approach outperforms state-of-the-art methods on the zero-shot base-to-novel generalization, cross-dataset transfer, as well as few-shot classification across 11 diverse datasets. Code is available at https://github.com/HHenryD/TAP.