Hierarchy-Aware Fine-Tuning of Vision-Language Models
This addresses the challenge of expensive and inconsistent predictions when adapting VLMs to structured taxonomies, though it appears incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of adapting vision-language models to hierarchical classification by proposing an efficient fine-tuning framework that enforces structural consistency with minimal parameter updates, achieving consistent improvements in Full-Path Accuracy and Tree-based Inconsistency Error across multiple benchmarks.
Vision-Language Models (VLMs) learn powerful multimodal representations through large-scale image-text pretraining, but adapting them to hierarchical classification is underexplored. Standard approaches treat labels as flat categories and require full fine-tuning, which is expensive and produces inconsistent predictions across taxonomy levels. We propose an efficient hierarchy-aware fine-tuning framework that updates a few parameters while enforcing structural consistency. We combine two objectives: Tree-Path KL Divergence (TP-KL) aligns predictions along the ground-truth label path for vertical coherence, while Hierarchy-Sibling Smoothed Cross-Entropy (HiSCE) encourages consistent predictions among sibling classes. Both losses work in the VLM's shared embedding space and integrate with lightweight LoRA adaptation. Experiments across multiple benchmarks show consistent improvements in Full-Path Accuracy and Tree-based Inconsistency Error with minimal parameter overhead. Our approach provides an efficient strategy for adapting VLMs to structured taxonomies.