Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning
This addresses the challenge of adapting vision-language models for specialized domains like surgery without losing general capabilities, though it appears incremental as it builds on existing adaptation methods.
The paper tackles the problem of fine-tuning vision-language models on domain-specific datasets, which can degrade their generalization, by proposing Chain-of-Adaptation (CoA), a framework that uses reinforcement learning to integrate domain knowledge while preserving multimodal priors, resulting in higher accuracy and stronger generalization on surgical benchmarks.
Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent reasoning and perceptual capabilities. CoA introduces a structured reasoning format that enhances domain alignment without sacrificing general multimodal competence by reinforcement learning. Experiments on standard surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-language abilities, providing a reliable pathway for domain specialization in VLMs.