Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuning
This addresses the limitation of current MLLMs in specialized domains, though it appears incremental as it builds on existing fine-tuning methods.
The paper tackled the problem of multimodal large language models (MLLMs) performing poorly in specialized domains like remote sensing and medical imaging, finding that textual domain knowledge injection yields little improvement. The proposed reinforcement fine-tuning framework that integrates domain knowledge as constraints and rewards achieved state-of-the-art results on multimodal domain tasks.
Multimodal large language models (MLLMs) have shown remarkable capabilities in multimodal perception and understanding tasks. However, their effectiveness in specialized domains, such as remote sensing and medical imaging, remains limited. A natural approach to domain adaptation is to inject domain knowledge through textual instructions, prompts, or auxiliary captions. Surprisingly, we find that such input-level domain knowledge injection yields little to no improvement on scientific multimodal tasks, even when the domain knowledge is explicitly provided. This observation suggests that current MLLMs fail to internalize domain-specific priors through language alone, and that domain knowledge must be integrated at the optimization level. Motivated by this insight, we propose a reinforcement fine-tuning framework that incorporates domain knowledge directly into the learning objective. Instead of treating domain knowledge as descriptive information, we encode it as domain-informed constraints and reward signals, shaping the model's behavior in the output space. Extensive experiments across multiple datasets in remote sensing and medical domains consistently demonstrate good performance gains, achieving state-of-the-art results on multimodal domain tasks. Our results highlight the necessity of optimization-level domain knowledge integration and reveal a fundamental limitation of textual domain conditioning in current MLLMs.