Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare
This addresses the problem of limited healthcare AI accessibility for Chinese-speaking users by providing a multi-modal model, though it is incremental as it adapts existing methods to a new language and domain.
The study tackled the lack of non-English and multi-modal models in healthcare by introducing Qilin-Med-VL, the first Chinese large vision-language model, which integrates textual and visual data analysis and is trained on a new dataset of over 1M image-text pairs to enhance medical caption generation and query answering.
Large Language Models (LLMs) have introduced a new era of proficiency in comprehending complex healthcare and biomedical topics. However, there is a noticeable lack of models in languages other than English and models that can interpret multi-modal input, which is crucial for global healthcare accessibility. In response, this study introduces Qilin-Med-VL, the first Chinese large vision-language model designed to integrate the analysis of textual and visual data. Qilin-Med-VL combines a pre-trained Vision Transformer (ViT) with a foundational LLM. It undergoes a thorough two-stage curriculum training process that includes feature alignment and instruction tuning. This method enhances the model's ability to generate medical captions and answer complex medical queries. We also release ChiMed-VL, a dataset consisting of more than 1M image-text pairs. This dataset has been carefully curated to enable detailed and comprehensive interpretation of medical data using various types of images.