VisionLLM-based Multimodal Fusion Network for Glottic Carcinoma Early Detection
This work addresses early detection of glottic carcinoma for medical diagnosis, but it appears incremental as it applies existing multimodal fusion techniques to a new dataset.
The paper tackles the problem of early detection of glottic carcinoma, which is challenging due to morphological similarities with vocal cord dysplasia, and proposes a VisionLLM-based multimodal fusion network (MMGC-Net) that achieves state-of-the-art performance on a private dataset of 5,799 image-text pairs.
The early detection of glottic carcinoma is critical for improving patient outcomes, as it enables timely intervention, preserves vocal function, and significantly reduces the risk of tumor progression and metastasis. However, the similarity in morphology between glottic carcinoma and vocal cord dysplasia results in suboptimal detection accuracy. To address this issue, we propose a vision large language model-based (VisionLLM-based) multimodal fusion network for glottic carcinoma detection, known as MMGC-Net. By integrating image and text modalities, multimodal models can capture complementary information, leading to more accurate and robust predictions. In this paper, we collect a private real glottic carcinoma dataset named SYSU1H from the First Affiliated Hospital of Sun Yat-sen University, with 5,799 image-text pairs. We leverage an image encoder and additional Q-Former to extract vision embeddings and the Large Language Model Meta AI (Llama3) to obtain text embeddings. These modalities are then integrated through a laryngeal feature fusion block, enabling a comprehensive integration of image and text features, thereby improving the glottic carcinoma identification performance. Extensive experiments on the SYSU1H dataset demonstrate that MMGC-Net can achieve state-of-the-art performance, which is superior to previous multimodal models.