CephGPT-4: An Interactive Multimodal Cephalometric Measurement and Diagnostic System with Visual Large Language Model
This work addresses the need for automated diagnostic tools in orthodontics, representing an incremental advancement by applying existing LMMs to a new medical domain with specific data.
The authors tackled the problem of automating cephalometric analysis and diagnosis in orthodontics by developing CephGPT-4, a multimodal model fine-tuned on a dataset of cephalometric images and doctor-patient dialogues, which demonstrated excellent performance and potential to revolutionize orthodontic applications.
Large-scale multimodal language models (LMMs) have achieved remarkable success in general domains. However, the exploration of diagnostic language models based on multimodal cephalometric medical data remains limited. In this paper, we propose a novel multimodal cephalometric analysis and diagnostic dialogue model. Firstly, a multimodal orthodontic medical dataset is constructed, comprising cephalometric images and doctor-patient dialogue data, with automatic analysis of cephalometric landmarks using U-net and generation of diagnostic reports. Then, the cephalometric dataset and generated diagnostic reports are separately fine-tuned on Minigpt-4 and VisualGLM. Results demonstrate that the CephGPT-4 model exhibits excellent performance and has the potential to revolutionize orthodontic measurement and diagnostic applications. These innovations hold revolutionary application potential in the field of orthodontics.