Comprehensive Evaluation of Multimodal AI Models in Medical Imaging Diagnosis: From Data Augmentation to Preference-Based Comparison
This work addresses the need for robust evaluation of AI models in medical diagnostics, providing a framework that could improve diagnostic accuracy for healthcare professionals, though it is incremental as it builds on existing multimodal and evaluation methods.
This study tackled the problem of evaluating multimodal AI models for medical imaging diagnosis by developing a framework that expanded 500 clinical cases to 3,000 through augmentation and compared models against physician diagnoses. The results showed that Llama 3.2-90B outperformed human diagnoses in 85.27% of cases, while specialized vision models like BLIP2 and Llava achieved preferences in 41.36% and 46.77% of cases, respectively.
This study introduces an evaluation framework for multimodal models in medical imaging diagnostics. We developed a pipeline incorporating data preprocessing, model inference, and preference-based evaluation, expanding an initial set of 500 clinical cases to 3,000 through controlled augmentation. Our method combined medical images with clinical observations to generate assessments, using Claude 3.5 Sonnet for independent evaluation against physician-authored diagnoses. The results indicated varying performance across models, with Llama 3.2-90B outperforming human diagnoses in 85.27% of cases. In contrast, specialized vision models like BLIP2 and Llava showed preferences in 41.36% and 46.77% of cases, respectively. This framework highlights the potential of large multimodal models to outperform human diagnostics in certain tasks.