Towards Selection of Large Multimodal Models as Engines for Burned-in Protected Health Information Detection in Medical Images
This addresses privacy and compliance in medical imaging by evaluating LMMs for PHI detection, but it is incremental as it benchmarks existing models without introducing new methods.
The study benchmarked three Large Multimodal Models (GPT-4o, Gemini 2.5 Flash, Qwen 2.5 7B) for detecting Protected Health Information in medical images, finding superior OCR performance (WER: 0.03-0.05, CER: 0.02-0.03) over traditional methods but inconsistent gains in overall PHI detection accuracy, with best results on complex imprint patterns.
The detection of Protected Health Information (PHI) in medical imaging is critical for safeguarding patient privacy and ensuring compliance with regulatory frameworks. Traditional detection methodologies predominantly utilize Optical Character Recognition (OCR) models in conjunction with named entity recognition. However, recent advancements in Large Multimodal Model (LMM) present new opportunities for enhanced text extraction and semantic analysis. In this study, we systematically benchmark three prominent closed and open-sourced LMMs, namely GPT-4o, Gemini 2.5 Flash, and Qwen 2.5 7B, utilizing two distinct pipeline configurations: one dedicated to text analysis alone and another integrating both OCR and semantic analysis. Our results indicate that LMM exhibits superior OCR efficacy (WER: 0.03-0.05, CER: 0.02-0.03) compared to conventional models like EasyOCR. However, this improvement in OCR performance does not consistently correlate with enhanced overall PHI detection accuracy. The strongest performance gains are observed on test cases with complex imprint patterns. In scenarios where text regions are well readable with sufficient contrast, and strong LMMs are employed for text analysis after OCR, different pipeline configurations yield similar results. Furthermore, we provide empirically grounded recommendations for LMM selection tailored to specific operational constraints and propose a deployment strategy that leverages scalable and modular infrastructure.