MIRAGE: Multimodal Identification and Recognition of Annotations in Indian General Prescriptions
This addresses the challenge of digitizing handwritten prescriptions for hospitals in India, though it is incremental as it builds on existing MLLM approaches.
The paper tackled the problem of extracting medication names and dosages from handwritten medical records in India, achieving 82% accuracy by fine-tuning multimodal large language models on a dataset of simulated records.
Hospitals in India still rely on handwritten medical records despite the availability of Electronic Medical Records (EMR), complicating statistical analysis and record retrieval. Handwritten records pose a unique challenge, requiring specialized data for training models to recognize medications and their recommendation patterns. While traditional handwriting recognition approaches employ 2-D LSTMs, recent studies have explored using Multimodal Large Language Models (MLLMs) for OCR tasks. Building on this approach, we focus on extracting medication names and dosages from simulated medical records. Our methodology MIRAGE (Multimodal Identification and Recognition of Annotations in indian GEneral prescriptions) involves fine-tuning the QWEN VL, LLaVA 1.6 and Idefics2 models on 743,118 high resolution simulated medical record images-fully annotated from 1,133 doctors across India. Our approach achieves 82% accuracy in extracting medication names and dosages.