Sumotosima: A Framework and Dataset for Classifying and Summarizing Otoscopic Images
This work addresses the lack of automated tools for otoscopic image analysis, benefiting medical professionals and patients by providing clear summaries, but it is incremental as it builds on existing deep learning and transformer methods.
The authors tackled the problem of classifying and summarizing otoscopic images by proposing Sumotosima, a resource-efficient deep learning and transformer-based framework, which achieved 98.03% accuracy in classification and outperformed GPT-4o and LLaVA by 88.53% and 107.57% in ROUGE scores for summarization.
Otoscopy is a diagnostic procedure to examine the ear canal and eardrum using an otoscope. It identifies conditions like infections, foreign bodies, ear drum perforations and ear abnormalities. We propose a novel resource efficient deep learning and transformer based framework, Sumotosima (Summarizer for otoscopic images), an end-to-end pipeline for classification followed by summarization. Our framework works on combination of triplet and cross-entropy losses. Additionally, we use Knowledge Enhanced Multimodal BART whose input is fused textual and image embedding. The objective is to provide summaries that are well-suited for patients, ensuring clarity and efficiency in understanding otoscopic images. Given the lack of existing datasets, we have curated our own OCASD (Otoscopic Classification And Summary Dataset), which includes 500 images with 5 unique categories annotated with their class and summaries by Otolaryngologists. Sumotosima achieved a result of 98.03%, which is 7.00%, 3.10%, 3.01% higher than K-Nearest Neighbors, Random Forest and Support Vector Machines, respectively, in classification tasks. For summarization, Sumotosima outperformed GPT-4o and LLaVA by 88.53% and 107.57% in ROUGE scores, respectively. We have made our code and dataset publicly available at https://github.com/anas2908/Sumotosima