Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis
This work addresses the problem of automating microscopy image analysis for medical experts, biologists, and materials scientists, but it is incremental as it primarily benchmarks existing models on a new domain without introducing novel methods.
The study tested large vision language models (VLMs) like ChatGPT, Gemini, and LLaVA, along with the Segment Anything Model (SAM), on microscopy image tasks including classification, segmentation, counting, and visual question answering, finding that while they showed some capability, their performance was significantly below that of domain experts, especially when handling impurities, defects, and overlaps.
Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.