Cytoarchitecture in Words: Weakly Supervised Vision-Language Modeling for Human Brain Microscopy
This work provides a practical recipe for integrating natural language interfaces into biomedical image analysis workflows, particularly for researchers in domains where fine-grained paired image-text annotations are scarce, by leveraging existing vision foundation models.
The authors developed a label-mediated method to generate meaningful captions for human brain microscopy images, linking images and text through labels without requiring curated paired image-text data. This method, applied across 57 brain areas, achieved 90.6% accuracy in matching cytoarchitectonic reference labels and 68.6% accuracy in recovering the brain area from descriptions in an 8-way test.
Foundation models increasingly offer potential to support interactive, agentic workflows that assist researchers during analysis and interpretation of image data. Such workflows often require coupling vision to language to provide a natural-language interface. However, paired image-text data needed to learn this coupling are scarce and difficult to obtain in many research and clinical settings. One such setting is microscopic analysis of cell-body-stained histological human brain sections, which enables the study of cytoarchitecture: cell density and morphology and their laminar and areal organization. Here, we propose a label-mediated method that generates meaningful captions from images by linking images and text only through a label, without requiring curated paired image-text data. Given the label, we automatically mine area descriptions from related literature and use them as synthetic captions reflecting canonical cytoarchitectonic attributes. An existing cytoarchitectonic vision foundation model (CytoNet) is then coupled to a large language model via an image-to-text training objective, enabling microscopy regions to be described in natural language. Across 57 brain areas, the resulting method produces plausible area-level descriptions and supports open-set use through explicit rejection of unseen areas. It matches the cytoarchitectonic reference label for in-scope patches with 90.6% accuracy and, with the area label masked, its descriptions remain discriminative enough to recover the area in an 8-way test with 68.6% accuracy. These results suggest that weak, label-mediated pairing can suffice to connect existing biomedical vision foundation models to language, providing a practical recipe for integrating natural-language in domains where fine-grained paired annotations are scarce.