Boosting Vision-Language Models for Histopathology Classification: Predict all at once
This work addresses the challenge of improving classification accuracy in histopathology for medical applications, representing an incremental advancement by extending existing vision-language models with a transductive method.
The paper tackles the problem of inductive classification in histopathology by introducing a transductive approach that leverages text-based predictions and patch affinities, achieving significant accuracy improvements over existing zero-shot methods across four datasets and five vision-language models.
The development of vision-language models (VLMs) for histo-pathology has shown promising new usages and zero-shot performances. However, current approaches, which decompose large slides into smaller patches, focus solely on inductive classification, i.e., prediction for each patch is made independently of the other patches in the target test data. We extend the capability of these large models by introducing a transductive approach. By using text-based predictions and affinity relationships among patches, our approach leverages the strong zero-shot capabilities of these new VLMs without any additional labels. Our experiments cover four histopathology datasets and five different VLMs. Operating solely in the embedding space (i.e., in a black-box setting), our approach is highly efficient, processing $10^5$ patches in just a few seconds, and shows significant accuracy improvements over inductive zero-shot classification. Code available at https://github.com/FereshteShakeri/Histo-TransCLIP.