The Importance of Downstream Networks in Digital Pathology Foundation Models
This work addresses the problem of biased evaluation in digital pathology for researchers and practitioners by revealing that traditional static downstream setups can skew performance comparisons, leading to a more accurate assessment of foundation models.
The study found that the performance of feature extractor models in digital pathology is highly sensitive to the configuration of downstream aggregation models, and when accounting for this, many current models show notably similar performance, as demonstrated by evaluating seven models across three datasets with 162 configurations.
Digital pathology has significantly advanced disease detection and pathologist efficiency through the analysis of gigapixel whole-slide images (WSI). In this process, WSIs are first divided into patches, for which a feature extractor model is applied to obtain feature vectors, which are subsequently processed by an aggregation model to predict the respective WSI label. With the rapid evolution of representation learning, numerous new feature extractor models, often termed foundational models, have emerged. Traditional evaluation methods rely on a static downstream aggregation model setup, encompassing a fixed architecture and hyperparameters, a practice we identify as potentially biasing the results. Our study uncovers a sensitivity of feature extractor models towards aggregation model configurations, indicating that performance comparability can be skewed based on the chosen configurations. By accounting for this sensitivity, we find that the performance of many current feature extractor models is notably similar. We support this insight by evaluating seven feature extractor models across three different datasets with 162 different aggregation model configurations. This comprehensive approach provides a more nuanced understanding of the feature extractors' sensitivity to various aggregation model configurations, leading to a fairer and more accurate assessment of new foundation models in digital pathology.