PhenoBench: A Comprehensive Benchmark for Cell Phenotyping
This work addresses the problem of inconsistent benchmarking for cell phenotyping in pathology, providing a comprehensive tool for researchers, though it is incremental as it builds on existing foundational models.
The authors tackled the lack of a unified benchmark for cell phenotyping in digital pathology by introducing PhenoBench, which includes a new dataset (PhenoCell) and evaluation code, revealing that foundational models perform poorly on this challenging task with F1 scores as low as 0.20 compared to over 0.70 on existing benchmarks.
Digital pathology has seen the advent of a wealth of foundational models (FM), yet to date their performance on cell phenotyping has not been benchmarked in a unified manner. We therefore propose PhenoBench: A comprehensive benchmark for cell phenotyping on Hematoxylin and Eosin (H&E) stained histopathology images. We provide both PhenoCell, a new H&E dataset featuring 14 granular cell types identified by using multiplexed imaging, and ready-to-use fine-tuning and benchmarking code that allows the systematic evaluation of multiple prominent pathology FMs in terms of dense cell phenotype predictions in different generalization scenarios. We perform extensive benchmarking of existing FMs, providing insights into their generalization behavior under technical vs. medical domain shifts. Furthermore, while FMs achieve macro F1 scores > 0.70 on previously established benchmarks such as Lizard and PanNuke, on PhenoCell, we observe scores as low as 0.20. This indicates a much more challenging task not captured by previous benchmarks, establishing PhenoCell as a prime asset for future benchmarking of FMs and supervised models alike. Code and data are available on GitHub.