Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis
This work addresses the problem of making powerful pre-trained pathology foundation models practical for clinical applications on standard hardware, representing an incremental improvement in domain-specific adaptation methods.
The paper tackles the challenge of adapting pathology foundation models for clinical tasks with weak labels by proposing TAPFM, a method that uses vision transformer attention for multiple instance learning aggregation and optimizes feature representations and attention weights, resulting in consistent outperformance of benchmarks on mutation prediction tasks for bladder cancer and lung adenocarcinoma.
Pathology foundation models (PFMs) have emerged as powerful tools for analyzing whole slide images (WSIs). However, adapting these pretrained PFMs for specific clinical tasks presents considerable challenges, primarily due to the availability of only weak (WSI-level) labels for gigapixel images, necessitating multiple instance learning (MIL) paradigm for effective WSI analysis. This paper proposes a novel approach for single-GPU \textbf{T}ask \textbf{A}daptation of \textbf{PFM}s (TAPFM) that uses vision transformer (\vit) attention for MIL aggregation while optimizing both for feature representations and attention weights. The proposed approach maintains separate computational graphs for MIL aggregator and the PFM to create stable training dynamics that align with downstream task objectives during end-to-end adaptation. Evaluated on mutation prediction tasks for bladder cancer and lung adenocarcinoma across institutional and TCGA cohorts, TAPFM consistently outperforms conventional approaches, with H-Optimus-0 (TAPFM) outperforming the benchmarks. TAPFM effectively handles multi-label classification of actionable mutations as well. Thus, TAPFM makes adaptation of powerful pre-trained PFMs practical on standard hardware for various clinical applications.