Efficient Whole Slide Pathology VQA via Token Compression
This addresses the problem of efficient generative analysis in pathology for medical professionals, though it is incremental as it builds on existing MLLM architectures with a novel compression technique.
The paper tackled the challenge of visual question answering (VQA) for whole-slide pathology images, which are large and computationally demanding, by proposing TCP-LLaVA, a model that uses token compression to reduce input length and computational cost, achieving higher VQA accuracy on ten TCGA tumor subtypes while cutting training resource consumption significantly.
Whole-slide images (WSIs) in pathology can reach up to 10,000 x 10,000 pixels, posing significant challenges for multimodal large language model (MLLM) due to long context length and high computational demands. Previous methods typically focus on patch-level analysis or slide-level classification using CLIP-based models with multi-instance learning, but they lack the generative capabilities needed for visual question answering (VQA). More recent MLLM-based approaches address VQA by feeding thousands of patch tokens directly into the language model, which leads to excessive resource consumption. To address these limitations, we propose Token Compression Pathology LLaVA (TCP-LLaVA), the first MLLM architecture to perform WSI VQA via token compression. TCP-LLaVA introduces a set of trainable compression tokens that aggregate visual and textual information through a modality compression module, inspired by the [CLS] token mechanism in BERT. Only the compressed tokens are forwarded to the LLM for answer generation, significantly reducing input length and computational cost. Experiments on ten TCGA tumor subtypes show that TCP-LLaVA outperforms existing MLLM baselines in VQA accuracy while reducing training resource consumption by a substantial margin.