When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
This addresses the problem of analyzing high-resolution medical images for cancer diagnosis, offering a novel method that could improve accuracy in pathology, though it is incremental as it builds on existing Transformer and LongNet techniques.
The paper tackles the challenge of processing gigapixel images for cancer diagnosis and prognosis in computational pathology by introducing LongViT, a vision Transformer that encodes these images end-to-end, outperforming previous state-of-the-art methods on tasks like cancer subtyping and survival prediction.
This technical report presents LongViT, a vision Transformer that can process gigapixel images in an end-to-end manner. Specifically, we split the gigapixel image into a sequence of millions of patches and project them linearly into embeddings. LongNet is then employed to model the extremely long sequence, generating representations that capture both short-range and long-range dependencies. The linear computation complexity of LongNet, along with its distributed algorithm, enables us to overcome the constraints of both computation and memory. We apply LongViT in the field of computational pathology, aiming for cancer diagnosis and prognosis within gigapixel whole-slide images. Experimental results demonstrate that LongViT effectively encodes gigapixel images and outperforms previous state-of-the-art methods on cancer subtyping and survival prediction. Code and models will be available at https://aka.ms/LongViT.