MTVNet: Mapping using Transformers for Volumes -- Network for Super-Resolution with Long-Range Interactions
This work addresses a bottleneck in 3D medical or scientific imaging by enabling more effective transformer use, though it is incremental as it adapts 2D transformer advances to 3D.
The authors tackled the problem of limited receptive field in volumetric super-resolution by proposing MTVNet, a multi-scale transformer-based model that uses hierarchical attention blocks and carrier tokens to enable long-range interactions, achieving state-of-the-art results on five 3D datasets with pronounced advantages for larger images.
Until now, it has been difficult for volumetric super-resolution to utilize the recent advances in transformer-based models seen in 2D super-resolution. The memory required for self-attention in 3D volumes limits the receptive field. Therefore, long-range interactions are not used in 3D to the extent done in 2D and the strength of transformers is not realized. We propose a multi-scale transformer-based model based on hierarchical attention blocks combined with carrier tokens at multiple scales to overcome this. Here information from larger regions at coarse resolution is sequentially carried on to finer-resolution regions to predict the super-resolved image. Using transformer layers at each resolution, our coarse-to-fine modeling limits the number of tokens at each scale and enables attention over larger regions than what has previously been possible. We experimentally compare our method, MTVNet, against state-of-the-art volumetric super-resolution models on five 3D datasets demonstrating the advantage of an increased receptive field. This advantage is especially pronounced for images that are larger than what is seen in popularly used 3D datasets. Our code is available at https://github.com/AugustHoeg/MTVNet