FeatUp: A Model-Agnostic Framework for Features at Any Resolution
This addresses a bottleneck in computer vision for researchers and practitioners by enabling high-resolution features without re-training, though it is incremental as it builds on existing feature upsampling techniques.
The paper tackles the problem of low spatial resolution in deep features for dense prediction tasks like segmentation and depth prediction, introducing FeatUp, a model-agnostic framework that restores spatial information and significantly outperforms existing upsampling and super-resolution methods in tasks such as class activation map generation and transfer learning.
Deep features are a cornerstone of computer vision research, capturing image semantics and enabling the community to solve downstream tasks even in the zero- or few-shot regime. However, these features often lack the spatial resolution to directly perform dense prediction tasks like segmentation and depth prediction because models aggressively pool information over large areas. In this work, we introduce FeatUp, a task- and model-agnostic framework to restore lost spatial information in deep features. We introduce two variants of FeatUp: one that guides features with high-resolution signal in a single forward pass, and one that fits an implicit model to a single image to reconstruct features at any resolution. Both approaches use a multi-view consistency loss with deep analogies to NeRFs. Our features retain their original semantics and can be swapped into existing applications to yield resolution and performance gains even without re-training. We show that FeatUp significantly outperforms other feature upsampling and image super-resolution approaches in class activation map generation, transfer learning for segmentation and depth prediction, and end-to-end training for semantic segmentation.