UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes
This addresses the need for more general and efficient vision models, though it appears incremental as it builds on existing paradigms with hybrid components.
The paper tackles the problem of developing a unified model for diverse computer vision tasks without task-specific modifications, achieving competitive and near state-of-the-art results on panoptic segmentation, depth prediction, and image colorization.
We introduce UViM, a unified approach capable of modeling a wide range of computer vision tasks. In contrast to previous models, UViM has the same functional form for all tasks; it requires no task-specific modifications which require extensive human expertise. The approach involves two components: (I) a base model (feed-forward) which is trained to directly predict raw vision outputs, guided by a learned discrete code and (II) a language model (autoregressive) that is trained to generate the guiding code. These components complement each other: the language model is well-suited to modeling structured interdependent data, while the base model is efficient at dealing with high-dimensional outputs. We demonstrate the effectiveness of UViM on three diverse and challenging vision tasks: panoptic segmentation, depth prediction and image colorization, where we achieve competitive and near state-of-the-art results. Our experimental results suggest that UViM is a promising candidate for a unified modeling approach in computer vision.