FlowFeat: Pixel-Dense Embedding of Motion Profiles
This work addresses the need for dense and versatile image representations in computer vision, particularly for tasks like video object segmentation, monocular depth estimation, and semantic segmentation, with incremental improvements over existing methods.
The paper tackled the problem of low-resolution feature grids in state-of-the-art networks like transformers for dense prediction tasks by introducing FlowFeat, a high-resolution and multi-task feature representation that embeds motion profiles, resulting in significant enhancement of five state-of-the-art encoders across three dense tasks.
Dense and versatile image representations underpin the success of virtually all computer vision applications. However, state-of-the-art networks, such as transformers, produce low-resolution feature grids, which are suboptimal for dense prediction tasks. To address this limitation, we present FlowFeat, a high-resolution and multi-task feature representation. The key ingredient behind FlowFeat is a novel distillation technique that embeds a distribution of plausible apparent motions, or motion profiles. By leveraging optical flow networks and diverse video data, we develop an effective self-supervised training framework that statistically approximates the apparent motion. With its remarkable level of spatial detail, FlowFeat encodes a compelling degree of geometric and semantic cues while exhibiting high temporal consistency. Empirically, FlowFeat significantly enhances the representational power of five state-of-the-art encoders and alternative upsampling strategies across three dense tasks: video object segmentation, monocular depth estimation and semantic segmentation. Training FlowFeat is computationally inexpensive and robust to inaccurate flow estimation, remaining highly effective even when using unsupervised flow networks. Our work takes a step forward towards reliable and versatile dense image representations.