Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners
This work addresses the lack of pixel-level spatio-temporal representations for dynamic scenes, enabling dense prediction tasks without requiring curated datasets.
LILA learns pixel-accurate feature descriptors from videos using linear in-context learning with noisy depth and motion cues, achieving temporally consistent embeddings that improve performance on video object segmentation, surface normal estimation, and semantic segmentation.
One of the most exciting applications of vision models involve pixel-level reasoning. Despite the abundance of vision foundation models, we still lack representations that effectively embed spatio-temporal properties of visual scenes at the pixel level. Existing frameworks either train on image-based pretext tasks, which do not account for dynamic elements, or on video sequences for action-level reasoning, which does not scale to dense pixel-level prediction. We present a framework that learns pixel-accurate feature descriptors from videos, LILA. The core element of our training framework is linear in-context learning. LILA leverages spatio-temporal cue maps -- depth and motion -- estimated with off-the-shelf networks. Despite the noisy nature of those cues, LILA trains effectively on uncurated video datasets, embedding semantic and geometric properties in a temporally consistent manner. We demonstrate compelling empirical benefits of the learned representation across a diverse suite of vision tasks: video object segmentation, surface normal estimation and semantic segmentation.