THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond
This addresses the need for efficient, generalizable video perception models for computer vision applications, though it builds incrementally on existing diffusion-based architectures.
The authors tackled the problem of unified human-centric video perception by developing THFM, a single model that handles both dense (depth, normals, segmentation, dense pose) and sparse (2D/3D keypoint) tasks, achieving performance on-par or surpassing specialized state-of-the-art models on various benchmarks despite being trained only on synthetic data.
We present THFM, a unified video foundation model for human-centric perception that jointly addresses dense tasks (depth, normals, segmentation, dense pose) and sparse tasks (2d/3d keypoint estimation) within a single architecture. THFM is derived from a pretrained text-to-video diffusion model, repurposed as a single-forward-pass perception model and augmented with learnable tokens for sparse predictions. Modulated by the text prompt, our single unified model is capable of performing various perception tasks. Crucially, our model is on-par or surpassing state-of-the-art specialized models on a variety of benchmarks despite being trained exclusively on synthetic data (i.e.~without training on real-world or benchmark specific data). We further highlight intriguing emergent properties of our model, which we attribute to the underlying diffusion-based video representation. For example, our model trained on videos with a single human in the scene generalizes to multiple humans and other object classes such as anthropomorphic characters and animals -- a capability that hasn't been demonstrated in the past.