Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators
This work addresses the challenge of visual interaction for AI systems, offering a novel approach to task learning from demonstrations, though it is incremental in applying autoregressive Transformers to video data.
The paper tackles the problem of enabling models to perform unseen tasks by watching demonstration videos, achieving zero-shot imitation of video semantics to new scenarios without fine-tuning, with results showing high-quality video generation that follows scaling laws.
People interact with the real-world largely dependent on visual signal, which are ubiquitous and illustrate detailed demonstrations. In this paper, we explore utilizing visual signals as a new interface for models to interact with the environment. Specifically, we choose videos as a representative visual signal. And by training autoregressive Transformers on video datasets in a self-supervised objective, we find that the model emerges a zero-shot capability to infer the semantics from a demonstration video, and imitate the semantics to an unseen scenario. This allows the models to perform unseen tasks by watching the demonstration video in an in-context manner, without further fine-tuning. To validate the imitation capacity, we design various evaluation metrics including both objective and subjective measures. The results show that our models can generate high-quality video clips that accurately align with the semantic guidance provided by the demonstration videos, and we also show that the imitation capacity follows the scaling law. Code and models have been open-sourced.