World Model on Million-Length Video And Language With Blockwise RingAttention
This work addresses the problem of long-context understanding for AI researchers and developers, enabling generally intelligent models to operate over extended temporal horizons, though it is incremental in building upon existing sequence modeling techniques.
The paper tackles the challenge of scaling sequence models to process million-token contexts by developing a comprehensive pipeline for training 1M context language and video-language models, resulting in new benchmarks in language retrieval and enhanced long video understanding capabilities, with open-source 7B parameter models.
Enabling long-context understanding remains a key challenge in scaling existing sequence models -- a crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, we aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities in long video understanding. We detail our long context data curation process, progressive context extension from 4K to 1M tokens, and present an efficient open-source implementation for scalable training on long sequences. Additionally, we open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens.