Pretraining Frame Preservation in Autoregressive Video Memory Compression
This addresses the challenge of enabling long history memory in autoregressive video models with low context cost, though it appears incremental as it builds on existing neural network structures and fine-tuning approaches.
The paper tackles the problem of compressing long videos into short contexts while preserving high-frequency details of individual frames, achieving compression of a 20-second video into a context of about 5k length with perceptually preserved frame retrieval.
We present PFP, a neural network structure to compress long videos into short contexts, with an explicit pretraining objective to preserve the high-frequency details of single frames at arbitrary temporal positions. The baseline model can compress a 20-second video into a context at about 5k length, where random frames can be retrieved with perceptually preserved appearances. Such pretrained models can be directly fine-tuned as memory encoders for autoregressive video models, enabling long history memory with low context cost and relatively low fidelity loss. We evaluate the framework with ablative settings and discuss the trade-offs of possible neural architecture designs.