Espresso: High Compression For Rich Extraction From Videos for Your Vision-Language Model
This work addresses the problem of handling long-form video inputs for vision-language models, offering a scalable alternative to pooling-based methods, though it appears incremental in its approach.
The paper tackles the challenge of extending vision-language models to long videos by introducing Espresso, an architecture that compresses spatial and temporal features into fixed-length sequences, enabling efficient video encoding while maintaining strong reasoning capabilities.
Recent advances in vision-language models (VLMs) have shown great promise in connecting images and text, but extending these models to long videos remains challenging due to the rapid growth in token counts. Models that compress videos by local aggregation in time or space have become popular for handling long-form inputs; however, these pooling-based projectors sacrifice the benefits of fixed-length representations that are crucial for streaming and efficient video understanding. We introduce $\texttt{Espresso}$, a new architecture that separately compresses spatial and temporal features into fixed-length sequences. $\texttt{Espresso}$ enables efficient video encoding while maintaining strong long-form reasoning capabilities. Experiments show that fixed-length compression combined with segment-wise processing offers a scalable and competitive alternative to pooling-based approaches. Our results demonstrate that fixed-length projectors, when properly designed and trained, remain a viable foundation for video-language modeling.