AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
This work addresses the computational barrier for long video understanding, potentially democratizing access by reducing costs, though it is incremental as it builds on existing MLLM frameworks.
The paper tackles the high computational and memory costs of long video understanding by proposing AuroraLong, which replaces the LLM component in MLLMs with a linear RNN language model to handle arbitrary-length inputs with constant hidden states, achieving performance comparable to Transformer-based models of similar size across multiple benchmarks.
The challenge of long video understanding lies in its high computational complexity and prohibitive memory cost, since the memory and computation required by transformer-based LLMs scale quadratically with input sequence length. We propose AuroraLong to address this challenge by replacing the LLM component in MLLMs with a linear RNN language model that handles input sequence of arbitrary length with constant-size hidden states. To further increase throughput and efficiency, we combine visual token merge with linear RNN models by reordering the visual tokens by their sizes in ascending order. Despite having only 2B parameters and being trained exclusively on public data, AuroraLong achieves performance comparable to Transformer-based models of similar size trained on private datasets across multiple video benchmarks. This demonstrates the potential of efficient, linear RNNs to democratize long video understanding by lowering its computational entry barrier. To our best knowledge, we are the first to use a linear RNN based LLM backbone in a LLaVA-like model for open-ended video understanding.