When LLaVA Meets Objects: Token Composition for Vision-Language-Models
This addresses efficiency issues for users of VLMs, though it is incremental as it builds on existing token-efficient methods.
The authors tackled the problem of high computational cost in autoregressive Vision Language Models (VLMs) due to many visual tokens, proposing Mask-LLaVA to combine multi-level visual features for a compact representation, achieving competitive results with only a fraction of tokens compared to baselines.
Current autoregressive Vision Language Models (VLMs) usually rely on a large number of visual tokens to represent images, resulting in a need for more compute especially at inference time. To address this problem, we propose Mask-LLaVA, a framework that leverages different levels of visual features to create a compact yet information-rich visual representation for autoregressive VLMs. Namely, we combine mask-based object representations together with global tokens and local patch tokens. While all tokens are used during training, it shows that the resulting model can flexibly drop especially the number of mask-based object-tokens at test time, allowing to adapt the number of tokens during inference without the need to retrain the model and without a significant drop in performance. We evaluate the proposed approach on a suite of standard benchmarks showing results competitive to current token efficient methods and comparable to the original LLaVA baseline using only a fraction of visual tokens. Our analysis demonstrates that combining multi-level features enables efficient learning with fewer tokens while allowing dynamic token selection at test time for good performance.