Rethinking Model Efficiency: Multi-Agent Inference with Large Models
This work addresses efficiency issues in vision-language models for applications requiring low latency, though it is incremental as it builds on existing model architectures with a novel inference strategy.
The paper tackles the bottleneck of end-to-end latency in vision-language models caused by sequential token generation, showing that large models with fewer output tokens can be more efficient than small models with longer sequences. Empirical results on real-world benchmarks confirm that large models achieve comparable performance with significantly fewer tokens, and a proposed multi-agent inference framework leverages this by transferring reasoning tokens from small models to approach large model performance.
Most vision-language models (VLMs) apply a large language model (LLM) as the decoder, where the response tokens are generated sequentially through autoregression. Therefore, the number of output tokens can be the bottleneck of the end-to-end latency. However, different models may require vastly different numbers of output tokens to achieve comparable performance. In this work, we conduct a comprehensive analysis of the latency across different components of VLMs on simulated data. The experiment shows that a large model with fewer output tokens can be more efficient than a small model with a long output sequence. The empirical study on diverse real-world benchmarks confirms the observation that a large model can achieve better or comparable performance as a small model with significantly fewer output tokens. To leverage the efficiency of large models, we propose a multi-agent inference framework that keeps large models with short responses but transfers the key reasoning tokens from the small model when necessary. The comparison on benchmark tasks demonstrates that by reusing the reasoning tokens from small models, it can help approach the performance of a large model with its own reasoning, which confirms the effectiveness of our proposal.