FrEVL: Leveraging Frozen Pretrained Embeddings for Efficient Vision-Language Understanding
This addresses efficiency challenges for practitioners deploying vision-language models, offering a viable alternative when computational constraints outweigh marginal performance gains, though it is incremental as it builds on existing embedding methods.
The paper tackles the high computational cost of vision-language models by proposing FrEVL, a framework that uses frozen pretrained embeddings, achieving 85-95% of state-of-the-art performance on benchmarks with only 68.4M trainable parameters and providing a 2.3x speedup with 52% lower energy consumption.
The deployment of vision-language models remains constrained by substantial computational requirements. We present \textbf{FrEVL}, a framework exploring whether frozen pretrained embeddings can support effective vision-language understanding. Our analysis reveals that frozen embeddings contain rich information for discriminative tasks, achieving 85\% to 95\% of state-of-the-art performance on standard benchmarks with only 68.4M trainable parameters. This performance dichotomy reveals a critical insight: frozen embedding effectiveness depends on alignment between pretraining objectives and downstream task requirements. When accounting for end-to-end computation including embedding extraction, FrEVL provides $2.3\times$ speedup with 52\% lower energy consumption, making it suitable for scenarios with pre-computable inputs or when deployment constraints outweigh marginal performance gains. Our evaluation provides practitioners with guidance on when frozen embedding approaches represent viable alternatives to full model deployment. We will release our complete implementation and evaluation framework to facilitate further research into efficient multi-modal understanding.