Ternarization of Vision Language Models for use on edge devices
This work addresses the challenge of running large vision language models on resource-constrained edge devices, though it is incremental as it builds on existing ternarization techniques.
The researchers tackled the problem of deploying vision language models on edge devices by compressing pre-trained models into ternary versions, achieving a ternary model with the fastest token generation speed while maintaining a good balance between memory usage and perplexity.
We propose a process to compress a pre-trained Vision Language Model into a ternary version of itself instead of training a ternary model from scratch. A new initialization scheme from pre-trained weights based on the k-means algorithm is proposed to reduce the ternarization time. We implement different custom operators for executing the ternary model on the TensorFlow Lite Engine. We compare the original model with its ternary and binary versions in terms of memory consumption, inference speed and perplexity. We find that the ternary model using our custom ternary matrix multiplication operator provides a good compromise in term of memory usage and perplexity, while having the fastest token generation speed.