Efficient Ternary Weight Embedding Model: Bridging Scalability and Performance
This work addresses scalability issues for embedding models in resource-constrained environments like real-time recommendation systems, representing an incremental improvement through ternarization.
The authors tackled the high memory and computational demands of embedding models by proposing a finetuning framework for ternary-weight embedding models, which reduced memory usage and latency while maintaining performance, as demonstrated on public text and vision datasets.
Embedding models have become essential tools in both natural language processing and computer vision, enabling efficient semantic search, recommendation, clustering, and more. However, the high memory and computational demands of full-precision embeddings pose challenges for deployment in resource-constrained environments, such as real-time recommendation systems. In this work, we propose a novel finetuning framework to ternary-weight embedding models, which reduces memory and computational overhead while maintaining high performance. To apply ternarization to pre-trained embedding models, we introduce self-taught knowledge distillation to finalize the ternary-weights of the linear layers. With extensive experiments on public text and vision datasets, we demonstrated that without sacrificing effectiveness, the ternarized model consumes low memory usage and has low latency in the inference stage with great efficiency. In practical implementations, embedding models are typically integrated with Approximate Nearest Neighbor (ANN) search. Our experiments combining ternary embedding with ANN search yielded impressive improvement in both accuracy and computational efficiency. The repository is available at here.