EmbeddingGemma: Powerful and Lightweight Text Representations
This provides a high-performance, cost-effective solution for low-latency and high-throughput applications like on-device use, though it is incremental in improving existing embedding methods.
The paper tackles the problem of creating efficient text embedding models by introducing EmbeddingGemma, a lightweight model that achieves state-of-the-art results on the Massive Text Embedding Benchmark, outperforming prior models with fewer than 500M parameters and offering comparable performance to larger models.
We introduce EmbeddingGemma, a new lightweight, open text embedding model based on the Gemma 3 language model family. Our innovative training recipe strategically captures knowledge from larger models via encoder-decoder initialization and geometric embedding distillation. We improve model robustness and expressiveness with a spread-out regularizer, and ensure generalizability by merging checkpoints from varied, optimized mixtures. Evaluated on the Massive Text Embedding Benchmark (MTEB) across multilingual, English, and code domains, EmbeddingGemma (300M) achieves state-of-the-art results. Notably, it outperforms prior top models, both proprietary and open, with fewer than 500M parameters, and provides performance comparable to models double its size, offering an exceptional performance-to-cost ratio. Remarkably, this lead persists when quantizing model weights or truncating embedding outputs. This makes EmbeddingGemma particularly well-suited for low-latency and high-throughput use cases such as on-device applications. We provide ablation studies exploring our key design choices. We release EmbeddingGemma to the community to promote further research.