Compressed Concatenation of Small Embedding Models
This enables more efficient deployment of embedding models in resource-constrained environments like browsers or edge devices, though it is incremental as it builds on existing concatenation and compression techniques.
The paper tackles the performance gap between small and large embedding models by showing that concatenating multiple small models' embeddings outperforms a single larger baseline, and introduces a lightweight decoder to compress the high-dimensional result, recovering 89% of original performance with 48x compression on MTEB retrieval tasks.
Embedding models are central to dense retrieval, semantic search, and recommendation systems, but their size often makes them impractical to deploy in resource-constrained environments such as browsers or edge devices. While smaller embedding models offer practical advantages, they typically underperform compared to their larger counterparts. To bridge this gap, we demonstrate that concatenating the raw embedding vectors of multiple small models can outperform a single larger baseline on standard retrieval benchmarks. To overcome the resulting high dimensionality of naive concatenation, we introduce a lightweight unified decoder trained with a Matryoshka Representation Learning (MRL) loss. This decoder maps the high-dimensional joint representation to a low-dimensional space, preserving most of the original performance without fine-tuning the base models. We also show that while concatenating more base models yields diminishing gains, the robustness of the decoder's representation under compression and quantization improves. Our experiments show that, on a subset of MTEB retrieval tasks, our concat-encode-quantize pipeline recovers 89\% of the original performance with a 48x compression factor when the pipeline is applied to a concatenation of four small embedding models.