Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models
For practitioners needing efficient long-sequence embeddings, this provides a practical alternative to transformers with substantially lower memory footprint.
This work proposes recurrent architectures (Mamba2, RWKV, xLSTM) for text embeddings, achieving competitive performance with transformer-based models while reducing memory usage to constant in input length via a vertically chunked inference strategy.
Transformer-based embedding models suffer from quadratic computational and linear memory complexity, limiting their utility for long sequences. We propose recurrent architectures as an efficient alternative, introducing a vertically chunked inference strategy that enables fast embedding generation with memory usage that becomes constant in the input length once it exceeds the vertical chunk size. By fine-tuning Mamba2 models, we demonstrate their viability as general-purpose text embedders, achieving competitive performance across a range of benchmarks while maintaining a substantially smaller memory footprint compared to transformer-based counterparts. We empirically validate the applicability of our inference strategy to Mamba2, RWKV, and xLSTM models, confirming consistent runtime-memory trade-offs across architectures and establishing recurrent models as a compelling alternative to transformers for efficient embedding generation.