Embedding Inversion via Conditional Masked Diffusion Language Models
This addresses the problem of reconstructing text from embeddings for users of embedding models, offering a more efficient approach compared to sequential methods.
The paper tackles the problem of embedding inversion by framing it as conditional masked diffusion, recovering all tokens in parallel through iterative denoising instead of sequential autoregressive generation. The method achieves up to 81.3% token accuracy on 32-token sequences across three embedding models with only 8 forward passes through a 78M parameter model.
We frame embedding inversion as conditional masked diffusion, recovering all tokens in parallel through iterative denoising rather than sequential autoregressive generation. A masked diffusion language model is conditioned on the target embedding via adaptive layer normalization, requiring only 8 forward passes through a 78M parameter model with no access to the target encoder. On 32-token sequences across three embedding models, the method achieves up to 81.3% token accuracy. Source code and live demo are available at https://github.com/jina-ai/embedding-inversion-demo.