As We May Search
For users concerned about privacy and control over sensitive data, this work provides a viable alternative to cloud-based retrieval-augmented generation systems, though the results are incremental as they apply existing methods to a new setting.
The paper proposes local-first information retrieval, where indexes, models, and inference run on user devices, and demonstrates that dense retrieval maintains over 91% nDCG@10 up to 100K documents and a 7B local language model reaches within 4 points of cloud baseline on answer quality, showing that the tradeoff is scope rather than quality.
The sensitive information in personal documents, legal files, and medical records is among the most valuable things to search, yet current retrieval-augmented generation systems still require sending content to remote servers. We propose local-first IR, a design philosophy where indexes, models, and inference reside on user devices, treating remote services as optional. This paper makes four contributions: (1) a framework organizing retrieval architectures along three dimensions: privacy and control, capability, and accessibility, (2) experiments on consumer hardware across five benchmarks, scaling from 1K to 1M documents with dense retrieval, BM25, and hybrid fusion. Dense retrieval keeps over 91% nDCG@10 up to 100K documents, with approximate HNSW indexes extending this to 1M with only 2% quality loss; a 7B local language model reaches within 4 points of a cloud baseline on answer quality, (3) competing perspectives for and against local-first IR, informed by experimental evidence, and (4) a research agenda identifying open problems. The real tradeoff is scope rather than quality: what matters is what you can search, not how well you can search it.