OnPrem.LLM: A Privacy-Conscious Document Intelligence Toolkit
This toolkit addresses privacy concerns for organizations handling sensitive data, offering a practical solution for local or hybrid deployments, though it is incremental as it builds on existing methods like RAG and quantization.
The authors tackled the problem of applying large language models to sensitive data in offline environments by developing OnPrem.LLM, a toolkit that provides privacy-preserving document intelligence pipelines, resulting in a system that supports multiple LLM backends and a no-code interface for non-technical users.
We present OnPrem$.$LLM, a Python-based toolkit for applying large language models (LLMs) to sensitive, non-public data in offline or restricted environments. The system is designed for privacy-preserving use cases and provides prebuilt pipelines for document processing and storage, retrieval-augmented generation (RAG), information extraction, summarization, classification, and prompt/output processing with minimal configuration. OnPrem$.$LLM supports multiple LLM backends -- including llama$.$cpp, Ollama, vLLM, and Hugging Face Transformers -- with quantized model support, GPU acceleration, and seamless backend switching. Although designed for fully local execution, OnPrem$.$LLM also supports integration with a wide range of cloud LLM providers when permitted, enabling hybrid deployments that balance performance with data control. A no-code web interface extends accessibility to non-technical users.