The Case for Instance-Optimized LLMs in OLAP Databases
This work addresses the scalability problem for database practitioners and researchers by making LLM-enhanced analytics more practical, though it is incremental as it builds on existing compression techniques.
The paper tackles the high computational and memory costs of deploying large language models (LLMs) at scale in OLAP databases by introducing IOLM-DB, a system that generates lightweight, query-specific models, achieving up to 76% reduction in model footprints and 3.31× throughput increase while maintaining accuracy.
Large Language Models (LLMs) can enhance analytics systems with powerful data summarization, cleaning, and semantic transformation capabilities. However, deploying LLMs at scale -- processing millions to billions of rows -- remains prohibitively expensive in computation and memory. We present IOLM-DB, a novel system that makes LLM-enhanced database queries practical through query-specific model optimization. Instead of using general-purpose LLMs, IOLM-DB generates lightweight, specialized models tailored to each query's specific needs using representative data samples. IOLM-DB reduces model footprints by up to 76% and increases throughput by up to 3.31$\times$ while maintaining accuracy through aggressive compression techniques, including quantization, sparsification, and structural pruning. We further show how our approach enables higher parallelism on existing hardware and seamlessly supports caching and batching strategies to reduce overheads. Our prototype demonstrates that leveraging LLM queries inside analytics systems is feasible at scale, opening new possibilities for future OLAP applications.