On Using Non-Volatile Memory in Apache Lucene
This is an incremental study for developers and researchers using Apache Lucene, focusing on optimizing search functionality with emerging memory technologies.
The paper tackled the problem of improving Apache Lucene's performance by using non-volatile memory (NVDIMM), showing modest improvements in committing, searching, and near-real-time searching, but indicating that bigger impacts require redesigning Lucene to access NVM as byte-addressable memory.
Apache Lucene is a widely popular information retrieval library used to provide search functionality in an extremely wide variety of applications. Naturally, it has to efficiently index and search large number of documents. With non-volatile memory in DIMM form factor (NVDIMM), software now has access to durable, byte-addressable memory with write latency within an order of magnitude of DRAM write latency. In this preliminary article, we present the first reported work on the impact of using NVDIMM on the performance of committing, searching, and near-real time searching in Apache Lucene. We show modest improvements by using NVM but, our empirical study suggests that bigger impact requires redesigning Lucene to access NVM as byte-addressable memory using loads and stores, instead of accessing NVM via the file system.