Ramdoot Pydipaty

Johnu George, Ce Gao, Richard Liu et al.

In this paper, we introduce Katib: a scalable, cloud-native, and production-ready hyperparameter tuning system that is agnostic of the underlying machine learning framework. Though there are multiple hyperparameter tuning systems available, this is the first one that caters to the needs of both users and administrators of the system. We present the motivation and design of the system and contrast it with existing hyperparameter tuning systems, especially in terms of multi-tenancy, scalability, fault-tolerance, and extensibility. It can be deployed on local machines, or hosted as a service in on-premise data centers, or in private/public clouds. We demonstrate the advantage of our system using experimental results as well as real-world, production use cases. Katib has active contributors from multiple companies and is open-sourced at \emph{https://github.com/kubeflow/katib} under the Apache 2.0 license.

Ramdoot Pydipaty, Amit Saha

Apache Lucene is a widely popular information retrieval library used to provide search functionality in an extremely wide variety of applications. Naturally, it has to efficiently index and search large number of documents. With non-volatile memory in DIMM form factor (NVDIMM), software now has access to durable, byte-addressable memory with write latency within an order of magnitude of DRAM write latency. In this preliminary article, we present the first reported work on the impact of using NVDIMM on the performance of committing, searching, and near-real time searching in Apache Lucene. We show modest improvements by using NVM but, our empirical study suggests that bigger impact requires redesigning Lucene to access NVM as byte-addressable memory using loads and stores, instead of accessing NVM via the file system.