NeurBench: A Benchmark Suite for Learned Database Components with Drift Modeling
For researchers and practitioners developing learned database components, NeurBench enables systematic robustness evaluation under diverse drift scenarios, addressing a gap in existing benchmarks.
NeurBench is a benchmark suite for evaluating learned database components under measurable and controllable data and workload drift, using a drift factor to quantify drift types and a generation framework to simulate realistic drift. Experiments show its effectiveness in generating realistic drift and providing performance insights.
Learned database components, which deeply integrate machine learning into their design, have been extensively studied in recent years. Given the dynamism of databases, where data and workloads continuously drift, it is crucial for learned database components to remain effective and efficient in the face of data and workload drift. Robustness, therefore, is a key factor in assessing their practical applicability. Although recent works examine learned database components under specific drift, they fail to enable systematic performance evaluations across a broad range of drift or under customized drift as needed. This paper presents NeurBench, a new benchmark suite that supports evaluating learned database components under measurable and controllable data and workload drift. We quantify diverse types of drift by introducing a key concept called the drift factor. Building on this formulation, we propose a drift-aware data and workload generation framework that effectively simulates real-world drift while preserving inherent correlations. Experimental results demonstrate the effectiveness of NeurBench in generating realistic data and workload drift, while providing insights into the performance of representative learned database components under different drift scenarios.