Sub-linear RACE Sketches for Approximate Kernel Density Estimation on Streaming Data
This addresses the scalability issue for kernel methods in machine learning applications, particularly for streaming data, though it is incremental as it builds on existing sketching and approximation techniques.
The paper tackles the problem of high memory and computation costs for approximate kernel density estimation on high-dimensional streaming data by proposing RACE, a sketching algorithm that compresses vectors into a small array of integer counters, achieving 10x better compression than competing methods on real-world datasets.
Kernel density estimation is a simple and effective method that lies at the heart of many important machine learning applications. Unfortunately, kernel methods scale poorly for large, high dimensional datasets. Approximate kernel density estimation has a prohibitively high memory and computation cost, especially in the streaming setting. Recent sampling algorithms for high dimensional densities can reduce the computation cost but cannot operate online, while streaming algorithms cannot handle high dimensional datasets due to the curse of dimensionality. We propose RACE, an efficient sketching algorithm for kernel density estimation on high-dimensional streaming data. RACE compresses a set of N high dimensional vectors into a small array of integer counters. This array is sufficient to estimate the kernel density for a large class of kernels. Our sketch is practical to implement and comes with strong theoretical guarantees. We evaluate our method on real-world high-dimensional datasets and show that our sketch achieves 10x better compression compared to competing methods.