Fishing in the Stream: Similarity Search over Endless Data
This addresses the problem of efficient similarity search in streaming data for applications requiring real-time processing, though it is incremental as it builds on existing LSH methods.
The paper tackles similarity search over endless data streams by introducing a time-sensitive approach that considers data quality and temporal characteristics, and proposes Stream-LSH, which bounds index size and increases the probability of finding similar items compared to alternatives, with empirical validation on real-world datasets.
Similarity search is the task of retrieving data items that are similar to a given query. In this paper, we introduce the time-sensitive notion of similarity search over endless data-streams (SSDS), which takes into account data quality and temporal characteristics in addition to similarity. SSDS is challenging as it needs to process unbounded data, while computation resources are bounded. We propose Stream-LSH, a randomized SSDS algorithm that bounds the index size by retaining items according to their freshness, quality, and dynamic popularity attributes. We analytically show that Stream-LSH increases the probability to find similar items compared to alternative approaches using the same space capacity. We further conduct an empirical study using real world stream datasets, which confirms our theoretical results.