Probabilistic Blocking with An Application to the Syrian Conflict
This work addresses database merging challenges for researchers or analysts dealing with conflict data, but it is incremental as it builds on existing locality sensitive hashing techniques.
The paper tackles entity resolution by introducing KLSH, DOPH, and weighted DOPH as blocking methods, and applies them to a subset of the Syrian conflict dataset to demonstrate their effectiveness.
Entity resolution seeks to merge databases as to remove duplicate entries where unique identifiers are typically unknown. We review modern blocking approaches for entity resolution, focusing on those based upon locality sensitive hashing (LSH). First, we introduce $k$-means locality sensitive hashing (KLSH), which is based upon the information retrieval literature and clusters similar records into blocks using a vector-space representation and projections. Second, we introduce a subquadratic variant of LSH to the literature, known as Densified One Permutation Hashing (DOPH). Third, we propose a weighted variant of DOPH. We illustrate each method on an application to a subset of the ongoing Syrian conflict, giving a discussion of each method.