Identifying Duplicate and Contradictory Information in Wikipedia
This addresses quality issues in Wikipedia for users and editors, but it is incremental as it applies existing methods to a new dataset.
The study tackled the problem of identifying duplicate and contradictory information in Wikipedia by applying near-duplicate detection techniques, resulting in the categorization of sentence clusters into six types, including identical sentences that quantify content copying and near-duplicate sentences highlighting contradictions.
Our study identifies sentences in Wikipedia articles that are either identical or highly similar by applying techniques for near-duplicate detection of web pages. This is accomplished with a MapReduce implementation of minhash to identify clusters of sentences with high Jaccard similarity. We show that these clusters can be categorized into six different types, two of which are particularly interesting: identical sentences quantify the extent to which content in Wikipedia is copied and pasted, and near-duplicate sentences that state contradictory facts point to quality issues in Wikipedia.