Is there something I'm missing? Topic Modeling in eDiscovery
This addresses the legal industry's need for efficient and defensible eDiscovery processes by demonstrating that computer-assisted searches can be considered complete based on topics, potentially reducing fears of missing critical information.
The paper tackles the problem of ensuring completeness in legal eDiscovery searches by showing that topic modeling can achieve topic-level completeness even with imperfect document recall. Using Latent Dirichlet Allocation, they found that categorizers achieving 80% recall captured all topics from the full relevant document set, with no new topics in the missed documents.
In legal eDiscovery, the parties are required to search through their electronically stored information to find documents that are relevant to a specific case. Negotiations over the scope of these searches are often based on a fear that something will be missed. This paper continues an argument that discovery should be based on identifying the facts of a case. If a search process is less than complete (if it has Recall less than 100%), it may still be complete in presenting all of the relevant available topics. In this study, Latent Dirichlet Allocation was used to identify 100 topics from all of the known relevant documents. The documents were then categorized to about 80% Recall (i.e., 80% of the relevant documents were found by the categorizer, designated the hit set and 20% were missed, designated the missed set). Despite the fact that less than all of the relevant documents were identified by the categorizer, the documents that were identified contained all of the topics derived from the full set of documents. This same pattern held whether the categorizer was a naïve Bayes categorizer trained on a random selection of documents or a Support Vector Machine trained with Continuous Active Learning (which focuses evaluation on the most-likely-to-be-relevant documents). No topics were identified in either categorizer's missed set that were not already seen in the hit set. Not only is a computer-assisted search process reasonable (as required by the Federal Rules of Civil Procedure), it is also complete when measured by topics.