Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review
This work improves the autonomy and reliability of document review systems, particularly for legal and information retrieval tasks, though it is incremental as it builds on an existing method.
The paper tackled the problem of reducing manual tuning in continuous active learning for technology-assisted review by eliminating topic- and dataset-specific parameters, requiring only initial user input and ongoing relevance feedback. The enhanced method consistently outperformed the original version and other methods across multiple datasets, achieving superior results on the majority of topics.
We enhance the autonomy of the continuous active learning method shown by Cormack and Grossman (SIGIR 2014) to be effective for technology-assisted review, in which documents from a collection are retrieved and reviewed, using relevance feedback, until substantially all of the relevant documents have been reviewed. Autonomy is enhanced through the elimination of topic-specific and dataset-specific tuning parameters, so that the sole input required by the user is, at the outset, a short query, topic description, or single relevant document; and, throughout the review, ongoing relevance assessments of the retrieved documents. We show that our enhancements consistently yield superior results to Cormack and Grossman's version of continuous active learning, and other methods, not only on average, but on the vast majority of topics from four separate sets of tasks: the legal datasets examined by Cormack and Grossman, the Reuters RCV1-v2 subject categories, the TREC 6 AdHoc task, and the construction of the TREC 2002 filtering test collection.