The Mafiascum Dataset: A Large Text Corpus for Deception Detection
This provides a new dataset for deception detection research, addressing a data scarcity problem for NLP researchers, though it is incremental as it applies existing methods to new data.
The paper tackles the lack of large-scale labeled deceptive text by introducing the Mafiascum dataset, a corpus of over 9000 documents from Mafia games, and uses it to train a logistic regression classifier that achieves an average precision of 0.39 and AUROC of 0.68 on long documents.
Detecting deception in natural language has a wide variety of applications, but because of its hidden nature there are currently no public, large-scale sources of labeled deceptive text. This work introduces the Mafiascum dataset [1], a collection of over 700 games of Mafia, in which players are randomly assigned either deceptive or non-deceptive roles and then interact via forum postings. Over 9000 documents were compiled from the dataset, which each contained all messages written by a single player in a single game. This corpus was used to construct a set of hand-picked linguistic features based on prior deception research, as well as a set of average word vectors enriched with subword information. A logistic regression classifier fit on a combination of these feature sets achieved an average precision of 0.39 (chance = 0.26) and an AUROC of 0.68 on 5000+ word documents. On 50+ word documents, an average precision of 0.29 (chance = 0.23) and an AUROC of 0.59 was achieved. [1] https://bitbucket.org/bopjesvla/thesis/src