Rethinking the Authorship Verification Experimental Setups
This work addresses experimental setup issues in authorship verification, which is important for forensic linguistics and digital security, though it is incremental as it builds on existing PAN datasets and BERT models.
The authors tackled inconsistent performance in authorship verification by creating five new dataset splits to isolate topic and style biases, and found that BERT-like models are competitive with state-of-the-art methods but biased toward named entities. Removing named entities improved results by achieving better generalization on a new DarkReddit dataset.
One of the main drivers of the recent advances in authorship verification is the PAN large-scale authorship dataset. Despite generating significant progress in the field, inconsistent performance differences between the closed and open test sets have been reported. To this end, we improve the experimental setup by proposing five new public splits over the PAN dataset, specifically designed to isolate and identify biases related to the text topic and to the author's writing style. We evaluate several BERT-like baselines on these splits, showing that such models are competitive with authorship verification state-of-the-art methods. Furthermore, using explainable AI, we find that these baselines are biased towards named entities. We show that models trained without the named entities obtain better results and generalize better when tested on DarkReddit, our new dataset for authorship verification.