A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification
This work addresses a gap for NLP researchers by providing a systematic evaluation of public models in discourse analysis, though it is incremental as it focuses on benchmarking rather than introducing new methods.
This paper tackled the lack of a comprehensive language model comparison for implicit discourse relation classification by fine-tuning seven pre-trained models on the PDTB-3 dataset, achieving a new state-of-the-art accuracy of 0.671 and revealing that sentence-level pre-training objectives underperform compared to masked language modeling with full attention.
Though discourse parsing can help multiple NLP fields, there has been no wide language model search done on implicit discourse relation classification. This hinders researchers from fully utilizing public-available models in discourse analysis. This work is a straightforward, fine-tuned discourse performance comparison of seven pre-trained language models. We use PDTB-3, a popular discourse relation annotated dataset. Through our model search, we raise SOTA to 0.671 ACC and obtain novel observations. Some are contrary to what has been reported before (Shi and Demberg, 2019b), that sentence-level pre-training objectives (NSP, SBO, SOP) generally fail to produce the best performing model for implicit discourse relation classification. Counterintuitively, similar-sized PLMs with MLM and full attention led to better performance.