IIE-NLP-Eyas at SemEval-2021 Task 4: Enhancing PLM for ReCAM with Special Tokens, Re-Ranking, Siamese Encoders and Back Translation
This work addresses a specific challenge in natural language processing for abstract concept understanding, but it is incremental as it builds on existing models and techniques.
The paper tackled the problem of reading comprehension of abstract meaning in natural language by enhancing a pre-trained language model (RoBERTa) with special tokens, re-ranking, Siamese encoders, and back translation, achieving eighth and tenth ranks on two subtasks in SemEval-2021 Task 4.
This paper introduces our systems for all three subtasks of SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning. To help our model better represent and understand abstract concepts in natural language, we well-design many simple and effective approaches adapted to the backbone model (RoBERTa). Specifically, we formalize the subtasks into the multiple-choice question answering format and add special tokens to abstract concepts, then, the final prediction of question answering is considered as the result of subtasks. Additionally, we employ many finetuning tricks to improve the performance. Experimental results show that our approaches achieve significant performance compared with the baseline systems. Our approaches achieve eighth rank on subtask-1 and tenth rank on subtask-2.