OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction
This work addresses the problem of predicting lexical complexity for natural language processing applications, but it is incremental as it builds on existing transformer models and methods.
The paper tackled lexical complexity prediction for words and multiword expressions by proposing an ensemble model that uses pretrained transformers and training methods like multi-step fine-tuning and adversarial training, achieving competitive results with a top-10 ranking in SemEval-2021 Task 1.
We propose an ensemble model for predicting the lexical complexity of words and multiword expressions (MWEs). The model receives as input a sentence with a target word or MWEand outputs its complexity score. Given that a key challenge with this task is the limited size of annotated data, our model relies on pretrained contextual representations from different state-of-the-art transformer-based language models (i.e., BERT and RoBERTa), and on a variety of training methods for further enhancing model generalization and robustness:multi-step fine-tuning and multi-task learning, and adversarial training. Additionally, we propose to enrich contextual representations by adding hand-crafted features during training. Our model achieved competitive results and ranked among the top-10 systems in both sub-tasks.