A Frustratingly Easy Improvement for Position Embeddings via Random Padding
This work addresses a specific bottleneck in extractive question answering for NLP practitioners, offering an incremental improvement to existing pre-trained models.
The paper tackles the problem of poorly trained rear position embeddings in Transformer models for extractive question answering, which occur due to fewer updates during training with varied context lengths. The proposed Random Padding method, which reorders input tokens during fine-tuning, significantly improves performance on instances with answers at rear positions, especially when models trained on short contexts are evaluated on long ones.
Position embeddings, encoding the positional relationships among tokens in text sequences, make great contributions to modeling local context features in Transformer-based pre-trained language models. However, in Extractive Question Answering, position embeddings trained with instances of varied context lengths may not perform well as we expect. Since the embeddings of rear positions are updated fewer times than the front position embeddings, the rear ones may not be properly trained. In this paper, we propose a simple but effective strategy, Random Padding, without any modifications to architectures of existing pre-trained language models. We adjust the token order of input sequences when fine-tuning, to balance the number of updating times of every position embedding. Experiments show that Random Padding can significantly improve model performance on the instances whose answers are located at rear positions, especially when models are trained on short contexts but evaluated on long contexts. Our code and data will be released for future research.