Relative Position Prediction as Pre-training for Text Encoders
This work addresses the need for more effective self-supervised learning methods in NLP, though it appears incremental as it adapts existing relative position encoding ideas.
The paper tackles the problem of improving text encoder pre-training by proposing a position-centric objective based on relative position prediction, which aims to achieve superior performance on downstream tasks.
Meaning is defined by the company it keeps. However, company is two-fold: It's based on the identity of tokens and also on their position (topology). We argue that a position-centric perspective is more general and useful. The classic MLM and CLM objectives in NLP are easily phrased as position predictions over the whole vocabulary. Adapting the relative position encoding paradigm in NLP to create relative labels for self-supervised learning, we seek to show superior pre-training judged by performance on downstream tasks.