SentPWNet: A Unified Sentence Pair Weighting Network for Task-specific Sentence Embedding
This work addresses a specific bottleneck in sentence embedding for NLP tasks like semantic text similarity, offering an incremental improvement over existing pair-based methods.
The paper tackled the problem of biased sentence representations in pair-based metric learning by proposing SentPWNet, a unified locality weighting and learning framework that updates weights based on neighboring spatial distribution, which consistently outperformed existing methods on four public datasets and a self-collected benchmark with 1.4 million places.
Pair-based metric learning has been widely adopted to learn sentence embedding in many NLP tasks such as semantic text similarity due to its efficiency in computation. Most existing works employed a sequence encoder model and utilized limited sentence pairs with a pair-based loss to learn discriminating sentence representation. However, it is known that the sentence representation can be biased when the sampled sentence pairs deviate from the true distribution of all sentence pairs. In this paper, our theoretical analysis shows that existing works severely suffered from a good pair sampling and instance weighting strategy. Instead of one time pair selection and learning on equal weighted pairs, we propose a unified locality weighting and learning framework to learn task-specific sentence embedding. Our model, SentPWNet, exploits the neighboring spatial distribution of each sentence as locality weight to indicate the informative level of sentence pair. Such weight is updated along with pair-loss optimization in each round, ensuring the model keep learning the most informative sentence pairs. Extensive experiments on four public available datasets and a self-collected place search benchmark with 1.4 million places clearly demonstrate that our model consistently outperforms existing sentence embedding methods with comparable efficiency.