Sen2Pro: A Probabilistic Perspective to Sentence Embedding from Pre-trained Language Model
This addresses the problem of uncertainty in sentence embeddings for NLP practitioners, offering a plug-and-play solution that is incremental but enhances existing methods.
The paper tackles the lack of uncertainty representation in sentence embeddings from pre-trained language models by proposing Sen2Pro, a probabilistic framework that models sentences as density distributions, achieving theoretical and practical improvements over point estimates.
Sentence embedding is one of the most fundamental tasks in Natural Language Processing and plays an important role in various tasks. The recent breakthrough in sentence embedding is achieved by pre-trained language models (PLMs). Despite its success, an embedded vector (Sen2Vec) representing a point estimate does not naturally express uncertainty in a taskagnostic way. This paper thereby proposes an efficient framework on probabilistic sentence embedding (Sen2Pro) from PLMs, and it represents a sentence as a probability density distribution in an embedding space to reflect both model uncertainty and data uncertainty (i.e., many-to-one nature) in the sentence representation. The proposed framework performs in a plug-and-play way without retraining PLMs anymore, and it is easy to implement and generally applied on top of any PLM. The superiority of Sen2Pro over Sen2Vec has been theoretically verified and practically illustrated on different NLP tasks.