Sen2Pro: A Probabilistic Perspective to Sentence Embedding from
  Pre-trained Language Model
Lingfeng Shen Haiyun Jiang Lemao Liu Shuming Shi
Sentence embedding is one of the most fundamental tasks in Natural Language
Processing and plays an important role in various tasks. The recent
breakthrough in sentence embedding is achieved by pre-trained language models
(PLMs). Despite its success, an embedded vector (Sen2Vec) representing a point
estimate does not naturally express uncertainty in a taskagnostic way. This
paper thereby proposes an efficient framework on probabilistic sentence
embedding (Sen2Pro) from PLMs, and it represents a sentence as a probability
density distribution in an embedding space to reflect both model uncertainty
and data uncertainty (i.e., many-to-one nature) in the sentence representation.
The proposed framework performs in a plug-and-play way without retraining PLMs
anymore, and it is easy to implement and generally applied on top of any PLM.
The superiority of Sen2Pro over Sen2Vec has been theoretically verified and
practically illustrated on different NLP tasks.