TaDSE: Template-aware Dialogue Sentence Embeddings
Minsik Oh Jiwei Li Guoyin Wang
Learning high quality sentence embeddings from dialogues has drawn increasing
attentions as it is essential to solve a variety of dialogue-oriented tasks
with low annotation cost. However, directly annotating and gathering utterance
relationships in conversations are difficult, while token-level annotations,
\eg, entities, slots and templates, are much easier to obtain. General sentence
embedding methods are usually sentence-level self-supervised frameworks and
cannot utilize token-level extra knowledge. In this paper, we introduce
Template-aware Dialogue Sentence Embedding (TaDSE), a novel augmentation method
that utilizes template information to effectively learn utterance
representation via self-supervised contrastive learning framework. TaDSE
augments each sentence with its corresponding template and then conducts
pairwise contrastive learning over both sentence and template. We further
enhance the effect with a synthetically augmented dataset that enhances
utterance-template relation, in which entity detection (slot-filling) is a
preliminary step. We evaluate TaDSE performance on five downstream benchmark
datasets. The experiment results show that TaDSE achieves significant
improvements over previous SOTA methods, along with a consistent Intent
Classification task performance improvement margin. We further introduce a
novel analytic instrument of Semantic Compression method, for which we discover
a correlation with uniformity and alignment. Our code will be released soon.