Slot Induction via Pre-trained Language Model Probing and Multi-level
  Contrastive Learning
Hoang H. Nguyen Chenwei Zhang Ye Liu Philip S. Yu
Recent advanced methods in Natural Language Understanding for Task-oriented
Dialogue (TOD) Systems (e.g., intent detection and slot filling) require a
large amount of annotated data to achieve competitive performance. In reality,
token-level annotations (slot labels) are time-consuming and difficult to
acquire. In this work, we study the Slot Induction (SI) task whose objective is
to induce slot boundaries without explicit knowledge of token-level slot
annotations. We propose leveraging Unsupervised Pre-trained Language Model
(PLM) Probing and Contrastive Learning mechanism to exploit (1) unsupervised
semantic knowledge extracted from PLM, and (2) additional sentence-level intent
label signals available from TOD. Our approach is shown to be effective in SI
task and capable of bridging the gaps with token-level supervised models on two
NLU benchmark datasets. When generalized to emerging intents, our SI objectives
also provide enhanced slot label representations, leading to improved
performance on the Slot Filling tasks.