Large Language Models Meet Open-World Intent Discovery and Recognition:
  An Evaluation of ChatGPT
Xiaoshuai Song Keqing He Pei Wang Guanting Dong Yutao Mou Jingang Wang Yunsen Xian Xunliang Cai Weiran Xu
The tasks of out-of-domain (OOD) intent discovery and generalized intent
discovery (GID) aim to extend a closed intent classifier to open-world intent
sets, which is crucial to task-oriented dialogue (TOD) systems. Previous
methods address them by fine-tuning discriminative models. Recently, although
some studies have been exploring the application of large language models
(LLMs) represented by ChatGPT to various downstream tasks, it is still unclear
for the ability of ChatGPT to discover and incrementally extent OOD intents. In
this paper, we comprehensively evaluate ChatGPT on OOD intent discovery and
GID, and then outline the strengths and weaknesses of ChatGPT. Overall, ChatGPT
exhibits consistent advantages under zero-shot settings, but is still at a
disadvantage compared to fine-tuned models. More deeply, through a series of
analytical experiments, we summarize and discuss the challenges faced by LLMs
including clustering, domain-specific understanding, and cross-domain
in-context learning scenarios. Finally, we provide empirical guidance for
future directions to address these challenges.