Empirical Study of Zero-Shot NER with ChatGPT
Tingyu Xie Qi Li Jian Zhang Yan Zhang Zuozhu Liu Hongwei Wang
Large language models (LLMs) exhibited powerful capability in various natural
language processing tasks. This work focuses on exploring LLM performance on
zero-shot information extraction, with a focus on the ChatGPT and named entity
recognition (NER) task. Inspired by the remarkable reasoning capability of LLM
on symbolic and arithmetic reasoning, we adapt the prevalent reasoning methods
to NER and propose reasoning strategies tailored for NER. First, we explore a
decomposed question-answering paradigm by breaking down the NER task into
simpler subproblems by labels. Second, we propose syntactic augmentation to
stimulate the model's intermediate thinking in two ways: syntactic prompting,
which encourages the model to analyze the syntactic structure itself, and tool
augmentation, which provides the model with the syntactic information generated
by a parsing tool. Besides, we adapt self-consistency to NER by proposing a
two-stage majority voting strategy, which first votes for the most consistent
mentions, then the most consistent types. The proposed methods achieve
remarkable improvements for zero-shot NER across seven benchmarks, including
Chinese and English datasets, and on both domain-specific and general-domain
scenarios. In addition, we present a comprehensive analysis of the error types
with suggestions for optimization directions. We also verify the effectiveness
of the proposed methods on the few-shot setting and other LLMs.