Improving Open Information Extraction with Large Language Models: A
  Study on Demonstration Uncertainty
Chen Ling Xujiang Zhao Xuchao Zhang Yanchi Liu Wei Cheng Haoyu Wang Zhengzhang Chen Takao Osaki Katsushi Matsuda Haifeng Chen Liang Zhao
Open Information Extraction (OIE) task aims at extracting structured facts
from unstructured text, typically in the form of (subject, relation, object)
triples. Despite the potential of large language models (LLMs) like ChatGPT as
a general task solver, they lag behind state-of-the-art (supervised) methods in
OIE tasks due to two key issues. First, LLMs struggle to distinguish irrelevant
context from relevant relations and generate structured output due to the
restrictions on fine-tuning the model. Second, LLMs generates responses
autoregressively based on probability, which makes the predicted relations lack
confidence. In this paper, we assess the capabilities of LLMs in improving the
OIE task. Particularly, we propose various in-context learning strategies to
enhance LLM's instruction-following ability and a demonstration uncertainty
quantification module to enhance the confidence of the generated relations. Our
experiments on three OIE benchmark datasets show that our approach holds its
own against established supervised methods, both quantitatively and
qualitatively.