Benchmarking Large Language Models with Augmented Instructions for
  Fine-grained Information Extraction
Jun Gao Huan Zhao Yice Zhang Wei Wang Changlong Yu Ruifeng Xu
Information Extraction (IE) is an essential task in Natural Language
Processing. Traditional methods have relied on coarse-grained extraction with
simple instructions. However, with the emergence of Large Language Models
(LLMs), there is a need to adapt IE techniques to leverage the capabilities of
these models. This paper introduces a fine-grained IE benchmark dataset
tailored for LLMs, employing augmented instructions for each information type,
which includes task descriptions, extraction rules, output formats, and
examples. Through extensive evaluations, we observe that encoder-decoder
models, particularly T5 and FLAN-T5, perform well in generalizing to unseen
information types, while ChatGPT exhibits greater adaptability to new task
forms. Our results also indicate that performance is not solely dictated by
model scale, and highlight the significance of architecture, data diversity,
and learning techniques. This work paves the way for a more refined and
versatile utilization of LLMs in Information Extraction.