Assessing the quality of information extraction
Filip Seitl Tomáš Kovářík Soheyla Mirshahi Jan Kryštůfek Rastislav Dujava Matúš Ondreička Herbert Ullrich Petr Gronat
Advances in large language models have notably enhanced the efficiency of
information extraction from unstructured and semi-structured data sources. As
these technologies become integral to various applications, establishing an
objective measure for the quality of information extraction becomes imperative.
However, the scarcity of labeled data presents significant challenges to this
endeavor. In this paper, we introduce an automatic framework to assess the
quality of the information extraction and its completeness. The framework
focuses on information extraction in the form of entity and its properties. We
discuss how to handle the input/output size limitations of the large language
models and analyze their performance when iteratively extracting the
information. Finally, we introduce metrics to evaluate the quality of the
extraction and provide an extensive discussion on how to interpret the metrics.