Fine-tuning and aligning question answering models for complex
  information extraction tasks
Matthias Engelbach Dennis Klau Felix Scheerer Jens Drawehn Maximilien Kintz
The emergence of Large Language Models (LLMs) has boosted performance and
possibilities in various NLP tasks. While the usage of generative AI models
like ChatGPT opens up new opportunities for several business use cases, their
current tendency to hallucinate fake content strongly limits their
applicability to document analysis, such as information retrieval from
documents. In contrast, extractive language models like question answering (QA)
or passage retrieval models guarantee query results to be found within the
boundaries of an according context document, which makes them candidates for
more reliable information extraction in productive environments of companies.
In this work we propose an approach that uses and integrates extractive QA
models for improved feature extraction of German business documents such as
insurance reports or medical leaflets into a document analysis solution. We
further show that fine-tuning existing German QA models boosts performance for
tailored extraction tasks of complex linguistic features like damage cause
explanations or descriptions of medication appearance, even with using only a
small set of annotated data. Finally, we discuss the relevance of scoring
metrics for evaluating information extraction tasks and deduce a combined
metric from Levenshtein distance, F1-Score, Exact Match and ROUGE-L to mimic
the assessment criteria from human experts.