Extracting Mathematical Concepts with Large Language Models
Valeria de Paiva Qiyue Gao Pavel Kovalev Lawrence S. Moss
We extract mathematical concepts from mathematical text using generative
large language models (LLMs) like ChatGPT, contributing to the field of
automatic term extraction (ATE) and mathematical text processing, and also to
the study of LLMs themselves. Our work builds on that of others in that we aim
for automatic extraction of terms (keywords) in one mathematical field,
category theory, using as a corpus the 755 abstracts from a snapshot of the
online journal "Theory and Applications of Categories", circa 2020. Where our
study diverges from previous work is in (1) providing a more thorough analysis
of what makes mathematical term extraction a difficult problem to begin with;
(2) paying close attention to inter-annotator disagreements; (3) providing a
set of guidelines which both human and machine annotators could use to
standardize the extraction process; (4) introducing a new annotation tool to
help humans with ATE, applicable to any mathematical field and even beyond
mathematics; (5) using prompts to ChatGPT as part of the extraction process,
and proposing best practices for such prompts; and (6) raising the question of
whether ChatGPT could be used as an annotator on the same level as human
experts. Our overall findings are that the matter of mathematical ATE is an
interesting field which can benefit from participation by LLMs, but LLMs
themselves cannot at this time surpass human performance on it.