Evaluating the Performance of LLMs on Technical Language Processing
  tasks
Andrew Kernycky David Coleman Christopher Spence Udayan Das
In this paper we present the results of an evaluation study of the
perfor-mance of LLMs on Technical Language Processing tasks. Humans are often
confronted with tasks in which they have to gather information from dispar-ate
sources and require making sense of large bodies of text. These tasks can be
significantly complex for humans and often require deep study including
rereading portions of a text. Towards simplifying the task of gathering
in-formation we evaluated LLMs with chat interfaces for their ability to
provide answers to standard questions that a human can be expected to answer
based on their reading of a body of text. The body of text under study is Title
47 of the United States Code of Federal Regulations (CFR) which describes
regula-tions for commercial telecommunications as governed by the Federal
Com-munications Commission (FCC). This has been a body of text of interest
be-cause our larger research concerns the issue of making sense of information
related to Wireless Spectrum Governance and usage in an automated manner to
support Dynamic Spectrum Access. The information concerning this wireless
spectrum domain is found in many disparate sources, with Title 47 of the CFR
being just one of many. Using a range of LLMs and providing the required CFR
text as context we were able to quantify the performance of those LLMs on the
specific task of answering the questions below.