Examining the Influence of Varied Levels of Domain Knowledge Base
  Inclusion in GPT-based Intelligent Tutors
Blake Castleman Mehmet Kerem Turkcan
Recent advancements in large language models (LLMs) have facilitated the
development of chatbots with sophisticated conversational capabilities.
However, LLMs exhibit frequent inaccurate responses to queries, hindering
applications in educational settings. In this paper, we investigate the
effectiveness of integrating a knowledge base (KB) with LLM intelligent tutors
to increase response reliability. To achieve this, we design a scaleable KB
that affords educational supervisors seamless integration of lesson curricula,
which is automatically processed by the intelligent tutoring system. We then
detail an evaluation, where student participants were presented with questions
about the artificial intelligence curriculum to respond to. GPT-4 intelligent
tutors with varying hierarchies of KB access and human domain experts then
assessed these responses. Lastly, students cross-examined the intelligent
tutors' responses to the domain experts' and ranked their various pedagogical
abilities. Results suggest that, although these intelligent tutors still
demonstrate a lower accuracy compared to domain experts, the accuracy of the
intelligent tutors increases when access to a KB is granted. We also observe
that the intelligent tutors with KB access exhibit better pedagogical abilities
to speak like a teacher and understand students than those of domain experts,
while their ability to help students remains lagging behind domain experts.