Domain Knowledge Distillation from Large Language Model: An Empirical
  Study in the Autonomous Driving Domain
Yun Tang Antonio A. Bruto da Costa Jason Zhang Irvine Patrick Siddartha Khastgir Paul Jennings
Engineering knowledge-based (or expert) systems require extensive manual
effort and domain knowledge. As Large Language Models (LLMs) are trained using
an enormous amount of cross-domain knowledge, it becomes possible to automate
such engineering processes. This paper presents an empirical automation and
semi-automation framework for domain knowledge distillation using prompt
engineering and the LLM ChatGPT. We assess the framework empirically in the
autonomous driving domain and present our key observations. In our
implementation, we construct the domain knowledge ontology by "chatting" with
ChatGPT. The key finding is that while fully automated domain ontology
construction is possible, human supervision and early intervention typically
improve efficiency and output quality as they lessen the effects of response
randomness and the butterfly effect. We, therefore, also develop a web-based
distillation assistant enabling supervision and flexible intervention at
runtime. We hope our findings and tools could inspire future research toward
revolutionizing the engineering of knowledge-based systems across application
domains.