Reliable Academic Conference Question Answering: A Study Based on Large
  Language Model
Zhiwei Huang Long Jin Junjie Wang Mingchen Tu Yin Hua Zhiqiang Liu Jiawei Meng Huajun Chen Wen Zhang
The rapid growth of computer science has led to a proliferation of research
presented at academic conferences, fostering global scholarly communication.
Researchers consistently seek accurate, current information about these events
at all stages. This data surge necessitates an intelligent question-answering
system to efficiently address researchers' queries and ensure awareness of the
latest advancements. The information of conferences is usually published on
their official website, organized in a semi-structured way with a lot of text.
To address this need, we have developed the ConferenceQA dataset for 7 diverse
academic conferences with human annotations. Firstly, we employ a combination
of manual and automated methods to organize academic conference data in a
semi-structured JSON format. Subsequently, we annotate nearly 100
question-answer pairs for each conference. Each pair is classified into four
different dimensions. To ensure the reliability of the data, we manually
annotate the source of each answer. In light of recent advancements, Large
Language Models (LLMs) have demonstrated impressive performance in various NLP
tasks. They have demonstrated impressive capabilities in information-seeking
question answering after instruction fine-tuning, and as such, we present our
conference QA study based on LLM. Due to hallucination and outdated knowledge
of LLMs, we adopt retrieval based methods to enhance LLMs' question-answering
abilities. We have proposed a structure-aware retrieval method, specifically
designed to leverage inherent structural information during the retrieval
process. Empirical validation on the ConferenceQA dataset has demonstrated the
effectiveness of this method. The dataset and code are readily accessible on
https://github.com/zjukg/ConferenceQA.