Qsnail: A Questionnaire Dataset for Sequential Question Generation
Yan Lei Liang Pang Yuanzhuo Wang Huawei Shen Xueqi Cheng
The questionnaire is a professional research methodology used for both
qualitative and quantitative analysis of human opinions, preferences,
attitudes, and behaviors. However, designing and evaluating questionnaires
demands significant effort due to their intricate and complex structure.
Questionnaires entail a series of questions that must conform to intricate
constraints involving the questions, options, and overall structure.
Specifically, the questions should be relevant and specific to the given
research topic and intent. The options should be tailored to the questions,
ensuring they are mutually exclusive, completed, and ordered sensibly.
Moreover, the sequence of questions should follow a logical order, grouping
similar topics together. As a result, automatically generating questionnaires
presents a significant challenge and this area has received limited attention
primarily due to the scarcity of high-quality datasets. To address these
issues, we present Qsnail, the first dataset specifically constructed for the
questionnaire generation task, which comprises 13,168 human-written
questionnaires gathered from online platforms. We further conduct experiments
on Qsnail, and the results reveal that retrieval models and traditional
generative models do not fully align with the given research topic and intents.
Large language models, while more closely related to the research topic and
intents, exhibit significant limitations in terms of diversity and specificity.
Despite enhancements through the chain-of-thought prompt and finetuning,
questionnaires generated by language models still fall short of human-written
questionnaires. Therefore, questionnaire generation is challenging and needs to
be further explored. The dataset is available at:
https://github.com/LeiyanGithub/qsnail.