Keeping Users Engaged During Repeated Administration of the Same
  Questionnaire: Using Large Language Models to Reliably Diversify Questions
Hye Sun Yun Mehdi Arjmand Phillip Raymond Sherlock Michael Paasche-Orlow James W. Griffith Timothy Bickmore
Standardized, validated questionnaires are vital tools in HCI research and
healthcare, offering dependable self-report data. However, their repeated use
in longitudinal or pre-post studies can induce respondent fatigue, impacting
data quality via response biases and decreased response rates. We propose
utilizing large language models (LLMs) to generate diverse questionnaire
versions while retaining good psychometric properties. In a longitudinal study,
participants engaged with our agent system and responded daily for two weeks to
either a standardized depression questionnaire or one of two LLM-generated
questionnaire variants, alongside a validated depression questionnaire.
Psychometric testing revealed consistent covariation between the external
criterion and the focal measure administered across the three conditions,
demonstrating the reliability and validity of the LLM-generated variants.
Participants found the repeated administration of the standardized
questionnaire significantly more repetitive compared to the variants. Our
findings highlight the potential of LLM-generated variants to invigorate
questionnaires, fostering engagement and interest without compromising
validity.