RACER: An LLM-powered Methodology for Scalable Analysis of
  Semi-structured Mental Health Interviews
Satpreet Harcharan Singh Kevin Jiang Kanchan Bhasin Ashutosh Sabharwal Nidal Moukaddam Ankit B Patel
Semi-structured interviews (SSIs) are a commonly employed data-collection
method in healthcare research, offering in-depth qualitative insights into
subject experiences. Despite their value, the manual analysis of SSIs is
notoriously time-consuming and labor-intensive, in part due to the difficulty
of extracting and categorizing emotional responses, and challenges in scaling
human evaluation for large populations. In this study, we develop RACER, a
Large Language Model (LLM) based expert-guided automated pipeline that
efficiently converts raw interview transcripts into insightful domain-relevant
themes and sub-themes. We used RACER to analyze SSIs conducted with 93
healthcare professionals and trainees to assess the broad personal and
professional mental health impacts of the COVID-19 crisis. RACER achieves
moderately high agreement with two human evaluators (72%), which approaches the
human inter-rater agreement (77%). Interestingly, LLMs and humans struggle with
similar content involving nuanced emotional, ambivalent/dialectical, and
psychological statements. Our study highlights the opportunities and challenges
in using LLMs to improve research efficiency and opens new avenues for scalable
analysis of SSIs in healthcare research.