Comparing Large Language Model AI and Human-Generated Coaching Messages
  for Behavioral Weight Loss
Zhuoran Huang Michael P. Berry Christina Chwyl Gary Hsieh Jing Wei Evan M. Forman
Automated coaching messages for weight control can save time and costs, but
their repetitive, generic nature may limit their effectiveness compared to
human coaching. Large language model (LLM) based artificial intelligence (AI)
chatbots, like ChatGPT, could offer more personalized and novel messages to
address repetition with their data-processing abilities. While LLM AI
demonstrates promise to encourage healthier lifestyles, studies have yet to
examine the feasibility and acceptability of LLM-based BWL coaching. 87 adults
in a weight-loss trial rated ten coaching messages' helpfulness (five
human-written, five ChatGPT-generated) using a 5-point Likert scale, providing
additional open-ended feedback to justify their ratings. Participants also
identified which messages they believed were AI-generated. The evaluation
occurred in two phases: messages in Phase 1 were perceived as impersonal and
negative, prompting revisions for Phase 2 messages. In Phase 1, AI-generated
messages were rated less helpful than human-written ones, with 66 percent
receiving a helpfulness rating of 3 or higher. However, in Phase 2, the AI
messages matched the human-written ones regarding helpfulness, with 82% scoring
three or above. Additionally, 50% were misidentified as human-written,
suggesting AI's sophistication in mimicking human-generated content. A thematic
analysis of open-ended feedback revealed that participants appreciated AI's
empathy and personalized suggestions but found them more formulaic, less
authentic, and too data-focused. This study reveals the preliminary feasibility
and acceptability of LLM AIs, like ChatGPT, in crafting potentially effective
weight control coaching messages. Our findings also underscore areas for future
enhancement.