Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing
  with Language Models
Paramveer S. Dhillon Somayeh Molaei Jiaqi Li Maximilian Golub Shaochun Zheng Lionel P. Robert
Advances in language modeling have paved the way for novel human-AI
co-writing experiences. This paper explores how varying levels of scaffolding
from large language models (LLMs) shape the co-writing process. Employing a
within-subjects field experiment with a Latin square design, we asked
participants (N=131) to respond to argumentative writing prompts under three
randomly sequenced conditions: no AI assistance (control), next-sentence
suggestions (low scaffolding), and next-paragraph suggestions (high
scaffolding). Our findings reveal a U-shaped impact of scaffolding on writing
quality and productivity (words/time). While low scaffolding did not
significantly improve writing quality or productivity, high scaffolding led to
significant improvements, especially benefiting non-regular writers and less
tech-savvy users. No significant cognitive burden was observed while using the
scaffolded writing tools, but a moderate decrease in text ownership and
satisfaction was noted. Our results have broad implications for the design of
AI-powered writing tools, including the need for personalized scaffolding
mechanisms.