Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models
It addresses the design of AI-powered writing tools for users, though it is incremental as it builds on existing co-writing research.
This paper investigated how different levels of AI scaffolding affect human-AI co-writing, finding that high scaffolding (next-paragraph suggestions) significantly improved writing quality and productivity, especially for non-regular writers and less tech-savvy users, while low scaffolding had no significant effect.
Advances in language modeling have paved the way for novel human-AI co-writing experiences. This paper explores how varying levels of scaffolding from large language models (LLMs) shape the co-writing process. Employing a within-subjects field experiment with a Latin square design, we asked participants (N=131) to respond to argumentative writing prompts under three randomly sequenced conditions: no AI assistance (control), next-sentence suggestions (low scaffolding), and next-paragraph suggestions (high scaffolding). Our findings reveal a U-shaped impact of scaffolding on writing quality and productivity (words/time). While low scaffolding did not significantly improve writing quality or productivity, high scaffolding led to significant improvements, especially benefiting non-regular writers and less tech-savvy users. No significant cognitive burden was observed while using the scaffolded writing tools, but a moderate decrease in text ownership and satisfaction was noted. Our results have broad implications for the design of AI-powered writing tools, including the need for personalized scaffolding mechanisms.