Parachute: Evaluating Interactive Human-LM Co-writing Systems
This addresses the need for better evaluation methods in human-AI interaction research, though it is incremental as it builds on existing evaluation concepts.
The paper tackles the lack of studies assessing interactive human-LM co-writing systems by proposing Parachute, a human-centered evaluation framework that integrates categorized practical metrics, and demonstrates its use with a case study.
A surge of advances in language models (LMs) has led to significant interest in using LMs to build co-writing systems, in which humans and LMs interactively contribute to a shared writing artifact. However, there is a lack of studies assessing co-writing systems in interactive settings. We propose a human-centered evaluation framework, Parachute, for interactive co-writing systems. Parachute showcases an integrative view of interaction evaluation, where each evaluation aspect consists of categorized practical metrics. Furthermore, we present Parachute with a use case to demonstrate how to evaluate and compare co-writing systems using Parachute.