From Metrics to Meaning: Time to Rethink Evaluation in Human-AI Collaborative Design
This work addresses the challenge of improving evaluation methods for human-AI systems in creative design, advocating for a holistic approach beyond traditional metrics, though it is incremental in refining existing evaluation frameworks.
The paper tackled the problem of evaluating human-AI collaborative design systems by conducting a large field study (n=808) and a lab study (n=12) using a co-creative car design tool, finding that exposure to AI-generated galleries significantly enhanced engagement and design quality compared to a random control.
As AI systems increasingly shape decision making in creative design contexts, understanding how humans engage with these tools has become a critical challenge for interactive intelligent systems research. This paper contributes a challenge to rethink how to evaluate human--AI collaborative systems, advocating for a more nuanced and multidimensional approach. Findings from one of the largest field studies to date (n = 808) of a human--AI co-creative system, The Genetic Car Designer, complemented by a controlled lab study (n = 12) are presented. The system is based on an interactive evolutionary algorithm where participants were tasked with designing a simple two dimensional representation of a car. Participants were exposed to galleries of design suggestions generated by an intelligent system, MAP--Elites, and a random control. Results indicate that exposure to galleries generated by MAP--Elites significantly enhanced both cognitive and behavioural engagement, leading to higher-quality design outcomes. Crucially for the wider community, the analysis reveals that conventional evaluation methods, which often focus on solely behavioural and design quality metrics, fail to capture the full spectrum of user engagement. By considering the human--AI design process as a changing emotional, behavioural and cognitive state of the designer, we propose evaluating human--AI systems holistically and considering intelligent systems as a core part of the user experience -- not simply a back end tool.