Reviving Reflection-in-Action: Instilling Designerly Thinking in AI-Supported Ideation through Multimodal Prompting
For AI-supported design tools, this work identifies a gap between objective performance gains from sketching and subjective user preference for text, suggesting the need to design for reflection-in-action.
The paper explores how multimodal prompting (text, sketch, sketch-plus-tags) affects design students' ideation with AI tools. Results show sketch modality tends to enhance fluency, but participants prefer text prompting, highlighting a tension between performance and preference.
Current AI-powered creativity support tools (AI-CSTs) primarily use text prompting to generate solution-oriented outputs. However, the potential value of multimodal prompting in designer-AI interaction, specifically the introduction of productive friction to encourage iteration and reflection, has not been fully explored. To address this, we developed SketchifAI, a prototype AI-CST, and evaluated it with design students. In a mixed-methods, within-participants study, we examined how different input modalities (text, sketch, and sketch-plus-tags) affected design students' perceived ability to express their intent, their perception of creativity support, and their divergent thinking performance. Our preliminary findings suggest that the sketch modality tended to enhance fluency, with inconclusive evidence for differences in variety, originality, or quality compared to text modality. Yet, paradoxically, participants showed a strong preference for text prompting. We discuss how AI tools might be designed to reintroduce reflection-through-sketching, ensuring that designer-AI interaction supports, rather than erodes, essential design skills in students.