Beyond Semantic Similarity: Open Challenges for Embedding-Based Creative Process Analysis Across AI Design Tools
This paper identifies open challenges for researchers and developers in accurately analyzing creative processes across different AI design tools, particularly when using embedding-based methods.
The paper argues that fixed embedding similarity, commonly used in AI-based creativity support tools (CSTs), can misrepresent creative dynamics by failing to detect creative pivots within superficially similar language. This limitation treats shifts in the problem being addressed as continued elaboration, hindering cross-domain comparison of creative processes.
AI-based creativity support tools (CSTs) are evaluated through domain-specific metrics, limiting cross-domain comparison of creative processes. Embedding-based protocol analysis offers a potential domain-agnostic analytical layer. However, we argue that fixed embedding similarity can misrepresent creative dynamics: it may not detect creative pivots that occur within superficially similar language, treating shifts in the problem being addressed as continued elaboration. We identify three open challenges stemming from this gap: aligning similarity measures with creative significance, segmenting and representing multimodal design traces, and evaluating agentic systems where embedding-based metrics enter the generation loop and shape agent behavior. We propose context-aware interventions using large language models as a direction for making trace analysis sensitive to session-specific creative dynamics.