Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping
This addresses the challenge of making UI/UX design more accessible and iterative for designers by reducing reliance on high-fidelity inputs, though it is incremental in benchmarking existing models.
The authors tackled the problem of automating web design prototyping from sketches by introducing Sketch2Code, a benchmark that evaluates Vision-Language Models (VLMs) on converting sketches into webpage prototypes, finding that even top models struggle with accuracy and effective interaction, but a user study showed a preference for proactive question-asking over passive feedback.
Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas. However, existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. To bridge this gap, we introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. Beyond end-to-end benchmarking, Sketch2Code supports interactive agent evaluation that mimics real-world design workflows, where a VLM-based agent iteratively refines its generations by communicating with a simulated user, either passively receiving feedback instructions or proactively asking clarification questions. We comprehensively analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs; even the most capable models struggle to accurately interpret sketches and formulate effective questions that lead to steady improvement. Nevertheless, a user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception, highlighting the need to develop more effective paradigms for multi-turn conversational agents.