Reverse Browser: Vector-Image-to-Code Generator
This work addresses the challenge of low fidelity in automated UI code generation for software engineers, though it appears incremental by focusing on input type and metrics.
The paper tackles the problem of converting user interface designs into code by using vector images as input instead of bitmaps, resulting in a new model and metric for improved fidelity.
Automating the conversion of user interface design into code (image-to-code or image-to-UI) is an active area of software engineering research. However, the state-of-the-art solutions do not achieve high fidelity to the original design, as evidenced by benchmarks. In this work, I approach the problem differently: I use vector images instead of bitmaps as model input. I create several large datasets for training machine learning models. I evaluate the available array of Image Quality Assessment (IQA) algorithms and introduce a new, multi-scale metric. I then train a large open-weights model and discuss its limitations.