ULDGNN: A Fragmented UI Layer Detector Based on Graph Neural Networks
This work addresses a domain-specific issue for developers using UI design software like Sketch to improve front-end code generation, but it is incremental as it builds on existing methods for layer detection and merging.
The paper tackles the problem of fragmented layers in UI design drafts degrading code generation quality by proposing a pipeline that automatically merges these layers using graph neural networks and rule-based algorithms, achieving 87% accuracy in detection on a new dataset.
While some work attempt to generate front-end code intelligently from UI screenshots, it may be more convenient to utilize UI design drafts in Sketch which is a popular UI design software, because we can access multimodal UI information directly such as layers type, position, size, and visual images. However, fragmented layers could degrade the code quality without being merged into a whole part if all of them are involved in the code generation. In this paper, we propose a pipeline to merge fragmented layers automatically. We first construct a graph representation for the layer tree of a UI draft and detect all fragmented layers based on the visual features and graph neural networks. Then a rule-based algorithm is designed to merge fragmented layers. Through experiments on a newly constructed dataset, our approach can retrieve most fragmented layers in UI design drafts, and achieve 87% accuracy in the detection task, and the post-processing algorithm is developed to cluster associative layers under simple and general circumstances.