Automatically Generating Codes from Graphical Screenshots Based on Deep Autocoder
This addresses a tedious task for front-end developers, but it is incremental as it builds on prior attempts by improving accuracy with attention mechanisms.
The paper tackles the problem of automatically converting GUI images to front-end code in software development, proposing PixCoder with an artificially supervised attention mechanism, achieving over 95% accuracy in generated code.
During software front-end development, the work to convert Graphical User Interface(GUI) image to the corresponding front-end code is an inevitable tedious work. There have been some attempts to make this work to be automatic. However, the GUI code generated by these models is not accurate due to the lack of attention mechanism guidance. To solve this problem, we propose PixCoder based on an artificially supervised attention mechanism. The approach is to train a neural network to predict the style sheets in the input GUI image and then output a vector. PixCoder generate the GUI code targeting specific platform according to the output vector. The experimental results have shown the accuracy of the GUI code generated by PixCoder is over 95%.