Computational Approaches for App-to-App Retrieval and Design Consistency Check
This addresses usability limitations for designers by providing open-source, simpler methods for computational design support, though it is incremental in leveraging existing models for a new application.
The paper tackled the problem of retrieving similar mobile user interfaces (UIs) and checking design consistency by proposing a zero-shot approach using visual models trained on web-scale images and enabling app-to-app retrieval, resulting in improved performance over existing specialized models.
Extracting semantic representations from mobile user interfaces (UI) and using the representations for designers' decision-making processes have shown the potential to be effective computational design support tools. Current approaches rely on machine learning models trained on small-sized mobile UI datasets to extract semantic vectors and use screenshot-to-screenshot comparison to retrieve similar-looking UIs given query screenshots. However, the usability of these methods is limited because they are often not open-sourced and have complex training pipelines for practitioners to follow, and are unable to perform screenshot set-to-set (i.e., app-to-app) retrieval. To this end, we (1) employ visual models trained with large web-scale images and test whether they could extract a UI representation in a zero-shot way and outperform existing specialized models, and (2) use mathematically founded methods to enable app-to-app retrieval and design consistency analysis. Our experiments show that our methods not only improve upon previous retrieval models but also enable multiple new applications.