Computational Approaches for App-to-App Retrieval and Design Consistency
  Check
Seokhyeon Park Wonjae Kim Young-Ho Kim Jinwook Seo
Extracting semantic representations from mobile user interfaces (UI) and
using the representations for designers' decision-making processes have shown
the potential to be effective computational design support tools. Current
approaches rely on machine learning models trained on small-sized mobile UI
datasets to extract semantic vectors and use screenshot-to-screenshot
comparison to retrieve similar-looking UIs given query screenshots. However,
the usability of these methods is limited because they are often not
open-sourced and have complex training pipelines for practitioners to follow,
and are unable to perform screenshot set-to-set (i.e., app-to-app) retrieval.
To this end, we (1) employ visual models trained with large web-scale images
and test whether they could extract a UI representation in a zero-shot way and
outperform existing specialized models, and (2) use mathematically founded
methods to enable app-to-app retrieval and design consistency analysis. Our
experiments show that our methods not only improve upon previous retrieval
models but also enable multiple new applications.