隨著當今互聯網的廣泛覆蓋及行動設備的使用越來越普遍，導致行動應用程式的開發與設計過程越發重要；在行動應用程式的設計過程中，為了加快UI設計師在搜尋相關應用程式UI設計案例以用來激發靈感，本文提出一種基於草圖檢索行動應用程式UI設計實例的方法，讓UI設計師只輸入根據初步想法隨手描繪出的草圖，即可搜尋到與輸入草圖相似的設計案例。
本研究提出的方法由兩個模型組成，分別是基於CycleGAN的影像到影像轉換模型，以及基於卷積自編碼器的檢索模型。首先，我們使用影像到影像轉換模型，將複雜的UI設計案例截圖轉換為與UI設計草圖相似的結果，稱生成草圖；接著，我們使用檢索模型從輸入的手繪草圖及生成草圖(UI設計案例截圖)中，提取特徵並將其轉換為低維的特徵向量，隨後，我們使用餘弦相似度來計算手繪草圖與生成草圖的特徵向量之間的相似值，且根據此值來排序及輸出與查詢案例最為接近的UI設計實例。
為了能更好的驗證及評估我們系統的性能，我們將另外兩個系統作為基準線的方法比較；根據實驗結果，我們的方法明顯優於其他兩個基準線，其中，系統對於top-1查詢結果的準確率為72.83%，而對於top-5與top-10查詢結果的準確率分別達到85.51%和92.39%。此結果表明，我們提出的基於草圖檢索UI設計案例方法，在行動應用程式開發過程中實現的潛力頗大，因為它有助於UI設計師在搜尋設計案例的過程中，減少搜尋到無關的設計案例的次數，從而節省工作時間並提高工作效率。

With the wide coverage of the Internet and widespread use of mobile devices nowadays, the process of designing and developing mobile applications (apps) has become increasingly important. To accelerate the process of searching and inspiration stimulating for a mobile apps user interface (UI) designer, this thesis presents a method for retrieving mobile apps UI design examples based on sketches. With such a method, the UI designers can search for design examples similar to the input sketch that is just drawn out based on preliminary ideas.
This thesis proposes a sketch-based image retrieval system consisting of two models that are the image-to-image translation model and the image retrieval model. The image-to-image translation model is based on the CycleGAN to transform a screenshot into a sketch. On the other hand, the image retrieval model based on a convolutional autoencoder extracts features from input data, and converts them into lower-dimensional latent vectors. Next, the similarity values between both the latent vectors of hand drawn sketches and generated sketches (screenshots) are calculated using cosine similarity. Afterward, the final query results will be ranked according to the similarity value calculated.
In order to evaluate the performance of our sketch-based image retrieval system, two baselines are implemented to compare with the system. According to the experimental results, our system achieves the precision of 72.83%, 85.51%, and 92.39% on top-1, top-5, and top-10 query outcomes, respectively, which is significantly superior to the two baselines do. Our proposed method can be implemented for retrieving relevant UI design examples from a low-fidelity sketch during the mobile apps UI design process. In consequence, such a method can reduce the working time and improve the work efficiency, because it helps UI designers diminish the number of times to search for irrelevant design examples in the course of developing a mobile apps UI design.

[1] A. Phongtraychack and D. Dolgaya, “Evolution of mobile applications,” in Proceedings of the MATEC Web of Conferences, pp. 1-7, 2018.
[2] A. Swearngin et al., “Rewire: Interface design assistance from examples,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, Quebec, Canada, pp. 1-12, 2018.
[3] S. R. Herring et al., “Getting inspired! Understanding how and why examples are used in creative design practice,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, Masschusetts, pp. 87-96, 2009.
[4] N. Bonnardel, “Creativity in design activities: The role of analogies in a constrained cognitive environment,” in Proceedings of the 3rd Conference on Creativity & Cognition, Loughborough, United Kingdom, pp. 158-165, 1999.
[5] M. Urbieta et al., “Improving mockup-based requirement specification with end-user annotations,” in Proceedings of the International Conference on Agile Software Development, Porto, Portugal, pp. 19-34, 2018.
[6] T. Memmel, C. Bock, and H. Reiterer, “Model-driven prototyping for corporate software specification,” in Proceedings of the IFIP International Conference on Engineering for Human-Computer Interaction, Rio de Janeiro, Brazil, pp. 158-174, 2007.
[7] B. Deka, Z. Huang, and R. Kumar, “ERICA: Interaction mining mobile apps,” in Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan, pp. 767-776, 2016.
[8] Waldo Team, “The world's largest library of mobile user flows,” [Online]. Available: https://uxarchive.com/. [Accessed September 22, 2021].
[9] B. Deka et al., “Rico: A mobile app dataset for building data-driven design applications,” in Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, Quebec, Canada, pp. 845-854, 2017.
[10] F. Huang, J. F. Canny, and J. Nichols, “Swire: Sketch-based user interface retrieval,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, United Kingdom, pp. 1-10, 2019.
[11] J. Canny, “A computational approach to edge detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-8, no. 6, pp. 679-698, 1986. doi:10.1109/TPAMI.1986.4767851
[12] S. Winkler and P. Mohandas, “The evolution of video quality measurement: From PSNR to hybrid metrics,” IEEE Transactions on Broadcasting, vol. 54, no. 3, pp. 660-668, 2008. doi:10.1109/TBC.2008.2000733
[13] Z. Wang et al., “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2004. doi:10.1109/TIP.2003.819861
[14] R. Hu and J. Collomosse, “A performance evaluation of gradient field hog descriptor for sketch based image retrieval,” Computer Vision and Image Understanding, vol. 117, no. 7, pp. 790-806, 2013. doi:10.1016/j.cviu.2013.02.005
[15] M. Eitz, J. Hays, and M. Alexa, “How do humans sketch objects?” ACM Transactions on Graphics, vol. 31, no. 4, pp. 1-10, 2012. doi:10.1145/ 2185520.2185540
[16] P. Sangkloy et al., “The sketchy database: Learning to retrieve badly drawn bunnies,” ACM Transactions on Graphics, vol. 35, no. 4, pp. 1-12, 2016. doi:10.1145/ 2897824.2925954
[17] L. Guo et al., “Sketch-based image retrieval using generative adversarial networks,” in Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, California, pp. 1267-1268, 2017.
[18] G. Pahariya, “Bi-Modal content based image retrieval using multi-class Cycle-GAN,” in Proceedings of the International Conference on Digital Image Computing: Techniques and Applications, Canberra, Australia, pp. 1-7, 2018.
[19] M. Jogin et al., “Feature extraction using convolution neural networks (CNN) and deep learning,” in Proceedings of the 3rd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology, Bangalore, India, pp. 2319-2323, 2018.
[20] Kdnuggets, “An intuitive explanation of convolutional neural networks,” [Online]. Available: https://www.kdnuggets.com/2016/11/intuitive-explanation-convolu-tional-neural-networks.html/3. [Accessed September 25, 2021].

[21] I. Goodfellow et al., “Generative adversarial nets, ” in Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Canada, vol. 2, pp. 2672-2680, 2014.
[22] P. Isola et al., “Image-to-image translation with conditional adversarial networks, ” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaii, pp. 1125-1134, 2017.
[23] J.-Y. Zhu et al., “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, pp. 2223-2232, 2017.
[24] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proceedings of the International Conference on Learning Representations, San Diego, California, arXiv preprint arXiv:1409.1556, 2015.
[25] M. Heusel et al., “GANs trained by a two time-scale update rule converge to a local nash equilibrium,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, California, pp. 6629-6640, 2017.
[26] T. Salimans et al., “Improved techniques for training GANs,” Advances in Neural Information Processing Systems, vol. 29, pp. 2234-2242, 2016.