The transformation of the graphical user interface created by the designer into the code for a specific platform is the responsibility of software engineers. Recent progress in machine learning made it possible to automate this transformation by leveraging machine learning methods. Traditional computer vision methods represent one of the ways to tackle code generation tasks. However, these methods rely on the use of absolute layout in the final compilation stage of code generation. Another way to tackle this task is to use the encoder-decoder model inspired by the image captioning task. Our proposed model uses encoder-decoder with an attention mechanism that is encouraged to focuses on a different subset of salient image features every time-step. Our model outperforms previously proposed models in the task of automatic code generation from graphical input, on pix2code benchmark dataset.

[1] T. Beltramelli, "pix2code: Generating code from a graphical user interface screenshot," CoRR, vol. abs/1705.07962, 2017.

[2] Z. Zhu, Z. Xue, and Z. Yuan, "Automatic graphics program generation using attention-based hierarchical decoder," CoRR, vol. abs/1810.11536, 2018.

[3] R. B. Girshick, "Fast R-CNN," CoRR, vol. abs/1504.08083, 2015.

[4] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, "Mask R-CNN," CoRR, vol, abs/1703.06870, 2017.

[5] J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," CoRR, vol. abs/1506.02640, 2015.

[6] K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio, "Show, attend and tell: Neural image caption generation with visual attention," CoRR, vol. abs/1502.03044, 2015.

[7] L. Zhou, C. Xu, P. A. Koch, and J. J. Corso, "Image caption generation with text-conditional semantic attention," CoRR, vol. abs/1606.04621, 2016.

[8] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, "Self-critical sequence training for image captioning," CoRR, vol. abs/1612.00563, 2016.

[9] Z. Gan, C. Gan, X. He, Y. Pu, K. Tran, J. Gao, L. Carin, and L. Deng, "Semantic compositional networks for visual captioning," CoRR, vol. abs/1611.08002, 2016.

[10] Y.Liu, Q. Hu, and K. Shu, "Improving pix2code based bi-directional lstm," 2018 IEEE International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), p. 220-223, 2018.

[11] D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," 2014.

[12] J.Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, "Long-term recurrent convolutional networks for visual recognition and description," CoRR, vol. abs/1411.4389, 2014.

[13] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image caption generator," CoRR, vol. abs/1411.4555, 2014.

[14] S. Hochreiter and J, Schmidhuber, "Long short-term memory," Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.

[15] L.J.Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," CoRR, vol, abs/1607.06450, 2016.

[16] K. Cho, B. van Merrienboer, Ç. Gülçehre, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using RNN encoder-decoder for statistical machine translation," CoRR, vol, abs/1406.1078, 2014.

[17] M.Luong, H. Pham, and C. D. Manning, "Effective approaches to attention-based neural machine translation," CoRR, vol. abs/1508.04025, 2015.

[18] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in International Conference on Learning and Representations, 2015.

[19] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M.S.Bernstein, A. C. Berg, and F. Li, "Imagenet large scale visual recognition challenge," CoRR, vol. abs/1409.0575, 2014.

[20] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," Journal of Machine Learning Research, vol. 15, pp. 1929-1958, 2014.