{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T23:39:01Z","timestamp":1790465941365,"version":"4.1.0"},"reference-count":50,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2018,1,11]],"date-time":"2018-01-11T00:00:00Z","timestamp":1515628800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["41501485"],"award-info":[{"award-number":["41501485"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61331017"],"award-info":[{"award-number":["61331017"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Recently, inspired by the power of deep learning, convolution neural networks can produce fantastic images at the pixel level. However, a significant limiting factor for previous approaches is that they focus on some simple datasets such as faces and bedrooms. In this paper, we propose a multiscale deep neural network to transform sketches into Chinese paintings. To synthesize more realistic imagery, we train the generative network by using both L1 loss and adversarial loss. Additionally, users can control the process of the synthesis since the generative network is feed-forward. This network can also be treated as neural style transfer by adding an edge detector. Furthermore, additional experiments on image colorization and image super-resolution demonstrate the universality of our proposed approach.<\/jats:p>","DOI":"10.3390\/a11010004","type":"journal-article","created":{"date-parts":[[2018,1,11]],"date-time":"2018-01-11T13:36:15Z","timestamp":1515677775000},"page":"4","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Transform a Simple Sketch to a Chinese Painting by a Multiscale Deep Neural Network"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8352-0081","authenticated-orcid":false,"given":"Daoyu","family":"Lin","sequence":"first","affiliation":[{"name":"Department of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China"},{"name":"The Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, Institute of Electronics, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Wang","sequence":"additional","affiliation":[{"name":"The Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, Institute of Electronics, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guangluan","family":"Xu","sequence":"additional","affiliation":[{"name":"The Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, Institute of Electronics, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jun","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China"},{"name":"The Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, Institute of Electronics, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kun","family":"Fu","sequence":"additional","affiliation":[{"name":"Department of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China"},{"name":"The Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, Institute of Electronics, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2018,1,11]]},"reference":[{"key":"ref_1","unstructured":"Jing, Y., Yang, Y., Feng, Z., Ye, J., and Song, M. (arXiv, 2017). Neural Style Transfer: A Review, arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gatys, L.A., Ecker, A.S., and Bethge, M. (arXiv, 2015). A neural algorithm of artistic style, arXiv.","DOI":"10.1167\/16.12.326"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Gatys, L.A., Ecker, A.S., and Bethge, M. (2016, January 27\u201330). Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.265"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Shih, Y., Paris, S., Barnes, C., Freeman, W.T., and Durand, F. (2014). Style Transfer for Headshot Portraits, Association for Computing Machinery (ACM).","DOI":"10.1145\/2601097.2601137"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Efros, A.A., and Freeman, W.T. (2001, January 12\u201317). Image quilting for texture synthesis and transfer. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA.","DOI":"10.1145\/383259.383296"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wei, L.Y., and Levoy, M. (2000, January 23\u201328). Fast texture synthesis using tree-structured vector quantization. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA.","DOI":"10.1145\/344779.345009"},{"key":"ref_7","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, Proceedings of the Neural Information Processing Systems, Lake Tahoe, Nevada, 3\u20136 December 2012, Curran Associates, Inc."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_9","unstructured":"Radford, A., Metz, L., and Chintala, S. (arXiv, 2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Johnson, J., Alahi, A., and Fei-Fei, L. (2016, January 8\u201316). Perceptual losses for real-time style transfer and super-resolution. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46475-6_43"},{"key":"ref_11","unstructured":"Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (arXiv, 2014). Generative Adversarial Networks, arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Sangkloy, P., Lu, J., Fang, C., Yu, F., and Hays, J. (arXiv, 2016). Scribbler: Controlling Deep Image Synthesis with Sketch and Color, arXiv.","DOI":"10.1109\/CVPR.2017.723"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"G\u00fc\u00e7l\u00fct\u00fcrk, Y., G\u00fc\u00e7l\u00fc, U., van Lier, R., and van Gerven, M.A. (2016, January 8\u201316). Convolutional sketch inversion. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46604-0_56"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1109\/MCG.2011.67","article-title":"Photosketcher: Interactive sketch-based image synthesis","volume":"31","author":"Eitz","year":"2011","journal-title":"IEEE Comput. Graph. Appl."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Chen, T., Cheng, M.M., Tan, P., Shamir, A., and Hu, S.M. (2009). Sketch2photo: Internet image montage. ACM Trans. Graph., 28.","DOI":"10.1145\/1661412.1618470"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Shih, Y., Paris, S., Durand, F., and Freeman, W.T. (2013). Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph., 32.","DOI":"10.1145\/2508363.2508419"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1145\/882262.882264","article-title":"Graphcut textures: Image and video synthesis using graph cuts","volume":"22","author":"Kwatra","year":"2003","journal-title":"ACM Trans. Graph."},{"key":"ref_19","unstructured":"Kingma, D.P., and Welling, M. (arXiv, 2013). Auto-encoding variational bayes, arXiv."},{"key":"ref_20","unstructured":"Rezende, D.J., Mohamed, S., and Wierstra, D. (2014, January 21\u201326). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. Proceedings of the 31st International Conference on Machine Learning, Beijing, China."},{"key":"ref_21","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, Proceedings of the Neural Information Processing Systems, Montreal, QC, Canada, 8\u201313 December 2014, Curran Associates, Inc."},{"key":"ref_22","unstructured":"Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., and Graves, A. (2016). Conditional image generation with pixelcnn decoders. Advances in Neural Information Processing Systems, Proceedings of the Neural Information Processing Systems, Barcelona, Spain, 5\u201310 December 2016, Curran Associates, Inc."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., and Salesin, D.H. (2001, January 12\u201317). Image analogies. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA.","DOI":"10.1145\/383259.383295"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Yan, X., Yang, J., Sohn, K., and Lee, H. (2016, January 8\u201316). Attribute2image: Conditional image generation from visual attributes. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46493-0_47"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"110:1","DOI":"10.1145\/2897824.2925974","article-title":"Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification","volume":"35","author":"Iizuka","year":"2016","journal-title":"ACM Trans. Graph."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_27","unstructured":"Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved Techniques for Training GANs. Advances in Neural Information Processing Systems, Proceedings of the Neural Information Processing Systems, Barcelona, Spain, 5\u201310 December 2016, Curran Associates, Inc."},{"key":"ref_28","unstructured":"Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (arXiv, 2016). InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets, arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Dixon, D., Prasad, M., and Hammond, T. (2010, January 7\u201311). iCanDraw: Using sketch recognition and corrective feedback to assist a user in drawing human faces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Denver, CO, USA.","DOI":"10.1145\/1753326.1753459"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (arXiv, 2016). Image-to-Image Translation with Conditional Adversarial Networks, arXiv.","DOI":"10.1109\/CVPR.2017.632"},{"key":"ref_31","unstructured":"Mirza, M., and Osindero, S. (arXiv, 2014). Conditional generative adversarial nets, arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., Krishnan, D., Taylor, G.W., and Fergus, R. (2010, January 13\u201318). Deconvolutional networks. Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA.","DOI":"10.1109\/CVPR.2010.5539957"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Odena, A., Dumoulin, V., and Olah, C. (2017, July 15). Deconvolution and Checkerboard Artifacts. Available online: http:\/\/distill.pub\/2016\/deconv-checkerboard\/.","DOI":"10.23915\/distill.00003"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Xie, S., and Tu, Z. (2015, January 7\u201313). Holistically-nested edge detection. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.164"},{"key":"ref_36","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (arXiv, 2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems, arXiv."},{"key":"ref_37","unstructured":"Ioffe, S., and Szegedy, C. (2015, January 6\u201311). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the 32nd International Conference on Machine Learning, Lille, France."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Li, C., and Wand, M. (2016, January 27\u201330). Combining markov random fields and convolutional neural networks for image synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.272"},{"key":"ref_39","unstructured":"Ulyanov, D., Lebedev, V., Vedaldi, A., and Lempitsky, V. (2016, January 20\u201322). Texture networks: Feed-forward synthesis of textures and stylized images. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Yang, Y., and Newsam, S. (2010, January 2\u20135). Bag-of-visual-words and spatial extensions for land-use classification. Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, San Jose, CA, USA.","DOI":"10.1145\/1869790.1869829"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zhang, R., Zhu, J.Y., Isola, P., Geng, X., Lin, A.S., Yu, T., and Efros, A.A. (2017). Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Trans. Graph., 36.","DOI":"10.1145\/3072959.3073703"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2861","DOI":"10.1109\/TIP.2010.2050625","article-title":"Image super-resolution via sparse representation","volume":"19","author":"Yang","year":"2010","journal-title":"IEEE Trans. Image Process."},{"key":"ref_43","unstructured":"Martin, D., Fowlkes, C., Tal, D., and Malik, J. (2001, January 7\u201314). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proceedings of the Eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada."},{"key":"ref_44","unstructured":"Chang, H., Yeung, D.Y., and Xiong, Y. (July, January 27). Super-resolution through neighbor embedding. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Glasner, D., Bagon, S., and Irani, M. (October, January 29). Super-resolution from a single image. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.","DOI":"10.1109\/ICCV.2009.5459271"},{"key":"ref_46","unstructured":"Zeyde, R., Elad, M., and Protter, M. (2010, January 24\u201330). On single image scale-up using sparse-representations. Proceedings of the International Conference on Curves and Surfaces, Avignon, France."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Bevilacqua, M., Roumy, A., Guillemot, C., and Morel, A. (2012, January 3\u20137). Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. Proceedings of the 23rd British Machine Vision Conference (BMVC), Guildford, UK.","DOI":"10.5244\/C.26.135"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Timofte, R., De Smet, V., and Van Gool, L. (2014, January 1\u20135). A+: Adjusted anchored neighborhood regression for fast super-resolution. Proceedings of the Asian Conference on Computer Vision, Singapore.","DOI":"10.1109\/ICCV.2013.241"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Huang, J.B., Singh, A., and Ahuja, N. (2015, January 7\u201312). Single image super-resolution from transformed self-exemplars. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299156"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Dong, C., Loy, C.C., He, K., and Tang, X. (2014, January 6\u201312). Learning a deep convolutional network for image super-resolution. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10593-2_13"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/11\/1\/4\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:50:54Z","timestamp":1760194254000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/11\/1\/4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,1,11]]},"references-count":50,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2018,1]]}},"alternative-id":["a11010004"],"URL":"https:\/\/doi.org\/10.3390\/a11010004","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,1,11]]}}}