{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T10:32:36Z","timestamp":1783074756559,"version":"3.54.6"},"reference-count":54,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T00:00:00Z","timestamp":1615334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Key Laboratory of E&amp;M (Zhejiang University of Technology), Ministry of Education &amp; Zhejiang Province","award":["Grant No. EM 2016070101"],"award-info":[{"award-number":["Grant No. EM 2016070101"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The goal of large-scale automatic paintings analysis is to classify and retrieve images using machine learning techniques. The traditional methods use computer vision techniques on paintings to enable computers to represent the art content. In this work, we propose using a graph convolutional network and artistic comments rather than the painting color to classify type, school, timeframe and author of the paintings by implementing natural language processing (NLP) techniques. First, we build a single artistic comment graph based on co-occurrence relations and document word relations and then train an art graph convolutional network (ArtGCN) on the entire corpus. The nodes, which include the words and documents in the topological graph are initialized using a one-hot representation; then, the embeddings are learned jointly for both words and documents, supervised by the known-class training labels of the paintings. Through extensive experiments on different classification tasks using different input sources, we demonstrate that the proposed methods achieve state-of-art performance. In addition, ArtGCN can learn word and painting embeddings, and we find that they have a major role in describing the labels and retrieval paintings, respectively.<\/jats:p>","DOI":"10.3390\/s21061940","type":"journal-article","created":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T20:51:42Z","timestamp":1615409502000},"page":"1940","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["How to Represent Paintings: A Painting Classification Using Artistic Comments"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0321-7750","authenticated-orcid":false,"given":"Wentao","family":"Zhao","sequence":"first","affiliation":[{"name":"College of Mechanical Engineering, Zhejiang University of Technology, Hangzhou 310023, China"},{"name":"School of Intelligent Transportation, Zhejiang Institute of Mechanical &amp; Electrical Engineering, Hangzhou 310053, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2363-9125","authenticated-orcid":false,"given":"Dalin","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinguo","family":"Qiu","sequence":"additional","affiliation":[{"name":"College of Mechanical Engineering, Zhejiang University of Technology, Hangzhou 310023, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Jiang","sequence":"additional","affiliation":[{"name":"College of Mechanical Engineering, Zhejiang University of Technology, Hangzhou 310023, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"41770","DOI":"10.1109\/ACCESS.2019.2907986","article-title":"Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings","volume":"7","author":"Sandoval","year":"2019","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.eswa.2018.07.026","article-title":"Fine-Tuning Convolutional Neural Networks for Fine Art Classification","volume":"114","author":"Cetinic","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"73694","DOI":"10.1109\/ACCESS.2019.2921101","article-title":"A Deep Learning Perspective on Beauty, Sentiment, and Remembrance of Art","volume":"7","author":"Cetinic","year":"2019","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.patrec.2019.11.008","article-title":"Learning the Principles of Art History with Convolutional Neural Networks","volume":"129","author":"Cetinic","year":"2020","journal-title":"Pattern Recognit. Lett."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Huckle, N., Garcia, N., and Nakashima, Y. (2020). Demographic Influences on Contemporary Art with Unsupervised Style Embeddings. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-030-66096-3_10"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Chen, L., and Yang, J. (2019, January 21\u201325). Recognizing the Style of Visual Arts via Adaptive Cross-Layer Correlation. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.","DOI":"10.1145\/3343031.3350977"},{"key":"ref_7","unstructured":"Wynen, D., Schmid, C., and Mairal, J. (2018). Unsupervised Learning of Artistic Styles with Archetypal Style Analysis. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.eswa.2017.11.056","article-title":"Categorizing Paintings in Art Styles Based on Qualitative Color Descriptors, Quantitative Global Features and Machine Learning (QArt-Learn)","volume":"97","author":"Falomir","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ma, D., Gao, F., Bai, Y., Lou, Y., Wang, S., Huang, T., and Duan, L.Y. (2017, January 3\u20137). From Part to Whole: Who Is behind the Painting?. Proceedings of the 25th ACM International Conference on Multimedia, Sliema, Malta.","DOI":"10.1145\/3123266.3123325"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mao, H., Cheung, M., and She, J. (2017, January 3\u20137). Deepart: Learning Joint Representations of Visual Arts. Proceedings of the 25th ACM International Conference on Multimedia, Sliema, Malta.","DOI":"10.1145\/3123266.3123405"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Tan, W.R., Chan, C.S., Aguirre, H.E., and Tanaka, K. (2016). Ceci n\u2019est Pas Une Pipe: A Deep Convolutional Network for Fine-Art Paintings Classification. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25\u201328 September 2016, IEEE.","DOI":"10.1109\/ICIP.2016.7533051"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Garcia, N., Ye, C., Liu, Z., Hu, Q., Otani, M., Chu, C., Nakashima, Y., and Mitamura, T. (2020). A Dataset and Baselines for Visual Question Answering on Art. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-030-66096-3_8"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sheng, S., and Moens, M.F. (2019, January 21\u201325). Generating Captions for Images of Ancient Artworks. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.","DOI":"10.1145\/3343031.3350972"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Baraldi, L., Cornia, M., Grana, C., and Cucchiara, R. (2018). Aligning Text and Document Illustrations: Towards Visually Explainable Digital Humanities. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 20\u201324 August 2018, IEEE.","DOI":"10.1109\/ICPR.2018.8545064"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1670671.1670672","article-title":"Impressionism, Expressionism, Surrealism: Automated Recognition of Painters and Schools of Art","volume":"7","author":"Shamir","year":"2010","journal-title":"ACM Trans. Appl. Percept."},{"key":"ref_16","unstructured":"Arora, R.S., and Elgammal, A. (2012). Towards Automated Classification of Fine-Art Painting Style: A Comparative Study. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan, 11\u201315 November 2012, IEEE."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Agarwal, S., Karnick, H., Pant, N., and Patel, U. (2015). Genre and Style Based Painting Classification. Proceedings of the 2015 IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 5\u20139 January 2015, IEEE.","DOI":"10.1109\/WACV.2015.84"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009). Imagenet: A Large-Scale Hierarchical Image Database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20\u201325 June 2009, IEEE.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3273022","article-title":"Omniart: A Large-Scale Artistic Benchmark","volume":"14","author":"Strezoski","year":"2018","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl. (TOMM)"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Seguin, B., Striolo, C., and Kaplan, F. (2016). Visual Link Retrieval in a Database of Paintings. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-319-46604-0_52"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2491","DOI":"10.1109\/TMM.2018.2801718","article-title":"Image Style Classification Based on Learnt Deep Correlation Features","volume":"20","author":"Chu","year":"2018","journal-title":"IEEE Trans. Multimed."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. arXiv.","DOI":"10.3115\/v1\/D14-1181"},{"key":"ref_23","unstructured":"Zhang, X., Zhao, J., and LeCun, Y. (2015). Character-Level Convolutional Networks for Text Classification. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Conneau, A., Schwenk, H., Barrault, L., and Lecun, Y. (2016). Very Deep Convolutional Networks for Text Classification. arXiv.","DOI":"10.18653\/v1\/E17-1104"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Tai, K.S., Socher, R., and Manning, C.D. (2015). Improved Semantic Representations from Tree-Structured Long Short-Term Memory Networks. arXiv.","DOI":"10.3115\/v1\/P15-1150"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.jbi.2017.07.006","article-title":"Recurrent Neural Networks for Classifying Relations in Clinical Notes","volume":"72","author":"Luo","year":"2017","journal-title":"J. Biomed. Inform."},{"key":"ref_27","unstructured":"Liu, P., Qiu, X., and Huang, X. (2016). Recurrent Neural Network for Text Classification with Multi-Task Learning. arXiv."},{"key":"ref_28","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2021, February 15). Attention Is All You Need. NIPS. Available online: https:\/\/arxiv.org\/pdf\/1706.03762.pdf."},{"key":"ref_29","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Tu, M., Wang, G., Huang, J., Tang, Y., He, X., and Zhou, B. (2019). Multi-Hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs. arXiv.","DOI":"10.18653\/v1\/P19-1260"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhang, M., Cui, Z., Neumann, M., and Chen, Y. (2018, January 2\u20137). An End-to-End Deep Learning Architecture for Graph Classification. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11782"},{"key":"ref_32","unstructured":"Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec, J. (2018). Hierarchical Graph Representation Learning with Differentiable Pooling. arXiv."},{"key":"ref_33","unstructured":"Cangea, C., Veli\u010dkovi\u0107, P., Jovanovi\u0107, N., Kipf, T., and Li\u00f2, P. (2018). Towards Sparse Hierarchical Graph Classifiers. arXiv."},{"key":"ref_34","unstructured":"Bianchi, F.M., Grattarola, D., Alippi, C., and Livi, L. (2019). Graph Neural Networks with Convolutional Arma Filters. arXiv."},{"key":"ref_35","unstructured":"Kipf, T.N., and Welling, M. (2016). Semi-Supervised Classification with Graph Convolutional Networks. arXiv."},{"key":"ref_36","unstructured":"Yao, L., Mao, C., and Luo, Y. (2018). Graph Convolutional Networks for Text Classification. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"8409","DOI":"10.1609\/aaai.v34i05.6359","article-title":"Tensor Graph Convolutional Networks for Text Classification","volume":"34","author":"Liu","year":"2020","journal-title":"AAAI"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Huang, L., Ma, D., Li, S., Zhang, X., and Wang, H. (2019). Text Level Graph Neural Network for Text Classification. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3\u20137 November 2019, Association for Computational Linguistics.","DOI":"10.18653\/v1\/D19-1345"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Garcia, N., and Vogiatzis, G. (2018, January 8\u201314). How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-11012-3_52"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1385","DOI":"10.1007\/s00138-014-0621-6","article-title":"Painting-91: A Large Scale Database for Computational Painting Categorization","volume":"25","author":"Khan","year":"2014","journal-title":"Mach. Vis. Appl."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Karayev, S., Trentacoste, M., Han, H., Agarwala, A., Darrell, T., Hertzmann, A., and Winnemoeller, H. (2013). Recognizing Image Style. arXiv.","DOI":"10.5244\/C.28.122"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1016\/j.eswa.2019.05.036","article-title":"Multitask Painting Categorization by Deep Multibranch Neural Network","volume":"135","author":"Bianco","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_43","unstructured":"Zhou, K., Dong, Y., Lee, W.S., Hooi, B., Xu, H., and Feng, J. (2020). Effective Training Strategies for Deep Graph Neural Networks. arXiv."},{"key":"ref_44","unstructured":"Zhao, L., and Akoglu, L. (2019). Pairnorm: Tackling Oversmoothing in Gnns. arXiv."},{"key":"ref_45","unstructured":"Rong, Y., Huang, W., Xu, T., and Huang, J. (2019). Dropedge: Towards Deep Graph Convolutional Networks on Node Classification. arXiv."},{"key":"ref_46","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Garcia, N., Renoust, B., and Nakashima, Y. (2019, January 10\u201313). Context-Aware Embeddings for Automatic Art Analysis. Proceedings of the 2019 on International Conference on Multimedia Retrieval, Ottawa, ON, Canada.","DOI":"10.1145\/3323873.3325028"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017, January 3\u20137). Bag of Tricks for Efficient Text Classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain.","DOI":"10.18653\/v1\/E17-2068"},{"key":"ref_50","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv."},{"key":"ref_51","first-page":"2579","article-title":"Visualizing Data Using T-SNE","volume":"9","author":"Hinton","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_52","unstructured":"Hamilton, W.L., Ying, R., and Leskovec, J. (2017, January 4\u20139). Inductive Representation Learning on Large Graphs. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_53","unstructured":"Chen, J., Ma, T., and Xiao, C. (2018). FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling. arXiv."},{"key":"ref_54","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/1940\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:33:21Z","timestamp":1760160801000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/1940"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,10]]},"references-count":54,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["s21061940"],"URL":"https:\/\/doi.org\/10.3390\/s21061940","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,10]]}}}