{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T05:45:46Z","timestamp":1771307146439,"version":"3.50.1"},"reference-count":61,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,12,21]],"date-time":"2019-12-21T00:00:00Z","timestamp":1576886400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,12,21]],"date-time":"2019-12-21T00:00:00Z","timestamp":1576886400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["18H03264"],"award-info":[{"award-number":["18H03264"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Multimed Info Retr"],"published-print":{"date-parts":[[2020,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In automatic art analysis, models that besides the visual elements of an artwork represent the relationships between the different artistic attributes could be very informative. Those kinds of relationships, however, usually appear in a very subtle way, being extremely difficult to detect with standard convolutional neural networks. In this work, we propose to capture contextual artistic information from fine-art paintings with a specific ContextNet network. As context can be obtained from multiple sources, we explore two modalities of ContextNets: one based on multitask learning and another one based on knowledge graphs. Once the contextual information is obtained, we use it to enhance visual representations computed with a neural network. In this way, we are able to (1) capture information about the content and the style with the visual representations and (2) encode relationships between different artistic attributes with the ContextNet. We evaluate our models on both painting classification and retrieval, and by visualising the resulting embeddings on a knowledge graph, we can confirm that our models represent specific stylistic aspects present in the data.<\/jats:p>","DOI":"10.1007\/s13735-019-00189-4","type":"journal-article","created":{"date-parts":[[2020,2,3]],"date-time":"2020-02-03T14:18:14Z","timestamp":1580739494000},"page":"17-30","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":35,"title":["ContextNet: representation and exploration for painting classification and retrieval in context"],"prefix":"10.1007","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9200-6359","authenticated-orcid":false,"given":"Noa","family":"Garcia","sequence":"first","affiliation":[]},{"given":"Benjamin","family":"Renoust","sequence":"additional","affiliation":[]},{"given":"Yuta","family":"Nakashima","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,12,21]]},"reference":[{"key":"189_CR1","first-page":"1","volume-title":"Encyclopedia of social network analysis and mining","author":"D Auber","year":"2018","unstructured":"Auber D, Archambault D, Bourqui R, Delest M, Dubois J, Lambert A, Mary P, Mathiaut M, M\u00e9lan\u00e7on G, Pinaud B, Renoust B, Vallet J (2018) Tulip 5. In: Alhajj R, Rokne J (eds) Encyclopedia of social network analysis and mining. Springer, New York, pp 1\u201328"},{"key":"189_CR2","first-page":"71","volume-title":"European conference on computer vision workshops","author":"Y Bar","year":"2014","unstructured":"Bar Y, Levy N, Wolf L (2014) Classification of artistic styles using binarized features derived from a deep neural network. In: Agapito L, Bronstein M, Rother C (eds) European conference on computer vision workshops. Springer, Cham, pp 71\u201384"},{"key":"189_CR3","unstructured":"Bilen H, Vedaldi A (2016) Integrated perception with recurrent multi-task neural networks. In: Advances in neural information processing systems, p 235\u2013243"},{"key":"189_CR4","doi-asserted-by":"crossref","unstructured":"Carlson A, Betteridge J, Kisiel B, Settles B, Hruschka\u00a0Jr, E.R, Mitchell T.M (2010) Toward an architecture for never-ending language learning. In: AAAI, vol\u00a05. Atlanta, p 3","DOI":"10.1609\/aaai.v24i1.7519"},{"key":"189_CR5","doi-asserted-by":"crossref","unstructured":"Carneiro G, da\u00a0Silva NP, Del\u00a0Bue A, Costeira JP (2012) Artistic image classification: an analysis on the printart database. In: European conference on computer vision, pp 143\u2013157","DOI":"10.1007\/978-3-642-33765-9_11"},{"issue":"1","key":"189_CR6","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1023\/A:1007379606734","volume":"28","author":"R Caruana","year":"1997","unstructured":"Caruana R (1997) Multitask learning. Mach Learn 28(1):41\u201375","journal-title":"Mach Learn"},{"key":"189_CR7","doi-asserted-by":"crossref","unstructured":"Chen X, Shrivastava A, Gupta A (2013) Neil: extracting visual knowledge from web data. In: Proceedings of the IEEE international conference on computer vision, pp 1409\u20131416","DOI":"10.1109\/ICCV.2013.178"},{"issue":"9","key":"189_CR8","doi-asserted-by":"publisher","first-page":"2491","DOI":"10.1109\/TMM.2018.2801718","volume":"20","author":"WT Chu","year":"2018","unstructured":"Chu WT, Wu YL (2018) Image style classification based on learnt deep correlation features. IEEE Trans Multimed 20(9):2491\u20132502","journal-title":"IEEE Trans Multimed"},{"key":"189_CR9","doi-asserted-by":"crossref","unstructured":"Collomosse J, Bui T, Wilber M.J, Fang C, Jin H (2017) Sketching with style: Visual search with sketches and aesthetic context. In: Proceedings of the IEEE international conference on computer vision, pp 2679\u20132687","DOI":"10.1109\/ICCV.2017.290"},{"key":"189_CR10","doi-asserted-by":"crossref","unstructured":"Crowley E, Zisserman A (2014) The state of the art: object retrieval in paintings using discriminative regions. In: Proceedings of the British machine vision conference. BMVA Press","DOI":"10.5244\/C.28.38"},{"key":"189_CR11","doi-asserted-by":"crossref","unstructured":"Crowley EJ, Parkhi OM, Zisserman A (2015) Face painting: querying art with photos. In: BMVC, pp 65\u20131","DOI":"10.5244\/C.29.65"},{"key":"189_CR12","unstructured":"Crowley E.J, Zisserman A (2016) The art of detection. In: European conference on computer vision. Springer, pp 721\u2013737"},{"issue":"1","key":"189_CR13","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1109\/TMM.2017.2724843","volume":"20","author":"P Cui","year":"2018","unstructured":"Cui P, Liu S, Zhu W (2018) General knowledge embedded image representation learning. IEEE Trans Multimed 20(1):198\u2013207","journal-title":"IEEE Trans Multimed"},{"key":"189_CR14","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1109\/TPAMI.1979.4766909","volume":"2","author":"DL Davies","year":"1979","unstructured":"Davies DL, Bouldin DW (1979) A cluster separation measure. IEEE Trans Pattern Anal Mach Intell 2:224\u2013227","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"189_CR15","doi-asserted-by":"crossref","unstructured":"Deng J, Ding N, Jia Y, Frome A, Murphy K, Bengio S, Li, Y, Neven H, Adam H (2014) Large-scale object classification using label relation graphs. In: European conference on computer vision. Springer, pp 48\u201364","DOI":"10.1007\/978-3-319-10590-1_4"},{"key":"189_CR16","unstructured":"Dwyer T, Marriott K, Stuckey P.J (2005) Fast node overlap removal. In: International symposium on graph drawing. Springer, pp 153\u2013164"},{"key":"189_CR17","doi-asserted-by":"crossref","unstructured":"Fergus R, Bernal H, Weiss Y, Torralba A (2010) Semantic label sharing for learning with many categories. In: European conference on computer vision. Springer, pp 762\u2013775","DOI":"10.1007\/978-3-642-15549-9_55"},{"key":"189_CR18","doi-asserted-by":"crossref","unstructured":"Garcia N, Renoust B, Nakashima Y (2019) Context-aware embeddings for automatic art analysis. In: Proceedings of the 2019 on international conference on multimedia retrieval. ACM, pp 25\u201333","DOI":"10.1145\/3323873.3325028"},{"key":"189_CR19","unstructured":"Garcia N, Renoust B, Nakashima Y (2019) Understanding art through multi-modal retrieval in paintings. arXiv preprint arXiv:1904.10615"},{"key":"189_CR20","doi-asserted-by":"crossref","unstructured":"Garcia N, Vogiatzis G (2018) How to read paintings: semantic art understanding with multi-modal retrieval. In: Proceedings of the European conference in computer vision workshops","DOI":"10.1007\/978-3-030-11012-3_52"},{"key":"189_CR21","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1016\/j.knosys.2018.03.022","volume":"151","author":"P Goyal","year":"2018","unstructured":"Goyal P, Ferrara E (2018) Graph embedding techniques, applications, and performance: a survey. Knowl Based Syst 151:78\u201394","journal-title":"Knowl Based Syst"},{"key":"189_CR22","doi-asserted-by":"crossref","unstructured":"Grover A, Leskovec J (2016) node2vec: scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 855\u2013864","DOI":"10.1145\/2939672.2939754"},{"key":"189_CR23","unstructured":"Hachul S, J\u00fcnger M (2004) Drawing large graphs with a potential-field-based multilevel algorithm. In: International symposium on graph drawing. Springer, pp 285\u2013295"},{"key":"189_CR24","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition","DOI":"10.1109\/CVPR.2016.90"},{"issue":"4","key":"189_CR25","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1109\/MSP.2008.923513","volume":"25","author":"CR Johnson","year":"2008","unstructured":"Johnson CR, Hendriks E, Berezhnoy IJ, Brevdo E, Hughes SM, Daubechies I, Li J, Postma E, Wang JZ (2008) Image processing for artist identification. IEEE Signal Process Mag 25(4):37\u201348","journal-title":"IEEE Signal Process Mag"},{"key":"189_CR26","doi-asserted-by":"crossref","unstructured":"Johnson J, Krishna R, Stark M, Li L.J, Shamma D, Bernstein M, Fei-Fei L (2015) Image retrieval using scene graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3668\u20133678","DOI":"10.1109\/CVPR.2015.7298990"},{"key":"189_CR27","doi-asserted-by":"crossref","unstructured":"Karayev S, Trentacoste M, Han H, Agarwala A, Darrell T, Hertzmann A, Winnemoeller H (2014) Recognizing image style. In: Proceedings of the British machine vision conference. BMVA Press","DOI":"10.5244\/C.28.122"},{"key":"189_CR28","doi-asserted-by":"crossref","unstructured":"Khan FS, Beigpour S, Van\u00a0de Weijer J, Felsberg M (2014) Painting-91: a large scale database for computational painting categorization. In: Machine vision and applications","DOI":"10.1007\/s00138-014-0621-6"},{"key":"189_CR29","unstructured":"Krishna R, Zhu Y, Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y, Li LJ, Shamma DA, Bernstein M, Fei-Fei L (2016) Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv:1602.07332"},{"key":"189_CR30","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097\u20131105"},{"issue":"3","key":"189_CR31","doi-asserted-by":"publisher","first-page":"853","DOI":"10.1111\/j.1467-8659.2009.01700.x","volume":"29","author":"A Lambert","year":"2010","unstructured":"Lambert A, Bourqui R, Auber D (2010) Winding roads: routing edges into bundles. Comput Graph Forum 29(3):853\u2013862","journal-title":"Comput Graph Forum"},{"key":"189_CR32","unstructured":"Long M, Wang J (2015) Learning multiple tasks with deep relationship networks, vol 3. CoRR, arXiv:abs\/1506.02117"},{"issue":"2","key":"189_CR33","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","volume":"60","author":"DG Lowe","year":"2004","unstructured":"Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91\u2013110","journal-title":"Int J Comput Vis"},{"key":"189_CR34","doi-asserted-by":"crossref","unstructured":"Ma D, Gao F, Bai Y, Lou Y, Wang S, Huang T, Duan LY (2017) From part to whole: who is behind the painting? In: Proceedings of the 2017 ACM on multimedia conference. ACM","DOI":"10.1145\/3123266.3123325"},{"key":"189_CR35","doi-asserted-by":"crossref","unstructured":"Mao H, Cheung M, She J (2017) Deepart: learning joint representations of visual arts. In: ACM on multimedia conference","DOI":"10.1145\/3123266.3123405"},{"key":"189_CR36","doi-asserted-by":"crossref","unstructured":"Marino K, Salakhutdinov R, Gupta A (2017) The more you know: Using knowledge graphs for image classification. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 20\u201328","DOI":"10.1109\/CVPR.2017.10"},{"key":"189_CR37","doi-asserted-by":"crossref","unstructured":"Mensink T, Van\u00a0Gemert J (2014) The rijksmuseum challenge: Museum-centered visual recognition. In: Proceedings of international conference on multimedia retrieval. ACM","DOI":"10.1145\/2578726.2578791"},{"issue":"11","key":"189_CR38","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/219717.219748","volume":"38","author":"GA Miller","year":"1995","unstructured":"Miller GA (1995) Wordnet: a lexical database for English. Commun ACM 38(11):39\u201341","journal-title":"Commun ACM"},{"key":"189_CR39","doi-asserted-by":"crossref","unstructured":"Renoust B, Oliveira\u00a0Franca M, Chan J, Garcia N, Le V, Uesaka A, Nakashima Y, Nagahara H, Wang J, Fujioka Y (2019) Historical and modern features for Buddha statue classification. In: Proceedings of 2019 ACM multimedia conference, SUMAC workshop. Association for Computing Machinery (ACM), pp 1\u20138","DOI":"10.1145\/3347317.3357239"},{"key":"189_CR40","unstructured":"Renoust B, Oliveira\u00a0Franca M, Chan J, Le V, Uesaka A, Nakashima Y, Nagahara H, Wang J, Fujioka Y (2019) Buda.art: a multimodal content-based analysis and retrieval system for Buddha statues. In: Proceedings of 2019 ACM multimedia conference. Association for Computing Machinery (ACM), pp 1\u20133"},{"key":"189_CR41","doi-asserted-by":"crossref","unstructured":"Rudd EM, G\u00fcnther M, Boult TE (2016) Moon: a mixed objective optimization network for the recognition of facial attributes. In: European conference on computer vision. Springer, pp 19\u201335","DOI":"10.1007\/978-3-319-46454-1_2"},{"key":"189_CR42","unstructured":"Ruder S (2017) An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098"},{"key":"189_CR43","doi-asserted-by":"crossref","unstructured":"Salakhutdinov R, Torralba A, Tenenbaum J (2011) Learning to share visual appearance for multiclass object detection. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1481\u20131488","DOI":"10.1109\/CVPR.2011.5995720"},{"key":"189_CR44","unstructured":"Saleh B, Elgammal AM (2015) Large-scale classification of fine-art paintings: learning the right metric on the right feature. CoRR"},{"key":"189_CR45","doi-asserted-by":"crossref","unstructured":"Sanakoyeu A, Kotovenko D, Lang S, Ommer B (2018) A style-aware content loss for real-time HD style transfer. In: Proceedings of the European conference on computer vision, vol\u00a02","DOI":"10.1007\/978-3-030-01237-3_43"},{"issue":"3","key":"189_CR46","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1145\/3068335","volume":"42","author":"E Schubert","year":"2017","unstructured":"Schubert E, Sander J, Ester M, Kriegel HP, Xu X (2017) Dbscan revisited, revisited: why and how you should (still) use DBSCAN. ACM Trans Database Syst (TODS) 42(3):19","journal-title":"ACM Trans Database Syst (TODS)"},{"key":"189_CR47","first-page":"753","volume-title":"European conference on computer vision workshops","author":"B Seguin","year":"2016","unstructured":"Seguin B, Striolo C, Kaplan F et al (2016) Visual link retrieval in a database of paintings. In: Hua G, J\u00e9gou H (eds) European conference on computer vision workshops. Springer, Cham, pp 753\u2013767"},{"key":"189_CR48","unstructured":"Sener O, Koltun V (2018) Multi-task learning as multi-objective optimization. In: Advances in neural information processing systems, pp 525\u2013536"},{"key":"189_CR49","doi-asserted-by":"crossref","unstructured":"Shamir L, Macura T, Orlov N, Eckley D.M, Goldberg I.G (2010) Impressionism, expressionism, surrealism: automated recognition of painters and schools of art. ACM Trans Appl Percept 6(2)","DOI":"10.1145\/1670671.1670672"},{"key":"189_CR50","unstructured":"Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations"},{"key":"189_CR51","unstructured":"Speer R, Havasi C (2012) Representing general relational knowledge in conceptnet 5. In: Proceedings of the eighth international conference on language resources and evaluation (LREC-2012), pp 3679\u20133686"},{"key":"189_CR52","doi-asserted-by":"crossref","unstructured":"Strezoski G, van Noord N, Worring M (2019) Learning task relatedness in multi-task learning for images in context. In: Proceedings of the 2019 on international conference on multimedia retrieval. ACM, pp 78\u201386","DOI":"10.1145\/3323873.3325009"},{"issue":"4","key":"189_CR53","first-page":"88","volume":"14","author":"G Strezoski","year":"2018","unstructured":"Strezoski G, Worring M (2018) Omniart: a large-scale artistic benchmark. ACM Trans Multimed Comput Commun Appl (TOMM) 14(4):88","journal-title":"ACM Trans Multimed Comput Commun Appl (TOMM)"},{"key":"189_CR54","doi-asserted-by":"crossref","unstructured":"Tan WR, Chan CS, Aguirre HE, Tanaka K (2016) Ceci n\u2019est pas une pipe: a deep convolutional network for fine-art paintings classification. In: ICIP","DOI":"10.1109\/ICIP.2016.7533051"},{"key":"189_CR55","doi-asserted-by":"crossref","unstructured":"Wang X, Ye Y, Gupta A (2018) Zero-shot recognition via semantic embeddings and knowledge graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6857\u20136866","DOI":"10.1109\/CVPR.2018.00717"},{"key":"189_CR56","doi-asserted-by":"publisher","unstructured":"Wattenberg M, Vigas F, Johnson I (2016) How to use t-sne effectively. Distill. https:\/\/doi.org\/10.23915\/distill.00002. http:\/\/distill.pub\/2016\/misread-tsne","DOI":"10.23915\/distill.00002"},{"key":"189_CR57","unstructured":"Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) Show, attend and tell: neural image caption generation with visual attention. In: International conference on machine learning, pp 2048\u20132057"},{"key":"189_CR58","unstructured":"Yang Y, Hospedales T (2016) Deep multi-task representation learning: a tensor factorisation approach. arXiv preprint arXiv:1605.06391"},{"key":"189_CR59","doi-asserted-by":"publisher","first-page":"1:1","DOI":"10.1145\/2978656","volume-title":"ACM Trans Multimed Comput Commun Appl","author":"H Zhang","year":"2016","unstructured":"Zhang H, Shang X, Luan H, Wang M, Chua TS (2016) Learning from collective intelligence: feature learning using social images and tags. ACM Trans Multimed Comput Commun Appl., pp 1:1\u20131:23. https:\/\/doi.org\/10.1145\/2978656"},{"issue":"2","key":"189_CR60","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1007\/s11263-012-0582-z","volume":"101","author":"T Zhang","year":"2013","unstructured":"Zhang T, Ghanem B, Liu S, Ahuja N (2013) Robust visual tracking via structured multi-task sparse learning. Int J Comput Vis 101(2):367\u2013383","journal-title":"Int J Comput Vis"},{"key":"189_CR61","doi-asserted-by":"crossref","unstructured":"Zhang Z, Luo P, Loy CC, Tang X (2014) Facial landmark detection by deep multi-task learning. In: European conference on computer vision. Springer, pp 94\u2013108","DOI":"10.1007\/978-3-319-10599-4_7"}],"container-title":["International Journal of Multimedia Information Retrieval"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s13735-019-00189-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s13735-019-00189-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s13735-019-00189-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T04:24:02Z","timestamp":1665721442000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s13735-019-00189-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,21]]},"references-count":61,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,3]]}},"alternative-id":["189"],"URL":"https:\/\/doi.org\/10.1007\/s13735-019-00189-4","relation":{},"ISSN":["2192-6611","2192-662X"],"issn-type":[{"value":"2192-6611","type":"print"},{"value":"2192-662X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12,21]]},"assertion":[{"value":"11 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 October 2019","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 December 2019","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 December 2019","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}