{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,16]],"date-time":"2025-12-16T12:45:30Z","timestamp":1765889130397,"version":"build-2065373602"},"reference-count":60,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2023,6,15]],"date-time":"2023-06-15T00:00:00Z","timestamp":1686787200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Hands represent an important aspect of pictorial narration but have rarely been addressed as an object of study in art history and digital humanities. Although hand gestures play a significant role in conveying emotions, narratives, and cultural symbolism in the context of visual art, a comprehensive terminology for the classification of depicted hand poses is still lacking. In this article, we present the process of creating a new annotated dataset of pictorial hand poses. The dataset is based on a collection of European early modern paintings, from which hands are extracted using human pose estimation (HPE) methods. The hand images are then manually annotated based on art historical categorization schemes. From this categorization, we introduce a new classification task and perform a series of experiments using different types of features, including our newly introduced 2D hand keypoint features, as well as existing neural network-based features. This classification task represents a new and complex challenge due to the subtle and contextually dependent differences between depicted hands. The presented computational approach to hand pose recognition in paintings represents an initial attempt to tackle this challenge, which could potentially advance the use of HPE methods on paintings, as well as foster new research on the understanding of hand gestures in art.<\/jats:p>","DOI":"10.3390\/jimaging9060120","type":"journal-article","created":{"date-parts":[[2023,6,16]],"date-time":"2023-06-16T01:31:31Z","timestamp":1686879091000},"page":"120","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["A Computational Approach to Hand Pose Recognition in Early Modern Paintings"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9467-8896","authenticated-orcid":false,"given":"Valentine","family":"Bernasconi","sequence":"first","affiliation":[{"name":"Digital Visual Studies, University of Zurich, 8006 Zurich, Switzerland"}]},{"given":"Eva","family":"Cetini\u0107","sequence":"additional","affiliation":[{"name":"Digital Society Initiative, University of Zurich, 8001 Zurich, Switzerland"}]},{"given":"Leonardo","family":"Impett","sequence":"additional","affiliation":[{"name":"Cambridge Digital Humanities, University of Cambridge, Cambridge CB2 1RX, UK"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Schmitt, J.C. (1990). La Raison des Gestes Dans L\u2019Occident M\u00e9di\u00e9val, Editions Gallimard.","DOI":"10.14375\/NP.9782070718450"},{"key":"ref_2","unstructured":"Wittkower, R. (1992). La Migration des Symboles, Thames & Hudson. Iconologia."},{"key":"ref_3","unstructured":"Dimova, T. (2020). Le Langage des Mains Dans L\u2019art: Histoire, Significations et Usages des Chirogrammes Picturaux aux XVIIe et XVIIIe Siecles, Brepols Publishers."},{"key":"ref_4","unstructured":"Bremmer, J., and Roodenburg, H. (1991). A Cultural History of Gesture. From Antiquity to the Present Day, Polity Press."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Agarwal, S., Karnick, H., Pant, N., and Patel, U. (2015, January 5\u20139). Genre and Style Based Painting Classification. Proceedings of the 2015 IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACV.2015.84"},{"key":"ref_6","unstructured":"Arora, R.S., and Elgammal, A. (2012, January 11\u201315). Towards automated classification of fine-art painting style: A comparative study. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.eswa.2018.07.026","article-title":"Fine-tuning convolutional neural networks for fine art classification","volume":"114","author":"Cetinic","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Tan, W.R., Chan, C.S., Aguirre, H.E., and Tanaka, K. (2016, January 25\u201328). Ceci n\u2019est pas une pipe: A deep convolutional network for fine-art paintings classification. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.","DOI":"10.1109\/ICIP.2016.7533051"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Hua, G., and J\u00e9gou, H. (2016, January 11\u201314). Visual Link Retrieval in a Database of Paintings. Proceedings of the Computer Vision\u2014ECCV 2016 Workshops, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46604-0"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Shen, X., Efros, A.A., and Aubry, M. (2019, January 15\u201320). Discovering Visual Patterns in Art Collections With Spatially-Consistent Feature Learning. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00950"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ufer, N., Simon, M., Lang, S., and Ommer, B. (2021). Large-scale interactive retrieval in art collections using multi-style feature aggregation. PLoS ONE, 16.","DOI":"10.1371\/journal.pone.0259718"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1325","DOI":"10.1007\/s11263-022-01576-x","article-title":"Spatially-consistent Feature Matching and Learning for Heritage Image Analysis","volume":"130","author":"Shen","year":"2022","journal-title":"Int. J. Comput. Vis."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1080\/01973762.2013.761111","article-title":"Nonverbal Communication in Medieval Illustrations Revisited by Computer Vision and Art History","volume":"29","author":"Bell","year":"2013","journal-title":"Vis. Resour."},{"key":"ref_14","first-page":"460","article-title":"Artistic Object Recognition by Unsupervised Style Adaptation","volume":"Volume 11363","author":"Jawahar","year":"2019","journal-title":"Computer Vision\u2014ACCV 2018"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Yin, R., Monson, E., Honig, E., Daubechies, I., and Maggioni, M. (2016, January 20\u201325). Object recognition in art drawings: Transfer of a neural network. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.","DOI":"10.1109\/ICASSP.2016.7472087"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Smir nov, S., and Eguizabal, A. (2018, January 22\u201324). Deep learning for object detection in fine-art paintings. Proceedings of the 2018 Metrology for Archaeology and Cultural Heritage (MetroArchaeo), Cassino, Italy.","DOI":"10.1109\/MetroArchaeo43810.2018.9089828"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Lin, H., Van Zuijlen, M., Wijntjes, M.W.A., Pont, S.C., and Bala, K. (2020). Insights from a Large-Scale Database of Material Depictions in Paintings. arXiv.","DOI":"10.1007\/978-3-030-68796-0_38"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Hua, G., and J\u00e9gou, H. (2016, January 8\u201310). Pose and Pathosformel in Aby Warburg\u2019s Bilderatlas. Proceedings of the Computer Vision\u2014ECCV 2016 Workshops, Amsterdam, The Netherlands. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-319-46604-0"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Marsocci, V., and Lastilla, L. (2021). POSE-ID-on\u2014A Novel Framework for Artwork Pose Clustering. ISPRS Int. J.-Geo-Inf., 10.","DOI":"10.3390\/ijgi10040257"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3569089","article-title":"Enhancing Human Pose Estimation in Ancient Vase Paintings via Perceptually-grounded Style Transfer Learning","volume":"16","author":"Madhu","year":"2022","journal-title":"J. Comput. Cult. Herit."},{"key":"ref_21","unstructured":"Ohrt, R., and Ohrt, R. (2020). Aby Warburg: Bilderatlas Mnemosyne: The Original, Hatje Cantz Verlag. Kulturgeschichte."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1109\/TPAMI.2019.2929257","article-title":"OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields","volume":"43","author":"Cao","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Guler, R.A., Neverova, N., and Kokkinos, I. (2018, January 18\u201323). DensePose: Dense Human Pose Estimation in the Wild. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00762"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014, January 6\u201312). Microsoft COCO: Common Objects in Context. Proceedings of the Computer Vision\u2014ECCV, Zurich, Switzerland. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-319-10599-4"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. (2014, January 23\u201328). 2D Human Pose Estimation: New Benchmark and State of the Art Analysis. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.471"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Simon, T., Joo, H., Matthews, I., and Sheikh, Y. (2017, January 21\u201326). Hand Keypoint Detection in Single Images using Multiview Bootstrapping. Proceedings of the CVPR, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.494"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1515\/mial-2019-0004","article-title":"Ikonographie Und Interaktion. Computergest\u00fctzte Analyse von Posen in Bildern der Heilsgeschichte","volume":"24","author":"Impett","year":"2019","journal-title":"Das Mittelalt."},{"key":"ref_28","unstructured":"Impett, L. (2020). The Routledge Companion to Digital Humanities and Art History, Routledge."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Bernasconi, V. (2022, January 22\u201325). GAB\u2014Gestures for Artworks Browsing. Proceedings of the 27th International Conference on Intelligent User Interfaces, Online. IUI \u201822 Companion.","DOI":"10.1145\/3490100.3516470"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Springstein, M., Schneider, S., Althaus, C., and Ewerth, R. (2022, January 10\u201314). Semi-Supervised Human Pose Estimation in Art-Historical Images. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal. MM \u201922.","DOI":"10.1145\/3503161.3548371"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Jenicek, T., and Chum, O. (2019, January 20\u201325). Linking Art through Human Poses. Proceedings of the 2019 International Conference on Document Analysis and Recognition (ICDAR), Sydney, Australia.","DOI":"10.1109\/ICDAR.2019.00216"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zhao, S., Akda\u011f Salah, A., and Salah, A.A. (2022, January 23\u201327). Automatic Analysis of Human Body Representations in Western Art. Proceedings of the Computer Vision\u2013ECCV 2022 Workshops, Tel Aviv, Israel. Proceedings, Part I.","DOI":"10.1007\/978-3-031-25056-9_19"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3458885","article-title":"A Dataset and a Convolutional Model for Iconography Classification in Paintings","volume":"14","author":"Milani","year":"2021","journal-title":"J. Comput. Cult. Herit."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Cetinic, E. (2021). Towards Generating and Evaluating Iconographic Image Captions of Artworks. J. Imaging, 7.","DOI":"10.3390\/jimaging7080123"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1109\/TCYB.2021.3129119","article-title":"Gesture-Based Human\u2013Machine Interaction: Taxonomy, Problem Definition, and Analysis","volume":"53","author":"Mastrogiovanni","year":"2023","journal-title":"IEEE Trans. Cybern."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.cviu.2015.08.004","article-title":"Recent methods and databases in vision-based hand gesture recognition: A review","volume":"141","author":"Pisharady","year":"2015","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1049\/iet-cvi.2017.0052","article-title":"Review of constraints on vision-based gesture recognition for human\u2013computer interaction","volume":"12","author":"Chakraborty","year":"2018","journal-title":"IET Computer Vision"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Oudah, M., Al-Naji, A., and Chahl, J. (2020). Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. J. Imaging, 6.","DOI":"10.3390\/jimaging6080073"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Ahmed, S., Kallu, K.D., Ahmed, S., and Cho, S.H. (2021). Hand Gestures Recognition Using Radar Sensors for Human-Computer-Interaction: A Review. Remote Sens., 13.","DOI":"10.3390\/rs13030527"},{"key":"ref_40","unstructured":"Zhang, F., Bazarevsky, V., Vaku nov, A., Tkachenka, A., Sung, G., Chang, C.L., and Grundmann, M. (2020). MediaPipe Hands: On-device Real-time Hand Tracking. arXiv."},{"key":"ref_41","unstructured":"M, S., Rakesh, S., Gupta, S., Biswas, S., and Das, P.P. (2015, January 16\u201319). Real-time hands-free immersive image navigation system using Microsoft Kinect 2.0 and Leap Motion Controller. Proceedings of the 2015 Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG), Patna, Bihar."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1110","DOI":"10.1109\/TMM.2013.2246148","article-title":"Robust Part-Based Hand Gesture Recognition Using Kinect Sensor","volume":"15","author":"Ren","year":"2013","journal-title":"IEEE Trans. Multimed."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Marin, G., Dominio, F., and Zanuttigh, P. (2014, January 27\u201330). Hand gesture recognition with leap motion and kinect devices. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.","DOI":"10.1109\/ICIP.2014.7025313"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.patcog.2017.10.033","article-title":"Convolutional Neural Networks and Long Short-Term Memory for skeleton-based human activity and hand gesture recognition","volume":"76","author":"Cabido","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"K\u00f6p\u00fckl\u00fc, O., Gunduz, A., Kose, N., and Rigoll, G. (2019, January 14\u201318). Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks. Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France.","DOI":"10.1109\/FG.2019.8756576"},{"key":"ref_46","unstructured":"Sung, G., Sokal, K., Uboweja, E., Bazarevsky, V., Baccash, J., Bazavan, E.G., Chang, C.L., and Grundmann, M. (2021). On-device Real-time Hand Gesture Recognition. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"113794","DOI":"10.1016\/j.eswa.2020.113794","article-title":"Sign Language Recognition: A Deep Survey","volume":"164","author":"Rastgoo","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1007\/s13042-017-0705-5","article-title":"A review of hand gesture and sign language recognition techniques","volume":"10","author":"Cheok","year":"2019","journal-title":"Int. J. Mach. Learn. Cybern."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Kumar, M., Gupta, P., Jha, R.K., Bhatia, A., Jha, K., and Shah, B.K. (2021, January 6\u20138). Sign Language Alphabet Recognition Using Convolution Neural Network. Proceedings of the 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India.","DOI":"10.1109\/ICICCS51141.2021.9432296"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Shin, J., Matsuoka, A., Hasan, M.A.M., and Srizon, A.Y. (2021). American Sign Language Alphabet Recognition by Extracting Feature from Hand Pose Estimation. Sensors, 21.","DOI":"10.3390\/s21175856"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Zhang, X., Huang, H., Tan, J., Xu, H., Yang, C., Peng, G., Wang, L., and Liu, J. (2021, January 10\u201317). Hand Image Understanding via Deep Multi-Task Learning. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.01109"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_53","unstructured":"Nagaraj, A. (2023, February 21). ASL Alphabet. Available online: https:\/\/www.kaggle.com\/datasets\/grassknoted\/asl-alphabet."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Lucaf\u00f2, C., Marzoli, D., Zdybek, P., Malatesta, G., Smerilli, F., Ferrara, C., and Tommasi, L. (2021). The Bias toward the Right Side of Others Is Stronger for Hands than for Feet. Symmetry, 13.","DOI":"10.3390\/sym13010146"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1007\/s00221-014-4124-5","article-title":"Both right- and left-handers show a bias to attend others\u2019 right arm","volume":"233","author":"Marzoli","year":"2015","journal-title":"Exp. Brain Res."},{"key":"ref_56","first-page":"553","article-title":"La pr\u00e9\u00e9minence de la main droite: \u00c9tude sur la polarit\u00e9 religieuse","volume":"68","author":"Hertz","year":"1909","journal-title":"Revue Philosophique de la France et de L\u2019\u00c9tranger"},{"key":"ref_57","unstructured":"Barasch, M. (1987). Giotto and the Language of Gesture, University Press. Cambridge studies in the history of art."},{"key":"ref_58","unstructured":"Bernasconi, V. (2022, June 07). La main baladeuse. Jeu de Paume en ligne 2022. as part of the online exhibition Contagions visuelles. Available online: https:\/\/jdp.visualcontagions.net\/nautilus."},{"key":"ref_59","unstructured":"Hughes, A. (BBC Science Focus Magazine, 2023). Why AI-generated hands are the stuff of nightmares, explained by a scientist, BBC Science Focus Magazine."},{"key":"ref_60","unstructured":"Chayka, K. (The New Yorker, 2023). The Uncanny Failure of A.I.-Generated Hands, The New Yorker."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/6\/120\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:55:51Z","timestamp":1760126151000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/6\/120"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,15]]},"references-count":60,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["jimaging9060120"],"URL":"https:\/\/doi.org\/10.3390\/jimaging9060120","relation":{},"ISSN":["2313-433X"],"issn-type":[{"type":"electronic","value":"2313-433X"}],"subject":[],"published":{"date-parts":[[2023,6,15]]}}}