{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T23:40:06Z","timestamp":1784677206480,"version":"3.55.0"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T00:00:00Z","timestamp":1708646400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Academic Research in Korea National University of Cultural Heritage"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Comput. Cult. Herit."],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>Conventional studies on the satisfaction of museum visitors focus on collecting information through surveys to provide a one-way service to visitors, and thus it is impossible to obtain feedback on the real-time satisfaction of visitors who are experiencing the museum exhibition program. In addition, museum practitioners lack research on automated ways to evaluate a produced content program's lifecycle and its appropriateness. To overcome these problems, we propose a novel multi-convolutional neural network, called VimoNet, which is able to recognize visitors emotions automatically in real-time based on their facial expressions and body gestures. Furthermore, we design a user preference model of content and a framework to obtain feedback on content improvement for providing personalized digital cultural heritage content to visitors. Specifically, we define seven emotions of visitors and build a dataset of visitor facial expressions and gestures with respect to the emotions. Using the dataset, we proceed with feature fusion of face and gesture images trained on the DenseNet-201 and VGG-16 models for generating a combined emotion recognition model. From the results of the experiment, VimoNet achieved a classification accuracy of 84.10%, providing 7.60% and 14.31% improvement, respectively, over a single face and body gesture-based method of emotion classification performance. It is thus possible to automatically capture the emotions of museum visitors via VimoNet, and we confirm its feasibility through a case study with respect to digital content of cultural heritage.<\/jats:p>","DOI":"10.1145\/3631123","type":"journal-article","created":{"date-parts":[[2023,10,30]],"date-time":"2023-10-30T21:56:24Z","timestamp":1698702984000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Real-time Multi-CNN-based Emotion Recognition System for Evaluating Museum Visitors\u2019 Satisfaction"],"prefix":"10.1145","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7333-9948","authenticated-orcid":false,"given":"Do Hyung","family":"Kwon","sequence":"first","affiliation":[{"name":"Korea National University of Cultural Heritage"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9034-5234","authenticated-orcid":false,"given":"Jeong Min","family":"Yu","sequence":"additional","affiliation":[{"name":"Korea National University of Cultural Heritage"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,2,23]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1002\/sce.20018"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","unstructured":"Vasiliki V. Daskalaki Maria C. Voutsa Christina Boutsouki and Leonidas Hatzithomas. 2020. Service quality visitor satisfaction and future behavior in the museum sector. 10.5281\/ZENODO.3603167","DOI":"10.5281\/ZENODO.3603167"},{"key":"e_1_3_1_4_2","volume-title":"Proceedings of the IEEE International Conference on RFID Technology and Applications (RFID-TA \u201921)","author":"Vena Arnaud","year":"2021","unstructured":"Arnaud Vena, Isabelle Illanes, Lucie Alidieres, Brice Sorli, and Francois Perea. 2021. RFID based indoor localization system to analyze visitor behavior in a museum. In Proceedings of the IEEE International Conference on RFID Technology and Applications (RFID-TA \u201921). 10.1109\/rfid-ta53372.2021.9617265"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.3390\/app12020533"},{"key":"e_1_3_1_6_2","volume-title":"Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom \u201816)","author":"Martella Claudio","year":"2016","unstructured":"Claudio Martella, Armando Miraglia, Marco Cattani, and Maarten van Steen. 2016. Leveraging proximity sensing to mine the behavior of museum visitors. In Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom \u201816). 10.1109\/percom.2016.7456513"},{"key":"e_1_3_1_7_2","article-title":"Early prediction of museum visitor engagement with multimodal","author":"Henderson Nathan","year":"2021","unstructured":"Nathan Henderson, Wookhee Min, Andrew Emerson, Jonathan Rowe, Seung Lee, James Minogue, and James Lester. 2021. Early prediction of museum visitor engagement with multimodal adversarial domain adaptation. International Educational Data Mining Society.","journal-title":"International Educational Data Mining Society"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2020.1818457"},{"key":"e_1_3_1_9_2","volume-title":"Proceedings of the International Conference on Electrical, Computer and Communication Engineering (ECCE \u201817)","author":"Rashed Md. Golam","year":"2017","unstructured":"Md. Golam Rashed, Dipankar Das, Yoshinori Kobayashi, and Yoshinori Kuno. 2017. Analysis and prediction of real museum visitorsinterests and preferences based on their behaviors. In Proceedings of the International Conference on Electrical, Computer and Communication Engineering (ECCE \u201817). 10.1109\/ecace.2017.7912947"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-014-2192-y"},{"key":"e_1_3_1_11_2","first-page":"32","article-title":"Visitor survey at the national museum of contemporary art, deoksugung: A study on the visitors of korean art museums with implications for museum","volume":"1","author":"Yang Jiyeon","year":"2001","unstructured":"Jiyeon Yang. 2001. Visitor survey at the national museum of contemporary art, deoksugung: A study on the visitors of korean art museums with implications for museum marketing. Kor. Assoc. Arts Manage. 1 (2001), 32\u201361.","journal-title":"Kor. Assoc. Arts Manage."},{"key":"e_1_3_1_12_2","volume-title":"Museums and Their Visitors","author":"Hopper-Greenhill Eilean","year":"1994","unstructured":"Eilean Hopper-Greenhill. 1994. Museums and Their Visitors. Routledge, London."},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocs.2021.101357"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.3233\/ais-170467"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.pmcj.2016.08.011"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.3390\/jimaging8010008"},{"key":"e_1_3_1_17_2","volume-title":"Extended Abstracts of the CHI Conference on Human Factors in Computing Systems","author":"Tseng Yuan-Chi","year":"2018","unstructured":"Yuan-Chi Tseng, An-Hou Tang, Yu-Hsuan Shih, and Sheng-Fu Liang. 2018. Using eye movement data and visit contexts to understand the experience of museum visitors. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 10.1145\/3170427.3188587"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/tip.2021.3049955"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.06.005"},{"key":"e_1_3_1_20_2","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR \u201820)","author":"Wang Kai","year":"2020","unstructured":"Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao. 2020. Suppressing uncertainties for large-scale facial expression recognition. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR \u201820). 10.1109\/cvpr42600.2020.00693"},{"key":"e_1_3_1_21_2","volume-title":"Proceedings of the IEEE International Conference on Image Processing (ICIP \u201819)","author":"Shen Zhijuan","year":"2019","unstructured":"Zhijuan Shen, Jun Cheng, Xiping Hu, and Qian Dong. 2019. Emotion recognition based on multi-view body gestures. In Proceedings of the IEEE International Conference on Image Processing (ICIP \u201819). 10.1109\/icip.2019.8803460"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/taffc.2018.2874986"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/2738221"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/tcds.2020.2990928"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/taslp.2021.3120586"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/access.2021.3135658"},{"key":"e_1_3_1_27_2","volume-title":"Proceedings of the International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES \u201821)","author":"Mahima M. A.","year":"2021","unstructured":"M. A. Mahima, Nidhi C. Patel, Srividhya Ravichandran, N. Aishwarya, and Sumana Maradithaya. 2021. A text-based hybrid approach for multiple emotion detection using contextual and semantic analysis. In Proceedings of the International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES \u201821). 10.1109\/icses52305.2021.9633843"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/tcyb.2020.2987064"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/tim.2022.3147876"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/taffc.2017.2781732"},{"key":"e_1_3_1_31_2","volume-title":"\u2013Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP \u201920)","author":"Sarkar Pritam","year":"2020","unstructured":"Pritam Sarkar and Ali Etemad. 2020. Self-supervised learning for ECG-based emotion recognition. In \u2013Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP \u201920). 10.1109\/icassp40776.2020.9053985"},{"key":"e_1_3_1_32_2","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Computer Science","author":"Chen Genlang","year":"2019","unstructured":"Genlang Chen, Yi Zhu, Zhiqing Hong, and Zhen Yang. 2019. EmotionalGAN. In Proceedings of the International Conference on Artificial Intelligence and Computer Science. 10.1145\/3349341.3349422"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.3390\/s22103749"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.3390\/s21041322"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","unstructured":"Steve Benford Anders Sundnes L\u00f8vlie Karin Ryding Paulina Rajkowska Edgar Bodiaj Dimitrios Paris Darzentas Harriet R. Cameron Jocelyn Spence Joy Egede and Bogdan Spanjevic. 2022. Sensitive pictures: Emotional interpretation in the museum. 10.48550\/ARXIV.2203.01041","DOI":"10.48550\/ARXIV.2203.01041"},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1007\/978-3-030-78092-0_35","volume-title":"Universal Access in Human-Computer Interaction. Design Methods and User Experience: Proceedings of the 15th International Conference (UAHCI \u201921), Held as Part of the 23rd HCI International Conference (HCII \u201921)","author":"Altieri Alex","year":"2021","unstructured":"Alex Altieri, Silvia Ceccacci, Luca Giraldi, Alma Leopardi, Maura Mengoni, and Abudukaiyoumu Talipu. 2021. Affective guide for museum: A system to suggest museum paths based on visitors\u2019 emotions. In Universal Access in Human-Computer Interaction. Design Methods and User Experience: Proceedings of the 15th International Conference (UAHCI \u201921), Held as Part of the 23rd HCI International Conference (HCII \u201921), 521\u2013532."},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12652-020-02381-5"},{"key":"e_1_3_1_38_2","article-title":"Inferring emotional state from facial micro-expressions","author":"Aiuti Alessandro","year":"2022","unstructured":"Alessandro Aiuti, Alessio Ferrato, Carla Limongelli, Mauro Mezzini, and Giuseppe Sansonetti. 2022. Inferring emotional state from facial micro-expressions. Socialize (2022).","journal-title":"Socialize"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066x.37.9.1019"},{"key":"e_1_3_1_40_2","volume-title":"Best Laid Schemes: The Psychology of the Emotions","author":"Oatley Keith","year":"1992","unstructured":"Keith Oatley. 1992. Best Laid Schemes: The Psychology of the Emotions. Cambridge University Press."},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1086\/209123"},{"key":"e_1_3_1_42_2","article-title":"Affect and consumer behavior","author":"Cohen Joel B.","year":"1991","unstructured":"Joel B. Cohen and Charles S. Areni. 1991. Affect and consumer behavior. In Handbook of Consumer Behavior. Prentice-Hall, Englewood Cliffs, NJ.","journal-title":"Handbook of Consumer Behavior"},{"key":"e_1_3_1_43_2","first-page":"207","article-title":"Universals and cultural differences in facial expression of emotion","volume":"19","author":"Ekman Paul","year":"1971","unstructured":"Paul Ekman. 1971. Universals and cultural differences in facial expression of emotion. Nebr. Symp. Motiv. 19 (1971), 207\u2013283.","journal-title":"Nebr. Symp. Motiv"},{"issue":"6","key":"e_1_3_1_44_2","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell James A.","year":"1980","unstructured":"James A. Russell. 1980. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 6 (1980), 1161\u20131178","journal-title":"J. Pers. Soc. Psychol."},{"issue":"4","key":"e_1_3_1_45_2","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1511\/2001.28.344","article-title":"The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice","volume":"89","author":"Plutchik Robert","year":"2001","unstructured":"Robert Plutchik. 2001. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am. Sci. 89, 4 (2001), 344\u2013350.","journal-title":"Am. Sci."},{"key":"e_1_3_1_46_2","first-page":"45","article-title":"Basic emotions","volume":"98","author":"Ekman Paul","year":"1999","unstructured":"Paul Ekman et\u00a0al. 1999. Basic emotions. In Handbook of Cognition and Emotion 98 (1999), 45\u201360.","journal-title":"Handbook of"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.13088\/jiis.2014.20.1.001"},{"key":"e_1_3_1_48_2","volume-title":"Proceedings of the International Conference on Multimodal Interaction","author":"Emerson Andrew","year":"2020","unstructured":"Andrew Emerson, Nathan Henderson, Jonathan Rowe, Wookhee Min, Seung Lee, James Minogue, and James Lester. 2020. Early prediction of visitor engagement in science museums with multimodal learning analytics. In Proceedings of the International Conference on Multimodal Interaction. 10.1145\/3382507.3418890"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-49161-1_37"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","unstructured":"Gao Huang Zhuang Liu Laurens van der Maaten and Kilian Q. Weinberger. 2016. Densely connected convolutional networks. arXiv:1608.06993. Retrieved from 10.48550\/ARXIV.1608.06993","DOI":"10.48550\/ARXIV.1608.06993"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv.1409.1556. Retrieved from 10.48550\/ARXIV.1409.1556","DOI":"10.48550\/ARXIV.1409.1556"},{"key":"e_1_3_1_52_2","volume-title":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition","author":"Viola Paul","year":"2001","unstructured":"Paul Viola and Michael Jones. 2001. Rapid object detection using boosted cascade of simple feature. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. 10.1109\/cvpr.2001.990517"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/taffc.2017.2740923"},{"key":"e_1_3_1_54_2","volume-title":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition\u2014Workshops","author":"Lucey Patrick","year":"2010","unstructured":"Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. 2010. The extended cohn-kanade dataset: A complete dataset for action unit and emotion-specified expression. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition\u2014Workshops. 10.1109\/cvprw.2010.5543262"},{"key":"e_1_3_1_55_2","volume-title":"Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition","author":"Lyons M.","unstructured":"M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba. Coding facial expressions with Gabor wavelets. In Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition. 10.1109\/afgr.1998.670949"},{"key":"e_1_3_1_56_2","volume-title":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops","author":"Glowinski Donald","year":"2008","unstructured":"Donald Glowinski, Antonio Camurri, Gualtiero Volpe, Nele Dael, and Klaus Scherer. 2008. Technique for automatic emotion recognition by body gesture analysis. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 10.1109\/cvprw.2008.4563173"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-74889-2_7"},{"key":"e_1_3_1_58_2","volume-title":"Proceedings of the 18th International Conference on Pattern Recognition (ICPR \u201906)","author":"Gunes H.","year":"2006","unstructured":"H. Gunes and M. Piccardi. 2006. A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In Proceedings of the 18th International Conference on Pattern Recognition (ICPR \u201906). 10.1109\/icpr.2006.39"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","unstructured":"Paul Ekman and Wallace V. Friesen. 1978. Facial action coding system. 10.1037\/t27734-000","DOI":"10.1037\/t27734-000"},{"key":"e_1_3_1_60_2","volume-title":"Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication","author":"Gunes H.","year":"2005","unstructured":"H. Gunes and M. Piccardi. 2005. Fusing face and body gesture for machine recognition of emotions. In Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication. 10.1109\/roman.2005.1513796"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"}],"container-title":["Journal on Computing and Cultural Heritage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3631123","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3631123","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:35:51Z","timestamp":1750178151000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3631123"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,23]]},"references-count":60,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3631123"],"URL":"https:\/\/doi.org\/10.1145\/3631123","relation":{},"ISSN":["1556-4673","1556-4711"],"issn-type":[{"value":"1556-4673","type":"print"},{"value":"1556-4711","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,23]]},"assertion":[{"value":"2022-08-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-05-21","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}