{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T09:13:15Z","timestamp":1774516395299,"version":"3.50.1"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,6,24]],"date-time":"2023-06-24T00:00:00Z","timestamp":1687564800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Comput. Cult. Herit."],"published-print":{"date-parts":[[2023,6,30]]},"abstract":"<jats:p>\n            The surge of Mobile\n            <jats:bold>Virtual Reality (VR)<\/jats:bold>\n            applications is getting growing attention among researchers and practitioners. The recent literature demonstrates its benefits when used for education purposes, since virtual immersion yields promising results for learning. Leveraging this trend, within the so called \u201cdigital didactics\u201d, the need to gauge VR\u2019s effectiveness in the didactic field has become paramount; so far, a method to connect traditional evaluation strategies to novel VR-based learning is still broadly missing. This paper investigates the problem of quantifying the learning outcomes and proposes a new didactic evaluation method for the\n            <jats:bold>Digital Cultural Heritage (DCH)<\/jats:bold>\n            learning. This research, conducted in a higher education institute, proposes three new Key Performance Indicators, referring to\n            <jats:bold>Revised Bloom\u2019s Taxonomy (RBT)<\/jats:bold>\n            :\n            <jats:bold>Mnemonic (M)<\/jats:bold>\n            ,\n            <jats:bold>Transversal (T)<\/jats:bold>\n            , and\n            <jats:bold>Disciplinary (D)<\/jats:bold>\n            . A questionnaire was administered by the same teacher who holds the course, to evaluate how well the application communicated information. The participants have been subdivided into two groups with the same knowledge base. The first group (1ACAT) that represents the \u201cVR group\u201d used the app at home to deepen their subject studies; while the second group (1AGR) that represents the \u201ccontrol group\u201d consulted and studied the app only before the test. The results have demonstrated that the \u201ccontrol group\u201d has a greater ability to support purely mnemonic topics (1ACAT 46.9%, 1AGR 53.1%), such as dates and simple definitions. The skills reached by the \u201cVR group\u201d attest to both transveral (1ACAT 52.9%, 1AGR 47.1%) and disciplinary (1ACAT 52.5%, 1AGR 47.5%) knowledge. These results validate the use of VR in teaching, demonstrating both experiential value and student involvement, but even confirming the compensatory function of VR if compared with the irreplaceable role of teachers in guiding learners to learn.\n          <\/jats:p>","DOI":"10.1145\/3593432","type":"journal-article","created":{"date-parts":[[2023,4,24]],"date-time":"2023-04-24T12:09:25Z","timestamp":1682338165000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["Evaluating Learning Outcomes of Virtual Reality Applications in Education: A Proposal for Digital Cultural Heritage"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5523-7174","authenticated-orcid":false,"given":"Marina","family":"Paolanti","sequence":"first","affiliation":[{"name":"University of Macerata, Department of Political Sciences, Communication andInternational Relations, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0660-342X","authenticated-orcid":false,"given":"Mariapaola","family":"Puggioni","sequence":"additional","affiliation":[{"name":"VRAI Vision Robotics and Artificial Intelligence Lab Dipartimento di Ingegneria dell\u2019Informazione (DII), Universit\u00e0 Politecnica delle Marche, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8893-9244","authenticated-orcid":false,"given":"Emanuele","family":"Frontoni","sequence":"additional","affiliation":[{"name":"University of Macerata, Department of Political Sciences, Communication and International Relations and VRAI Vision Robotics and Artificial Intelligence Lab Dipartimento di Ingegneria dell\u2019Informazione (DII), Universit\u00e0 Politecnica delle Marche, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1169-4795","authenticated-orcid":false,"given":"Lorella","family":"Giannandrea","sequence":"additional","affiliation":[{"name":"University of Macerata, Department of Education, Cultural Heritage and Tourism, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9160-834X","authenticated-orcid":false,"given":"Roberto","family":"Pierdicca","sequence":"additional","affiliation":[{"name":"Dipartimento di Ingegneria Civile, Edile e dell\u2019Architettura (DICEA), Universit\u00e0 Politecnica delle Marche, Italy"}]}],"member":"320","published-online":{"date-parts":[[2023,6,24]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1108\/10748120110424816"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1089\/cyber.2018.0150"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.103647"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2019.2937525"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.103710"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460657"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359590"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1023\/B:HIGH.0000009822.49980.30"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2016.08.019"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11528-008-0108-2"},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1007\/978-3-319-60922-5_30","volume-title":"Int. Conf. Augmented Reality, Virtual Reality and Computer Graphics","author":"Pierdicca Roberto","year":"2017","unstructured":"Roberto Pierdicca, Emanuele Frontoni, Rama Pollini, Matteo Trani, and Lorenzo Verdini. 2017. The use of augmented reality glasses for the application in industry 4.0. In Int. Conf. Augmented Reality, Virtual Reality and Computer Graphics. Springer, 389\u2013401."},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.469"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procir.2019.02.025"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377143"},{"key":"e_1_3_3_16_2","article-title":"Virtual reality and augmented reality in education","author":"Boyles Brian","year":"2017","unstructured":"Brian Boyles. 2017. Virtual reality and augmented reality in education. Center For Teaching Excellence, United States Military Academy, West Point, NY (2017).","journal-title":"Center For Teaching Excellence, United States Military Academy, West Point, NY"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1080\/09523987.2019.1583461"},{"key":"e_1_3_3_18_2","first-page":"412","volume-title":"Proc. 27th Int. Conf. Computers in Education","author":"Kr\u00fcger Jule M.","year":"2019","unstructured":"Jule M. Kr\u00fcger, Alexander Buchholz, and Daniel Bodemer. 2019. Augmented reality in education: Three unique characteristics from a user\u2019s perspective. In Proc. 27th Int. Conf. Computers in Education. 412\u2013422."},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.3991\/ijet.v14i03.9289"},{"issue":"3","key":"e_1_3_3_20_2","first-page":"286","article-title":"Learning by searching: A learning environment that provides searching and analysis facilities for supporting trend analysis activities","volume":"16","author":"Yin Chengjiu","year":"2013","unstructured":"Chengjiu Yin, Han-Yu Sung, Gwo-Jen Hwang, Sachio Hirokawa, Hui-Chun Chu, Brendan Flanagan, and Yoshiyuki Tabata. 2013. Learning by searching: A learning environment that provides searching and analysis facilities for supporting trend analysis activities. J. Educational Technol. Soc. 16, 3 (2013), 286\u2013300.","journal-title":"J. Educational Technol. Soc."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2010.27"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2010.13"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2013.37"},{"key":"e_1_3_3_24_2","doi-asserted-by":"crossref","unstructured":"K. E. Stavroulia M. Christofi E. Baka D. Michael-Grigoriou N. Magnenat-Thalmann and A. Lanitis. 2019. Assessing the emotional impact of virtual reality-based teacher training. The International Journal of Information and Learning Technology .","DOI":"10.1108\/IJILT-11-2018-0127"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453073"},{"key":"e_1_3_3_26_2","first-page":"1","volume-title":"Proc. 28th Int. Conf. Information Systems Development (ISD2019 Toulon, France)","author":"Wohlgenannt Isabell","year":"2019","unstructured":"Isabell Wohlgenannt, Jennifer Fromm, Stefan Stieglitz, Jaziar Radianti, and Tim A. Majchrzak. 2019. Virtual reality in higher education: Preliminary results from a design-science-research project. In Proc. 28th Int. Conf. Information Systems Development (ISD2019 Toulon, France). 1\u201310."},{"key":"e_1_3_3_27_2","first-page":"1380","volume-title":"Int. Conf. Society for Information Technology and Teacher Education","author":"Chang George","year":"2010","unstructured":"George Chang, Patricia Morreale, and Padmavathi Medicherla. 2010. Applications of augmented reality systems in education. In Int. Conf. Society for Information Technology and Teacher Education. Association for the Advancement of Computing in Education (AACE), 1380\u20131385."},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1504\/IJTEL.2011.042102"},{"key":"e_1_3_3_29_2","first-page":"241","volume-title":"The Educational Forum","author":"Dewey John","year":"1986","unstructured":"John Dewey. 1986. Experience and education. In The Educational Forum, Vol. 50. Taylor & Francis, 241\u2013252."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-69744-2_18"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.103778"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3145534"},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1007\/978-3-319-40651-0_17","volume-title":"Int. Conf. Augmented Reality, Virtual Reality and Computer Graphics","author":"Naspetti Simona","year":"2016","unstructured":"Simona Naspetti, Roberto Pierdicca, Serena Mandolesi, Marina Paolanti, Emanuele Frontoni, and Raffaele Zanoli. 2016. Automatic analysis of eye-tracking data for augmented reality applications: A prospective outlook. In Int. Conf. Augmented Reality, Virtual Reality and Computer Graphics. Springer, 217\u2013230."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-10-2419-1_2"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctvjf9vz4"},{"key":"e_1_3_3_36_2","first-page":"20","article-title":"Taxonomy of educational objectives. Vol. 1: Cognitive domain","author":"Bloom Benjamin S.","year":"1956","unstructured":"Benjamin S. Bloom. 1956. Taxonomy of educational objectives. Vol. 1: Cognitive domain. New York: McKay (1956), 20\u201324.","journal-title":"New York: McKay"},{"key":"e_1_3_3_37_2","first-page":"113","volume-title":"Int. Conf. Computer Engineering (ICENCO)","author":"Sayed Neven A. M. El","year":"2010","unstructured":"Neven A. M. El Sayed, Hala H. Zayed, and Mohamed I. Sharawy. 2010. ARSC: Augmented reality student card. In Int. Conf. Computer Engineering (ICENCO). IEEE, 113\u2013120."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(01)00075-6"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1012049406877"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.4018\/978-1-60566-940-3.ch012"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1166\/asl.2011.2043"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1080\/1475939X.2020.1737210"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3352933"},{"issue":"1","key":"e_1_3_3_44_2","first-page":"5","article-title":"Teaching and learning physics using 3D virtual learning environment: A case study of combined virtual reality and virtual laboratory in secondary school","volume":"39","author":"Bogusevschi Diana","year":"2020","unstructured":"Diana Bogusevschi, Cristina Muntean, and Gabriel-Miro Muntean. 2020. Teaching and learning physics using 3D virtual learning environment: A case study of combined virtual reality and virtual laboratory in secondary school. J. Comput. Math. Sci. Teaching 39, 1 (2020), 5\u201318.","journal-title":"J. Comput. Math. Sci. Teaching"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.4018\/978-1-5225-0680-5.ch009"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.culher.2013.04.004"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/129888.129892"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10055-017-0318-z"},{"key":"e_1_3_3_49_2","first-page":"322","volume-title":"Int. Conf. Educational Innovation through Technology (EITT)","author":"Yang Xiaozhe","year":"2017","unstructured":"Xiaozhe Yang, Pei-Yu Cheng, and Xue Yang. 2017. The impact of three types of virtual reality scene on learning. In Int. Conf. Educational Innovation through Technology (EITT). IEEE, 322\u2013324."},{"key":"e_1_3_3_50_2","volume-title":"Proc. 53rd Annu. Hawaii Int. Conf. System Sciences","author":"Holopainen Jani","year":"2020","unstructured":"Jani Holopainen, Antti Juhani L\u00e4htev\u00e4noja, Osmo Mattila, Ilona S\u00f6dervik, Essi P\u00f6yry, Petri Parvinen, et\u00a0al. 2020. Exploring the learning outcomes with various technologies: Proposing design principles for virtual reality learning environments. In Proc. 53rd Annu. Hawaii Int. Conf. System Sciences. University of Hawaii."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.13189\/ujer.2020.080214"},{"key":"e_1_3_3_52_2","first-page":"1","article-title":"Designing and validating educational standards for E-teaching in virtual learning environments (VLEs), based on revised Bloom\u2019s taxonomy","author":"Barari Nori","year":"2020","unstructured":"Nori Barari, Morteza RezaeiZadeh, Abasalt Khorasani, and Farnoosh Alami. 2020. Designing and validating educational standards for E-teaching in virtual learning environments (VLEs), based on revised Bloom\u2019s taxonomy. Interactive Learn. Environ. (2020), 1\u201313.","journal-title":"Interactive Learn. Environ."},{"issue":"2","key":"e_1_3_3_53_2","first-page":"7","article-title":"Narrative and content combine in a learning game for virtual heritage","volume":"9","author":"Jacobson Jeffrey","year":"2009","unstructured":"Jeffrey Jacobson, Kerry Handron, and Lynn Holden. 2009. Narrative and content combine in a learning game for virtual heritage. Distance Educ. 9, 2 (2009), 7\u201326.","journal-title":"Distance Educ."},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2018.09.002"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.103784"},{"key":"e_1_3_3_56_2","first-page":"53","article-title":"Using structural equation modelling to study the influence of perceived usefulness and perceived compatibility on students\u2019 attitudes towards using Ipad","author":"Zogheib Bashar","year":"2019","unstructured":"Bashar Zogheib. 2019. Using structural equation modelling to study the influence of perceived usefulness and perceived compatibility on students\u2019 attitudes towards using Ipad. Innovations, Technologies Res. Educ. (2019), 53\u201366.","journal-title":"Innovations, Technologies Res. Educ."},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.2307\/249008"},{"key":"e_1_3_3_58_2","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1007\/978-3-030-25999-0_28","volume-title":"Int. Conf. Augmented Reality, Virtual Reality and Computer Graphics","author":"Frontoni Emanuele","year":"2019","unstructured":"Emanuele Frontoni, Marina Paolanti, Mariapaola Puggioni, Roberto Pierdicca, and Michele Sasso. 2019. Measuring and assessing augmented reality potential for educational purposes: SmartMarca project. In Int. Conf. Augmented Reality, Virtual Reality and Computer Graphics. Springer, 319\u2013334."},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3051275"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.4236\/jss.2017.51004"},{"key":"e_1_3_3_61_2","volume-title":"2nd Int. Conf. Education Management Technology, Shanghai, China","author":"Muniandy Balakrishnan","year":"2011","unstructured":"Balakrishnan Muniandy, Mei Yean Ong, Kia Kien Phua, and Saw Lan Ong. 2011. User acceptance of a key performance indicators monitoring system (KPI-MS) in higher education: An application of the technology acceptance model. In 2nd Int. Conf. Education Management Technology, Shanghai, China."},{"key":"e_1_3_3_62_2","doi-asserted-by":"crossref","unstructured":"B. Muniandy S. M. Y. Ong K. K. Phua and S. L. Ong. 2011. Assessing Key Performance Indicators Monitoring System (KPI-MS) of a university using technology acceptance model. International Journal of Social Science and Humanity 1 3 (2011) 171.","DOI":"10.7763\/IJSSH.2011.V1.30"},{"issue":"3","key":"e_1_3_3_63_2","first-page":"53","article-title":"Measurement tool for assessing research information management system success","volume":"10","author":"Hasan Mahmudul","year":"2018","unstructured":"Mahmudul Hasan, Nurazean Maarop, Ganthan Narayana Samy, Roslina Mohammad, Nurulhuda Firdaus Azmi, Noor Hafizah Hassan, and Nabilah Abdul Ghaffar. 2018. Measurement tool for assessing research information management system success. J. Telecommunication, Electron. Comput. Eng. (JTEC) 10, 3-2 (2018), 53\u201357.","journal-title":"J. Telecommunication, Electron. Comput. Eng. (JTEC)"},{"issue":"10","key":"e_1_3_3_64_2","first-page":"7","article-title":"Psychometric theory\u201425 years ago and now","volume":"4","author":"Nunnally Jum C.","year":"1975","unstructured":"Jum C. Nunnally. 1975. Psychometric theory\u201425 years ago and now. Educational Researcher 4, 10 (1975), 7\u201321.","journal-title":"Educational Researcher"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2017.08.010"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.103758"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.13023"},{"key":"e_1_3_3_68_2","first-page":"1","article-title":"Virtual reality in K-12 and higher education: A systematic review of the literature from 2000 to 2019","author":"Luo Heng","year":"2021","unstructured":"Heng Luo, Gege Li, Qinna Feng, Yuqin Yang, and Mingzhang Zuo. 2021. Virtual reality in K-12 and higher education: A systematic review of the literature from 2000 to 2019. J. Comput. Assisted Learn. (2021), 1\u201315.","journal-title":"J. Comput. Assisted Learn."},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11423-020-09736-x"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.04.010"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2018.04.019"}],"container-title":["Journal on Computing and Cultural Heritage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3593432","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3593432","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:38:00Z","timestamp":1750178280000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3593432"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,24]]},"references-count":70,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6,30]]}},"alternative-id":["10.1145\/3593432"],"URL":"https:\/\/doi.org\/10.1145\/3593432","relation":{},"ISSN":["1556-4673","1556-4711"],"issn-type":[{"value":"1556-4673","type":"print"},{"value":"1556-4711","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,24]]},"assertion":[{"value":"2022-02-24","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-22","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}