{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T12:25:31Z","timestamp":1776083131150,"version":"3.50.1"},"reference-count":42,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,9,1]],"date-time":"2025-09-01T00:00:00Z","timestamp":1756684800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Advanced Knowledge Center in Immersive Technologies (AKCIT)","award":["308783\/2020-4"],"award-info":[{"award-number":["308783\/2020-4"]}]},{"name":"Bahia State Research Support Foundation (FAPESB)","award":["308783\/2020-4"],"award-info":[{"award-number":["308783\/2020-4"]}]},{"name":"National Council for Scientific and Technological Development","award":["308783\/2020-4"],"award-info":[{"award-number":["308783\/2020-4"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>This study presents preliminary design guidelines to support the evaluation of industrial safety training using immersive technologies, with a focus on high-risk work environments such as working at height. Although virtual reality has been widely adopted for training, few studies have explored its use for behavior-level evaluation, corresponding to Level 3 of the Kirkpatrick Model. Addressing this gap, the study adopts the Design Science Research methodology, combining a systematic literature review with expert focus group analysis to develop a conceptual framework for training evaluation. The results identify key elements necessary for immersive training evaluations, including scenario configuration, ethical procedures, recruitment, equipment selection, experimental design, and implementation strategies. The resulting guidelines are organized into six categories: scenario configuration, ethical procedures, recruitment, equipment selection, experimental design, and implementation strategies. These guidelines represent a DSR-based conceptual artifact to inform future empirical studies and support the structured assessment of immersive safety training interventions. The study also highlights the potential of integrating behavioral and physiological indicators to support immersive evaluations of behavioral change, offering an expert-informed and structured foundation for future empirical studies in high-risk industrial contexts.<\/jats:p>","DOI":"10.3390\/informatics12030088","type":"journal-article","created":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T15:02:57Z","timestamp":1756825377000},"page":"88","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Preliminary Design Guidelines for Evaluating Immersive Industrial Safety Training"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-8359-3886","authenticated-orcid":false,"given":"Andr\u00e9","family":"Cordeiro","sequence":"first","affiliation":[{"name":"Institute of Pharmaceutical Technology, Oswaldo Cruz Foundation\u2014Fiocruz, Rio de Janeiro 21040-360, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5388-3092","authenticated-orcid":false,"given":"Regina","family":"Leite","sequence":"additional","affiliation":[{"name":"Construction Department, Federal Institute of Bahia, Salvador 40110-150, Brazil"}]},{"given":"Lucas","family":"Almeida","sequence":"additional","affiliation":[{"name":"PGGDDesign, Federal University of Paran\u00e1, Curitiba 80320-230, Brazil"}]},{"given":"Cintia","family":"Neves","sequence":"additional","affiliation":[{"name":"Institute for Science, Innovation and Technology in Industry 4.0 (INCITE INDUSTRIA 4.0), SENAI CIMATEC University, Salvador 41650-010, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5065-7938","authenticated-orcid":false,"given":"Tiago","family":"Silva","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Industrial Engineering (DEMI), NOVA School of Science and Technology (FCT), NOVA University Lisbon, 2829-516 Caparica, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9213-9602","authenticated-orcid":false,"given":"Alexandre","family":"Siqueira","sequence":"additional","affiliation":[{"name":"College of Engineering, University of Florida, Gainesville, FL 32611, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1039-3939","authenticated-orcid":false,"given":"Marcio","family":"Catapan","sequence":"additional","affiliation":[{"name":"PGGDDesign, Federal University of Paran\u00e1, Curitiba 80320-230, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6505-6636","authenticated-orcid":false,"given":"Ingrid","family":"Winkler","sequence":"additional","affiliation":[{"name":"Institute for Science, Innovation and Technology in Industry 4.0 (INCITE INDUSTRIA 4.0), SENAI CIMATEC University, Salvador 41650-010, Brazil"},{"name":"Advanced Knowledge Center in Immersive Technology (AKCIT), Goiania 574001-970, Brazil"}]}],"member":"1968","published-online":{"date-parts":[[2025,9,1]]},"reference":[{"key":"ref_1","unstructured":"International Labour Organization (ILO) (2023). Relevant SDG Targets Related to Health and Safety at the Workplace, International Labour Organization."},{"key":"ref_2","unstructured":"Tribunal Superior do Trabalho (TST) (2022). Acidentes de Trabalho Matam ao Menos Uma Pessoa a Cada 3 h 47 min No Brasil, Tribunal Superior do Trabalho."},{"key":"ref_3","unstructured":"SmartLab (2025, July 03). Observat\u00f3rio de Seguran\u00e7a e Sa\u00fade no Trabalho. Available online: https:\/\/smartlabbr.org\/."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"105678","DOI":"10.1016\/j.ssci.2022.105678","article-title":"Transfer study of safety training based on mapping knowledge domain\u2014Overview, factors and future","volume":"148","author":"Chen","year":"2022","journal-title":"Saf. Sci."},{"key":"ref_5","unstructured":"Kirkpatrick, D.L., and Kirkpatrick, W.K. (2016). Kirkpatrick\u2019s Four Levels of Training Evaluation, Association for Talent Development."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/j.ssci.2019.03.002","article-title":"Formulating a learner model for evaluating construction workers\u2019 learning ability during safety training","volume":"116","author":"Zhang","year":"2019","journal-title":"Saf. Sci."},{"key":"ref_7","unstructured":"Kirkpatrick, D.L., and Kirkpatrick, J.D. (2006). Evaluating Training Programs, Tata McGraw-Hill."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1080\/10494820.2019.1587636","article-title":"The use of a virtual reality training system to improve technical skill in the maintenance of live-line power distribution networks","volume":"29","author":"Ayala","year":"2021","journal-title":"Interact. Learn. Environ."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"919","DOI":"10.1007\/s10055-020-00498-8","article-title":"A review of the evidence for the effectiveness of training with virtual reality technology","volume":"25","author":"Abich","year":"2021","journal-title":"Virtual Real."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Jerald, J. (2015). The VR Book: Human-Centered Design for Virtual Reality, ACM & Morgan & Claypool.","DOI":"10.1145\/2792790"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"133","DOI":"10.12753\/2066-026X-15-020","article-title":"A literature review on immersive virtual reality in education: State of the art and perspectives","volume":"1","author":"Freina","year":"2015","journal-title":"Elearning Softw. Educ."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1127","DOI":"10.1111\/jcal.12670","article-title":"Virtual reality enhances safety training in the maritime industry: An organizational training experiment with a non-WEIRD sample","volume":"38","author":"Makransky","year":"2022","journal-title":"J. Comput. Assist. Learn."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Henriques, A.C., and Winkler, I. (2021). The advancement of virtual reality in automotive market research: Challenges and opportunities. Appl. Sci., 11.","DOI":"10.3390\/app112411610"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Checa, D., Saucedo-Dorantes, J.J., Osornio-Rios, R.A., Antonino-Daviu, J.A., and Bustillo, A. (2022). Virtual reality training application for the condition-based maintenance of induction motors. Appl. Sci., 12.","DOI":"10.3390\/app12010414"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Cordeiro, A., Leite, R., Gregory, L., Catapan, M., Siqueira, A., Silva, T., Brand\u00e3o, M., Ambrosio, P., Reis, P., and Winkler, I. (2025, January 27\u201329). Immersive Technologies to Height Safety: Training Evaluation and Insights. Proceedings of the 2025 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR), Lisbon, Portugal.","DOI":"10.1109\/AIxVR63409.2025.00082"},{"key":"ref_16","unstructured":"Creswell, J.W., and Creswell, J.D. (2017). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, SAGE Publications."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"337","DOI":"10.25300\/MISQ\/2013\/37.2.01","article-title":"Positioning and presenting design science research for maximum impact","volume":"37","author":"Gregor","year":"2013","journal-title":"MIS Q."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Offermann, P., Blom, S., Sch\u00f6nherr, M., and Bub, U. (2010). Artifact types in information systems design science\u2014A literature review. Global Perspectives on Design Science Research: DESRIST 2010, St. Gallen, Switzerland, Springer.","DOI":"10.1007\/978-3-642-13335-0_6"},{"key":"ref_19","first-page":"564","article-title":"Effectiveness of industrial training using virtual reality to mitigate risks associated with the work environment: A literature review","volume":"10","author":"Cordeiro","year":"2023","journal-title":"Blucher Eng. Proc."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1007\/s10055-022-00685-9","article-title":"The sentiment of a virtual rock concert","volume":"27","author":"Slater","year":"2023","journal-title":"Virtual Real."},{"key":"ref_21","unstructured":"Samsung (2025, August 27). Galaxy Watch7\u2014Health Monitoring Specifications. Available online: https:\/\/www.samsung.com\/us\/watches\/galaxy-watch7\/."},{"key":"ref_22","unstructured":"Pico, X.R. (2025, July 14). Pico 4 Enterprise Specifications. Available online: https:\/\/www.picoxr.com\/global\/products\/pico4-enterprise."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1287\/mnsc.46.2.186.11926","article-title":"A theoretical extension of the technology acceptance model: Four longitudinal field studies","volume":"46","author":"Venkatesh","year":"2000","journal-title":"Manag. Sci."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"425","DOI":"10.2307\/30036540","article-title":"User acceptance of information technology: Toward a unified view","volume":"27","author":"Venkatesh","year":"2003","journal-title":"MIS Q."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1162\/105474605323384654","article-title":"The factor structure of the presence questionnaire","volume":"14","author":"Witmer","year":"2005","journal-title":"Presence"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1007\/s10055-016-0295-7","article-title":"A questionnaire for measuring presence in virtual environments: Factor analysis and Brazilian adaptation","volume":"20","author":"Silva","year":"2016","journal-title":"Virtual Real."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"349","DOI":"10.3389\/fpsyg.2020.00349","article-title":"Questionnaire Measures and Physiological Correlates of Presence: A Systematic Review","volume":"11","author":"Grassini","year":"2020","journal-title":"Front. Psychol."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Archer, N.S., Bluff, A., Eddy, A., Nikhil, C.K., Hazell, N., Frank, D., and Johnston, A. (2022). Odour enhances the sense of presence in a virtual reality environment. PLoS ONE, 17.","DOI":"10.1371\/journal.pone.0265039"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1007\/s10055-024-00944-x","article-title":"Design and emotional responses: Is there coherence between what is said and what is felt? A study using biofeedback and virtual reality","volume":"28","author":"Tavares","year":"2024","journal-title":"Virtual Real."},{"key":"ref_30","first-page":"538","article-title":"Physiological reaction as an objective measure of presence in virtual environments","volume":"11","author":"Meehan","year":"2002","journal-title":"Presence"},{"key":"ref_31","unstructured":"Spanlang, B., Fr\u00f6hlich, T., Descalzo, F.F., Antley, A., and Slater, M. (2007, January 25\u201327). The Making of a Presence Experiment: Responses to Virtual Fire. Proceedings of the 10th Annual International Workshop on Presence, Barcelona, Spain."},{"key":"ref_32","unstructured":"Narciso, D.G. (2022). Realistic Multisensory Virtual Reality Simulator for Firefighter Training: Comparing the Physiological Response of Trainees in Real and Virtual Environments. [Ph.D. Thesis, Universidade De Tr\u00e1s-Os-Montes e Alto Douro]."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1264","DOI":"10.1177\/0269881119860156","article-title":"It feels real: Physiological responses to a stressful virtual reality environment and its impact on working memory","volume":"33","author":"Martens","year":"2019","journal-title":"J. Psychopharmacol."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Slater, M., Banakou, D., Beacco, A., Gallego, J., Macia-Varela, F., and Oliva, R. (2022). A Separate Reality: An Update on Place Illusion and Plausibility in Virtual Reality. Front. Virtual Real., 3.","DOI":"10.3389\/frvir.2022.914392"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Llinares, C., Higuera-Trujillo, J.L., and Monta\u00f1ana, A. (2024). A Comparative Study of Real and Virtual Environment via Psychological and Physiological Responses. Appl. Sci., 14.","DOI":"10.3390\/app14010232"},{"key":"ref_36","first-page":"271","article-title":"Gamified immersive safety training in virtual reality: A case from the railway industry","volume":"36","author":"Katende","year":"2024","journal-title":"J. Workplace Learn."},{"key":"ref_37","first-page":"1205","article-title":"Effects of point-and-level based gamification on motivation and performance in a VR assembly task","volume":"38","author":"Lee","year":"2022","journal-title":"J. Comput. Assist. Learn."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Carnell, S., De Siqueira, A.G., Miles, A., and Lok, B. (2022). Informing and evaluating educational applications with the Kirkpatrick model in virtual environments: Using a virtual human scenario to measure communication skills behavior change. Front. Virtual Real., 3.","DOI":"10.3389\/frvir.2022.810797"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Winkler, I., Ambr\u00f3sio, P.E., Leite, R.M.C., Cordeiro, A.M., Almeida, L.G.G., Thasla, Y., Siqueira, A.G., Catapan, M.F., and Berretta, L.O. (2025, January 9\u201313). Biofeedback na avalia\u00e7\u00e3o da experi\u00eancia do usu\u00e1rio em ambientes imersivos. Proceedings of the Anais do 25\u00ba Simp\u00f3sio Brasileiro de Computa\u00e7\u00e3o Aplicada \u00e0 Sa\u00fade\u2014SBCAS 2025, Porto Alegre, Brazil.","DOI":"10.5753\/sbc.16731.5.5"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1016\/j.learninstruc.2017.12.007","article-title":"Adding immersive virtual reality to a science lab simulation causes more presence but less learning","volume":"60","author":"Makransky","year":"2019","journal-title":"Learn. Instr."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Nasri, M., Kosa, M., Chukoskie, L., Moghaddam, M., and Harteveld, C. (2024, January 21\u201325). Exploring eye tracking to detect cognitive load in complex virtual reality training. Proceedings of the 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Bellevue, WA, USA.","DOI":"10.1109\/ISMAR-Adjunct64951.2024.00022"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Doolani, S., Wessels, C., Kanal, V., Sevastopoulos, C., Jaiswal, A., Nambiappan, H., and Makedon, F. (2020). A review of extended reality (XR) technologies for manufacturing training. Technologies, 8.","DOI":"10.3390\/technologies8040077"}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/12\/3\/88\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:37:28Z","timestamp":1760035048000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/12\/3\/88"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,1]]},"references-count":42,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["informatics12030088"],"URL":"https:\/\/doi.org\/10.3390\/informatics12030088","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,1]]}}}