{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T17:43:49Z","timestamp":1793382229112,"version":"4.5.0"},"reference-count":53,"publisher":"Public Library of Science (PLoS)","issue":"8","license":[{"start":{"date-parts":[[2023,8,2]],"date-time":"2023-08-02T00:00:00Z","timestamp":1690934400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Simulation West Network as part of Health Education England 2022"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Immersive technology is becoming more widespread in simulation-based medical education with applications that both supplement and replace traditional teaching methods. There is a lack of validated measures that capture user experience to inform of the technology utility. We aimed to establish a consensus of items and domains that different simulation experts would include in a measure for immersive technology use.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A 3-stage modified Delphi using online software was conducted to support the conceptual framework for the proposed measure. The first round was informed by prior work on immersive technology in simulation. In the first round, participants were asked to describe what we could measure in simulation-based education and technology. Thematic analysis generated key themes that were presented to the participants in the second round. Ranking of importance in round 2 was determined by mean rank scores. The final round was an online meeting for final consensus discussion and most important domains by experts were considered.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>A total of 16 simulation experts participated in the study. A consensus was reached on the ideal measure in immersive technology simulation that would be a user questionnaire and domains of interest would be: what was learnt, the degree of immersion experienced, fidelity provided, debrief, psychological safety and patient safety. No consensus was reached with the barriers that this technology introduces in education.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec004\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>There is varied opinion on what we should prioritise in measuring the experience in simulation practice. Importantly, this study identified key areas that aids our understanding on how we can measure new technology in educational settings. Synthesising these results in to a multidomain instrument requires a systematic approach to testing in future research.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0275766","type":"journal-article","created":{"date-parts":[[2023,8,2]],"date-time":"2023-08-02T13:39:04Z","timestamp":1690983544000},"page":"e0275766","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":20,"title":["Evaluating user experience with immersive technology in simulation-based education: A modified Delphi study with qualitative analysis"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3638-5454","authenticated-orcid":true,"given":"Chris","family":"Jacobs","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Georgia","family":"Foote","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5051-5921","authenticated-orcid":true,"given":"Michael","family":"Williams","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2023,8,2]]},"reference":[{"key":"pone.0275766.ref001","first-page":"2012","article-title":"Innovative Simulation Strategies in Education","author":"M Aebersold","year":"2012","journal-title":"Nursing Research and Practice"},{"issue":"3","key":"pone.0275766.ref002","doi-asserted-by":"crossref","DOI":"10.1097\/ACM.0000000000000130","article-title":"Reconsidering Fidelity in Simulation-Based Training.","volume":"89","author":"SJ Hamstra","year":"2014","journal-title":"Academic Medicine."},{"issue":"48","key":"pone.0275766.ref003","doi-asserted-by":"crossref","DOI":"10.1097\/MD.0000000000023374","article-title":"Chest compressions quality during sudden cardiac arrest scenario performed in virtual reality A crossover study in a training environment.","volume":"99","author":"F Jaskiewicz","year":"2020","journal-title":"MEDICINE."},{"issue":"3","key":"pone.0275766.ref004","first-page":"163","article-title":"Developing measures of immersion and motivation for learning technologies in healthcare simulation: a pilot study.","volume":"10","author":"C Jacobs","year":"2022","journal-title":"Journal of Advances in Medical Education & Professionalism."},{"key":"pone.0275766.ref005","doi-asserted-by":"crossref","DOI":"10.1002\/9781119604020","volume-title":"Healthcare simulation at a glance","author":"K Forrest","year":"2019"},{"key":"pone.0275766.ref006","doi-asserted-by":"crossref","unstructured":"Zsigmond I, Buhai A, editors. Augmented Reality in Medical Education, an Empirical Study. 21st International Conference on Computational Science and Its Applications (ICCSA); 2021 Sep 13\u201316; Cagliari, ITALY2021.","DOI":"10.1007\/978-3-030-87016-4_45"},{"key":"pone.0275766.ref007","first-page":"2","article-title":"Revisiting Milgram and Kishino\u2019s Reality-Virtuality Continuum.","author":"R Skarbez","year":"2021","journal-title":"Frontiers in Virtual Reality."},{"issue":"3","key":"pone.0275766.ref008","first-page":"350","article-title":"Neuroanatomy Learning: Augmented Reality vs.","volume":"13","author":"D Henssen","year":"2020","journal-title":"Cross-Sections. Anatomical Sciences Education"},{"issue":"1","key":"pone.0275766.ref009","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1007\/s40670-019-00834-x","article-title":"Mixed Reality Anatomy Using Microsoft HoloLens and Cadaveric Dissection: A Comparative Effectiveness Study.","volume":"30","author":"M Stojanovska","year":"2020","journal-title":"Medical Science Educator"},{"key":"pone.0275766.ref010","doi-asserted-by":"crossref","first-page":"100429","DOI":"10.1016\/j.edurev.2021.100429","article-title":"A systematic review of immersive technology applications for medical practice and education\u2014Trends, application areas, recipients, teaching contents, evaluation methods, and performance.","volume":"35","author":"YM Tang","year":"2022","journal-title":"Educational Research Review"},{"key":"pone.0275766.ref011","author":"C. Jacobs","year":"2022","journal-title":"Immersive technology in healthcare education: a scoping review osf.io\/tpjyw"},{"issue":"9","key":"pone.0275766.ref012","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1037\/0003-066X.50.9.741","article-title":"Validity of psychological assessment: Validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning","volume":"50","author":"S. Messick","year":"1995","journal-title":"American Psychologist."},{"issue":"1","key":"pone.0275766.ref013","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1186\/s41077-016-0033-y","article-title":"Validation of educational assessments: a primer for simulation and beyond.","volume":"1","author":"DA Cook","year":"2016","journal-title":"Advances in Simulation"},{"key":"pone.0275766.ref014","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5116\/ijme.5e01.eb1a","article-title":"Augmented reality and mixed reality for healthcare education beyond surgery: an integrative review","volume":"11","author":"J Gerup","year":"2020","journal-title":"Int J Med Educ"},{"key":"pone.0275766.ref015","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.techfore.2019.07.002","article-title":"Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process","volume":"147","author":"I Belton","year":"2019","journal-title":"Technological Forecasting and Social Change"},{"issue":"1","key":"pone.0275766.ref016","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1186\/s12874-016-0165-8","article-title":"Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology.","volume":"16","author":"BH Eubank","year":"2016","journal-title":"BMC Medical Research Methodology"},{"issue":"6","key":"pone.0275766.ref017","doi-asserted-by":"crossref","first-page":"1485","DOI":"10.1068\/a39267","article-title":"Using the Delphi Technique in Normative Planning Research: Methodological Design Considerations.","volume":"40","author":"N Novakowski","year":"2008","journal-title":"Environment and Planning A: Economy and Space."},{"issue":"1","key":"pone.0275766.ref018","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.amjsurg.2019.10.017","article-title":"Priorities in surgical simulation research: What do the experts say?","volume":"220","author":"NE Anton","year":"2020","journal-title":"The American Journal of Surgery"},{"key":"pone.0275766.ref019","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.ecns.2020.09.001","article-title":"2020 to 2023 Research Priorities Advance INACSL Core Values","volume":"47","author":"A Franklin","year":"2020","journal-title":"Clin Simul Nurs"},{"key":"pone.0275766.ref020","article-title":"Enhancing education, clinical practice and staff wellbeing.","author":"B Baxendale","year":"2020","journal-title":"A national vision for the role of simulation and immersive learning technologies in health and care. Health Education England"},{"issue":"4","key":"pone.0275766.ref021","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1111\/tct.13338","article-title":"A remote access mixed reality teaching ward round.","volume":"18","author":"L Bala","year":"2021","journal-title":"CLINICAL TEACHER"},{"key":"pone.0275766.ref022","first-page":"2","article-title":"A Postphenomenological Framework for Studying User Experience of Immersive Virtual Reality.","author":"J Vindenes","year":"2021","journal-title":"Frontiers in Virtual Reality."},{"issue":"3","key":"pone.0275766.ref023","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1007\/s10055-006-0035-5","article-title":"The virtual playground: an educational virtual reality environment for evaluating interactivity and conceptual learning.","volume":"10","author":"M Roussou","year":"2006","journal-title":"Virtual Reality"},{"issue":"3","key":"pone.0275766.ref024","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1287\/mnsc.9.3.458","article-title":"An experimental application of the Delphi method to the use of experts","volume":"9","author":"N Dalkey","year":"1963","journal-title":"Management science"},{"issue":"2","key":"pone.0275766.ref025","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/S0020-7489(00)00044-4","article-title":"A critical review of the Delphi technique as a research methodology for nursing","volume":"38","author":"S Keeney","year":"2001","journal-title":"International Journal of Nursing Studies"},{"issue":"7","key":"pone.0275766.ref026","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1080\/13611260500069947","article-title":"The Delphi technique in health sciences education research.","volume":"27","author":"MR de Villiers","year":"2005","journal-title":"Medical Teacher."},{"issue":"5","key":"pone.0275766.ref027","doi-asserted-by":"crossref","first-page":"e011780","DOI":"10.1136\/bmjopen-2016-011780","article-title":"Moving forward through consensus: protocol for a modified Delphi approach to determine the top research priorities in the field of orthopaedic oncology","volume":"6","author":"P Schneider","year":"2016","journal-title":"BMJ Open"},{"issue":"3","key":"pone.0275766.ref028","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1177\/1049732317697102","article-title":"A Guide to Field Notes for Qualitative Research: Context and Conversation.","volume":"28","author":"J Phillippi","year":"2018","journal-title":"Qualitative Health Research."},{"issue":"9","key":"pone.0275766.ref029","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1136\/emj.2006.037309","article-title":"Determination of required anatomical knowledge for clinical practice in emergency medicine: national curriculum planning using a modified Delphi technique","volume":"23","author":"D Kilroy","year":"2006","journal-title":"Emergency Medicine Journal"},{"issue":"4","key":"pone.0275766.ref030","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1007\/s10755-006-9022-8","article-title":"Idea Generation and Exploration: Benefits and Limitations of the Policy Delphi Research Method.","volume":"31","author":"KK Franklin","year":"2007","journal-title":"Innovative Higher Education."},{"key":"pone.0275766.ref031","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1007\/978-0-306-47630-3_7","volume-title":"Principles of Forecasting: A Handbook for Researchers and Practitioners.","author":"G Rowe","year":"2001"},{"key":"pone.0275766.ref032","first-page":"1","article-title":"Learning from 360-degree film in healthcare simulation: a mixed methods pilot","author":"C Jacobs","year":"2022","journal-title":"Journal of Visual Communication in Medicine"},{"issue":"3","key":"pone.0275766.ref033","doi-asserted-by":"crossref","first-page":"152","DOI":"10.3163\/1536-5050.103.3.010","article-title":"Bloom\u2019s taxonomy of cognitive learning objectives.","volume":"103","author":"NE Adams","year":"2015","journal-title":"J Med Libr Assoc"},{"issue":"1","key":"pone.0275766.ref034","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1037\/1524-9220.5.1.40","article-title":"Clothing and embodiment: Men managing body image and appearance.","volume":"5","author":"H Frith","year":"2004","journal-title":"Psychology of men & masculinity."},{"key":"pone.0275766.ref035","first-page":"1","article-title":"The online survey as a qualitative research tool","author":"V Braun","year":"2020","journal-title":"International Journal of Social Research Methodology"},{"key":"pone.0275766.ref036","doi-asserted-by":"crossref","first-page":"26152","DOI":"10.3402\/qhw.v9.26152","article-title":"What can \"thematic analysis\" offer health and wellbeing researchers?","volume":"9","author":"V Braun","year":"2014","journal-title":"International journal of qualitative studies on health and well-being"},{"issue":"1","key":"pone.0275766.ref037","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2431-6-1","article-title":"Identification of pain indicators for infants at risk for neurological impairment: A Delphi consensus study","volume":"6","author":"B Stevens","year":"2006","journal-title":"BMC Pediatrics"},{"key":"pone.0275766.ref038","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","author":"JR Landis","year":"1977","journal-title":"biometrics"},{"key":"pone.0275766.ref039","first-page":"2020","article-title":"Security guide. Zoom Video Communications Inc","volume":"9","author":"Z. Inc","year":"2016","journal-title":"Retrieved October"},{"key":"pone.0275766.ref040","doi-asserted-by":"crossref","first-page":"1609406919874596","DOI":"10.1177\/1609406919874596","article-title":"Using Zoom Videoconferencing for Qualitative Data Collection: Perceptions and Experiences of Researchers and Participants.","volume":"18","author":"MM Archibald","year":"2019","journal-title":"International Journal of Qualitative Methods"},{"issue":"6","key":"pone.0275766.ref041","doi-asserted-by":"crossref","first-page":"e20476","DOI":"10.1371\/journal.pone.0020476","article-title":"Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review.","volume":"6","author":"R Boulkedid","year":"2011","journal-title":"PLOS ONE."},{"key":"pone.0275766.ref042","first-page":"47","article-title":"The Delphi Technique: Methodological Considerations and the Need for Reporting Guidelines in Medical Journals.","volume":"4","author":"IP Catalin Toma","year":"2016","journal-title":"International Journal of Public Health Research"},{"issue":"1","key":"pone.0275766.ref043","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1186\/s12874-019-0886-6","article-title":"A modified Delphi study to identify the features of high quality measurement plans for healthcare improvement projects.","volume":"20","author":"T Woodcock","year":"2020","journal-title":"BMC Medical Research Methodology"},{"key":"pone.0275766.ref044","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.chb.2018.04.019","article-title":"The state of immersive technology research: A literature analysis.","volume":"86","author":"A Suh","year":"2018","journal-title":"Computers in Human Behavior"},{"issue":"7","key":"pone.0275766.ref045","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1016\/j.jclinepi.2010.02.006","article-title":"The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes","volume":"63","author":"LB Mokkink","year":"2010","journal-title":"Journal of Clinical Epidemiology"},{"key":"pone.0275766.ref046","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1037\/h0040957","article-title":"Construct validity in psychological tests.","volume":"52","author":"LJ Cronbach","year":"1955","journal-title":"Psychological Bulletin"},{"issue":"7","key":"pone.0275766.ref047","first-page":"598","article-title":"The Delphi Method.","volume":"22","author":"S Iqbal","year":"2009","journal-title":"The Psychologist"},{"issue":"4","key":"pone.0275766.ref048","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1080\/0144341970170401","article-title":"Delphi: a technique to harness expert opinion for critical decision\u2010making tasks in education.","volume":"17","author":"MJ Clayton","year":"1997","journal-title":"Educational Psychology"},{"issue":"1","key":"pone.0275766.ref049","doi-asserted-by":"crossref","first-page":"e173","DOI":"10.1111\/eje.12000","article-title":"A Delphi approach to define learning outcomes and assessment","volume":"17","author":"I Tonni","year":"2013","journal-title":"European Journal of Dental Education"},{"issue":"2222\u20130682 (Print","key":"pone.0275766.ref050","author":"P Nasa","year":"2021","journal-title":"Delphi methodology in healthcare research: How to decide its appropriateness"},{"issue":"2","key":"pone.0275766.ref051","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1177\/0890334419830990","article-title":"Reflexivity in Qualitative Research.","volume":"35","author":"JE Dodgson","year":"2019","journal-title":"Journal of Human Lactation"},{"issue":"5","key":"pone.0275766.ref052","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1177\/1468794113488126","article-title":"Skype interviewing: Reflections of two PhD researchers.","volume":"14","author":"H Deakin","year":"2014","journal-title":"Qualitative research."},{"issue":"10","key":"pone.0275766.ref053","doi-asserted-by":"crossref","first-page":"1187","DOI":"10.1287\/mnsc.28.10.1187","article-title":"Group Confidence Pressures in Iterative Decisions.","volume":"28","author":"DM Boje","year":"1982","journal-title":"Management Science"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0275766","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,2]],"date-time":"2023-08-02T13:39:52Z","timestamp":1690983592000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0275766"}},"subtitle":[],"editor":[{"given":"Soham","family":"Bandyopadhyay","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2023,8,2]]},"references-count":53,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,8,2]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0275766","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.09.26.509545","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,2]]}}}