{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T13:06:51Z","timestamp":1788959211936,"version":"build-2803163510"},"reference-count":122,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"\n Background<\/jats:title>\n Clinical educators regularly use simulation-based education to help learners develop clinical reasoning, procedural skills, and communication strategies. However, differences between simulated and live clinical environments, such as missing case details or workflow interruptions, can reduce realism and divert learners\u2019 attention from the intended learning goals. While published design guidelines recommend pilot testing simulations, they do not explain how to systematically identify usability problems. Human factors evaluation methods can address this gap by identifying issues before piloting and providing design insights.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This tutorial describes our usability testing protocol for evaluating educational simulations before piloting with learners. Using a telemedicine maternal care simulation as an example, we show how mixed methods usability testing can identify technical, workflow, and assessment-related issues in simulation design.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We recruited obstetricians and family practitioners to test our simulation. To measure multiple usability dimensions concurrently, we used several published instruments, including an Agile Task Analysis (ATA), the Single Ease Question (SEQ), an adapted System Usability Scale (SUS), the Satisfaction With Simulation Experience Scale (SSES), and the NASA (National Aeronautics and Space Administration) Task Load Index (NASA-TLX). We also developed a competency assessment rubric to measure telemedicine competencies. We recorded observations on the ATA while testers used a think-aloud protocol. We then debriefed testers and administered the posttest questionnaires. We summarized quantitative data using descriptive statistics and analyzed qualitative data using theory-based deductive coding with published usability heuristics.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Twelve testers identified 52 usability issues over 2 testing cycles; 15 issues required correction before piloting with learners. Revisions included changes to standardized patient scripts, synthetic patient data, and scoring rubric instructions. Testers assigned a median SEQ score of 6 or higher to 4 of the 6 workflow steps; most struggled with unfamiliar telemedicine tasks or completing a virtual physical examination. SUS scores ranged from 65 to 100. The median SUS score was 77.5, indicating above-average usability. The NASA-TLX subscale scores were highest for mental workload (median 60, IQR 56.3-63.8) compared with other subscales, such as physical or temporal workload (median 15.00, IQR 11.3-18.8 and median 20.00, IQR 11.3-43.8, respectively), suggesting the simulation imposed a high mental demand. The overall SSES score of 43.5 out of 50 suggested faculty testers thought the simulation offered valuable learning opportunities.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Following usability testing, we successfully piloted the simulation with 23 obstetrics and family medicine residents without encountering any implementation issues. Incorporating usability evaluation methods throughout the simulation development life cycle helped identify and mitigate design problems that might otherwise limit simulation fidelity and instructional effectiveness. Our tutorial showcases a modular mixed methods approach that can be adapted to study a range of simulation types and research questions related to workflow, technology, and learning objectives.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/102697","type":"journal-article","created":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T16:30:08Z","timestamp":1785169808000},"page":"e102697","source":"Crossref","is-referenced-by-count":0,"title":["How to Conduct Usability Testing of Medical Education Simulations: Methods Tutorial Using a Telemedicine Simulation for Maternal Care"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6170-7964","authenticated-orcid":false,"given":"Blake J","family":"Lesselroth","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0047-4447","authenticated-orcid":false,"given":"Alexander","family":"Douglas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0772-9075","authenticated-orcid":false,"given":"Helen","family":"Monkman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7077-7267","authenticated-orcid":false,"given":"Romaric","family":"Marcilly","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-8676-4243","authenticated-orcid":false,"given":"Karalane","family":"Bellavia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6825-7640","authenticated-orcid":false,"given":"Mataeo","family":"Anderson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1931-6588","authenticated-orcid":false,"given":"Jacob","family":"Van Buren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-1258-5225","authenticated-orcid":false,"given":"Charles","family":"Parsons","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3587-0774","authenticated-orcid":false,"given":"Sonakshi","family":"Mohanty","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2606-0813","authenticated-orcid":false,"given":"Juell","family":"Homco","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1428-6919","authenticated-orcid":false,"given":"Morgan","family":"Richards","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-8387-6012","authenticated-orcid":false,"given":"Karen P","family":"Gold","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,9]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1097\/MD.0000000000038813"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecns.2021.08.009"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1136\/qshc.2009.038562"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1177\/1046878109355683"},{"key":"ref5","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-43826-5","author":"Gliva-McConvey, G","year":"2020","journal-title":"Comprehensive Healthcare Simulation: Implementing Best Practices in Standardized Patient Methodology"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1136\/bmjstel-2017-000232"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.54531\/nyvm5886"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.6434"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1186\/s41077-017-0043-4"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1097\/SIH.0b013e3180315539"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare12141417"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1097\/SIH.0b013e3180f637f5"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare11030296"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1007\/s44186-023-00127-3"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1007\/s42600-024-00340-z"},{"key":"ref16","first-page":"33","year":"2019","journal-title":"Ergonomics of Human-System Interaction\u2014Part 210: Human-Centred Design for Interactive Systems"},{"key":"ref17","author":"Barnum, CM","year":"2010","journal-title":"Usability Testing Essentials: Ready, Set...Test!"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2022.2093704"},{"key":"ref19","first-page":"4823","author":"Shackel, B","year":"1984","journal-title":"Visual Display Terminals: Usability Issues and Health Concerns"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI250242"},{"key":"ref21","author":"Nielsen, J","year":"1994","journal-title":"Usability Engineering. 1st Edition"},{"key":"ref22","author":"Harrington, L","year":"2014","journal-title":"Usability Evaluation Handbook for Electronic Health Records"},{"key":"ref23","author":"Albert, W","year":"2013","journal-title":"Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.1089\/tmj.2020.29040.rb"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1055\/s-0041-1726484"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1007\/s13721-021-00300-y"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocac250"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1177\/21501327211019286"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1016\/j.prrv.2020.06.009"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.2196\/29099"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1016\/j.pop.2022.04.003"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1089\/tmj.2019.0261"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1016\/j.hjdsi.2021.100522"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-022-07564-8"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.2196\/12515"},{"key":"ref36","unstructured":"Telehealth competenciesAssociation of American Medical Colleges2026-08-14https:\/\/www.aamc.org\/data-reports\/report\/telehealth-competencies"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.15766\/mep_2374-8265.10867"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.2433"},{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.15766\/mep_2374-8265.11047"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1016\/j.hjdsi.2025.100768"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI250327"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI250237"},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI260637"},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.1001\/jamahealthforum.2022.5541"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.1186\/s12884-024-06535-7"},{"key":"ref46","doi-asserted-by":"publisher","DOI":"10.1097\/AOG.0000000000005976"},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.1016\/j.focus.2025.100362"},{"key":"ref48","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.p1878"},{"key":"ref49","doi-asserted-by":"publisher","DOI":"10.3122\/jabfm.2024.240198R1"},{"key":"ref50","first-page":"93","author":"Quesenbery, W","year":"2014","journal-title":"Content and Complexity: Information Design in Technical Communication"},{"key":"ref51","author":"Davis, FD","year":"1985","journal-title":"A Technology Acceptance Model for Empirically Testing New End-User Information Systems: Theory and Results"},{"key":"ref52","author":"Sauro, J","year":"2011","journal-title":"A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices"},{"key":"ref53","doi-asserted-by":"publisher","DOI":"10.1016\/j.nedt.2011.01.004"},{"key":"ref54","author":"Hart, SG","year":"1986","journal-title":"NASA Task Load Index (TLX): Paper and Pencil Package, Version 1.0"},{"key":"ref55","doi-asserted-by":"publisher","DOI":"10.1080\/00140130903067797"},{"key":"ref56","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.32917"},{"key":"ref57","author":"Lesselroth, BJ","year":"2020","journal-title":"Patient-Centered Digital Healthcare Technology: Novel Applications for Next Generation Healthcare Systems"},{"key":"ref58","first-page":"479","author":"Kirwan, B","year":"2020","journal-title":"Contemporary Ergonomics"},{"key":"ref59","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1201\/9780203489925-48","author":"Stanton, NA","year":"2004","journal-title":"Handbook of Human Factors and Ergonomics Methods"},{"key":"ref60","doi-asserted-by":"publisher","DOI":"10.1590\/1981-5271v43suplemento1-20190238.ING"},{"key":"ref61","doi-asserted-by":"publisher","DOI":"10.3946\/kjme.2018.76"},{"key":"ref62","unstructured":"SauroJDumasJComparison of three one-question, post-task usability questionnaires2009Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2009)April 4-9, 2009Boston, Massachusetts15991608"},{"key":"ref63","unstructured":"SauroJLewisJHow the SEQ correlates with other task metricsMeasuringU20262026-02-05https:\/\/measuringu.com\/how-the-seq-correlates-with-other-task-metrics\/"},{"key":"ref64","unstructured":"SauroJ10 things to know about the Single Ease Question (SEQ)MeasuringU20122026-02-04https:\/\/measuringu.com\/seq10\/"},{"key":"ref65","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2025.104822"},{"key":"ref66","first-page":"4","author":"Brooke, J","year":"1996","journal-title":"Usability Evaluation in Industry"},{"key":"ref67","unstructured":"LewisJRSauroJThe factor structure of the System Usability Scale2009First International Conference on Human Centered Design (HCD 2009)July 19-24, 2009San Diego, California94103"},{"key":"ref68","doi-asserted-by":"publisher","DOI":"10.5430\/jnep.v2n3p74"},{"key":"ref69","doi-asserted-by":"publisher","DOI":"10.1177\/154193120605000909"},{"key":"ref70","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2014.956151"},{"key":"ref71","doi-asserted-by":"publisher","DOI":"10.1177\/1541931215591373"},{"key":"ref72","first-page":"702","volume":"2010","author":"Russ, AL","year":"2010","journal-title":"AMIA Annu Symp Proc"},{"key":"ref73","author":"Krug, S","year":"2009","journal-title":"Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems"},{"key":"ref74","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2004.01.003"},{"key":"ref75","doi-asserted-by":"publisher","DOI":"10.1006\/jbin.2001.1021"},{"key":"ref76","unstructured":"ZoomZoom Video Communications, Inc20132026-07-29https:\/\/zoom.us\/"},{"key":"ref77","doi-asserted-by":"publisher","DOI":"10.3233\/shti250255"},{"key":"ref78","unstructured":"LearningSpaceCAE Healthcare2026-08-14https:\/\/www.caehealthcare.com\/learningspace\/"},{"key":"ref79","author":"Saldana, J","year":"2011","journal-title":"Fundamentals of Qualitative Research"},{"key":"ref80","doi-asserted-by":"publisher","DOI":"10.1016\/s1532-0464(03)00060-1"},{"key":"ref81","author":"Shneiderman, B","year":"2010","journal-title":"Designing the User Interface: Strategies for Effective Human-Computer Interaction"},{"key":"ref82","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2011.08.005"},{"key":"ref83","doi-asserted-by":"publisher","DOI":"10.1055\/s-0038-1638836"},{"key":"ref84","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI200327"},{"key":"ref85","doi-asserted-by":"publisher","DOI":"10.1007\/s10648-023-09819-0"},{"key":"ref86","doi-asserted-by":"publisher","DOI":"10.1080\/07853890.2024.2421441"},{"key":"ref87","author":"Sauro, J","year":"2016","journal-title":"Quantifying the User Experience: Practical Statistics for User Research"},{"key":"ref88","doi-asserted-by":"publisher","DOI":"10.1080\/01421590500046924"},{"key":"ref89","doi-asserted-by":"publisher","DOI":"10.1111\/j.1743-498X.2007.00161.x"},{"key":"ref90","doi-asserted-by":"crossref","DOI":"10.1201\/9781482269505","author":"Carayon, P","year":"2006","journal-title":"Handbook of Human Factors and Ergonomics in Health Care and Patient Safety"},{"key":"ref91","doi-asserted-by":"publisher","DOI":"10.1016\/j.apergo.2013.04.023"},{"key":"ref92","author":"Kohn, LT","year":"2000","journal-title":"To Err is Human: Building a Safer Health System"},{"key":"ref93","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI230365"},{"key":"ref94","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI220582"},{"key":"ref95","doi-asserted-by":"publisher","DOI":"10.1016\/j.actpsy.2025.105075"},{"key":"ref96","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2016.1246710"},{"key":"ref97","doi-asserted-by":"publisher","DOI":"10.1111\/medu.14941"},{"key":"ref98","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecns.2017.05.015"},{"key":"ref99","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI260636"},{"key":"ref100","unstructured":"NielsenJDiscount usability: 20 yearsNielsen Norman Group20092009-02-06https:\/\/www.nngroup.com\/articles\/discount-usability-20-years\/"},{"key":"ref101","doi-asserted-by":"publisher","DOI":"10.2196\/21161"},{"key":"ref102","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2019.1700644"},{"issue":"3","key":"ref103","first-page":"178","volume":"23","author":"Xiao, YM","year":"2005","journal-title":"Zhonghua Lao Dong Wei Sheng Zhi Ye Bing Za Zhi"},{"key":"ref104","doi-asserted-by":"publisher","DOI":"10.1111\/j.1464-0597.2004.00161.x"},{"key":"ref105","doi-asserted-by":"publisher","DOI":"10.14408\/kjems.2014.18.2.007"},{"key":"ref106","doi-asserted-by":"publisher","DOI":"10.15452\/cejnm.2021.12.0038"},{"key":"ref107","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2011.555303"},{"key":"ref108","unstructured":"LewisJRSauroJTen things to know about the RITE methodMeasuringU2026-02-06https:\/\/measuringu.com\/rite-method\/"},{"key":"ref109","doi-asserted-by":"publisher","DOI":"10.1007\/s10055-022-00632-8"},{"key":"ref110","doi-asserted-by":"publisher","DOI":"10.1016\/j.teln.2025.04.017"},{"key":"ref111","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-05645-2"},{"key":"ref112","doi-asserted-by":"publisher","DOI":"10.2196\/56844"},{"key":"ref113","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2023.1336897"},{"key":"ref114","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0238437"},{"key":"ref115","unstructured":"JeffriesPRRizzoloMADesigning and implementing models for the innovative use of simulation to teach nursing care of ill adults and children: a national, multi-site, multi-method studyNLN\/Laerdal Project Summary Report20062026-08-14https:\/\/www.nln.org\/docs\/default-source\/uploadedfiles\/professional-development-programs\/read-the-nln-laerdal-project-summary-report-pdf.pdf"},{"key":"ref116","doi-asserted-by":"publisher","DOI":"10.1016\/j.tipsro.2022.09.001"},{"key":"ref117","doi-asserted-by":"publisher","DOI":"10.2196\/38259"},{"key":"ref118","unstructured":"NielsenJWhy you only need to test with 5 usersNielsen Norman Group20002026-08-14https:\/\/www.nngroup.com\/articles\/why-you-only-need-to-test-with-5-users\/"},{"key":"ref119","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169166"},{"key":"ref120","doi-asserted-by":"publisher","DOI":"10.1145\/765891.765936"},{"key":"ref121","unstructured":"SauroJA brief history of the magic number 5 in usability testingMeasuringU2026-08-14https:\/\/measuringu.com\/five-history\/"},{"key":"ref122","author":"Gwet, KL","year":"2014","journal-title":"Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e102697","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T12:20:14Z","timestamp":1788956414000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e102697"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,9]]},"references-count":122,"URL":"https:\/\/doi.org\/10.2196\/102697","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.102697","asserted-by":"object"}]},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,9]]}}}