{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T02:52:15Z","timestamp":1791341535020,"version":"4.3.1"},"reference-count":17,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T00:00:00Z","timestamp":1791331200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T00:00:00Z","timestamp":1791331200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100025450","name":"Independence Blue Cross Foundation","doi-asserted-by":"publisher","award":["529800"],"award-info":[{"award-number":["529800"]}],"id":[{"id":"10.13039\/100025450","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Adv Simul"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>In the health professions, artificial intelligence simulation-based experiences (AI-SBEs) are increasingly used to evaluate learner performance and provide feedback. Benefits of AI-SBEs include providing instantaneous feedback, allowing more practice opportunities than feasible using traditional methods, and reducing educator burden. However, methods for calibrating AI feedback are underdeveloped.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>This study evaluated the feedback provided by a generative AI-based software program (SimConverse) when assessing simulated patient interactions using the Weight Sensitivity Instrument (WSI), a checklist used to assess learners\u2019 weight bias-related clinical behaviors. A two-phase interrater reliability (IRR) approach was used. First, iterative prompt engineering was guided by AI-human IRR results. Four authors and four volunteers generated 31 interactions in SimConverse. Interaction transcripts were downloaded, and another author with expertise in the WSI manually reviewed and scored the transcripts. Percent agreement and Gwet\u2019s AC1 were calculated to compare AI and expert assessments, and the research team reviewed feedback statements and agreement statistics to identify rubric instructions in need of revision. After revising the rubric in SimConverse, four authors produced 10 additional SBE interactions. Expert and AI assessments were explored. In phase two, the AI-SBE was piloted with 60 medical students in a larger intervention study. A random sample of 30 student transcripts were assessed for AI-human IRR, and students\u2019 comments about the AI-SBE were reviewed.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Of the 11 WSI items, expert and AI agreement was consistently good or very good for 7 items across all rounds (Gwet\u2019s AC1=0.71 and higher). One item initially showed poor agreement (Gwet\u2019s AC1=0.56), but agreement improved after prompt engineering (final Gwet\u2019s AC1=0.89). The improved item assessed whether students made specific statements (asked if weight loss was a priority). Three other items had persistent poor agreement (final Gwet\u2019s AC1=0.42, 0.66, 0.64). These items required broader contextual assessment of the transcripts. Most open-ended student reactions to the AI-SBE were positive.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Findings suggest that AI scoring and feedback within AI-SBEs requires iterative prompt engineering and educator\/researcher guidance. Tuning was more successful for content-specific items than context-dependent items. Future work could explore best practices for prompt refinement, particularly for more complex and context-dependent rubric items.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s41077-026-00464-5","type":"journal-article","created":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T02:25:44Z","timestamp":1791339944000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["How does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability"],"prefix":"10.1186","volume":"11","author":[{"given":"Margaret","family":"Brace","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gail","family":"Furman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa","family":"Diewald","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rebecca","family":"Shenkman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie","family":"Felgoise","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tracy","family":"Oliver","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,10,7]]},"reference":[{"key":"464_CR1","doi-asserted-by":"publisher","first-page":"e43311","DOI":"10.2196\/43311","volume":"25","author":"T Stamer","year":"2023","unstructured":"Stamer T, Steinh\u00e4user J, Fl\u00e4gel K. Artificial Intelligence Supporting the Training of Communication Skills in the Education of Health Care Professions: Scoping Review. J Med Internet Res. 2023;25:e43311. https:\/\/doi.org\/10.2196\/43311.","journal-title":"J Med Internet Res"},{"issue":"6","key":"464_CR2","doi-asserted-by":"publisher","first-page":"102582","DOI":"10.1016\/j.outlook.2025.102582","volume":"73","author":"R Atkins","year":"2025","unstructured":"Atkins R, Brown KM, Mudd SS, Ghobadi K, Baker DJ, Szanton S. Reimagining nursing education: Leveraging competency-based education, artificial intelligence, and simulation for a diverse and practice-ready workforce. Nurs Outlook. 2025;73(6):102582. https:\/\/doi.org\/10.1016\/j.outlook.2025.102582.","journal-title":"Nurs Outlook"},{"key":"464_CR3","doi-asserted-by":"publisher","unstructured":"Hallquist E, Gupta I, Montalbano M, Loukas M. Applications of Artificial Intelligence in Medical Education: A Systematic Review. Cureus. 2025 Mar;1. https:\/\/doi.org\/10.7759\/cureus.79878.","DOI":"10.7759\/cureus.79878"},{"key":"464_CR4","doi-asserted-by":"publisher","unstructured":"Ting PW, Wolffsohn JS. Artificial intelligence-driven patient history and symptoms combined with slit-lamp eye simulation for enhancing the clinical training of students. Clin Experimental Optometry. 2025;1\u201310. https:\/\/doi.org\/10.1080\/08164622.2025.2544809.","DOI":"10.1080\/08164622.2025.2544809"},{"issue":"6","key":"464_CR5","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1080\/0142159X.2023.2180340","volume":"45","author":"MG Tolsgaard","year":"2023","unstructured":"Tolsgaard MG, Pusic MV, Sebok-Syer SS, Gin B, Svendsen MB, Syer MD, et al. The fundamentals of Artificial Intelligence in medical education research: AMEE Guide 156. Med Teach. 2023;45(6):565\u201373. https:\/\/doi.org\/10.1080\/0142159X.2023.2180340.","journal-title":"Med Teach"},{"key":"464_CR6","doi-asserted-by":"publisher","unstructured":"Henderson M, Bearman M, Chung J, Fawns T, Buckingham Shum S, Matthews KE, et al. Comparing Generative AI and teacher feedback: student perceptions of usefulness and trustworthiness. Assess Evaluation High Educ. 2025;1\u201316. https:\/\/doi.org\/10.1080\/02602938.2025.2502582.","DOI":"10.1080\/02602938.2025.2502582"},{"key":"464_CR7","doi-asserted-by":"publisher","unstructured":"Lee SS, Moore RL. Harnessing Generative AI (GenAI) for Automated Feedback in Higher Education: A Systematic Review. OLJ. 2024;28(3). https:\/\/doi.org\/10.24059\/olj.v28i3.4593.","DOI":"10.24059\/olj.v28i3.4593"},{"key":"464_CR8","doi-asserted-by":"publisher","first-page":"e59435","DOI":"10.2196\/59435","volume":"27","author":"C Wang","year":"2025","unstructured":"Wang C, Li S, Lin N, Zhang X, Han Y, Wang X, et al. Application of Large Language Models in Medical Training Evaluation\u2014Using ChatGPT as a Standardized Patient: Multimetric Assessment. J Med Internet Res. 2025;27:e59435. https:\/\/doi.org\/10.2196\/59435.","journal-title":"J Med Internet Res"},{"key":"464_CR9","doi-asserted-by":"publisher","first-page":"e59213","DOI":"10.2196\/59213","volume":"10","author":"F Holderried","year":"2024","unstructured":"Holderried F, Stegemann-Philipps C, Herrmann-Werner A, Festl-Wietek T, Holderried M, Eickhoff C, et al. A Language Model\u2013Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study. JMIR Med Educ. 2024;10:e59213. https:\/\/doi.org\/10.2196\/59213.","journal-title":"JMIR Med Educ"},{"key":"464_CR10","doi-asserted-by":"publisher","unstructured":"Oliver T, Furman G, Shenkman R, Diewald L, Brace M, Mariani B. Development and Psychometric Testing of a Weight Sensitivity Instrument to Assess Weight Bias. Journal of Nursing Measurement. 2026. https:\/\/doi.org\/10.1891\/jnm-2025-0162.","DOI":"10.1891\/jnm-2025-0162"},{"key":"464_CR11","doi-asserted-by":"publisher","unstructured":"Kottner J, Audig\u00e9 L, Brorson S, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96-106. doi:10.1016\/j.jclinepi.2010.03.002 https:\/\/doi.org\/10.1016\/j.jclinepi.2010.03.002","DOI":"10.1016\/j.jclinepi.2010.03.002"},{"issue":"4","key":"464_CR12","doi-asserted-by":"publisher","first-page":"499","DOI":"10.1177\/2192568220911648","volume":"10","author":"JR Dettori","year":"2020","unstructured":"Dettori JR, Norvell DC. Kappa and Beyond: Is There Agreement? Global Spine J. 2020;10(4):499\u2013501. https:\/\/doi.org\/10.1177\/2192568220911648.","journal-title":"Global Spine J"},{"issue":"1","key":"464_CR13","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1186\/1471-2288-13-61","volume":"13","author":"N Wongpakaran","year":"2013","unstructured":"Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen\u2019s Kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(1):61. https:\/\/doi.org\/10.1186\/1471-2288-13-61.","journal-title":"BMC Med Res Methodol"},{"issue":"5","key":"464_CR14","doi-asserted-by":"publisher","first-page":"718","DOI":"10.1080\/02602938.2025.2459641","volume":"50","author":"T Corbin","year":"2025","unstructured":"Corbin T, Tai J, Flenady G. Understanding the place and value of GenAI feedback: a recognition-based framework. Assess Evaluation High Educ. 2025;50(5):718\u201331. https:\/\/doi.org\/10.1080\/02602938.2025.2459641.","journal-title":"Assess Evaluation High Educ"},{"key":"464_CR15","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2024.2445037","author":"K Masters","year":"2025","unstructured":"Masters K, MacNeil H, Benjamin J, Carver T, Nemethy K, Valanci-Aroesty S, et al. Artificial intelligence in health professions education assessment: AMEE Guide 178. Med Teach. 2025. https:\/\/doi.org\/10.1080\/0142159X.2024.2445037.","journal-title":"Med Teach"},{"key":"464_CR16","doi-asserted-by":"publisher","unstructured":"Hicke Y, Geathers J, Vu K, Sewell J, Cardie C, Talwalkar J et al. MedSimAI: Simulation and Formative Feedback Generation to Enhance Deliberate Practice in Medical Education. In: Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference . 2026:415\u201324. Available from: http:\/\/arxiv.org\/abs\/2503.05793https:\/\/doi.org\/10.1145\/3785022.3785092. [Cited 4 May 2026].","DOI":"10.1145\/3785022.3785092"},{"key":"464_CR17","doi-asserted-by":"publisher","first-page":"1704820","DOI":"10.3389\/feduc.2025.1704820","volume":"10","author":"LH Alghamdi","year":"2025","unstructured":"Alghamdi LH, Alghizzi TM. Educators\u2019 reflections on AI-automated feedback in higher education: a structured integrative review of potentials, pitfalls, and ethical dimensions. Front Educ. 2025;10:1704820. https:\/\/doi.org\/10.3389\/feduc.2025.1704820.","journal-title":"Front Educ"}],"container-title":["Advances in Simulation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41077-026-00464-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s41077-026-00464-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41077-026-00464-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T02:25:45Z","timestamp":1791339945000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s41077-026-00464-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,7]]},"references-count":17,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["464"],"URL":"https:\/\/doi.org\/10.1186\/s41077-026-00464-5","relation":{},"ISSN":["2059-0628"],"issn-type":[{"value":"2059-0628","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,7]]},"assertion":[{"value":"30 March 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 July 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 October 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This study was approved by the Villanova University Institutional Review Board, IRB-FY2025-214, and data were only collected from consenting participants.","order":1,"name":"Ethics","label":"Ethics approval and consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","label":"Consent for publication","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"70"}}