{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T19:02:21Z","timestamp":1790276541582,"version":"4.1.0"},"reference-count":27,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language model (LLM)\u2013powered virtual standardized patients (VSPs) enable scalable clinical skills practice, but the validity of AI-generated scores relative to faculty ratings remains unclear.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to assess agreement between LLM-generated and faculty ratings of history-taking and communication performance and to examine the influence of rater and case heterogeneity.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n In this cross-sectional study, 92 fourth-year medical students completed one of three 15-minute voice-based VSP cases (fever, diarrhea, and cough). Ten blinded faculty raters scored performance (0\u2010100 points total; 0\u201050 points per domain). AI scores were generated by DeepSeek-V3 using a calibrated prompt. Agreement was evaluated using mixed-effects models, intraclass correlation coefficients (ICC [2,1]), Spearman correlations, mean absolute error (MAE), Bland-Altman analysis, and variance partition coefficients (VPC).<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Median total scores were similar for AI and faculty (median 93.0, IQR 89.0-95.0 vs median 94.0, IQR 91.0-95.0). Rater variability accounted for 37% of residual variance in faculty total scores (VPC=0.37). AI total scores were positively associated with faculty total scores (\u03b2=0.37, 95% CI 0.26\u20100.48;\n P<\/jats:italic>\n <.001; Spearman \u03c1=0.50, 95% CI 0.34\u20100.65). Absolute agreement was moderate (ICC[2,1]=0.51, 95% CI 0.34\u20100.65), with MAE of 3.11 points. Mixed-effects Bland-Altman analysis showed a small, not statistically significant mean bias (1.26 points, 95% CI \u22120.48 to 3.01;\n P<\/jats:italic>\n =.16) and 95% limits of agreement from \u22124.95 to 7.48 (width=12.43 points), with proportional bias (\u03b2_proportional bias=\u22120.55;\n P<\/jats:italic>\n <.001). Agreement was stronger for information gathering (\u03b2=0.46; \u03c1=0.49; ICC=0.54; VPC=0.23) than for communication (\u03b2=0.27; \u03c1=0.28; ICC=0.29; VPC=0.52). A sensitivity analysis in the lowest quartile showed attenuated but consistent agreement (ICC=0.38).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n LLM-based scoring in a VSP showed moderate agreement with faculty ratings, performing better for information gathering than for communication. Due to rater and case heterogeneity, ceiling effects, and proportional bias, this method is suitable for formative use and enhanced sampling in programmatic assessment but not for independent, high-stakes summative decisions.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/95578","type":"journal-article","created":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T12:35:09Z","timestamp":1785501309000},"page":"e95578-e95578","source":"Crossref","is-referenced-by-count":2,"title":["Evaluating Large Language Model\u2013Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4334-587X","authenticated-orcid":false,"given":"Xiaoxing","family":"Gao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2978-6484","authenticated-orcid":false,"given":"Xiaoming","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5989-7711","authenticated-orcid":false,"given":"Rongrong","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-4109-076X","authenticated-orcid":false,"given":"Li","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4916-1409","authenticated-orcid":false,"given":"Huiting","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7147-8277","authenticated-orcid":false,"given":"Bingqing","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9690-8555","authenticated-orcid":false,"given":"Chong","family":"Wei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1829-5491","authenticated-orcid":false,"given":"Wei","family":"Qiu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5831-4227","authenticated-orcid":false,"given":"Mengyu","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5355-3405","authenticated-orcid":false,"given":"Xuefeng","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,24]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-015-0296-3","article-title":"Virtual patients--what are we talking about? A framework to classify the meanings of the term in healthcare education","volume":"15","author":"Kononowicz","journal-title":"BMC Med Educ"},{"issue":"7","key":"R2","doi-asserted-by":"publisher","DOI":"10.2196\/14676","article-title":"Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration","volume":"21","author":"Kononowicz","journal-title":"J Med Internet Res"},{"issue":"4","key":"R3","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1111\/j.1365-2923.2008.03286.x","article-title":"Virtual patients: a critical literature review and proposed next steps","volume":"43","author":"Cook","journal-title":"Med Educ"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/48291","article-title":"Large language models in medical education: opportunities, challenges, and future directions","volume":"9","author":"Abd-Alrazaq","journal-title":"JMIR Med Educ"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","first-page":"680","DOI":"10.3390\/app16020680","article-title":"Can ChatGPT replace the teacher in assessment? A review of research on the use of large language models in grading and providing feedback","volume":"16","author":"Jukiewicz","journal-title":"Appl Sci"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1007\/s10459-013-9458-4","article-title":"What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment","volume":"19","author":"Cook","journal-title":"Adv Health Sci Educ Theory Pract"},{"issue":"2","key":"R7","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10459-022-10096-9","article-title":"Pass\/fail decisions and standards: the impact of differential examiner stringency on OSCE outcomes","volume":"27","author":"Homer","journal-title":"Adv Health Sci Educ Theory Pract"},{"issue":"12","key":"R8","doi-asserted-by":"publisher","first-page":"1181","DOI":"10.1111\/j.1365-2923.2011.04075.x","article-title":"A systematic review of the reliability of objective structured clinical examination scores","volume":"45","author":"Brannick","journal-title":"Med Educ"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-07241-4","article-title":"Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination","volume":"25","author":"Tekin","journal-title":"BMC Med Educ"},{"key":"R10","doi-asserted-by":"crossref","unstructured":"Schi\u00f6tt J Ivegren W Borg A Parodis I Skantze G . Using LLMs to grade clinical reasoning for medical students in virtual patient dialogues. In: Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics; 2025. URL: https:\/\/aclanthology.org\/2025.sigdial-1.56\/ [Accessed 12-08-2026]","DOI":"10.18653\/v1\/2025.sigdial-1.56"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1177\/0049124104268644","article-title":"Multimodel inference: understanding AIC and BIC in model selection","volume":"33","author":"Burnham","journal-title":"Sociol Methods Res"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/58753","article-title":"Enhancing medical interview skills through AI-simulated patient interactions: nonrandomized controlled trial","volume":"10","author":"Yamamoto","journal-title":"JMIR Med Educ"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.2196\/73419","article-title":"Development and validation of a large language model-based system for medical history-taking training: prospective multicase study on evaluation stability, human-AI consistency, and transparency","volume":"11","author":"Liu","journal-title":"JMIR Med Educ"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Zhou K Giorgi JM Mani P Xu P Liang D Tan C . From feedback to checklists: grounded evaluation of AI-generated clinical notes. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2025. [doi: 10.18653\/v1\/2025.emnlp-industry.104]","DOI":"10.18653\/v1\/2025.emnlp-industry.104"},{"issue":"9","key":"R15","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2024-096799","article-title":"Validity evidence for communication skills assessment in health professions education: a scoping review","volume":"15","author":"Dorrestein","journal-title":"BMJ Open"},{"issue":"7","key":"R16","doi-asserted-by":"publisher","first-page":"564","DOI":"10.3109\/0142159X.2013.789134","article-title":"Assessment in the post-psychometric era: learning to love the subjective and collective","volume":"35","author":"Hodges","journal-title":"Med Teach"},{"issue":"8","key":"R17","doi-asserted-by":"publisher","first-page":"800","DOI":"10.1111\/j.1365-2923.2008.03113.x","article-title":"Is it me or is it them? Factors that influence the passing of underperforming students","volume":"42","author":"Cleland","journal-title":"Med Educ"},{"issue":"2","key":"R18","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1037\/spq0000518","article-title":"Using many-facet Rasch measurement and generalizability theory to explore rater effects for direct behavior rating-multi-item scales","volume":"38","author":"Anthony","journal-title":"Sch Psychol"},{"issue":"7","key":"R19","doi-asserted-by":"publisher","first-page":"3910","DOI":"10.3758\/s13428-022-01997-z","article-title":"A Bayesian many-facet Rasch model with Markov modeling for rater severity drift","volume":"55","author":"Uto","journal-title":"Behav Res Methods"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2023-100830","article-title":"Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards","volume":"30","author":"Roberts","journal-title":"BMJ Health Care Inform"},{"issue":"11","key":"R21","doi-asserted-by":"publisher","first-page":"981","DOI":"10.1111\/medu.14221","article-title":"Workplace-based assessments in postgraduate medical education: a hermeneutic review","volume":"54","author":"Prentice","journal-title":"Med Educ"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-021-03077-w","article-title":"OSCE rater cognition - an international multi-centre qualitative study","volume":"22","author":"Hyde","journal-title":"BMC Med Educ"},{"issue":"2","key":"R23","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","journal-title":"JMIR Med Educ"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.2196\/56342","article-title":"Assessing the ability of a large language model to score free-text medical student clinical notes: quantitative study","volume":"10","author":"Burke","journal-title":"JMIR Med Educ"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.2196\/83376","article-title":"Evaluating the pediatric behavior guidance of students based on actual clinical transcripts scored by faculty and large language models: pilot comparative study","volume":"12","author":"Dhillon","journal-title":"JMIR Med Educ"},{"key":"R27","unstructured":"Sun X . AI score and human score of medical students\u2019 VSP interview. Zenodo. 2026. URL: https:\/\/zenodo.org\/records\/20774435 [Accessed 26-08-2026]"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T18:42:40Z","timestamp":1790275360000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e95578"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,24]]},"references-count":27,"URL":"https:\/\/doi.org\/10.2196\/95578","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,24]]},"article-number":"v12i7e95578"}}