{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T07:22:33Z","timestamp":1791530553014,"version":"4.3.4"},"reference-count":54,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Strengthening the global health workforce is central to achieving universal health coverage, but health systems cannot improve what they cannot measure. Valid and scalable assessment of clinical competency is essential for monitoring workforce readiness and ensuring that expanded service coverage translates into high-quality care. Traditional standardized patients, however, remain resource-intensive, difficult to scale, and vulnerable to evaluator-related bias. Recent advances in AI have enabled AI-led simulated standardized patients (SSPs) that may address these limitations.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n Although existing reviews have examined AI-SSPs broadly within medical education, their use as clinical competency assessment systems remains less well characterized. This scoping review aimed to address this gap by systematically mapping the scope, design features, and validation approaches of AI-SSP tools for clinical competency assessment.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We registered the protocol prospectively with the Open Science Framework and conducted a scoping review following the Joanna Briggs Institute Manual for Evidence Synthesis. We searched MEDLINE, Embase, CINAHL Complete, Education Source, Web of Science Core Collection, Inspec, ERIC, and Cochrane CENTRAL, supplemented by manual searching. Eligible primary studies involved an AI-based SSP interaction that yielded a quantitative score of respondents' clinical competency. Two reviewers independently screened records and resolved conflicts through discussion. Data were charted on study characteristics and populations, frontend platform and interface features, backend AI architectures, performance scoring mechanisms, and tool evaluation methods and outcomes.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Of 5987 database records and 4 identified through other methods, 21 studies describing 20 unique AI-SSP systems were included. Studies were published between 2008 and 2026, with 15 (71%) published in 2024 or later. Most studies were conducted in high-income settings and involved medical students. Systems shifted over time from rule-based virtual patient to large language model\u2013enabled conversational platforms after 2022. History-taking was the most frequently assessed competency, and checklist coverage was the most common scoring approach. Among studies comparing AI-generated scores with human ratings, findings were mixed: some reported moderate to strong agreement, while others showed score inflation and inconsistent performance. Most studies focused on feasibility; fewer evaluated large-scale use or sustained real-world implementation.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This review is innovative in shifting attention from AI-SSPs as educational simulations to AI-SSPs as emerging infrastructures for clinical competency assessment. Unlike prior reviews of virtual patients or AI in medical education, this review maps not only system design and learner interaction but also which competencies are assessed, how scores are generated, and how the assessment evidence has been validated. AI-SSPs could make clinical competency assessment more scalable and potentially more standardized, although more evidence is still needed to determine whether they achieve validity and reliability across diverse learner or practitioner groups.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/92826","type":"journal-article","created":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T06:00:06Z","timestamp":1782972006000},"page":"e92826-e92826","source":"Crossref","is-referenced-by-count":1,"title":["AI for Clinical Competency Assessment: Scoping Review of Methods and Applications"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7244-8555","authenticated-orcid":false,"given":"Wenjia (Stella)","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9652-6653","authenticated-orcid":false,"given":"Benjamin B","family":"Daniels","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3059-1369","authenticated-orcid":false,"given":"Carol","family":"Mita","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0923-573X","authenticated-orcid":false,"given":"Hoang","family":"Nguyen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5386-4502","authenticated-orcid":false,"given":"David B","family":"Duong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,14]]},"reference":[{"issue":"9","key":"R1","doi-asserted-by":"publisher","first-page":"e839","DOI":"10.1016\/S2214-109X(17)30295-4","article-title":"All roads lead to universal health coverage","volume":"5","author":"Ghebreyesus","journal-title":"Lancet Glob Health"},{"issue":"6","key":"R2","doi-asserted-by":"publisher","first-page":"435","DOI":"10.2471\/BLT.09.066290","article-title":"Priorities for research into human resources for health in low- and middle-income countries","volume":"88","author":"Ranson","journal-title":"Bull World Health Organ"},{"issue":"9449","key":"R3","doi-asserted-by":"publisher","first-page":"1984","DOI":"10.1016\/S0140-6736(04)17482-5","article-title":"Human resources for health: overcoming the crisis","volume":"364","author":"Chen","journal-title":"Lancet"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Daniels B Das J Gatti R Yi Chang A . Caseloads and competence in sub-saharan africa: a fundamental reassessment of the human resources crisis in primary health care. World Bank; 2025. URL: https:\/\/hdl.handle.net\/10986\/42721 [doi: 10.1596\/1813-9450-11043]","DOI":"10.1596\/1813-9450-11043"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"Slawomirski L Klazinga N . The economics of patient safety: from analysis to action. OECD Publishing; Aug 2022. [doi: 10.1787\/761f2da8-en]","DOI":"10.1787\/761f2da8-en"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0245439","article-title":"Impact of integrating objective structured clinical examination into academic student assessment: large-scale experience in a French medical school","volume":"16","author":"Matet","journal-title":"PLoS ONE"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"192","DOI":"10.5144\/0256-4947.2008.192","article-title":"Objective structured clinical examinations as an assessment method in residency training: practical considerations","volume":"28","author":"Hijazi","journal-title":"Ann Saudi Med"},{"issue":"2","key":"R8","doi-asserted-by":"publisher","first-page":"156","DOI":"10.4103\/ijcfm.ijcfm_12_22","article-title":"Challenges of objective structured clinical examination as a tool in medical assessment","volume":"8","author":"Behera","journal-title":"Indian Journal of Community and Family Medicine"},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Conner R Gatti R Andrews K Avitabile C Sharma J Chang AY . The quality of health and education systems across Africa: evidence from a decade of service delivery indicators surveys. World Bank Group. Nov 2021. URL: https:\/\/documents.worldbank.org\/en\/publication\/documents-reports\/documentdetail\/380481637326015488 [Accessed 12-02-2026]","DOI":"10.1596\/978-1-4648-1675-8"},{"issue":"8S","key":"R10","doi-asserted-by":"publisher","first-page":"S16","DOI":"10.1097\/ACM.0000000000005245","article-title":"\u201cMy assessments are biased!\u201d measurement and sociocultural approaches to achieve fairness in assessment in medical education","volume":"98","author":"Hauer","journal-title":"Acad Med"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1016\/j.pec.2023.107934","article-title":"A gender biased hidden curriculum of clinical vignettes in undergraduate medical training","volume":"116","author":"Arsever","journal-title":"Patient Educ Couns"},{"issue":"12","key":"R12","doi-asserted-by":"publisher","first-page":"1438","DOI":"10.1097\/ACM.0000000000005794","article-title":"Bias in observed assessments in medical education: a scoping review","volume":"99","author":"Ismaeel","journal-title":"Acad Med"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.3352\/jeehp.2017.14.34","article-title":"The sights and insights of examiners in objective structured clinical examinations","volume":"14","author":"Chong","journal-title":"J Educ Eval Health Prof"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-017-0908-1","article-title":"Examiner effect on the objective structured clinical exam - a study at five medical schools","volume":"17","author":"Schleicher","journal-title":"BMC Med Educ"},{"issue":"10","key":"R15","doi-asserted-by":"publisher","first-page":"1589","DOI":"10.1097\/ACM.0b013e3181edfe13","article-title":"Computerized virtual patients in health professions education: a systematic review and meta-analysis","volume":"85","author":"Cook","journal-title":"Acad Med"},{"issue":"7","key":"R16","doi-asserted-by":"publisher","DOI":"10.2196\/14676","article-title":"Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration","volume":"21","author":"Kononowicz","journal-title":"J Med Internet Res"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.2196\/79039","article-title":"Large language model-based virtual patient systems for history-taking in medical education: comprehensive systematic review","volume":"14","author":"Li","journal-title":"JMIR Med Inform"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.2196\/79091","article-title":"Embracing the future of medical education with large language model-based virtual patients: scoping review","volume":"27","author":"Zeng","journal-title":"J Med Internet Res"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01283-x","article-title":"Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education","volume":"6","author":"Yu","journal-title":"Commun Med (Lond)"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"7","DOI":"10.3390\/aimed1010007","article-title":"Large language model-based virtual patients for simulated clinical learning: a scoping review","volume":"1","author":"Gandhi","journal-title":"AI in Medicine"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/71125","article-title":"Applications, challenges, and prospects of generative artificial intelligence empowering medical education: scoping review","volume":"11","author":"Lin","journal-title":"JMIR Med Educ"},{"issue":"3","key":"R22","doi-asserted-by":"publisher","first-page":"1803","DOI":"10.1007\/s40670-025-02373-0","article-title":"Artificial intelligence in medical education: a scoping review of the evidence for efficacy and future directions","volume":"35","author":"Shaw","journal-title":"Med Sci Educ"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-06719-5","article-title":"A systematic review of the impact of artificial intelligence on educational outcomes in health professions education","volume":"25","author":"Feigerlova","journal-title":"BMC Med Educ"},{"key":"R24","unstructured":"JBI Manual for Evidence Synthesis. URL: https:\/\/synthesismanual.jbi.global [Accessed 07-06-2026]"},{"issue":"10","key":"R25","doi-asserted-by":"publisher","first-page":"2119","DOI":"10.11124\/JBIES-20-00167","article-title":"Updated methodological guidance for the conduct of scoping reviews","volume":"18","author":"Peters","journal-title":"JBI Evid Synth"},{"issue":"7","key":"R26","doi-asserted-by":"publisher","first-page":"467","DOI":"10.7326\/M18-0850","article-title":"PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation","volume":"169","author":"Tricco","journal-title":"Ann Intern Med"},{"issue":"2","key":"R27","doi-asserted-by":"publisher","first-page":"174","DOI":"10.5195\/jmla.2021.962","article-title":"PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews","volume":"109","author":"Rethlefsen","journal-title":"J Med Libr Assoc"},{"key":"R28","unstructured":"A scoping review of AI-assisted vignettes for clinical competency assessment. OSF. URL: https:\/\/osf.io\/bu5en\/ [Accessed 27-07-2026]"},{"key":"R29","volume":"372","author":"Page","journal-title":"BMJ"},{"issue":"3","key":"R30","doi-asserted-by":"publisher","first-page":"e66","DOI":"10.1080\/01421590801910216","article-title":"The use of a virtual patient case in an OSCE-based exam--a pilot study","volume":"30","author":"Courteille","journal-title":"Med Teach"},{"key":"R31","volume":"169","author":"Oliven","journal-title":"Stud Health Technol Inform"},{"issue":"4","key":"R32","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1097\/SIH.0000000000000373","article-title":"Virtual standardized patient simulation: case development and pilot application to high-value care","volume":"14","author":"Bond","journal-title":"Simul Healthc"},{"issue":"9","key":"R33","doi-asserted-by":"publisher","first-page":"1053","DOI":"10.1080\/0142159X.2019.1616683","article-title":"Using virtual standardized patients to accurately assess information gathering skills in medical students","volume":"41","author":"Maicher","journal-title":"Med Teach"},{"issue":"7","key":"R34","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-019-1262-0","article-title":"MCRDR knowledge-based 3D dialogue simulation in clinical training and assessment","volume":"43","author":"Yang","journal-title":"J Med Syst"},{"key":"R35","doi-asserted-by":"crossref","unstructured":"Lee C Low MYH Soon BTC Lu L Lee H Patel NS . Development of a chatbot to train physiotherapy students in clinical questioning and reasoning. Presented at: 2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE); Dec 4-7, 2022:477-481; Hung Hom, Hong Kong. [doi: 10.1109\/TALE54877.2022.00084]","DOI":"10.1109\/TALE54877.2022.00084"},{"issue":"1","key":"R36","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-06399-7","article-title":"Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial","volume":"24","author":"Br\u00fcgge","journal-title":"BMC Med Educ"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/53961","article-title":"A generative pretrained transformer (GPT)-powered Chatbot as a simulated patient to practice history taking: prospective, mixed methods study","volume":"10","author":"Holderried","journal-title":"JMIR Med Educ"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.2196\/59213","article-title":"A language model-powered simulated patient with automated feedback for history taking: prospective study","volume":"10","author":"Holderried","journal-title":"JMIR Med Educ"},{"key":"R39","unstructured":"Li Y Zeng C Zhong J Zhang R Zhang M Zou L . Leveraging large language model as simulated patients for clinical education. arXiv. Preprint posted online on Apr 13, 2024. [doi: 10.48550\/arXiv.2404.13066]"},{"issue":"4","key":"R40","doi-asserted-by":"publisher","first-page":"1871","DOI":"10.1007\/s40670-025-02394-9","article-title":"Exploration of the role of ChatGPT in teaching communication skills for medical students: a pilot study","volume":"35","author":"Chiu","journal-title":"Med Sci Educ"},{"key":"R41","doi-asserted-by":"publisher","DOI":"10.1016\/j.hlc.2025.06.625","article-title":"Large language models deliver interactive learning cases accurately and with rapid appropriate feedback","volume":"34","author":"Gorcilov","journal-title":"Heart Lung Circ"},{"key":"R42","unstructured":"Haut K Hasan M Carroll T Epstein R Sen T Hoque E . AI standardized patient improves human conversations in advanced cancer care. arXiv. Preprint posted online on May 5, 2025. URL: http:\/\/arxiv.org\/abs\/2505.02694 [Accessed 04-08-2026]"},{"key":"R43","unstructured":"Haut K . USIHC: understand, simulate, and improve human compassion in serious illness communication (SIC) [Dissertation]. University of Rochester; 2025. :145. URL: http:\/\/hdl.handle.net\/1802\/38511 [Accessed 04-08-2026]"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.2196\/73419","article-title":"Development and validation of a large language model-based system for medical history-taking training: prospective multicase study on evaluation stability, human-AI consistency, and transparency","volume":"11","author":"Liu","journal-title":"JMIR Med Educ"},{"issue":"3","key":"R45","doi-asserted-by":"publisher","first-page":"1169","DOI":"10.1007\/s40670-025-02343-6","article-title":"Use of large language models for rapid quantitative feedback in case-based learning: a pilot study","volume":"35","author":"Qian","journal-title":"Med Sci Educ"},{"key":"R46","unstructured":"Zheng W Turner L Kropczynski J Ozer M Nguyen T Halse S . LLM-as-a-fuzzy-judge: fine-tuning large language models as a clinical evaluation judge with fuzzy logic. arXiv. Preprint posted online on Jun 12, 2025. [doi: 10.48550\/arXiv.2506.11221]"},{"issue":"4","key":"R47","doi-asserted-by":"publisher","DOI":"10.1016\/j.chest.2025.07.2506","article-title":"Use of artificial intelligence to identify student behaviors associated with improved diagnostic accuracy of pulmonary diseases","volume":"168","author":"Zhou","journal-title":"Chest"},{"key":"R48","doi-asserted-by":"publisher","DOI":"10.1016\/j.nedt.2025.106968","article-title":"AI-enhanced virtual reality simulation for nursing students\u2019 empathy: automated scoring and inter-rater reliability in a randomised controlled study","volume":"159","author":"Chen","journal-title":"Nurse Educ Today"},{"key":"R49","doi-asserted-by":"crossref","unstructured":"Hicke Y et al. MedSimAI: simulation and formative feedback generation to enhance deliberate practice in medical education. arXiv. Preprint posted online on Mar 1, 2026. [doi: 10.48550\/arXiv.2503.05793]","DOI":"10.1145\/3785022.3785092"},{"key":"R50","doi-asserted-by":"crossref","unstructured":"Reynolds EC Benton E Lu J Schnable C Southwood R . AI versus human consensus: correlation of rubric-based scoring in pharmacy counseling. Presented at: 2025 ASHP Midyear Clinical Meeting; Dec 7-10, 2025; Las Vegas, Nevada, United States. [doi: 10.1093\/ajhp\/zxaf258]","DOI":"10.1093\/ajhp\/zxaf258"},{"issue":"22","key":"R51","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare12222241","article-title":"Enhancing clinical reasoning with virtual patients: a hybrid systematic review combining human reviewers and ChatGPT","volume":"12","author":"Garc\u00eda-Torres","journal-title":"Healthcare (Basel)"},{"issue":"1","key":"R52","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-07241-4","article-title":"Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination","volume":"25","author":"Tekin","journal-title":"BMC Med Educ"},{"issue":"3","key":"R53","doi-asserted-by":"publisher","first-page":"101493","DOI":"10.1016\/j.giq.2020.101493","article-title":"Data governance: organizing data for trustworthy artificial Intelligence","volume":"37","author":"Janssen","journal-title":"Gov Inf Q"},{"issue":"3","key":"R54","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.105902","article-title":"When compliance is not safety: the regulatory blind spot in ai companion chatbots","volume":"18","author":"Zavaleta-Monestel","journal-title":"Cureus"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T06:02:24Z","timestamp":1786946544000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e92826"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,14]]},"references-count":54,"URL":"https:\/\/doi.org\/10.2196\/92826","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,14]]},"article-number":"v12i9e92826"}}