{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T23:09:53Z","timestamp":1792796993236,"version":"4.5.0"},"reference-count":36,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T00:00:00Z","timestamp":1789948800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Physician\u2013patient verbal interactions are fundamental to clinical care and medical education, yet their evaluation is time-intensive and subject to inter-rater variability. We evaluated whether large language models (LLMs) can accurately assess transcribed standardized patient-physician verbal interactions and generate useful narrative feedback.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>In this single-center study, 12 standardized patient\u2013medical student encounters were independently evaluated by ChatGPT-4o, Gemini 3 Pro, and a physician reviewer using a gold standard rubric composed of 55 history-taking elements and 19 presentation elements assessing diagnostic reasoning, management planning, and patient counseling. Agreement was assessed using Cohen\u2019s kappa and intraclass correlation coefficients. Narrative feedback generated by the LLMs was independently evaluated by ChatGPT, Gemini, Claude, and a physician reviewer.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Across raters, mean scores were similar, with 35\u201336 of 55 history-taking elements and 13\u201314 of 19 presentation elements identified as present. Agreement was high for history-taking elements (\n                      <jats:italic>\u03ba<\/jats:italic>\n                      \u202f=\u202f0.84\u20130.89; intraclass correlation coefficients = 0.93\u20130.97) and lower for presentation elements (\n                      <jats:italic>\u03ba<\/jats:italic>\n                      \u202f=\u202f0.62\u20130.72; intraclass correlation coefficients = 0.61\u20130.83). Both LLMs generated detailed, accurate, and clinically useful narrative feedback.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>These findings demonstrate that LLMs can reliably evaluate simulated physician\u2013patient interactions and may support standardized assessment of clinical communication for research, quality improvement, and medical education.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frai.2026.1936749","type":"journal-article","created":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T05:35:56Z","timestamp":1789968956000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Large language model evaluation and feedback of transcribed simulated physician-patient verbal interactions"],"prefix":"10.3389","volume":"9","author":[{"given":"Eric J.","family":"Nolan","sequence":"first","affiliation":[{"name":"Division of Hospital Medicine, Department of Internal Medicine, College of Medicine, The Ohio State University Wexner Medical Center","place":["Columbus, OH, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Harry B.","family":"Burke","sequence":"additional","affiliation":[{"name":"George Washington University School of Medicine","place":["Washington, DC, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,21]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"1181","DOI":"10.1111\/j.1365-2923.2011.04075.x","article-title":"A systematic review of the reliability of objective structured clinical examination scores","volume":"45","author":"Brannick","year":"2011","journal-title":"Med. Educ."},{"key":"ref2","doi-asserted-by":"publisher","first-page":"910","DOI":"10.1136\/amiajnl-2013-002321","article-title":"QNOTE: an instrument for measuring the quality of EHR clinical notes","volume":"21","author":"Burke","year":"2014","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref3","doi-asserted-by":"publisher","first-page":"e56342","DOI":"10.2196\/56342","article-title":"Assessing the ability of a large language model to score free-text medical student clinical notes: quantitative study","volume":"10","author":"Burke","year":"2024","journal-title":"JMIR Med. Educ."},{"key":"ref5","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1136\/amiajnl-2014-002726","article-title":"Electronic health records improve note quality","volume":"22","author":"Burke","year":"2015","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"830","DOI":"10.1186\/s12909-025-07433-y","article-title":"The impact of doctor-patient communication on patient satisfaction in outpatient settings: implications for medical training and practice","volume":"25","author":"Chen","year":"2025","journal-title":"BMC Med. Educ."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"902","DOI":"10.1097\/ACM.0000000000002618","article-title":"Clinical reasoning assessment methods: a scoping review and practical guidance","volume":"94","author":"Daniel","year":"2019","journal-title":"Acad. Med."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"1670","DOI":"10.1053\/j.gastro.2021.07.037","article-title":"A review of the evidence and recommendations on communication skills and the patient-provider relationship: a Rome foundation working team report","volume":"161","author":"Drossman","year":"2021","journal-title":"Gastroenterology"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1056\/NEJMra054784","article-title":"Assessment in medical education","volume":"356","author":"Epstein","year":"2007","journal-title":"N. Engl. J. Med."},{"key":"ref10","doi-asserted-by":"publisher","first-page":"1586","DOI":"10.1177\/10966218251363802","article-title":"Using large language models to analyze symptom discussions and recommendations in clinical encounters","volume":"28","author":"Fenton","year":"2025","journal-title":"J. Palliat. Med."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"CD012418","DOI":"10.1002\/14651858.CD012418.pub2","article-title":"Interventions for improving medical students' interpersonal communication in medical consultations","volume":"2","author":"Gilligan","year":"2021","journal-title":"Cochrane Database Syst. Rev."},{"key":"ref12","doi-asserted-by":"publisher","first-page":"617","DOI":"10.1007\/s10459-007-9068-0","article-title":"Undesired variance due to examiner stringency\/leniency effect in communication skill scores assessed in OSCEs","volume":"13","author":"Harasym","year":"2008","journal-title":"Adv. Health Sci. Educ. Theory Pract."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1186\/s12909-020-02110-8","article-title":"Validity, reliability and feasibility of a new observation rating tool and a post encounter rating tool for the assessment of clinical reasoning skills of medical students during their internal medicine clerkship: a pilot study","volume":"20","author":"Haring","year":"2020","journal-title":"BMC Med. Educ."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"e82818","DOI":"10.2196\/82818","article-title":"Evaluation of an AI-based Chatbot providing real-time feedback in communication training for mental health care professionals: proof-of-concept observational study","volume":"27","author":"Herschbach","year":"2025","journal-title":"J. Med. Internet Res."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"108718","DOI":"10.1016\/j.pec.2025.108718","article-title":"Systematic review of observational coding systems to assess patient-healthcare professional interactions","volume":"135","author":"Hillen","year":"2025","journal-title":"Patient Educ. Couns."},{"key":"ref16","doi-asserted-by":"publisher","first-page":"806","DOI":"10.1093\/postmj\/qgaf195","article-title":"Artificial intelligence for assessment in competency-based medical education: current practices and future directions","volume":"102","author":"Hui","year":"2026","journal-title":"Postgrad. Med. J."},{"key":"ref17","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1055\/s-0039-1692164","article-title":"Challenges and opportunities to improve the clinician experience reviewing electronic Progress notes","volume":"10","author":"Hultman","year":"2019","journal-title":"Appl. Clin. Inform."},{"key":"ref18","doi-asserted-by":"publisher","first-page":"1159","DOI":"10.1111\/medu.14347","article-title":"Machine learning to extract communication and history-taking skills in OSCE transcripts","volume":"54","author":"Jani","year":"2020","journal-title":"Med. Educ."},{"key":"ref19","doi-asserted-by":"publisher","first-page":"e27378","DOI":"10.1002\/pbc.27378","article-title":"Advancing the field of communication research in pediatric oncology: a systematic review of the literature analyzing medical dialogue","volume":"65","author":"Kaye","year":"2018","journal-title":"Pediatr. Blood Cancer"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting Intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J. Chiropr. Med."},{"key":"ref21","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1186\/s12909-020-02171-9","article-title":"A randomised controlled trial of feedback to improve patient satisfaction and consultation skills in medical students","volume":"20","author":"Lai","year":"2020","journal-title":"BMC Med. Educ."},{"key":"ref22","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"1183","DOI":"10.1001\/jama.290.9.1183","article-title":"Physician clinical performance assessment: prospects and barriers","volume":"290","author":"Landon","year":"2003","journal-title":"JAMA"},{"key":"ref24","first-page":"757","article-title":"Poor physician-patient communication and medical error","volume":"103","author":"Lazris","year":"2021","journal-title":"Am. Fam. Physician"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1001\/jama.277.7.553","article-title":"Physician-patient communication. The relationship with malpractice claims among primary care physicians and surgeons","volume":"277","author":"Levinson","year":"1997","journal-title":"JAMA"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"989","DOI":"10.1111\/j.1365-2923.2009.03438.x","article-title":"The effect of differential rater function over time (DRIFT) on objective structured clinical examination ratings","volume":"43","author":"McLaughlin","year":"2009","journal-title":"Med. Educ."},{"key":"ref27","doi-asserted-by":"publisher","first-page":"112058","DOI":"10.1016\/j.jclinepi.2025.112058","article-title":"The measurement properties reliability and measurement error explained - a COSMIN perspective","volume":"190","author":"Mokkink","year":"2026","journal-title":"J. Clin. Epidemiol."},{"key":"ref28","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1097\/MLR.0000000000001598","article-title":"Will improvements in patient experience with care impact clinical and quality of care outcomes?: a systematic review","volume":"59","author":"Navarro","year":"2021","journal-title":"Med. Care"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1186\/s12911-026-03414-3","article-title":"Accuracy of large language model transcription of simulated physician-patient verbal interactions","volume":"26","author":"Nolan","year":"2026","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"ref30","doi-asserted-by":"publisher","first-page":"1772","DOI":"10.1080\/0142159X.2025.2504106","article-title":"Using large language models (LLMs) to apply analytic rubrics to score post-encounter notes","volume":"47","author":"Runyon","year":"2025","journal-title":"Med. Teach."},{"key":"ref31","doi-asserted-by":"publisher","first-page":"901","DOI":"10.1186\/s12909-023-04688-1","article-title":"Assessor discomfort and failure to fail in clinical performance assessments","volume":"23","author":"Scarff","year":"2023","journal-title":"BMC Med. Educ."},{"key":"ref32","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat. Med."},{"key":"ref33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3686952","article-title":"Commsense: a wearable sensing computational framework for evaluating patient-clinician interactions","volume":"8","author":"Wang","year":"2024","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"e59435","DOI":"10.2196\/59435","article-title":"Application of large language models in medical training evaluation-using ChatGPT as a standardized patient: multimetric assessment","volume":"27","author":"Wang","year":"2025","journal-title":"J. Med. Internet Res."},{"key":"ref35","doi-asserted-by":"publisher","first-page":"825","DOI":"10.3389\/fpsyg.2020.00825","article-title":"Do we overestimate the within-variability? The impact of measurement error on intraclass coefficient estimation","volume":"11","author":"Wilms","year":"2020","journal-title":"Front. Psychol."},{"key":"ref36","doi-asserted-by":"publisher","first-page":"1631565","DOI":"10.3389\/fmed.2025.1631565","article-title":"Leveraging LLM to identify missed information in patient-physician communication: improving healthcare service quality","volume":"12","author":"Zhou","year":"2025","journal-title":"Front Med (Lausanne)."},{"key":"ref37","doi-asserted-by":"publisher","first-page":"826","DOI":"10.1097\/MLR.0b013e31819a5acc","article-title":"Physician communication and patient adherence to treatment: a meta-analysis","volume":"47","author":"Zolnierek","year":"2009","journal-title":"Med. Care"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1936749\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T05:35:57Z","timestamp":1789968957000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1936749\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,21]]},"references-count":36,"alternative-id":["10.3389\/frai.2026.1936749"],"URL":"https:\/\/doi.org\/10.3389\/frai.2026.1936749","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,21]]},"article-number":"1936749"}}