{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T21:38:15Z","timestamp":1792618695812,"version":"4.5.0"},"reference-count":56,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners\u2019 summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and support feedback. Still, their test-retest reliability and concordance with expert ratings in this context remain unclear.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate the test-retest reliability\u2014specifically, whether the models produce stable ratings across repetitions\u2014of LLM-generated medical summary statement ratings, and to measure their concordance with human expert assessments across different prompting strategies.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We conducted an experimental comparative study using 122 rubric-scored summary statements (English and German) from 6 internal medicine VP cases previously rated by 2 expert raters. Statements were evaluated along 4 ternary components (semantic qualifiers, transformation, narrowing, and global rating), and 2 binary components (factual accuracy and person). Four LLMs (Anthropic\u2019s Claude, OpenAI\u2019s GPT-4o, Mistral AI\u2019s Mistral Large, and Mistral AI\u2019s Mistral 7B) were each run 5 times under 4 prompting strategies (zero-shot, zero-shot with chain-of-thought [CoT], few-shot, and few-shot+CoT). The test-retest reliability was assessed with intraclass correlation coefficients (ICCs), Fleiss kappa (\u03ba), and concordance with human ratings with accuracy (%) and chance-corrected interrater agreement (Gwet AC1 and AC2).<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Large closed-weights models (Claude, GPT-4o, and Mistral Large) showed high test-retest reliability (ICC\u22650.92; Fleiss \u03ba=0.59\u20101.00). Claude was essentially deterministic (ICC and Fleiss \u03ba\u22650.99); GPT-4o and Mistral Large reached ICC\u22650.97 and Fleiss \u03ba\u22650.80 except under zero-shot. Mistral 7B, a small open-weights model, had high ICCs (\u22650.88) but low Fleiss \u03ba (0.32\u20100.52). Few-shot+CoT prompting achieved the best concordance with human ratings for all the models (means across runs and rubric components), peaking for Mistral Large (accuracy 77%; Gwet AC1 and AC2 0.85, 95% CI 0.83\u20100.87), followed by Claude (71%; 0.77, 0.74\u20100.81), GPT-4o (67%; 0.72, 0.68\u20100.77), and Mistral 7B (60%; 0.64, 0.56\u20100.73). Concordance was higher for the binary rubric components (person and factual accuracy) than for the ternary components. Extreme deviations and invalid outputs were rare but more frequent in ternary components and the smaller local model.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Large, general-purpose LLMs can function as reliable, rubric-guided raters of VP-based clinical summary statements, achieving high concordance with expert ratings under clearly defined criteria in binary rubric components and when using few-shot with CoT prompting. The performance on more nuanced, ternary reasoning dimensions was more variable. LLMs can support scalable, rubric-based formative feedback on clinical reasoning within VP systems but should complement rather than replace human judgment.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/94666","type":"journal-article","created":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T13:15:09Z","timestamp":1787750109000},"page":"e94666-e94666","source":"Crossref","is-referenced-by-count":0,"title":["Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5224-1522","authenticated-orcid":false,"given":"Patrick","family":"Stadler","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2656-7417","authenticated-orcid":false,"given":"Piotr","family":"Bucichowski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8710-8257","authenticated-orcid":false,"given":"Ylva","family":"Holzhausen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4335-5162","authenticated-orcid":false,"given":"Inga","family":"Hege","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1441-7512","authenticated-orcid":false,"given":"Harm","family":"Peters","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0310-2956","authenticated-orcid":false,"given":"Julius Josef","family":"Kaminski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,21]]},"reference":[{"issue":"10 Suppl","key":"R1","doi-asserted-by":"publisher","first-page":"S109","DOI":"10.1097\/00001888-199810000-00062","article-title":"The importance of early problem representation during case presentations","volume":"73","author":"Chang","journal-title":"Acad Med"},{"issue":"21","key":"R2","doi-asserted-by":"publisher","first-page":"2217","DOI":"10.1056\/NEJMra054782","article-title":"Educational strategies to promote clinical diagnostic reasoning","volume":"355","author":"Bowen","journal-title":"N Engl J Med"},{"key":"R3","doi-asserted-by":"crossref","unstructured":"Bowen JL Ten Cate O Prerequisites for learning clinical reasoning. Principles and practice of case-based clinical reasoning education. In: Ten Cate O Custers EJFM Durning SJ , editors. Innovation and Change in Professional Education. Vol 15. Springer; 2018:47-63. [doi: 10.1007\/978-3-319-64828-6_4]","DOI":"10.1007\/978-3-319-64828-6_4"},{"issue":"4","key":"R4","doi-asserted-by":"publisher","first-page":"688","DOI":"10.1016\/j.jbi.2011.03.008","article-title":"Summarization of clinical information: a conceptual model","volume":"44","author":"Feblowitz","journal-title":"J Biomed Inform"},{"issue":"13","key":"R5","doi-asserted-by":"publisher","first-page":"3041","DOI":"10.1007\/s11606-023-08359-1","article-title":"Reasoning on rounds: a framework for teaching diagnostic reasoning in the inpatient setting","volume":"38","author":"Penner","journal-title":"J Gen Intern Med"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1097\/ACM.0000000000000800","article-title":"The development and preliminary validation of a rubric to assess medical students\u2019 written summary statements in virtual patient cases","volume":"91","author":"Smith","journal-title":"Acad Med"},{"issue":"5","key":"R7","doi-asserted-by":"publisher","first-page":"457","DOI":"10.3109\/0142159X.2014.956052","article-title":"Thirty years of illness scripts: theoretical origins and practical applications","volume":"37","author":"Custers","journal-title":"Med Teach"},{"issue":"6","key":"R8","doi-asserted-by":"publisher","DOI":"10.3205\/zma001291","article-title":"Medical knowledge and teamwork predict the quality of case summary statements as an indicator of clinical reasoning in undergraduate medical students","volume":"36","author":"F\u00fcrstenberg","journal-title":"GMS J Med Educ"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1097\/ACM.0000000000001421","article-title":"The causes of errors in clinical reasoning: cognitive biases, knowledge deficits, and dual process thinking","volume":"92","author":"Norman","journal-title":"Acad Med"},{"issue":"7","key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/14676","article-title":"Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration","volume":"21","author":"Kononowicz","journal-title":"J Med Internet Res"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1097\/ACM.0000000000005908","article-title":"The use of virtual patients to provide feedback on clinical reasoning: a systematic review","volume":"100","author":"Jay","journal-title":"Acad Med"},{"issue":"9","key":"R12","doi-asserted-by":"publisher","first-page":"1346","DOI":"10.1097\/ACM.0000000000001578","article-title":"Feedback for learners in medical education: what is known? A scoping review","volume":"92","author":"Bing-You","journal-title":"Acad Med"},{"issue":"6","key":"R13","doi-asserted-by":"publisher","first-page":"921","DOI":"10.1080\/0142159X.2025.2569623","article-title":"Formative assessment and feedback in medical education: a practical guide: AMEE Guide No. 189","volume":"48","author":"Lipnevich","journal-title":"Med Teach"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-020-02297-w","article-title":"Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach","volume":"20","author":"Hege","journal-title":"BMC Med Educ"},{"issue":"11S","key":"R15","doi-asserted-by":"publisher","first-page":"S90","DOI":"10.1097\/ACM.0000000000005357","article-title":"Automated patient note grading: examining scoring reliability and feasibility","volume":"98","author":"Bond","journal-title":"Acad Med"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.2196\/49724","article-title":"Task-specific transformer-based language models in health care: scoping review","volume":"12","author":"Cho","journal-title":"JMIR Med Inform"},{"key":"R17","unstructured":"Nori H Daswani M Kelly C et al. Sequential diagnosis with language models. arXiv. Preprint posted online on Jun 27, 2025. [doi: 10.48550\/arXiv.2506.22405]"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02005-2","article-title":"Evaluating clinical AI summaries with large language models as judges","volume":"8","author":"Croxford","journal-title":"NPJ Digit Med"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2024.100213","article-title":"Applying large language models and chain-of-thought for automatic scoring","volume":"6","author":"Lee","journal-title":"Comput Educ Artif Intell"},{"issue":"11","key":"R20","doi-asserted-by":"publisher","first-page":"1772","DOI":"10.1080\/0142159X.2025.2504106","article-title":"Using large language models (LLMs) to apply analytic rubrics to score post-encounter notes","volume":"47","author":"Runyon","journal-title":"Med Teach"},{"issue":"12","key":"R21","doi-asserted-by":"publisher","DOI":"10.1056\/AIcs2400631","article-title":"Rubrics to prompts: assessing medical student post-encounter notes with AI","volume":"1","author":"Jamieson","journal-title":"NEJM AI"},{"issue":"6","key":"R22","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2025.101260","article-title":"Unleashing the potential of prompt engineering for large language models","volume":"6","author":"Chen","journal-title":"Patterns (N Y)"},{"issue":"7972","key":"R23","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"key":"R24","unstructured":"Han T Adams LC Papaioannou JM et al. MedAlpaca\u2013an open-source collection of medical conversational AI models and training data. arXiv. Preprint posted online on Apr 14, 2023. [doi: 10.48550\/arXiv.2304.08247]"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00455-y","article-title":"Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction","volume":"4","author":"Rasmy","journal-title":"NPJ Digit Med"},{"key":"R26","volume":"6","author":"Lin","journal-title":"Proc Mach Learn Syst"},{"key":"R27","unstructured":"CASUS. CASUS online. URL: https:\/\/crt.casus.net\/pmw2\/app\/homepage.html [Accessed 14-09-2026]"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Kojima T Gu S Reid M Matsuo Y Iwasawa Y . Large language models are zero-shot reasoners. Presented at: Advances in Neural Information Processing Systems 35; Nov 28 to Dec 9, 2022:22199-22213; New Orleans, LA. 2022.[doi: 10.52202\/068431-1613]","DOI":"10.52202\/068431-1613"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Jiang Y Wang Y Zeng X et al. FollowBench: a multi-level fine-grained constraints following benchmark for large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1. Association for Computational Linguistics; 2024:4667-4688. [doi: 10.18653\/v1\/2024.acl-long.257]","DOI":"10.18653\/v1\/2024.acl-long.257"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Kwon W Li Z Zhuang S et al. Efficient memory management for large language model serving with pagedattention. In: SOSP \u201923: Proceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery; 2023:611-626. [doi: 10.1145\/3600006.3613165]","DOI":"10.1145\/3600006.3613165"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"At\u0131l B Aykent S Chittams A . Non-determinism of \u201cdeterministic\u201d LLM system settings in hosted environments. In: Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems. 2025:135-148. [doi: 10.18653\/v1\/2025.eval4nlp-1.12]","DOI":"10.18653\/v1\/2025.eval4nlp-1.12"},{"key":"R32","unstructured":"R Core Team. The R Language Definition. Vienna Austria R Found Stat Comput. 2000. URL: https:\/\/cran.r-project.org\/doc\/manuals\/r-devel\/R-lang.html"},{"issue":"2","key":"R33","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","journal-title":"J Chiropr Med"},{"issue":"6","key":"R34","doi-asserted-by":"publisher","first-page":"1219","DOI":"10.1037\/a0037489","article-title":"Interrater agreement statistics with skewed data: evaluation of alternatives to Cohen\u2019s kappa","volume":"82","author":"Xu","journal-title":"J Consult Clin Psychol"},{"issue":"Pt 1","key":"R35","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"Gwet","journal-title":"Br J Math Stat Psychol"},{"key":"R36","unstructured":"Gwet KL . Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters. Advanced Analytics, LLC; 2014. URL: https:\/\/books.google.co.in\/books\/about\/Handbook_of_Inter_Rater_Reliability_4th.html?id=fac9BQAAQBAJ&redir_esc=y [Accessed 07-10-2026]"},{"issue":"1","key":"R37","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-13-61","article-title":"A comparison of Cohen\u2019s Kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples","volume":"13","author":"Wongpakaran","journal-title":"BMC Med Res Methodol"},{"issue":"7","key":"R38","doi-asserted-by":"publisher","first-page":"3326","DOI":"10.3758\/s13428-022-01950-0","article-title":"Better to be in agreement than in bad company","volume":"55","author":"Silveira","journal-title":"Behav Res"},{"issue":"11","key":"R39","doi-asserted-by":"publisher","first-page":"1115","DOI":"10.1089\/acm.2017.0297","article-title":"Appropriate statistics for determining chance-removed interpractitioner agreement","volume":"25","author":"Popplewell","journal-title":"J Altern Complement Med"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","journal-title":"Biometrics"},{"key":"R41","doi-asserted-by":"crossref","unstructured":"Se\u00dfler K F\u00fcrstenberg M B\u00fchler B Kasneci E . Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In: LAK \u201925. Association for Computing Machinery; 2025:462-472. [doi: 10.1145\/3706468.3706527]","DOI":"10.1145\/3706468.3706527"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.3389\/feduc.2023.1272229","article-title":"Is GPT-4 a reliable rater? Evaluating consistency in GPT-4\u2019s text ratings","volume":"8","author":"Hackl","journal-title":"Front Educ"},{"issue":"1","key":"R43","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-07624-7","article-title":"Comparison of accuracy and consistency of AI Language models when answering standardised dental MCQs","volume":"25","author":"Alshammari","journal-title":"BMC Med Educ"},{"issue":"1","key":"R44","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/s11606-024-09050-9","article-title":"Comparing scoring consistency of large language models with faculty for formative assessments in medical education","volume":"40","author":"Sreedhar","journal-title":"J Gen Intern Med"},{"key":"R45","doi-asserted-by":"crossref","unstructured":"Xiao C Ma W Song Q et al. Human-AI collaborative essay scoring: a dual-process framework with llms. In: LAK \u201925. Association for Computing Machinery; 2025:293-305. [doi: 10.1145\/3706468.3706507]","DOI":"10.1145\/3706468.3706507"},{"issue":"2","key":"R46","doi-asserted-by":"publisher","first-page":"669","DOI":"10.1007\/s10758-025-09836-8","article-title":"Unveiling scoring processes: dissecting the differences between LLMs and human graders in automatic scoring","volume":"31","author":"Wu","journal-title":"Tech Know Learn"},{"issue":"4","key":"R47","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1111\/jep.13969","article-title":"The inseparability of context and clinical reasoning","volume":"30","author":"Olson","journal-title":"J Eval Clin Pract"},{"key":"R48","unstructured":"Yuan Z Yuan H Tan C Wang W Huang S . How well do large language models perform in arithmetic tasks? arXiv. Preprint posted online on Mar 16, 2023. [doi: 10.48550\/ARXIV.2304.02015]"},{"issue":"1","key":"R49","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1080\/0142159X.2025.2519639","article-title":"Bytes versus brains: a comparative study of AI-generated feedback and human tutor feedback in medical education","volume":"48","author":"Ali","journal-title":"Med Teach"},{"issue":"5","key":"R50","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1007\/s12599-019-00595-2","article-title":"Hybrid Intelligence","volume":"61","author":"Dellermann","journal-title":"Bus Inf Syst Eng"},{"key":"R51","doi-asserted-by":"crossref","unstructured":"Lai V Ngo N Pouran Ben Veyseh A et al. ChatGPT beyond English: towards a comprehensive evaluation of large language models in multilingual learning. Presented at: Findings of the Association for Computational Linguistics: EMNLP 2023; Dec 6-10, 2023:13171-13189; Singapore. [doi: 10.18653\/v1\/2023.findings-emnlp.878]","DOI":"10.18653\/v1\/2023.findings-emnlp.878"},{"key":"R52","doi-asserted-by":"crossref","unstructured":"Shojaee P Mirzadeh I Alizadeh K Horton M Bengio S Farajtabar M . The illusion of thinking: understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv. Preprint posted online on Jun 7, 2025. [doi: 10.48550\/ARXIV.2506.06941]","DOI":"10.70777\/si.v2i6.15919"},{"key":"R53","unstructured":"Chen Y Benton J Radhakrishnan A et al. Reasoning models don\u2019t always say what they think. arXiv. Preprint posted online on May 8, 2025. [doi: 10.48550\/ARXIV.2505.05410]"},{"key":"R54","unstructured":"International Collection of Virtual Patients (iCoViP). URL: https:\/\/icovip.eu [Accessed 08-10-2026]"},{"key":"R55","unstructured":"CASUS. URL: https:\/\/icovip.casus.net [Accessed 08-10-2026]"},{"key":"R56","unstructured":"MedicalSummaryStatements_LLMs. GitHub. URL: https:\/\/github.com\/DSFZ-Team\/MedicalSummaryStatements_LLMs [Accessed 08-10-2026]"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T20:45:22Z","timestamp":1792615522000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e94666"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,21]]},"references-count":56,"URL":"https:\/\/doi.org\/10.2196\/94666","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,21]]},"article-number":"v12i10 Supple94666"}}