{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T07:18:05Z","timestamp":1790666285143,"version":"4.1.0"},"reference-count":26,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Standard setting is essential for a defensible assessment in medical education. The modified Angoff method requires several expert judges, and determining the minimally competent candidate is cognitively challenging. However, empirical evidence on the role of AI in standard setting is unclear.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to examine the role of large language models (LLMs) in the modified Angoff standard setting method for multiple-choice questions in a basic medical science examination compared with faculty judges.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n This study was conducted in year 2 of the MD program in the United Arab Emirates. Ten faculty judges and 5 LLMs (GPT-5.2, Grok, DeepSeek, MedGemma, and Claude 4.5 Sonnet) determined the Angoff cutoff scores for a summative examination (120 multiple-choice questions). Standardized prompts were used for the LLMs to mimic the same information provided for faculty judges. Generalizability (\n G<\/jats:italic>\n ) theory analyses were performed using a fully crossed item \u00d7 rater design to estimate variance components,\n G<\/jats:italic>\n and \u03a6 coefficients, decision study, and root mean squared error (RMSE) of the Angoff cutoff scores.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Faculty-generated Angoff estimates (mean 69.13, SD 10.92) were comparable to LLM-generated estimates (mean 68.94, SD 12.39). A 2-tailed paired-sample\n t<\/jats:italic>\n test revealed no statistically significant difference between the 2 groups (95% CI \u22122.21 to 2.91;\n t<\/jats:italic>\n 119<\/jats:sub>\n =0.27;\n P<\/jats:italic>\n =.79). Generalizability theory analysis demonstrated moderate reliability for the faculty panel (\n G<\/jats:italic>\n coefficient=0.738; \u03a6 coefficient=0.692). Despite comprising only 5 LLMs, the LLM panel demonstrated higher reliability (\n G<\/jats:italic>\n coefficient=0.823; \u03a6 coefficient=0.815), lower RMSE (1.28 vs 2.33), higher item-related variance (46.85% vs 18.36%), and lower rater-related variance (2.64% vs 16.40%) than the faculty panel. Pass rates were similar using LLM- and faculty-derived cutoff scores (52\/73, 71.2%). LLMs differed in their minimally competent candidate conceptualization and approaches to determining item-level percentages of correct responses. Furthermore, the correlations between Angoff estimates and item-related\n P<\/jats:italic>\n values were larger in LLMs than in faculty judges (\n r<\/jats:italic>\n =0.552 vs 0.437).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n \n In this single-institution study, the evaluated LLMs generated modified Angoff estimates that were broadly comparable to those of faculty judges. Generalizability theory analyses demonstrated higher\n G<\/jats:italic>\n and \u03a6 coefficients, lower rater-related variance, and lower RMSE for the evaluated LLM outputs under standardized prompting conditions, indicating greater consistency in the generated estimates. Application of the resulting cutoff scores produced pass and fail rates similar to those derived from faculty judges. These findings suggest the future application of LLMs as decision assistance tools for modified Angoff standard setting while maintaining expert human oversight.\n <\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/98766","type":"journal-article","created":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T20:41:52Z","timestamp":1788986512000},"page":"e98766-e98766","source":"Crossref","is-referenced-by-count":1,"title":["AI-Assisted Angoff Standard Setting for Multiple-Choice Examinations in Medical Education: Comparative Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1482-1996","authenticated-orcid":false,"given":"Salah Eldin","family":"Kassab","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9660-9149","authenticated-orcid":false,"given":"Mariam","family":"Shadan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5833-9160","authenticated-orcid":false,"given":"Arina","family":"Ziganshina","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4350-0393","authenticated-orcid":false,"given":"Shifan","family":"Khanday","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6096-6139","authenticated-orcid":false,"given":"Hossam","family":"Hamdy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,9]]},"reference":[{"issue":"5","key":"R1","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1046\/j.1365-2923.2003.01495.x","article-title":"Setting standards on educational tests","volume":"37","author":"Norcini","journal-title":"Med Educ"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1207\/s15324818ame1001_3","article-title":"The credibility and comparability of standards","volume":"10","author":"Norcini","journal-title":"Appl Meas Educ"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1207\/s15324818ame0501_5","article-title":"The reproducibility of standards over groups and occasions","volume":"5","author":"Norcini","journal-title":"Appl Meas Educ"},{"issue":"7","key":"R4","doi-asserted-by":"publisher","first-page":"1309","DOI":"10.1002\/jdd.12609","article-title":"Adaptive e-learning platforms can improve student performance and engagement in dental education","volume":"85","author":"Yakin","journal-title":"J Dent Educ"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6920-6-46","article-title":"Standard setting: comparison of two methods","volume":"6","author":"George","journal-title":"BMC Med Educ"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-016-0656-7","article-title":"Insights into the Angoff method: results from a simulation study","volume":"16","author":"Shulruf","journal-title":"BMC Med Educ"},{"issue":"12","key":"R7","doi-asserted-by":"publisher","first-page":"1199","DOI":"10.1111\/j.1365-2923.2011.04073.x","article-title":"Comparison of two methods of standard setting: the performance of the three-level Angoff method","volume":"45","author":"Jalili","journal-title":"Med Educ"},{"issue":"10 Suppl","key":"R8","doi-asserted-by":"publisher","first-page":"S85","DOI":"10.1097\/00001888-200310001-00027","article-title":"Establishing passing standards for classroom achievement tests in medical education: a comparative study of four methods","volume":"78","author":"Downing","journal-title":"Acad Med"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-08300-6","article-title":"Angoff methods in standard setting in health professional education: a systematic review and meta-analysis","volume":"25","author":"Sridharan","journal-title":"BMC Med Educ"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1111\/j.1745-3984.1988.tb00291.x","article-title":"The effect of various factors on standard setting","volume":"25","author":"Norcini","journal-title":"J Educ Meas"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.55016\/ojs\/ajer.v52i1.55111","article-title":"Setting cut-scores: a critical review of the Angoff and modified Angoff methods","volume":"52","author":"Ricker","journal-title":"Alta J Educ Res"},{"issue":"2","key":"R12","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1111\/emip.12399","article-title":"Evaluating panelists\u2019 understanding of standard setting data","volume":"40","author":"Baron","journal-title":"Educ Meas"},{"issue":"8","key":"R13","doi-asserted-by":"publisher","first-page":"2574","DOI":"10.1002\/bcp.70137","article-title":"Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: from exam takers to exam shapers","volume":"92","author":"G\u00e9rard","journal-title":"Br J Clin Pharmacol"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-05365-7","article-title":"Artificial intelligence and medical education: application in classroom instruction and student assessment using a pharmacology & therapeutics case study","volume":"24","author":"Sridharan","journal-title":"BMC Med Educ"},{"issue":"9","key":"R15","doi-asserted-by":"publisher","first-page":"1410","DOI":"10.1080\/0142159X.2024.2445037","article-title":"Artificial intelligence in health professions education assessment: AMEE guide no. 178","volume":"47","author":"Masters","journal-title":"Med Teach"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1007\/s11528-016-0091-y","article-title":"The Substitution Augmentation Modification Redefinition (SAMR) model: a critical review and suggestions for its use","volume":"60","author":"Hamilton","journal-title":"TechTrends"},{"issue":"7","key":"R17","doi-asserted-by":"publisher","first-page":"1101","DOI":"10.1080\/0142159X.2025.2581163","article-title":"The role of artificial intelligence in co-creation of health professions education: integrating innovation into collaboration: AMEE guide no. 190","volume":"48","author":"Suliman","journal-title":"Med Teach"},{"issue":"11","key":"R18","doi-asserted-by":"publisher","first-page":"832","DOI":"10.1046\/j.1365-2923.1999.00487.x","article-title":"Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judges","volume":"33","author":"Verhoeven","journal-title":"Med Educ"},{"issue":"2","key":"R19","doi-asserted-by":"publisher","first-page":"154","DOI":"10.3109\/01421590903196979","article-title":"A standard setting method with the best performing students as point of reference: practical and affordable","volume":"32","author":"Cohen-Schotanus","journal-title":"Med Teach"},{"key":"R20","doi-asserted-by":"crossref","unstructured":"Champlain AF . Standard setting methods in medical education: high\u2010stakes assessment. In: Swanwick T Forrest K O\u2019Brien BC , editors. Understanding Medical Education: Evidence, Theory, and Practice. 3rd ed. John Wiley & Sons; 2018. [doi: 10.1002\/9781119373780.ch24]","DOI":"10.1002\/9781119373780.ch24"},{"issue":"8","key":"R21","doi-asserted-by":"publisher","first-page":"810","DOI":"10.1111\/j.1365-2923.2008.03103.x","article-title":"The practical value of the standard error of measurement in borderline pass\/fail decisions","volume":"42","author":"Hays","journal-title":"Med Educ"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/j.jsp.2013.11.008","article-title":"Generalizability theory: a practical guide to study design, implementation, and interpretation","volume":"52","author":"Briesch","journal-title":"J Sch Psychol"},{"issue":"9","key":"R23","doi-asserted-by":"publisher","first-page":"860","DOI":"10.1046\/j.1365-2923.2002.01301.x","article-title":"Panel expertise for an Angoff standard setting procedure in progress testing: item writers compared to recently graduated students","volume":"36","author":"Verhoeven","journal-title":"Med Educ"},{"key":"R24","unstructured":"Angoff WH . Scales, Norms, and Equivalent Scores. Educational Testing Service; 1984."},{"issue":"3","key":"R25","doi-asserted-by":"publisher","first-page":"425","DOI":"10.3102\/00346543064003425","article-title":"Validating the performance standards associated with passing scores","volume":"64","author":"Kane","journal-title":"Rev Educ Res"},{"key":"R26","unstructured":"Luckin R Holmes W Griffiths M Forcier LB . Intelligence Unleashed. An Argument for AI in Education. Pearson Education; 2016. ISBN: 9780992424886"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T20:41:53Z","timestamp":1788986513000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e98766"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,9]]},"references-count":26,"URL":"https:\/\/doi.org\/10.2196\/98766","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,9]]},"article-number":"v12i5e98766"}}