{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T14:12:49Z","timestamp":1788876769240,"version":"build-2803163510"},"reference-count":66,"publisher":"Wiley","license":[{"start":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T00:00:00Z","timestamp":1788825600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"},{"start":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T00:00:00Z","timestamp":1788825600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["asmepublications.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Education"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Artificial intelligence (AI) is increasingly used to generate, score and interpret educational assessment, yet these applications are being adopted in a largely unregulated environment. This creates a paradox: Whereas AI systems used in clinical care are subject to formal scrutiny for safety, performance and monitoring, AI systems used to inform consequential decisions about learner progression and future clinical practice are not. Existing validity frameworks remain useful but may not fully account for AI\u2010specific threats.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We conducted a state\u2010of\u2010the\u2010science conceptual review examining validity for AI\u2010based assessment through Kane's four inferences: scoring, generalisation, extrapolation and implications. We integrated conceptual and empirical literature from medical education and adjacent AI evaluation fields to identify major threats, evidence needs and practical implications for defensible AI\u2010based assessment use.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>AI introduces distinct validity threats across the inferential chain. For scoring, key risks include construct contamination, prompt instability and limited explainability. For generalisation, domain shift, rater culture differences and temporal drift threaten reproducibility across settings and time. For extrapolation, AI may alter the construct being assessed, making explicit construct definition essential to defensible interpretation. For implications, risks include subgroup inequities, automation bias, deskilling, weak accountability and erosion of academic integrity. Across these inferences, we identified seven cross\u2010cutting mechanisms that can operate as either threats or affordances depending on design and governance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Validity for AI\u2010based assessment should be approached as ongoing governance of a sociotechnical system rather than one\u2010time validation of a static instrument or assessment process. Defensible adoption requires explicit validity arguments, auditability, multi\u2010site evaluation and continuous consequence monitoring that is aware of the specific AI affordances and threats.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1111\/medu.70309","type":"journal-article","created":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T14:04:11Z","timestamp":1788876251000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Developing validity arguments for artificial intelligence\u2010based assessment: Balancing affordances and threats"],"prefix":"10.1111","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9197-5564","authenticated-orcid":false,"given":"Martin G.","family":"Tolsgaard","sequence":"first","affiliation":[{"name":"Copenhagen Academy for Medical Education and Simulation (CAMES), Copenhagen University Hospital Rigshospitalet University of Copenhagen  Copenhagen Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lawrence","family":"Grierson","sequence":"additional","affiliation":[{"name":"Department of Family Medicine McMaster University  Hamilton Ontario Canada"},{"name":"McMaster Education Research, Innovation, and Theory (MERIT) Centre McMaster University  Hamilton Ontario Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5236-6598","authenticated-orcid":false,"given":"Martin V.","family":"Pusic","sequence":"additional","affiliation":[{"name":"Boston Children's Hospital, Harvard Medical School  Boston Massachusetts USA"},{"name":"American Board of Medical Specialties  Chicago Illinois USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4567-1313","authenticated-orcid":false,"given":"Laurah","family":"Turner","sequence":"additional","affiliation":[{"name":"Department of Medical Education University of Cincinnati College of Medicine  Cincinnati Ohio USA"},{"name":"Department of Biostatistics, Health Informatics, and Data Science University of Cincinnati College of Medicine  Cincinnati Ohio USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,9,8]]},"reference":[{"issue":"9","key":"e_1_2_16_2_1","doi-asserted-by":"crossref","DOI":"10.1001\/jamahealthforum.2025.3351","article-title":"Benefit\u2010risk reporting for FDA\u2010cleared AI\u2010enabled medical devices","volume":"6","author":"Lin JC","year":"2025","journal-title":"JAMA Health Forum"},{"key":"e_1_2_16_3_1","doi-asserted-by":"crossref","DOI":"10.1136\/bmj-2024-081554","article-title":"FUTURE\u2010AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare","volume":"388","author":"Lekadir K","year":"2025","journal-title":"BMJ"},{"key":"e_1_2_16_4_1","unstructured":"European Medicines Agency (EMA).Medical devices.https:\/\/www.ema.europa.eu\/en\/human-regulatory-overview\/medical-devices. Accessed February 27 2026."},{"issue":"1","key":"e_1_2_16_5_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/jedm.12000","article-title":"Validating the interpretations and uses of test scores","volume":"50","author":"Kane MT","year":"2013","journal-title":"J Educ Meas"},{"key":"e_1_2_16_6_1","doi-asserted-by":"crossref","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson A","year":"2023","journal-title":"JMIR Med Educ"},{"issue":"6","key":"e_1_2_16_7_1","doi-asserted-by":"crossref","first-page":"2046","DOI":"10.1016\/j.jsurg.2021.06.013","article-title":"Natural language processing to estimate clinical competency committee ratings","volume":"78","author":"Abbott KL","year":"2021","journal-title":"J Surg Educ"},{"key":"e_1_2_16_8_1","doi-asserted-by":"publisher","DOI":"10.1056\/AIcs2400631"},{"issue":"3","key":"e_1_2_16_9_1","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1093\/ehjdh\/ztad029","article-title":"ChatGPT takes on the European Exam in Core Cardiology: an artificial intelligence success story?","volume":"4","author":"Skalidis I","year":"2023","journal-title":"Eur Heart J Digit Health"},{"issue":"3","key":"e_1_2_16_10_1","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1111\/medu.14696","article-title":"Exploring how feedback reflects entrustment decisions using artificial intelligence","volume":"56","author":"Gin BC","year":"2022","journal-title":"Med Educ"},{"issue":"5","key":"e_1_2_16_11_1","doi-asserted-by":"crossref","first-page":"534","DOI":"10.1097\/ACM.0000000000005634","article-title":"Using natural language processing to evaluate the quality of supervisor narrative comments in competency\u2010based medical education","volume":"99","author":"Spadafore M","year":"2024","journal-title":"Acad Med"},{"key":"e_1_2_16_12_1","doi-asserted-by":"publisher","DOI":"10.1111\/medu.15190"},{"key":"e_1_2_16_13_1","unstructured":"WuD HaredashtFN MaharajSK et al.First do NOHARM: towards clinically safe large language models. arXiv.2025;2512.01241."},{"issue":"11","key":"e_1_2_16_14_1","doi-asserted-by":"crossref","first-page":"1262","DOI":"10.1111\/medu.15474","article-title":"Why we should stop writing commentaries about AI","volume":"58","author":"Johnsson V","year":"2024","journal-title":"Med Educ"},{"issue":"9","key":"e_1_2_16_15_1","doi-asserted-by":"crossref","first-page":"S34","DOI":"10.1097\/ACM.0000000000006104","article-title":"It takes more than enthusiasm: the missing infrastructure to unlock AI's potential in medical education","volume":"100","author":"Turner L","year":"2025","journal-title":"Acad Med"},{"issue":"1","key":"e_1_2_16_16_1","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1111\/medu.15161","article-title":"Researching models of innovation and adoption in health professions education","volume":"58","author":"Pusic MV","year":"2024","journal-title":"Med Educ"},{"key":"e_1_2_16_17_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591\u2010024\u201003328\u20105"},{"key":"e_1_2_16_18_1","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2500144"},{"key":"e_1_2_16_19_1","article-title":"Judging LLM\u2010as\u2010a\u2010judge with MT\u2010Bench and Chatbot Arena","volume":"36","author":"Zheng L","year":"2023","journal-title":"Adv Neural Inf Proces Syst"},{"issue":"6671","key":"e_1_2_16_20_1","doi-asserted-by":"crossref","first-page":"669","DOI":"10.1126\/science.adi6000","article-title":"Prediction\u2010powered inference","volume":"382","author":"Angelopoulos AN","year":"2023","journal-title":"Science"},{"key":"e_1_2_16_21_1","unstructured":"AngelopoulosAN DuchiJC ZrnicT.PPI++: efficient prediction\u2010powered inference. arXiv preprint.2023."},{"key":"e_1_2_16_22_1","unstructured":"BoyeauP AngelopoulosAN YosefN MalikJ JordanMI.AutoEval Done Right: using autolabelers for evaluation. arXiv preprint.2024."},{"key":"e_1_2_16_23_1","doi-asserted-by":"crossref","unstructured":"Saad\u2010FalconJ KhattabO PottsC ZahariaM.ARES: an automated evaluation framework for retrieval\u2010augmented generation systems. In: Proceedings of NAACL2024.","DOI":"10.18653\/v1\/2024.naacl-long.20"},{"key":"e_1_2_16_24_1","doi-asserted-by":"crossref","unstructured":"ChagantyA MussmannS LiangP.The price of debiasing automatic metrics in natural language evaluation. In: Proceedings of ACL2018.","DOI":"10.18653\/v1\/P18-1060"},{"key":"e_1_2_16_25_1","doi-asserted-by":"crossref","unstructured":"PanicksseryA BowmanSR FengS.LLM evaluators recognize and favor their own generations. arXiv preprint.2024.","DOI":"10.52202\/079017-2197"},{"key":"e_1_2_16_26_1","unstructured":"JungJ BrahmanF ChoiY.Trust or Escalate: LLM judges with provable guarantees for human agreement. arXiv preprint.2024."},{"issue":"1","key":"e_1_2_16_27_1","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1007\/s11701-025-02994-y","article-title":"Closing the data gap: leveraging pretrained neural networks for robotic surgical assessment on limited clinical data","volume":"20","author":"Hashemi N","year":"2025","journal-title":"J Robot Surg"},{"key":"e_1_2_16_28_1","doi-asserted-by":"publisher","DOI":"10.1093\/acamed\/wvaf065"},{"key":"e_1_2_16_29_1","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0000000000005527"},{"issue":"1","key":"e_1_2_16_30_1","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1007\/s11701-023-01807-4","article-title":"Surgical gestures can be used to assess surgical competence in robot\u2010assisted surgery: a validity investigating study of simulated RARP","volume":"18","author":"Olsen RG","year":"2024","journal-title":"J Robot Surg"},{"key":"e_1_2_16_31_1","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2025.2555353"},{"issue":"6","key":"e_1_2_16_32_1","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1080\/0142159X.2023.2180340","article-title":"The fundamentals of artificial intelligence in medical education research: AMEE Guide No. 156","volume":"45","author":"Tolsgaard MG","year":"2023","journal-title":"Med Teach"},{"key":"e_1_2_16_33_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598\u2010023\u201029490\u20103"},{"key":"e_1_2_16_34_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598\u2010023\u201029105\u2010x"},{"issue":"1","key":"e_1_2_16_35_1","doi-asserted-by":"crossref","first-page":"2074","DOI":"10.1038\/s41598-025-86536-4","article-title":"Clinical validation of explainable AI for fetal growth scans through multi\u2010level, cross\u2010institutional prospective end\u2010user evaluation","volume":"15","author":"Bashir Z","year":"2025","journal-title":"Sci Rep"},{"issue":"4","key":"e_1_2_16_36_1","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1002\/jvc2.59","article-title":"Generalizability and usefulness of artificial intelligence for skin cancer diagnostics: an algorithm validation study","volume":"1","author":"Ternov NK","year":"2022","journal-title":"JEADV Clin Pract"},{"issue":"5","key":"e_1_2_16_37_1","doi-asserted-by":"crossref","first-page":"1057","DOI":"10.1007\/s10459-020-10009-8","article-title":"The role of data science and machine learning in health professions education: practical applications, theoretical contributions, and epistemic beliefs","volume":"25","author":"Tolsgaard MG","year":"2020","journal-title":"Adv Health Sci Educ Theory Pract"},{"issue":"2","key":"e_1_2_16_38_1","doi-asserted-by":"crossref","DOI":"10.1097\/AS9.0000000000000442","article-title":"Leveraging education science for AI\u2010clinician collaboration in the patient care ecosystem","volume":"5","author":"Tolsgaard MG","year":"2024","journal-title":"Ann Surg Open"},{"key":"e_1_2_16_39_1","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1007\/978-3-031-72111-3_59","volume-title":"Medical Image Computing and Computer Assisted Intervention \u2013 MICCAI 2024","author":"Lin M","year":"2024"},{"key":"e_1_2_16_40_1","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1038\/s41746-022-00566-0","article-title":"Machine learning for technical skill assessment in surgery: a systematic review","volume":"5","author":"Lam K","year":"2022","journal-title":"NPJ Digit Med"},{"issue":"1","key":"e_1_2_16_41_1","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1038\/s41746-022-00738-y","article-title":"Surgical gestures as a method to quantify surgical performance and predict patient outcomes","volume":"5","author":"Ma R","year":"2022","journal-title":"NPJ Digit Med"},{"key":"e_1_2_16_42_1","doi-asserted-by":"crossref","first-page":"835","DOI":"10.1007\/s40593-021-00268-w","article-title":"Early prediction of writing quality using keystroke logging","volume":"32","author":"Conijn R","year":"2022","journal-title":"Int J Artif Intell Educ"},{"key":"e_1_2_16_43_1","unstructured":"NguyenT PentapalliLS SieverdingM et al.Jailbreak detection in clinical training LLMs using feature\u2010based predictive models. arXiv. 2025;2505.00010."},{"key":"e_1_2_16_44_1","unstructured":"ManheimD GarrabrantS.Categorizing variants of Goodhart's law. arXiv.2018;1803.04585.10.48550\/arXiv.1803.04585"},{"key":"e_1_2_16_45_1","unstructured":"GanguliD LovittL KernionJ et al.Red teaming language models to reduce harms: methods scaling behaviors and lessons learned. arXiv.2022;2209.07858."},{"issue":"1","key":"e_1_2_16_46_1","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: a systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"Goddard K","year":"2012","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"e_1_2_16_47_1","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1177\/0018720810376055","article-title":"Complacency and bias in human use of automation: an attentional integration","volume":"52","author":"Parasuraman R","year":"2010","journal-title":"Hum Factors"},{"issue":"8","key":"e_1_2_16_48_1","doi-asserted-by":"crossref","first-page":"1506","DOI":"10.1177\/0018720814535628","article-title":"The retention of manual flying skills in the automated cockpit","volume":"56","author":"Casner SM","year":"2014","journal-title":"Hum Factors"},{"key":"e_1_2_16_49_1","unstructured":"CharetteRN.Air France Flight 447 crash causes in part point to automation paradox. IEEE Spectrum. July 10 2012."},{"key":"e_1_2_16_50_1","doi-asserted-by":"crossref","unstructured":"ShawSD NaveG.Thinking fast slow and artificial: how AI is reshaping human reasoning and the rise of cognitive surrender. OSF Preprints2026.10.31234\/osf.io\/yk25n_v1","DOI":"10.31234\/osf.io\/yk25n_v1"},{"issue":"11","key":"e_1_2_16_51_1","first-page":"1828","article-title":"Breast Cancer Surveillance Consortium. Diagnostic accuracy of digital screening mammography with and without computer\u2010aided detection. JAMA","volume":"175","author":"Lehman CD","year":"2015","journal-title":"Intern Med"},{"issue":"8","key":"e_1_2_16_52_1","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1038\/s41568-018-0016-5","article-title":"Artificial intelligence in radiology","volume":"18","author":"Hosny A","year":"2018","journal-title":"Nat Rev Cancer"},{"issue":"12","key":"e_1_2_16_53_1","doi-asserted-by":"crossref","first-page":"2293","DOI":"10.1038\/s41562-024-02024-1","article-title":"When combinations of humans and AI are useful: a systematic review and meta\u2010analysis","volume":"8","author":"Vaccaro M","year":"2024","journal-title":"Nat Hum Behav"},{"key":"e_1_2_16_54_1","doi-asserted-by":"publisher","DOI":"10.1016\/S2468\u20101253(25)00133\u20105"},{"key":"e_1_2_16_55_1","article-title":"The tragic crash of Flight AF447 shows the unlikely but catastrophic consequences of automation","author":"Oliver N","year":"2017","journal-title":"Harv Bus Rev"},{"key":"e_1_2_16_56_1","doi-asserted-by":"crossref","first-page":"6310","DOI":"10.1038\/s41598-020-62877-0","article-title":"Habitual use of GPS negatively impacts spatial memory during self\u2010guided navigation","volume":"10","author":"Dahmani L","year":"2020","journal-title":"Sci Rep"},{"key":"e_1_2_16_57_1","unstructured":"KosmynaN HauptmannE YuanYT et al.Your brain on ChatGPT: accumulation of cognitive debt when using an AI assistant for essay writing task. arXiv.2025;2506.08872."},{"key":"e_1_2_16_58_1","doi-asserted-by":"publisher","DOI":"10.2196\/106486"},{"issue":"4","key":"e_1_2_16_59_1","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1080\/00461520.2011.611369","article-title":"The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems","volume":"46","author":"VanLehn K","year":"2011","journal-title":"Educ Psychol"},{"issue":"1","key":"e_1_2_16_60_1","doi-asserted-by":"crossref","first-page":"7764","DOI":"10.1038\/s41598-021-87148-4","article-title":"Rethinking GPS navigation: creating cognitive maps through auditory clues","volume":"11","author":"Clemenson GD","year":"2021","journal-title":"Sci Rep"},{"issue":"3","key":"e_1_2_16_61_1","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1037\/0033-2909.112.3.527","article-title":"An argument\u2010based approach to validity","volume":"112","author":"Kane MT","year":"1992","journal-title":"Psychol Bull"},{"issue":"1","key":"e_1_2_16_62_1","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1097\/SLA.0000000000006367","article-title":"Opportunities and applications of educational technologies in surgical education and assessment","volume":"281","author":"Fried GM","year":"2025","journal-title":"Ann Surg"},{"key":"e_1_2_16_63_1","article-title":"Article 13: transparency and provision of information to deployers","author":"Regulation (EU) 2024\/1689","year":"2024","journal-title":"Off J Eur Union"},{"key":"e_1_2_16_64_1","doi-asserted-by":"publisher","DOI":"10.3389\/feduc.2022.857496"},{"issue":"7","key":"e_1_2_16_65_1","doi-asserted-by":"crossref","DOI":"10.1016\/j.patter.2023.100790","article-title":"The path toward equal performance in medical machine learning","volume":"4","author":"Petersen E","year":"2023","journal-title":"Patterns"},{"key":"e_1_2_16_66_1","first-page":"3315","article-title":"Equality of opportunity in supervised learning","volume":"29","author":"Hardt M","year":"2016","journal-title":"Adv Neural Inf Proces Syst"},{"issue":"6","key":"e_1_2_16_67_1","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi N","year":"2022","journal-title":"ACM Comput Surv"}],"container-title":["Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/medu.70309","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/medu.70309","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/medu.70309","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T14:04:21Z","timestamp":1788876261000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/10.1111\/medu.70309"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,8]]},"references-count":66,"alternative-id":["10.1111\/medu.70309"],"URL":"https:\/\/doi.org\/10.1111\/medu.70309","archive":["Portico"],"relation":{},"ISSN":["0308-0110","1365-2923"],"issn-type":[{"value":"0308-0110","type":"print"},{"value":"1365-2923","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,8]]},"assertion":[{"value":"2026-03-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-20","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-09-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"medu.70309"}}