{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T15:01:24Z","timestamp":1780498884973,"version":"3.54.1"},"reference-count":49,"publisher":"Oxford University Press (OUP)","license":[{"start":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T00:00:00Z","timestamp":1777507200000},"content-version":"am","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"National Board of Medical Examiners (NBME) Edward J. Stemmler, MD Medical Education Research Fund grant"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>Medical knowledge is a critical competency for physicians. Assessment of medical knowledge is distinguished between acquisition and application. Studies have demonstrated the association in acquisition throughout training; however, few have assessed the association between acquisition and application to practice. The purpose of this study was to explore the relationship between medical knowledge acquisition and application across a nationally representative data set.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Method<\/jats:title>\n                    <jats:p>This was a multi-institutional, multispecialty retrospective study of data from medical school graduates at 7 institutions entering residency training from 2016 to 2018. Medical knowledge acquisition was assessed using scores on United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge. Medical knowledge application was assessed using Accreditation Council for Graduate Medical Education milestone ratings during internship. Mixed-effects regression was used to estimate the predictive association between USMLE performance and Accreditation Council for Graduate Medical Education Milestones, clustering specialty, and program effects using coefficients reflecting beta estimates.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Data were analyzed from 3430 medical school graduates. Failing USMLE Step 1 was associated with a higher risk of being rated as \u201cnot yet level 1\u201d in patient care milestones at mid-year (odds ratio\u2009=\u20094.19, P = .007), whereas failure of USMLE Step 2 was associated with the same in medical knowledge (odds ratio\u2009=\u20097.56, P\u2009=\u2009.15). Higher USMLE scores were associated with higher milestone ratings, though the effect size was very small (standard coefficient range .01\u2013.19). There were no significant associations between USMLE scores and end-of-year milestone ratings for any analyses.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Previous data suggest USMLE scores provide value in identifying future standardized examination (ie, medical knowledge acquisition) performance. These findings demonstrate less compelling evidence using USMLE to predict application of knowledge into practice. USMLE pass\/fail status is the most predictive metric of medical knowledge application and may be useful in creating targeted remediation plans.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/acamed\/wvag131","type":"journal-article","created":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T04:47:25Z","timestamp":1777610845000},"source":"Crossref","is-referenced-by-count":0,"title":["Does USMLE performance predict residency milestone ratings in medical knowledge and patient care? A multisite, multispecialty study"],"prefix":"10.1093","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1138-9416","authenticated-orcid":false,"given":"Jeanne M","family":"Farnan","sequence":"first","affiliation":[{"name":"University of Chicago Pritzker School of Medicine , Chicago, IL,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8327-8002","authenticated-orcid":false,"given":"Sally A","family":"Santen","sequence":"additional","affiliation":[{"name":"University of Cincinnati , Cincinnati, OH,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8583-4335","authenticated-orcid":false,"given":"Yoon Soo","family":"Park","sequence":"additional","affiliation":[{"name":"University of Illinois College of Medicine , Chicago, IL,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5486-6123","authenticated-orcid":false,"given":"Tonya","family":"Fancher","sequence":"additional","affiliation":[{"name":"University of California Davis School of Medicine , Sacramento, CA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jose A Negrete","family":"Manriquez","sequence":"additional","affiliation":[{"name":"David Geffen School of Medicine at UCLA , Los Angeles, CA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8408-2581","authenticated-orcid":false,"given":"Luan","family":"Lawson","sequence":"additional","affiliation":[{"name":"Virginia Commonwealth University School of Medicine , Richmond, VA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-9006-1857","authenticated-orcid":false,"given":"Sean O","family":"Hogan","sequence":"additional","affiliation":[{"name":"Accreditation Council for Graduate Medical Education , Chicago, IL,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3266-9289","authenticated-orcid":false,"given":"Michael S","family":"Ryan","sequence":"additional","affiliation":[{"name":"University of Virginia School of Medicine , Charlottesville, VA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"for the Medical School Outcomes Milestone Study Group","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katherine","family":"Berg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anita","family":"Wilson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tonya","family":"Fancher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marjorie","family":"Weseterfelt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeanne","family":"Farnan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laurah","family":"Lukin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoon Soo","family":"Park","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mike","family":"Ryan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Martindale","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sally","family":"Santen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sean","family":"Hogan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,4,30]]},"reference":[{"key":"2026060310032296700_wvag131-B1","doi-asserted-by":"publisher","first-page":"1057","DOI":"10.1001\/jama.1935.02760130007002","article-title":"The history of medical licensure","volume":"104","author":"Sigerist","year":"1935","journal-title":"JAMA"},{"key":"2026060310032296700_wvag131-B2","doi-asserted-by":"publisher","first-page":"1051","DOI":"10.1056\/NEJMsr1200117","article-title":"The next GME accreditation system\u2013rationale and benefits","volume":"366","author":"Nasca","year":"2012","journal-title":"N Engl J Med"},{"key":"2026060310032296700_wvag131-B3","author":"AAMC. AACOM. ACGME Foundational Competencies for Undergraduate Medical Education Initiative","year":"2023"},{"key":"2026060310032296700_wvag131-B4","author":"Holmboe","year":"2020"},{"key":"2026060310032296700_wvag131-B5","author":"Curriculum Reports","year":"2024"},{"key":"2026060310032296700_wvag131-B6","doi-asserted-by":"publisher","first-page":"908","DOI":"10.1007\/s11999-015-4317-7","article-title":"Does residency selection criteria predict performance in orthopaedic surgery residency?","volume":"474","author":"Raman","year":"2016","journal-title":"Clin Orthop Relat Res"},{"key":"2026060310032296700_wvag131-B7","doi-asserted-by":"publisher","first-page":"615","DOI":"10.1186\/s12909-023-04602-9","article-title":"Investigating factors that influence residency program selection among medical students","volume":"23","author":"Alaqeel","year":"2023","journal-title":"BMC Med Educ"},{"key":"2026060310032296700_wvag131-B8","doi-asserted-by":"publisher","first-page":"386","DOI":"10.1177\/00031348241262427","article-title":"Evaluating the factors influencing residency match for surgical specialty applicants and programs: challenges and future directions","volume":"91","author":"Patel","year":"2025","journal-title":"Am Surg"},{"key":"2026060310032296700_wvag131-B9","doi-asserted-by":"publisher","first-page":"652","DOI":"10.4300\/JGME-D-22-00955.1","article-title":"A systematic review of metrics utilized in the selection and prediction of future performance of residents in the United States","volume":"15","author":"Lipman","year":"2023","journal-title":"J Grad Med Educ"},{"key":"2026060310032296700_wvag131-B10","author":"National Resident Matching Program","year":"2020"},{"key":"2026060310032296700_wvag131-B11","year":"2024"},{"key":"2026060310032296700_wvag131-B12","doi-asserted-by":"publisher","first-page":"1742","DOI":"10.1097\/ACM.0000000000004227","article-title":"Association between USMLE step 1 scores and in-training examination performance: a meta-analysis","volume":"96","author":"Panda","year":"2021","journal-title":"Acad Med J Assoc Am Med Coll"},{"key":"2026060310032296700_wvag131-B13","doi-asserted-by":"publisher","first-page":"1388","DOI":"10.1097\/ACM.0000000000003382","article-title":"Correlations between the USMLE step examinations, American College of Physicians in-training examination, and ABIM internal medicine certification examination","volume":"95","author":"McDonald","year":"2020","journal-title":"Acad Med J Assoc Am Med Coll"},{"key":"2026060310032296700_wvag131-B14","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1016\/j.jsurg.2010.06.010","article-title":"Using United States Medical Licensing Examination (USMLE) examination results to predict later in-training examination performance among general surgery residents","volume":"67","author":"Spurlock","year":"2010","journal-title":"J Surg Educ"},{"key":"2026060310032296700_wvag131-B15","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/j.jemermed.2008.04.010","article-title":"Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents?","volume":"38","author":"Thundiyil","year":"2010","journal-title":"J Emerg Med"},{"key":"2026060310032296700_wvag131-B16","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1186\/s12909-023-04530-8","article-title":"USMLE step 1 and step 2 CK as indicators of resident performance","volume":"23","author":"Lombardi","year":"2023","journal-title":"BMC Med Educ"},{"key":"2026060310032296700_wvag131-B17","doi-asserted-by":"publisher","first-page":"1676","DOI":"10.1016\/j.jsurg.2021.01.020","article-title":"Predictive factors of first time pass rate on the American Board of Surgery certification in general surgery exams: a systematic review","volume":"78","author":"Nguyen","year":"2021","journal-title":"J Surg Educ"},{"key":"2026060310032296700_wvag131-B18","doi-asserted-by":"publisher","first-page":"1157","DOI":"10.1097\/ACM.0000000000000310","article-title":"The relationship between licensing examination performance and the outcomes of care by international medical school graduates","volume":"89","author":"Norcini","year":"2014","journal-title":"Acad Med"},{"key":"2026060310032296700_wvag131-B19","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1097\/ACM.0000000000005480","article-title":"The associations between United States medical licensing examination performance and outcomes of patient care","volume":"99","author":"Norcini","year":"2024","journal-title":"Acad Med"},{"key":"2026060310032296700_wvag131-B20","doi-asserted-by":"publisher","first-page":"1780","DOI":"10.1097\/ACM.0000000000001747","article-title":"Exploring the relationships between USMLE performance and disciplinary action in practice: a validity study of score inferences from a licensure examination","volume":"92","author":"Cuddy","year":"2017","journal-title":"Acad Med J Assoc Am Med Coll"},{"key":"2026060310032296700_wvag131-B21","doi-asserted-by":"publisher","first-page":"1504","DOI":"10.1097\/ACM.0000000000004775","article-title":"An examination of the associations among USMLE step 3 scores and the likelihood of disciplinary action in practice","volume":"97","author":"Cuddy","year":"2022","journal-title":"Acad Med J Assoc Am Med Coll"},{"key":"2026060310032296700_wvag131-B22","doi-asserted-by":"publisher","first-page":"1427","DOI":"10.1097\/ACM.0000000000004910","article-title":"Drawing a line between 2 points: challenges and opportunities in linking assessments with key educational outcomes","volume":"97","author":"Park","year":"2022","journal-title":"Acad Med"},{"key":"2026060310032296700_wvag131-B23","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1080\/10401334.2025.2477094","article-title":"Gaslighting the racially and ethnically minoritized medical student: how US medical school assessment practices perpetuate systemic inequities","volume":"38","author":"Ryan","year":"2026","journal-title":"Teach Learn Med"},{"key":"2026060310032296700_wvag131-B24","doi-asserted-by":"publisher","first-page":"696","DOI":"10.4300\/JGME-D-20-00013.1","article-title":"A critical disconnect: residency selection factors lack correlation with intern performance","volume":"12","author":"Burkhardt","year":"2020","journal-title":"J Grad Med Educ"},{"key":"2026060310032296700_wvag131-B25","doi-asserted-by":"publisher","first-page":"1324","DOI":"10.1097\/ACM.0000000000004207","article-title":"Exploring the association between USMLE scores and ACGME milestone ratings: a validity study using national data from emergency medicine","volume":"96","author":"Hamstra","year":"2021","journal-title":"Acad Med"},{"key":"2026060310032296700_wvag131-B26","doi-asserted-by":"publisher","first-page":"213","DOI":"10.4300\/JGME-D-20-00603.1","article-title":"Association between internal medicine residency applicant characteristics and performance on ACGME milestones during intern year","volume":"13","author":"Golden","year":"2021","journal-title":"J Grad Med Educ"},{"key":"2026060310032296700_wvag131-B27","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1016\/j.acap.2020.03.010","article-title":"Application factors associated with clinical performance during pediatric internship","volume":"20","author":"Gross","year":"2020","journal-title":"Acad Pediatr"},{"key":"2026060310032296700_wvag131-B28","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1097\/ACM.0000000000005891","article-title":"Holistic review in applicant selection: a scoping review","volume":"100","author":"Gottlieb","year":"2025","journal-title":"Acad Med J Assoc Am Med Coll"},{"key":"2026060310032296700_wvag131-B29","doi-asserted-by":"publisher","first-page":"362","DOI":"10.1097\/ACM.0b013e3181970c6b","article-title":"Selection criteria for residency: results of a national program directors survey","volume":"84","author":"Green","year":"2009","journal-title":"Acad Med"},{"key":"2026060310032296700_wvag131-B30","doi-asserted-by":"publisher","first-page":"e11550","DOI":"10.7759\/cureus.11550","article-title":"Red flags, geography, exam scores, and other factors used by program directors in determining which applicants are offered an interview for anesthesiology residency","volume":"12","author":"Vinagre","year":"2020","journal-title":"Cureus"},{"key":"2026060310032296700_wvag131-B31","author":"Facebook, Twitter, LinkedIn","year":"2024"},{"key":"2026060310032296700_wvag131-B32","doi-asserted-by":"publisher","first-page":"102840","DOI":"10.1016\/j.acap.2025.102840","article-title":"The landscape of pediatric residency coaching programs in the United States","volume":"25","author":"Thomas","year":"2025","journal-title":"Acad Pediatr"},{"key":"2026060310032296700_wvag131-B33","doi-asserted-by":"publisher","first-page":"430","DOI":"10.1097\/ACM.0000000000001825","article-title":"A novel pediatric residency coaching program: outcomes after one year","volume":"93","author":"Rassbach","year":"2018","journal-title":"Acad Med J Assoc Am Med Coll"},{"key":"2026060310032296700_wvag131-B34","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1007\/s40037-020-00612-1","article-title":"You can have both: coaching to promote clinical competency and professional identity formation","volume":"10","author":"Parsons","year":"2021","journal-title":"Perspect Med Educ"},{"key":"2026060310032296700_wvag131-B35","author":"Academic coaching in medical education","year":"2024"},{"key":"2026060310032296700_wvag131-B36","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1097\/ACM.0000000000000500","article-title":"The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE step 1 examination","volume":"90","author":"Kay","year":"2015","journal-title":"Acad Med"},{"key":"2026060310032296700_wvag131-B37","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1186\/s12909-020-1974-3","article-title":"Do USMLE steps, and ITE score predict the American Board of Internal Medicine certifying exam results?","volume":"20","author":"Rayamajhi","year":"2020","journal-title":"BMC Med Educ"},{"key":"2026060310032296700_wvag131-B38","doi-asserted-by":"publisher","first-page":"918","DOI":"10.1001\/jamasurg.2022.3340","article-title":"Characteristics associated with outstanding general surgery residency graduate performance, as rated by surgical educators","volume":"157","author":"Purdy","year":"2022","journal-title":"JAMA Surg"},{"key":"2026060310032296700_wvag131-B39","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1016\/j.urology.2023.11.036","article-title":"Effect of United States medical licensing examination score cutoffs on recruitment of underrepresented applicants in the urology match","volume":"187","author":"Kaldany","year":"2024","journal-title":"Urology"},{"key":"2026060310032296700_wvag131-B40","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.jamcollsurg.2020.02.042","article-title":"Underrepresented minorities in general surgery residency: analysis of interviewed applicants, residents, and core teaching faculty","volume":"231","author":"Jarman","year":"2020","journal-title":"J Am Coll Surg"},{"key":"2026060310032296700_wvag131-B41","author":"The American Board of Pediatrics"},{"key":"2026060310032296700_wvag131-B42"},{"key":"2026060310032296700_wvag131-B43","doi-asserted-by":"publisher","first-page":"201","DOI":"10.4300\/JGME-D-21-00546.1","article-title":"ACGME milestones in the real world: a qualitative study exploring response process evidence","volume":"14","author":"Maranich","year":"2022","journal-title":"J Grad Med Educ"},{"key":"2026060310032296700_wvag131-B44","author":"Common Questions. USMLE"},{"key":"2026060310032296700_wvag131-B45","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1016\/j.jsurg.2014.06.003","article-title":"The predictive value of application variables on the global rating of applicants to a general surgery residency program","volume":"72","author":"Sharp","year":"2015","journal-title":"J Surg Educ"},{"key":"2026060310032296700_wvag131-B46","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/j.urology.2020.08.033","article-title":"USMLE step 1 scoring changes and the urology residency application process: program directors\u2019 perspectives","volume":"145","author":"Chisholm","year":"2020","journal-title":"Urology"},{"key":"2026060310032296700_wvag131-B47","doi-asserted-by":"publisher","first-page":"15","DOI":"10.5811\/westjem.2021.3.50897","article-title":"Emergency medicine program directors\u2019 perspectives on changes to step 1 scoring: does it help or hurt applicants?","volume":"23","author":"Glassman","year":"2021","journal-title":"West J Emerg Med"},{"key":"2026060310032296700_wvag131-B48","doi-asserted-by":"publisher","first-page":"e257","DOI":"10.3928\/01477447-20220425-03","article-title":"Perspectives of orthopedic surgery program directors on the USMLE step 1 scoring change","volume":"45","author":"White-Dzuro","year":"2022","journal-title":"Orthopedics"},{"key":"2026060310032296700_wvag131-B49","doi-asserted-by":"publisher","first-page":"1034","DOI":"10.1001\/jama.2024.28397","article-title":"Diminishing objectivity in the residency application process","volume":"333","author":"Agolia","year":"2025","journal-title":"JAMA. Published"}],"container-title":["Academic Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/academicmedicine\/advance-article-pdf\/doi\/10.1093\/acamed\/wvag131\/68198742\/wvag131.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/academicmedicine\/advance-article-pdf\/doi\/10.1093\/acamed\/wvag131\/68198742\/wvag131.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T14:03:29Z","timestamp":1780495409000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/academicmedicine\/advance-article\/doi\/10.1093\/acamed\/wvag131\/8666122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,30]]},"references-count":49,"URL":"https:\/\/doi.org\/10.1093\/acamed\/wvag131","relation":{},"ISSN":["1938-808X"],"issn-type":[{"value":"1938-808X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,30]]},"article-number":"wvag131"}}