{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T13:01:26Z","timestamp":1772715686231,"version":"3.50.1"},"reference-count":64,"publisher":"Journal of Graduate Medical Education","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:p>\n                    <jats:bold>Background<\/jats:bold>\n                    The Accreditation Council for Graduate Medical Education (ACGME) Milestones use has been formative and low-stakes to date, and transitioning to higher-stakes applications in a truly competency-based medical education (CBME) system requires extensive validity evidence. Surgical specialties, with their unique demands for procedural skills and operative experience, represent a critical context for evaluating the validity of Milestones.\n                  <\/jats:p>\n                  <jats:p>\n                    <jats:bold>Objective<\/jats:bold>\n                    To synthesize studies reporting validity evidence for the ACGME Milestones in surgical specialties.\n                  <\/jats:p>\n                  <jats:p>\n                    <jats:bold>Methods<\/jats:bold>\n                    This systematic review was conducted based on Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. A systematic literature search was conducted across 8 databases and references to identify studies that reported validity evidence for Milestones in surgical specialties. Literature was reviewed for inclusion using Covidence and coded based on Messick\u2019s framework. The quality of the studies was evaluated using the Medical Education Research Study Quality Instrument.\n                  <\/jats:p>\n                  <jats:p>\n                    <jats:bold>Results<\/jats:bold>\n                    A total of 114 studies were included from 2013 to 2023. The primary source of validity evidence (n=45, 39.5%) was relations to other variables (knowledge and skills, learner characteristics, patient\/health care, social-emotional variables), followed by response processes (n=38, 33.3%: interrater reliability, rating processes, structure of Clinical Competency Committee, rater training, longitudinal reliability, straightlining) and consequences (n=29, 25.4%: value and utility, intended use, anticipated impact). Only 12 studies (10.5%) reported internal structure evidence.\n                  <\/jats:p>\n                  <jats:p>\n                    <jats:bold>Conclusions<\/jats:bold>\n                    This study provides insights into understanding what constitutes validity evidence within the context of ACGME Milestones in surgical specialties. This review highlights areas where further research is needed to support the moderate to high-stakes use of Milestones in a CBME system.\n                  <\/jats:p>","DOI":"10.4300\/jgme-d-24-00524.1","type":"journal-article","created":{"date-parts":[[2025,4,15]],"date-time":"2025-04-15T11:37:56Z","timestamp":1744717076000},"page":"162-171","source":"Crossref","is-referenced-by-count":2,"title":["Construct Validity Evidence for ACGME Milestones in Surgical Specialties: A Systematic Review"],"prefix":"10.4300","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9320-8921","authenticated-orcid":true,"given":"Ting","family":"Sun","sequence":"first","affiliation":[{"name":"Ting Sun, PhD, is an Assistant Professor, Office of Surgical Education, Department of Surgery, University of Utah School of Medicine, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0680-366X","authenticated-orcid":true,"given":"Stanley J.","family":"Hamstra","sequence":"additional","affiliation":[{"name":"Stanley J. Hamstra, PhD, is a Professor, Department of Surgery, University of Toronto, Toronto, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7039-4717","authenticated-orcid":true,"given":"Kenji","family":"Yamazaki","sequence":"additional","affiliation":[{"name":"Kenji Yamazaki, PhD, is Senior Outcomes Research Scientist, Accreditation Council for Graduate Medical Education, Chicago, Illinois, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5914-9499","authenticated-orcid":true,"given":"Katherine Jiawen","family":"Ren","sequence":"additional","affiliation":[{"name":"Katherine Jiawen Ren, MA, is a Doctoral Student, Department of Educational Leadership, Cato College of Education, University of North Carolina at Charlotte, Charlotte, North Carolina, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4198-8142","authenticated-orcid":true,"given":"Brigitte","family":"Smith","sequence":"additional","affiliation":[{"name":"Brigitte Smith, MD, MHPE, is an Associate Professor, Division of Vascular Surgery, Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"2856","reference":[{"issue":"(8)","key":"i1949-8357-17-2-162-b01","series-title":"Med Teach","first-page":"676","article-title":"The role of assessment in competency-based medical education","volume":"32","author":"Holmboe","year":"2010","unstructured":"Holmboe\n              ES\n            , SherbinoJ, LongDM, SwingSR, FrankJR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682. doi:10.3109\/0142159x.2010.500704"},{"issue":"(11)","key":"i1949-8357-17-2-162-b02","series-title":"N Engl J Med","first-page":"1051","article-title":"The next GME accreditation system\u2014rationale and benefits","volume":"366","author":"Nasca","year":"2012","unstructured":"Nasca\n              T\n            , PhilibertI, BrighamT, FlynnTC. The next GME accreditation system\u2014rationale and benefits. N Engl J Med. 2012;366(11):1051-1056. doi:10.1056\/NEJMsr1200117"},{"key":"i1949-8357-17-2-162-b03","series-title":"The Milestones Guidebook. Accreditation Council for Graduate Medical Education","first-page":"6","author":"Edgar","year":"2020","unstructured":"Edgar\n              L\n            , McLeanS, HoganSO, HamstraS, HolmboeES. The Milestones Guidebook. Accreditation Council for Graduate Medical Education; 2020:6-19. Accessed February 11, 2025. https:\/\/www.acgme.org\/globalassets\/milestonesguidebook.pdf."},{"issue":"(7)","key":"i1949-8357-17-2-162-b04","series-title":"Acad Med","first-page":"813","article-title":"Association of surgical resident competency ratings with patient outcomes","volume":"98","author":"Kendrick","year":"2023","unstructured":"Kendrick\n              DE\n            , ThelenAE, ChenX, et al. Association of surgical resident competency ratings with patient outcomes. Acad Med. 2023;98(7):813-820. doi:10.1097\/acm.0000000000005157"},{"issue":"(9)","key":"i1949-8357-17-2-162-b05","series-title":"Med Teach","first-page":"904","article-title":"Competencies, milestones, and EPAs\u2014are those who ignore the past condemned to repeat it?","volume":"38","author":"Klamen","year":"2016","unstructured":"Klamen\n              DL\n            , WilliamsRG, RobertsN, CiancioloAT. Competencies, milestones, and EPAs\u2014are those who ignore the past condemned to repeat it?Med Teach. 2016;38(9):904-910. doi:10.3109\/0142159x.2015.1132831"},{"issue":"(4)","key":"i1949-8357-17-2-162-b06","series-title":"Acad Med","first-page":"404","article-title":"Placing constraints on the use of the ACGME Milestones: a commentary on the limitations of global performance ratings","volume":"90","author":"Williams","year":"2015","unstructured":"Williams\n              RG\n            , DunningtonGL, MellingerJD, KlamenDL. Placing constraints on the use of the ACGME Milestones: a commentary on the limitations of global performance ratings. Acad Med. 2015;90(4):404-407. doi:10.1097\/acm.0000000000000507"},{"key":"i1949-8357-17-2-162-b07","series-title":"Standards for Educational and Psychological Testing","article-title":"American Psychological Association, National Council on Measurement in Education","year":"2014","unstructured":"American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. American Educational Research Association; 2014."},{"issue":"(9)","key":"i1949-8357-17-2-162-b08","series-title":"Am Psychol","first-page":"741","article-title":"Validity of psychological assessment: validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning","volume":"50","author":"Messick","year":"1995","unstructured":"Messick\n              S\n            \n          . Validity of psychological assessment: validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning. Am Psychol. 1995;50(9):741. doi:10.1037\/\/0003-066x.50.9.741"},{"key":"i1949-8357-17-2-162-b09","series-title":"BMJ","first-page":"n160","article-title":"PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews","volume":"372","author":"Page","year":"2021","unstructured":"Page\n              MJ\n            , MoherD, BossuytPM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. doi: 10.1136\/bmj.n160"},{"key":"i1949-8357-17-2-162-b10","article-title":"ACGME Data Resource Book: Academic Year 2022-2023","unstructured":"Accreditation Council for Graduate Medical Education. ACGME Data Resource Book: Academic Year 2022-2023. Accessed January 29, 2025. https:\/\/www.acgme.org\/globalassets\/pfassets\/publicationsbooks\/2022-2023_acgme_databook_document.pdf"},{"key":"i1949-8357-17-2-162-b11","series-title":"Milestones Bibliography","article-title":"Accreditation Council for Graduate Medical Education","unstructured":"Accreditation Council for Graduate Medical Education. Milestones Bibliography. Published 2021 December. Accessed January 29, 2025. https:\/\/www.acgme.org\/globalassets\/pdfs\/milestones\/milestones-bibliography\u2013-december-2021-final.pdf"},{"issue":"(2)","key":"i1949-8357-17-2-162-b12","series-title":"Adv Health Sci Educ Theory Pract","first-page":"233","article-title":"What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment","volume":"19","author":"Cook","year":"2014","unstructured":"Cook\n              DA\n            , ZendejasB, HamstraSJ, HatalaR, BrydgesR. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract. 2014;19(2):233-250. doi:10.1007\/s10459-013-9458-4"},{"issue":"(suppl 2)","key":"i1949-8357-17-2-162-b13","series-title":"J Grad Med Educ","first-page":"75","article-title":"A validity framework for effective analysis and interpretation of milestones data","volume":"13","author":"Hamstra","year":"2021","unstructured":"Hamstra\n              SJ\n            , YamazakiK. A validity framework for effective analysis and interpretation of milestones data. J Grad Med Educ. 2021;13(suppl 2):75-80. doi:10.4300\/jgme-d-20-01039.1"},{"issue":"(1)","key":"i1949-8357-17-2-162-b14","series-title":"Rev Educ Res","first-page":"24","article-title":"Methodological guidance paper: high-quality meta-analysis in a systematic review","volume":"90","author":"Pigott","year":"2020","unstructured":"Pigott\n              TD\n            , PolaninJR. Methodological guidance paper: high-quality meta-analysis in a systematic review. Rev Educ Res. 2020;90(1):24-46. doi:10.3102\/0034654319877153"},{"issue":"(8)","key":"i1949-8357-17-2-162-b15","series-title":"Acad Med","first-page":"1067","article-title":"Appraising the quality of medical education research methods: the medical education research study quality instrument and the Newcastle\u2013 Ottawa scale-education","volume":"90","author":"Cook","year":"2015","unstructured":"Cook\n              DA\n            , ReedDA. Appraising the quality of medical education research methods: the medical education research study quality instrument and the Newcastle\u2013 Ottawa scale-education. Acad Med. 2015;90(8): 1067-1076. doi:10.1097\/acm.0000000000000786"},{"issue":"(6)","key":"i1949-8357-17-2-162-b16","series-title":"J Surg Educ","first-page":"e150","article-title":"Using the ACGME Milestones for resident self-evaluation and faculty engagement","volume":"73","author":"Meier","year":"2016","unstructured":"Meier\n              AH\n            , GruessnerA, CooneyRN. Using the ACGME Milestones for resident self-evaluation and faculty engagement. J Surg Educ. 2016;73(6): e150-e157. doi:10.1016\/j.jsurg.2016.09.001"},{"issue":"(3)","key":"i1949-8357-17-2-162-b17","series-title":"Am J Surg","first-page":"505","article-title":"Is there a gender bias in milestones evaluations in general surgery residency training?","volume":"221","author":"Kwasny","year":"2021","unstructured":"Kwasny\n              L\n            , ShebrainS, MuneneG, SawyerR. Is there a gender bias in milestones evaluations in general surgery residency training?Am J Surg. 2021;221(3):505-508. doi:10.1016\/j.amjsurg.2020.12.020"},{"issue":"(2)","key":"i1949-8357-17-2-162-b18","series-title":"Am J Surg","first-page":"260","article-title":"A taxonomy of perioperative surgical learning: trending resident skill acquisition","volume":"213","author":"Hardaway","year":"2017","unstructured":"Hardaway\n              JC\n            , BassonMD, AliM, et al. A taxonomy of perioperative surgical learning: trending resident skill acquisition. Am J Surg. 2017;213(2):260-267. doi:10.1016\/j.amjsurg.2016.09.045"},{"issue":"(4)","key":"i1949-8357-17-2-162-b19","series-title":"J Surg Educ","first-page":"1076","article-title":"Ability of ophthalmology residents to self-assess their performance through established milestones","volume":"76","author":"Srikumaran","year":"2019","unstructured":"Srikumaran\n              D\n            , TianJ, RamuluP, et al. Ability of ophthalmology residents to self-assess their performance through established milestones. J Surg Educ. 2019;76(4): 1076-1087. doi:10.1016\/j.jsurg.2018.12.004"},{"issue":"(10)","key":"i1949-8357-17-2-162-b20","series-title":"Acad Med","first-page":"1522","article-title":"A national study of longitudinal consistency in ACGME Milestone ratings by clinical competency committees: exploring an aspect of validity in the assessment of residents\u2019 competence","volume":"94","author":"Hamstra","year":"2019","unstructured":"Hamstra\n              SJ\n            , YamazakiK, BartonMA, SantenSA, BeesonMS, HolmboeES. A national study of longitudinal consistency in ACGME Milestone ratings by clinical competency committees: exploring an aspect of validity in the assessment of residents\u2019 competence. Acad Med. 2019;94(10):1522-1531. doi:10.1097\/acm.0000000000002820"},{"issue":"(6)","key":"i1949-8357-17-2-162-b21","series-title":"J Surg Educ","first-page":"1334","article-title":"Implementation and initial construct validity evidence of a tool, myTIPreport, for interactive workplace feedback on ACGME Milestones","volume":"77","author":"Husk","year":"2020","unstructured":"Husk\n              KE\n            , LearmanLA, FieldC, ConnollyA. Implementation and initial construct validity evidence of a tool, myTIPreport, for interactive workplace feedback on ACGME Milestones. J Surg Educ. 2020; 77(6):1334-1340. doi:10.1016\/j.jsurg.2020.05.002"},{"key":"i1949-8357-17-2-162-b22","series-title":"Int J Med Educ","first-page":"11","article-title":"A pilot study of orthopaedic resident self-assessment using a milestones\u2019 survey just prior to milestones implementation","volume":"7","author":"Bradley","year":"2016","unstructured":"Bradley\n              KE\n            , AndolsekKM. A pilot study of orthopaedic resident self-assessment using a milestones\u2019 survey just prior to milestones implementation. Int J Med Educ. 2016;7:11-18. doi:10.5116\/ijme.5682.6dfd"},{"issue":"(12)","key":"i1949-8357-17-2-162-b23","series-title":"J Pediatr Surg","first-page":"845","article-title":"Pediatric surgery milestones 2.0: a primer","volume":"57","author":"Alaish","year":"2022","unstructured":"Alaish\n              SM\n            , ArcaMJ, BucherBT, et al. Pediatric surgery milestones 2.0: a primer. J Pediatr Surg. 2022;57(12): 845-851. doi:10.1016\/j.jpedsurg.2022.04.021"},{"key":"i1949-8357-17-2-162-b24","series-title":"World Neurosurg","first-page":"499","article-title":"Does one size fit all? Examining the application of neurosurgery residency milestones developed in the United States to a Taiwanese culture","volume":"104","author":"Lee","year":"2017","unstructured":"Lee\n              CY\n            , LaiHY, LeeCH, LeeST. Does one size fit all? Examining the application of neurosurgery residency milestones developed in the United States to a Taiwanese culture. World Neurosurg. 2017;104: 499-508. doi:10.1016\/j.wneu.2017.04.129"},{"issue":"(4)","key":"i1949-8357-17-2-162-b25","series-title":"Acad Med","first-page":"569","article-title":"An empirical investigation into milestones factor structure using national data derived from clinical competency committees","volume":"97","author":"Yamazaki","year":"2022","unstructured":"Yamazaki\n              K\n            , HolmboeES, HamstraSJ. An empirical investigation into milestones factor structure using national data derived from clinical competency committees. Acad Med. 2022;97(4):569-576. doi:10.1097\/acm.0000000000004218"},{"issue":"(1)","key":"i1949-8357-17-2-162-b26","series-title":"Ann Surg","first-page":"180","article-title":"The association of ACGME Milestones with performance on American Board of Surgery assessments: a national investigation of surgical trainees","volume":"279","author":"Weaver","year":"2024","unstructured":"Weaver\n              ML\n            , CarterT, YamazakiK, et al. The association of ACGME Milestones with performance on American Board of Surgery assessments: a national investigation of surgical trainees. Ann Surg. 2024; 279(1):180-186. doi:10.1097\/sla.0000000000005998"},{"key":"i1949-8357-17-2-162-b27","series-title":"J Surg Res","first-page":"34","article-title":"Application factors may not be predictors of success among general surgery residents as measured by ACGME Milestones","volume":"253","author":"Hayek","year":"2020","unstructured":"Hayek\n              SA\n            , WickizerAP, LaneSM, et al. Application factors may not be predictors of success among general surgery residents as measured by ACGME Milestones. J Surg Res. 2020;253:34-40. doi:10.1016\/j.jss.2020. 03.029"},{"issue":"(3)","key":"i1949-8357-17-2-162-b28","series-title":"Surg","first-page":"488","article-title":"Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents","volume":"163","author":"Stride","year":"2018","unstructured":"Stride\n              HP\n            , GeorgeBC, WilliamsRG, et al. Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents. Surg. 2018; 163(3):488-494. doi:10.1016\/j.surg.2017.10.011"},{"issue":"(1)","key":"i1949-8357-17-2-162-b29","series-title":"J Surg Educ","first-page":"147","article-title":"The effect and use of milestones in the assessment of neurological surgery residents and residency programs","volume":"75","author":"Conforti","year":"2018","unstructured":"Conforti\n              LN\n            , YaghmourNA, HamstraSJ, et al. The effect and use of milestones in the assessment of neurological surgery residents and residency programs. J Surg Educ. 2018;75(1):147-155. doi:10.1016\/j.jsurg.2017.06.001"},{"issue":"(11)","key":"i1949-8357-17-2-162-b30","series-title":"Med Teach","first-page":"1228","article-title":"A multispecialty ethnographic study of clinical competency committees (CCCs)","volume":"44","author":"Ekpenyong","year":"2022","unstructured":"Ekpenyong\n              A\n            , EdgarL, WilkersonL, HolmboeES. A multispecialty ethnographic study of clinical competency committees (CCCs). Med Teach. 2022;44(11): 1228-1236. doi:10.1080\/0142159x.2022.2072281"},{"issue":"(4)","key":"i1949-8357-17-2-162-b31","series-title":"J Grad Med Educ","first-page":"439","article-title":"Rising to the challenge: residency programs\u2019 experience with implementing milestones-based assessment","volume":"11","author":"Dzara","year":"2019","unstructured":"Dzara\n              K\n            , HuthK, KesselheimJC, SchumacherDJ. Rising to the challenge: residency programs\u2019 experience with implementing milestones-based assessment. J Grad Med Educ. 2019;11(4):439-446. doi:10.4300\/jgme-d-18-00717.1"},{"issue":"(5)","key":"i1949-8357-17-2-162-b32","series-title":"J Surg Educ","first-page":"1259","article-title":"Qualitative study of orthopedic surgery milestones 1.0: burdens and benefits","volume":"79","author":"Balach","year":"2022","unstructured":"Balach\n              T\n            , ConfortiLN, SanghaS, EdgarL, AmesE. Qualitative study of orthopedic surgery milestones 1.0: burdens and benefits. J Surg Educ. 2022;79(5):1259-1269. doi:10.1016\/j.jsurg.2022.05.002"},{"issue":"(8)","key":"i1949-8357-17-2-162-b01","series-title":"Med Teach","first-page":"676","article-title":"The role of assessment in competency-based medical education","volume":"32","author":"Holmboe","year":"2010","unstructured":"Holmboe\n              ES\n            , SherbinoJ, LongDM, SwingSR, FrankJR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682. doi:10.3109\/0142159x.2010.500704"},{"issue":"(11)","key":"i1949-8357-17-2-162-b02","series-title":"N Engl J Med","first-page":"1051","article-title":"The next GME accreditation system\u2014rationale and benefits","volume":"366","author":"Nasca","year":"2012","unstructured":"Nasca\n              T\n            , PhilibertI, BrighamT, FlynnTC. The next GME accreditation system\u2014rationale and benefits. N Engl J Med. 2012;366(11):1051-1056. doi:10.1056\/NEJMsr1200117"},{"key":"i1949-8357-17-2-162-b03","series-title":"The Milestones Guidebook. Accreditation Council for Graduate Medical Education","first-page":"6","author":"Edgar","year":"2020","unstructured":"Edgar\n              L\n            , McLeanS, HoganSO, HamstraS, HolmboeES. The Milestones Guidebook. Accreditation Council for Graduate Medical Education; 2020:6-19. Accessed February 11, 2025. https:\/\/www.acgme.org\/globalassets\/milestonesguidebook.pdf."},{"issue":"(7)","key":"i1949-8357-17-2-162-b04","series-title":"Acad Med","first-page":"813","article-title":"Association of surgical resident competency ratings with patient outcomes","volume":"98","author":"Kendrick","year":"2023","unstructured":"Kendrick\n              DE\n            , ThelenAE, ChenX, et al. Association of surgical resident competency ratings with patient outcomes. Acad Med. 2023;98(7):813-820. doi:10.1097\/acm.0000000000005157"},{"issue":"(9)","key":"i1949-8357-17-2-162-b05","series-title":"Med Teach","first-page":"904","article-title":"Competencies, milestones, and EPAs\u2014are those who ignore the past condemned to repeat it?","volume":"38","author":"Klamen","year":"2016","unstructured":"Klamen\n              DL\n            , WilliamsRG, RobertsN, CiancioloAT. Competencies, milestones, and EPAs\u2014are those who ignore the past condemned to repeat it?Med Teach. 2016;38(9):904-910. doi:10.3109\/0142159x.2015.1132831"},{"issue":"(4)","key":"i1949-8357-17-2-162-b06","series-title":"Acad Med","first-page":"404","article-title":"Placing constraints on the use of the ACGME Milestones: a commentary on the limitations of global performance ratings","volume":"90","author":"Williams","year":"2015","unstructured":"Williams\n              RG\n            , DunningtonGL, MellingerJD, KlamenDL. Placing constraints on the use of the ACGME Milestones: a commentary on the limitations of global performance ratings. Acad Med. 2015;90(4):404-407. doi:10.1097\/acm.0000000000000507"},{"key":"i1949-8357-17-2-162-b07","series-title":"Standards for Educational and Psychological Testing","article-title":"American Psychological Association, National Council on Measurement in Education","year":"2014","unstructured":"American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. American Educational Research Association; 2014."},{"issue":"(9)","key":"i1949-8357-17-2-162-b08","series-title":"Am Psychol","first-page":"741","article-title":"Validity of psychological assessment: validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning","volume":"50","author":"Messick","year":"1995","unstructured":"Messick\n              S\n            \n          . Validity of psychological assessment: validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning. Am Psychol. 1995;50(9):741. doi:10.1037\/\/0003-066x.50.9.741"},{"key":"i1949-8357-17-2-162-b09","series-title":"BMJ","first-page":"n160","article-title":"PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews","volume":"372","author":"Page","year":"2021","unstructured":"Page\n              MJ\n            , MoherD, BossuytPM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. doi: 10.1136\/bmj.n160"},{"key":"i1949-8357-17-2-162-b10","article-title":"ACGME Data Resource Book: Academic Year 2022-2023","unstructured":"Accreditation Council for Graduate Medical Education. ACGME Data Resource Book: Academic Year 2022-2023. Accessed January 29, 2025. https:\/\/www.acgme.org\/globalassets\/pfassets\/publicationsbooks\/2022-2023_acgme_databook_document.pdf"},{"key":"i1949-8357-17-2-162-b11","series-title":"Milestones Bibliography","article-title":"Accreditation Council for Graduate Medical Education","unstructured":"Accreditation Council for Graduate Medical Education. Milestones Bibliography. Published 2021 December. Accessed January 29, 2025. https:\/\/www.acgme.org\/globalassets\/pdfs\/milestones\/milestones-bibliography\u2013-december-2021-final.pdf"},{"issue":"(2)","key":"i1949-8357-17-2-162-b12","series-title":"Adv Health Sci Educ Theory Pract","first-page":"233","article-title":"What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment","volume":"19","author":"Cook","year":"2014","unstructured":"Cook\n              DA\n            , ZendejasB, HamstraSJ, HatalaR, BrydgesR. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract. 2014;19(2):233-250. doi:10.1007\/s10459-013-9458-4"},{"issue":"(suppl 2)","key":"i1949-8357-17-2-162-b13","series-title":"J Grad Med Educ","first-page":"75","article-title":"A validity framework for effective analysis and interpretation of milestones data","volume":"13","author":"Hamstra","year":"2021","unstructured":"Hamstra\n              SJ\n            , YamazakiK. A validity framework for effective analysis and interpretation of milestones data. J Grad Med Educ. 2021;13(suppl 2):75-80. doi:10.4300\/jgme-d-20-01039.1"},{"issue":"(1)","key":"i1949-8357-17-2-162-b14","series-title":"Rev Educ Res","first-page":"24","article-title":"Methodological guidance paper: high-quality meta-analysis in a systematic review","volume":"90","author":"Pigott","year":"2020","unstructured":"Pigott\n              TD\n            , PolaninJR. Methodological guidance paper: high-quality meta-analysis in a systematic review. Rev Educ Res. 2020;90(1):24-46. doi:10.3102\/0034654319877153"},{"issue":"(8)","key":"i1949-8357-17-2-162-b15","series-title":"Acad Med","first-page":"1067","article-title":"Appraising the quality of medical education research methods: the medical education research study quality instrument and the Newcastle\u2013 Ottawa scale-education","volume":"90","author":"Cook","year":"2015","unstructured":"Cook\n              DA\n            , ReedDA. Appraising the quality of medical education research methods: the medical education research study quality instrument and the Newcastle\u2013 Ottawa scale-education. Acad Med. 2015;90(8): 1067-1076. doi:10.1097\/acm.0000000000000786"},{"issue":"(6)","key":"i1949-8357-17-2-162-b16","series-title":"J Surg Educ","first-page":"e150","article-title":"Using the ACGME Milestones for resident self-evaluation and faculty engagement","volume":"73","author":"Meier","year":"2016","unstructured":"Meier\n              AH\n            , GruessnerA, CooneyRN. Using the ACGME Milestones for resident self-evaluation and faculty engagement. J Surg Educ. 2016;73(6): e150-e157. doi:10.1016\/j.jsurg.2016.09.001"},{"issue":"(3)","key":"i1949-8357-17-2-162-b17","series-title":"Am J Surg","first-page":"505","article-title":"Is there a gender bias in milestones evaluations in general surgery residency training?","volume":"221","author":"Kwasny","year":"2021","unstructured":"Kwasny\n              L\n            , ShebrainS, MuneneG, SawyerR. Is there a gender bias in milestones evaluations in general surgery residency training?Am J Surg. 2021;221(3):505-508. doi:10.1016\/j.amjsurg.2020.12.020"},{"issue":"(2)","key":"i1949-8357-17-2-162-b18","series-title":"Am J Surg","first-page":"260","article-title":"A taxonomy of perioperative surgical learning: trending resident skill acquisition","volume":"213","author":"Hardaway","year":"2017","unstructured":"Hardaway\n              JC\n            , BassonMD, AliM, et al. A taxonomy of perioperative surgical learning: trending resident skill acquisition. Am J Surg. 2017;213(2):260-267. doi:10.1016\/j.amjsurg.2016.09.045"},{"issue":"(4)","key":"i1949-8357-17-2-162-b19","series-title":"J Surg Educ","first-page":"1076","article-title":"Ability of ophthalmology residents to self-assess their performance through established milestones","volume":"76","author":"Srikumaran","year":"2019","unstructured":"Srikumaran\n              D\n            , TianJ, RamuluP, et al. Ability of ophthalmology residents to self-assess their performance through established milestones. J Surg Educ. 2019;76(4): 1076-1087. doi:10.1016\/j.jsurg.2018.12.004"},{"issue":"(10)","key":"i1949-8357-17-2-162-b20","series-title":"Acad Med","first-page":"1522","article-title":"A national study of longitudinal consistency in ACGME Milestone ratings by clinical competency committees: exploring an aspect of validity in the assessment of residents\u2019 competence","volume":"94","author":"Hamstra","year":"2019","unstructured":"Hamstra\n              SJ\n            , YamazakiK, BartonMA, SantenSA, BeesonMS, HolmboeES. A national study of longitudinal consistency in ACGME Milestone ratings by clinical competency committees: exploring an aspect of validity in the assessment of residents\u2019 competence. Acad Med. 2019;94(10):1522-1531. doi:10.1097\/acm.0000000000002820"},{"issue":"(6)","key":"i1949-8357-17-2-162-b21","series-title":"J Surg Educ","first-page":"1334","article-title":"Implementation and initial construct validity evidence of a tool, myTIPreport, for interactive workplace feedback on ACGME Milestones","volume":"77","author":"Husk","year":"2020","unstructured":"Husk\n              KE\n            , LearmanLA, FieldC, ConnollyA. Implementation and initial construct validity evidence of a tool, myTIPreport, for interactive workplace feedback on ACGME Milestones. J Surg Educ. 2020; 77(6):1334-1340. doi:10.1016\/j.jsurg.2020.05.002"},{"key":"i1949-8357-17-2-162-b22","series-title":"Int J Med Educ","first-page":"11","article-title":"A pilot study of orthopaedic resident self-assessment using a milestones\u2019 survey just prior to milestones implementation","volume":"7","author":"Bradley","year":"2016","unstructured":"Bradley\n              KE\n            , AndolsekKM. A pilot study of orthopaedic resident self-assessment using a milestones\u2019 survey just prior to milestones implementation. Int J Med Educ. 2016;7:11-18. doi:10.5116\/ijme.5682.6dfd"},{"issue":"(12)","key":"i1949-8357-17-2-162-b23","series-title":"J Pediatr Surg","first-page":"845","article-title":"Pediatric surgery milestones 2.0: a primer","volume":"57","author":"Alaish","year":"2022","unstructured":"Alaish\n              SM\n            , ArcaMJ, BucherBT, et al. Pediatric surgery milestones 2.0: a primer. J Pediatr Surg. 2022;57(12): 845-851. doi:10.1016\/j.jpedsurg.2022.04.021"},{"key":"i1949-8357-17-2-162-b24","series-title":"World Neurosurg","first-page":"499","article-title":"Does one size fit all? Examining the application of neurosurgery residency milestones developed in the United States to a Taiwanese culture","volume":"104","author":"Lee","year":"2017","unstructured":"Lee\n              CY\n            , LaiHY, LeeCH, LeeST. Does one size fit all? Examining the application of neurosurgery residency milestones developed in the United States to a Taiwanese culture. World Neurosurg. 2017;104: 499-508. doi:10.1016\/j.wneu.2017.04.129"},{"issue":"(4)","key":"i1949-8357-17-2-162-b25","series-title":"Acad Med","first-page":"569","article-title":"An empirical investigation into milestones factor structure using national data derived from clinical competency committees","volume":"97","author":"Yamazaki","year":"2022","unstructured":"Yamazaki\n              K\n            , HolmboeES, HamstraSJ. An empirical investigation into milestones factor structure using national data derived from clinical competency committees. Acad Med. 2022;97(4):569-576. doi:10.1097\/acm.0000000000004218"},{"issue":"(1)","key":"i1949-8357-17-2-162-b26","series-title":"Ann Surg","first-page":"180","article-title":"The association of ACGME Milestones with performance on American Board of Surgery assessments: a national investigation of surgical trainees","volume":"279","author":"Weaver","year":"2024","unstructured":"Weaver\n              ML\n            , CarterT, YamazakiK, et al. The association of ACGME Milestones with performance on American Board of Surgery assessments: a national investigation of surgical trainees. Ann Surg. 2024; 279(1):180-186. doi:10.1097\/sla.0000000000005998"},{"key":"i1949-8357-17-2-162-b27","series-title":"J Surg Res","first-page":"34","article-title":"Application factors may not be predictors of success among general surgery residents as measured by ACGME Milestones","volume":"253","author":"Hayek","year":"2020","unstructured":"Hayek\n              SA\n            , WickizerAP, LaneSM, et al. Application factors may not be predictors of success among general surgery residents as measured by ACGME Milestones. J Surg Res. 2020;253:34-40. doi:10.1016\/j.jss.2020. 03.029"},{"issue":"(3)","key":"i1949-8357-17-2-162-b28","series-title":"Surg","first-page":"488","article-title":"Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents","volume":"163","author":"Stride","year":"2018","unstructured":"Stride\n              HP\n            , GeorgeBC, WilliamsRG, et al. Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents. Surg. 2018; 163(3):488-494. doi:10.1016\/j.surg.2017.10.011"},{"issue":"(1)","key":"i1949-8357-17-2-162-b29","series-title":"J Surg Educ","first-page":"147","article-title":"The effect and use of milestones in the assessment of neurological surgery residents and residency programs","volume":"75","author":"Conforti","year":"2018","unstructured":"Conforti\n              LN\n            , YaghmourNA, HamstraSJ, et al. The effect and use of milestones in the assessment of neurological surgery residents and residency programs. J Surg Educ. 2018;75(1):147-155. doi:10.1016\/j.jsurg.2017.06.001"},{"issue":"(11)","key":"i1949-8357-17-2-162-b30","series-title":"Med Teach","first-page":"1228","article-title":"A multispecialty ethnographic study of clinical competency committees (CCCs)","volume":"44","author":"Ekpenyong","year":"2022","unstructured":"Ekpenyong\n              A\n            , EdgarL, WilkersonL, HolmboeES. A multispecialty ethnographic study of clinical competency committees (CCCs). Med Teach. 2022;44(11): 1228-1236. doi:10.1080\/0142159x.2022.2072281"},{"issue":"(4)","key":"i1949-8357-17-2-162-b31","series-title":"J Grad Med Educ","first-page":"439","article-title":"Rising to the challenge: residency programs\u2019 experience with implementing milestones-based assessment","volume":"11","author":"Dzara","year":"2019","unstructured":"Dzara\n              K\n            , HuthK, KesselheimJC, SchumacherDJ. Rising to the challenge: residency programs\u2019 experience with implementing milestones-based assessment. J Grad Med Educ. 2019;11(4):439-446. doi:10.4300\/jgme-d-18-00717.1"},{"issue":"(5)","key":"i1949-8357-17-2-162-b32","series-title":"J Surg Educ","first-page":"1259","article-title":"Qualitative study of orthopedic surgery milestones 1.0: burdens and benefits","volume":"79","author":"Balach","year":"2022","unstructured":"Balach\n              T\n            , ConfortiLN, SanghaS, EdgarL, AmesE. Qualitative study of orthopedic surgery milestones 1.0: burdens and benefits. J Surg Educ. 2022;79(5):1259-1269. doi:10.1016\/j.jsurg.2022.05.002"}],"container-title":["Journal of Graduate Medical Education"],"original-title":[],"link":[{"URL":"https:\/\/jgme.kglmeridian.com\/view\/journals\/jgme\/17\/2\/article-p162.xml","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/meridian.allenpress.com\/jgme\/article-pdf\/17\/2\/162\/3507903\/i1949-8357-17-2-162.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/jgme.kglmeridian.com\/downloadpdf\/journals\/jgme\/17\/2\/article-p162.xml","content-type":"text\/html","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/jgme.kglmeridian.com\/downloadpdf\/journals\/jgme\/17\/2\/article-p162.xml","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T00:54:30Z","timestamp":1770771270000},"score":1,"resource":{"primary":{"URL":"https:\/\/jgme.kglmeridian.com\/view\/journals\/jgme\/17\/2\/article-p162.xml"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4]]},"references-count":64,"journal-issue":{"issue":"2"},"URL":"https:\/\/doi.org\/10.4300\/jgme-d-24-00524.1","relation":{},"ISSN":["1949-8357","1949-8349"],"issn-type":[{"value":"1949-8357","type":"electronic"},{"value":"1949-8349","type":"print"}],"subject":[],"published":{"date-parts":[[2025,4]]}}}