{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T15:12:26Z","timestamp":1788448346039,"version":"build-2803163510"},"reference-count":56,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T00:00:00Z","timestamp":1784160000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Programmatic assessment (PA) is widely endorsed as a way to organize assessment around competency development in postgraduate medical education (PGME), yet no systematic review has synthesized the empirical evidence in this setting. We therefore synthesized the evidence on its contribution to competency development in PGME and on the factors that shape its implementation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A systematic review was conducted following PRISMA 2020 guidelines (PROSPERO: CRD420261349850). We searched MEDLINE, Embase, Scopus, Web of Science, and EBSCOhost from January 2005 onward. Each included study was mapped against four operational features of PA derived from van der Vleuten et al. Methodological quality was assessed using the Mixed Methods Appraisal Tool (MMAT), and findings were synthesized using a narrative approach.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Twenty studies met inclusion criteria. Four reported competency-related outcomes, all with observational designs. Two evaluated purposefully designed PA systems, and two analyzed milestone-based data generated within regulated competency-based training without an explicitly designed PA system. These studies reported increases in competency ratings and improved examination outcomes over training, though none could establish a causal link to the assessment system. The remaining 16 studies focused on implementation processes, stakeholder perceptions, and measurement properties. Challenges including feedback documentation barriers, misalignment between intended and perceived assessment stakes, and variable stakeholder engagement were reported consistently across settings. Methodological quality was uneven across the included studies (nine rated High, six Moderate, and five Low on a descriptive MMAT-based quality index).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The empirical evidence for PA in PGME remains limited relative to its theoretical foundations. Most evidence reflects how stakeholders experience the system or whether assessment tools produce reliable data, rather than whether the system improves competency or patient care. Future research should extend outcome measurement beyond system-internal metrics and account for the contextual specificity of PA across diverse educational and regulatory environments.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fmed.2026.1873126","type":"journal-article","created":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T11:10:21Z","timestamp":1784200221000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Programmatic assessment and competency development in postgraduate medical education: a systematic review and narrative synthesis"],"prefix":"10.3389","volume":"13","author":[{"given":"\u0130brahim Ula\u015f","family":"\u00d6zturan","sequence":"first","affiliation":[{"name":"Department of Medical Education, Ac\u0131badem University","place":["\u0130stanbul, T\u00fcrkiye"]},{"name":"Department of Emergency Medicine, Kocaeli University","place":["Kocaeli, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"P\u0131nar","family":"Daylan","sequence":"additional","affiliation":[{"name":"Department of Medical Education, Kocaeli University","place":["Kocaeli, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cansu","family":"Alye\u015fil \u00d6zturan","sequence":"additional","affiliation":[{"name":"Kocaeli City Hospital, Ministry of Health","place":["Kocaeli, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dilek","family":"Kitap\u00e7\u0131o\u011flu","sequence":"additional","affiliation":[{"name":"Department of Medical Education, Ac\u0131badem University","place":["\u0130stanbul, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Levent","family":"Alt\u0131nta\u015f","sequence":"additional","affiliation":[{"name":"Department of Medical Education, Ac\u0131badem University","place":["\u0130stanbul, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,7,16]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"638","DOI":"10.3109\/0142159X.2010.501190","article-title":"Competency-based medical education: theory to practice","volume":"32","author":"Frank","year":"2010","journal-title":"Med Teach"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"1176","DOI":"10.1111\/j.1365-2929.2005.02341.x","article-title":"Entrustability of professional activities and competency-based training","volume":"39","author":"Ten Cate","year":"2005","journal-title":"Med Educ"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"568","DOI":"10.1080\/0142159X.2017.1315069","article-title":"Implementing competency-based medical education: moving forward","volume":"39","author":"Frank","year":"2017","journal-title":"Med Teach"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"676","DOI":"10.3109\/0142159X.2010.500704","article-title":"The role of assessment in competency-based medical education","volume":"32","author":"Holmboe","year":"2010","journal-title":"Med Teach"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"855","DOI":"10.1080\/01421590701775453","article-title":"Workplace-based assessment as an educational tool: AMEE guide no. 31","volume":"29","author":"Norcini","year":"2007","journal-title":"Med Teach"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"205","DOI":"10.3109\/0142159X.2012.652239","article-title":"A model for programmatic assessment fit for purpose","volume":"34","author":"Van der Vleuten","year":"2012","journal-title":"Med Teach"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"478","DOI":"10.3109\/0142159X.2011.565828","article-title":"Programmatic assessment: from assessment of learning to assessment for learning","volume":"33","author":"Schuwirth","year":"2011","journal-title":"Med Teach"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1111\/j.1365-2929.2005.02094.x","article-title":"Assessing professional competence: from methods to programs","volume":"39","author":"Van der Vleuten","year":"2005","journal-title":"Med Educ"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1111\/j.1365-2929.2004.02059.x","article-title":"The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study","volume":"39","author":"Driessen","year":"2005","journal-title":"Med Educ"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"1002","DOI":"10.1097\/ACM.0000000000002743","article-title":"A core components framework for evaluating implementation of competency-based medical education programs","volume":"94","author":"Van Melle","year":"2019","journal-title":"Acad Med"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"206","DOI":"10.3109\/0142159X.2011.551559","article-title":"Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 conference","volume":"33","author":"Norcini","year":"2011","journal-title":"Med Teach"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1186\/1472-6920-13-123","article-title":"Programmatic assessment of competency-based workplace learning: when theory meets practice","volume":"13","author":"Bok","year":"2013","journal-title":"BMC Med Educ"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"1139","DOI":"10.1080\/0142159X.2021.1957088","article-title":"Ottawa 2020 consensus statement for programmatic assessment - 1. Agreement on the principles","volume":"43","author":"Heeneman","year":"2020","journal-title":"Med Teach"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"1149","DOI":"10.1080\/0142159X.2021.1956681","article-title":"Ottawa 2020 consensus statements for programmatic assessment - 2. Implementation and practice","volume":"43","author":"Torre","year":"2020","journal-title":"Med Teach"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"487","DOI":"10.1111\/medu.12645","article-title":"The impact of programmatic assessment on student learning: theory versus practice","volume":"49","author":"Heeneman","year":"2015","journal-title":"Med Educ"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"990","DOI":"10.1016\/j.bja.2022.02.038","article-title":"A mobile application to facilitate implementation of programmatic assessment in anaesthesia training","volume":"128","author":"Marty","year":"2022","journal-title":"Br J Anaesth"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"E729","DOI":"10.46374\/VolXXVI_Issue3_Moore","article-title":"Fine-tuning large language models to enhance programmatic assessment in graduate medical education","volume":"26","author":"Booth","year":"2024","journal-title":"J Educ Perioper Med"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"1081","DOI":"10.1213\/ANE.0000000000006667","article-title":"Development and pilot testing of a programmatic system for competency assessment in US anesthesiology residency training","volume":"138","author":"Woodworth","year":"2024","journal-title":"Anesth Analg"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"285","DOI":"10.3233\/EFI-180221","article-title":"The mixed methods appraisal tool (MMAT) version 2018 for information professionals and researchers","volume":"34","author":"Hong","year":"2018","journal-title":"Educ Inf"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref21","volume-title":"Guidance on the Conduct of Narrative Synthesis in Systematic Reviews: A Product from the ESRC Methods Programme. Version 1","author":"Popay","year":"2006"},{"key":"ref22","volume-title":"Kirkpatrick\u2019s Four Levels of Training Evaluation","author":"Kirkpatrick","year":"2016"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1111\/j.1365-2923.2011.04076.x","article-title":"Kirkpatrick\u2019s levels and education 'evidence\u2019","volume":"46","author":"Yardley","year":"2012","journal-title":"Med Educ"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1007\/s10488-010-0319-7","article-title":"Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda","volume":"38","author":"Proctor","year":"2011","journal-title":"Admin Pol Ment Health"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/chp.20001","article-title":"Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities","volume":"29","author":"Moore","year":"2009","journal-title":"J Contin Educ Heal Prof"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"e1001895","DOI":"10.1371\/journal.pmed.1001895","article-title":"Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual)","volume":"12","author":"Lewin","year":"2015","journal-title":"PLoS Med"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"1363222","DOI":"10.3389\/fmed.2024.1363222","article-title":"Comparison of OBGYN postgraduate curricula and assessment methods between Canada and the Netherlands: an auto-ethnographic study","volume":"11","author":"Paternotte","year":"2024","journal-title":"Front Med"},{"key":"ref28","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1097\/ACM.0000000000002959","article-title":"Synthesizing and reporting milestones-based learner analytics: validity evidence from a longitudinal cohort of internal medicine residents","volume":"95","author":"Park","year":"2020","journal-title":"Acad Med"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"e2137179","DOI":"10.1001\/jamanetworkopen.2021.37179","article-title":"Longitudinal reliability of milestones-based learning trajectories in family medicine residents","volume":"4","author":"Park","year":"2021","journal-title":"JAMA Netw Open"},{"key":"ref30","doi-asserted-by":"publisher","first-page":"694","DOI":"10.1111\/acem.12393","article-title":"Programmatic assessment of level 1 milestones in incoming interns","volume":"21","author":"Hauff","year":"2014","journal-title":"Acad Emerg Med"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"10","DOI":"10.17169\/fqs-12.1.1589","article-title":"Autoethnography: an overview","volume":"12","author":"Ellis","year":"2011","journal-title":"Forum Qual Soc Res"},{"key":"ref32","doi-asserted-by":"publisher","first-page":"103550","DOI":"10.1016\/j.jsurg.2025.103550","article-title":"Competency-based training for orthopedic surgery: an evaluation of the Australian orthopedic association training program 2017-2021","volume":"82","author":"Ashman","year":"2025","journal-title":"J Surg Educ"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"900","DOI":"10.1097\/ACM.0000000000000707","article-title":"The McMaster modular assessment program (McMAP): a theoretically grounded work-based assessment system for an emergency medicine residency program","volume":"90","author":"Chan","year":"2015","journal-title":"Acad Med"},{"key":"ref34","doi-asserted-by":"publisher","first-page":"367","DOI":"10.4300\/JGME-D-18-00372.1","article-title":"Milestones 2.0: a step forward","volume":"10","author":"Edgar","year":"2018","journal-title":"J Grad Med Educ"},{"key":"ref35","doi-asserted-by":"publisher","first-page":"1522","DOI":"10.1097\/ACM.0000000000002820","article-title":"A national study of longitudinal consistency in ACGME milestone ratings by clinical competency committees: exploring an aspect of validity in the assessment of residents\u2019 competence","volume":"94","author":"Hamstra","year":"2019","journal-title":"Acad Med"},{"key":"ref36","doi-asserted-by":"publisher","first-page":"S7","DOI":"10.1097\/ACM.0000000000005603","article-title":"The next era of assessment: can ensuring high-quality, equitable patient care be the defining characteristic?","volume":"99","author":"Schumacher","year":"2024","journal-title":"Acad Med"},{"key":"ref37","doi-asserted-by":"publisher","first-page":"654","DOI":"10.1111\/medu.13532","article-title":"Stakes in the eye of the beholder: an international study of learners\u2019 perceptions within programmatic assessment","volume":"52","author":"Schut","year":"2018","journal-title":"Med Educ"},{"key":"ref38","doi-asserted-by":"publisher","first-page":"478","DOI":"10.1080\/10401334.2023.2229803","article-title":"Assessment-seeking strategies: navigating the decision to initiate workplace-based assessment","volume":"36","author":"Gauthier","year":"2024","journal-title":"Teach Learn Med"},{"key":"ref39","doi-asserted-by":"publisher","first-page":"434","DOI":"10.1080\/10401334.2019.1574581","article-title":"Attending emergency physicians\u2019 perceptions of a programmatic workplace-based assessment system: the McMaster modular assessment program (McMAP)","volume":"31","author":"Acai","year":"2019","journal-title":"Teach Learn Med"},{"key":"ref40","doi-asserted-by":"publisher","first-page":"1087","DOI":"10.1111\/jep.13328","article-title":"Competency-based education calls for programmatic assessment: but what does this look like in practice?","volume":"26","author":"Rich","year":"2020","journal-title":"J Eval Clin Pract"},{"key":"ref41","doi-asserted-by":"publisher","first-page":"57","DOI":"10.4300\/JGME-D-24-00639.1","article-title":"Implementing the 5 core components of competency-based medical education in US emergency medicine residency programs","volume":"17","author":"Caretta-Weyer","year":"2025","journal-title":"J Grad Med Educ"},{"key":"ref42","doi-asserted-by":"publisher","first-page":"685","DOI":"10.1097\/ACM.0000000000001066","article-title":"Implementing competency-based medical education in a postgraduate family medicine residency training program: a stepwise approach, facilitating factors, and processes or steps that would have been helpful","volume":"91","author":"Schultz","year":"2016","journal-title":"Acad Med"},{"key":"ref43","doi-asserted-by":"publisher","first-page":"84","DOI":"10.4300\/JGME-D-17-00094.1","article-title":"Programmatic assessment in emergency medicine: implementation of best practices","volume":"10","author":"Perry","year":"2018","journal-title":"J Grad Med Educ"},{"key":"ref44","doi-asserted-by":"publisher","first-page":"1515","DOI":"10.1097\/ACM.0000000000000754","article-title":"Developing and successfully implementing a competency-based portfolio assessment system in a postgraduate family medicine residency program","volume":"90","author":"McEwen","year":"2015","journal-title":"Acad Med"},{"key":"ref45","doi-asserted-by":"publisher","first-page":"293","DOI":"10.3390\/educsci12050293","article-title":"Do resident archetypes influence the functioning of programs of assessment?","volume":"12","author":"Rich","year":"2022","journal-title":"Educ Sci"},{"key":"ref46","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1080\/10401334.2016.1146607","article-title":"Competency evaluations in the next accreditation system: contributing to guidelines and implications","volume":"28","author":"Park","year":"2016","journal-title":"Teach Learn Med"},{"key":"ref47","doi-asserted-by":"publisher","first-page":"1164","DOI":"10.1111\/medu.12289","article-title":"Validity in work-based assessment: expanding our horizons","volume":"47","author":"Govaerts","year":"2013","journal-title":"Med Educ"},{"key":"ref48","doi-asserted-by":"publisher","first-page":"1709","DOI":"10.1016\/j.jsurg.2024.08.008","article-title":"Evaluating the validity of National Multiassessment System in postgraduate surgical training: a retrospective cohort study","volume":"81","author":"O\u2019Keeffe","year":"2024","journal-title":"J Surg Educ"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"157","DOI":"10.4300\/JGME-D-12-00380.1","article-title":"Nuts and bolts of entrustable professional activities","volume":"5","author":"ten Cate","year":"2013","journal-title":"J Grad Med Educ"},{"key":"ref50","doi-asserted-by":"publisher","first-page":"599","DOI":"10.22454\/FamMed.2022.854689","article-title":"A comparison of resident-completed and preceptor-completed formative workplace-based assessments in a competency-based medical education program","volume":"54","author":"Lee","year":"2022","journal-title":"Fam Med"},{"key":"ref51","doi-asserted-by":"publisher","first-page":"188","DOI":"10.1097\/ACM.0000000000004750","article-title":"Development, implementation, and meta-evaluation of a national approach to programmatic assessment in Canadian family medicine residency training","volume":"98","author":"Ross","year":"2023","journal-title":"Acad Med"},{"key":"ref52","doi-asserted-by":"publisher","first-page":"219","DOI":"10.4103\/tjem.tjem_45_23","article-title":"Scholarly impact of the dissertation requirement for postgraduate medical education and factors affecting transformation into publication: a bibliometric analysis of 2434 dissertations in the field of emergency medicine","volume":"23","author":"\u00d6zturan","year":"2023","journal-title":"Turk J Emerg Med"},{"key":"ref53","doi-asserted-by":"publisher","first-page":"659","DOI":"10.1001\/jamainternmed.2016.9623","article-title":"Racial disparities in medical student membership in the alpha omega alpha honor society","volume":"177","author":"Boatright","year":"2017","journal-title":"JAMA Intern Med"},{"key":"ref54","doi-asserted-by":"publisher","first-page":"1286","DOI":"10.1097\/ACM.0000000000002323","article-title":"How small differences in assessed clinical performance amplify to large differences in grades and awards: a cascade with serious consequences for students underrepresented in medicine","volume":"93","author":"Teherani","year":"2018","journal-title":"Acad Med"},{"key":"ref55","doi-asserted-by":"publisher","first-page":"S98","DOI":"10.1097\/ACM.0000000000003717","article-title":"Medical education\u2019s wicked problem: achieving equity in assessment for medical learners","volume":"95","author":"Lucey","year":"2020","journal-title":"Acad Med"},{"key":"ref56","doi-asserted-by":"publisher","first-page":"674","DOI":"10.1097\/ACM.0000000000004574","article-title":"Operationalizing programmatic assessment: the CBME programmatic assessment practice guidelines","volume":"97","author":"Rich","year":"2022","journal-title":"Acad Med"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1873126\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T11:10:24Z","timestamp":1784200224000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1873126\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,16]]},"references-count":56,"alternative-id":["10.3389\/fmed.2026.1873126"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1873126","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,16]]},"article-number":"1873126"}}