{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T06:12:45Z","timestamp":1788415965493,"version":"build-2803163510"},"reference-count":33,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T00:00:00Z","timestamp":1788393600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Programmatic assessment (PA) is widely endorsed in postgraduate medical education, but empirical accounts of real-world implementation in emergency medicine (EM) remain limited and are concentrated in North American settings. We piloted and evaluated the design, implementation, and outcomes of a PA model in an EM residency at a high-volume tertiary care center in T\u00fcrkiye.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A mixed-methods study was conducted over an eight-month pilot (November 2024\u2013June 2025). Program design and evaluation followed a Theory of Change framework applied across three sequential phases: a structured gap analysis informed by external medical-education experts, competency-based blueprint development across six domains, and pilot implementation in a small core-faculty and resident cohort. Outcomes included feasibility (assessment volume and participation), cohort-level educational impact (longitudinal performance via paired-sample comparisons), and acceptability (as measured by an end-of-pilot satisfaction survey and qualitative feedback).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>A total of 3,107 assessment data points were documented across all six competency domains. Among 16 residents with paired baseline and follow-up data, cohort-level relative performance increased significantly in five of six domains. Evidence-Based Medicine and Lifelong Learning did not change significantly. The survey response rate was 89.5%, and 82.4% of respondents endorsed continuing the program. Acceptability data converged across streams: residents simultaneously valued the multisource perspective and voiced concerns about feedback culture, with the burden of form completion as the dominant practical concern.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Implementation was feasible, accompanied by positive change in cohort-level relative performance, and was broadly acceptable. The most informative finding was a tension between the system's formative design intent and how some residents experienced it, which directly informs next-cycle priorities focused on feedback culture, technology integration, and sustainability.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fmed.2026.1881471","type":"journal-article","created":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T05:15:34Z","timestamp":1788412534000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Design, implementation, and evaluation of a programmatic assessment model in an emergency medicine residency: a mixed-methods study"],"prefix":"10.3389","volume":"13","author":[{"given":"\u0130brahim Ula\u015f","family":"\u00d6zturan","sequence":"first","affiliation":[{"name":"Department of Medical Education, Acibadem University","place":["Istanbul, T\u00fcrkiye"]},{"name":"Department of Emergency Medicine, Kocaeli University Faculty of Medicine","place":["Kocaeli, T\u00fcrkiye"]},{"name":"International FAIMER Institute, Foundation for Advancement of International Medical Education and Research","place":["Philadelphia, PA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dilek","family":"Kitapcioglu","sequence":"additional","affiliation":[{"name":"Department of Medical Education, Acibadem University","place":["Istanbul, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Prattama Santoso","family":"Utomo","sequence":"additional","affiliation":[{"name":"International FAIMER Institute, Foundation for Advancement of International Medical Education and Research","place":["Philadelphia, PA, United States"]},{"name":"Department of Medical Education and Bioethics, Faculty of Medicine, Public Health and Nursing, Universitas Gadjah Mada","place":["Yogyakarta, Indonesia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cansu","family":"Alyesil Ozturan","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Kocaeli City Hospital, Ministry of Health","place":["Kocaeli, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nurettin \u00d6zg\u00fcr","family":"Do\u011fan","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Kocaeli University Faculty of Medicine","place":["Kocaeli, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elif","family":"Yaka","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Kocaeli University Faculty of Medicine","place":["Kocaeli, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Serkan","family":"Y\u0131lmaz","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Kocaeli University Faculty of Medicine","place":["Kocaeli, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Murat","family":"Pekdemir","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Kocaeli University Faculty of Medicine","place":["Kocaeli, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Levent","family":"Alt\u0131nta\u015f","sequence":"additional","affiliation":[{"name":"Department of Medical Education, Acibadem University","place":["Istanbul, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,3]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1111\/j.1365-2929.2005.02094.x","article-title":"Assessing professional competence: from methods to programs","volume":"39","author":"van Der Vleuten","year":"2005","journal-title":"Med Educ"},{"key":"B2","doi-asserted-by":"publisher","first-page":"205","DOI":"10.3109\/0142159X.2012.652239","article-title":"A model for programmatic assessment fit for purpose","volume":"34","author":"van Der Vleuten","year":"2012","journal-title":"Med Teach"},{"key":"B3","doi-asserted-by":"publisher","first-page":"478","DOI":"10.3109\/0142159X.2011.565828","article-title":"Programmatic assessment: from assessment of learning to assessment for learning","volume":"33","author":"Schuwirth","year":"2011","journal-title":"Med Teach"},{"key":"B4","doi-asserted-by":"publisher","first-page":"1139","DOI":"10.1080\/0142159X.2021.1957088","article-title":"Ottawa 2020 Consensus statement for programmatic assessment\u20141. Agreement on the principles","volume":"43","author":"Heeneman","year":"2021","journal-title":"Med Teach"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1873126","DOI":"10.3389\/fmed.2026.1873126","article-title":"Programmatic assessment and competency development in postgraduate medical education: a systematic review and narrative synthesis","volume":"13","author":"\u00d6zturan","year":"2026","journal-title":"Front Med"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1087","DOI":"10.1111\/jep.13328","article-title":"Competency-based education calls for programmatic assessment: but what does this look like in practice?","volume":"26","author":"Rich","year":"2020","journal-title":"J Eval Clin Pract"},{"key":"B7","doi-asserted-by":"publisher","first-page":"57","DOI":"10.4300\/JGME-D-24-00639.1","article-title":"Implementing the 5 core components of competency-based medical education in US emergency medicine residency programs","volume":"17","author":"Caretta-Weyer","year":"2025","journal-title":"J Grad Med Educ"},{"key":"B8","doi-asserted-by":"publisher","first-page":"685","DOI":"10.1097\/ACM.0000000000001066","article-title":"Implementing competency-based medical education in a postgraduate family medicine residency training program: a stepwise approach, facilitating factors, and processes or steps that would have been helpful","volume":"91","author":"Schultz","year":"2016","journal-title":"Acad Med"},{"key":"B9","doi-asserted-by":"publisher","first-page":"1149","DOI":"10.1080\/0142159X.2021.1956681","article-title":"Ottawa 2020 Consensus statement for programmatic assessment\u20142. Implementation and practice","volume":"43","author":"Torre","year":"2021","journal-title":"Med Teach"},{"key":"B10","doi-asserted-by":"publisher","first-page":"694","DOI":"10.1111\/acem.12393","article-title":"Programmatic assessment of level 1 milestones in incoming interns","volume":"21","author":"Hauff","year":"2014","journal-title":"Acad Emerg Med"},{"key":"B11","doi-asserted-by":"publisher","first-page":"84","DOI":"10.4300\/JGME-D-17-00685.1","article-title":"Programmatic assessment in emergency medicine: implementation of best practices","volume":"10","author":"Perry","year":"2018","journal-title":"J Grad Med Educ"},{"key":"B12","doi-asserted-by":"publisher","first-page":"900","DOI":"10.1097\/ACM.0000000000000707","article-title":"The McMaster modular assessment program (McMAP): a theoretically grounded work-based assessment system for an emergency medicine residency program","volume":"90","author":"Chan","year":"2015","journal-title":"Acad Med"},{"key":"B13","doi-asserted-by":"publisher","first-page":"434","DOI":"10.1080\/10401334.2019.1574581","article-title":"Attending emergency physicians\u2019 perceptions of a programmatic workplace-based assessment system: the McMaster modular assessment program (McMAP)","volume":"31","author":"Acai","year":"2019","journal-title":"Teach Learn Med"},{"key":"B14","doi-asserted-by":"publisher","first-page":"103550","DOI":"10.1016\/j.jsurg.2025.103550","article-title":"Competency-based training for orthopedic surgery: an evaluation of the Australian orthopedic association training program 2017\u20132021","volume":"82","author":"Ashman","year":"2025","journal-title":"J Surg Educ"},{"key":"B15","doi-asserted-by":"publisher","first-page":"e2137179","DOI":"10.1001\/jamanetworkopen.2021.37179","article-title":"Longitudinal reliability of milestones-based learning trajectories in family medicine residents","volume":"4","author":"Park","year":"2021","journal-title":"JAMA Netw Open"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1081","DOI":"10.1213\/ANE.0000000000006667","article-title":"Development and pilot testing of a programmatic system for competency assessment in US anesthesiology residency training","volume":"138","author":"Woodworth","year":"2024","journal-title":"Anesth Analg"},{"key":"B17","doi-asserted-by":"publisher","first-page":"293","DOI":"10.3390\/educsci12050293","article-title":"Do resident archetypes influence the functioning of programs of assessment?","volume":"12","author":"Rich","year":"2022","journal-title":"Educ Sci"},{"key":"B18","doi-asserted-by":"publisher","first-page":"478","DOI":"10.1080\/10401334.2023.2229803","article-title":"Assessment-seeking strategies: navigating the decision to initiate workplace-based assessment","volume":"36","author":"Gauthier","year":"2024","journal-title":"Teach Learn Med"},{"key":"B19","doi-asserted-by":"publisher","first-page":"654","DOI":"10.1111\/medu.13532","article-title":"Stakes in the eye of the beholder: an international study of learners\u2019 perceptions within programmatic assessment","volume":"52","author":"Schut","year":"2018","journal-title":"Med Educ"},{"key":"B20","volume-title":"How to Develop an Evidence-informed Theory of Change for Health: WHO Technical Guidance","year":"2024"},{"key":"B21","volume-title":"Acil T\u0131p Uzmanl\u0131k E\u011fitimi K\u0131lavuzu [Emergency Medicine Specialty Training Guide]","year":"2021"},{"key":"B22","volume-title":"TUKMOS \u2014 Acil T\u0131p Uzmanl\u0131k E\u011fitimi \u00c7ekirdek M\u00fcfredat\u0131 [Core Curriculum for Emergency Medicine Residency Training]","year":"2019"},{"key":"B23","year":""},{"key":"B24","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using thematic analysis in psychology","volume":"3","author":"Braun","year":"2006","journal-title":"Qual Res Psychol"},{"key":"B25","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1080\/14780887.2020.1769238","article-title":"One size fits all? What counts as quality practice in (reflexive) thematic analysis?","volume":"18","author":"Braun","year":"2021","journal-title":"Qual Res Psychol"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1609406919899220","DOI":"10.1177\/1609406919899220","article-title":"Intercoder reliability in qualitative research: debates and practical guidelines","volume":"19","author":"O\u2019Connor","year":"2020","journal-title":"Int J Qual Methods"},{"key":"B27","doi-asserted-by":"publisher","first-page":"2134","DOI":"10.1111\/1475-6773.12117","article-title":"Achieving integration in mixed methods designs\u2014principles and practices","volume":"48","author":"Fetters","year":"2013","journal-title":"Health Serv Res"},{"key":"B28","doi-asserted-by":"publisher","first-page":"1515","DOI":"10.1097\/ACM.0000000000000754","article-title":"Developing and successfully implementing a competency-based portfolio assessment system in a postgraduate family medicine residency program","volume":"90","author":"McEwen","year":"2015","journal-title":"Acad Med"},{"key":"B29","doi-asserted-by":"publisher","first-page":"990","DOI":"10.1016\/j.bja.2022.02.038","article-title":"A mobile application to facilitate implementation of programmatic assessment in anesthesia training","volume":"128","author":"Marty","year":"2022","journal-title":"Br J Anaesth"},{"key":"B30","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1097\/ACM.0000000000002495","article-title":"The power of subjectivity in the assessment of medical trainees","volume":"94","author":"Ten Cate","year":"2019","journal-title":"Acad Med"},{"key":"B31","doi-asserted-by":"publisher","first-page":"E729","DOI":"10.46374\/VolXXVI_Issue3_Moore","article-title":"Fine-tuning large language models to enhance programmatic assessment in graduate medical education","volume":"26","author":"Booth","year":"2024","journal-title":"J Educ Perioper Med"},{"key":"B32","doi-asserted-by":"publisher","first-page":"1251","DOI":"10.1097\/ACM.0000000000002750","article-title":"Competency-based medical education in a norm-referenced world: a root cause analysis of challenges to the competency-based paradigm in medical school","volume":"98","author":"Ryan","year":"2023","journal-title":"Acad Med"},{"key":"B33","doi-asserted-by":"publisher","first-page":"1461","DOI":"10.1097\/ACM.0000000000002750","article-title":"SQUIRE-EDU (Standards for QUality Improvement Reporting Excellence in Education): publication guidelines for educational improvement","volume":"94","author":"Ogrinc","year":"2019","journal-title":"Acad Med"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1881471\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T05:15:35Z","timestamp":1788412535000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1881471\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,3]]},"references-count":33,"alternative-id":["10.3389\/fmed.2026.1881471"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1881471","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,3]]},"article-number":"1881471"}}