{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T12:29:52Z","timestamp":1786710592218,"version":"build-2736575974"},"reference-count":16,"publisher":"Journal of Graduate Medical Education","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,8]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec id=\"st1\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Graduate medical education (GME) leaders face challenges in evaluating program performance due to fragmented data systems and limited resources. Dashboards have been proposed, but few are scalable, participant-informed, and feasible without new infrastructure.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"st2\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To develop and implement a centralized, participant-informed GME Program Scorecard (GPS) that enables continuous program evaluation and improvement using existing data systems without new data entry.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"st3\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>The GPS was developed at a large academic medical center in 2022-2023. A consensus development panel of 20 GME administrators identified 42 candidate metrics across 13 systems. Program directors (PDs) from the largest residency programs were surveyed (11; 9 responses, 81.8%) to rate each metric\u2019s value. The 17 most supported metrics were integrated into a Tableau-based dashboard. The GPS was piloted with 4 programs (3 Accreditation Council for Graduate Medical Education [ACGME]-accredited, 1 non-ACGME-accredited), involving 8 PDs and program coordinators. Feedback was collected via structured forms and a focus group. Metrics were rated on a 1 to 5 scale, with lower scores indicating higher priority.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"st4\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Pre-intervention survey results guided prioritization, with 92.9% of metrics receiving a mean score &lt;3. Pilot users reported high acceptability and identified design refinements incorporated before institutional launch. Development required 2920 analyst hours and $1,600 in licenses. Since launching in January 2024, 309 users (61.4% of 503 potential users) have accessed the GPS with 3933 total views.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"st5\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The GPS demonstrated feasibility, requiring 2920 analyst hours and $1,600 in licenses, and with 61.4% of potential users accessing the dashboard within 22 months of launch.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.4300\/jgme-d-25-00843.1","type":"journal-article","created":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T11:31:19Z","timestamp":1786707079000},"page":"448-457","source":"Crossref","is-referenced-by-count":0,"title":["Where Are We and Where Are We Going? A GME Program Scorecard for Continuous Programmatic Improvement"],"prefix":"10.4300","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2073-0759","authenticated-orcid":true,"given":"Pooja M.","family":"Varman","sequence":"first","affiliation":[{"name":"Pooja M. Varman, MD, MS, is a PGY-2 Resident, Surgical Education Research Fellow, Department of General Surgery, Cleveland Clinic, Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicole E.","family":"Brooks","sequence":"additional","affiliation":[{"name":"Nicole E. Brooks, MD, MEHP, is a PGY-4 Resident, Department of General Surgery, Cleveland Clinic, Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathleen","family":"Bauer","sequence":"additional","affiliation":[{"name":"Kathleen Bauer, MEd, is Program Manager, Graduate Medical Education, Cleveland Clinic, Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James K.","family":"Stoller","sequence":"additional","affiliation":[{"name":"James K. Stoller, MD, MS, is Chairman, Education Institute, Jean Wall Bennett Professor of Medicine, Senior Associate Dean and Endowed Chair, Samson Global Leadership, Cleveland Clinic Lerner College of Medicine of Case Western Reserve University School of Medicine, Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Judith C.","family":"French","sequence":"additional","affiliation":[{"name":"Judith C. French, PhD, is an Associate Professor of Surgery and Vice Chair of Education, Department of General Surgery, Cleveland Clinic and Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeremy M.","family":"Lipman","sequence":"additional","affiliation":[{"name":"Jeremy M. Lipman, MD, MHPE, is a Professor of Surgery, Director of Graduate Medical Education, and Designated Institutional Official, Cleveland Clinic and Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"2856","reference":[{"issue":"12","key":"i1949-8357-18-4-448-b01","series-title":"Acad Med","first-page":"1223","article-title":"Meritocracy in graduate medical education? Some suggestions for creating a report card","volume":"73","author":"Iverson","year":"1998","unstructured":"Iverson DJ. Meritocracy in graduate medical education? Some suggestions for creating a report card. Acad Med. 1998;73(12):1223-1225. doi:10.1097\/00001888-199812000-00008"},{"issue":"6","key":"i1949-8357-18-4-448-b02","series-title":"Acad Med","first-page":"608","article-title":"Program report cards: evaluation across multiple residency programs at one institution","volume":"82","author":"Phitayakorn","year":"2007","unstructured":"Phitayakorn R, Levitan N, Shuck JM. Program report cards: evaluation across multiple residency programs at one institution. Acad Med. 2007;82(6):608-615. doi:10.1097\/ACM.0b013e3180556906"},{"issue":"4","key":"i1949-8357-18-4-448-b03","series-title":"Acad Med","first-page":"419","article-title":"Gathering trainee feedback to improve programs with low annual ACGME survey content area compliance: a pilot study","volume":"99","author":"Hoffert","year":"2024","unstructured":"Hoffert MM, Pfeiffer L, Hepke M, et al. Gathering trainee feedback to improve programs with low annual ACGME survey content area compliance: a pilot study. Acad Med. 2024;99(4):419-423. doi:10.1097\/ACM.0000000000005464"},{"issue":"5","key":"i1949-8357-18-4-448-b04","series-title":"Acad Med","first-page":"671","article-title":"Defining and adopting clinical performance measures in graduate medical education: where are we now and where are we going?","volume":"94","author":"Smirnova","year":"2019","unstructured":"Smirnova A, Sebok-Syer SS, Chahine S, et al. Defining and adopting clinical performance measures in graduate medical education: where are we now and where are we going? Acad Med. 2019;94(5):671-677. doi:10.1097\/ACM.0000000000002620"},{"issue":"7","key":"i1949-8357-18-4-448-b05","series-title":"Acad Med","first-page":"975","article-title":"Illuminating graduate medical education outcomes in order to improve them","volume":"93","author":"Weinstein","year":"2018","unstructured":"Weinstein DF, Thibault GE. Illuminating graduate medical education outcomes in order to improve them. Acad Med. 2018;93(7):975-978. doi:10.1097\/ACM.0000000000002244"},{"issue":"2","key":"i1949-8357-18-4-448-b06","series-title":"Acad Med","first-page":"139","article-title":"Improving graduate medical education by aggregating data across the medical education continuum","volume":"99","author":"Thelen","year":"2024","unstructured":"Thelen AE, George BC, Burkhardt JC, Khamees D, Haas MRC, Weinstein D. Improving graduate medical education by aggregating data across the medical education continuum. Acad Med. 2024;99(2):139-145. doi:10.1097\/ACM.0000000000005313"},{"issue":"3","key":"i1949-8357-18-4-448-b07","series-title":"J Grad Med Educ","first-page":"384","article-title":"Facilitating institutional oversight and program improvement through educational competency committees","volume":"8","author":"Andolsek","year":"2016","unstructured":"Andolsek KM, Fortune RF, Nagler A, Stancil C, Kuhn C, McNeill D. Facilitating institutional oversight and program improvement through educational competency committees. J Grad Med Educ. 2016;8(3):384-389. doi:10.4300\/JGME-D-15-00307.1"},{"issue":"4","key":"i1949-8357-18-4-448-b08","series-title":"Acad Med","first-page":"404","article-title":"Placing constraints on the use of the ACGME Milestones: a commentary on the limitations of global performance ratings","volume":"90","author":"Williams","year":"2015","unstructured":"Williams RG, Dunnington GL, Mellinger JD, Klamen DL. Placing constraints on the use of the ACGME Milestones: a commentary on the limitations of global performance ratings. Acad Med. 2015;90(4):404-407. doi:10.1097\/ACM.0000000000000507"},{"issue":"8","key":"i1949-8357-18-4-448-b09","series-title":"Med Teach","first-page":"855","article-title":"Twelve tips to promote successful development of a learner performance dashboard within a medical education program","volume":"40","author":"Boscardin","year":"2018","unstructured":"Boscardin C, Fergus KB, Hellevig B, Hauer KE. Twelve tips to promote successful development of a learner performance dashboard within a medical education program. Med Teach. 2018;40(8):855-861. doi:10.1080\/0142159X.2017.1396306"},{"issue":"suppl 4","key":"i1949-8357-18-4-448-b10","series-title":"J Grad Med Educ","first-page":"169","article-title":"Putting time in perspective: an integrated graduate medical education institutional dashboard and report card","volume":"11","author":"Andrada","year":"2019","unstructured":"Andrada J, Teo J, Neo J, Yeo H, Leng LB. Putting time in perspective: an integrated graduate medical education institutional dashboard and report card. J Grad Med Educ. 2019;11(suppl 4):169-176. doi:10.4300\/JGME-D-18-00482"},{"key":"i1949-8357-18-4-448-b11","article-title":"ACGME Institutional Requirements","unstructured":"Accreditation Council for Graduate Medical Education. ACGME Institutional Requirements. Accessed June 2, 2026. https:\/\/www.acgme.org\/globalassets\/pfassets\/programrequirements\/2025-reformatted-requirements\/institutionalrequirements_2025_reformatted.pdf"},{"issue":"5","key":"i1949-8357-18-4-448-b12","series-title":"Acad Med","first-page":"787","article-title":"Perspective: the ACGME toolbox: half empty or half full?","volume":"85","author":"Green","year":"2010","unstructured":"Green ML, Holmboe E. Perspective: the ACGME toolbox: half empty or half full? Acad Med. 2010;85(5): 787-790. doi:10.1097\/ACM.0b013e3181d737a6"},{"issue":"1","key":"i1949-8357-18-4-448-b13","series-title":"BMC Med Educ","first-page":"245","article-title":"Achieving educational mission and vision with an educational scorecard","volume":"18","author":"Huntington","year":"2018","unstructured":"Huntington J, Dick JF, Ryder HF. Achieving educational mission and vision with an educational scorecard. BMC Med Educ. 2018;18(1):245. doi:10.1186\/s12909-018-1354-4"},{"issue":"5","key":"i1949-8357-18-4-448-b14","series-title":"Acad Med","first-page":"663","article-title":"Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research","volume":"91","author":"Waggoner","year":"2016","unstructured":"Waggoner J, Carline JD, Durning SJ. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad Med. 2016;91(5):663-668. doi:10.1097\/ACM.0000000000001092"},{"key":"i1949-8357-18-4-448-b15","article-title":"A usability instrument for evaluating websites\u2014navigation elements. Paper presented at Australasian Computer Human Interaction Conference Australasian Computer Human Interaction Conference","author":"Fisher","year":"2004","unstructured":"Fisher J, Bentley J, Turner R, Craig A. A usability instrument for evaluating websites\u2014navigation elements. Paper presented at Australasian Computer Human Interaction Conference Australasian Computer Human Interaction Conference; January 2004."},{"issue":"1","key":"i1949-8357-18-4-448-b16","series-title":"J Grad Med Educ","first-page":"119","article-title":"The annual institutional review\u2014key performance measures and processes","volume":"13","author":"Amos","year":"2021","unstructured":"Amos S, Wiggins JB, Shaw EK, Hannah WN. The annual institutional review\u2014key performance measures and processes. J Grad Med Educ. 2021;13(1):119-122. doi:10.4300\/JGME-D-20-00491.1"}],"container-title":["Journal of Graduate Medical Education"],"original-title":[],"link":[{"URL":"https:\/\/jgme.kglmeridian.com\/view\/journals\/jgme\/18\/4\/article-p448.xml","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/jgme.kglmeridian.com\/downloadpdf\/journals\/jgme\/18\/4\/article-p448.xml","content-type":"text\/html","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/jgme.kglmeridian.com\/downloadpdf\/journals\/jgme\/18\/4\/article-p448.xml","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T11:31:35Z","timestamp":1786707095000},"score":1,"resource":{"primary":{"URL":"https:\/\/jgme.kglmeridian.com\/view\/journals\/jgme\/18\/4\/article-p448.xml"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8]]},"references-count":16,"journal-issue":{"issue":"4"},"URL":"https:\/\/doi.org\/10.4300\/jgme-d-25-00843.1","relation":{},"ISSN":["1949-8357","1949-8349"],"issn-type":[{"value":"1949-8357","type":"electronic"},{"value":"1949-8349","type":"print"}],"subject":[],"published":{"date-parts":[[2026,8]]}}}