{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T19:01:40Z","timestamp":1783710100245,"version":"3.55.0"},"reference-count":11,"publisher":"Journal of Graduate Medical Education","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,2,1]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Otolaryngology\u2013head and neck surgery is in the first wave of residency training programs in Canada to adopt Competence by Design (CBD), a model of competency-based medical education. CBD is built on frequent, low-stakes assessments and requires an increase in the number of feedback interactions. The University of Toronto otolaryngology\u2013head and neck surgery residents piloted the CBD model but were completing only 1 assessment every 4 weeks, which was insufficient to support CBD.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This project aimed to increase assessment completion to once per resident per week using quality improvement methodology.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Stakeholder engagement activities had residents and faculty characterize barriers to assessment completion. Brief electronic assessment forms were completed by faculty on residents' personal mobile devices in face-to-face encounters, and the number completed per resident was tracked for 10 months during the 2016\u20132017 pilot year. Response to the intervention was analyzed using statistical process control charts.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The first bundled intervention\u2014a rule set dictating which clinical instance should be assessed, combined with a weekly reminder implemented for 10 weeks\u2014was unsuccessful in increasing the frequency of assessments. The second intervention was a leaderboard, designed on an audit-and-feedback system, which sent weekly comparison e-mails of each resident's completion rate to all residents and the program director. The leaderboard demonstrated significant improvement from baseline over 10 weeks, increasing the assessment completion rate from 0.22 to 2.87 assessments per resident per week.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>A resident-designed audit-and-feedback leaderboard system improved the frequency of CBD assessment completion.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.4300\/jgme-d-19-00358.1","type":"journal-article","created":{"date-parts":[[2020,1,2]],"date-time":"2020-01-02T16:31:41Z","timestamp":1577982701000},"page":"46-50","source":"Crossref","is-referenced-by-count":6,"title":["Feedback Frequency in Competence by Design: A Quality Improvement Initiative"],"prefix":"10.4300","volume":"12","author":[{"given":"Neil","family":"Arnstead","sequence":"first","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paolo","family":"Campisi","sequence":"additional","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Susan Glover","family":"Takahashi","sequence":"additional","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chris J.","family":"Hong","sequence":"additional","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Florence","family":"Mok","sequence":"additional","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher W.","family":"Noel","sequence":"additional","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Siu","sequence":"additional","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian M.","family":"Wong","sequence":"additional","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric","family":"Monteiro","sequence":"additional","affiliation":[{"name":"All authors are with University of Toronto, Ontario, Canada. Neil Arnstead, MD, is a Resident, Department of Otolaryngology\u2013Head and Neck Surgery; Paolo Campisi, MD, MSc, FRCSC, is Professor and Director for Postgraduate Education, Department of Otolaryngology\u2013Head and Neck Surgery, and Staff Otolaryngologist, Hospital for Sick Children, Toronto, Ontario, Canada; Susan Glover Takahashi, PhD, MA(E"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"2856","reference":[{"key":"2020111914555346400_i1949-8357-12-1-46-b01","article-title":"Competence by design","author":"Royal College of Physicians and Surgeons of Canada","year":"2014"},{"issue":"8","key":"2020111914555346400_i1949-8357-12-1-46-b02","doi-asserted-by":"publisher","first-page":"638","DOI":"10.3109\/0142159X.2010.501190","article-title":"Competency-based medical education: theory to practice","volume":"32","author":"Frank","year":"2010","journal-title":"Med Teach"},{"issue":"6","key":"2020111914555346400_i1949-8357-12-1-46-b03","doi-asserted-by":"publisher","first-page":"542","DOI":"10.1097\/ACM.0b013e31805559c7","article-title":"Competency-based postgraduate training: can we bridge the gap between theory and clinical practice?","volume":"82","author":"ten Cate","year":"2007","journal-title":"Acad Med"},{"key":"2020111914555346400_i1949-8357-12-1-46-b04","unstructured":"Equator Network. SQUIRE 2.0 Standards for quality improvement reporting excellence: revised publication guidelines from a detailed consensus process. http:\/\/www.equator-network.org\/reporting-guidelines\/squire. Accessed December 6, 2019."},{"key":"2020111914555346400_i1949-8357-12-1-46-b05","unstructured":"Langley \n              G, \n            \n            \n              Moen \n              R, \n            \n            \n              Noaln \n              T, \n            \n            \n              Norman \n              C, \n            \n            \n              Provost \n              L. \n            \n          \n          The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. 2nd ed. San Francisco, CA: Jossey-Bass;2009."},{"key":"2020111914555346400_i1949-8357-12-1-46-b06","article-title":"ISMP Medication Safety Alert!","year":"1999","journal-title":"Inst Safe Medicat Pract Newsl"},{"issue":"3","key":"2020111914555346400_i1949-8357-12-1-46-b07","doi-asserted-by":"publisher","first-page":"555","DOI":"10.4300\/jgme-d-13-00409.1","article-title":"The milestones passport: a learner-centered application of the milestone framework to prompt real-time feedback in the emergency department","volume":"6","author":"Yarris","year":"2014","journal-title":"J Grad Med Educ"},{"issue":"12","key":"2020111914555346400_i1949-8357-12-1-46-b08","first-page":"1260","article-title":"Field notes: assisting achievement and documenting competence","volume":"55","author":"Donoff","year":"2009","journal-title":"Can Fam Physician"},{"issue":"2","key":"2020111914555346400_i1949-8357-12-1-46-b09","doi-asserted-by":"publisher","first-page":"673","DOI":"10.1097\/01.prs.0000475797.69478.0e","article-title":"Comprehensive observations of resident evolution: a novel method for assessing procedure-based residency training","volume":"137","author":"Cooney","year":"2016","journal-title":"Plast Reconstr Surg"},{"issue":"1","key":"2020111914555346400_i1949-8357-12-1-46-b10","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1016\/j.jamcollsurg.2018.02.014","article-title":"Comprehensive multicenter graduate surgical education initiative incorporating entrustable professional activities, continuous quality improvement cycles, and a web-based platform to enhance teaching and learning","volume":"227","author":"Anderson","year":"2018","journal-title":"J Am Coll Surg"},{"issue":"4","key":"2020111914555346400_i1949-8357-12-1-46-b11","doi-asserted-by":"publisher","first-page":"430","DOI":"10.4300\/jgme-d-18-01006.1","article-title":"Workplace-based assessments using pediatric critical care entrustable professional activities","volume":"11","author":"Emke","year":"2019","journal-title":"J Grad Med Educ"}],"container-title":["Journal of Graduate Medical Education"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/meridian.allenpress.com\/jgme\/article-pdf\/12\/1\/46\/2426818\/jgme-d-19-00358_1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/meridian.allenpress.com\/jgme\/article-pdf\/12\/1\/46\/2426818\/jgme-d-19-00358_1.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T19:58:33Z","timestamp":1761163113000},"score":1,"resource":{"primary":{"URL":"https:\/\/jgme.kglmeridian.com\/view\/journals\/jgme\/12\/1\/article-p46.xml"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,1]]},"references-count":11,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,2,1]]}},"URL":"https:\/\/doi.org\/10.4300\/jgme-d-19-00358.1","relation":{},"ISSN":["1949-8349","1949-8357"],"issn-type":[{"value":"1949-8349","type":"print"},{"value":"1949-8357","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,1]]}}}