{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T00:09:02Z","timestamp":1771459742225,"version":"3.50.1"},"reference-count":30,"publisher":"California Digital Library (CDL)","issue":"1","license":[{"start":{"date-parts":[[2019,12,19]],"date-time":"2019-12-19T00:00:00Z","timestamp":1576713600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["WestJEM"],"abstract":"<jats:p>Introduction: Evaluators use assessment data to make judgments on resident performance within the Accreditation Council for Graduate Medical Education (ACGME) milestones framework. While workplace-based narrative assessments (WBNA) offer advantages to rating scales, validity evidence for their use in assessing the milestone sub-competencies is lacking. This study aimed to determine the frequency of sub-competencies assessed through WBNAs in an emergency medicine (EM) residency program. \n\nMethods: We performed a retrospective analysis of WBNAs of postgraduate year (PGY) 2-4 residents. A shared mental model was established by reading and discussing the milestones framework, and we created a guide for coding WBNAs to the milestone sub-competencies in an iterative process. Once inter-rater reliability was satisfactory, raters coded each WBNA to the 23 EM milestone sub-competencies.\n\nResults: We analyzed 2517 WBNAs. An average of 2.04 sub-competencies were assessed per WBNA. The sub-competencies most frequently identified were multitasking, medical knowledge, practice-based performance improvement, patient-centered communication, and team management. The sub-competencies least frequently identified were pharmacotherapy, airway management, anesthesia and acute pain management, goal-directed focused ultrasound, wound management, and vascular access. Overall, the frequency with which WBNAs assessed individual sub-competencies was low, with 14 of the 23 sub-competencies being assessed in less than 5% of WBNAs. \n\nConclusion: WBNAs identify few milestone sub-competencies. Faculty assessed similar sub-competencies related to interpersonal and communication skills, practice-based learning and improvement, and medical knowledge, while neglecting sub-competencies related to patient care and procedural skills. These findings can help shape faculty development programs designed to improve assessments of specific workplace behaviors and provide more robust data for the summative assessment of residents.<\/jats:p>","DOI":"10.5811\/westjem.2019.12.44468","type":"journal-article","created":{"date-parts":[[2020,1,3]],"date-time":"2020-01-03T19:05:28Z","timestamp":1578078328000},"page":"173-179","source":"Crossref","is-referenced-by-count":5,"title":["Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?"],"prefix":"10.5811","volume":"21","author":[{"given":"David","family":"Diller","sequence":"first","affiliation":[{"name":"LAC+USC Medical Center, Keck School of Medicine of the University of Southern California, Department of Emergency Medicine, Los Angeles, California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shannon","family":"Cooper","sequence":"additional","affiliation":[{"name":"Henry Ford Allegiance Health, Department of Emergency Medicine, Jackson, Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aarti","family":"Jain","sequence":"additional","affiliation":[{"name":"LAC+USC Medical Center, Keck School of Medicine of the University of Southern California, Department of Emergency Medicine, Los Angeles, California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chun","family":"Lam","sequence":"additional","affiliation":[{"name":"LAC+USC Medical Center, Keck School of Medicine of the University of Southern California, Department of Emergency Medicine, Los Angeles, California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeff","family":"Riddell","sequence":"additional","affiliation":[{"name":"LAC+USC Medical Center, Keck School of Medicine of the University of Southern California, Department of Emergency Medicine, Los Angeles, California"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"29705","published-online":{"date-parts":[[2019,12,19]]},"reference":[{"key":"ref0","doi-asserted-by":"publisher","unstructured":"Nasca TJ, Philibert I, Brigham T, et al. The next GME accreditation system--rationale and benefits. N Engl J Med. 2012;366(11):1051-6.","DOI":"10.1056\/nejmsr1200117"},{"key":"ref1","unstructured":"Team NK. Exploring the ACGME core competencies. Available at: https:\/\/knowledgeplus.nejm.org\/blog\/exploring-acgme-core-competencies\/. Accessed 1\/25\/2019."},{"key":"ref2","doi-asserted-by":"crossref","unstructured":"Carter WA. Milestone myths and misperceptions. J Grad Med Educ. 2014;6(1):18-20.","DOI":"10.4300\/JGME-D-13-00438.1"},{"key":"ref3","doi-asserted-by":"publisher","unstructured":"Dehon E, Jones J, Puskarich M, et al. Use of emergency medicine milestones as items on end-of-shift evaluations results in overestimates of residents\u2019 proficiency level. J Grad Med Educ. 2015;7(2):192-6.","DOI":"10.4300\/jgme-d-14-00438.1"},{"key":"ref4","doi-asserted-by":"crossref","unstructured":"Schott M, Kedia R, Promes SB, et al. Direct observation assessment of milestones: problems with reliability. West J Emerg Med. 2015;16(6):871-6.","DOI":"10.5811\/westjem.2015.9.27270"},{"key":"ref5","doi-asserted-by":"publisher","unstructured":"Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316-26.","DOI":"10.1001\/jama.2009.1365"},{"key":"ref6","doi-asserted-by":"publisher","unstructured":"Ginsburg S, van der Vleuten CPM, Eva KW. The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Acad Med. 2017;92(11):1617-21.","DOI":"10.1097\/acm.0000000000001669"},{"key":"ref7","doi-asserted-by":"publisher","unstructured":"Hanson JL, Rosenberg AA, Lane JL. Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States. Front Psychol. 2013;4:668.","DOI":"10.3389\/fpsyg.2013.00668"},{"key":"ref8","doi-asserted-by":"publisher","unstructured":"Hatala R, Sawatsky AP, Dudek N, et al. Using in-training evaluation report (ITER) Qualitative comments to assess medical students and residents: a systematic review. Acad Med. 2017;92(6):868-79.","DOI":"10.1097\/acm.0000000000001506"},{"key":"ref9","doi-asserted-by":"publisher","unstructured":"Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med. 2013;88(10):1539-44.","DOI":"10.1097\/acm.0b013e3182a36c3d"},{"key":"ref10","doi-asserted-by":"publisher","unstructured":"Cohen G, Blumberg P, Ryan N, P. S. Do final grades reflect written qualitative evaluations of student performance? Teaching and Learning in Medicine. 1993;5(1):10-15.","DOI":"10.1080\/10401339309539580"},{"key":"ref11","doi-asserted-by":"crossref","unstructured":"Frohna A, Stern D. The nature of qualitative comments in evaluating professionalism. Med Educ. 2005;39(8):763-8.","DOI":"10.1111\/j.1365-2929.2005.02234.x"},{"key":"ref12","doi-asserted-by":"publisher","unstructured":"Guerrasio J, Cumbler E, Trosterman A, et al. Determining need for remediation through postrotation evaluations. J Grad Med Educ. 2012;4(1):47-51.","DOI":"10.4300\/jgme-d-11-00145.1"},{"key":"ref13","doi-asserted-by":"crossref","unstructured":"Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med. 2010;85(7):1212-20.","DOI":"10.1097\/ACM.0b013e3181d85a4e"},{"key":"ref14","doi-asserted-by":"publisher","unstructured":"Govaerts MJ, van der Vleuten CP, Schuwirth LW, et al. The use of observational diaries in in-training evaluation: student perceptions. Adv Health Sci Educ Theory Pract. 2005;10(3):171-88.","DOI":"10.1007\/s10459-005-0398-5"},{"key":"ref15","doi-asserted-by":"publisher","unstructured":"Lefebvre C, Hiestand B, Glass C, et al. Examining the effects of narrative commentary on evaluators\u2019 summative assessments of resident performance. Eval Health Prof. 2018:163278718820415.","DOI":"10.1177\/0163278718820415"},{"key":"ref16","doi-asserted-by":"publisher","unstructured":"Ginsburg S, van der Vleuten C, Eva KW, et al. Hedging to save face: a linguistic analysis of written comments on in-training evaluation reports. Adv Health Sci Educ Theory Pract. 2016;21(1):175-88.","DOI":"10.1007\/s10459-015-9622-0"},{"key":"ref17","doi-asserted-by":"publisher","unstructured":"Ginsburg S, Gold W, Cavalcanti RB, et al. Competencies \u201cplus\u201d: the nature of written comments on internal medicine residents\u2019 evaluation forms. Acad Med. 2011;86(10 Suppl):S30-34.","DOI":"10.1097\/acm.0b013e31822a6d92"},{"key":"ref18","doi-asserted-by":"publisher","unstructured":"Lye PS, Biernat KA, Bragg DS, et al. A pleasure to work with--an analysis of written comments on student evaluations. Ambul Pediatr. 2001;1(3):128-31.","DOI":"10.1367\/1539-4409(2001)001<0128:aptwwa>2.0.co;2"},{"key":"ref19","doi-asserted-by":"crossref","unstructured":"Kiefer CS, Colletti JE, Bellolio MF, et al. The \u201cgood\u201d dean\u2019s letter. Acad Med. 2010;85(11):1705-8.","DOI":"10.1097\/ACM.0b013e3181f55a10"},{"key":"ref20","doi-asserted-by":"publisher","unstructured":"Canavan C, Holtman MC, Richmond M, et al. The quality of written comments on professional behaviors in a developmental multisource feedback program. Acad Med. 2010;85(10 Suppl):S106-9.","DOI":"10.1097\/acm.0b013e3181ed4cdb"},{"key":"ref21","unstructured":"Beeson M, Christopher T, Heidt J, et al. The Emergency Medicine Milestone Project: A Joint Initiative of the Accreditation Council for Graduate Medical Education and the American Board of Emergency Medicine. Available at: https:\/\/www.acgme.org\/Portals\/0\/PDFs\/Milestones\/EmergencyMedicineMilestones.pdf. Accessed January 25, 2019."},{"key":"ref22","doi-asserted-by":"publisher","unstructured":"van der Vleuten CP, Schuwirth LW, Scheele F, et al. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. 2010;24(6):703-19.","DOI":"10.1016\/j.bpobgyn.2010.04.001"},{"key":"ref23","doi-asserted-by":"crossref","unstructured":"Miller GE. The assessment of clinical skills\/competence\/performance. Acad Med. 1990;65(9 Suppl):S63-7.","DOI":"10.1097\/00001888-199009000-00045"},{"key":"ref24","doi-asserted-by":"crossref","unstructured":"Park YS, Zar FA, Norcini JJ, et al. Competency evaluations in the Next Accreditation System: contributing to guidelines and implications. Teach Learn Med. 2016;28(2):135-45.","DOI":"10.1080\/10401334.2016.1146607"},{"key":"ref25","doi-asserted-by":"publisher","unstructured":"Tekian A, Borhani M, Tilton S, et al. What do quantitative ratings and qualitative comments tell us about general surgery residents\u2019 progress toward independent practice? Evidence from a 5-year longitudinal cohort. Am J Surg. 2019;217(2):288-95.","DOI":"10.1016\/j.amjsurg.2018.09.031"},{"key":"ref26","doi-asserted-by":"publisher","unstructured":"Ginsburg S, Regehr G, Lingard L, et al. Reading between the lines: faculty interpretations of narrative evaluation comments. Med Educ. 2015;49(3):296-306.","DOI":"10.1111\/medu.12637"},{"key":"ref27","doi-asserted-by":"publisher","unstructured":"Ekpenyong A, Baker E, Harris I, et al. How do clinical competency committees use different sources of data to assess residents\u2019 performance on the internal medicine milestones? A mixed methods pilot study. Med Teach. 2017;39(10):1074-83.","DOI":"10.1080\/0142159x.2017.1353070"},{"key":"ref28","doi-asserted-by":"publisher","unstructured":"Dayal A, O\u2019Connor DM, Qadri U, et al. Comparison of male vs female resident milestone evaluations by faculty during emergency medicine residency training. JAMA Intern Med. 2017;177(5):651-7.","DOI":"10.1001\/jamainternmed.2016.9616"},{"key":"ref29","doi-asserted-by":"crossref","unstructured":"Chary M, Parikh S, Manini A, et al. A review of natural language processing in medical education. West J Emerg Med. 2019;20(1):78-86.","DOI":"10.5811\/westjem.2018.11.39725"}],"container-title":["Western Journal of Emergency Medicine"],"original-title":[],"deposited":{"date-parts":[[2020,3,9]],"date-time":"2020-03-09T22:23:01Z","timestamp":1583792581000},"score":1,"resource":{"primary":{"URL":"https:\/\/escholarship.org\/uc\/item\/1ds5s744"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,19]]},"references-count":30,"issue-title":["CDEM\/CORD Special Issue 21.1"],"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,12,20]]},"published-print":{"date-parts":[[2020,1,10]]}},"URL":"https:\/\/doi.org\/10.5811\/westjem.2019.12.44468","relation":{},"ISSN":["1936-900X"],"issn-type":[{"value":"1936-900X","type":"print"}],"subject":[],"published":{"date-parts":[[2019,12,19]]}}}