{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T12:30:20Z","timestamp":1786710620576,"version":"build-2736575974"},"reference-count":8,"publisher":"Journal of Graduate Medical Education","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,8]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec id=\"st1\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Summarizing resident assessments for Clinical Competency Committee or mentoring meetings is challenging. This especially applies to raters\u2019 free-text comments, which often provide valuable feedback.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"st2\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To assess feasibility and accuracy of using a large language model (LLM) to label free-text comments as \u201cStrengths\u201d or \u201cAreas for Improvement\u201d from end-of-rotation assessments of residents.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"st3\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We performed a mixed-methods study of residents\u2019 end-of-rotation assessments completed between July 1, 2024 and June 30, 2025. Following data de-identification, we used a private (paid) LLM account (ChatGPT-4o) to compile numerical and label free-text (\u201cStrengths,\u201d \u201cAreas for Improvement\u201d) feedback. We assessed feasibility using time spent de-identifying data, time for LLMs to return labeled free-text comments, and LLM recall (ability to accurately extract\/label free-text feedback). We performed 2 confirmatory trials (free LLMs: MS CoPilot, ChatGPT-4) to ascertain reproducibility. Two subject matter experts labeled free-text comments to determine LLM labeling accuracy. The study was performed in 2025.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"st4\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Our dataset represented 277 individual assessments for 25 unique residents. Feasibility data demonstrated data de-identification time of \u223c60 minutes and LLM processing\/labeling times of &lt;7 minutes per LLM. All LLMs consistently compiled all free-text entries, accurately labeling 59.7% of free-text comments. Human analysis of comments (476) identified 388 (81.5%) \u201cStrengths\u201d and 88 (18.5%) \u201cAreas for Improvement.\u201d Of these, LLM recall was 67.8% for \u201cStrengths\u201d and 23.9% for \u201cAreas for Improvement.\u201d<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"st5\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>We designed prompts that feasibly labeled end-of-rotation assessment free-text comments across 3 LLMs with recall for \u201cStrengths\u201d 2.5 times higher than recall for \u201cAreas for Improvement.\u201d<\/jats:p>\n                  <\/jats:sec>","DOI":"10.4300\/jgme-d-25-01049.1","type":"journal-article","created":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T11:31:32Z","timestamp":1786707092000},"page":"477-482","source":"Crossref","is-referenced-by-count":0,"title":["Leveraging Large Language Models to Summarize Data and Free-Text Comments From Resident Assessments"],"prefix":"10.4300","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5475-206X","authenticated-orcid":true,"given":"Carisa M.","family":"Cooney","sequence":"first","affiliation":[{"name":"Carisa M. Cooney, MPH, is an Associate Professor and Director of Education Innovation, Department of Plastic and Reconstructive Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9068-9371","authenticated-orcid":true,"given":"Rebecca","family":"Slattery","sequence":"additional","affiliation":[{"name":"Rebecca Slattery is Program Manager, Department of Plastic and Reconstructive Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2786-287X","authenticated-orcid":true,"given":"Damon S.","family":"Cooney","sequence":"additional","affiliation":[{"name":"Damon S. Cooney, MD, PhD, is an Associate Professor, Department of Plastic and Reconstructive Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0703-6251","authenticated-orcid":true,"given":"Scott D.","family":"Lifchez","sequence":"additional","affiliation":[{"name":"Scott D. Lifchez, MD, is Interim Chair, Program Director, and a Professor, Department of Plastic and Reconstructive Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julie E.","family":"Park","sequence":"additional","affiliation":[{"name":"Julie E. Park, MD, FACS, is Division Chief, Plastic Surgery, University Hospital Ahuja Medical Center, Division Chief, Plastic and Reconstructive Surgery, University Hospital Cleveland Medical Center, and a Professor, Case Western Reserve University School of Medicine, Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"2856","reference":[{"key":"i1949-8357-18-4-477-b01","article-title":"ACGME Common Program Requirements (Residency) Section 5.1","unstructured":"Accreditation Council for Graduate Medical Education. ACGME Common Program Requirements (Residency) Section 5.1. Accessed July 14, 2025. https:\/\/www.acgme.org\/globalassets\/pfassets\/programrequirements\/2025-reformatted-requirements\/cprresidency_2025_reformatted.pdf"},{"issue":"1","key":"i1949-8357-18-4-477-b02","series-title":"BMC Med Educ","first-page":"192","article-title":"Implementation of written structured feedback into a surgical OSCE","volume":"21","author":"Sterz","year":"2021","unstructured":"Sterz J, Lin\u00dfen S, Stefanescu MC, Schreckenbach T, Seifert LB, Ruesseler M. Implementation of written structured feedback into a surgical OSCE. BMC Med Educ. 2021;21(1):192. doi:10.1186\/s12909-021-02581-3"},{"issue":"6","key":"i1949-8357-18-4-477-b03","series-title":"J Surg Educ","first-page":"e72","article-title":"Natural language processing and assessment of resident feedback quality","volume":"78","author":"Solano","year":"2021","unstructured":"Solano QP, Hayward L, Chopra Z, et al. Natural language processing and assessment of resident feedback quality. J Surg Educ. 2021;78(6):e72-e77. doi:10.1016\/j.jsurg.2021.05.012"},{"issue":"2","key":"i1949-8357-18-4-477-b04","series-title":"Am J Surg","first-page":"369","article-title":"Natural language processing and entrustable professional activity text feedback in surgery: a machine learning model of resident autonomy","volume":"221","author":"Stahl","year":"2021","unstructured":"Stahl CC, Jung SA, Rosser AA, et al. Natural language processing and entrustable professional activity text feedback in surgery: a machine learning model of resident autonomy. Am J Surg. 2021;221(2):369-375. doi:10.1016\/j.amjsurg.2020.11.044"},{"key":"i1949-8357-18-4-477-b05","series-title":"Hybrid Computational Intelligence for Pattern Analysis and Understanding, Computational Intelligence Applications for Text and Sentiment Data Analysis","first-page":"1","article-title":"Chapter 1: sentiment analysis and computational intelligence","author":"Das","year":"2023","unstructured":"Das D, Kumar Kolya A, Sarkar S, Basu A. Chapter 1: sentiment analysis and computational intelligence. In: Das D, Kumar Kolya A, Basu A, Sarkar S, eds. Hybrid Computational Intelligence for Pattern Analysis and Understanding, Computational Intelligence Applications for Text and Sentiment Data Analysis. Academic Press; 2023:1-16."},{"key":"i1949-8357-18-4-477-b06","article-title":"ACGME Common Program Requirements (Residency) Section 5.1.d","unstructured":"Accreditation Council for Graduate Medical Education. ACGME Common Program Requirements (Residency) Section 5.1.d. Accessed July 14, 2025. https:\/\/www.acgme.org\/globalassets\/pfassets\/programrequirements\/2025-reformatted-requirements\/cprresidency_2025_reformatted.pdf"},{"issue":"1","key":"i1949-8357-18-4-477-b07","series-title":"BMC Med Educ","first-page":"197","article-title":"Sharing perspectives on feedback: a combined resident-faculty workshop","volume":"22","author":"Kim","year":"2022","unstructured":"Kim B, Rajagopalan A, Tabasky EM, Reddy SS, Topor DR. Sharing perspectives on feedback: a combined resident-faculty workshop. BMC Med Educ. 2022;22(1): 197. doi:10.1186\/s12909-022-03253-6"},{"key":"i1949-8357-18-4-477-b08","series-title":"arXiv","article-title":"A review of sparse expert models in deep learning","author":"Fedus","year":"2022","unstructured":"Fedus W, Dean J, Zoph B. A review of sparse expert models in deep learning. arXiv. 2022;preprint arXiv: 2209.01667. doi:10.48550\/arXiv.2209.01667"}],"container-title":["Journal of Graduate Medical Education"],"original-title":[],"link":[{"URL":"https:\/\/jgme.kglmeridian.com\/view\/journals\/jgme\/18\/4\/article-p477.xml","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/jgme.kglmeridian.com\/downloadpdf\/journals\/jgme\/18\/4\/article-p477.xml","content-type":"text\/html","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/jgme.kglmeridian.com\/downloadpdf\/journals\/jgme\/18\/4\/article-p477.xml","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T11:31:32Z","timestamp":1786707092000},"score":1,"resource":{"primary":{"URL":"https:\/\/jgme.kglmeridian.com\/view\/journals\/jgme\/18\/4\/article-p477.xml"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8]]},"references-count":8,"journal-issue":{"issue":"4"},"URL":"https:\/\/doi.org\/10.4300\/jgme-d-25-01049.1","relation":{},"ISSN":["1949-8357","1949-8349"],"issn-type":[{"value":"1949-8357","type":"electronic"},{"value":"1949-8349","type":"print"}],"subject":[],"published":{"date-parts":[[2026,8]]}}}