{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T06:32:41Z","timestamp":1791354761561,"version":"4.3.1"},"reference-count":50,"publisher":"Ubiquity Press, Ltd.","issue":"1","license":[{"start":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T00:00:00Z","timestamp":1702857600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Research Foundation Flanders (FWO)","award":["S003219N"],"award-info":[{"award-number":["S003219N"]}]},{"name":"Strategic Basic Research (SBO)","award":["S003219N"],"award-info":[{"award-number":["S003219N"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Perspect Med Educ"],"abstract":"<jats:p>Introduction: Manually analysing the quality of large amounts of written feedback comments is time-consuming and demands extensive resources and human effort. Therefore, this study aimed to explore whether a state-of-the-art large language model (LLM) could be fine-tuned to identify the presence of four literature-derived feedback quality criteria (performance, judgment, elaboration and improvement) and the seven CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional) in written feedback comments.\nMethods: A set of 2,349 labelled feedback comments of five healthcare educational programs in Flanders (Belgium) (specialistic medicine, general practice, midwifery, speech therapy and occupational therapy) was split into 12,452 sentences to create two datasets for the machine learning analysis. The Dutch BERT models BERTje and RobBERT were used to train four multiclass-multilabel classification models: two to identify the four feedback quality criteria and two to identify the seven CanMEDS roles.\nResults: The classification models trained with BERTje and RobBERT to predict the presence of the four feedback quality criteria attained macro average F1-scores of 0.73 and 0.76, respectively. The F1-score of the model predicting the presence of the CanMEDS roles trained with BERTje was 0.71 and 0.72 with RobBERT.\nDiscussion: The results showed that a state-of-the-art LLM is able to identify the presence of the four feedback quality criteria and the CanMEDS roles in written feedback comments. This implies that the quality analysis of written feedback comments can be automated using an LLM, leading to savings of time and resources.<\/jats:p>","DOI":"10.5334\/pme.1056","type":"journal-article","created":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T08:36:02Z","timestamp":1702888562000},"source":"Crossref","is-referenced-by-count":15,"title":["Automating the Identification of Feedback Quality Criteria and the CanMEDS Roles in Written Feedback Comments Using Natural Language Processing"],"prefix":"10.5334","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4435-1462","authenticated-orcid":false,"given":"Sofie","family":"Van Ostaeyen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-1070","authenticated-orcid":false,"given":"Loic","family":"De Langhe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6090-5552","authenticated-orcid":false,"given":"Orph\u00e9e","family":"De Clercq","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2915-6987","authenticated-orcid":false,"given":"Mieke","family":"Embo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3615-7347","authenticated-orcid":false,"given":"Tammy","family":"Schellens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9544-4197","authenticated-orcid":false,"given":"Martin","family":"Valcke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"3285","published-online":{"date-parts":[[2023,12,18]]},"reference":[{"issue":"7","key":"key20231218083558_B1","doi-asserted-by":"crossref","first-page":"1002","DOI":"10.1097\/ACM.0000000000002743","article-title":"A Core Components Framework for Evaluating Implementation of Competency-Based Medical Education Programs","volume":"94","year":"2019","journal-title":"Acad Med"},{"issue":"5","key":"key20231218083558_B2","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1097\/00001888-200205000-00003","article-title":"Shifting Paradigms: From Flexner to Competencies","volume":"77","year":"2002","journal-title":"Acad Med"},{"issue":"8","key":"key20231218083558_B3","doi-asserted-by":"crossref","first-page":"638","DOI":"10.3109\/0142159X.2010.501190","article-title":"Competency-based medical education: Theory to practice","volume":"32","year":"2010","journal-title":"Med Teach"},{"issue":"6","key":"key20231218083558_B4","doi-asserted-by":"crossref","first-page":"594","DOI":"10.1080\/0142159X.2017.1315077","article-title":"Implementing competency-based medical education: What changes in curricular structure and processes are needed?","volume":"39","year":"2017","journal-title":"Med Teach"},{"issue":"4","key":"key20231218083558_B5","doi-asserted-by":"crossref","first-page":"e32","DOI":"10.36834\/cmej.57323","article-title":"Assessing the quality of feedback to general internal medicine residents in a competency-based environment","volume":"10","year":"2019","journal-title":"Can Med Educ J"},{"issue":"1","key":"key20231218083558_B6","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1007\/s13187-018-1456-z","article-title":"Exploring How the New Entrustable Professional Activity Assessment Tools Affect the Quality of Feedback Given to Medical Oncology Residents","volume":"35","year":"2020","journal-title":"J Cancer Educ"},{"key":"key20231218083558_B7","doi-asserted-by":"crossref","unstructured":"Ginsburg S. Hidden in plain sight: the untapped potential of written assessment comments. PhD thesis. Maastricht University, Maastricht. 2016. DOI: 10.26481\/dis.20160901sg","DOI":"10.26481\/dis.20160901sg"},{"issue":"11","key":"key20231218083558_B8","doi-asserted-by":"crossref","first-page":"1617","DOI":"10.1097\/ACM.0000000000001669","article-title":"The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data","volume":"92","year":"2017","journal-title":"Acad Med"},{"issue":"3","key":"key20231218083558_B9","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1177\/0163278718820415","article-title":"Examining the effects of narrative commentary on evaluators\u2019 summative assessments of resident performance","volume":"43","year":"2020","journal-title":"Eval Heal Prof"},{"issue":"7","key":"key20231218083558_B10","first-page":"81","article-title":"Numbers encapsulate, words elaborate: toward the best use of comments for assessment and feedback on entrustment ratings","volume":"96","year":"2021","journal-title":"Acad Med"},{"issue":"10","key":"key20231218083558_B11","first-page":"106","article-title":"The quality of written comments on professional behaviors in a developmental multisource feedback program","volume":"85","year":"2010","journal-title":"Acad Med"},{"issue":"3","key":"key20231218083558_B12","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1111\/medu.12637","article-title":"Reading between the lines: faculty interpretations of narrative evaluation comments","volume":"49","year":"2015","journal-title":"Med Educ"},{"issue":"6","key":"key20231218083558_B13","doi-asserted-by":"crossref","first-page":"560","DOI":"10.1111\/medu.12678","article-title":"A contemporary approach to validity arguments: A practical guide to Kane\u2019s framework","volume":"49","year":"2015","journal-title":"Med Educ"},{"key":"key20231218083558_B14","first-page":"17","volume-title":"Educational design a CanMEDS guide for the health professions","year":"2011"},{"issue":"260","key":"key20231218083558_B15","first-page":"1","article-title":"An online Delphi study to investigate the completeness of the CanMEDS roles and the relevance, formulation, and measurability of their key competencies within eight healthcare disciplines in Flanders","volume":"22","year":"2022","journal-title":"BMC Med Educ"},{"issue":"6","key":"key20231218083558_B16","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1007\/S40037-015-0231-7","article-title":"Guidelines: the do\u2019s, don\u2019ts and don\u2019t knows of feedback for clinical education","volume":"4","year":"2015","journal-title":"Perspect Med Educ"},{"issue":"145","key":"key20231218083558_B17","first-page":"1","article-title":"Does what we write matter? Determining the features of high-and low-quality summative written comments of students on the internal medicine clerkship using pile-sort and consensus analysis: a mixed-methods study","volume":"16","year":"2016","journal-title":"BMC Med Educ"},{"issue":"7","key":"key20231218083558_B18","doi-asserted-by":"crossref","first-page":"973","DOI":"10.1007\/s11606-015-3237-2","article-title":"The quality of written feedback by attendings of internal medicine residents","volume":"30","year":"2015","journal-title":"J Gen Intern Med"},{"issue":"6","key":"key20231218083558_B19","doi-asserted-by":"crossref","first-page":"929","DOI":"10.1007\/s11606-019-04946-3","article-title":"Milestone implementation\u2019s impact on narrative comments and perception of feedback for internal medicine residents: a mixed methods study","volume":"34","year":"2019","journal-title":"J Gen Intern Med"},{"issue":"11","key":"key20231218083558_B20","doi-asserted-by":"crossref","first-page":"1712","DOI":"10.1097\/ACM.0000000000003315","article-title":"Resident Perceptions of Assessment and Feedback in Competency-Based Medical Education: A Focus Group Study of One Internal Medicine Residency Program","volume":"95","year":"2020","journal-title":"Acad Med"},{"issue":"6","key":"key20231218083558_B21","doi-asserted-by":"crossref","first-page":"628","DOI":"10.4103\/1735-9066.197671","article-title":"Feedback in the nonshifting context of the midwifery clinical education in Indonesia: A mixed methods study","volume":"21","year":"2016","journal-title":"Iran J Nurs Midwifery Res"},{"issue":"4","key":"key20231218083558_B22","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1097\/ACM.0000000000001927","article-title":"The feedback tango: An integrative review and analysis of the content of the teacher-learner feedback exchange","volume":"93","year":"2018","journal-title":"Acad Med"},{"issue":"3","key":"key20231218083558_B23","doi-asserted-by":"crossref","first-page":"1069","DOI":"10.1002\/nop2.717","article-title":"Challenges in the assessment of nursing students in clinical placements: Exploring perceptions among nurse mentors","volume":"8","year":"2021","journal-title":"Nurs Open"},{"issue":"8","key":"key20231218083558_B24","doi-asserted-by":"crossref","first-page":"909","DOI":"10.1080\/0142159X.2020.1763285","article-title":"Taking stock of what is known about faculty development in competency-based medical education: A scoping review paper","volume":"42","year":"2020","journal-title":"Med Teach"},{"issue":"2","key":"key20231218083558_B25","doi-asserted-by":"crossref","first-page":"545","DOI":"10.1213\/ANE.0000000000005265","article-title":"Using machine learning to evaluate attending feedback on resident performance","volume":"132","year":"2021","journal-title":"Anesth Analg"},{"issue":"278","key":"key20231218083558_B26","first-page":"1","article-title":"Feedback using an ePortfolio for medicine long cases: quality not quantity","volume":"16","year":"2016","journal-title":"BMC Med Educ"},{"issue":"1","key":"key20231218083558_B27","first-page":"1","article-title":"I did not check if the teacher gave feedback\u2019: a qualitative analysis of Taiwanese postgraduate year 1 trainees\u2019 talk around e-portfolio feedback-seeking behaviours","volume":"9","year":"2019","journal-title":"BMJ Open"},{"key":"key20231218083558_B28","unstructured":"Meriam-Webster. Artificial intelligence. https:\/\/www.merriam-webster.com\/dictionary\/artificial%20intelligence#:~:text=noun,to%20imitate%20intelligent%20human%20behavior (accessed 12 January 2023)."},{"key":"key20231218083558_B29","volume-title":"Speech and Language Processing","year":"2023","edition":"3rd ed."},{"issue":"2","key":"key20231218083558_B30","first-page":"1","article-title":"Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education","volume":"8","year":"2022","journal-title":"JMIR Med Educ"},{"issue":"10","key":"key20231218083558_B31","doi-asserted-by":"crossref","first-page":"1457","DOI":"10.1097\/ACM.0000000000004153","article-title":"Using Natural Language Processing to Automatically Assess Feedback Quality: Findings from 3 Surgical Residencies","volume":"96","year":"2021","journal-title":"Acad Med"},{"issue":"6","key":"key20231218083558_B32","doi-asserted-by":"crossref","first-page":"e72","DOI":"10.1016\/j.jsurg.2021.05.012","article-title":"Natural Language Processing and Assessment of Resident Feedback Quality","volume":"78","year":"2021","journal-title":"J Surg Educ"},{"key":"key20231218083558_B33","first-page":"47","article-title":"Attention Is All You Need","year":"2017"},{"key":"key20231218083558_B34","unstructured":"OpenAI. ChatGPT: Optimizing Language Models for Dialogue. https:\/\/openai.com\/blog\/chatgpt\/ (Accessed 16 March 2023)."},{"key":"key20231218083558_B35","journal-title":"A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles"},{"key":"key20231218083558_B36","doi-asserted-by":"crossref","unstructured":"The Royal College of Physicians and Surgeons of Canada. CanMEDS framework. https:\/\/www.royalcollege.ca\/ca\/en\/canmeds\/canmeds-framework.html (accessed 3 May 2021). DOI: 10.1177\/001316446002000104","DOI":"10.1177\/001316446002000104"},{"issue":"1","key":"key20231218083558_B37","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales. Educ","volume":"20","year":"1960","journal-title":"Psychol. Meas"},{"key":"key20231218083558_B38","volume-title":"Natural Language Processing with Transformers","year":"2022"},{"key":"key20231218083558_B39","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","year":"2019"},{"issue":"4","key":"key20231218083558_B40","doi-asserted-by":"crossref","first-page":"897","DOI":"10.3390\/psych3040056","article-title":"Automated Essay Scoring Using Transformer Models","volume":"3","year":"2021","journal-title":"Psych"},{"key":"key20231218083558_B41","article-title":"BERTje: A Dutch BERT Model","year":"2019","journal-title":"arXiv"},{"key":"key20231218083558_B42","article-title":"AlBERTo: Italian BERT language understanding model for NLP challenging tasks based on tweets","year":"2019"},{"key":"key20231218083558_B43","first-page":"3255","article-title":"RobBERT: A Dutch RoBERTa-based language model","year":"2020"},{"key":"key20231218083558_B44","first-page":"1015","article-title":"Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation","year":"2006"},{"issue":"4","key":"key20231218083558_B45","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1007\/s10459-012-9409-5","article-title":"Dissecting the doctor: From character to characteristics in North American medical education","volume":"18","year":"2013","journal-title":"Adv Health Sci Educ Theory Pract"},{"issue":"5","key":"key20231218083558_B46","doi-asserted-by":"crossref","first-page":"390","DOI":"10.3109\/0142159X.2014.890281","article-title":"Understanding, teaching and assessing the elements of the CanMEDS Professional Role: Canadian Program Directors\u2019 views","volume":"36","year":"2014","journal-title":"Med Teach"},{"issue":"12","key":"key20231218083558_B47","doi-asserted-by":"crossref","first-page":"1223","DOI":"10.1111\/medu.14911","article-title":"Predictors of faculty narrative evaluation quality in medical school clerkships","volume":"56","year":"2022","journal-title":"Med Educ"},{"key":"key20231218083558_B48","first-page":"3","article-title":"Towards automated content analysis of educational feedback: A multi-language study","year":"2022","journal-title":"Comput. Educ: AI"},{"issue":"7","key":"key20231218083558_B49","doi-asserted-by":"crossref","first-page":"1082","DOI":"10.1097\/ACM.0000000000003047","article-title":"Taken Out of Context: Hazards in the Interpretation of Written Assessment Comments","volume":"95","year":"2020","journal-title":"Acad Med"},{"issue":"1","key":"key20231218083558_B50","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/s10459-015-9622-0","article-title":"Hedging to save face: A linguistic analysis of written comments on in-training evaluation reports","volume":"21","year":"2016","journal-title":"Adv Health Sci Educ Theory Pract"}],"container-title":["Perspectives on Medical Education"],"original-title":[],"link":[{"URL":"https:\/\/account.pmejournal.org\/index.php\/up-j-pme\/article\/download\/1056\/1601","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/account.pmejournal.org\/index.php\/up-j-pme\/article\/download\/1056\/1602","content-type":"text\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/account.pmejournal.org\/index.php\/up-j-pme\/article\/download\/1056\/1603","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/account.pmejournal.org\/index.php\/up-j-pme\/article\/download\/1056\/1604","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/account.pmejournal.org\/index.php\/up-j-pme\/article\/download\/1056\/1605","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/account.pmejournal.org\/index.php\/up-j-pme\/article\/download\/1056\/1606","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/account.pmejournal.org\/index.php\/up-j-pme\/article\/download\/1056\/1601","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T10:44:05Z","timestamp":1782816245000},"score":1,"resource":{"primary":{"URL":"https:\/\/account.pmejournal.org\/index.php\/up-j-pme\/article\/view\/1056"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,18]]},"references-count":50,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1,4]]}},"URL":"https:\/\/doi.org\/10.5334\/pme.1056","relation":{},"ISSN":["2212-277X"],"issn-type":[{"value":"2212-277X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,18]]}}}