{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,3]],"date-time":"2025-07-03T00:45:39Z","timestamp":1751503539930},"reference-count":42,"publisher":"Georg Thieme Verlag KG","issue":"05","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\n          Objectives\u2003Clinical Competency Committee (CCC) members employ varied approaches to the review process. This makes the design of a competency assessment dashboard that fits the needs of all members difficult. This work details a user-centered evaluation of a dashboard currently utilized by the Internal Medicine Clinical Competency Committee (IM CCC) at the University of Cincinnati College of Medicine and generated design recommendations.<\/jats:p><jats:p>\n          Methods\u2003Eleven members of the IM CCC participated in semistructured interviews with the research team. These interviews were recorded and transcribed for analysis. The three design research methods used in this study included process mapping (workflow diagrams), affinity diagramming, and a ranking experiment.<\/jats:p><jats:p>\n          Results\u2003Through affinity diagramming, the research team identified and organized opportunities for improvement about the current system expressed by study participants. These areas include a time-consuming preprocessing step, lack of integration of data from multiple sources, and different workflows for each step in the review process. Finally, the research team categorized nine dashboard components based on rankings provided by the participants.<\/jats:p><jats:p>\n          Conclusion\u2003We successfully conducted user-centered evaluation of an IM CCC dashboard and generated four recommendations. Programs should integrate quantitative and qualitative feedback, create multiple views to display these data based on user roles, work with designers to create a usable, interpretable dashboard, and develop a strong informatics pipeline to manage the system. To our knowledge, this type of user-centered evaluation has rarely been attempted in the medical education domain. Therefore, this study provides best practices for other residency programs to evaluate current competency assessment tools and to develop new ones.<\/jats:p>","DOI":"10.1055\/s-0043-1777103","type":"journal-article","created":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T00:06:53Z","timestamp":1703117213000},"page":"996-1007","source":"Crossref","is-referenced-by-count":3,"title":["User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard"],"prefix":"10.1055","volume":"14","author":[{"given":"Scott","family":"Vennemeyer","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States"}]},{"given":"Benjamin","family":"Kinnear","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, College of Medicine, University of Cincinnati, Ohio, United States"},{"name":"Department of Internal Medicine, College of Medicine, University of Cincinnati, Ohio, United States"}]},{"given":"Andy","family":"Gao","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States"},{"name":"Medical Sciences Baccalaureate Program, College of Medicine, University of Cincinnati, Ohio, United States"}]},{"given":"Siyi","family":"Zhu","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States"},{"name":"School of Design, College of Design, Architecture, Art, and Planning (DAAP), University of Cincinnati, Ohio, United States"}]},{"given":"Anunita","family":"Nattam","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States"},{"name":"Medical Sciences Baccalaureate Program, College of Medicine, University of Cincinnati, Ohio, United States"}]},{"given":"Michelle I.","family":"Knopp","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine, College of Medicine, University of Cincinnati, Ohio, United States"},{"name":"Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Department of Pediatrics, College of Medicine, University of Cincinnati, Ohio, United States"}]},{"given":"Eric","family":"Warm","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine, College of Medicine, University of Cincinnati, Ohio, United States"}]},{"given":"Danny T.Y.","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States"},{"name":"Department of Pediatrics, College of Medicine, University of Cincinnati, Ohio, United States"},{"name":"Medical Sciences Baccalaureate Program, College of Medicine, University of Cincinnati, Ohio, United States"},{"name":"School of Design, College of Design, Architecture, Art, and Planning (DAAP), University of Cincinnati, Ohio, United States"}]}],"member":"194","published-online":{"date-parts":[[2023,12,20]]},"reference":[{"issue":"03","key":"ref3","doi-asserted-by":"crossref","first-page":"205","DOI":"10.3109\/0142159X.2012.652239","article-title":"A model for programmatic assessment fit for purpose","volume":"34","author":"C PM van der Vleuten","year":"2012","journal-title":"Med Teach"},{"issue":"03","key":"ref4","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1097\/ACM.0000000000001995","article-title":"Translating theory into practice: implementing a program of assessment","volume":"93","author":"K E Hauer","year":"2018","journal-title":"Acad Med"},{"issue":"01","key":"ref5","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1007\/S40037-020-00625-W","article-title":"Where the rubber meets the road - An integrative review of programmatic assessment in health care professions education","volume":"10","author":"S Schut","year":"2021","journal-title":"Perspect Med Educ"},{"issue":"07","key":"ref6","doi-asserted-by":"crossref","first-page":"723","DOI":"10.1111\/medu.13884","article-title":"Some assembly required: tracing the interpretative work of Clinical Competency Committees","volume":"53","author":"R Pack","year":"2019","journal-title":"Med Educ"},{"issue":"05","key":"ref7","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1080\/0142159X.2017.1299924","article-title":"Competency milestones for medical students: design, implementation, and analysis at one medical school","volume":"39","author":"K D Lomis","year":"2017","journal-title":"Med Teach"},{"issue":"01","key":"ref8","doi-asserted-by":"crossref","first-page":"68","DOI":"10.3109\/0142159X.2013.849801","article-title":"Automatic scoring of medical students' clinical notes to monitor learning in the workplace","volume":"36","author":"A Spickard III","year":"2014","journal-title":"Med Teach"},{"issue":"04","key":"ref9","doi-asserted-by":"crossref","first-page":"119","DOI":"10.4236\/ib.2013.54015","article-title":"The conceptual model of influencing factors and influencing mechanism on team decision-making quality mediated by information sharing","volume":"05","author":"P Dai","year":"2013","journal-title":"IB"},{"issue":"04","key":"ref10","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1177\/1046496496274003","article-title":"Information exchange and use in small group decision making","volume":"27","author":"A R Dennis","year":"1996","journal-title":"Small Group Res"},{"issue":"11","key":"ref11","doi-asserted-by":"crossref","first-page":"1110","DOI":"10.1080\/0142159X.2018.1474191","article-title":"Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education","volume":"40","author":"B Kinnear","year":"2018","journal-title":"Med Teach"},{"key":"ref12","doi-asserted-by":"crossref","first-page":"29838","DOI":"10.3402\/meo.v21.29838","article-title":"Resident dashboards: helping your clinical competency committee visualize trainees' key performance indicators","volume":"21","author":"K A Friedman","year":"2016","journal-title":"Med Educ Online"},{"issue":"08","key":"ref13","doi-asserted-by":"crossref","first-page":"855","DOI":"10.1080\/0142159X.2017.1396306","article-title":"Twelve tips to promote successful development of a learner performance dashboard within a medical education program","volume":"40","author":"C Boscardin","year":"2018","journal-title":"Med Teach"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/978-981-15-5069-0_2","volume-title":"Overview of data visualization","author":"Q Li","year":"2020"},{"issue":"01","key":"ref15","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1007\/s10278-017-0015-1","article-title":"Minimizing barriers in learning for on-call radiology residents-end-to-end web-based resident feedback system","volume":"31","author":"H H Choi","year":"2018","journal-title":"J Digit Imaging"},{"issue":"03","key":"ref16","doi-asserted-by":"crossref","first-page":"361","DOI":"10.4300\/JGME-D-16-00469.1","article-title":"Automated reporting of trainee metrics using electronic clinical systems","volume":"9","author":"J C Levin","year":"2017","journal-title":"J Grad Med Educ"},{"issue":"01","key":"ref17","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1097\/ALN.0000000000000071","article-title":"Automated near-real-time clinical performance feedback for anesthesiology residents: one piece of the milestones puzzle","volume":"120","author":"J M Ehrenfeld","year":"2014","journal-title":"Anesthesiology"},{"issue":"04","key":"ref18","doi-asserted-by":"crossref","first-page":"61","DOI":"10.7812\/TPP\/15-041","article-title":"Navigating the next accreditation system: a dashboard for the milestones","volume":"19","author":"S Johna","year":"2015","journal-title":"Perm J"},{"issue":"02","key":"ref19","doi-asserted-by":"crossref","first-page":"673","DOI":"10.1097\/01.prs.0000475797.69478.0e","article-title":"Comprehensive observations of resident evolution: a novel method for assessing procedure-based residency training","volume":"137","author":"C M Cooney","year":"2016","journal-title":"Plast Reconstr Surg"},{"issue":"05","key":"ref20","doi-asserted-by":"crossref","first-page":"1443","DOI":"10.1148\/rg.2018170117","article-title":"Radiology resident assessment and feedback dashboard","volume":"38","author":"A B Durojaiye","year":"2018","journal-title":"Radiographics"},{"issue":"01","key":"ref21","first-page":"e16","article-title":"Developing a dashboard to meet Competence Committee needs: a design-based research project","volume":"11","author":"B Thoma","year":"2020","journal-title":"Can Med Educ J"},{"issue":"08","key":"ref22","doi-asserted-by":"crossref","first-page":"1177","DOI":"10.1007\/s11606-014-2801-5","article-title":"Entrustment and mapping of observable practice activities for resident assessment","volume":"29","author":"E J Warm","year":"2014","journal-title":"J Gen Intern Med"},{"issue":"10","key":"ref23","doi-asserted-by":"crossref","first-page":"1398","DOI":"10.1097\/ACM.0000000000001292","article-title":"Entrusting observable practice activities and milestones over the 36 months of an internal medicine residency","volume":"91","author":"E J Warm","year":"2016","journal-title":"Acad Med"},{"key":"ref24","first-page":"286","volume-title":"Assessment through entrustable professional activities","author":"H C Chen","year":"2018"},{"issue":"02","key":"ref25","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1080\/10401334.2019.1653764","article-title":"Linking workplace-based assessment to ACGME milestones: a comparison of mapping strategies in two specialties","volume":"32","author":"M Kelleher","year":"2020","journal-title":"Teach Learn Med"},{"issue":"14","key":"ref26","doi-asserted-by":"crossref","first-page":"3670","DOI":"10.1007\/s11606-022-07492-7","article-title":"Developing the expected entrustment score: accounting for variation in resident assessment","volume":"37","author":"D P Schauer","year":"2022","journal-title":"J Gen Intern Med"},{"issue":"05","key":"ref27","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1055\/s-0039-1698466","article-title":"Usability testing of an interactive dashboard for surgical quality improvement in a large congenital heart center","volume":"10","author":"D TY Wu","year":"2019","journal-title":"Appl Clin Inform"},{"issue":"S 02","key":"ref28","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/s11577-017-0454-1","article-title":"How to construct a mixed methods research design","volume":"69","author":"J Schoonenboom","year":"2017","journal-title":"Kolner Z Soz Sozialpsychol (Aufl)"},{"key":"ref29","first-page":"96","volume-title":"Affinity diagram","author":"N R Tague","year":"2005","edition":"2nd ed."},{"issue":"01","key":"ref32","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1093\/bmb\/lds020","article-title":"Ordinal preference elicitation methods in health economics and health services research: using discrete choice experiments and ranking methods","volume":"103","author":"S Ali","year":"2012","journal-title":"Br Med Bull"},{"issue":"03","key":"ref33","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1136\/jamia.2010.004333","article-title":"Traversing the many paths of workflow research: developing a conceptual framework of workflow terminology through a systematic literature review","volume":"17","author":"K M Unertl","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"04","key":"ref35","doi-asserted-by":"crossref","first-page":"665","DOI":"10.1017\/S1049096502001129","article-title":"Asking questions: techniques for semistructured interviews","volume":"35","author":"B L Leech","year":"2002","journal-title":"PS Polit Sci Polit"},{"issue":"12","key":"ref37","doi-asserted-by":"crossref","first-page":"692","DOI":"10.1007\/BF02599012","article-title":"Performance on the internal medicine second-year residency in-training examination predicts the outcome of the ABIM certifying examination","volume":"9","author":"H Waxman","year":"1994","journal-title":"J Gen Intern Med"},{"key":"ref38","first-page":"c1988","volume-title":"MKSAP: Medical Knowledge Self-Assessment Program VIII","author":"American College of Physicians","year":"1988"},{"issue":"07","key":"ref39","doi-asserted-by":"crossref","first-page":"921","DOI":"10.1007\/s11606-008-0588-y","article-title":"The ambulatory long-block: an accreditation council for graduate medical education (ACGME) educational innovations project (EIP)","volume":"23","author":"E J Warm","year":"2008","journal-title":"J Gen Intern Med"},{"issue":"06","key":"ref40","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1007\/S40037-021-00681-W","article-title":"Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program","volume":"10","author":"M Kelleher","year":"2021","journal-title":"Perspect Med Educ"},{"issue":"11","key":"ref41","doi-asserted-by":"crossref","first-page":"1617","DOI":"10.1097\/ACM.0000000000001669","article-title":"The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data","volume":"92","author":"S Ginsburg","year":"2017","journal-title":"Acad Med"},{"key":"ref42","doi-asserted-by":"crossref","first-page":"668","DOI":"10.3389\/fpsyg.2013.00668","article-title":"Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States","volume":"4","author":"J L Hanson","year":"2013","journal-title":"Front Psychol"},{"issue":"11","key":"ref43","doi-asserted-by":"crossref","first-page":"1154","DOI":"10.1080\/0142159X.2017.1368467","article-title":"Numerical versus narrative: a comparison between methods to measure medical student performance during clinical clerkships","volume":"39","author":"J Bartels","year":"2017","journal-title":"Med Teach"},{"issue":"07","key":"ref44","doi-asserted-by":"crossref","first-page":"641","DOI":"10.3109\/0142159X.2014.973388","article-title":"Twelve Tips for programmatic assessment","volume":"37","author":"C PM Van Der Vleuten","year":"2015","journal-title":"Med Teach"},{"issue":"11","key":"ref45","doi-asserted-by":"crossref","first-page":"1253","DOI":"10.1007\/s11606-011-1696-7","article-title":"A multiple choice testing program coupled with a year-long elective experience is associated with improved performance on the internal medicine in-training examination","volume":"26","author":"B R Mathis","year":"2011","journal-title":"J Gen Intern Med"},{"issue":"02","key":"ref46","doi-asserted-by":"crossref","first-page":"269","DOI":"10.4300\/JGME-D-09-00102.1","article-title":"Multisource feedback in the ambulatory setting","volume":"2","author":"E J Warm","year":"2010","journal-title":"J Grad Med Educ"},{"issue":"10","key":"ref47","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1097\/ACM.0000000000000424","article-title":"Connecting resident education to patient outcomes: the evolution of a quality improvement curriculum in an internal medicine residency","volume":"89","author":"M A Zafar","year":"2014","journal-title":"Acad Med"},{"key":"ref48","volume-title":"Flask Web Development","author":"M Grinberg","year":"2014","edition":"1st ed."}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0043-1777103.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T00:07:10Z","timestamp":1703117230000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0043-1777103"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10]]},"references-count":42,"journal-issue":{"issue":"05","published-online":{"date-parts":[[2023,10,18]]},"published-print":{"date-parts":[[2023,10]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0043-1777103","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10]]}}}