{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T19:54:35Z","timestamp":1791230075588,"version":"4.1.0"},"reference-count":43,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Competency-based medical education increasingly uses digital workplace assessment platforms to collect longitudinal evidence of trainees\u2019 readiness for progressive responsibility. Entrustable professional activities (EPAs) translate competencies into observable clinical work, but individual workplace observations are context-dependent and may not support dependable progression decisions. Empirical guidance remains limited on how many observations, faculty raters, EPA titles, and workplace settings are needed for a dependable EPA portfolio.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to estimate EPA portfolio dependability for competency-based progression review using nationwide data from a digital workplace assessment platform in otorhinolaryngology-head and neck surgery residency training.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We conducted a retrospective, nationwide cross-sectional generalizability and decision study using routinely collected EPA-based workplace assessment data from Taiwan\u2019s E-MyWay platform. The study included completed EPA observations from accredited otorhinolaryngology-head and neck surgery residency programs between August 2022 and May 2026. Faculty-assigned entrustment-supervision ratings were scored on a 5-level national EPA scale. The resident was the object of measurement, whereas faculty rater, EPA title, and workplace setting were measurement facets. The generalizability study partitioned rating variance into resident, facet, interaction, and residual components, which were subsequently used in decision-study projections of the generalizability coefficient, Phi coefficient, and absolute standard error of measurement under alternative portfolio configurations.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The analytic sample included 45,526 EPA observations involving 466 residents and 448 faculty raters across 35 training programs, 11 EPA titles, and 5 workplace settings. Residents had a median of 87 (IQR 47-124) observations and were assessed by a median of 9 (IQR 6-12) faculty raters across 11 (IQR 10-11) EPA titles and 5 (IQR 4-5) workplace settings. Single EPA observations were insufficient for resident-level summative interpretation. In the primary generalizability study, residual\/encounter-level variance was the largest component, accounting for 40.1% of total variance; resident variance accounted for 26.9%, faculty rater variance accounted for 10.8%, and resident \u00d7 faculty rater interaction for 10.4%. A study-defined well-distributed 10-observation portfolio achieved a generalizability coefficient of 0.77 but an absolute dependability coefficient (Phi) of 0.65. A 20-observation portfolio across 5 faculty raters, 5 EPA titles, and 3 workplace settings yielded a Phi of 0.76, and a 30-observation portfolio across 6 faculty raters, 6 EPA titles, and 4 workplace settings yielded a Phi of 0.80.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Nationwide digital workplace assessment data showed that EPA portfolio dependability reflected both observation counts and sampling breadth across faculty raters, EPA titles, and workplace settings. These study-defined portfolio configurations may help clinical competency committees, residency programs, and specialty societies monitor portfolio completeness before progression deliberation. They should complement, rather than replace, narrative feedback, case complexity, performance trajectory, and committee judgment.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/104628","type":"journal-article","created":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T19:05:06Z","timestamp":1787771106000},"page":"e104628-e104628","source":"Crossref","is-referenced-by-count":0,"title":["Dependability of Entrustable Professional Activity Portfolios in Otorhinolaryngology Residency: Nationwide Generalizability and Decision Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3635-4815","authenticated-orcid":false,"given":"Jeng-Wen","family":"Chen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9468-9969","authenticated-orcid":false,"given":"Ching-Lin","family":"Hsieh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4344-4766","authenticated-orcid":false,"given":"Chun-Hsiang","family":"Chang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8118-3547","authenticated-orcid":false,"given":"Ya-Hui","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8583-8459","authenticated-orcid":false,"given":"Wei-Chung","family":"Hsu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6288-9218","authenticated-orcid":false,"given":"Pa-Chun","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0837-1597","authenticated-orcid":false,"given":"Li-Jen","family":"Liao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9414-8660","authenticated-orcid":false,"given":"Chun-Hou","family":"Liao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8278-0033","authenticated-orcid":false,"given":"Mingchih","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4071-3895","authenticated-orcid":false,"given":"Okki Dhona","family":"Laksmita","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,5]]},"reference":[{"issue":"4","key":"R1","doi-asserted-by":"publisher","DOI":"10.2196\/15655","article-title":"Developing a competency-based learning and assessment system for residency training: analysis study of user requirements and acceptance","volume":"22","author":"Hsiao","journal-title":"J Med Internet Res"},{"issue":"11","key":"R2","doi-asserted-by":"publisher","first-page":"1261","DOI":"10.1097\/ACM.0000000000005305","article-title":"The assessment burden in competency-based medical education: how programs are adapting","volume":"98","author":"Szulewski","journal-title":"Acad Med"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/87605","article-title":"Medical students\u2019 acceptance of digital entrustable professional activities: results of a cohort study","volume":"12","author":"Domann","journal-title":"JMIR Med Educ"},{"issue":"12","key":"R4","doi-asserted-by":"publisher","first-page":"1178","DOI":"10.1097\/00001888-200012000-00009","article-title":"Competency-based residency training: the next advance in graduate medical education","volume":"75","author":"Long","journal-title":"Acad Med"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"612","DOI":"10.1186\/s12909-024-05609-6","article-title":"Evaluating competency-based medical education: a systematized review of current practices","volume":"24","author":"Alharbi","journal-title":"BMC Med Educ"},{"issue":"6","key":"R6","doi-asserted-by":"publisher","first-page":"542","DOI":"10.1097\/ACM.0b013e31805559c7","article-title":"Competency-based postgraduate training: can we bridge the gap between theory and clinical practice?","volume":"82","author":"ten Cate","journal-title":"Acad Med"},{"issue":"10","key":"R7","doi-asserted-by":"publisher","first-page":"1106","DOI":"10.1080\/0142159X.2020.1838465","article-title":"The recommended description of an entrustable professional activity: AMEE Guide No. 140","volume":"43","author":"Ten Cate","journal-title":"Med Teach"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/54112","article-title":"Transforming the future of digital health education: redesign of a graduate program using competency mapping","volume":"10","author":"Mun","journal-title":"JMIR Med Educ"},{"issue":"6","key":"R9","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1111\/medu.14072","article-title":"Beyond summative decision making: illuminating the broader roles of competence committees","volume":"54","author":"Pack","journal-title":"Med Educ"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/82326","article-title":"Quantifying emergency medicine residency learning curves using natural language processing: retrospective cohort study","volume":"11","author":"Preiksaitis","journal-title":"JMIR Med Educ"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Nel D Marty AP Frick S Hennus MP Linsenmeyer M . Addressing practical and conceptual challenges in workplace-based assessment. In: ten Cate O Burch VC Chen HC Chou FC Hennus MP , editors. Entrustable Professional Activities and Entrustment Decision-Making in Health Professions Education. 1st ed. Ubiquity Press; 2024:237-247. [doi: 10.5334\/bdc.t]","DOI":"10.5334\/bdc.t"},{"issue":"7","key":"R12","doi-asserted-by":"publisher","first-page":"569","DOI":"10.3109\/0142159X.2013.798403","article-title":"What is wrong with assessment in postgraduate training? Lessons from clinical practice and educational research","volume":"35","author":"Driessen","journal-title":"Med Teach"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"134","DOI":"10.1080\/0142159X.2024.2323177","article-title":"In EPAs we trust, is quality and safety a must? A cross-specialty analysis of entrustable professional activity guides","volume":"47","author":"Brown","journal-title":"Med Teach"},{"issue":"7","key":"R14","doi-asserted-by":"publisher","first-page":"825","DOI":"10.1111\/medu.15291","article-title":"How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study","volume":"58","author":"Ryan","journal-title":"Med Educ"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1177\/01945998211033235","article-title":"Milestones 2.0: otolaryngology resident competency in the postpandemic era","volume":"166","author":"Cabrera-Muffly","journal-title":"Otolaryngol Head Neck Surg"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1177\/01945998221128292","article-title":"A needs assessment for the future of otolaryngology education","volume":"169","author":"Chen","journal-title":"Otolaryngol Head Neck Surg"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","first-page":"1135","DOI":"10.1186\/s12909-025-07679-6","article-title":"Factors associated with attainment of ad hoc entrustability among Taiwan otolaryngology resident physicians: a nationwide cross-sectional study","volume":"25","author":"Lin","journal-title":"BMC Med Educ"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.2196\/81718","article-title":"Automated evaluation of reflection and feedback quality in workplace-based assessments by using natural language processing: cross-sectional competency-based medical education study","volume":"11","author":"Chen","journal-title":"JMIR Med Educ"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.2196\/95066","article-title":"Learning analytics of a national entrustable professional activities platform: cross-sectional study of system-level constraints on advanced entrustment in competency-based medical education","volume":"12","author":"Chen","journal-title":"JMIR Med Educ"},{"issue":"4","key":"R20","doi-asserted-by":"publisher","first-page":"1242","DOI":"10.1002\/ohn.1104","article-title":"EMYWAY workplace-based entrustable professional activities assessments in otolaryngology residency training: a nationwide experience","volume":"172","author":"Guo","journal-title":"Otolaryngol Head Neck Surg"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/41671","article-title":"Empowering health care education through learning analytics: in-depth scoping review","volume":"25","author":"Bojic","journal-title":"J Med Internet Res"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1016\/j.nedt.2022.105409","article-title":"Implementing an entrustable professional activities programmatic assessments for nurse practitioner training in emergency care: a pilot study","volume":"115","author":"Chiang","journal-title":"Nurse Educ Today"},{"issue":"4","key":"R23","doi-asserted-by":"publisher","first-page":"616","DOI":"10.1097\/ACM.0000000000002997","article-title":"A reliability analysis of entrustment-derived workplace-based assessments","volume":"95","author":"Kelleher","journal-title":"Acad Med"},{"issue":"11","key":"R24","doi-asserted-by":"publisher","first-page":"2684","DOI":"10.1007\/s11606-021-07151-3","article-title":"WBAs in UME-how many are needed? A reliability analysis of 5 AAMC core EPAs implemented in the internal medicine clerkship","volume":"37","author":"Dunne","journal-title":"J Gen Intern Med"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/j.jsp.2013.11.008","article-title":"Generalizability theory: a practical guide to study design, implementation, and interpretation","volume":"52","author":"Briesch","journal-title":"J Sch Psychol"},{"issue":"7S","key":"R26","doi-asserted-by":"publisher","first-page":"S64","DOI":"10.1097\/ACM.0000000000004112","article-title":"Constructing a validity map for a workplace-based assessment system: cross-walking Messick and Kane","volume":"96","author":"Kinnear","journal-title":"Acad Med"},{"issue":"2","key":"R27","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1097\/ACM.0000000000005719","article-title":"Psychometric properties of entrustable professional activity-based objective structured clinical examinations during transition from undergraduate to graduate medical education: a generalizability study","volume":"100","author":"Suneja","journal-title":"Acad Med"},{"issue":"6","key":"R28","doi-asserted-by":"publisher","first-page":"78","DOI":"10.36834\/cmej.75876","article-title":"The quality of assessment for learning score for evaluating written feedback in anesthesiology postgraduate medical education: a generalizability and decision study","volume":"14","author":"Choo","journal-title":"Can Med Educ J"},{"issue":"6","key":"R29","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1097\/SIH.0000000000000667","article-title":"Preparing and presenting validation studies: a guide for the perplexed","volume":"17","author":"Calhoun","journal-title":"Simul Healthc"},{"issue":"8","key":"R30","doi-asserted-by":"publisher","first-page":"1197","DOI":"10.1097\/ACM.0000000000003921","article-title":"Generalizability of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale to assess medical student performance on core EPAs in the workplace: findings from one institution","volume":"96","author":"Ryan","journal-title":"Acad Med"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1186\/s12909-018-1294-z","article-title":"Development, administration, and validity evidence of a subspecialty preparatory test toward licensure: a pilot study","volume":"18","author":"Johnson","journal-title":"BMC Med Educ"},{"issue":"11","key":"R32","doi-asserted-by":"publisher","first-page":"1709","DOI":"10.1016\/j.jsurg.2024.08.008","article-title":"Evaluating the validity of national multiassessment system in postgraduate surgical training: a retrospective cohort study","volume":"81","author":"O\u2019Keeffe","journal-title":"J Surg Educ"},{"issue":"10","key":"R33","doi-asserted-by":"publisher","first-page":"2296","DOI":"10.1002\/lary.27085","article-title":"Validity evidence as a key marker of quality of technical skill assessment in OTL-HNS","volume":"128","author":"Labb\u00e9","journal-title":"Laryngoscope"},{"issue":"6","key":"R34","doi-asserted-by":"publisher","first-page":"560","DOI":"10.1111\/medu.12678","article-title":"A contemporary approach to validity arguments: a practical guide to Kane\u2019s framework","volume":"49","author":"Cook","journal-title":"Med Educ"},{"issue":"2","key":"R35","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1097\/ACM.0000000000002469","article-title":"Key factors in clinical competency committee members\u2019 decisions regarding residents\u2019 readiness to serve as supervisors: a national study","volume":"94","author":"Schumacher","journal-title":"Acad Med"},{"issue":"1","key":"R36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s40037-018-0490-1","article-title":"Design and evaluation of a clinical competency committee","volume":"8","author":"Duitsman","journal-title":"Perspect Med Educ"},{"issue":"111922","key":"R37","doi-asserted-by":"publisher","first-page":"111922","DOI":"10.1016\/j.jclinane.2025.111922","article-title":"Reliability and utility of anesthesiology entrustable professional activities assessed with a mobile web application","volume":"106","author":"Tanaka","journal-title":"J Clin Anesth"},{"issue":"11","key":"R38","doi-asserted-by":"publisher","first-page":"1712","DOI":"10.1097\/ACM.0000000000003315","article-title":"Resident perceptions of assessment and feedback in competency-based medical education: a focus group study of one internal medicine residency program","volume":"95","author":"Branfield Day","journal-title":"Acad Med"},{"issue":"5","key":"R39","doi-asserted-by":"publisher","first-page":"794","DOI":"10.1097\/ACM.0000000000001902","article-title":"Assessment of technical skills competence in the operating room: a systematic and scoping review","volume":"93","author":"Fahim","journal-title":"Acad Med"},{"issue":"7S","key":"R40","doi-asserted-by":"publisher","first-page":"S70","DOI":"10.1097\/ACM.0000000000004090","article-title":"Validity evidence for assessing entrustable professional activities during undergraduate medical education","volume":"96","author":"Violato","journal-title":"Acad Med"},{"issue":"5","key":"R41","doi-asserted-by":"publisher","first-page":"585","DOI":"10.1097\/ACM.0000000000005907","article-title":"Implementing core entrustable professional activities in undergraduate medical education: a psychometric study","volume":"100","author":"Violato","journal-title":"Acad Med"},{"issue":"1","key":"R42","doi-asserted-by":"publisher","DOI":"10.1111\/tct.13668","article-title":"Do summative entrustment decisions actually lead to entrustment?","volume":"21","author":"Sigurdsson","journal-title":"Clin Teach"},{"issue":"10","key":"R43","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1111\/medu.14287","article-title":"Translating outcome frameworks to assessment programmes: implications for validity","volume":"54","author":"Tavares","journal-title":"Med Educ"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T19:24:10Z","timestamp":1791228250000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e104628"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,5]]},"references-count":43,"URL":"https:\/\/doi.org\/10.2196\/104628","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,5]]},"article-number":"v12i4e104628"}}