{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T07:01:43Z","timestamp":1790146903779,"version":"4.0.1"},"reference-count":30,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T00:00:00Z","timestamp":1790121600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"\n Background<\/jats:title>\n Korean Medicine clinical practice guidelines (KM-CPGs) undergo mandatory external peer review of their drafts during development, yet reviewers lack a standardized, validated instrument to guide this task. Widely used appraisal tools assess the quality of completed guidelines rather than support the section-level review of a draft. We aimed to develop an external peer review instrument for KM-CPG drafts (KM-EPRI) and to assess its content validity.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n In this two-phase methodological study, a draft instrument was first derived by directed qualitative content analysis of the external review comments produced for previously published KM-CPGs, with each comment coded into one of four predefined review modules: the clinical questions (CQ), the introductory components (CPG-I), the development methods (CPG-M), and the recommendations (CPG-R). The content validity and acceptability of the items were then assessed through a modified Delphi study with a multidisciplinary panel of KM clinicians and evidence-based medicine methodologists. Consensus was defined a priori as an item-level content validity index (I-CVI)\n \n \n \u2265<\/mml:mo>\n <\/mml:math>\n <\/jats:inline-formula>\n 0.80. The scale-level content validity index (S-CVI\/Ave), Kendall\u2019s coefficient of concordance (\n \n \n W<\/mml:mi>\n <\/mml:math>\n <\/jats:inline-formula>\n ), and the interquartile range (IQR) were reported as supporting measures.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n From 3,630 external review comments, 2,073 content-related comments were consolidated and refined into a finalized instrument of 117 review items across 4 modules and 33 subdomains. All 10 invited experts participated (response rate 100%, no attrition), and every item reached the consensus threshold in a single round under a pre-defined early-stopping rule. All 117 items had an I-CVI\n \n \n \u2265<\/mml:mo>\n <\/mml:math>\n <\/jats:inline-formula>\n 0.80, the S-CVI\/Ave was 0.976, and the IQR was\n \n \n \u2264<\/mml:mo>\n <\/mml:math>\n <\/jats:inline-formula>\n 1 for every item (median 0.75). Kendall\u2019s\n \n \n W<\/mml:mi>\n <\/mml:math>\n <\/jats:inline-formula>\n was low (0.196), reflecting a ceiling effect from uniformly high ratings rather than disagreement among panelists.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusion<\/jats:title>\n KM-EPRI can serve as a consensus-based, structured reference for the external peer review of KM-CPG drafts during development. Its effect on review quality and inter-rater reliability remains to be evaluated in practice.<\/jats:p>\n <\/jats:sec>","DOI":"10.3389\/fmed.2026.1926917","type":"journal-article","created":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T06:19:35Z","timestamp":1790144375000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["An external peer review instrument for Korean Medicine clinical practice guideline drafts (KM-EPRI): development and content validation"],"prefix":"10.3389","volume":"13","author":[{"given":"Hee Jung","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Korean Medicine, College of Korean Medicine, Daegu Haany University","place":["Gyeongsan, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoon Jae","family":"Lee","sequence":"additional","affiliation":[{"name":"Jaseng Spine and Joint Research Institute, Jaseng Medical Foundation","place":["Seoul, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyo-Weon","family":"Suh","sequence":"additional","affiliation":[{"name":"Clinical Evidence Research Team, Division of Healthcare Research, National Evidence-Based Healthcare Collaborating Agency","place":["Seoul, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keumji","family":"Kim","sequence":"additional","affiliation":[{"name":"Korean Medicine Clinical Practice Guidelines Team, National Institute for Korean Medicine Development","place":["Seoul, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jihyun","family":"Sang","sequence":"additional","affiliation":[{"name":"College of Korean Medicine, Sangji University","place":["Wonju, Gangwon, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junhyeok","family":"Yi","sequence":"additional","affiliation":[{"name":"National Agency for Korean Medicine Innovative Technologies Development, National Institute for Korean Medicine Development","place":["Seoul, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seungwon","family":"Shin","sequence":"additional","affiliation":[{"name":"College of Korean Medicine, Sangji University","place":["Wonju, Gangwon, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,23]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","DOI":"10.17226\/13058","volume-title":"Clinical Practice Guidelines We Can Trust","author":"Graham","year":"2011"},{"key":"B2","doi-asserted-by":"publisher","first-page":"525","DOI":"10.7326\/0003-4819-156-7-201204030-00009","article-title":"Guidelines International Network: toward international standards for clinical practice guidelines","volume":"156","author":"Qaseem","year":"2012","journal-title":"Ann Intern Med"},{"key":"B3","volume-title":"Developing NICE Guidelines: The Manual (PMG20)","year":"2014"},{"key":"B4","volume-title":"SIGN 50: A Guideline Developer\u2019s Handbook","year":"2025"},{"key":"B5","volume-title":"WHO Handbook for Guideline Development","year":"2014"},{"key":"B6","volume-title":"Procedures and Requirements for Meeting the NHMRC Standards for Clinical Practice Guidelines","year":"2025"},{"key":"B7","volume-title":"U.S. Preventive Services Task Force Procedure Manual","year":"2023"},{"key":"B8","doi-asserted-by":"publisher","first-page":"100924","DOI":"10.1016\/j.imr.2023.100924","article-title":"Development of clinical practice guidelines for Korean medicine: towards evidence-based complementary and alternative medicine","volume":"12","author":"Shin","year":"2023","journal-title":"Integr Med Res"},{"key":"B9","volume-title":"NECA\u2019s Handbook for Clinical Practice Guideline Developer","author":"Kim","year":"2015"},{"key":"B10","volume-title":"Handbook for Clinical Practice Guideline Developer","author":"Kim","year":"2022"},{"key":"B11","volume-title":"Korean Medicine Clinical Practice Guideline Development Manual","author":"Kim","year":"2022"},{"key":"B12","volume-title":"Clinical Practice Guidelines DB","year":"2026"},{"key":"B13","doi-asserted-by":"publisher","first-page":"E839","DOI":"10.1503\/cmaj.090449","article-title":"AGREE II: advancing guideline development, reporting and evaluation in health care","volume":"182","author":"Brouwers","year":"2010","journal-title":"Can Med Assoc J"},{"key":"B14","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1186\/1472-6963-5-18","article-title":"Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines","volume":"5","author":"MacDermid","year":"2005","journal-title":"BMC Health Serv Res"},{"key":"B15","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1007\/s11606-007-0440-9","article-title":"Rigorous development does not ensure that guidelines are acceptable to a panel of knowledgeable providers","volume":"23","author":"Nuckols","year":"2008","journal-title":"J Gen Intern Med"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1277","DOI":"10.1177\/1049732305276687","article-title":"Three approaches to qualitative content analysis","volume":"15","author":"Hsieh","year":"2005","journal-title":"Qual Health Res"},{"key":"B17","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1111\/j.1365-2648.2007.04569.x","article-title":"The qualitative content analysis process","volume":"62","author":"Elo","year":"2008","journal-title":"J Adv Nurs"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1008","DOI":"10.1046\/j.1365-2648.2000.01567.x","article-title":"Research guidelines for the Delphi survey technique","volume":"32","author":"Hasson","year":"2000","journal-title":"J Adv Nurs"},{"key":"B19","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1111\/j.1365-2648.2006.03716.x","article-title":"Consulting the oracle: ten lessons from using the Delphi technique in nursing research","volume":"53","author":"Keeney","year":"2006","journal-title":"J Adv Nurs"},{"key":"B20","doi-asserted-by":"publisher","first-page":"e0304651","DOI":"10.1371\/journal.pone.0304651","article-title":"Delphi studies in social, health sciences\u2014recommendations for an interdisciplinary standardized reporting (DELPHISTAR): results of a Delphi study","volume":"19","author":"Niederberger","year":"2024","journal-title":"PLoS One"},{"key":"B21","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.im.2003.11.002","article-title":"The Delphi method as a research tool: an example, design considerations and applications","volume":"42","author":"Okoli","year":"2004","journal-title":"Inf Manage"},{"key":"B22","volume-title":"The RAND\/UCLA Appropriateness Method User\u2019s Manual","author":"Fitch","year":"2001"},{"key":"B23","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1016\/j.jclinepi.2013.12.002","article-title":"Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies","volume":"67","author":"Diamond","year":"2014","journal-title":"J Clin Epidemiol"},{"key":"B24","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1097\/00006199-198611000-00017","article-title":"Determination and quantification of content validity","volume":"35","author":"Lynn","year":"1986","journal-title":"Nurs Res"},{"key":"B25","doi-asserted-by":"publisher","first-page":"489","DOI":"10.1002\/nur.20147","article-title":"The content validity index: are you sure you know what\u2019s being reported? Critique and recommendations","volume":"29","author":"Polit","year":"2006","journal-title":"Res Nurs Health"},{"key":"B26","doi-asserted-by":"publisher","first-page":"459","DOI":"10.1002\/nur.20199","article-title":"Is the CVI an acceptable indicator of content validity? Appraisal and recommendations","volume":"30","author":"Polit","year":"2007","journal-title":"Res Nurs Health"},{"key":"B27","doi-asserted-by":"publisher","first-page":"763","DOI":"10.1111\/j.1540-5915.1997.tb01330.x","article-title":"Managing Delphi surveys using nonparametric statistical techniques","volume":"28","author":"Schmidt","year":"1997","journal-title":"Decis Sci"},{"key":"B28","doi-asserted-by":"publisher","first-page":"1525","DOI":"10.1016\/j.techfore.2012.04.013","article-title":"Consensus measurement in Delphi studies: review and implications for future quality assurance","volume":"79","author":"von der Gracht","year":"2012","journal-title":"Technol Forecast Soc Change"},{"key":"B29","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1198\/108571105X46642","article-title":"Species associations: the Kendall coefficient of concordance revisited","volume":"10","author":"Legendre","year":"2005","journal-title":"J Agric Biol Environ Stat"},{"key":"B30","volume-title":"OH (CCO) PEBC Handbook","year":"2020"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1926917\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T06:19:38Z","timestamp":1790144378000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1926917\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,23]]},"references-count":30,"alternative-id":["10.3389\/fmed.2026.1926917"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1926917","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,23]]},"article-number":"1926917"}}