{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T05:50:00Z","timestamp":1791352200990,"version":"4.3.1"},"reference-count":29,"publisher":"Wiley","license":[{"start":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T00:00:00Z","timestamp":1791244800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"},{"start":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T00:00:00Z","timestamp":1791244800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["J Evidence Based Medicine"],"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Aim<\/jats:title>\n                    <jats:p>The proliferation of clinical practice guidelines presents significant challenges for ensuring evidence\u2010based care, since evaluating their trustworthiness with the Reporting Items for practice Guidelines in HealThcare (RIGHT) criteria is both time\u2010consuming and resource\u2010intensive. To address this, the Artificial intelligence\u2010empowered DeVelopment and AssessmeNt of healthCare guidElines and stanDards (ADVANCED) working group developed ad^Writer, an artificial intelligence system, designed to automate guideline reporting quality assessment. This study aimed to determine the agreement and reliability of ad^Writer compared with human evaluations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Guidelines were identified from four published systematic reviews, and through direct communication with the original authors, 60 were ultimately included. These guidelines were evaluated using ad^Writer with two different large language models, each performing three independent assessments. A majority rule was applied to synthesize results. Results from systematic reviews, which served as the reference standard. Agreement, consistency, and stability of the artificial intelligence\u2010based assessments were examined using Bland\u2013Altman plots, accuracy measures, and intraclass correlation coefficient (ICC), with a focus on identifying patterns of performance across individual RIGHT items.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Both models showed good agreement with human but had wide limits of agreement. DeepSeek performed better (mean difference = \u22121.40, standard deviation = 5.00, 95% limits of agreement ([\u221211.22, 8.42]) and had higher accuracy scores. Both achieved \u2265\u00a075% accuracy on 23 items (65.71%), however, the results varied. ChatGPT showed higher consistency (ICC = 0.93) compared to DeepSeek (ICC = 0.88).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The ad^Writer system leverages large language models to demonstrate the potential for rapid, accurate, and reproducible guideline quality assessments based on RIGHT. By serving as a supportive tool for manual evaluation, this approach enhances the efficiency of the guideline appraisal process, saving time and human resources for stakeholders.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1111\/jebm.70193","type":"journal-article","created":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T04:50:01Z","timestamp":1791348601000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A Comparative Study of ad^Writer and Human Evaluations of Guideline Reporting Quality: Performance of ChatGPT and DeepSeek"],"prefix":"10.1111","author":[{"given":"Jie","family":"Zhang","sequence":"first","affiliation":[{"name":"Research Unit of Evidence\u2010Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences Lanzhou University Lanzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qianling","family":"Shi","sequence":"additional","affiliation":[{"name":"The First School of Clinical Medicine Lanzhou University  Lanzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hui","family":"Liu","sequence":"additional","affiliation":[{"name":"Research Unit of Evidence\u2010Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences Lanzhou University Lanzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xufei","family":"Luo","sequence":"additional","affiliation":[{"name":"Research Unit of Evidence\u2010Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences Lanzhou University Lanzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhenhua","family":"Yang","sequence":"additional","affiliation":[{"name":"Vincent V.C. Woo Chinese Medicine Clinical Research Institute, School of Chinese Medicine Hong Kong Baptist University  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zeming","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science Hong Kong Baptist University  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bingyi","family":"Wang","sequence":"additional","affiliation":[{"name":"Research Unit of Evidence\u2010Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences Lanzhou University Lanzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yishan","family":"Qin","sequence":"additional","affiliation":[{"name":"Research Unit of Evidence\u2010Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences Lanzhou University Lanzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Honghao","family":"Lai","sequence":"additional","affiliation":[{"name":"Department of Health Policy and Management, School of Public Health Lanzhou University  Lanzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yaolong","family":"Chen","sequence":"additional","affiliation":[{"name":"Research Unit of Evidence\u2010Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences Lanzhou University Lanzhou China"},{"name":"Evidence\u2010Based Medicine Center, School of Basic Medical Sciences Lanzhou University  Lanzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"the ADVANCED Working Group","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,10,6]]},"reference":[{"issue":"4","key":"e_1_2_6_2_1","first-page":"403","article-title":"How to Differentiate Clinical Practice Guidelines From Expert Consensus","volume":"10","author":"Chen Y.","year":"2019","journal-title":"Medical College Journal"},{"issue":"37","key":"e_1_2_6_3_1","first-page":"2912","article-title":"Evaluation and ranking for scientific, transparent and applicable of Chinese guidelines and consensus published in the medical journals in 2022","volume":"103","author":"Research Unit of Evidence\u2010Based Evaluation and Guidelines (2021RU017)","year":"2023","journal-title":"Chinese Medical Journal"},{"issue":"12","key":"e_1_2_6_4_1","first-page":"733","article-title":"The Management of Gout","volume":"78","author":"Peters C.","year":"2023","journal-title":"Revue M\u00e9dicale de Li\u00e8ge"},{"issue":"5","key":"e_1_2_6_5_1","doi-asserted-by":"crossref","first-page":"641","DOI":"10.3904\/kjim.2023.206","article-title":"Korean Guidelines for the Management of Gout","volume":"38","author":"Lee J. J.","year":"2023","journal-title":"Korean Journal of Internal Medicine"},{"issue":"6","key":"e_1_2_6_6_1","doi-asserted-by":"crossref","first-page":"752","DOI":"10.1136\/ard-2023-224771","article-title":"2023 EULAR Recommendations on Imaging in Diagnosis and Management of Crystal\u2010Induced Arthropathies in Clinical Practice","volume":"83","author":"Mandl P.","year":"2024","journal-title":"Annals of the Rheumatic Diseases"},{"issue":"12","key":"e_1_2_6_7_1","doi-asserted-by":"crossref","first-page":"1430","DOI":"10.1097\/CM9.0000000000002713","article-title":"Development of the Scientific, Transparent and Applicable Rankings (STAR) Tool for Clinical Practice Guidelines","volume":"136","author":"Yang N.","year":"2023","journal-title":"Chinese Medical Journal"},{"key":"e_1_2_6_8_1","unstructured":"Equator Network.Reporting Guidelines https:\/\/www.equator\u2010network.org\/reporting\u2010guidelines\/."},{"issue":"2","key":"e_1_2_6_9_1","doi-asserted-by":"crossref","first-page":"128","DOI":"10.7326\/M16-1565","article-title":"A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement","volume":"166","author":"Chen Y.","year":"2017","journal-title":"Annals of Internal Medicine"},{"key":"e_1_2_6_10_1","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1016\/j.jclinepi.2021.08.013","article-title":"Clinical Epidemiology in China Series. Paper 4: The Reporting and Methodological Quality of Chinese Clinical Practice Guidelines Published Between 2014 and 2018: A Systematic Review","volume":"140","author":"Zhou Q.","year":"2021","journal-title":"Journal of Clinical Epidemiology"},{"issue":"9","key":"e_1_2_6_11_1","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2024-086301","article-title":"Clinical Practice Guidelines and Expert Consensus Statements on Rehabilitation for Patients With COVID\u201019: A Systematic Review","volume":"14","author":"Zhang Y.","year":"2024","journal-title":"Bmj Open"},{"key":"e_1_2_6_12_1","first-page":"1877","article-title":"Language Models Are Few\u2010Shot Learners","volume":"33","author":"Brown T.","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_6_13_1","unstructured":"OpenAI J.Achiam S.Adler et\u00a0al. \u201cGPT 4 Technical Report \u201darXiv last modified March 4 2024 https:\/\/arxiv.org\/abs\/2303.08774."},{"key":"e_1_2_6_14_1","unstructured":"OpenAI A.Hurst A.Lerer et\u00a0al. \u201cGPT 4o System Card \u201darXiv published October 25 2024 https:\/\/arxiv.org\/abs\/2410.21276."},{"key":"e_1_2_6_15_1","unstructured":"DeepSeek\u2010AI A.Liu B.Feng et\u00a0al. \u201cDeepSeek V3 Technical Report \u201darXiv last modified February 18 2025 https:\/\/arxiv.org\/abs\/2412.19437."},{"key":"e_1_2_6_16_1","unstructured":"Advanced Artificial intelligence\u2010empowered DeVelopment and AssessmeNt of healthCare guidElines and stanDards https:\/\/www.advancedguides.ai\/."},{"issue":"10","key":"e_1_2_6_17_1","doi-asserted-by":"crossref","first-page":"3283","DOI":"10.1038\/s41591-025-03953-8","article-title":"The STARD\u2010AI Reporting Guideline for Diagnostic Accuracy Studies Using Artificial Intelligence","volume":"31","author":"Sounderajah V.","year":"2025","journal-title":"Nature Medicine"},{"issue":"6","key":"e_1_2_6_18_1","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1016\/j.jclinepi.2010.09.007","article-title":"Optimism Bias Leads to Inconclusive Results: An Empirical Study","volume":"64","author":"Djulbegovic B.","year":"2011","journal-title":"Journal of Clinical Epidemiology"},{"issue":"3","key":"e_1_2_6_19_1","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1002\/uog.21929","article-title":"Expected Value Bias in Routine Third Trimester Growth Scans","volume":"55","author":"Drukker L.","year":"2020","journal-title":"Ultrasound in Obstetrics and Gynecology"},{"issue":"1","key":"e_1_2_6_20_1","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1186\/s13643-021-01645-1","article-title":"Reporting Quality of Clinical Practice Guidelines Regarding Gout and Hyperuricemia According to the RIGHT Checklist: Systematic Review","volume":"10","author":"Wang C.","year":"2021","journal-title":"Systematic Reviews"},{"issue":"8","key":"e_1_2_6_21_1","doi-asserted-by":"crossref","first-page":"633","DOI":"10.21037\/atm-20-7000","article-title":"Quality and Consistency of Clinical Practice Guidelines for Treating Children With COVID\u201019","volume":"9","author":"Li Q.","year":"2021","journal-title":"Annals of Translational Medicine"},{"issue":"2","key":"e_1_2_6_22_1","doi-asserted-by":"crossref","first-page":"774","DOI":"10.21037\/apm-21-2851","article-title":"Quality Appraisal of Clinical Practice Guidelines on Physical Restraints in ICU: A Systematic Review","volume":"11","author":"Lei R.","year":"2022","journal-title":"Annals of Palliative Medicine"},{"key":"e_1_2_6_23_1","doi-asserted-by":"crossref","DOI":"10.1016\/j.arr.2024.102559","article-title":"Recommendations and Quality of Multimorbidity Guidelines: A Systematic Review","volume":"102","author":"Wang Z.","year":"2024","journal-title":"Ageing Research Reviews"},{"issue":"51","key":"e_1_2_6_24_1","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2309583120","article-title":"Systematic Testing of Three Language Models Reveals Low Language Accuracy, Absence of Response Stability, and a Yes Response Bias","volume":"120","author":"Dentella V.","year":"2023","journal-title":"Proceedings of the National Academy of Sciences of the United States of America"},{"key":"e_1_2_6_25_1","unstructured":"R.LiuandN. B.Shah \u201cReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing \u201darXiv published June 1 2023 https:\/\/arxiv.org\/abs\/2306.00622."},{"key":"e_1_2_6_26_1","unstructured":"A.Goldberg I.Ullah T. G. H.Khuong et\u00a0al. \u201cUsefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment \u201darXiv last modified November 8 2024 https:\/\/arxiv.org\/abs\/2411.03417."},{"issue":"4","key":"e_1_2_6_27_1","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1038\/s41562-024-01847-2","article-title":"How to Write Effective Prompts for Large Language Models","volume":"8","author":"Lin Z.","year":"2024","journal-title":"Nature Human Behaviour"},{"issue":"4","key":"e_1_2_6_28_1","doi-asserted-by":"crossref","DOI":"10.1016\/j.acalib.2023.102720","article-title":"The CLEAR Path: A Framework for Enhancing Information Literacy Through Prompt Engineering","volume":"49","author":"Lo L. S.","year":"2023","journal-title":"Journal of Academic Librarianship"},{"key":"e_1_2_6_29_1","doi-asserted-by":"crossref","DOI":"10.2196\/50638","article-title":"Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial","volume":"25","author":"Mesko B.","year":"2023","journal-title":"Journal of Medical Internet Research"},{"issue":"8","key":"e_1_2_6_30_1","first-page":"1187","article-title":"A Study on Performance Improvement of Prompt Engineering for Generative AI With a Large Language Model","volume":"22","author":"Park D.","year":"2023","journal-title":"Journal of Web Engineering"}],"container-title":["Journal of Evidence-Based Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/jebm.70193","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/jebm.70193","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/jebm.70193","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T04:50:12Z","timestamp":1791348612000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/jebm.70193"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,6]]},"references-count":29,"alternative-id":["10.1111\/jebm.70193"],"URL":"https:\/\/doi.org\/10.1111\/jebm.70193","archive":["Portico"],"relation":{},"ISSN":["1756-5383","1756-5391"],"issn-type":[{"value":"1756-5383","type":"print"},{"value":"1756-5391","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,6]]},"assertion":[{"value":"2025-09-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-19","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-10-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70193"}}