{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T13:57:45Z","timestamp":1788184665762,"version":"build-2803163510"},"reference-count":39,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Surgical site infection (SSI) is the most prevalent type of health care\u2013associated infection that leads to increased morbidity and mortality and a significant economic burden. Effective prevention of SSI relies on surgeons strictly following the latest clinical guidelines and implementing standardized and multilevel intervention strategies. However, the frequent updates to clinical guidelines render the processes of acquisition and interpretation quite time-consuming and intricate. The emergence of artificial intelligence (AI) chatbots offers both possibilities and challenges to address these issues in the surgical field.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to test the multidimensional capability of state-of-the-art AI chatbots for generating proper recommendations and corresponding rationales concordant with the global guideline for the prevention of SSI.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Referred by other authoritative guidelines, recommendations and corresponding rationales from the 2018 World Health Organization global guidelines were refined and selected as benchmarks. Then, they were rephrased into a combined format of closed-ended queries for recommendations and open-ended queries for corresponding rationales, whereafter input into ChatGPT-4o (OpenAI), OpenAI-o1 (OpenAI), Claude 3.5 Sonnet (Anthropic), and Gemini 1.5 Pro (Google) 3 times. All responses were individually evaluated in 10 evaluation metrics based on the QUEST dimensions by 4 multidisciplinary senior surgeons using a 5-point Likert scale. The multidimensional performances among chatbots were compared, and the interrater agreements were calculated.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n A total of 300 responses to 25 queries were generated by the 4 chatbots. The interrater agreements of the evaluators ranged from moderate to good (0.54\u20100.87). In response to recommendations, the average accuracy, consistency, and harm scores for all chatbots were 4.03 (SD 1.09), 4.07 (SD 0.88), and 4.29 (SD 1.01), respectively. In responses for rationales, 4 subdimensions, including harm (mean 4.22, SD 0.97), relevance (mean 4.15, SD 0.83), fabrication and falsification (mean 4.12, SD 1.02), and understanding and reasoning (mean 4.04, SD 0.92) averagely scored \u22654. In contrast, consistency (mean 3.94, SD 0.72), clarity (mean 3.94, SD 0.89), comprehensiveness (mean 3.85, SD 0.83), and accuracy (mean 3.74, SD 0.91) performed at a moderate level. For the whole responses, the average self-awareness and trust and confidence scores for all chatbots were 3.84 (SD 0.89) and 3.88 (SD 0.91), respectively. Based on the average scores of the subdimensions, Claude 3.5 Sonnet and ChatGPT-4o were the top 2 outperformed models.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The performance of AI chatbots in providing responses regarding well-established global guidelines in the prevention of SSI was acceptable, demonstrating immense potential in clinical applications. Nonetheless, a critical issue is the necessity of enhancing the stability of chatbots, as inaccurate responses can lead to severe consequences for SSI. Despite its limitations, it is anticipated that AI will trigger far-reaching changes in how clinicians access and use medical information.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/75567","type":"journal-article","created":{"date-parts":[[2025,7,31]],"date-time":"2025-07-31T23:11:08Z","timestamp":1754003468000},"page":"e75567-e75567","source":"Crossref","is-referenced-by-count":11,"title":["Evaluating the Performance of State-of-the-Art Artificial Intelligence Chatbots Based on the WHO Global Guidelines for the Prevention of Surgical Site Infection: Cross-Sectional Study"],"prefix":"10.2196","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5016-858X","authenticated-orcid":false,"given":"Tianyi","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-0745-4427","authenticated-orcid":false,"given":"Ruiyuan","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3644-928X","authenticated-orcid":false,"given":"Baodong","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-5268-3638","authenticated-orcid":false,"given":"Congying","family":"Zou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0095-9476","authenticated-orcid":false,"given":"Ning","family":"Fan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5668-9527","authenticated-orcid":false,"given":"Shuo","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3271-1953","authenticated-orcid":false,"given":"Aobo","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3022-9281","authenticated-orcid":false,"given":"Yu","family":"Xi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1403-4159","authenticated-orcid":false,"given":"Lei","family":"Zang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,7,31]]},"reference":[{"key":"R1","unstructured":"Global guidelines on the prevention of surgical site infection. World Health Organization. 2018. URL: https:\/\/www.who.int\/publications\/i\/item\/9789241550475#cms [Accessed 27-11-2024]"},{"issue":"8","key":"R2","doi-asserted-by":"publisher","first-page":"784","DOI":"10.1001\/jamasurg.2017.0904","article-title":"Centers for Disease Control and Prevention guideline for the prevention of surgical site infection, 2017","volume":"152","author":"Berr\u00edos-Torres","journal-title":"JAMA Surg"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1177\/00469580231162549","article-title":"Global incidence of surgical site infection among patients: systematic review and meta-analysis","volume":"60","author":"Mengistu","journal-title":"Inquiry"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1186\/s13756-019-0638-8","article-title":"APSIC guidelines for the prevention of surgical site infections","volume":"8","author":"Ling","journal-title":"Antimicrob Resist Infect Control"},{"key":"R5","unstructured":"Surgical site infections: prevention and treatment. National Institute for Health and Care Excellence. 2019. URL: https:\/\/www.nice.org.uk\/guidance\/ng125 [Accessed 27-11-2024]"},{"issue":"5","key":"R6","doi-asserted-by":"publisher","first-page":"695","DOI":"10.1017\/ice.2023.67","article-title":"Strategies to prevent surgical site infections in acute-care hospitals: 2022 update","volume":"44","author":"Calderwood","journal-title":"Infect Control Hosp Epidemiol"},{"key":"R7","unstructured":"What are the key infection prevention and control recommendations to inform a surgical site infection (SSI) prevention quality improvement tool? National Services Scotland. Health Protection Scotland; 2018. URL: https:\/\/www.nss.nhs.scot\/media\/2208\/surgical-site-infection-ssi-prevention-quality-improvement-tool.pdf [Accessed 27-11-2024]"},{"issue":"13","key":"R8","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","journal-title":"N Engl J Med"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.2196\/56930","article-title":"Roles, users, benefits, and limitations of chatbots in health care: rapid review","volume":"26","author":"Laymouna","journal-title":"J Med Internet Res"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/47479","article-title":"Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument","volume":"25","author":"Walker","journal-title":"J Med Internet Res"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","journal-title":"NPJ Digit Med"},{"issue":"6","key":"R12","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare11060887","article-title":"ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns","volume":"11","author":"Sallam","journal-title":"Healthcare (Basel)"},{"issue":"3","key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(23)00023-7","article-title":"ChatGPT: friend or foe?","volume":"5","author":"The Lancet Digital Health","journal-title":"Lancet Digit Health"},{"issue":"7624","key":"R14","doi-asserted-by":"publisher","first-page":"806","DOI":"10.1136\/bmj.39335.541782.AD","article-title":"Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies","volume":"335","author":"Elm","journal-title":"BMJ"},{"issue":"2","key":"R15","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1016\/j.jhin.2016.12.016","article-title":"World Health Organization: global guidelines for the prevention of surgical site infection","volume":"95","author":"Leaper","journal-title":"J Hosp Infect"},{"key":"R16","unstructured":"The FORRESTER wave: AI foundation models for language, Q2 2024. FORRESTER. 2024. URL: https:\/\/www.forrester.com\/report\/the-forrester-wave-tm-ai-foundation-models-for-language-q2-2024\/RES180932?ref_search=0_1731504324998 [Accessed 30-11-2024]"},{"issue":"8","key":"R17","doi-asserted-by":"publisher","first-page":"1215","DOI":"10.1093\/ecco-jcc\/jjae040","article-title":"Accuracy of information given by ChatGPT for patients with inflammatory bowel disease in relation to ECCO guidelines","volume":"18","author":"Sciberras","journal-title":"J Crohns Colitis"},{"issue":"11","key":"R18","doi-asserted-by":"publisher","first-page":"1684","DOI":"10.1016\/j.spinee.2023.07.015","article-title":"Thromboembolic prophylaxis in spine surgery: an analysis of ChatGPT recommendations","volume":"23","author":"Duey","journal-title":"Spine J"},{"issue":"4","key":"R19","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1016\/S1473-3099(23)00113-5","article-title":"ChatGPT and antimicrobial advice: the end of the consulting infection doctor?","volume":"23","author":"Howard","journal-title":"Lancet Infect Dis"},{"issue":"6","key":"R20","doi-asserted-by":"publisher","first-page":"1790","DOI":"10.1007\/s11695-023-06603-5","article-title":"Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery","volume":"33","author":"Samaan","journal-title":"Obes Surg"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","first-page":"721","DOI":"10.3350\/cmh.2023.0089","article-title":"Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma","volume":"29","author":"Yeo","journal-title":"Clin Mol Hepatol"},{"issue":"9","key":"R22","doi-asserted-by":"publisher","first-page":"640","DOI":"10.1097\/BRS.0000000000004915","article-title":"Performance of ChatGPT on NASS clinical guidelines for the diagnosis and treatment of low back pain","volume":"49","author":"Shrestha","journal-title":"Spine (Phila Pa 1986)"},{"issue":"2","key":"R23","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"Alkaissi","journal-title":"Cureus"},{"key":"R24","unstructured":"Introducing ChatGPT. OpenAI. 2022. URL: https:\/\/openai.com\/blog\/chatgpt [Accessed 05-12-2024]"},{"issue":"10","key":"R25","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and reliability of chatbot responses to physician questions","volume":"6","author":"Goodman","journal-title":"JAMA Netw Open"},{"issue":"2","key":"R26","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.34616","article-title":"Use of ChatGPT in academic publishing: a rare case of seronegative systemic lupus erythematosus in a patient with HIV infection","volume":"15","author":"Manohar","journal-title":"Cureus"},{"issue":"2","key":"R27","doi-asserted-by":"publisher","first-page":"444","DOI":"10.1002\/ase.2261","article-title":"Initial impressions of ChatGPT for anatomy education","volume":"17","author":"Mogali","journal-title":"Anat Sci Educ"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1136\/ard-2024-226202","article-title":"Assessing the performance of AI chatbots in answering patients\u2019 common questions about low back pain","volume":"84","author":"Scaff","journal-title":"Ann Rheum Dis"},{"issue":"6630","key":"R29","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1126\/science.adg7879","article-title":"ChatGPT is fun, but not an author","volume":"379","author":"Thorp","journal-title":"Science"},{"issue":"7945","key":"R30","doi-asserted-by":"publisher","first-page":"620","DOI":"10.1038\/d41586-023-00107-z","article-title":"ChatGPT listed as author on research papers: many scientists disapprove","volume":"613","author":"Stokel-Walker","journal-title":"Nature New Biol"},{"issue":"8","key":"R31","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1001\/jama.2023.1344","article-title":"Nonhuman \u201cauthors\u201d and implications for the integrity of scientific publication and medical knowledge","volume":"329","author":"Flanagin","journal-title":"JAMA"},{"issue":"2","key":"R32","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/j.arthro.2024.07.040","article-title":"Currently available large language models do not provide musculoskeletal treatment recommendations that are concordant with evidence-based clinical practice guidelines","volume":"41","author":"Nwachukwu","journal-title":"Arthroscopy"},{"issue":"16","key":"R33","doi-asserted-by":"publisher","first-page":"1349","DOI":"10.1001\/jama.2023.5321","article-title":"AI-generated medical advice-GPT and beyond","volume":"329","author":"Haupt","journal-title":"JAMA"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.2196\/64406","article-title":"Ethical considerations in human-centered AI: advancing oncology chatbots through large language models","volume":"5","author":"Chow","journal-title":"JMIR Bioinform Biotechnol"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2023.1166014","article-title":"Impact of ChatGPT on medical chatbots as a disruptive technology","volume":"6","author":"Chow","journal-title":"Front Artif Intell"},{"key":"R36","unstructured":"Zhu K Wang J Zhou J et al. PromptBench: towards evaluating the robustness of large language models on adversarial prompts. arXiv. Preprint posted online on 2024. [doi: 10.48550\/arXiv.2306.04528]"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/50638","article-title":"Prompt engineering as an important emerging skill for medical professionals: tutorial","volume":"25","author":"Mesk\u00f3","journal-title":"J Med Internet Res"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.2196\/60501","article-title":"Prompt engineering paradigms for medical applications: scoping review","volume":"26","author":"Zaghir","journal-title":"J Med Internet Res"},{"issue":"2","key":"R39","doi-asserted-by":"publisher","DOI":"10.1002\/imt2.178","article-title":"ChatGPT and generative AI are revolutionizing the scientific community: a Janus-faced conundrum","volume":"3","author":"Pu","journal-title":"Imeta"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2025\/1\/e75567","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:06:38Z","timestamp":1782842798000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2025\/1\/e75567"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,31]]},"references-count":39,"URL":"https:\/\/doi.org\/10.2196\/75567","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,31]]},"article-number":"v27i8e75567"}}