{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T21:06:28Z","timestamp":1790197588495,"version":"4.1.0"},"reference-count":19,"publisher":"BMJ","issue":"5","license":[{"start":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T00:00:00Z","timestamp":1717027200000},"content-version":"unspecified","delay-in-days":29,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Susan and Luke Simons Directorship for Melanoma","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}]},{"DOI":"10.13039\/100007316","name":"Division of Cancer Prevention, National Cancer Institute","doi-asserted-by":"publisher","award":["NCI R01CA227481"],"award-info":[{"award-number":["NCI R01CA227481"]}],"id":[{"id":"10.13039\/100007316","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Van Stephenson Melanoma Fund","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}]},{"name":"James C. Bradford Melanoma Fund","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["J Immunother Cancer"],"accepted":{"date-parts":[[2024,5,14]]},"published-print":{"date-parts":[[2024,5]]},"abstract":"<jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Artificial intelligence (AI) chatbots have become a major source of general and medical information, though their accuracy and completeness are still being assessed. Their utility to answer questions surrounding immune-related adverse events (irAEs), common and potentially dangerous toxicities from cancer immunotherapy, are not well defined.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We developed 50 distinct questions with answers in available guidelines surrounding 10 irAE categories and queried two AI chatbots (ChatGPT and Bard), along with an additional 20 patient-specific scenarios. Experts in irAE management scored answers for accuracy and completion using a Likert scale ranging from 1 (least accurate\/complete) to 4 (most accurate\/complete). Answers across categories and across engines were compared.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Overall, both engines scored highly for accuracy (mean scores for ChatGPT and Bard were 3.87 vs 3.5, p&lt;0.01) and completeness (3.83 vs 3.46, p&lt;0.01). Scores of 1\u20132 (completely or mostly inaccurate or incomplete) were particularly rare for ChatGPT (6\/800 answer-ratings, 0.75%). Of the 50 questions, all eight physician raters gave ChatGPT a rating of 4 (fully accurate or complete) for 22 questions (for accuracy) and 16 questions (for completeness). In the 20 patient scenarios, the average accuracy score was 3.725 (median 4) and the average completeness was 3.61 (median 4).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>AI chatbots provided largely accurate and complete information regarding irAEs, and wildly inaccurate information (\u201challucinations\u201d) was uncommon. However, until accuracy and completeness increases further, appropriate guidelines remain the gold standard to follow<\/jats:p>\n               <\/jats:sec>","DOI":"10.1136\/jitc-2023-008599","type":"journal-article","created":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T18:57:55Z","timestamp":1717095475000},"page":"e008599","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":38,"title":["Use of artificial intelligence chatbots in clinical management of immune-related adverse events"],"prefix":"10.1136","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-7784-634X","authenticated-orcid":false,"given":"Hannah","family":"Burnette","sequence":"first","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7605-8497","authenticated-orcid":false,"given":"Aliyah","family":"Pabani","sequence":"additional","affiliation":[{"name":"Department of Oncology, Johns Hopkins University, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mitchell S","family":"von Itzstein","sequence":"additional","affiliation":[{"name":"Harold C Simmons Comprehensive Cancer Center, The University of Texas Southwestern Medical Center, Dallas, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8150-1963","authenticated-orcid":false,"given":"Benjamin","family":"Switzer","sequence":"additional","affiliation":[{"name":"Department of Medicine, Roswell Park Comprehensive Cancer Center, Buffalo, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Run","family":"Fan","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fei","family":"Ye","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9803-3497","authenticated-orcid":false,"given":"Igor","family":"Puzanov","sequence":"additional","affiliation":[{"name":"Department of Medicine, Roswell Park Comprehensive Cancer Center, Buffalo, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3470-8686","authenticated-orcid":false,"given":"Jarushka","family":"Naidoo","sequence":"additional","affiliation":[{"name":"RCSI Cancer Centre, Beaumont Hospital, Dublin, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8322-475X","authenticated-orcid":false,"given":"Paolo A","family":"Ascierto","sequence":"additional","affiliation":[{"name":"Department of Melanoma, Cancer Immunotherapy and Development Therapeutics, Istituto Nazionale Tumori IRCCS Fondazione Pascale, Napoli, Campania, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7812-6741","authenticated-orcid":false,"given":"David E","family":"Gerber","sequence":"additional","affiliation":[{"name":"Harold C Simmons Comprehensive Cancer Center, The University of Texas Southwestern Medical Center, Dallas, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8132-7069","authenticated-orcid":false,"given":"Marc S","family":"Ernstoff","sequence":"additional","affiliation":[{"name":"ImmunoOncology Branch (IOB), Developmental Therapeutics Program, Cancer Therapy and Diagnosis Division, National Cancer Institute (NCI), National Institutes of Health, Bethesda, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Douglas B","family":"Johnson","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2024,5,30]]},"reference":[{"key":"2025100312181131000_12.5.e008599.1","doi-asserted-by":"crossref","DOI":"10.1148\/radiol.230163","article-title":"Chatgpt and other large language models are double-edged swords","volume":"307","author":"Shen","year":"2023","journal-title":"Radiology"},{"key":"2025100312181131000_12.5.e008599.2","doi-asserted-by":"crossref","first-page":"1437","DOI":"10.1001\/jamaoncol.2023.2947","article-title":"Assessment of artificial intelligence Chatbot responses to top searched queries about cancer","volume":"9","author":"Pan","year":"2023","journal-title":"JAMA Oncol"},{"key":"2025100312181131000_12.5.e008599.3","doi-asserted-by":"crossref","first-page":"1459","DOI":"10.1001\/jamaoncol.2023.2954","article-title":"Use of artificial intelligence Chatbots for cancer treatment information","volume":"9","author":"Chen","year":"2023","journal-title":"JAMA Oncol"},{"key":"2025100312181131000_12.5.e008599.4","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and reliability of Chatbot responses to physician questions","volume":"6","author":"Goodman","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2025100312181131000_12.5.e008599.5","doi-asserted-by":"crossref","DOI":"10.1155\/2020\/4790254","article-title":"The epidemiology of migraine headache in Arab countries: A systematic review","volume":"2020","author":"El-Metwally","year":"2020","journal-title":"ScientificWorldJournal"},{"key":"2025100312181131000_12.5.e008599.6","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence Chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"2025100312181131000_12.5.e008599.7","doi-asserted-by":"crossref","unstructured":"Maida E , Moccia M , Palladino R , et al . Chatgpt vs. neurologists: a cross-sectional study investigating preference, satisfaction ratings and perceived empathy in responses among people living with multiple sclerosis. J Neurol 2024. doi:10.1007\/s00415-024-12328-x","DOI":"10.1007\/s00415-024-12328-x"},{"key":"2025100312181131000_12.5.e008599.8","doi-asserted-by":"crossref","first-page":"1702","DOI":"10.1001\/jama.2018.13995","article-title":"Immune checkpoint inhibitor toxicity in 2018","volume":"320","author":"Johnson","year":"2018","journal-title":"JAMA"},{"key":"2025100312181131000_12.5.e008599.9","doi-asserted-by":"crossref","DOI":"10.1136\/jitc-2021-002435","article-title":"Society for Immunotherapy of cancer (SITC) clinical practice guideline on immune checkpoint inhibitor-related adverse events","volume":"9","author":"Brahmer","year":"2021","journal-title":"J Immunother Cancer"},{"key":"2025100312181131000_12.5.e008599.10","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.2017.77.6385"},{"key":"2025100312181131000_12.5.e008599.11","doi-asserted-by":"crossref","first-page":"230","DOI":"10.6004\/jnccn.2020.0012","article-title":"NCCN guidelines insights: management of Immunotherapy-related toxicities, version 1.2020","volume":"18","author":"Thompson","year":"2020","journal-title":"J Natl Compr Canc Netw"},{"key":"2025100312181131000_12.5.e008599.12","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"2025100312181131000_12.5.e008599.13","doi-asserted-by":"crossref","first-page":"807","DOI":"10.1002\/pros.24699","article-title":"Performance of large language models on benign prostatic hyperplasia frequently asked questions","volume":"84","author":"Zhang","year":"2024","journal-title":"Prostate"},{"key":"2025100312181131000_12.5.e008599.14","doi-asserted-by":"crossref","first-page":"495","DOI":"10.3233\/ADR-230191","article-title":"Chatgpt as a diagnostic aid in Alzheimer\u2019s disease: an exploratory study","volume":"8","author":"El Haj","year":"2024","journal-title":"J Alzheimers Dis Rep"},{"key":"2025100312181131000_12.5.e008599.15","article-title":"Accuracy of information given by ChatGPT for patients with inflammatory bowel disease in relation to ECCO guidelines","volume":"2024","author":"Sciberras","year":"2024","journal-title":"J Crohns Colitis"},{"key":"2025100312181131000_12.5.e008599.16","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1093\/oncolo\/oyae009","article-title":"Comparison of large language models in answering Immuno-oncology questions: a cross-sectional study","volume":"29","author":"Iannantuono","year":"2024","journal-title":"Oncologist"},{"key":"2025100312181131000_12.5.e008599.17","doi-asserted-by":"crossref","first-page":"454","DOI":"10.1016\/j.clgc.2023.12.017","article-title":"Urological cancers and ChatGPT: assessing the quality of information and possible risks for patients","volume":"22","author":"Ozgor","year":"2024","journal-title":"Clin Genitourin Cancer"},{"key":"2025100312181131000_12.5.e008599.18","doi-asserted-by":"crossref","first-page":"2757","DOI":"10.1002\/lary.31243","article-title":"Exploring the role of artificial intelligence Chatbots in preoperative counseling for head and neck cancer surgery","volume":"134","author":"Lee","year":"2024","journal-title":"Laryngoscope"},{"key":"2025100312181131000_12.5.e008599.19","doi-asserted-by":"publisher","DOI":"10.1016\/j.psychres.2023.115334"}],"container-title":["Journal for ImmunoTherapy of Cancer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/jitc-2023-008599","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,3]],"date-time":"2025-10-03T19:18:18Z","timestamp":1759519098000},"score":1,"resource":{"primary":{"URL":"https:\/\/jitc.bmj.com\/lookup\/doi\/10.1136\/jitc-2023-008599"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5]]},"references-count":19,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,5,30]]},"published-print":{"date-parts":[[2024,5]]}},"alternative-id":["10.1136\/jitc-2023-008599"],"URL":"https:\/\/doi.org\/10.1136\/jitc-2023-008599","relation":{},"ISSN":["2051-1426"],"issn-type":[{"value":"2051-1426","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5]]}}}