{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T15:55:00Z","timestamp":1792252500108,"version":"4.4.0"},"reference-count":32,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>ChatGPT-4 is the latest release of a novel artificial intelligence (AI) chatbot able to answer freely formulated and complex questions. In the near future, ChatGPT could become the new standard for health care professionals and patients to access medical information. However, little is known about the quality of medical information provided by the AI.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>We aimed to assess the reliability of medical information provided by ChatGPT.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Medical information provided by ChatGPT-4 on the 5 hepato-pancreatico-biliary (HPB) conditions with the highest global disease burden was measured with the Ensuring Quality Information for Patients (EQIP) tool. The EQIP tool is used to measure the quality of internet-available information and consists of 36 items that are divided into 3 subsections. In addition, 5 guideline recommendations per analyzed condition were rephrased as questions and input to ChatGPT, and agreement between the guidelines and the AI answer was measured by 2 authors independently. All queries were repeated 3 times to measure the internal consistency of ChatGPT.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Five conditions were identified (gallstone disease, pancreatitis, liver cirrhosis, pancreatic cancer, and hepatocellular carcinoma). The median EQIP score across all conditions was 16 (IQR 14.5-18) for the total of 36 items. Divided by subsection, median scores for content, identification, and structure data were 10 (IQR 9.5-12.5), 1 (IQR 1-1), and 4 (IQR 4-5), respectively. Agreement between guideline recommendations and answers provided by ChatGPT was 60% (15\/25). Interrater agreement as measured by the Fleiss \u03ba was 0.78 (P&lt;.001), indicating substantial agreement. Internal consistency of the answers provided by ChatGPT was 100%.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>ChatGPT provides medical information of comparable quality to available static internet information. Although currently of limited quality, large language models could become the future standard for patients and health care professionals to gather medical information.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/47479","type":"journal-article","created":{"date-parts":[[2023,6,15]],"date-time":"2023-06-15T05:04:40Z","timestamp":1686805480000},"page":"e47479","source":"Crossref","is-referenced-by-count":363,"title":["Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument"],"prefix":"10.2196","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-6278-6399","authenticated-orcid":false,"given":"Harriet Louise","family":"Walker","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7366-296X","authenticated-orcid":false,"given":"Shahi","family":"Ghani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7109-3545","authenticated-orcid":false,"given":"Christoph","family":"Kuemmerli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7493-2850","authenticated-orcid":false,"given":"Christian Andreas","family":"Nebiker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8552-8538","authenticated-orcid":false,"given":"Beat Peter","family":"M\u00fcller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0898-3270","authenticated-orcid":false,"given":"Dimitri Aristotle","family":"Raptis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0818-9835","authenticated-orcid":false,"given":"Sebastian Manuel","family":"Staubli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2023,6,30]]},"reference":[{"key":"ref1","unstructured":"ChatGPTOpenAI2023-06-20https:\/\/chat.openai.com\/chat"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.2196\/45312"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.2196\/46885"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.3390\/bdcc7020062"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.37074\/jalt.2023.6.1.9"},{"key":"ref6","unstructured":"OpenAIWikipedia2023-06-20https:\/\/en.wikipedia.org\/wiki\/OpenAI"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(23)00019-5"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(23)00023-7"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-023-00288-7"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.4018\/IJSPPC.320225"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(22)01438-6"},{"key":"ref12","unstructured":"Digestive diseases \u2014 Level 2 causeInstitute for Health Metrics and Evaluation Global Burden of Disease (GBD)2023-06-20https:\/\/www.healthdata.org\/results\/gbd_summaries\/2019\/digestive-diseases-level-2-cause"},{"key":"ref13","unstructured":"Total cancers \u2014 CauseInstitute for Health Metrics and Evaluation Global Burden of Disease (GBD)2023-06-20https:\/\/www.healthdata.org\/results\/gbd_summaries\/2019\/total-cancers-level-2-cause"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1111\/j.1369-7625.2004.00273.x"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1016\/j.pec.2007.11.018"},{"key":"ref16","unstructured":"Clinical Guidance \u2014 Gallstone disease: diagnosis and managementNational Institute for Health and Care Excellence2023-06-20https:\/\/www.nice.org.uk\/guidance\/cg188\/chapter\/1-Recommendations"},{"key":"ref17","unstructured":"Pancreatitis NICE guideline [NG104]National Institute for Health and Care Excellence2023-06-20https:\/\/www.nice.org.uk\/guidance\/ng104\/chapter\/Recommendations"},{"key":"ref18","unstructured":"Cirrhosis in over 16s: assessment and managementNational Institute for Health and Care Excellence2023-06-20https:\/\/www.nice.org.uk\/guidance\/ng50\/chapter\/Recommendations"},{"key":"ref19","unstructured":"Pancreatic cancer in adults: diagnosis and managementNational Institute for Health and Care Excellence2023-06-20https:\/\/www.nice.org.uk\/guidance\/ng85\/chapter\/Recommendations-for-research"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1016\/j.jhep.2018.03.019"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1016\/j.hpb.2019.03.355"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.2196\/22618"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2020-040487"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijporl.2022.111224"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1007\/s11695-017-2983-0"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1007\/s12029-022-00879-z"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1038\/s41571-023-00746-1"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbr.2021.113684"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1186\/s13054-023-04380-2"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.35179"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.4334162"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.2196\/41430"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2023\/1\/e47479","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:55:20Z","timestamp":1782842120000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2023\/1\/e47479"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,30]]},"references-count":32,"URL":"https:\/\/doi.org\/10.2196\/47479","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.47479","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,30]]}}}