{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T18:25:48Z","timestamp":1777573548252,"version":"3.51.4"},"reference-count":36,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T00:00:00Z","timestamp":1715904000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"National Key R&D Program for Young Scientists","award":["2022YFF0712000"],"award-info":[{"award-number":["2022YFF0712000"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["72074006"],"award-info":[{"award-number":["72074006"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"General funding of the China Postdoctoral Science Foundation","award":["2023M740154"],"award-info":[{"award-number":["2023M740154"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,6,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Synthesizing and evaluating inconsistent medical evidence is essential in evidence-based medicine. This study aimed to employ ChatGPT\u00a0as a sophisticated scientific reasoning engine to identify conflicting clinical evidence and summarize unresolved questions to inform further research.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We evaluated ChatGPT\u2019s effectiveness in identifying conflicting evidence and investigated its principles of logical reasoning. An automated framework was developed to generate a PubMed dataset focused on controversial clinical topics. ChatGPT analyzed this dataset to identify consensus and controversy, and to formulate unsolved research questions. Expert evaluations were conducted 1) on the consensus and controversy for factual consistency, comprehensiveness, and potential harm and, 2) on the research questions for relevance, innovation, clarity, and specificity.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The gpt-4-1106-preview model achieved a 90% recall rate in detecting inconsistent claim pairs within a ternary assertions setup. Notably, without explicit reasoning prompts, ChatGPT provided sound reasoning for the assertions between claims and hypotheses, based on an analysis grounded in relevance, specificity, and certainty. ChatGPT\u2019s conclusions of consensus and controversies in clinical literature were comprehensive and factually consistent. The research questions proposed by ChatGPT received high expert ratings.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Our experiment implies that, in evaluating the relationship between evidence and claims, ChatGPT considered more detailed information beyond a straightforward assessment of sentimental orientation. This ability to process intricate information and conduct scientific reasoning regarding sentiment is noteworthy, particularly as this pattern emerged without explicit guidance or directives in prompts, highlighting ChatGPT\u2019s inherent logical reasoning capabilities.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>This study demonstrated ChatGPT\u2019s capacity to evaluate and interpret scientific claims. Such proficiency can be generalized to broader clinical research literature. ChatGPT effectively aids in facilitating clinical studies by proposing unresolved challenges based on analysis of existing studies. However, caution is advised as ChatGPT\u2019s outputs are inferences drawn from the input literature and could be harmful to clinical practice.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae100","type":"journal-article","created":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T16:45:46Z","timestamp":1715964346000},"page":"1551-1560","source":"Crossref","is-referenced-by-count":17,"title":["Utilizing ChatGPT as a scientific reasoning engine to differentiate conflicting evidence and summarize challenges in controversial clinical questions"],"prefix":"10.1093","volume":"31","author":[{"given":"Shiyao","family":"Xie","sequence":"first","affiliation":[{"name":"Institute of Medical Technology, Peking University Health Science Center , Beijing, 100191, China"},{"name":"National Institute of Health Data Science, Peking University , Beijing, 100191, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenjing","family":"Zhao","sequence":"additional","affiliation":[{"name":"Institute of Medical Technology, Peking University Health Science Center , Beijing, 100191, China"},{"name":"National Institute of Health Data Science, Peking University , Beijing, 100191, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guanghui","family":"Deng","sequence":"additional","affiliation":[{"name":"School of Health Humanities, Peking University , Beijing, 100191, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guohua","family":"He","sequence":"additional","affiliation":[{"name":"Department of Pediatric Nephrology and Rheumatology, Sun Yat-sen University First Affiliated Hospital , Guangzhou, Guangdong, 510062, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Na","family":"He","sequence":"additional","affiliation":[{"name":"Department of Pharmacy, Peking University Third Hospital , Beijing, 100089, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhenhua","family":"Lu","sequence":"additional","affiliation":[{"name":"Department of Gastrointestinal Cancer Translational Research Laboratory, Peking University Cancer Hospital , Beijing, 100143, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Weihua","family":"Hu","sequence":"additional","affiliation":[{"name":"Department of Epidemiology and Biostatistics, School of Public Health, Peking University , Beijing, 100191, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mingming","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Cardiology and Institute of Vascular Medicine, Peking University Third Hospital , Beijing, 100089, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8436-778X","authenticated-orcid":false,"given":"Jian","family":"Du","sequence":"additional","affiliation":[{"name":"Institute of Medical Technology, Peking University Health Science Center , Beijing, 100191, China"},{"name":"National Institute of Health Data Science, Peking University , Beijing, 100191, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2024,5,17]]},"reference":[{"issue":"2","key":"2024062008073569900_ocae100-B1","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1001\/jama.294.2.218","article-title":"Contradicted and initially stronger effects in highly cited clinical research","volume":"294","author":"Ioannidis","year":"2005","journal-title":"JAMA"},{"issue":"6","key":"2024062008073569900_ocae100-B2","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1016\/j.jclinepi.2004.10.019","article-title":"Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials","volume":"58","author":"Ioannidis","year":"2005","journal-title":"J Clin Epidemiol"},{"issue":"6","key":"2024062008073569900_ocae100-B3","doi-asserted-by":"crossref","first-page":"e333","DOI":"10.1016\/S2589-7500(23)00083-3","article-title":"Ethics of large language models in medicine and medical research","volume":"5","author":"Li","year":"2023","journal-title":"Lancet Digit Health"},{"key":"2024062008073569900_ocae100-B4","doi-asserted-by":"crossref","first-page":"e45183","DOI":"10.7554\/eLife.45183","article-title":"A comprehensive review of randomized clinical trials in three medical journals reveals 396 medical reversals","volume":"8","author":"Herrera-Perez","year":"2019","journal-title":"Elife"},{"issue":"3","key":"2024062008073569900_ocae100-B5","doi-asserted-by":"crossref","first-page":"e2005468","DOI":"10.1371\/journal.pbio.2005468","article-title":"Meta-research: why research on research matters","volume":"16","author":"Ioannidis","year":"2018","journal-title":"PLoS Biol"},{"issue":"1","key":"2024062008073569900_ocae100-B6","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1093\/ije\/dyv184","article-title":"Why do we think we know what we know? A metaknowledge analysis of the salt controversy","volume":"45","author":"Trinquart","year":"2016","journal-title":"Int J Epidemiol"},{"issue":"2","key":"2024062008073569900_ocae100-B7","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1177\/0003122421996323","article-title":"Creative destruction: the structural consequences of scientific curation","volume":"86","author":"McMahan","year":"2021","journal-title":"Am Sociol Rev"},{"key":"2024062008073569900_ocae100-B8","volume-title":"Pattern -Based Extraction of Argumentation from the Scientific Literature","author":"White","year":"2010"},{"key":"2024062008073569900_ocae100-B9","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1186\/s13326-016-0083-z","article-title":"A corpus of potentially contradictory research claims from cardiovascular research abstracts","volume":"7","author":"Alamri","year":"2016","journal-title":"J Biomed Semantics"},{"key":"2024062008073569900_ocae100-B10","doi-asserted-by":"crossref","first-page":"103275","DOI":"10.1016\/j.jbi.2019.103275","article-title":"Towards a characterization of apparent contradictions in the biomedical literature using context analysis","volume":"98","author":"Rosemblat","year":"2019","journal-title":"J Biomed Inform"},{"key":"2024062008073569900_ocae100-B11","first-page":"237","article-title":"Controversial trials first: identifying disagreement between clinical guidelines and new evidence","volume":"2021","author":"Borchert","year":"2022","journal-title":"AMIA Annu Symp Proc"},{"key":"2024062008073569900_ocae100-B12","first-page":"694","article-title":"Detecting contradictory COVID-19 drug efficacy claims from biomedical literature","author":"Sosa"},{"issue":"1","key":"2024062008073569900_ocae100-B13","doi-asserted-by":"crossref","first-page":"vbab012","DOI":"10.1093\/bioadv\/vbab012","article-title":"Identifying and classifying goals for scientific knowledge","volume":"1","author":"Boguslav","year":"2021","journal-title":"Bioinform Adv"},{"key":"2024062008073569900_ocae100-B14","doi-asserted-by":"crossref","first-page":"104405","DOI":"10.1016\/j.jbi.2023.104405","article-title":"Creating an ignorance-base: exploring known unknowns in the scientific literature","volume":"143","author":"Boguslav","year":"2023","journal-title":"J Biomed Inform"},{"issue":"11","key":"2024062008073569900_ocae100-B15","first-page":"11982","article-title":"A search engine for discovery of scientific challenges and directions","volume":"36","author":"Lahav","year":"2022","journal-title":"Proc AAAI Conf Artif Intell"},{"issue":"1","key":"2024062008073569900_ocae100-B16","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","year":"2023","journal-title":"Commun Med"},{"issue":"8","key":"2024062008073569900_ocae100-B17","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2024062008073569900_ocae100-B18","first-page":"1998","author":"Agrawal","year":"2022"},{"key":"2024062008073569900_ocae100-B19","first-page":"1","author":"Li"},{"issue":"12","key":"2024062008073569900_ocae100-B20","doi-asserted-by":"crossref","first-page":"e855","DOI":"10.1016\/S2589-7500(23)00202-9","article-title":"Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders","volume":"5","author":"Vaid","year":"2023","journal-title":"Lancet Digit Health"},{"key":"2024062008073569900_ocae100-B21","author":"Wang"},{"key":"2024062008073569900_ocae100-B22","author":"Qi"},{"key":"2024062008073569900_ocae100-B23","author":"OpenAI","year":"2023"},{"issue":"1","key":"2024062008073569900_ocae100-B24","doi-asserted-by":"crossref","first-page":"4164","DOI":"10.1038\/s41598-023-31412-2","article-title":"Evaluating the use of large language model in identifying top research questions in gastroenterology","volume":"13","author":"Lahat","year":"2023","journal-title":"Sci Rep"},{"issue":"10","key":"2024062008073569900_ocae100-B25","doi-asserted-by":"crossref","first-page":"104604","DOI":"10.1016\/j.respol.2022.104604","article-title":"Artificial intelligence in science: an emerging general method of invention","volume":"51","author":"Bianchini","year":"2022","journal-title":"Res Policy"},{"issue":"1","key":"2024062008073569900_ocae100-B26","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1038\/s41746-023-00896-7","article-title":"Evaluating large language models on medical evidence summarization","volume":"6","author":"Tang","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"12","key":"2024062008073569900_ocae100-B27","doi-asserted-by":"crossref","first-page":"2983","DOI":"10.1038\/s41591-023-02594-z","article-title":"Large language models should be used as scientific reasoning engines, not knowledge databases","volume":"29","author":"Truhn","year":"2023","journal-title":"Nat Med"},{"key":"2024062008073569900_ocae100-B28","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1186\/1471-2105-8-423","article-title":"PubMed related articles: a probabilistic topic-based model for content similarity","volume":"8","author":"Lin","year":"2007","journal-title":"BMC Bioinform"},{"issue":"2","key":"2024062008073569900_ocae100-B29","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1111\/ijmr.12335","article-title":"Using review articles to address societal grand challenges","volume":"25","author":"Kunisch","year":"2023","journal-title":"Int J Manag Rev"},{"key":"2024062008073569900_ocae100-B30","author":"Zhao","year":"2024"},{"key":"2024062008073569900_ocae100-B31","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei"},{"key":"2024062008073569900_ocae100-B32","first-page":"10776","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2023","author":"Sainz"},{"key":"2024062008073569900_ocae100-B33","doi-asserted-by":"crossref","first-page":"b2680","DOI":"10.1136\/bmj.b2680","article-title":"How citation distortions create unfounded authority: analysis of a citation network","volume":"339","author":"Greenberg","year":"2009","journal-title":"BMJ"},{"issue":"1","key":"2024062008073569900_ocae100-B34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/meet.14505001084","article-title":"Automated citation sentiment analysis: what can we learn from biomedical researchers","volume":"50","author":"Yu","year":"2013","journal-title":"Proc Am Soc Inf Sci Technol"},{"issue":"21","key":"2024062008073569900_ocae100-B35","doi-asserted-by":"crossref","first-page":"2436","DOI":"10.1016\/j.jacc.2020.09.584","article-title":"Ticagrelor or prasugrel in patients with non\u2013ST-segment elevation acute coronary syndromes","volume":"76","author":"Valina","year":"2020","journal-title":"J Am Coll Cardiol"},{"issue":"11","key":"2024062008073569900_ocae100-B36","doi-asserted-by":"crossref","first-page":"780","DOI":"10.1161\/CIRCULATIONAHA.116.023402","article-title":"Pharmacodynamic comparison of prasugrel versus ticagrelor in patients with type 2 diabetes mellitus and coronary artery disease","volume":"134","author":"Franchi","year":"2016","journal-title":"Circulation"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/7\/1551\/58243695\/ocae100.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/7\/1551\/58243695\/ocae100.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,20]],"date-time":"2024-06-20T08:11:26Z","timestamp":1718871086000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/7\/1551\/7676020"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,17]]},"references-count":36,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,5,17]]},"published-print":{"date-parts":[[2024,6,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae100","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,7]]},"published":{"date-parts":[[2024,5,17]]}}}