{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T11:21:05Z","timestamp":1782300065378,"version":"3.54.5"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,8,24]],"date-time":"2023-08-24T00:00:00Z","timestamp":1692835200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,24]],"date-time":"2023-08-24T00:00:00Z","timestamp":1692835200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000092","name":"U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["4R00LM013001"],"award-info":[{"award-number":["4R00LM013001"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["5R01LM009886"],"award-info":[{"award-number":["5R01LM009886"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2145640"],"award-info":[{"award-number":["2145640"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2019844"],"award-info":[{"award-number":["2019844"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study demonstrates that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical evidence summarization. Our findings reveal that LLMs could be susceptible to generating factually inconsistent summaries and making overly convincing or uncertain statements, leading to potential harm due to misinformation. Moreover, we find that models struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts.<\/jats:p>","DOI":"10.1038\/s41746-023-00896-7","type":"journal-article","created":{"date-parts":[[2023,8,24]],"date-time":"2023-08-24T01:01:56Z","timestamp":1692838916000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":354,"title":["Evaluating large language models on medical evidence summarization"],"prefix":"10.1038","volume":"6","author":[{"given":"Liyan","family":"Tang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8197-1465","authenticated-orcid":false,"given":"Zhaoyi","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4318-5987","authenticated-orcid":false,"given":"Betina","family":"Idnay","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1418-3103","authenticated-orcid":false,"given":"Jordan G.","family":"Nestor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6900-5596","authenticated-orcid":false,"given":"Ali","family":"Soroush","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9643-3024","authenticated-orcid":false,"given":"Pierre A.","family":"Elias","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziyang","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ying","family":"Ding","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Greg","family":"Durrett","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2817-9124","authenticated-orcid":false,"given":"Justin F.","family":"Rousseau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9624-0214","authenticated-orcid":false,"given":"Chunhua","family":"Weng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9309-8331","authenticated-orcid":false,"given":"Yifan","family":"Peng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,8,24]]},"reference":[{"key":"896_CR1","unstructured":"Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems Vol. 35 (eds Koyejo, S. et al.) 24824\u201324837 (Curran Associates, Inc., 2022)."},{"key":"896_CR2","unstructured":"Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems Vol. 33 (eds Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H.) 1877\u20131901 (Curran Associates, Inc., 2020)."},{"key":"896_CR3","unstructured":"Chowdhery, A. et al. PaLM: scaling language modeling with pathways. Preprint at https:\/\/arxiv.org\/abs\/2204.02311 (2022)."},{"key":"896_CR4","unstructured":"Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems Vol. 35 (eds Koyejo, S. et al.) 22199\u201322213 (Curran Associates, Inc., 2022)."},{"key":"896_CR5","unstructured":"Ouyang, L. et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems Vol. 35 (eds Koyejo, S. et al.) 27730\u201327744 (Curran Associates, Inc., 2022)."},{"key":"896_CR6","unstructured":"Goyal, T., Li, J. J. & Durrett, G. News summarization and evaluation in the era of GPT-3. Preprint at https:\/\/arxiv.org\/abs\/2209.12356 (2022)."},{"key":"896_CR7","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1038\/s41746-023-00819-6","volume":"6","author":"CA Gao","year":"2023","unstructured":"Gao, C. A. et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. npj Digit. Med. 6, 75 (2023).","journal-title":"npj Digit. Med."},{"key":"896_CR8","doi-asserted-by":"publisher","first-page":"e1001419","DOI":"10.1371\/journal.pmed.1001419","volume":"10","author":"EM Beller","year":"2013","unstructured":"Beller, E. M. et al. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 10, e1001419 (2013).","journal-title":"PLoS Med."},{"key":"896_CR9","unstructured":"OpenAI. Introducing ChatGPT. https:\/\/openai.com\/blog\/chatgpt (2023)."},{"key":"896_CR10","unstructured":"Lin, C.-Y. ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out. 8 74\u201381, Barcelona, Spain (Association for Computational Linguistics, 2004)."},{"key":"896_CR11","unstructured":"Banerjee, S. & Lavie, A. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization 65\u201372, Ann Arbor, Michigan (Association for Computational Linguistics, 2005)."},{"key":"896_CR12","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. BLEU: a method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting on Association for Computational Linguistics 311\u2013318 (Association for Computational Linguistics, 2002).","DOI":"10.3115\/1073083.1073135"},{"key":"896_CR13","doi-asserted-by":"crossref","unstructured":"Grusky, M., Naaman, M. & Artzi, Y. Newsroom: a dataset of 1.3 million summaries with diverse extractive strategies. In Proc. 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1 (Long Papers) 708\u2013719 (Association for Computational Linguistics, 2018).","DOI":"10.18653\/v1\/N18-1065"},{"key":"896_CR14","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1162\/tacl_a_00373","volume":"9","author":"AR Fabbri","year":"2021","unstructured":"Fabbri, A. R. et al. SummEval: re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguist. 9, 391\u2013409 (2021).","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"896_CR15","doi-asserted-by":"crossref","unstructured":"Tang, L. et al. Understanding factual errors in summarization: errors, summarizers, datasets, error detectors. In Proc. 61st Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers) 11626\u201311644 (Association for Computational Linguistics, Toronto, Canada, 2023).","DOI":"10.18653\/v1\/2023.acl-long.650"},{"key":"896_CR16","first-page":"CD013304","volume":"12","author":"V M\u00fchlbauer","year":"2021","unstructured":"M\u00fchlbauer, V. et al. Antipsychotics for agitation and psychosis in people with Alzheimer\u2019s disease and vascular dementia. Cochrane Database Syst. Rev. 12, CD013304 (2021).","journal-title":"Cochrane Database Syst. Rev."},{"key":"896_CR17","first-page":"CD013267","volume":"8","author":"J Luoa","year":"2023","unstructured":"Luoa, J. et al. Endovascular therapy versus medical treatment for symptomatic intracranial artery stenosis. Cochrane Database Syst. Rev. 8, CD013267 (2023).","journal-title":"Cochrane Database Syst. Rev."},{"key":"896_CR18","first-page":"CD004250","volume":"1","author":"A Gross","year":"2015","unstructured":"Gross, A. et al. Exercises for mechanical neck disorders. Cochrane Database Syst. Rev. 1, CD004250 (2015).","journal-title":"Cochrane Database Syst. Rev."},{"key":"896_CR19","first-page":"CD011968","volume":"9","author":"T Kamo","year":"2022","unstructured":"Kamo, T. et al. Repetitive peripheral magnetic stimulation for impairment and disability in people after stroke. Cochrane Database Syst. Rev. 9, CD011968 (2022).","journal-title":"Cochrane Database Syst. Rev."},{"key":"896_CR20","doi-asserted-by":"crossref","unstructured":"Zhang, Y. et al. SummN: a multi-stage summarization framework for long input dialogues and documents. In Proc. 60th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers) 1592\u20131604 (Association for Computational Linguistics, 2022).","DOI":"10.18653\/v1\/2022.acl-long.112"},{"key":"896_CR21","doi-asserted-by":"crossref","unstructured":"Zhang, Y. et al. An exploratory study on long dialogue summarization: what works and what\u2019s next. In Findings of the Association for Computational Linguistics: EMNLP 2021 4426\u20134433 (Association for Computational Linguistics, 2021).","DOI":"10.18653\/v1\/2021.findings-emnlp.377"},{"key":"896_CR22","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023).","journal-title":"Nature"},{"key":"896_CR23","doi-asserted-by":"crossref","unstructured":"Tang, L. et al. EchoGen: generating conclusions from echocardiogram notes. In Proc. 21st Workshop on Biomedical Language Processing 359\u2013368 (Association for Computational Linguistics, 2022).","DOI":"10.18653\/v1\/2022.bionlp-1.35"},{"key":"896_CR24","doi-asserted-by":"crossref","unstructured":"Jeblick, K. et al. ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Preprint at https:\/\/arxiv.org\/abs\/2212.14882 (2022).","DOI":"10.1007\/s00330-023-10213-1"},{"key":"896_CR25","doi-asserted-by":"publisher","first-page":"31","DOI":"10.5032\/jae.1994.04031","volume":"35","author":"D Clason","year":"1994","unstructured":"Clason, D. & Dormody, T. Analyzing data measured by individual Likert-type items. J. Agric. Educ. 35, 31\u201335 (1994).","journal-title":"J. Agric. Educ."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00896-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00896-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00896-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,19]],"date-time":"2023-11-19T10:02:57Z","timestamp":1700388177000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00896-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,24]]},"references-count":25,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["896"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00896-7","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.04.22.23288967","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,24]]},"assertion":[{"value":"25 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 August 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"158"}}