{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T20:59:03Z","timestamp":1791406743706,"version":"4.3.3"},"reference-count":36,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T00:00:00Z","timestamp":1781654400000},"content-version":"vor","delay-in-days":43,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100004319","name":"Pfizer Inc","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100004319","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,5,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>This study evaluated the quality and trustworthiness of large language model (LLM)-generated scientific and plain language summaries (PLS) from clinical oncology literature, focusing on faithfulness (absence of hallucinations), relevance, and readability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Ten LLM-generated scientific summaries and PLS from the INSIDE (artificial INtelligence to Support Informed DEcision making) prostate cancer dataset. For comparison, expert-written PLS from the BioLaySumm dataset were used. A panel of 5 LLMs and 3 human experts verified faithfulness. Verification was performed on original facts and facts modified with varying levels of error (subtle, moderate, contradictory). Readability was assessed using Flesch-Kincaid Reading Ease (FRE) scores.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Fact verification against the summaries was \u223c100%, confirming accurate fact extraction. LLM panel vs human panel agreement was substantial (kappa 0.67), outperforming agreement among the interhuman (0.43 [95% CI, 0.34\u20130.52]) and inter-LLM (0.40 [0.38\u20130.42]) panels. Large language model scientific summaries showed high faithfulness (88.9% [88.0\u201389.8]) and low hallucinations (9.6% [6.5\u201312.7]) compared to human-written PLS (61.6% [60.1\u201363.1] faithfulness; 40.6% [37.8\u2013 43.4] hallucinations). The LLMs detected errors sensitively with scores decreasing as fact modifications became more severe. Finally, LLM-generated PLS were more readable than human-written versions (FRE 42.3 [interquartile range, IQR 35.27\u201349.41] vs 28.8 [IQR 21.02\u201336.18]).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>A panel of LLMs reliably assessed the faithfulness of scientific summaries to their original source and thus can help increase reliability for clinical use. The lower faithfulness in human-written PLS likely reflects extrinsic hallucinations added for context.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The study demonstrates a novel approach to automatically assess the quality and trustworthiness of LLM-generated scientific and PLS via faithfulness, relevance, and readability.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooag078","type":"journal-article","created":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T11:22:15Z","timestamp":1778498535000},"source":"Crossref","is-referenced-by-count":1,"title":["Enhancing the quality and trustworthiness of large language model-generated summaries of clinical oncology literature"],"prefix":"10.1093","volume":"9","author":[{"given":"Arnulf","family":"Stenzl","sequence":"first","affiliation":[{"name":"Department of Urology, University of T\u00fcbingen , T\u00fcbingen 72076,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eamonn","family":"Rogers","sequence":"additional","affiliation":[{"name":"Department of Urology, University College Hospital , Galway H91 YR71,","place":["Ireland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sophia","family":"Ananiadou","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Manchester , Manchester M13 9PL,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanshan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Health Information Management, University of Pittsburgh , Pittsburgh, PA 15260,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew J","family":"Armstrong","sequence":"additional","affiliation":[{"name":"Departments of Medicine, Urology, Pharmacology, and Cancer Biology, Duke Cancer Institute Center for Prostate and Urologic Cancers, Duke University , Durham, NC 27710,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Sboner","sequence":"additional","affiliation":[{"name":"Department of Pathology and Laboratory Medicine, Weill Cornell Medicine , New York, NY 10065,","place":["United States"]},{"name":"Institute for Computational Biomedicine, Weill Cornell Medicine , New York, NY 10021,","place":["United States"]},{"name":"Meyer Cancer Center, Weill Cornell Medicine , New York, NY 10065,","place":["United States"]},{"name":"Englander Institute for Precision Medicine, Weill Cornell Medicine , New York, NY 10021,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giovanni","family":"Cacciamani","sequence":"additional","affiliation":[{"name":"Keck School of Medicine, University of Southern California , Los Angeles, CA 90033,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bob J A","family":"Schijvenaars","sequence":"additional","affiliation":[{"name":"Digital Science , London, EC1M 5NR,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kausar Riaz","family":"Ahmed","sequence":"additional","affiliation":[{"name":"Pfizer Inc. , Silver Spring, MD 20910,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hanna","family":"Thomsen","sequence":"additional","affiliation":[{"name":"Pfizer Inc. , Bothell, WA 98021,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Timothy","family":"Wiemken","sequence":"additional","affiliation":[{"name":"Pfizer Inc. , New York, NY 10001,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Antonio","family":"Campello","sequence":"additional","affiliation":[{"name":"Digital Science , London, EC1M 5NR,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3938-2627","authenticated-orcid":false,"given":"Cora N","family":"Sternberg","sequence":"additional","affiliation":[{"name":"Englander Institute for Precision Medicine, Weill Cornell Medicine , New York, NY 10021,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,6,16]]},"reference":[{"key":"2026061717334791400_ooag078-B1","doi-asserted-by":"crossref","first-page":"e54345","DOI":"10.2196\/54345","article-title":"Reference hallucination score for medical artificial intelligence chatbots: development and usability study","volume":"12","author":"Aljamaan","year":"2024","journal-title":"JMIR Med Inform."},{"key":"2026061717334791400_ooag078-B2","doi-asserted-by":"crossref","first-page":"689","DOI":"10.1186\/s12909-023-04698-z","article-title":"Revolutionizing healthcare: the role of artificial intelligence in clinical practice","volume":"23","author":"Alowais","year":"2023","journal-title":"BMC Med Educ"},{"key":"2026061717334791400_ooag078-B3","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1111\/bju.15662","article-title":"Application of artificial intelligence to overcome clinical information overload in urological cancer","volume":"130","author":"Stenzl","year":"2022","journal-title":"BJU Int"},{"key":"2026061717334791400_ooag078-B4","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.1016\/j.euf.2024.05.022","article-title":"Artificial INtelligence to support informed DEcision-making (inside) for improved literature analysis in oncology","volume":"10","author":"Stenzl","year":"2024","journal-title":"Eur Urol Focus"},{"key":"2026061717334791400_ooag078-B5","doi-asserted-by":"crossref","first-page":"100687","DOI":"10.1016\/j.modpat.2024.100687","article-title":"Generative artificial intelligence in pathology and medicine: a deeper dive","volume":"38","author":"Rashidi","year":"2025","journal-title":"Mod Pathol"},{"key":"2026061717334791400_ooag078-B6","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"2026061717334791400_ooag078-B7","doi-asserted-by":"crossref","first-page":"e59823","DOI":"10.2196\/59823","article-title":"The clinicians\u2019 guide to large language models: a general perspective with a focus on hallucinations","volume":"14","author":"Roustan","year":"2025","journal-title":"Interact J Med Res."},{"key":"2026061717334791400_ooag078-B8","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1186\/s12916-019-1426-2","article-title":"Key challenges for delivering clinical impact with artificial intelligence","volume":"17","author":"Kelly","year":"2019","journal-title":"BMC Med"},{"key":"2026061717334791400_ooag078-B9","doi-asserted-by":"publisher","author":"Kim","year":"2025","DOI":"10.1101\/2025.02.28.25323115"},{"key":"2026061717334791400_ooag078-B10","doi-asserted-by":"publisher","author":"Goldsack","year":"2024","DOI":"10.18653\/v1\/2024.bionlp-1.10"},{"key":"2026061717334791400_ooag078-B11","doi-asserted-by":"crossref","first-page":"110382","DOI":"10.1016\/j.clinimag.2024.110382","article-title":"Comparative diagnostic accuracy of GPT-4o and LLaMA 3-70b: proprietary vs open-source large language models in radiology","volume":"118","author":"Li","year":"2025","journal-title":"Clin Imaging"},{"key":"2026061717334791400_ooag078-B12","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.jaci.2025.02.004","article-title":"Evaluating large language model performance to support the diagnosis and management of patients with primary immune disorders","volume":"156","author":"Rider","year":"2025","journal-title":"J Allergy Clin Immunol"},{"key":"2026061717334791400_ooag078-B13","doi-asserted-by":"publisher","author":"He","DOI":"10.48550\/arXiv.2111.09543"},{"key":"2026061717334791400_ooag078-B14","doi-asserted-by":"crossref","first-page":"799","DOI":"10.1186\/s12885-023-11325-z","article-title":"Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters","volume":"23","author":"Li","year":"2023","journal-title":"BMC Cancer"},{"key":"2026061717334791400_ooag078-B15","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem Med (Zagreb)"},{"key":"2026061717334791400_ooag078-B16","year":"2025"},{"key":"2026061717334791400_ooag078-B17","author":"Kincaid","year":"2025"},{"key":"2026061717334791400_ooag078-B18","doi-asserted-by":"crossref","first-page":"84","DOI":"10.4103\/1357-6283.210517","article-title":"Assessing reading levels of health information: uses and limitations of Flesch formula","volume":"30","author":"Jindal","year":"2017","journal-title":"Educ Health (Abingdon)"},{"key":"2026061717334791400_ooag078-B19","doi-asserted-by":"crossref","first-page":"ooaf082","DOI":"10.1093\/jamiaopen\/ooaf082","article-title":"Comparing AI- versus clinician-authored summaries of simulated primary care electronic health records","volume":"8","author":"Shemtob","year":"2025","journal-title":"JAMIA Open"},{"key":"2026061717334791400_ooag078-B20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1056\/AIra2401164","article-title":"The use of artificial intelligence for cancer therapeutic decision-making","volume":"2","author":"Elemento","year":"2025","journal-title":"NEJM AI"},{"key":"2026061717334791400_ooag078-B21","doi-asserted-by":"crossref","first-page":"ooaf023","DOI":"10.1093\/jamiaopen\/ooaf023","article-title":"Using artificial intelligence to expedite and enhance plain language summary abstract writing of scientific content","volume":"8","author":"McMinn","year":"2025","journal-title":"JAMIA Open"},{"key":"2026061717334791400_ooag078-B22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1056\/AIoa2300151","article-title":"Comparative evaluation of LLMs in clinical oncology","volume":"1","author":"Rydzewski","year":"2024","journal-title":"NEJM AI"},{"key":"2026061717334791400_ooag078-B23","doi-asserted-by":"crossref","first-page":"101593","DOI":"10.1016\/j.jacadv.2025.101593","article-title":"Embracing generative artificial intelligence in clinical research and beyond: opportunities, challenges, and solutions","volume":"4","author":"Foote","year":"2025","journal-title":"JACC Adv"},{"key":"2026061717334791400_ooag078-B24","doi-asserted-by":"publisher","author":"Tonmoy","year":"2024","DOI":"10.48550\/arXiv.2401.01313"},{"key":"2026061717334791400_ooag078-B25","author":"Vishwanath","year":"2024"},{"key":"2026061717334791400_ooag078-B26","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1038\/s41586-024-07421-0","article-title":"Detecting hallucinations in large language models using semantic entropy","volume":"630","author":"Farquhar","year":"2024","journal-title":"Nature"},{"key":"2026061717334791400_ooag078-B27","doi-asserted-by":"publisher","author":"Ah","year":"2023","DOI":"10.48550\/arXiv.2311.01463"},{"key":"2026061717334791400_ooag078-B28","first-page":"11588","article-title":"Emergence of artificial generative intelligence and its potential impact on urology","volume":"30","author":"Javid","year":"2023","journal-title":"Can J Urol"},{"key":"2026061717334791400_ooag078-B29","doi-asserted-by":"publisher","author":"Rawte","year":"2023","DOI":"10.18653\/v1\/2023.emnlp-main.155"},{"key":"2026061717334791400_ooag078-B30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2026061717334791400_ooag078-B31","doi-asserted-by":"crossref","first-page":"110307","DOI":"10.1016\/j.compeleceng.2025.110307","article-title":"The paradigm of hallucinations in AI-driven cybersecurity systems: understanding taxonomy, classification outcomes, and mitigations","volume":"124","author":"Sood","year":"2025","journal-title":"Comput Electr Eng"},{"key":"2026061717334791400_ooag078-B32","doi-asserted-by":"publisher","author":"Zhang","year":"2019","DOI":"10.48550\/arXiv.1904.09675"},{"key":"2026061717334791400_ooag078-B33","doi-asserted-by":"publisher","author":"Papineni","year":"2002","DOI":"10.3115\/1073083.1073135"},{"key":"2026061717334791400_ooag078-B34","author":"Lin","year":"2004"},{"key":"2026061717334791400_ooag078-B35","doi-asserted-by":"crossref","first-page":"100070","DOI":"10.1016\/j.nlp.2024.100070","article-title":"A survey of text summarization: techniques, evaluation and challenges","volume":"7","author":"Wibawa","year":"2024","journal-title":"Nat Lang Process J"},{"key":"2026061717334791400_ooag078-B36","doi-asserted-by":"crossref","first-page":"e27725","DOI":"10.7554\/eLife.27725","article-title":"The readability of scientific texts is decreasing over time","volume":"6","author":"Plav\u00e9n-Sigray","year":"2017","journal-title":"Elife."}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/3\/ooag078\/68542923\/ooag078.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/3\/ooag078\/68542923\/ooag078.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T21:33:58Z","timestamp":1781732038000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooag078\/8709993"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,5]]},"references-count":36,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,5,5]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooag078","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,6]]},"published":{"date-parts":[[2026,5,5]]},"article-number":"ooag078"}}