{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T12:20:17Z","timestamp":1775910017399,"version":"3.50.1"},"reference-count":45,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T00:00:00Z","timestamp":1735516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01 HL167974"],"award-info":[{"award-number":["R01 HL167974"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01 HL169345"],"award-info":[{"award-number":["R01 HL169345"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01 AR077604"],"award-info":[{"award-number":["R01 AR077604"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01 EB002524"],"award-info":[{"award-number":["R01 EB002524"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01 AR079431"],"award-info":[{"award-number":["R01 AR079431"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["P41 EB027060"],"award-info":[{"award-number":["P41 EB027060"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["75N92020C00008"],"award-info":[{"award-number":["75N92020C00008"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["75N92020C00021"],"award-info":[{"award-number":["75N92020C00021"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Stanford Center for Artificial Intelligence and Medicine"},{"DOI":"10.13039\/100020670","name":"Stanford Institute for Human Centered AI","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100020670","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Stanford Center for Digital Health"},{"DOI":"10.13039\/100019607","name":"Stanford Cardiovascular Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100019607","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Stanford Center for Precision Health and Integrated Diagnostics"},{"name":"GE Healthcare, Philips and Amazon"},{"name":"ProMedica Foundation"},{"name":"One Medical and Stanford University"},{"name":"Accelerate Foundation Models Academic Research"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Brief hospital course (BHC) summaries are clinical documents that summarize a patient\u2019s hospital stay. While large language models (LLMs) depict remarkable capabilities in automating real-world tasks, their capabilities for healthcare applications such as synthesizing BHCs from clinical notes have not been shown. We introduce a novel preprocessed dataset, the MIMIC-IV-BHC, encapsulating clinical note and BHC pairs to adapt LLMs for BHC synthesis. Furthermore, we introduce a benchmark of the summarization performance of 2 general-purpose LLMs and 3 healthcare-adapted LLMs.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Using clinical notes as input, we apply prompting-based (using in-context learning) and fine-tuning-based adaptation strategies to 3 open-source LLMs (Clinical-T5-Large, Llama2-13B, and FLAN-UL2) and 2 proprietary LLMs (Generative Pre-trained Transformer [GPT]-3.5 and GPT-4). We evaluate these LLMs across multiple context-length inputs using natural language similarity metrics. We further conduct a clinical study with 5 clinicians, comparing clinician-written and LLM-generated BHCs across 30 samples, focusing on their potential to enhance clinical decision-making through improved summary quality. We compare reader preferences for the original and LLM-generated summary using Wilcoxon signed-rank tests. We further request optional qualitative feedback from clinicians to gain deeper insights into their preferences, and we present the frequency of common themes arising from these comments.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The Llama2-13B fine-tuned LLM outperforms other domain-adapted models given quantitative evaluation metrics of Bilingual Evaluation Understudy (BLEU) and Bidirectional Encoder Representations from Transformers (BERT)-Score. GPT-4 with in-context learning shows more robustness to increasing context lengths of clinical note inputs than fine-tuned Llama2-13B. Despite comparable quantitative metrics, the reader study depicts a significant preference for summaries generated by GPT-4 with in-context learning compared to both Llama2-13B fine-tuned summaries and the original summaries (P&amp;lt;.001), highlighting the need for qualitative clinical evaluation.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion and Conclusion<\/jats:title>\n                  <jats:p>We release a foundational clinically relevant dataset, the MIMIC-IV-BHC, and present an open-source benchmark of LLM performance in BHC synthesis from clinical notes. We observe high-quality summarization performance for both in-context proprietary and fine-tuned open-source LLMs using both quantitative metrics and a qualitative clinical reader study. Our research effectively integrates elements from the data assimilation pipeline: our methods use (1) clinical data sources to integrate, (2) data translation, and (3) knowledge creation, while our evaluation strategy paves the way for (4) deployment.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae312","type":"journal-article","created":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T20:14:21Z","timestamp":1735589661000},"page":"470-479","source":"Crossref","is-referenced-by-count":23,"title":["A dataset and benchmark for hospital course summarization with adapted large language models"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-2120-5722","authenticated-orcid":false,"given":"Asad","family":"Aali","sequence":"first","affiliation":[{"name":"Department of Radiology, Stanford University , Stanford, CA 94304,","place":["United States"]},{"name":"Department of Electrical and Computer Engineering, The University of Texas at Austin , Austin, TX 78712,","place":["United States"]}]},{"given":"Dave","family":"Van Veen","sequence":"additional","affiliation":[{"name":"Center for Artificial Intelligence in Medicine and Imaging, Stanford University , Palo Alto, CA 94304,","place":["United States"]},{"name":"Department of Electrical Engineering, Stanford University , Stanford, CA 94304,","place":["United States"]}]},{"given":"Yamin Ishraq","family":"Arefeen","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, The University of Texas at Austin , Austin, TX 78712,","place":["United States"]}]},{"given":"Jason","family":"Hom","sequence":"additional","affiliation":[{"name":"Department of Medicine, Stanford University , Stanford, CA 94304,","place":["United States"]}]},{"given":"Christian","family":"Bluethgen","sequence":"additional","affiliation":[{"name":"Department of Medicine, Stanford University , Stanford, CA 94304,","place":["United States"]},{"name":"University Hospital Zurich , Zurich 8091,","place":["Switzerland"]}]},{"given":"Eduardo Pontes","family":"Reis","sequence":"additional","affiliation":[{"name":"Center for Artificial Intelligence in Medicine and Imaging, Stanford University , Palo Alto, CA 94304,","place":["United States"]},{"name":"Albert Einstein Israelite Hospital , S\u00e3o Paulo 05652-900,","place":["Brazil"]}]},{"given":"Sergios","family":"Gatidis","sequence":"additional","affiliation":[{"name":"Department of Radiology, Stanford University , Stanford, CA 94304,","place":["United States"]},{"name":"Center for Artificial Intelligence in Medicine and Imaging, Stanford University , Palo Alto, CA 94304,","place":["United States"]}]},{"given":"Namuun","family":"Clifford","sequence":"additional","affiliation":[{"name":"School of Nursing, The University of Texas at Austin , Austin, TX 78712,","place":["United States"]}]},{"given":"Joseph","family":"Daws","sequence":"additional","affiliation":[{"name":"One Medical , San Francisco, CA 94111,","place":["United States"]}]},{"given":"Arash S","family":"Tehrani","sequence":"additional","affiliation":[{"name":"One Medical , San Francisco, CA 94111,","place":["United States"]}]},{"given":"Jangwon","family":"Kim","sequence":"additional","affiliation":[{"name":"Amazon , Seattle, WA 98109,","place":["United States"]}]},{"given":"Akshay S","family":"Chaudhari","sequence":"additional","affiliation":[{"name":"Department of Radiology, Stanford University , Stanford, CA 94304,","place":["United States"]},{"name":"Center for Artificial Intelligence in Medicine and Imaging, Stanford University , Palo Alto, CA 94304,","place":["United States"]},{"name":"Department of Biomedical Data Science, Stanford University , Stanford, CA 94304,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2024,12,30]]},"reference":[{"key":"2025021811394929600_ocae312-B1","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1093\/jamia\/ocaa325","article-title":"Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review","volume":"28","author":"Moy","year":"2021","journal-title":"J Am Med Inform Assoc."},{"key":"2025021811394929600_ocae312-B2","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1001\/jamainternmed.2019.0095","article-title":"Assessment of inpatient time allocation among first-year internal medicine residents using time-motion observations","volume":"179","author":"Chaiyachati","year":"2019","journal-title":"JAMA Intern Med."},{"key":"2025021811394929600_ocae312-B3","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1097\/ACM.0000000000001148","article-title":"How do residents spend their shift time? A time and motion study with a particular focus on the use of computers","volume":"91","author":"Mamykina","year":"2016","journal-title":"Acad Med."},{"key":"2025021811394929600_ocae312-B4","author":"Albers","year":"2020"},{"key":"2025021811394929600_ocae312-B5","doi-asserted-by":"crossref","DOI":"10.3399\/BJGPO.2023.0116","article-title":"Transforming healthcare documentation: harnessing the potential of AI to generate discharge summaries","volume":"8","author":"Clough","year":"2024","journal-title":"BJGP Open."},{"key":"2025021811394929600_ocae312-B6","doi-asserted-by":"crossref","first-page":"831","DOI":"10.1001\/jama.297.8.831","article-title":"Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care","volume":"297","author":"Kripalani","year":"2007","journal-title":"JAMA"},{"key":"2025021811394929600_ocae312-B7","doi-asserted-by":"crossref","first-page":"e107","DOI":"10.1016\/S2589-7500(23)00021-3","article-title":"ChatGPT: the future of discharge summaries?","volume":"5","author":"Patel","year":"2023","journal-title":"Lancet Digit Health"},{"key":"2025021811394929600_ocae312-B8","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2025021811394929600_ocae312-B9","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med."},{"key":"2025021811394929600_ocae312-B10","first-page":"1443","article-title":"FDA perspective on the regulation of artificial intelligence in health care and biomedicine","volume":"331","author":"Warraich","year":"2024","journal-title":"JAMA"},{"key":"2025021811394929600_ocae312-B11","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025021811394929600_ocae312-B12","first-page":"4171","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","volume":"1","author":"Devlin","year":"2019","journal-title":"Proc Conf North Am Chapter Assoc Comput Linguist"},{"key":"2025021811394929600_ocae312-B13","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2025021811394929600_ocae312-B14","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025021811394929600_ocae312-B15","first-page":"1","article-title":"Palm: scaling language modeling with pathways","volume":"24","author":"Chowdhery","year":"2023","journal-title":"J Mach Learn Res"},{"key":"2025021811394929600_ocae312-B16","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J Mach Learn Res"},{"key":"2025021811394929600_ocae312-B17","author":"Zhang"},{"key":"2025021811394929600_ocae312-B18","first-page":"1","article-title":"Pre-trained language models in biomedical domain: a systematic survey","volume":"56","author":"Wang","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2025021811394929600_ocae312-B19","first-page":"4794","article-title":"What\u2019s in a summary? Laying the groundwork for advances in hospital-course summarization","volume":"2021","author":"Adams","year":"2021","journal-title":"Proc Conf North Am Chapter Assoc Comput Linguist"},{"key":"2025021811394929600_ocae312-B20","doi-asserted-by":"crossref","first-page":"104358","DOI":"10.1016\/j.jbi.2023.104358","article-title":"Discharge summary hospital course summarisation of in patient electronic health record text with clinical concept guided deep pre-trained transformer models","volume":"141","author":"Searle","year":"2023","journal-title":"J Biomed Inform."},{"key":"2025021811394929600_ocae312-B21","doi-asserted-by":"crossref","first-page":"1995","DOI":"10.1093\/jamia\/ocad177","article-title":"A method to automate the discharge summary hospital course for neurology patients","volume":"30","author":"Hartman","year":"2023","journal-title":"J Am Med Inform Assoc."},{"key":"2025021811394929600_ocae312-B22","author":"Jung"},{"key":"2025021811394929600_ocae312-B23","doi-asserted-by":"publisher","DOI":"10.13026\/fh2q-4148","article-title":"MIMIC-IV-Ext-BHC: labeled clinical notes dataset for hospital course summarization","author":"Aali","year":"2024","journal-title":"PhysioNet"},{"key":"2025021811394929600_ocae312-B24","article-title":"MIMIC-IV-Note: deidentified free-text clinical notes","author":"Johnson","year":"2023","journal-title":"PhysioNet"},{"key":"2025021811394929600_ocae312-B25","article-title":"Clinical-t5: large language models built using mimic clinical text","author":"Lehman","year":"2023","journal-title":"PhysioNet"},{"key":"2025021811394929600_ocae312-B26","author":"Lampinen"},{"key":"2025021811394929600_ocae312-B27","author":"Touvron"},{"key":"2025021811394929600_ocae312-B28","author":"OpenAI","year":"2022"},{"key":"2025021811394929600_ocae312-B29","author":"OpenAI"},{"key":"2025021811394929600_ocae312-B30","author":"Tay"},{"key":"2025021811394929600_ocae312-B31","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025021811394929600_ocae312-B32","author":"Hu"},{"key":"2025021811394929600_ocae312-B33","article-title":"QLoRA: efficient finetuning of quantized LLMs","volume":"36","author":"Dettmers","year":"2024","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025021811394929600_ocae312-B34","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1038\/s42256-023-00626-4","article-title":"Parameter-efficient fine-tuning of large-scale pre-trained language models","volume":"5","author":"Ding","year":"2023","journal-title":"Nat Mach Intell."},{"key":"2025021811394929600_ocae312-B35","first-page":"311","article-title":"BLEU: a method for automatic evaluation of machine translation","volume":"40","author":"Papineni","year":"2002","journal-title":"Assoc Comput Linguist"},{"key":"2025021811394929600_ocae312-B36","first-page":"74","article-title":"ROUGE: a package for automatic evaluation of summaries","volume":"1","author":"Lin","year":"2004","journal-title":"Assoc Comput Linguist"},{"key":"2025021811394929600_ocae312-B37","first-page":"187","article-title":"Exploring correlation between ROUGE and human evaluation on meeting summaries","volume":"18","author":"Liu","year":"2009","journal-title":"IEEE Trans Audio Speech Lang Process"},{"key":"2025021811394929600_ocae312-B38","author":"Zhang"},{"key":"2025021811394929600_ocae312-B39","first-page":"449","author":"Van Veen"},{"key":"2025021811394929600_ocae312-B40","first-page":"469","author":"Chen"},{"key":"2025021811394929600_ocae312-B41","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1186\/s42492-023-00136-5","article-title":"Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential","volume":"6","author":"Lyu","year":"2023","journal-title":"Vis Comput Ind Biomed Art."},{"key":"2025021811394929600_ocae312-B42","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1080\/08820538.2023.2209166","article-title":"ChatGPT and ophthalmology: exploring its potential with discharge summaries and operative notes","volume":"38","author":"Singh","year":"2023","journal-title":"Semin Ophthalmol."},{"key":"2025021811394929600_ocae312-B43","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1038\/s41746-023-00989-3","article-title":"DRG-LLaMA: tuning LLaMA model to predict diagnosis-related group for hospitalized patients","volume":"7","author":"Wang","year":"2024","journal-title":"NPJ Digit Med."},{"key":"2025021811394929600_ocae312-B44","first-page":"2682","author":"Koh"},{"key":"2025021811394929600_ocae312-B45","doi-asserted-by":"crossref","first-page":"22021","DOI":"10.1609\/aaai.v38i20.30205","article-title":"MedAlign: a clinician-generated dataset for instruction following with electronic medical records","volume":"38","author":"Fleming","year":"2024","journal-title":"AAAI."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/3\/470\/61297015\/ocae312.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/3\/470\/61297015\/ocae312.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,18]],"date-time":"2025-02-18T11:40:21Z","timestamp":1739878821000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/3\/470\/7934937"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,30]]},"references-count":45,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,12,30]]},"published-print":{"date-parts":[[2025,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae312","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,3]]},"published":{"date-parts":[[2024,12,30]]}}}