{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T18:20:10Z","timestamp":1787854810532,"version":"build-2784847793"},"reference-count":46,"publisher":"Public Library of Science (PLoS)","issue":"8","license":[{"start":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T00:00:00Z","timestamp":1787788800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100008935","name":"Center of Excellence in Regulatory Science and Innovation","doi-asserted-by":"publisher","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}],"id":[{"id":"10.13039\/100008935","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["P30CA082103"],"award-info":[{"award-number":["P30CA082103"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Chimeric Antigen Receptor T-cell (CAR\u2011T) therapy, genetically engineered patient T cells targeting tumor antigens, has transformed care for hematologic malignancies but requires careful tracking of adverse events (AEs) often documented only in unstructured electronic health record (EHR)\u00a0notes. We evaluated a Large Language Model (LLM)\u2013based approach in UCSF\u2019s secure environment to extract AEs, dates, grades, and interventions within 30 days post\u2011infusion for six commercial CAR\u2011T products (2012\u20132023), benchmarking against two evaluators. Using GPT\u20114\u20110314 in a zero\u2011shot setting with four prompts (prespecified AEs, non\u2011prespecified AEs, CRS, ICANS), we compared outputs against dual annotations on a random sample of 50 notes using accuracy, precision, recall, F1, and Cohen\u2019s kappa. From 4,762 progress notes for 293 patients (median age 65.6), CRS occurred in 80.2% (median onset 4 days); neutropenia 70.0% (16 days); neutropenic fever 64.8% (4 days); ICANS in 34.8%. Interventions included tocilizumab and corticosteroids. Grades were frequently undocumented (CRS 62.3%, ICANS 56.1%); documented cases were mainly CRS grade 1 (59.4%) and ICANS grade 2 (28.0%). Performance was high on CRS and ICANS grading (accuracy of 0.97 and 0.91, respectively). Moderate performances were assessed for prespecified AE extraction (accuracies 0.62\u20130.76), and non\u2011prespecified AEs (accuracies 0.76\u20130.84). Inter\u2011rater reliability (IRR) was strong for CRS\/ICANS presence and grade (kappa 0.86\u20130.96), moderate for dates and interventions, and weaker for broader AE attributes. LLM\u2011derived insights can augment AE monitoring and real\u2011world evidence generation by unlocking unstructured clinical detail and characteristic timelines after CAR T. However, performance varied for broader AE attributes, warranting cautious use. Performance was highest for detecting and grading CRS and ICANS, with strong to near-perfect IRR. While cautious use of LLMs is warranted due to the variable performance observed in this study, these results support further evaluation of supervised CRS\/ICANS extraction in controlled EHR-based research or pilot settings, with monitoring and prospective validation.<\/jats:p>","DOI":"10.1371\/journal.pdig.0001426","type":"journal-article","created":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T17:33:32Z","timestamp":1787852012000},"page":"e0001426","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Extracting adverse event nature, severity, timelines and resulting interventions from clinical notes of patients receiving CAR-T cell therapy using large language models"],"prefix":"10.1371","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9091-0571","authenticated-orcid":true,"given":"Jordan","family":"Guillot","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brenda","family":"Miao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arvind","family":"Suresh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Madhumita","family":"Sushil","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher Y. K.","family":"Williams","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rohit","family":"Vashisht","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tomiko T.","family":"Oskotsky","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marina","family":"Sirota","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Atul J.","family":"Butte","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,8,27]]},"reference":[{"issue":"26","key":"pdig.0001426.ref001","doi-asserted-by":"crossref","first-page":"2531","DOI":"10.1056\/NEJMoa1707447","article-title":"Axicabtagene ciloleucel CAR T-cell therapy in refractory large B-cell lymphoma","volume":"377","author":"SS Neelapu","year":"2017","journal-title":"N Engl J Med"},{"issue":"5","key":"pdig.0001426.ref002","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1056\/NEJMoa1709866","article-title":"Tisagenlecleucel in Children and Young Adults with B-Cell Lymphoblastic Leukemia","volume":"378","author":"SL Maude","year":"2018","journal-title":"N Engl J Med"},{"issue":"1","key":"pdig.0001426.ref003","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1056\/NEJMoa1804980","article-title":"Tisagenlecleucel in Adult Relapsed or Refractory Diffuse Large B-Cell Lymphoma","volume":"380","author":"SJ Schuster","year":"2019","journal-title":"N Engl J Med"},{"issue":"10297","key":"pdig.0001426.ref004","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1016\/S0140-6736(21)00933-8","article-title":"Ciltacabtagene autoleucel, a B-cell maturation antigen-directed chimeric antigen receptor T-cell therapy in patients with relapsed or refractory multiple myeloma (CARTITUDE-1): a phase 1b\/2 open-label study","volume":"398","author":"JG Berdeja","year":"2021","journal-title":"Lancet"},{"issue":"7655","key":"pdig.0001426.ref005","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1038\/nature22395","article-title":"Therapeutic T cell engineering","volume":"545","author":"M Sadelain","year":"2017","journal-title":"Nature"},{"issue":"22","key":"pdig.0001426.ref006","doi-asserted-by":"crossref","first-page":"1924","DOI":"10.1001\/jama.2024.19462","article-title":"CAR T Cells and T-Cell Therapies for Cancer: A Translational Science Review","volume":"332","author":"JN Brudno","year":"2024","journal-title":"JAMA"},{"key":"pdig.0001426.ref007","first-page":"21","article-title":"CAR T-Cell Therapy: Adverse Events and Management","volume":"10","author":"S Adkins","year":"2019","journal-title":"J Adv Pract Oncol"},{"issue":"1","key":"pdig.0001426.ref008","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1186\/s12929-023-00982-8","article-title":"Complete spectrum of adverse events associated with chimeric antigen receptor (CAR)-T cell therapies","volume":"30","author":"C Yang","year":"2023","journal-title":"J Biomed Sci"},{"issue":"26","key":"pdig.0001426.ref009","doi-asserted-by":"crossref","first-page":"3321","DOI":"10.1182\/blood-2016-04-703751","article-title":"Toxicities of chimeric antigen receptor T cells: recognition and management","volume":"127","author":"JN Brudno","year":"2016","journal-title":"Blood"},{"issue":"6","key":"pdig.0001426.ref010","doi-asserted-by":"crossref","DOI":"10.1016\/S2352-3026(24)00077-2","article-title":"Immune effector cell-associated haematotoxicity after CAR T-cell therapy: from mechanism to management","volume":"11","author":"K Rejeski","year":"2024","journal-title":"Lancet Haematol"},{"issue":"1","key":"pdig.0001426.ref011","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1186\/s40425-018-0343-9","article-title":"Cytokine release syndrome","volume":"6","author":"A Shimabukuro-Vornhagen","year":"2018","journal-title":"J Immunother Cancer"},{"issue":"4","key":"pdig.0001426.ref012","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1016\/j.bbmt.2018.12.758","article-title":"ASTCT Consensus Grading for Cytokine Release Syndrome and Neurologic Toxicity Associated with Immune Effector Cells","volume":"25","author":"DW Lee","year":"2019","journal-title":"Biol Blood Marrow Transplant"},{"issue":"1","key":"pdig.0001426.ref013","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1038\/nrclinonc.2017.148","article-title":"Chimeric antigen receptor T-cell therapy - assessment and management of toxicities","volume":"15","author":"SS Neelapu","year":"2018","journal-title":"Nat Rev Clin Oncol"},{"issue":"2","key":"pdig.0001426.ref014","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1038\/s41408-022-00629-1","article-title":"Incidence and management of CAR-T neurotoxicity in patients with multiple myeloma treated with ciltacabtagene autoleucel in CARTITUDE studies","volume":"12","author":"AD Cohen","year":"2022","journal-title":"Blood Cancer J"},{"issue":"6","key":"pdig.0001426.ref015","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1158\/2159-8290.CD-16-0040","article-title":"Identification of Predictive Biomarkers for Cytokine Release Syndrome after Chimeric Antigen Receptor T-cell Therapy for Acute Lymphoblastic Leukemia","volume":"6","author":"DT Teachey","year":"2016","journal-title":"Cancer Discov"},{"key":"pdig.0001426.ref016","doi-asserted-by":"crossref","first-page":"103971","DOI":"10.1016\/j.ijmedinf.2019.103971","article-title":"A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis","volume":"132","author":"IJB Young","year":"2019","journal-title":"Int J Med Inform"},{"issue":"4","key":"pdig.0001426.ref017","article-title":"Extraction of Information Related to Adverse Drug Events from Electronic Health Record Notes: Design of an End-to-End Model Based on Deep Learning","volume":"6","author":"F Li","year":"2018","journal-title":"JMIR Med Inform"},{"key":"pdig.0001426.ref018","doi-asserted-by":"crossref","first-page":"106649","DOI":"10.1016\/j.compbiomed.2023.106649","article-title":"Natural Language Processing in Electronic Health Records in relation to healthcare decision-making: A systematic review","volume":"155","author":"E Hossain","year":"2023","journal-title":"Comput Biol Med"},{"key":"pdig.0001426.ref019","article-title":"GPT-4 Technical Report [Internet]","author":"OpenAI, Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, et al","year":"2024","journal-title":"arXiv"},{"key":"pdig.0001426.ref020","first-page":"1877","volume-title":"Language Models are Few-Shot Learners. In: Advances in Neural Information Processing Systems [Internet]","author":"T Brown","year":"2020"},{"key":"pdig.0001426.ref021","article-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","author":"J Devlin","year":"2019","journal-title":"arXiv"},{"key":"pdig.0001426.ref022","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1038\/s41746-018-0029-1","article-title":"Scalable and accurate deep learning with electronic health records","volume":"1","author":"A Rajkomar","year":"2018","journal-title":"NPJ Digit Med"},{"issue":"13","key":"pdig.0001426.ref023","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1001\/jama.2017.18391","article-title":"Big Data and Machine Learning in Health Care","volume":"319","author":"AL Beam","year":"2018","journal-title":"JAMA"},{"key":"pdig.0001426.ref024","article-title":"ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission","author":"K Huang","year":"2020","journal-title":"arXiv"},{"key":"pdig.0001426.ref025","doi-asserted-by":"crossref","first-page":"104603","DOI":"10.1016\/j.jbi.2024.104603","article-title":"Extracting adverse drug events from clinical Notes: A systematic review of approaches used","volume":"151","author":"S Modi","year":"2024","journal-title":"J Biomed Inform"},{"issue":"9","key":"pdig.0001426.ref026","doi-asserted-by":"crossref","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Y Hu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"pdig.0001426.ref027","doi-asserted-by":"crossref","first-page":"67","DOI":"10.3310\/nihropenres.13504.2","article-title":"Natural language processing for detecting adverse drug events: A systematic review protocol","volume":"3","author":"I Guellil","year":"2024","journal-title":"NIHR Open Res"},{"issue":"3","key":"pdig.0001426.ref028","doi-asserted-by":"crossref","DOI":"10.1093\/jamiaopen\/ooad045","article-title":"A certified de-identification system for all clinical text documents for information extraction at scale","volume":"6","author":"L Radhakrishnan","year":"2023","journal-title":"JAMIA Open"},{"key":"pdig.0001426.ref029","doi-asserted-by":"crossref","first-page":"6086","DOI":"10.1038\/s41598-024-56706-x","article-title":"Evaluation metrics and statistical tests for machine learning","volume":"14","author":"O Rainio","year":"2024","journal-title":"Sci Rep"},{"issue":"3","key":"pdig.0001426.ref030","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"ML McHugh","year":"2012","journal-title":"Biochem Med (Zagreb)"},{"issue":"1","key":"pdig.0001426.ref031","doi-asserted-by":"crossref","DOI":"10.1136\/bmjhci-2024-101139","article-title":"Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation","volume":"32","author":"V Ntinopoulos","year":"2025","journal-title":"BMJ Health Care Inform"},{"issue":"7","key":"pdig.0001426.ref032","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pdig.0000919","article-title":"Using large language models to extract information from pediatric clinical reports","volume":"4","author":"K Danhauser","year":"2025","journal-title":"PLOS Digit Health"},{"key":"pdig.0001426.ref033","doi-asserted-by":"crossref","DOI":"10.2196\/65454","article-title":"Classifying Unstructured Text in Electronic Health Records for Mental Health Prediction Models: Large Language Model Evaluation Study","volume":"13","author":"NC Cardamone","year":"2025","journal-title":"JMIR Med Inform"},{"issue":"4","key":"pdig.0001426.ref034","doi-asserted-by":"crossref","first-page":"1505","DOI":"10.1007\/s10620-025-09539-1","article-title":"Large Language Models can Identify the Presence of MASH and Extract VCTE Measurements from Unstructured Documentation","volume":"71","author":"AT Far","year":"2025","journal-title":"Dig Dis Sci"},{"issue":"6","key":"pdig.0001426.ref035","doi-asserted-by":"crossref","first-page":"1391","DOI":"10.1002\/cpt.3226","article-title":"Algorithmic identification of treatment-emergent adverse events from clinical notes using large language models: a pilot study in inflammatory bowel disease","volume":"115","author":"AL Silverman","year":"2024","journal-title":"Clin Pharmacol Ther"},{"issue":"4","key":"pdig.0001426.ref036","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1007\/s40264-024-01505-6","article-title":"Leveraging Natural Language Processing and Machine Learning Methods for Adverse Drug Event Detection in Electronic Health\/Medical Records: A Scoping Review","volume":"48","author":"S Golder","year":"2025","journal-title":"Drug Saf"},{"key":"pdig.0001426.ref037","doi-asserted-by":"crossref","DOI":"10.2196\/71949","article-title":"Identifying Adverse Drug Events in Clinical Text Using Fine-Tuned Clinical Language Models: Machine Learning Study","volume":"9","author":"E Kopacheva","year":"2025","journal-title":"JMIR Form Res"},{"issue":"1","key":"pdig.0001426.ref038","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1038\/s43856-025-01116-x","article-title":"Introducing mCODEGPT as a zero-shot information extraction from clinical free text data tool for cancer research","volume":"5","author":"K Zhang","year":"2025","journal-title":"Commun Med (Lond)"},{"key":"pdig.0001426.ref039","article-title":"irAE-GPT: Leveraging large language models to identify immune-related adverse events in electronic health records and clinical trial datasets","author":"CA Bejan","year":"2025","journal-title":"medRxiv"},{"issue":"15","key":"pdig.0001426.ref040","doi-asserted-by":"crossref","first-page":"5490","DOI":"10.3390\/jcm14155490","article-title":"Large Language Models for Adverse Drug Events: A Clinical Perspective","volume":"14","author":"MM Zitu","year":"2025","journal-title":"J Clin Med"},{"key":"pdig.0001426.ref041","article-title":"A Large Language Model for Extracting Post-marketing Adverse Drug Events from Clinical Notes in the Electronic Health Record","author":"D Ludwig","year":"2026","journal-title":"Drug Saf"},{"issue":"10254","key":"pdig.0001426.ref042","doi-asserted-by":"crossref","first-page":"839","DOI":"10.1016\/S0140-6736(20)31366-0","article-title":"Lisocabtagene maraleucel for patients with relapsed or refractory large B-cell lymphomas (TRANSCEND NHL 001): a multicentre seamless design study","volume":"396","author":"JS Abramson","year":"2020","journal-title":"Lancet"},{"key":"pdig.0001426.ref043","doi-asserted-by":"crossref","first-page":"1392831","DOI":"10.3389\/fneur.2024.1392831","article-title":"Incidence of immune effector cell-associated neurotoxicity among patients treated with CAR T-cell therapy for hematologic malignancies: systematic review and meta-analysis","volume":"15","author":"MW Han","year":"2024","journal-title":"Front Neurol"},{"issue":"7","key":"pdig.0001426.ref044","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1016\/j.jtct.2025.03.011","article-title":"Cytokine Release Syndrome and Neurotoxicity Following CD19 CAR-T in B-Cell Lymphoma","volume":"31","author":"R Shouval","year":"2025","journal-title":"Transplant Cell Ther"},{"issue":"9","key":"pdig.0001426.ref045","article-title":"Real-World Evidence of Axicabtagene Ciloleucel for the Treatment of Large B Cell Lymphoma in the United States","volume":"28","author":"CA Jacobson","year":"2022","journal-title":"Transplant Cell Ther"},{"issue":"20","key":"pdig.0001426.ref046","doi-asserted-by":"crossref","first-page":"5346","DOI":"10.1182\/bloodadvances.2023012549","article-title":"Optimizing the post-CAR T monitoring period in recipients of axicabtagene ciloleucel, tisagenlecleucel, and lisocabtagene maraleucel","volume":"8","author":"N Ahmed","year":"2024","journal-title":"Blood Adv"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001426","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T17:33:42Z","timestamp":1787852022000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001426"}},"subtitle":[],"editor":[{"given":"Wisit","family":"Cheungpasitporn","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,8,27]]},"references-count":46,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,8,27]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001426","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,27]]}}}