{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,9]],"date-time":"2026-08-09T11:15:12Z","timestamp":1786274112781,"version":"3.56.0"},"reference-count":46,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T00:00:00Z","timestamp":1669766400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR003528"],"award-info":[{"award-number":["U01TR003528"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM013337"],"award-info":[{"award-number":["R01LM013337"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,1,18]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Clinical knowledge-enriched transformer models (eg, ClinicalBERT) have state-of-the-art results on clinical natural language processing (NLP) tasks. One of the core limitations of these transformer models is the substantial memory consumption due to their full self-attention mechanism, which leads to the performance degradation in long clinical texts. To overcome this, we propose to leverage long-sequence transformer models (eg, Longformer and BigBird), which extend the maximum input sequence length from 512 to 4096, to enhance the ability to model long-term dependencies in long clinical texts.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and methods<\/jats:title><jats:p>Inspired by the success of long-sequence transformer models and the fact that clinical notes are mostly long, we introduce 2 domain-enriched language models, Clinical-Longformer and Clinical-BigBird, which are pretrained on a large-scale clinical corpus. We evaluate both language models using 10 baseline tasks including named entity recognition, question answering, natural language inference, and document classification tasks.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT and other short-sequence transformers in all 10 downstream tasks and achieve new state-of-the-art results.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>Our pretrained language models provide the bedrock for clinical NLP using long texts. We have made our source code available at https:\/\/github.com\/luoyuanlab\/Clinical-Longformer, and the pretrained models available for public download at: https:\/\/huggingface.co\/yikuan8\/Clinical-Longformer.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>This study demonstrates that clinical knowledge-enriched long-sequence transformers are able to learn long-term dependencies in long clinical text. Our methods can also inspire the development of other domain-enriched long-sequence transformers.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocac225","type":"journal-article","created":{"date-parts":[[2022,12,1]],"date-time":"2022-12-01T04:43:18Z","timestamp":1669869798000},"page":"340-347","source":"Crossref","is-referenced-by-count":128,"title":["A comparative study of pretrained language models for long clinical text"],"prefix":"10.1093","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7546-9979","authenticated-orcid":false,"given":"Yikuan","family":"Li","sequence":"first","affiliation":[{"name":"Division of Health and Biomedical Informatics, Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University , Chicago, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ramsey M","family":"Wehbe","sequence":"additional","affiliation":[{"name":"Division of Cardiology, Department of Medicine, Feinberg School of Medicine, Northwestern University , Chicago, Illinois, USA"},{"name":"Bluhm Cardiovascular Institute Center for Artificial Intelligence, Northwestern Medicine , Chicago, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2613-2541","authenticated-orcid":false,"given":"Faraz S","family":"Ahmad","sequence":"additional","affiliation":[{"name":"Division of Health and Biomedical Informatics, Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University , Chicago, Illinois, USA"},{"name":"Division of Cardiology, Department of Medicine, Feinberg School of Medicine, Northwestern University , Chicago, Illinois, USA"},{"name":"Bluhm Cardiovascular Institute Center for Artificial Intelligence, Northwestern Medicine , Chicago, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9884-9683","authenticated-orcid":false,"given":"Hanyin","family":"Wang","sequence":"additional","affiliation":[{"name":"Division of Health and Biomedical Informatics, Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University , Chicago, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuan","family":"Luo","sequence":"additional","affiliation":[{"name":"Division of Health and Biomedical Informatics, Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University , Chicago, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,11,30]]},"reference":[{"key":"2023011811021176900_ocac225-B1","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inform Process Syst"},{"key":"2023011811021176900_ocac225-B2","first-page":"4171","author":"Devlin"},{"key":"2023011811021176900_ocac225-B3","author":"Liu","year":"2019"},{"issue":"12","key":"2023011811021176900_ocac225-B4","doi-asserted-by":"crossref","first-page":"1632","DOI":"10.1093\/jamia\/ocz164","article-title":"Traditional Chinese medicine clinical records classification with BERT and domain specific corpora","volume":"26","author":"Yao","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2023011811021176900_ocac225-B5","author":"Zhang"},{"key":"2023011811021176900_ocac225-B6"},{"issue":"1","key":"2023011811021176900_ocac225-B7","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1093\/jamiaopen\/ooz072","article-title":"Adapting and evaluating a deep learning language model for clinical why-question answering","volume":"3","author":"Wen","year":"2020","journal-title":"JAMIA Open"},{"key":"2023011811021176900_ocac225-B8","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inform Process Syst"},{"issue":"9","key":"2023011811021176900_ocac225-B9","doi-asserted-by":"crossref","first-page":"3596","DOI":"10.1109\/JBHI.2021.3062322","article-title":"Limitations of transformers on clinical text classification","volume":"25","author":"Gao","year":"2021","journal-title":"IEEE J Biomed Health Inform"},{"key":"2023011811021176900_ocac225-B10","first-page":"94","author":"Huang"},{"issue":"1","key":"2023011811021176900_ocac225-B11","doi-asserted-by":"crossref","first-page":"e0262182","DOI":"10.1371\/journal.pone.0262182","article-title":"Unstructured clinical notes within the 24 hours since admission predict short, mid & long-term mortality in adult ICU patients","volume":"17","author":"Mahbub","year":"2022","journal-title":"PLoS One"},{"key":"2023011811021176900_ocac225-B12","author":"Ainslie"},{"key":"2023011811021176900_ocac225-B13","author":"Beltagy","year":"2020"},{"key":"2023011811021176900_ocac225-B14","first-page":"17283","article-title":"Big bird: transformers for longer sequences","volume":"33","author":"Zaheer","year":"2020","journal-title":"Adv Neural Inform Process Syst"},{"issue":"4","key":"2023011811021176900_ocac225-B15","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2023011811021176900_ocac225-B16","first-page":"72","author":"Alsentzer"},{"key":"2023011811021176900_ocac225-B17","first-page":"1500","author":"Smit"},{"key":"2023011811021176900_ocac225-B18","author":"He"},{"key":"2023011811021176900_ocac225-B19","author":"Michalopoulos","year":"2021"},{"key":"2023011811021176900_ocac225-B20","first-page":"1208","author":"Zhou"},{"key":"2023011811021176900_ocac225-B21","first-page":"2330","author":"Agrawal"},{"key":"2023011811021176900_ocac225-B22","first-page":"2978","author":"Dai"},{"key":"2023011811021176900_ocac225-B23","author":"Kitaev"},{"issue":"1","key":"2023011811021176900_ocac225-B24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"key":"2023011811021176900_ocac225-B25","author":"Wang","year":"2020"},{"issue":"2","key":"2023011811021176900_ocac225-B26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/978-3-031-02154-1","article-title":"Ontology-based Interpretation of Natural Language","volume":"7","author":"Cimiano","year":"2014","journal-title":"Synthesis Lectures on Human Language Technologies"},{"key":"2023011811021176900_ocac225-B27","first-page":"2357","author":"Pampari"},{"key":"2023011811021176900_ocac225-B28","author":"Yue"},{"key":"2023011811021176900_ocac225-B29","first-page":"6102","author":"Kang"},{"key":"2023011811021176900_ocac225-B30","author":"Soni","year":"2020"},{"issue":"10","key":"2023011811021176900_ocac225-B31","doi-asserted-by":"crossref","first-page":"1745","DOI":"10.1093\/bioinformatics\/bty869","article-title":"Cross-type biomedical named entity recognition with deep multi-task learning","volume":"35","author":"Wang","year":"2019","journal-title":"Bioinformatics"},{"issue":"S10","key":"2023011811021176900_ocac225-B32","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1186\/s12859-019-2813-6","article-title":"Collabonet: collaboration of deep neural networks for biomedical named entity recognition","volume":"20","author":"Yoon","year":"2019","journal-title":"BMC Bioinformatics"},{"issue":"5","key":"2023011811021176900_ocac225-B33","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1197\/jamia.M2444","article-title":"Evaluating the state-of-the-art in automatic de-identification","volume":"14","author":"Uzuner","year":"2007","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2023011811021176900_ocac225-B34","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2023011811021176900_ocac225-B35","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 challenge","volume":"20","author":"Sun","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2023011811021176900_ocac225-B36","doi-asserted-by":"crossref","first-page":"S20","DOI":"10.1016\/j.jbi.2015.07.020","article-title":"Annotating longitudinal clinical narratives for de-identification: the 2014 i2b2\/UTHealth corpus","volume":"58","author":"Stubbs","year":"2015","journal-title":"J Biomed Informatics"},{"key":"2023011811021176900_ocac225-B37","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1007\/978-94-017-2390-9_10","volume-title":"Natural language processing using very large corpora","author":"Ramshaw","year":"1999"},{"key":"2023011811021176900_ocac225-B38","author":"Li","year":"2018"},{"key":"2023011811021176900_ocac225-B39","first-page":"368","article-title":"Early prediction of acute kidney injury in critical care setting using clinical notes and structured multivariate physiological measurements","volume":"264","author":"Sun","year":"2019","journal-title":"MedInfo"},{"issue":"2","key":"2023011811021176900_ocac225-B40","doi-asserted-by":"crossref","first-page":"168","DOI":"10.5626\/JCSE.2012.6.2.168","article-title":"Design and development of a multimodal biomedical information retrieval system","volume":"6","author":"Demner-Fushman","year":"2012","journal-title":"J Comput Sci Eng"},{"issue":"1","key":"2023011811021176900_ocac225-B41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41597-019-0322-0","article-title":"MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports","volume":"6","author":"Johnson","year":"2019","journal-title":"Sci Data"},{"key":"2023011811021176900_ocac225-B42","author":"Li","year":"2020"},{"key":"2023011811021176900_ocac225-B43","author":"Wang","year":"2018"},{"key":"2023011811021176900_ocac225-B44","first-page":"1586","author":"Romanov"},{"key":"2023011811021176900_ocac225-B45","author":"Pappagari","year":"2019"},{"key":"2023011811021176900_ocac225-B46","first-page":"4163","author":"Jiao"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/2\/340\/48754733\/ocac225.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/2\/340\/48754733\/ocac225.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,12]],"date-time":"2023-03-12T03:03:23Z","timestamp":1678590203000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/2\/340\/6855145"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,30]]},"references-count":46,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,11,30]]},"published-print":{"date-parts":[[2023,1,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac225","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,2,1]]},"published":{"date-parts":[[2022,11,30]]}}}