{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T19:43:13Z","timestamp":1765827793357,"version":"3.37.3"},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2019,6,24]],"date-time":"2019-06-24T00:00:00Z","timestamp":1561334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Our objective is to develop algorithms for encoding clinical text into representations that can be used for a variety of phenotyping tasks.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Obtaining large datasets to take advantage of highly expressive deep learning methods is difficult in clinical natural language processing (NLP). We address this difficulty by pretraining a clinical text encoder on billing code data, which is typically available in abundance. We explore several neural encoder architectures and deploy the text representations obtained from these encoders in the context of clinical text classification tasks. While our ultimate goal is learning a universal clinical text encoder, we also experiment with training a phenotype-specific encoder. A universal encoder would be more practical, but a phenotype-specific encoder could perform better for a specific task.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We successfully train several clinical text encoders, establish a new state-of-the-art on comorbidity data, and observe good performance gains on substance misuse data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>We find that pretraining using billing codes is a promising research direction. The representations generated by this type of pretraining have universal properties, as they are highly beneficial for many phenotyping tasks. Phenotype-specific pretraining is a viable route for trading the generality of the pretrained encoder for better performance on a specific phenotyping task.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>We successfully applied our approach to many phenotyping tasks. We conclude by discussing potential limitations of our approach.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocz072","type":"journal-article","created":{"date-parts":[[2019,4,29]],"date-time":"2019-04-29T03:08:10Z","timestamp":1556507290000},"page":"1272-1278","source":"Crossref","is-referenced-by-count":14,"title":["Toward a clinical text encoder: pretraining for clinical natural language processing with applications to substance misuse"],"prefix":"10.1093","volume":"26","author":[{"given":"Dmitriy","family":"Dligach","sequence":"first","affiliation":[{"name":"Department of Computer Science, Loyola University Chicago, Chicago, Illinois, USA"},{"name":"Department of Public Health Sciences, Stritch School of Medicine, Loyola University, Maywood, Illinois, USA"},{"name":"Center for Health Outcomes and Informatics Research, Loyola University, Maywood, Illinois, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6368-4652","authenticated-orcid":false,"given":"Majid","family":"Afshar","sequence":"additional","affiliation":[{"name":"Department of Public Health Sciences, Stritch School of Medicine, Loyola University, Maywood, Illinois, USA"},{"name":"Center for Health Outcomes and Informatics Research, Loyola University, Maywood, Illinois, USA"}]},{"given":"Timothy","family":"Miller","sequence":"additional","affiliation":[{"name":"Computational Health Informatics Program (CHIP), Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA"}]}],"member":"286","published-online":{"date-parts":[[2019,6,24]]},"reference":[{"author":"Rajpurkar","key":"2020110613110073100_ocz072-B1"},{"year":"2018","author":"Hassan","key":"2020110613110073100_ocz072-B2"},{"key":"2020110613110073100_ocz072-B3","first-page":"97","article-title":"Learning transferable features with deep adaptation networks","author":"Long","year":"2015"},{"year":"2014","author":"Razavian","key":"2020110613110073100_ocz072-B4"},{"year":"2013","author":"Mikolov","key":"2020110613110073100_ocz072-B5"},{"first-page":"1532","year":"2014","author":"Pennington","key":"2020110613110073100_ocz072-B6"},{"first-page":"1188","year":"2014","author":"Le","key":"2020110613110073100_ocz072-B7"},{"author":"Peters","key":"2020110613110073100_ocz072-B8"},{"author":"Howard","key":"2020110613110073100_ocz072-B9"},{"year":"2018","author":"Devlin","key":"2020110613110073100_ocz072-B10"},{"year":"2018","author":"Akbik","key":"2020110613110073100_ocz072-B11"},{"first-page":"119","year":"2018","author":"Dligach","key":"2020110613110073100_ocz072-B12"},{"year":"2018","author":"Radford","key":"2020110613110073100_ocz072-B13"},{"first-page":"787","year":"2017","author":"Choi","key":"2020110613110073100_ocz072-B14"},{"first-page":"301","year":"2016","author":"Choi","key":"2020110613110073100_ocz072-B15"},{"year":"2017","author":"Lipton","key":"2020110613110073100_ocz072-B16"},{"key":"2020110613110073100_ocz072-B17","doi-asserted-by":"crossref","first-page":"26094.","DOI":"10.1038\/srep26094","article-title":"Deep patient: an unsupervised representation to predict the future of patients from the electronic health records","volume":"6","author":"Miotto","year":"2016","journal-title":"Sci Rep"},{"first-page":"22","year":"2017","author":"Nguyen","key":"2020110613110073100_ocz072-B18"},{"first-page":"30","year":"2016","author":"Pham","key":"2020110613110073100_ocz072-B19"},{"year":"2018","author":"Sushil","key":"2020110613110073100_ocz072-B20"},{"issue":"10","key":"2020110613110073100_ocz072-B21","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"2020110613110073100_ocz072-B22","doi-asserted-by":"crossref","first-page":"160035.","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"issue":"12","key":"2020110613110073100_ocz072-B23","doi-asserted-by":"crossref","DOI":"10.1097\/MLR.0b013e31825f64d0","article-title":"Systematic review of comorbidity indices for administrative data","volume":"50","author":"Sharabiani","year":"2012","journal-title":"Med Care"},{"year":"2017","author":"Weiss","key":"2020110613110073100_ocz072-B24"},{"issue":"5","key":"2020110613110073100_ocz072-B25","doi-asserted-by":"crossref","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"year":"2016","author":"Substance Use and Mental Health Services Administration","key":"2020110613110073100_ocz072-B26"},{"first-page":"791","year":"1993","author":"Saunders","key":"2020110613110073100_ocz072-B27"},{"issue":"3","key":"2020110613110073100_ocz072-B28","doi-asserted-by":"crossref","DOI":"10.1093\/jamia\/ocy166","article-title":"Natural language processing and machine learning to identify alcohol misuse from the electronic health record in trauma patients: development and internal validation","volume":"26","author":"Afshar","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613110073100_ocz072-B29","first-page":"281","article-title":"Random search for hyper-parameter optimization","volume":"13","author":"Bergstra","year":"2012","journal-title":"J Mach Learn Res"},{"year":"2018","author":"Yao","key":"2020110613110073100_ocz072-B30"},{"issue":"4","key":"2020110613110073100_ocz072-B31","doi-asserted-by":"crossref","DOI":"10.1016\/S1364-6613(99)01294-2","article-title":"Catastrophic forgetting in connectionist networks","volume":"3","author":"French","year":"1999","journal-title":"Trends Cogn Sci"},{"year":"2019","author":"Lee","key":"2020110613110073100_ocz072-B32"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/11\/1272\/34151828\/ocz072.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/11\/1272\/34151828\/ocz072.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T19:36:04Z","timestamp":1604691364000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/11\/1272\/5522436"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,24]]},"references-count":32,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2019,6,24]]},"published-print":{"date-parts":[[2019,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz072","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"type":"print","value":"1067-5027"},{"type":"electronic","value":"1527-974X"}],"subject":[],"published-other":{"date-parts":[[2019,11]]},"published":{"date-parts":[[2019,6,24]]}}}