{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T22:37:13Z","timestamp":1793140633663,"version":"4.5.0"},"reference-count":20,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,1,8]],"date-time":"2020-01-08T00:00:00Z","timestamp":1578441600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2020,1,8]],"date-time":"2020-01-08T00:00:00Z","timestamp":1578441600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100017183","name":"Janssen Scientific Affairs, LLC","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100017183","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Stroke severity is an important predictor of patient outcomes and is commonly measured with the National Institutes of Health Stroke Scale (NIHSS) scores. Because these scores are often recorded as free text in physician reports, structured real-world evidence databases seldom include the severity. The aim of this study was to use machine learning models to impute NIHSS scores for all patients with newly diagnosed stroke from multi-institution electronic health record (EHR) data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      NIHSS scores available in the Optum\u00a9 de-identified Integrated Claims-Clinical dataset were extracted from physician notes by applying natural language processing (NLP) methods. The cohort analyzed in the study consists of the 7149 patients with an inpatient or emergency room diagnosis of ischemic stroke, hemorrhagic stroke, or transient ischemic attack and a corresponding NLP-extracted NIHSS score. A subset of these patients (\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u20091033, 14%) were held out for independent validation of model performance and the remaining patients (\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u20096116, 86%) were used for training the model. Several machine learning models were evaluated, and parameters optimized using cross-validation on the training set. The model with optimal performance, a random forest model, was ultimately evaluated on the holdout set.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Leveraging machine learning we identified the main factors in electronic health record data for assessing stroke severity, including death within the same month as stroke occurrence, length of hospital stay following stroke occurrence, aphagia\/dysphagia diagnosis, hemiplegia diagnosis, and whether a patient was discharged to home or self-care. Comparing the imputed NIHSS scores to the NLP-extracted NIHSS scores on the holdout data set yielded an R\n                      <jats:sup>2<\/jats:sup>\n                      (coefficient of determination) of 0.57, an R (Pearson correlation coefficient) of 0.76, and a root-mean-squared error of 4.5.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Machine learning models built on EHR data can be used to determine proxies for stroke severity. This enables severity to be incorporated in studies of stroke patient outcomes using administrative and EHR databases.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12911-019-1010-x","type":"journal-article","created":{"date-parts":[[2020,1,8]],"date-time":"2020-01-08T12:02:33Z","timestamp":1578484953000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":113,"title":["Assessing stroke severity using electronic health record data: a machine learning approach"],"prefix":"10.1186","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1364-6743","authenticated-orcid":false,"given":"Emily","family":"Kogan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathryn","family":"Twyman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jesse","family":"Heap","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dejan","family":"Milentijevic","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer H.","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark","family":"Alberts","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,1,8]]},"reference":[{"issue":"3","key":"1010_CR1","doi-asserted-by":"publisher","first-page":"918","DOI":"10.1161\/01.str.0000441948.35804.77","volume":"45","author":"IL Katzan","year":"2014","unstructured":"Katzan IL, Spertus J, Bettger JP, Bravata DM, Reeves MJ, Smith EE, et al. Risk adjustment of ischemic stroke outcomes for comparing hospital performance: a statement for healthcare professionals from the American Heart Association\/American Stroke Association. Stroke. 2014;45(3):918\u201344.","journal-title":"Stroke"},{"key":"1010_CR2","unstructured":"National Center for Health Statistics. Health, United States, 2016: with Chartbook on long-term trends in health. Hyattsville: National Center for Health Statistics; 2017."},{"key":"1010_CR3","doi-asserted-by":"publisher","unstructured":"Rost NS, Bottle A, Lee JM, Randall M, Middleton S, Shaw L, et al. Stroke severity is a crucial predictor of outcome: an international prospective validation study. J Am Heart Assoc. 2016;5(1). https:\/\/doi.org\/10.1161\/JAHA.115.002433.","DOI":"10.1161\/JAHA.115.002433"},{"issue":"1","key":"1010_CR4","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1161\/xJAHA.111.000034","volume":"1","author":"GC Fonarow","year":"2012","unstructured":"Fonarow GC, Saver JL, Smith EE, Broderick JP, Kleindorfer DO, Sacco RL, et al. Relationship of national institutes of health stroke scale to 30-day mortality in medicare beneficiaries with acute ischemic stroke. J Am Heart Assoc. 2012;1(1):42\u201350.","journal-title":"J Am Heart Assoc"},{"key":"1010_CR5","doi-asserted-by":"publisher","first-page":"192","DOI":"10.3389\/fneur.2017.00192","volume":"8","author":"TG Phan","year":"2017","unstructured":"Phan TG, Clissold BB, Ma H, Ly JV, Srikanth V. Predicting disability after ischemic stroke based on comorbidity index and stroke severity-from the virtual international stroke trials archive-acute collaboration. Front Neurol. 2017;8:192.","journal-title":"Front Neurol"},{"issue":"15","key":"1010_CR6","first-page":"44","volume":"24","author":"AV Hage","year":"2011","unstructured":"Hage AV. The NIH stroke scale: a window into neurological status. Nurs Spectr. 2011;24(15):44\u20139.","journal-title":"Nurs Spectr"},{"key":"1010_CR7","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-58280-1_16","volume-title":"Handbook of large-scale distributed computing in smart healthcare","author":"OW Samuel","year":"2017","unstructured":"Samuel OW, Fang P, Chen S, Geng Y, Li G. Activity recognition based on pattern recognition of myoelectric signals for rehabilitation. In: Khan SU, Zomaya AY, Abbas A, editors. Handbook of large-scale distributed computing in smart healthcare. Basel: Springer International Publishing AG; 2017. https:\/\/doi.org\/10.1007\/978-3-319-58280-1_16."},{"issue":"5","key":"1010_CR8","doi-asserted-by":"publisher","first-page":"1589","DOI":"10.1161\/STR.0000000000000014","volume":"45","author":"GC Fonarow","year":"2014","unstructured":"Fonarow GC, Alberts MJ, Broderick JP, Jauch EC, Kleindorfer DO, Saver JL, et al. Stroke outcomes measures must be appropriately risk adjusted to ensure quality care of patients: a presidential advisory from the American Heart Association\/American Stroke Association. Stroke. 2014;45(5):1589\u2013601.","journal-title":"Stroke"},{"issue":"11","key":"1010_CR9","doi-asserted-by":"publisher","first-page":"1292","DOI":"10.1016\/j.jclinepi.2015.01.009","volume":"68","author":"SF Sung","year":"2015","unstructured":"Sung SF, Hsieh CY, Kao Yang YH, Lin HJ, Chen CH, Chen YW, et al. Developing a stroke severity index based on administrative data was feasible using data mining techniques. J Clin Epidemiol. 2015;68(11):1292\u2013300.","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"1010_CR10","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1186\/s12913-016-1769-8","volume":"16","author":"SF Sung","year":"2016","unstructured":"Sung SF, Hsieh CY, Lin HJ, Chen YW, Chen CH, Kao Yang YH, et al. Validity of a stroke severity index for administrative claims data research: a retrospective cohort study. BMC Health Serv Res. 2016;16(1):509.","journal-title":"BMC Health Serv Res"},{"issue":"4","key":"1010_CR11","doi-asserted-by":"publisher","first-page":"438","DOI":"10.1002\/pds.3944","volume":"25","author":"SF Sung","year":"2016","unstructured":"Sung SF, Chen SC, Hsieh CY, Li CY, Lai EC, Hu YH. A comparison of stroke severity proxy measures for claims data research: a population-based cohort study. Pharmacoepidemiol Drug Saf. 2016;25(4):438\u201343.","journal-title":"Pharmacoepidemiol Drug Saf"},{"key":"1010_CR12","unstructured":"45 CFR 164.514(b)(1) Implementation specifications: Requirements for de-identification of protected health information. Federal Government of the United States. https:\/\/www.govinfo.gov\/content\/pkg\/CFR-2017-title45-vol1\/pdf\/CFR-2017-title45-vol1-sec164-514.pdf. Accessed 15 Feb 2019."},{"key":"1010_CR13","unstructured":"Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Information Insurance Portability and Accountability Act (HIPAA) Privacy Rule (Dated as September 4, 2012, as first released on November 26, 2012)."},{"key":"1010_CR14","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1016\/j.diabres.2016.09.012","volume":"121","author":"AP Nunes","year":"2016","unstructured":"Nunes AP, Yang J, Radican L, Engel SS, Kurtyka K, Tunceli K, et al. Assessing occurrence of hypoglycemia and its severity from electronic health records of patients with type 2 diabetes mellitus. Diabetes Res Clin Pract. 2016;121:192\u2013203.","journal-title":"Diabetes Res Clin Pract"},{"issue":"11","key":"1010_CR15","doi-asserted-by":"publisher","first-page":"1130","DOI":"10.1097\/01.mlr.0000182534.19832.83","volume":"43","author":"H Quan","year":"2005","unstructured":"Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43(11):1130\u20139.","journal-title":"Med Care"},{"key":"1010_CR16","unstructured":"CMS\u2019 ICD-9-CM to and from ICD-10-CM and ICD-10-PCS Crosswalk or General Equivalence Mappings. http:\/\/www.nber.org\/data\/icd9-icd-10-cm-and-pcs-crosswalk-general-equivalence-mapping.html. Accessed 12 Apr 2018."},{"issue":"11","key":"1010_CR17","doi-asserted-by":"publisher","first-page":"3211","DOI":"10.1161\/STROKEAHA.113.002881","volume":"44","author":"M Reeves","year":"2013","unstructured":"Reeves M, Khoury J, Alwell K, Moomaw C, Flaherty M, Woo D, et al. Distribution of National Institutes of Health stroke scale in the cincinnati\/northern Kentucky stroke study. Stroke. 2013;44(11):3211\u20133.","journal-title":"Stroke"},{"issue":"11","key":"1010_CR18","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.1016\/S1474-4422(13)70223-0","volume":"12","author":"MD Ginsberg","year":"2013","unstructured":"Ginsberg MD, Palesch YY, Hill MD, Martin RH, Moy CS, Barsan WG, et al. High-dose albumin treatment for acute ischaemic stroke (ALIAS) part 2: a randomised, double-blind, phase 3, placebo-controlled trial. Lancet Neurol. 2013;12(11):1049\u201358.","journal-title":"Lancet Neurol"},{"issue":"6","key":"1010_CR19","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1016\/S1474-4422(15)00047-2","volume":"14","author":"GW Albers","year":"2015","unstructured":"Albers GW, von Kummer R, Truelsen T, Jensen J-KS, Ravn GM, Gr\u00f8nning BA, et al. Safety and efficacy of desmoteplase given 3\u20139 h after ischaemic stroke in patients with occlusion or high-grade stenosis in major cerebral arteries (DIAS-3): a double-blind, randomised, placebo-controlled phase 3 trial. Lancet Neurol. 2015;14(6):575\u201384.","journal-title":"Lancet Neurol"},{"issue":"6 Suppl 3","key":"1010_CR20","doi-asserted-by":"publisher","first-page":"S90","DOI":"10.1161\/CIRCOUTCOMES.115.001775","volume":"8","author":"MJ Reeves","year":"2015","unstructured":"Reeves MJ, Smith EE, Fonarow GC, Zhao X, Thompson M, Peterson ED, et al. Variation and trends in the documentation of National Institutes of Health stroke scale in GWTG-stroke hospitals. Circ Cardiovasc Qual Outcomes. 2015;8(6 Suppl 3):S90\u20138.","journal-title":"Circ Cardiovasc Qual Outcomes"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-019-1010-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12911-019-1010-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-019-1010-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,6]],"date-time":"2021-01-06T19:35:05Z","timestamp":1609961705000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-019-1010-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,8]]},"references-count":20,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["1010"],"URL":"https:\/\/doi.org\/10.1186\/s12911-019-1010-x","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,8]]},"assertion":[{"value":"2 August 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 December 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 January 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"No patient\u2019s personal identifying information is included in this manuscript. All study data were accessed with protocols compliant with US patient confidentiality requirements, including the Health Insurance Portability and Accountability Act of 1996 regulations. Because this study was non-interventional and used only statistically deidentified patient records, it was exempt from institutional review board review.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"EK, JH and DM are employees of Janssen; KT and JHL were employees of Janssen during the study and manuscript development; and MA is an employee of Hartford HealthCare.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"8"}}