{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T15:35:42Z","timestamp":1773416142363,"version":"3.50.1"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T00:00:00Z","timestamp":1573171200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T00:00:00Z","timestamp":1573171200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Sidra Medicine","award":["SDR200043"],"award-info":[{"award-number":["SDR200043"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n              <jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Predictive modeling with longitudinal electronic health record (EHR) data offers great promise for accelerating personalized medicine and better informs clinical decision-making. Recently, deep learning models have achieved state-of-the-art performance for many healthcare prediction tasks. However, deep models lack interpretability, which is integral to successful decision-making and can lead to better patient care. In this paper, we build upon the contextual decomposition (CD) method, an algorithm for producing importance scores from long short-term memory networks (LSTMs). We extend the method to bidirectional LSTMs (BiLSTMs) and use it in the context of predicting future clinical outcomes using patients\u2019 EHR historical visits.<\/jats:p>\n              <\/jats:sec>\n              <jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We use a real EHR dataset comprising 11071 patients, to evaluate and compare CD interpretations from LSTM and BiLSTM models. First, we train LSTM and BiLSTM models for the task of predicting which pre-school children with respiratory system-related complications will have asthma at school-age. After that, we conduct quantitative and qualitative analysis to evaluate the CD interpretations produced by the contextual decomposition of the trained models. In addition, we develop an interactive visualization to demonstrate the utility of CD scores in explaining predicted outcomes.<\/jats:p>\n              <\/jats:sec>\n              <jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Our experimental evaluation demonstrate that whenever a clear visit-level pattern exists, the models learn that pattern and the contextual decomposition can appropriately attribute the prediction to the correct pattern. In addition, the results confirm that the CD scores agree to a large extent with the importance scores generated using logistic regression coefficients. Our main insight was that rather than interpreting the attribution of individual visits to the predicted outcome, we could instead attribute a model\u2019s prediction to a group of visits.<\/jats:p>\n              <\/jats:sec>\n              <jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>We presented a quantitative and qualitative evidence that CD interpretations can explain patient-specific predictions using CD attributions of individual visits or a group of visits.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-019-0951-4","type":"journal-article","created":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T15:27:31Z","timestamp":1573226851000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Interpreting patient-Specific risk prediction using contextual decomposition of BiLSTMs: application to children with asthma"],"prefix":"10.1186","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3235-0860","authenticated-orcid":false,"given":"Rawan","family":"AlSaad","sequence":"first","affiliation":[]},{"given":"Qutaibah","family":"Malluhi","sequence":"additional","affiliation":[]},{"given":"Ibrahim","family":"Janahi","sequence":"additional","affiliation":[]},{"given":"Sabri","family":"Boughorbel","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,11,8]]},"reference":[{"issue":"10","key":"951_CR1","doi-asserted-by":"publisher","first-page":"1419","DOI":"10.1093\/jamia\/ocy068","volume":"25","author":"C Xiao","year":"2018","unstructured":"Xiao C, Choi E, Sun J. Opportunities and challenges in developing deep learning models using electronic health records data: A systematic review. J Am Med Assoc. 2018; 25(10):1419\u201328. \nhttp:\/\/doi.org\/10.1093\/jamia\/ocy068\n\n.","journal-title":"J Am Med Assoc"},{"issue":"44","key":"951_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12911-018-0620-z","volume":"18","author":"SB Golas","year":"2018","unstructured":"Golas SB, Shibahara T, Agboola S, et al.A machine learning model to predict the risk of 30-day readmissions in patients with heart failure: A retrospective analysis of electronic medical records data. BMC Med Inform Decis Making. 2018; 18(44):1\u201317. \nhttps:\/\/doi.org\/10.1186\/s12911-018-0620-z\n\n.","journal-title":"BMC Med Inform Decis Making"},{"key":"951_CR3","doi-asserted-by":"publisher","unstructured":"Cheng Y, Wang F, Zhang P, Hu J. Risk Prediction with Electronic Health Records: A Deep Learning Approach. Society for Industrial and Applied Mathematics; 2016. \nhttps:\/\/doi.org\/10.1137\/1.9781611974348.49\n\n.","DOI":"10.1137\/1.9781611974348.49"},{"issue":"5","key":"951_CR4","doi-asserted-by":"publisher","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","volume":"22","author":"Benjamin Shickel","year":"2018","unstructured":"Shickel B, Tighe PJ, Bihorac A, Rashidi P, J Biomed Health Inform IEEE. Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis. 2017:1\u201314. \nhttps:\/\/doi.org\/10.1109\/JBHI.2017.2767063\n\n.","journal-title":"IEEE Journal of Biomedical and Health Informatics"},{"issue":"2","key":"951_CR5","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1176\/appi.ajp.2016.16101169","volume":"174","author":"DE Adkins","year":"2017","unstructured":"Adkins DE. Machine Learning and Electronic Health Records: A Paradigm Shift. Am J Psychiatr. 2017; 174(2):93\u20134. \nhttps:\/\/doi.org\/10.1176\/appi.ajp.2016.16101169\n\n.","journal-title":"Am J Psychiatr"},{"key":"951_CR6","doi-asserted-by":"publisher","unstructured":"Lipton ZC, Kale DC, Elkan C, Wetzel R. Learning to Diagnose with LSTM Recurrent Neural Networks. 2015:1\u201318. \nhttps:\/\/doi.org\/10.14722\/ndss.2015.23268\n\n.","DOI":"10.14722\/ndss.2015.23268"},{"key":"951_CR7","doi-asserted-by":"publisher","unstructured":"Esteban C, Staeck O, Yang Y, Tresp V. Predicting Clinical Events by Combining Static and Dynamic Information Using Recurrent Neural Networks. In: 2016 IEEE International Conference on Healthcare Informatics (ICHI): 2016. \nhttps:\/\/doi.org\/10.1109\/ichi.2016.16\n\n.","DOI":"10.1109\/ichi.2016.16"},{"key":"951_CR8","unstructured":"Pham T, Tran T, Phung D, Venkatesh S. DeepCare : A Deep Dynamic Memory Model for Predictive Medicine. 2017; i:1\u201327. http:\/\/arxiv.org\/abs\/arXiv:1602.00357v2."},{"key":"951_CR9","doi-asserted-by":"publisher","unstructured":"Jagannatha AN, Yu H. Bidirectional RNN for Medical Event Detection in Electronic Health Records. Association for Computational Linguistics; 2016. \nhttps:\/\/doi.org\/10.18653\/v1\/n16-1056\n\n.","DOI":"10.18653\/v1\/n16-1056"},{"key":"951_CR10","unstructured":"Liu J, Zhang Z, Razavian N. Deep EHR: Chronic Disease Prediction Using Medical Notes. In: Proceedings of the 3rd Machine Learning for Healthcare Conference. PMLR 85: 2018. p. 440\u2013464. http:\/\/arxiv.org\/abs\/arXiv:1808.04928v1."},{"issue":"1","key":"951_CR11","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1007\/s40264-018-0765-9","volume":"42","author":"S Wunnava","year":"2019","unstructured":"Wunnava S, Qin X, Kakar T, Sen C, Rundensteiner EA, Kong X. Adverse Drug Event Detection from Electronic Health Records Using Hierarchical Recurrent Neural Networks with Dual-Level Embedding. Drug Safety. 2019; 42(1):113\u201322. \nhttps:\/\/doi.org\/10.1007\/s40264-018-0765-9\n\n.","journal-title":"Drug Safety"},{"key":"951_CR12","doi-asserted-by":"publisher","unstructured":"Ahmad MA, Eckert C, Teredesai A. Interpretable Machine Learning in Healthcare. In: Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics - BCB \u201918: 2018. p. 559\u2013560. \nhttps:\/\/doi.org\/10.1145\/3233547.3233667\n\n.","DOI":"10.1145\/3233547.3233667"},{"key":"951_CR13","unstructured":"Murdoch WJ, Singh C, Kumbier K, Abbasi-Asl R, Yu B. Interpretable machine learning: definitions, methods, and applications. 2019:1\u201311. http:\/\/arxiv.org\/abs\/1901.04592."},{"key":"951_CR14","unstructured":"Baumel T, Nassour-Kassis J, Cohen R, Elhadad M, Elhadad N. Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment. 2017. http:\/\/arxiv.org\/abs\/1709.09587."},{"key":"951_CR15","doi-asserted-by":"publisher","unstructured":"Ma F, Chitta R, Zhou J, You Q, Sun T, Gao J. Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks. 2017. \nhttps:\/\/doi.org\/10.1145\/3097983.3098088\n\n.","DOI":"10.1145\/3097983.3098088"},{"key":"951_CR16","doi-asserted-by":"publisher","unstructured":"Choi E, Bahadori MT, Song L, Stewart WF, Sun J. GRAM: Graph-based Attention Model for Healthcare Representation Learning. 2016:1\u201315. \nhttps:\/\/doi.org\/10.1145\/3097983.3098126\n\n.","DOI":"10.1145\/3097983.3098126"},{"key":"951_CR17","unstructured":"Choi E, Bahadori MT, Kulas JA, Schuetz A, Stewart WF, Sun J. RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism (NIPS). 2016. http:\/\/arxiv.org\/abs\/1608.05745."},{"issue":"1","key":"951_CR18","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1109\/tvcg.2018.2865027","volume":"25","author":"J Choo","year":"2018","unstructured":"Choo J, Kwon BC, Choi E, Kim YB, Kim JT, Choi M-J, Kwon S, Sun J. RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records. IEEE Trans Vis Comput Graph. 2018; 25(1):299\u2013309. \nhttps:\/\/doi.org\/10.1109\/tvcg.2018.2865027\n\n.","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"951_CR19","doi-asserted-by":"publisher","first-page":"65333","DOI":"10.1109\/ACCESS.2018.2875677","volume":"6","author":"J Zhang","year":"2018","unstructured":"Zhang J, Kowsari K, Harrison JH, Lobo JM, Barnes LE. Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record. IEEE Access. 2018; 6:65333\u201346. \nhttps:\/\/doi.org\/10.1109\/ACCESS.2018.2875677\n\n.","journal-title":"IEEE Access"},{"key":"951_CR20","doi-asserted-by":"publisher","unstructured":"Xu Y, Biswal S, Deshpande SR, Maher KO, Sun J. RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data. 2018. \nhttps:\/\/doi.org\/10.1145\/3219819.3220051\n\n.","DOI":"10.1145\/3219819.3220051"},{"key":"951_CR21","unstructured":"Che Z, Purushotham S, Khemani R, Liu Y. Distilling Knowledge from Deep Networks with Applications to Healthcare Domain. 2015. http:\/\/arxiv.org\/abs\/1512.03542."},{"key":"951_CR22","first-page":"371","volume":"2016","author":"Z Che","year":"2016","unstructured":"Che Z, Purushotham S, Khemani R, Liu Y. Interpretable Deep Models for ICU Outcome Prediction,. AMIA Annu Symp Proc AMIA Symp. 2016; 2016:371\u201380.","journal-title":"AMIA Annu Symp Proc AMIA Symp"},{"key":"951_CR23","unstructured":"Murdoch WJ, Liu PJ, Yu B. Beyond word importance: Contextual decomposition to extract interactions from lstms. arXiv preprint arXiv:1801.05453. 2018."},{"issue":"8","key":"951_CR24","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997; 9(8):1735\u201380. \nhttps:\/\/doi.org\/10.1162\/neco.1997.9.8.1735\n\n.","journal-title":"Neural Comput"},{"issue":"11","key":"951_CR25","doi-asserted-by":"publisher","first-page":"2673","DOI":"10.1109\/78.650093","volume":"45","author":"M Schuster","year":"1997","unstructured":"Schuster M, Paliwal KK. Bidirectional recurrent neural networks. IEEE Trans Sig Process. 1997; 45(11):2673\u201381. \nhttps:\/\/doi.org\/10.1109\/78.650093\n\n.","journal-title":"IEEE Trans Sig Process"},{"key":"951_CR26","doi-asserted-by":"publisher","unstructured":"DeShazo JP, Hoffman MA. A comparison of a multistate inpatient EHR database to the HCUP nationwide inpatient sample. BMC Health Serv Res. 2015; 15(1). \nhttps:\/\/doi.org\/10.1186\/s12913-015-1025-7\n\n.","DOI":"10.1186\/s12913-015-1025-7"},{"key":"951_CR27","unstructured":"Kingma DP, Ba J. Adam: A method for stochastic optimization. 2014. http:\/\/arxiv.org\/abs\/1412.6980."},{"issue":"3","key":"951_CR28","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1007\/s00787-010-0087-7","volume":"19","author":"M Szumilas","year":"2010","unstructured":"Szumilas M. Explaining odds ratios,. J Can Acad Child Adolesc Psychiatry. 2010; 19(3):227\u20139.","journal-title":"J Can Acad Child Adolesc Psychiatry"},{"issue":"2","key":"951_CR29","doi-asserted-by":"publisher","first-page":"183","DOI":"10.2500\/aap.2007.28.2953","volume":"28","author":"M Morais-Almeida","year":"2007","unstructured":"Morais-Almeida M, Gaspar A, Pires G, Prates S, Rosado-Pinto J. Risk factors for asthma symptoms at school age: an 8-year prospective study,. Allergy Asthma Proc. 2007; 28(2):183\u20139.","journal-title":"Allergy Asthma Proc"},{"key":"951_CR30","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1111\/j.1752-699X.2008.00095.x","volume":"2","author":"A Bjerg","year":"2008","unstructured":"Bjerg A, R\u00f6nmark E. Asthma in school age: prevalence and risk factors by time and by age. The Clinical Respiratory Journal. 2008; 2:123\u20136. \nhttps:\/\/doi.org\/10.1111\/j.1752-699X.2008.00095.x\n\n.","journal-title":"The Clinical Respiratory Journal"},{"key":"951_CR31","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.rmed.2017.11.002","volume":"133","author":"SS Szentpetery","year":"2017","unstructured":"Szentpetery SS, Gruzieva O, Forno E, Han Y-Y, Bergstr\u00f6m A, Kull I, Acosta-P\u00e9rez E, Col\u00f3n-Semidey A, Alvarez M, Canino GJ, Mel\u00e9n E, Celed\u00f3n JC. Combined effects of multiple risk factors on asthma in school-aged children,. Respir Med. 2017; 133:16\u201321. \nhttps:\/\/doi.org\/10.1016\/j.rmed.2017.11.002\n\n.","journal-title":"Respir Med"},{"issue":"1","key":"951_CR32","doi-asserted-by":"publisher","first-page":"79","DOI":"10.4104\/pcrj.2010.00076","volume":"20","author":"Y Juhn","year":"2010","unstructured":"Juhn Y, Kung A, Voigt R, Johnson S. Characterisation of children\u2019s asthma status by ICD-9 code and criteria-based medical record review. Prim Care Respir J. 2010; 20(1):79\u201383. \nhttps:\/\/doi.org\/10.4104\/pcrj.2010.00076\n\n.","journal-title":"Prim Care Respir J"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-019-0951-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12911-019-0951-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-019-0951-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,7]],"date-time":"2020-11-07T00:22:57Z","timestamp":1604708577000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-019-0951-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,8]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["951"],"URL":"https:\/\/doi.org\/10.1186\/s12911-019-0951-4","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,8]]},"assertion":[{"value":"28 March 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 October 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 November 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This research was approved by the Institutional Review Board at Sidra Medicine (protocol number 1804023464). Informed consent was exempted because of the retrospective nature of this research. Patient data were anonymized and de-identified by the data provider.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"214"}}