{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T14:48:40Z","timestamp":1780498120364,"version":"3.54.1"},"reference-count":60,"publisher":"Georg Thieme Verlag KG","issue":"02","funder":[{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"crossref","award":["R01LM011370"],"award-info":[{"award-number":["R01LM011370"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2017,4]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>Objectives: This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance.<\/jats:p><jats:p>Methods: We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) \u2018other\u2019 diagnosis.<\/jats:p><jats:p>Results: On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser).<\/jats:p><jats:p>Conclusion: In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.<\/jats:p><jats:p>Citation: Ferraro JP, Ye Y, Gesteland PH, Haug PJ, Tsui F(R), Cooper GF, Van Bree R, Ginter T, Nowalk AJ, Wagner M. The effects of natural language processing on cross-institutional portability of influenza case detection for disease surveillance. Appl Clin Inform 2017; 8: 560\u2013580 https:\/\/doi.org\/10.4338\/ACI-2016-12-RA-0211<\/jats:p>","DOI":"10.4338\/aci-2016-12-ra-0211","type":"journal-article","created":{"date-parts":[[2017,5,31]],"date-time":"2017-05-31T11:28:04Z","timestamp":1496230084000},"page":"560-580","source":"Crossref","is-referenced-by-count":9,"title":["The effects of natural language processing on cross-institutional portability of influenza case detection for disease surveillance"],"prefix":"10.4338","volume":"08","author":[{"given":"Jeffrey","family":"Ferraro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ye","family":"Ye","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Per","family":"Gesteland","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Haug","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fuchiang","family":"Tsui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregory","family":"Cooper","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rudy Van","family":"Bree","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Ginter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Nowalk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Wagner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"194","published-online":{"date-parts":[[2017,12,21]]},"reference":[{"key":"10.4338\/ACI-2016-12-RA-0211-1","doi-asserted-by":"crossref","unstructured":"Shaikh AT, Ferland L, Hood-Cree R, Shaffer L, McNabb SJ. Disruptive Innovation Can Prevent the Next Pandemic. Frontiers in public health 2015; 3","DOI":"10.3389\/fpubh.2015.00215"},{"key":"10.4338\/ACI-2016-12-RA-0211-2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2006.09.003"},{"key":"10.4338\/ACI-2016-12-RA-0211-3","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMra1208802"},{"key":"10.4338\/ACI-2016-12-RA-0211-4","doi-asserted-by":"crossref","unstructured":"Veenema T, T\u00f5ke J. Early detection and surveillance for biopreparedness and emerging infectious diseases. Online journal of issues in nursing 2006; 11(1)","DOI":"10.3912\/OJIN.Vol11No01Man02"},{"key":"10.4338\/ACI-2016-12-RA-0211-5","doi-asserted-by":"publisher","DOI":"10.1089\/bsp.2011.0088"},{"key":"10.4338\/ACI-2016-12-RA-0211-6","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.j280"},{"issue":"14","key":"10.4338\/ACI-2016-12-RA-0211-7","first-page":"373","volume":"64","author":"Clemmons","year":"2015","journal-title":"MMWR Morb Mortal Wkly Rep"},{"key":"10.4338\/ACI-2016-12-RA-0211-8","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2334-14-381"},{"issue":"41","key":"10.4338\/ACI-2016-12-RA-0211-9","first-page":"1421","volume":"60","author":"Control","year":"2011","journal-title":"MMWR: Morbidity and mortality weekly report"},{"key":"10.4338\/ACI-2016-12-RA-0211-10","doi-asserted-by":"crossref","unstructured":"Dixon BE, Siegel JA, Oemig TV, Grannis SJ. Towards Interoperability for public health surveillance: experiences from two states. Online journal of public health informatics 2013; 5(1)","DOI":"10.5210\/ojphi.v5i1.4395"},{"key":"10.4338\/ACI-2016-12-RA-0211-11","unstructured":"Gesteland PH, Wagner MM, Chapman WW, Espino JU, Tsui F-C, Gardner RM, Rolfs RT, Dato V, James BC, Haug PJ. Rapid deployment of an electronic disease surveillance system in the state of Utah for the 2002 Olympic winter games. Proc AMIA Symp 2002: 285-289"},{"key":"10.4338\/ACI-2016-12-RA-0211-12","unstructured":"Centers for Disease Control and Prevention, National Syndromic Surveillance Program (NSSP) - BioSense Platform 2003 [updated March 31, 2016 accessed Apr 2016]. Available from: http:\/\/www.cdc.gov\/nssp\/biosense\/index.html"},{"issue":"1","key":"10.4338\/ACI-2016-12-RA-0211-13","first-page":"i32","volume":"80","author":"Lombardo","year":"2013","journal-title":"J Urban Health"},{"key":"10.4338\/ACI-2016-12-RA-0211-14","first-page":"A5136","volume":"185","author":"Ferraro","year":"2012","journal-title":"Am J Respir Crit Care Med"},{"key":"10.4338\/ACI-2016-12-RA-0211-15","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2013.3299"},{"key":"10.4338\/ACI-2016-12-RA-0211-16","doi-asserted-by":"crossref","unstructured":"Moore CR, Farrag A, Ashkin E. Using Natural Language Processing to Extract Abnormal Results From Cancer Screening Reports. J Patient Saf 2014","DOI":"10.1097\/PTS.0000000000000127"},{"key":"10.4338\/ACI-2016-12-RA-0211-17","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2013-001934"},{"key":"10.4338\/ACI-2016-12-RA-0211-18","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.h1885"},{"key":"10.4338\/ACI-2016-12-RA-0211-19","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ajp.2014.14030423"},{"key":"10.4338\/ACI-2016-12-RA-0211-20","doi-asserted-by":"crossref","unstructured":"Pathak J, Kho AN, Denny JC. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. The Oxford University Press; 2013","DOI":"10.1136\/amiajnl-2013-002428"},{"key":"10.4338\/ACI-2016-12-RA-0211-21","unstructured":"Chapman WW, Dowling JN, Ivanov O, Gesteland PH, Olszewski R, Espino JU, Wagner MM, editors. Evaluating natural language processing applications applied to outbreak and disease surveillance. Proceedings of 36th symposium on the interface: computing science and statistics. 2004; Citeseer"},{"key":"10.4338\/ACI-2016-12-RA-0211-22","doi-asserted-by":"crossref","unstructured":"Chapman WW, Gundlapalli AV, South BR, Dowling JN. Natural language processing for biosurveillance. In: Castillo-Chavez C, Chen H, Lober WB, Thurmond M, Zeng D, editors. Infectious Disease Informatics and Biosurveillance: Springer; 2011. p. 279-310","DOI":"10.1007\/978-1-4419-6892-0_13"},{"key":"10.4338\/ACI-2016-12-RA-0211-23","doi-asserted-by":"publisher","DOI":"10.1002\/pds.3418"},{"key":"10.4338\/ACI-2016-12-RA-0211-24","unstructured":"Gundlapalli AV, Carter ME, Palmer M, Ginter T, Redd A, Pickard S, Shen S, South B, Divita G, Duvall S. Using natural language processing on the free text of clinical documents to screen for evidence of homelessness among US veterans. AMIA Annu Symp Proc 2013; Nov 16 2013: 537-546"},{"key":"10.4338\/ACI-2016-12-RA-0211-25","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2009.08.007"},{"key":"10.4338\/ACI-2016-12-RA-0211-26","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-156-1-201201030-00003"},{"key":"10.4338\/ACI-2016-12-RA-0211-27","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-12-212"},{"key":"10.4338\/ACI-2016-12-RA-0211-28","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2011-000465"},{"issue":"1","key":"10.4338\/ACI-2016-12-RA-0211-29","first-page":"256","volume":"45","author":"Daum\u00e9","year":"2007","journal-title":"Proc 45th Ann Meeting of the Assoc Compuational Linguistics"},{"key":"10.4338\/ACI-2016-12-RA-0211-30","unstructured":"Dredze M, Blitzer J, Talukdar PP, Ganchev K, Graca J, Pereira FC. Frustratingly Hard Domain Adaptation for Dependency Parsing. Conference on Empirical Methods in Natural Language Processing 2007: 1051-1055"},{"key":"10.4338\/ACI-2016-12-RA-0211-31","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2012-001453"},{"key":"10.4338\/ACI-2016-12-RA-0211-32","doi-asserted-by":"crossref","unstructured":"Teixeira PL, Wei W-Q, Cronin RM, Mo H, VanHouten JP, Carroll RJ, LaRose E, Bastarache LA, Rosenbloom ST, Edwards TL. Evaluating electronic health record data sources and algorithmic approaches to identify hypertensive individuals. Journal of the American Medical Informatics Association 2016: ocw071","DOI":"10.1093\/jamia\/ocw071"},{"key":"10.4338\/ACI-2016-12-RA-0211-33","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2011-000583"},{"key":"10.4338\/ACI-2016-12-RA-0211-34","doi-asserted-by":"crossref","unstructured":"Tsui F, Wagner M, Cooper G, Que J, Harkema H, Dowling J, Sriburadej T, Li Q, Espino J, Voorhees R. Probabilistic case detection for disease surveillance using data in electronic medical records. Online J Public Health Inform 2011; 3(3)","DOI":"10.5210\/ojphi.v3i3.3793"},{"key":"10.4338\/ACI-2016-12-RA-0211-35","unstructured":"Russell S, Norvig P. Artificial Intelligence: A Modern Approach. Prentice Hall; 2009. p. 272-319"},{"key":"10.4338\/ACI-2016-12-RA-0211-36","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkh061"},{"key":"10.4338\/ACI-2016-12-RA-0211-37","unstructured":"Samore MH. Natutal language processing: Can it help detect cases and characterize outbreaks? Advances in Disease Surveillance 2008; 5(59)"},{"key":"10.4338\/ACI-2016-12-RA-0211-38","doi-asserted-by":"crossref","unstructured":"Pineda AL, Tsui F-C, Visweswaran S, Cooper GF. Detection of patients with influenza syndrome using machine-learning models learned from emergency department reports. Online J Public Health Inform 2013; 5(1)","DOI":"10.5210\/ojphi.v5i1.4446"},{"key":"10.4338\/ACI-2016-12-RA-0211-39","first-page":"160","volume":"2016","author":"Mehrabi","year":"2016","journal-title":"AMIA Summits on Translational Science Proceedings"},{"key":"10.4338\/ACI-2016-12-RA-0211-40","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaci.2016.12.327"},{"key":"10.4338\/ACI-2016-12-RA-0211-41","unstructured":"Liu H, Bielinski SJ, Sohn S, Murphy S, Kavishwar BW, Jonnalagadda SR, Ravikumar KE, Wu ST, Kullo IJ, Chute CG. An information extraction framework for cohort identification using electronic health records. AMIA Jt Summits Transl Sci Proc 2013: 149-153"},{"key":"10.4338\/ACI-2016-12-RA-0211-42","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.2009.001560"},{"key":"10.4338\/ACI-2016-12-RA-0211-43","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324904003523"},{"key":"10.4338\/ACI-2016-12-RA-0211-44","doi-asserted-by":"publisher","DOI":"10.1197\/jamia.M3037"},{"key":"10.4338\/ACI-2016-12-RA-0211-45","doi-asserted-by":"crossref","unstructured":"Darwiche A. Modeling and reasoning with Bayesian networks: Cambridge University Press; 2009","DOI":"10.1017\/CBO9780511811357"},{"key":"10.4338\/ACI-2016-12-RA-0211-46","unstructured":"Ferraro JP, Allen TL, Briggs B, Haug P, Post H, editors. Development and function of a real-time web-based screening system for emergency department patients with occult septic shock. 2008 Annual Meeting - Socity for Academic Emergency Medicine; 2008; Washington, DC"},{"key":"10.4338\/ACI-2016-12-RA-0211-47","unstructured":"J Leng, S Shen, A Gundlapalli, South B, editors. The Extensible Human Oracle Suite of Tools (eHOST) for Annotation of Clinical Narratives. AMIA Spring Congress; 2010; Phoenix, AZ"},{"key":"10.4338\/ACI-2016-12-RA-0211-48","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"issue":"4","key":"10.4338\/ACI-2016-12-RA-0211-49","first-page":"309","volume":"9","author":"Cooper","year":"1992","journal-title":"Machine learning."},{"key":"10.4338\/ACI-2016-12-RA-0211-50","doi-asserted-by":"crossref","unstructured":"Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Series B Stat Methodol 1995: 289-300","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"10.4338\/ACI-2016-12-RA-0211-51","unstructured":"Press WH, Teukolsky SA, Vetterling WT, Flannery BP. Numerical Recipes in C: The Art of Scientific Computing. 3rd ed. New York, NY: Cambridge University Press; 2007"},{"key":"10.4338\/ACI-2016-12-RA-0211-52","doi-asserted-by":"crossref","unstructured":"DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988: 837-845","DOI":"10.2307\/2531595"},{"key":"10.4338\/ACI-2016-12-RA-0211-53","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.2009.002733"},{"key":"10.4338\/ACI-2016-12-RA-0211-54","unstructured":"Patterson O, Hurdle JF, editors. Document clustering of clinical narratives: a systematic study of clinical sublanguages. AMIA Annu Symp Proc; 2011; Citeseer"},{"key":"10.4338\/ACI-2016-12-RA-0211-55","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2012-001453"},{"key":"10.4338\/ACI-2016-12-RA-0211-56","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2014.08.011"},{"key":"10.4338\/ACI-2016-12-RA-0211-57","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2015.08.019"},{"key":"10.4338\/ACI-2016-12-RA-0211-58","unstructured":"Shi Y, Sha F. Information-theoretical learning of discriminative clusters for unsupervised domain adaptation. Proceedings of International Conference on Machine Learning 2012: 1079-1086"},{"key":"10.4338\/ACI-2016-12-RA-0211-59","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-009-5152-4"},{"key":"10.4338\/ACI-2016-12-RA-0211-60","unstructured":"Blitzer J, Kakade S, Foster DP, editors. Domain adaptation with coupled subspaces. International Conference on Artificial Intelligence and Statistics; 2011"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.4338\/ACI-2016-12-RA-0211.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,29]],"date-time":"2022-07-29T00:18:53Z","timestamp":1659053933000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.4338\/ACI-2016-12-RA-0211"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,4]]},"references-count":60,"journal-issue":{"issue":"02","published-online":{"date-parts":[[2017,12,21]]},"published-print":{"date-parts":[[2017,4]]}},"URL":"https:\/\/doi.org\/10.4338\/aci-2016-12-ra-0211","relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,4]]}}}