{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,20]],"date-time":"2026-06-20T22:10:39Z","timestamp":1781993439685,"version":"3.54.5"},"reference-count":24,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T00:00:00Z","timestamp":1702857600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Sentinel Initiative","award":["75F40119F19002"],"award-info":[{"award-number":["75F40119F19002"]}]},{"name":"Sentinel Initiative","award":["75F40119D10037"],"award-info":[{"award-number":["75F40119D10037"]}]},{"DOI":"10.13039\/100000038","name":"Food and Drug Administration","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000038","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100013017","name":"Vanderbilt University Medical Center","doi-asserted-by":"publisher","award":["UL1 TR002243"],"award-info":[{"award-number":["UL1 TR002243"]}],"id":[{"id":"10.13039\/100013017","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,2,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>Automated phenotyping algorithms can reduce development time and operator dependence compared to manually developed algorithms. One such approach, PheNorm, has performed well for identifying chronic health conditions, but its performance for acute conditions is largely unknown. Herein, we implement and evaluate PheNorm applied to symptomatic COVID-19 disease to investigate its potential feasibility for rapid phenotyping of acute health conditions.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and methods<\/jats:title>\n                  <jats:p>PheNorm is a general-purpose automated approach to creating computable phenotype algorithms based on natural language processing, machine learning, and (low cost) silver-standard training labels. We applied PheNorm to cohorts of potential COVID-19 patients from 2 institutions and used gold-standard manual chart review data to investigate the impact on performance of alternative feature engineering options and implementing externally trained models without local retraining.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Models at each institution achieved AUC, sensitivity, and positive predictive value of 0.853, 0.879, 0.851 and 0.804, 0.976, and 0.885, respectively, at quantiles of model-predicted risk that maximize F1. We report performance metrics for all combinations of silver labels, feature engineering options, and models trained internally versus externally.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Phenotyping algorithms developed using PheNorm performed well at both institutions. Performance varied with different silver-standard labels and feature engineering options. Models developed locally at one site also worked well when implemented externally at the other site.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>PheNorm models successfully identified an acute health condition, symptomatic COVID-19. The simplicity of the PheNorm approach allows it to be applied at multiple study sites with substantially reduced overhead compared to traditional approaches.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad241","type":"journal-article","created":{"date-parts":[[2023,12,19]],"date-time":"2023-12-19T00:12:54Z","timestamp":1702944774000},"page":"574-582","source":"Crossref","is-referenced-by-count":10,"title":["Data-driven automated classification algorithms for acute health conditions: applying PheNorm to COVID-19 disease"],"prefix":"10.1093","volume":"31","author":[{"given":"Joshua C","family":"Smith","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian D","family":"Williamson","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute , Seattle, WA 98101, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David J","family":"Cronkite","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute , Seattle, WA 98101, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jill M","family":"Whitaker","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael F","family":"McLemore","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua T","family":"Osmanski","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Winter","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arvind","family":"Ramaprasan","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute , Seattle, WA 98101, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ann","family":"Kelley","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute , Seattle, WA 98101, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mary","family":"Shea","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute , Seattle, WA 98101, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saranrat","family":"Wittayanukorn","sequence":"additional","affiliation":[{"name":"Center for Drug Evaluation and Research, US Food and Drug Administration , Silver Spring, MD 20903, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danijela","family":"Stojanovic","sequence":"additional","affiliation":[{"name":"Center for Drug Evaluation and Research, US Food and Drug Administration , Silver Spring, MD 20903, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yueqin","family":"Zhao","sequence":"additional","affiliation":[{"name":"Center for Drug Evaluation and Research, US Food and Drug Administration , Silver Spring, MD 20903, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sengwee","family":"Toh","sequence":"additional","affiliation":[{"name":"Harvard Pilgrim Health Care Institute , Boston, MA 02215, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin B","family":"Johnson","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David M","family":"Aronoff","sequence":"additional","affiliation":[{"name":"Department of Medicine, Indiana University School of Medicine , Indianapolis, IN 46202, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8471-0928","authenticated-orcid":false,"given":"David S","family":"Carrell","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute , Seattle, WA 98101, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,12,18]]},"reference":[{"issue":"5","key":"2024021710220045100_ocad241-B1","doi-asserted-by":"crossref","first-page":"793","DOI":"10.1093\/jamia\/ocaa028","article-title":"Using and improving distributed data networks to generate actionable evidence: the case of real-world outcomes in the Food and Drug Administration\u2019s Sentinel system","volume":"27","author":"Brown","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2024021710220045100_ocad241-B2","doi-asserted-by":"crossref","first-page":"101822","DOI":"10.1016\/j.artmed.2020.101822","article-title":"Automated machine learning: review of the state-of-the-art and opportunities for healthcare","volume":"104","author":"Waring","year":"2020","journal-title":"Artif Intell Med"},{"issue":"5","key":"2024021710220045100_ocad241-B3","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1007\/s40264-022-01158-3","article-title":"Applying machine learning in distributed data networks for pharmacoepidemiologic and pharmacovigilance studies: opportunities, challenges, and considerations","volume":"45","author":"Wong","year":"2022","journal-title":"Drug Saf"},{"issue":"22","key":"2024021710220045100_ocad241-B4","doi-asserted-by":"crossref","first-page":"2091","DOI":"10.1056\/NEJMp1809643","article-title":"The FDA Sentinel Initiative\u2014an evolving national resource","volume":"379","author":"Platt","year":"2018","journal-title":"N Engl J Med"},{"issue":"3","key":"2024021710220045100_ocad241-B5","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1002\/cpt.320","article-title":"The FDA\u2019s sentinel initiative\u2014a comprehensive approach to medical product surveillance","volume":"99","author":"Ball","year":"2016","journal-title":"Clin Pharmacol Ther"},{"issue":"e2","key":"2024021710220045100_ocad241-B6","doi-asserted-by":"crossref","first-page":"e226","DOI":"10.1136\/amiajnl-2013-001926","article-title":"Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory","volume":"20","author":"Richesson","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"2024021710220045100_ocad241-B7","doi-asserted-by":"crossref","first-page":"1255","DOI":"10.1093\/jamia\/ocz066","article-title":"High-throughput multimodal automated phenotyping (MAP) with application to PheWAS","volume":"26","author":"Liao","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"2024021710220045100_ocad241-B8","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1093\/jamia\/ocaa104","article-title":"PheMap: a multi-resource knowledge base for high-throughput phenotyping within electronic health records","volume":"27","author":"Zheng","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2024021710220045100_ocad241-B9","doi-asserted-by":"crossref","first-page":"1265","DOI":"10.1093\/jamia\/ocaa343","article-title":"A high-throughput phenotyping algorithm is portable from adult to pediatric populations","volume":"28","author":"Geva","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024021710220045100_ocad241-B10","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1093\/jamia\/ocx111","article-title":"Enabling phenotypic big data with PheNorm","volume":"25","author":"Yu","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2024021710220045100_ocad241-B11","doi-asserted-by":"crossref","first-page":"1166","DOI":"10.1093\/jamia\/ocw028","article-title":"Learning statistical models of phenotypes using noisy labeled training data","volume":"23","author":"Agarwal","year":"2016","journal-title":"J Am Med Inform Assoc"},{"issue":"7","key":"2024021710220045100_ocad241-B12","doi-asserted-by":"crossref","first-page":"e0218919","DOI":"10.1371\/journal.pone.0218919","article-title":"A diagnostic accuracy study validating cardiovascular ICD-9-CM codes in healthcare administrative databases. The Umbria Data-Value Project","volume":"14","author":"Cozzolino","year":"2019","journal-title":"PLoS One"},{"issue":"1","key":"2024021710220045100_ocad241-B13","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.ahj.2004.02.013","article-title":"Accuracy of Medicare claims-based diagnosis of acute myocardial infarction: estimating positive predictive value on the basis of review of hospital records","volume":"148","author":"Kiyota","year":"2004","journal-title":"Am Heart J"},{"key":"2024021710220045100_ocad241-B14","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.2147\/CLEP.S335621","article-title":"Positive predictive value of COVID-19 ICD-10 diagnosis codes across calendar time and clinical setting","volume":"13","author":"Lynch","year":"2021","journal-title":"Clin Epidemiol"},{"key":"2024021710220045100_ocad241-B15","author":"National Institutes of Health. Information on COVID-19 Treatment, Prevention and Research"},{"issue":"e1","key":"2024021710220045100_ocad241-B16","doi-asserted-by":"crossref","first-page":"e143","DOI":"10.1093\/jamia\/ocw135","article-title":"Surrogate-assisted feature extraction for high-throughput phenotyping","volume":"24","author":"Yu","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2024021710220045100_ocad241-B17","doi-asserted-by":"crossref","first-page":"993","DOI":"10.1093\/jamia\/ocv034","article-title":"Toward high-throughput phenotyping: unbiased automated feature extraction and selection from knowledge sources","volume":"22","author":"Yu","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024021710220045100_ocad241-B18","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1038\/s41746-021-00542-0","article-title":"Broadening the reach of the FDA Sentinel system: a roadmap for integrating electronic health record data in a causal analysis framework","volume":"4","author":"Desai","year":"2021","journal-title":"NPJ Digit Med"},{"issue":"1","key":"2024021710220045100_ocad241-B19","doi-asserted-by":"crossref","first-page":"11","DOI":"10.3390\/jpm6010011","article-title":"The Biobank portal for partners personalized medicine: a query tool for working with consented Biobank samples, genotypes, and phenotypes using i2b2","volume":"6","author":"Gainer","year":"2016","journal-title":"J Pers Med"},{"key":"2024021710220045100_ocad241-B20","author":"Smith"},{"key":"2024021710220045100_ocad241-B21","author":"Office for Human Research Protections (OHRP), US Department of Health and Human Services (HHS)"},{"key":"2024021710220045100_ocad241-B22","author":"Rosati","year":"2018"},{"key":"2024021710220045100_ocad241-B23"},{"key":"2024021710220045100_ocad241-B24","author":"National Library of Medicine"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/3\/574\/56691580\/ocad241.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/3\/574\/56691580\/ocad241.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,17]],"date-time":"2024-02-17T10:23:08Z","timestamp":1708165388000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/3\/574\/7477863"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,18]]},"references-count":24,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,12,18]]},"published-print":{"date-parts":[[2024,2,16]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad241","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,3,1]]},"published":{"date-parts":[[2023,12,18]]}}}