{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T04:20:00Z","timestamp":1773807600305,"version":"3.50.1"},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2022,10,29]],"date-time":"2022-10-29T00:00:00Z","timestamp":1667001600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100008028","name":"UW School of Medicine and Public Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100008028","id-type":"DOI","asserted-by":"publisher"}]},{"name":"the Wisconsin Partnership Program","award":["4755"],"award-info":[{"award-number":["4755"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,1,18]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>To develop a machine learning framework to forecast emergency department (ED) crowding and to evaluate model performance under spatial and temporal data drift.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>We obtained 4 datasets, identified by the location: 1\u2014large academic hospital and 2\u2014rural hospital, and time period: pre-coronavirus disease (COVID) (January 1, 2019\u2013February 1, 2020) and COVID-era (May 15, 2020\u2013February 1, 2021). Our primary target was a binary outcome that is equal to 1 if the number of patients with acute respiratory illness that were ED boarding for more than 4\u00a0h was above a prescribed historical percentile. We trained a random forest and used the area under the curve (AUC) to evaluate out-of-sample performance for 2 experiments: (1) we evaluated the impact of sudden temporal drift by training models using pre-COVID data and testing them during the COVID-era, (2) we evaluated the impact of spatial drift by testing models trained at location 1 on data from location 2, and vice versa.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The baseline AUC values for ED boarding ranged from 0.54 (pre-COVID at location 2) to 0.81 (COVID-era at location 1). Models trained with pre-COVID data performed similarly to COVID-era models (0.82 vs 0.78 at location 1). Models that were transferred from location 2 to location 1 performed worse than models trained at location 1 (0.51 vs 0.78).<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion and Conclusion<\/jats:title><jats:p>Our results demonstrate that ED boarding is a predictable metric for ED crowding, models were not significantly impacted by temporal data drift, and any attempts at implementation must consider spatial data drift.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocac214","type":"journal-article","created":{"date-parts":[[2022,10,29]],"date-time":"2022-10-29T13:47:58Z","timestamp":1667051278000},"page":"292-300","source":"Crossref","is-referenced-by-count":4,"title":["Multisite evaluation of prediction models for emergency department crowding before and during the COVID-19 pandemic"],"prefix":"10.1093","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7863-3447","authenticated-orcid":false,"given":"Ari J","family":"Smith","sequence":"first","affiliation":[{"name":"Department of Industrial and Systems Engineering, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4584-3808","authenticated-orcid":false,"given":"Brian W","family":"Patterson","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"},{"name":"BerbeeWalsh Department of Emergency Medicine, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"},{"name":"Department of Biostatistics and Medical Informatics, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"}]},{"given":"Michael S","family":"Pulia","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"},{"name":"BerbeeWalsh Department of Emergency Medicine, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"}]},{"given":"John","family":"Mayer","sequence":"additional","affiliation":[{"name":"Marshfield Clinic Research Institute , Marshfield, Wisconsin, USA"}]},{"given":"Rebecca J","family":"Schwei","sequence":"additional","affiliation":[{"name":"BerbeeWalsh Department of Emergency Medicine, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"}]},{"given":"Radha","family":"Nagarajan","sequence":"additional","affiliation":[{"name":"Marshfield Clinic Research Institute , Marshfield, Wisconsin, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7577-7530","authenticated-orcid":false,"given":"Frank","family":"Liao","sequence":"additional","affiliation":[{"name":"Applied Data Science, Information Services, UW-Health , Madison, Wisconsin, USA"}]},{"given":"Manish N","family":"Shah","sequence":"additional","affiliation":[{"name":"BerbeeWalsh Department of Emergency Medicine, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0904-4467","authenticated-orcid":false,"given":"Justin J","family":"Boutilier","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"},{"name":"BerbeeWalsh Department of Emergency Medicine, University of Wisconsin\u2013Madison , Madison, Wisconsin, USA"}]}],"member":"286","published-online":{"date-parts":[[2022,10,29]]},"reference":[{"issue":"6","key":"2023011811053528600_ocac214-B1","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1136\/emj.2008.062380","article-title":"A systematic review of models for forecasting the number of emergency department visits","volume":"26","author":"Wargon","year":"2009","journal-title":"Emerg Med J"},{"issue":"2","key":"2023011811053528600_ocac214-B2","doi-asserted-by":"crossref","first-page":"e0192568","DOI":"10.1371\/journal.pone.0192568","article-title":"Seasonality in trauma admissions \u2013 are daylight and weather variables better predictors than general cyclic effects?","volume":"13","author":"R\u00f8islien","year":"2018","journal-title":"PLoS One"},{"issue":"1","key":"2023011811053528600_ocac214-B3","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1186\/s12873-019-0256-z","article-title":"Real-time forecasting of emergency department arrivals using prehospital data","volume":"19","author":"Asheim","year":"2019","journal-title":"BMC Emerg Med"},{"issue":"2","key":"2023011811053528600_ocac214-B4","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1016\/j.annemergmed.2008.03.014","article-title":"Systematic review of emergency department crowding: causes, effects, and solutions","volume":"52","author":"Hoot","year":"2008","journal-title":"Ann Emerg Med"},{"issue":"6","key":"2023011811053528600_ocac214-B5","doi-asserted-by":"crossref","first-page":"605","DOI":"10.1016\/j.annemergmed.2012.10.026","article-title":"Effect of emergency department crowding on outcomes of admitted patients","volume":"61","author":"Sun","year":"2013","journal-title":"Ann Emerg Med"},{"key":"2023011811053528600_ocac214-B6","doi-asserted-by":"publisher","DOI":"10.1056\/CAT.21.0217","article-title":"Emergency department crowding: the canary in the health care system","author":"Kelen","journal-title":"NEJM Catal Innov Care Deliv"},{"issue":"4","key":"2023011811053528600_ocac214-B7","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/j.1553-2712.2009.00356.x","article-title":"Forecasting models of emergency department crowding","volume":"16","author":"Schweigler","year":"2009","journal-title":"Acad Emerg Med"},{"issue":"2","key":"2023011811053528600_ocac214-B8","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1016\/j.annemergmed.2007.12.011","article-title":"Forecasting emergency department crowding: a discrete event simulation","volume":"52","author":"Hoot","year":"2008","journal-title":"Ann Emerg Med"},{"issue":"4","key":"2023011811053528600_ocac214-B9","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1016\/j.annemergmed.2009.06.006","article-title":"Forecasting emergency department crowding: an external, multicenter evaluation","volume":"54","author":"Hoot","year":"2009","journal-title":"Ann Emerg Med"},{"key":"2023011811053528600_ocac214-B10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-227X-9-1","article-title":"Forecasting daily attendances at an emergency department to aid resource planning","volume":"9","author":"Sun","year":"2009","journal-title":"BMC Emerg Med"},{"issue":"3","key":"2023011811053528600_ocac214-B11","doi-asserted-by":"crossref","first-page":"e5650","DOI":"10.2196\/medinform.5650","article-title":"Forecasting daily patient outflow from a ward having no real-time clinical data","volume":"4","author":"Gopakumar","year":"2016","journal-title":"JMIR Med Inform"},{"issue":"2","key":"2023011811053528600_ocac214-B12","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1111\/j.1553-2712.2007.00032.x","article-title":"Forecasting daily patient volumes in the emergency department","volume":"15","author":"Jones","year":"2008","journal-title":"Acad Emerg Med"},{"issue":"1","key":"2023011811053528600_ocac214-B13","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1186\/s13054-019-2351-7","article-title":"Emergency department triage prediction of clinical outcomes using machine learning models","volume":"23","author":"Raita","year":"2019","journal-title":"Crit Care"},{"issue":"3","key":"2023011811053528600_ocac214-B14","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1111\/acem.12876","article-title":"Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach","volume":"23","author":"Taylor","year":"2016","journal-title":"Acad Emerg Med"},{"key":"2023011811053528600_ocac214-B15","doi-asserted-by":"crossref","first-page":"104146","DOI":"10.1016\/j.ijmedinf.2020.104146","article-title":"Emergency department disposition prediction using a deep neural network with integrated clinical narratives and structured data","volume":"139","author":"Chen","year":"2020","journal-title":"Int J Med Inform"},{"issue":"1","key":"2023011811053528600_ocac214-B16","doi-asserted-by":"crossref","first-page":"1381","DOI":"10.1038\/s41598-021-80985-3","article-title":"Predicting adult neuroscience intensive care unit admission from emergency department triage using a retrospective, tabular-free text machine learning approach","volume":"11","author":"Klang","year":"2021","journal-title":"Sci Rep"},{"issue":"1","key":"2023011811053528600_ocac214-B17","doi-asserted-by":"crossref","first-page":"50","DOI":"10.7326\/M20-4096","article-title":"Effect of timing of and adherence to social distancing measures on COVID-19 burden in the United States: a simulation modeling approach","volume":"174","author":"Alagoz","year":"2021","journal-title":"Ann Intern Med"},{"issue":"6","key":"2023011811053528600_ocac214-B18","doi-asserted-by":"crossref","first-page":"6413","DOI":"10.1007\/s10489-021-02743-2","article-title":"Machine learning techniques to predict different levels of hospital care of CoVid-19","volume":"52","author":"Hern\u00e1ndez-Pereira","year":"2022","journal-title":"Appl Intell"},{"issue":"2","key":"2023011811053528600_ocac214-B19","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1007\/s10729-020-09542-0","article-title":"From predictions to prescriptions: a data-driven response to COVID-19","volume":"24","author":"Bertsimas","year":"2021","journal-title":"Health Care Manag Sci"},{"issue":"4","key":"2023011811053528600_ocac214-B20","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1016\/j.annemergmed.2020.07.022","article-title":"Development and validation of the quick COVID-19 severity index: a prognostic tool for early clinical decompensation","volume":"76","author":"Haimovich","year":"2020","journal-title":"Ann Emerg Med"},{"issue":"5","key":"2023011811053528600_ocac214-B21","doi-asserted-by":"crossref","first-page":"e0250890","DOI":"10.1371\/journal.pone.0250890","article-title":"Influenza forecasting for French regions combining EHR, web and climatic data sources with a machine learning ensemble approach","volume":"16","author":"Poirier","year":"2021","journal-title":"PLoS One"},{"issue":"1","key":"2023011811053528600_ocac214-B22","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1111\/j.1553-2712.2004.tb01369.x","article-title":"Estimating the degree of emergency department overcrowding in academic medical centers: results of the National ED Overcrowding Study (NEDOCS)","volume":"11","author":"Weiss","year":"2004","journal-title":"Acad Emerg Med"},{"issue":"4","key":"2023011811053528600_ocac214-B23","doi-asserted-by":"crossref","first-page":"748","DOI":"10.5811\/westjem.2020.5.47606","article-title":"Electronic health record-based surveillance for community transmitted COVID-19 in the emergency department","volume":"21","author":"Pulia","year":"2020","journal-title":"WestJEM"},{"key":"2023011811053528600_ocac214-B24","volume-title":"Applied Logistic Regression","author":"Hosmer","year":"1989"},{"issue":"1","key":"2023011811053528600_ocac214-B25","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the Lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J R Stat Soc B Methodol"},{"issue":"3","key":"2023011811053528600_ocac214-B26","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1080\/00031305.1992.10475879","article-title":"An introduction to kernel and nearest-neighbor nonparametric regression","volume":"46","author":"Altman","year":"1992","journal-title":"Am Statist"},{"issue":"1","key":"2023011811053528600_ocac214-B27","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach Learn"},{"issue":"8","key":"2023011811053528600_ocac214-B28","doi-asserted-by":"crossref","first-page":"e0203316","DOI":"10.1371\/journal.pone.0203316","article-title":"Emergency department crowding: a systematic review of causes, consequences and solutions","volume":"13","author":"Morley","year":"2018","journal-title":"PLoS One"},{"issue":"3","key":"2023011811053528600_ocac214-B29","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1197\/jamia.M2772","article-title":"Forecasting emergency department crowding: a prospective, real-time evaluation","volume":"16","author":"Hoot","year":"2009","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"2023011811053528600_ocac214-B30","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern Med"},{"issue":"7","key":"2023011811053528600_ocac214-B31","doi-asserted-by":"crossref","first-page":"1129","DOI":"10.1513\/AnnalsATS.202006-698OC","article-title":"Evaluating a widely implemented proprietary deterioration index model among hospitalized patients with COVID-19","volume":"18","author":"Singh","year":"2021","journal-title":"Ann Am Thorac Soc"},{"issue":"15","key":"2023011811053528600_ocac214-B32","doi-asserted-by":"crossref","first-page":"465","DOI":"10.15585\/mmwr.mm6915e4","article-title":"Geographic differences in COVID-19 cases, deaths, and incidence \u2013 United States, February 12\u2013April 7, 2020","volume":"69","author":"Bialek","year":"2020","journal-title":"MMWR Morb Mortal Wkly Rep"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocac214\/46684142\/ocac214.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/2\/292\/48754906\/ocac214.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/2\/292\/48754906\/ocac214.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,6]],"date-time":"2024-10-06T20:14:28Z","timestamp":1728245668000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/2\/292\/6779990"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,29]]},"references-count":32,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,10,29]]},"published-print":{"date-parts":[[2023,1,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac214","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,2,1]]},"published":{"date-parts":[[2022,10,29]]}}}