{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T00:14:06Z","timestamp":1783037646855,"version":"3.54.6"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T00:00:00Z","timestamp":1658793600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T00:00:00Z","timestamp":1658793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Machine learning for hospital operations is under-studied. We present a prediction pipeline that uses live electronic health-records for patients in a UK teaching hospital\u2019s emergency department (ED) to generate short-term, probabilistic forecasts of emergency admissions. A set of XGBoost classifiers applied to 109,465 ED visits yielded AUROCs from 0.82 to 0.90 depending on elapsed visit-time at the point of prediction. Patient-level probabilities of admission were aggregated to forecast the number of admissions among current ED patients and, incorporating patients yet to arrive, total emergency admissions within specified time-windows. The pipeline gave a mean absolute error (MAE) of 4.0 admissions (mean percentage error of 17%) versus 6.5 (32%) for a benchmark metric. Models developed with 104,504 later visits during the Covid-19 pandemic gave AUROCs of 0.68\u20130.90 and MAE of 4.2 (30%) versus a 4.9 (33%) benchmark. We discuss how we surmounted challenges of designing and implementing models for real-time use, including temporal framing, data preparation, and changing operational conditions.<\/jats:p>","DOI":"10.1038\/s41746-022-00649-y","type":"journal-article","created":{"date-parts":[[2022,7,27]],"date-time":"2022-07-27T09:49:00Z","timestamp":1658915340000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":71,"title":["Machine learning for real-time aggregated prediction of hospital admission for emergency patients"],"prefix":"10.1038","volume":"5","author":[{"given":"Zella","family":"King","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4156-3419","authenticated-orcid":false,"given":"Joseph","family":"Farrington","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin","family":"Utley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Enoch","family":"Kung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samer","family":"Elkhodair","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steve","family":"Harris","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Sekula","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan","family":"Gillham","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3073-3128","authenticated-orcid":false,"given":"Kezhi","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1882-5476","authenticated-orcid":false,"given":"Sonya","family":"Crowe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,7,26]]},"reference":[{"key":"649_CR1","doi-asserted-by":"publisher","first-page":"719","DOI":"10.1038\/s41551-018-0305-z","volume":"2","author":"KH Yu","year":"2018","unstructured":"Yu, K. H., Beam, A. L. & Kohane, I. S. Artificial intelligence in healthcare. Nat. Biomed. Eng. 2, 719\u2013731 (2018).","journal-title":"Nat. Biomed. Eng."},{"key":"649_CR2","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1038\/s42256-020-0176-3","volume":"2","author":"OS Pianykh","year":"2020","unstructured":"Pianykh, O. S. et al. Improving healthcare operations management with machine learning. Nat. Mach. Intell. 2, 266\u2013273 (2020).","journal-title":"Nat. Mach. Intell."},{"key":"649_CR3","doi-asserted-by":"publisher","first-page":"332","DOI":"10.1197\/j.aem.2006.11.011","volume":"14","author":"T Falvo","year":"2007","unstructured":"Falvo, T. et al. The opportunity loss of boarding admitted patients in the emergency department. Acad. Emerg. Med. J. Soc. Acad. Emerg. Med. 14, 332\u2013337 (2007).","journal-title":"Acad. Emerg. Med. J. Soc. Acad. Emerg. Med."},{"key":"649_CR4","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1016\/j.jemermed.2010.06.028","volume":"42","author":"ES Powell","year":"2012","unstructured":"Powell, E. S. et al. The relationship between inpatient discharge timing and emergency department boarding. J. Emerg. Med. 42, 186\u2013196 (2012).","journal-title":"J. Emerg. Med."},{"key":"649_CR5","doi-asserted-by":"publisher","first-page":"e015676","DOI":"10.1136\/bmjopen-2016-015676","volume":"7","author":"N Stylianou","year":"2017","unstructured":"Stylianou, N., Fackrell, R. & Vasilakis, C. Are medical outliers associated with worse patient outcomes? A retrospective study within a regional NHS hospital using routine data. BMJ Open 7, e015676 (2017).","journal-title":"BMJ Open"},{"key":"649_CR6","doi-asserted-by":"publisher","first-page":"189","DOI":"10.15441\/ceem.18.022","volume":"6","author":"P McKenna","year":"2019","unstructured":"McKenna, P. et al. Emergency department and hospital crowding: causes, consequences, and cures. Clin. Exp. Emerg. Med. 6, 189\u2013195 (2019).","journal-title":"Clin. Exp. Emerg. Med."},{"key":"649_CR7","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1136\/emermed-2020-210192","volume":"39","author":"C Patry","year":"2021","unstructured":"Patry, C. et al. Older medical outliers on surgical wards: impact on 6-month outcomes. Emerg. Med. J. 39, 181\u2013185 (2021).","journal-title":"Emerg. Med. J."},{"key":"649_CR8","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1111\/rssa.12008","volume":"177","author":"SK Sahu","year":"2014","unstructured":"Sahu, S. K., Baffour, B., Harper, P. R., Minty, J. H. & Sarran, C. A hierarchical Bayesian model for improving short-term forecasting of hospital demand by including meteorological information. J. R. Stat. Soc. Ser. A Stat. Soc. 177, 39\u201361 (2014).","journal-title":"J. R. Stat. Soc. Ser. A Stat. Soc."},{"key":"649_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0201016","volume":"13","author":"WS Hong","year":"2018","unstructured":"Hong, W. S., Haimovich, A. D. & Taylor, R. A. Predicting hospital admission at emergency department triage using machine learning. PLoS ONE 13, 1\u201313 (2018).","journal-title":"PLoS ONE"},{"key":"649_CR10","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1023\/A:1019767900627","volume":"5","author":"PR Harper","year":"2002","unstructured":"Harper, P. R. A framework for operational modelling of hospital resources. Health Care Manag. Sci. 5, 165\u2013173 (2002).","journal-title":"Health Care Manag. Sci."},{"key":"649_CR11","doi-asserted-by":"publisher","first-page":"424","DOI":"10.12968\/bjhc.2011.17.9.424","volume":"17","author":"R Jones","year":"2011","unstructured":"Jones, R. Volatility in bed occupancy for emergency admissions. Br. J. Healthc. Manag. 17, 424\u2013430 (2011).","journal-title":"Br. J. Healthc. Manag."},{"key":"649_CR12","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1111\/j.1538-4632.2000.tb00425.x","volume":"32","author":"P Congdon","year":"2000","unstructured":"Congdon, P. A Bayesian approach to prediction using the gravity model, with an application to patient flow modeling. Geogr. Anal. 32, 205\u2013224 (2000).","journal-title":"Geogr. Anal."},{"key":"649_CR13","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.artmed.2009.09.001","volume":"48","author":"P Demeester","year":"2010","unstructured":"Demeester, P., Souffriau, W., De Causmaecker, P. & Vanden Berghe, G. A hybrid tabu search algorithm for automatically assigning patients to beds. Artif. Intell. Med. 48, 61\u201370 (2010).","journal-title":"Artif. Intell. Med."},{"key":"649_CR14","doi-asserted-by":"publisher","first-page":"1156","DOI":"10.1111\/acem.12244","volume":"20","author":"JS Peck","year":"2013","unstructured":"Peck, J. S. et al. Generalizability of a simple approach for predicting hospital sdmission from an emergency department. Acad. Emerg. Med. 20, 1156\u20131163 (2013).","journal-title":"Acad. Emerg. Med."},{"key":"649_CR15","doi-asserted-by":"publisher","first-page":"844","DOI":"10.1111\/j.1553-2712.2011.01125.x","volume":"18","author":"Y Sun","year":"2011","unstructured":"Sun, Y., Heng, B. H., Tay, S. Y. & Seow, E. Predicting hospital admissions at emergency department triage using routine administrative data. Acad. Emerg. Med 18, 844\u2013850 (2011).","journal-title":"Acad. Emerg. Med"},{"key":"649_CR16","doi-asserted-by":"publisher","first-page":"10458","DOI":"10.1109\/ACCESS.2018.2808843","volume":"6","author":"B Graham","year":"2018","unstructured":"Graham, B., Bond, R., Quinn, M. & Mulvenna, M. Using data mining to predict hospital admissions from the emergency department. IEEE Access 6, 10458\u201310469 (2018).","journal-title":"IEEE Access"},{"key":"649_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13054-019-2351-7","volume":"23","author":"Y Raita","year":"2019","unstructured":"Raita, Y. et al. Emergency department triage prediction of clinical outcomes using machine learning models. Crit. Care 23, 1\u201313 (2019).","journal-title":"Crit. Care"},{"key":"649_CR18","doi-asserted-by":"publisher","first-page":"308","DOI":"10.1136\/emermed-2014-203819","volume":"34","author":"Y Barak-Corren","year":"2017","unstructured":"Barak-Corren, Y., Israelit, S. H. & Reis, B. Y. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg. Med. J. 34, 308\u2013314 (2017).","journal-title":"Emerg. Med. J."},{"key":"649_CR19","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1136\/emermed-2016-205846","volume":"35","author":"JA Lucke","year":"2018","unstructured":"Lucke, J. A. et al. Early prediction of hospital admission for emergency department patients: a comparison between patients younger or older than 70 years. Emerg. Med. J. 35, 18\u201327 (2018).","journal-title":"Emerg. Med. J."},{"key":"649_CR20","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1109\/JBHI.2020.2990309","volume":"25","author":"R El-Bouri","year":"2021","unstructured":"El-Bouri, R., Eyre, D. W., Watkinson, P., Zhu, T. & Clifton, D. A. Hospital admission location prediction via deep interpretable networks for the year-round improvement of emergency patient care. IEEE J. Biomed. Health Inform. 25, 289\u2013300 (2021).","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"649_CR21","doi-asserted-by":"publisher","unstructured":"Kanter, J. M., Gillespie, O. & Veeramachaneni, K. Label, segment, featurize: a cross domain framework for prediction engineering. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) 430\u2013439. https:\/\/doi.org\/10.1109\/DSAA.2016.54 (2016).","DOI":"10.1109\/DSAA.2016.54"},{"key":"649_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-021-00529-x","volume":"4","author":"SM Lauritsen","year":"2021","unstructured":"Lauritsen, S. M. et al. The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards. Npj Digit. Med. 4, 1\u201312 (2021).","journal-title":"Npj Digit. Med."},{"key":"649_CR23","unstructured":"Nestor, B. et al. Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks. Preprint at https:\/\/arxiv.org\/abs\/1908.00690 (2019)."},{"key":"649_CR24","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1038\/s41591-020-0789-4","volume":"26","author":"SL Hyland","year":"2020","unstructured":"Hyland, S. L. et al. Early prediction of circulatory failure in the intensive care unit using machine learning. Nat. Med. 26, 364\u2013373 (2020).","journal-title":"Nat. Med."},{"key":"649_CR25","doi-asserted-by":"publisher","first-page":"e020124","DOI":"10.1136\/bmjopen-2017-020124","volume":"8","author":"BA Fritz","year":"2018","unstructured":"Fritz, B. A. et al. Using machine learning techniques to develop forecasting algorithms for postoperative complications: protocol for a retrospective study. BMJ Open 8, e020124 (2018).","journal-title":"BMJ Open"},{"key":"649_CR26","doi-asserted-by":"publisher","first-page":"905","DOI":"10.1016\/S2213-2600(18)30300-X","volume":"6","author":"A Meyer","year":"2018","unstructured":"Meyer, A. et al. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir. Med. 6, 905\u2013914 (2018).","journal-title":"Lancet Respir. Med."},{"key":"649_CR27","doi-asserted-by":"publisher","first-page":"E1045","DOI":"10.1111\/j.1553-2712.2012.01435.x","volume":"19","author":"JS Peck","year":"2012","unstructured":"Peck, J. S., Benneyan, J. C., Nightingale, D. J. & Gaehde, S. A. Predicting emergency department inpatient admissions to improve same-day patient flow. Acad. Emerg. Med 19, E1045\u2013E1054 (2012).","journal-title":"Acad. Emerg. Med"},{"key":"649_CR28","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1023\/A:1023333002675","volume":"6","author":"M Utley","year":"2003","unstructured":"Utley, M., Gallivan, S., Treasure, T. & Valencia, O. Analytical methods for calculating the capacity required to operate an effective booked admissions policy for elective inpatient services. Health Care Manag. Sci. 6, 97\u2013104 (2003).","journal-title":"Health Care Manag. Sci."},{"key":"649_CR29","doi-asserted-by":"crossref","unstructured":"Seneviratne, M. G., Shah, N. H. & Chu, L. Bridging the implementation gap of machine learning in healthcare. BMJ Innov. 6, (2020).","DOI":"10.1136\/bmjinnov-2019-000359"},{"key":"649_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-00318-y","volume":"3","author":"RC Li","year":"2020","unstructured":"Li, R. C., Asch, S. M. & Shah, N. H. Developing a delivery science for artificial intelligence in healthcare. Npj Digit. Med. 3, 1\u20133 (2020).","journal-title":"Npj Digit. Med."},{"key":"649_CR31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-021-00537-x","volume":"4","author":"Y Barak-Corren","year":"2021","unstructured":"Barak-Corren, Y. et al. Prediction across healthcare settings: a case study in predicting emergency department disposition. Npj Digit. Med. 4, 1\u20137 (2021).","journal-title":"Npj Digit. Med."},{"key":"649_CR32","first-page":"1","volume":"0","author":"T England","year":"2019","unstructured":"England, T. et al. Near real-time bed modelling feasibility study. J. Simul. 0, 1\u201312 (2019).","journal-title":"J. Simul."},{"key":"649_CR33","unstructured":"Wang, H. & Abraham, Z. Concept drift detection for streaming data. Preprint at http:\/\/arxiv.org\/abs\/1504.01044 (2015)."},{"key":"649_CR34","unstructured":"\u017dliobait\u0117, I. Learning under concept drift: an overview. Preprint at http:\/\/arxiv.org\/abs\/1010.4784 (2010)."},{"key":"649_CR35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-021-00509-1","volume":"4","author":"JP Richardson","year":"2021","unstructured":"Richardson, J. P. et al. Patient apprehensions about the use of artificial intelligence in healthcare. Npj Digit. Med. 4, 1\u20136 (2021).","journal-title":"Npj Digit. Med."},{"key":"649_CR36","doi-asserted-by":"publisher","first-page":"501","DOI":"10.21037\/atm.2020.03.63","volume":"8","author":"F Cabitza","year":"2020","unstructured":"Cabitza, F., Campagner, A. & Balsano, C. Bridging the \u201clast mile\u201d gap between AI implementation and operation: \u201cdata awareness\u201d that matters. Ann. Transl. Med. 8, 501 (2020).","journal-title":"Ann. Transl. Med."},{"key":"649_CR37","doi-asserted-by":"publisher","first-page":"34265","DOI":"10.1136\/bmjopen-2019-034265","volume":"10","author":"K Kohler","year":"2020","unstructured":"Kohler, K. & Ercole, A. Can network science reveal structure in a complex healthcare system? A network analysis using data from emergency surgical services. BMJ Open 10, 34265 (2020).","journal-title":"BMJ Open"},{"key":"649_CR38","doi-asserted-by":"publisher","first-page":"e16323","DOI":"10.2196\/16323","volume":"21","author":"E Coiera","year":"2019","unstructured":"Coiera, E. The last mile: where artificial intelligence meets reality. J. Med. Internet Res. 21, e16323 (2019).","journal-title":"J. Med. Internet Res."},{"key":"649_CR39","doi-asserted-by":"publisher","first-page":"1903","DOI":"10.21105\/joss.01903","volume":"4","author":"M Lang","year":"2019","unstructured":"Lang, M. et al. mlr3: A modern object-oriented machine learning framework in R. J. Open Source Softw. 4, 1903 (2019).","journal-title":"J. Open Source Softw."},{"key":"649_CR40","doi-asserted-by":"publisher","unstructured":"Chen, T. & Guestrin, C. XGBoost: a scalable tree boosting system. https:\/\/doi.org\/10.1145\/2939672.2939785 (2016).","DOI":"10.1145\/2939672.2939785"},{"key":"649_CR41","first-page":"1","volume":"55","author":"MB Wilk","year":"1968","unstructured":"Wilk, M. B. & Gnanadesikan, R. Probability plotting methods for the analysis for the analysis of data. Biometrika 55, 1\u201317 (1968).","journal-title":"Biometrika"},{"key":"649_CR42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v033.i01","volume":"33","author":"J Friedman","year":"2010","unstructured":"Friedman, J., Hastie, T. & Tibshirani, R. Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33, 1\u201322 (2010).","journal-title":"J. Stat. Softw"},{"key":"649_CR43","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1016\/j.orhc.2017.08.003","volume":"15","author":"C Pagel","year":"2017","unstructured":"Pagel, C. et al. Development, implementation and evaluation of a tool for forecasting short term demand for beds in an intensive care unit. Oper. Res. Health Care 15, 19\u201331 (2017).","journal-title":"Oper. Res. Health Care"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00649-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00649-y","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00649-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T03:10:37Z","timestamp":1669345837000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00649-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,26]]},"references-count":43,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["649"],"URL":"https:\/\/doi.org\/10.1038\/s41746-022-00649-y","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.03.07.22271999","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,26]]},"assertion":[{"value":"7 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 July 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"104"}}