{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T05:51:38Z","timestamp":1770270698970,"version":"3.49.0"},"reference-count":40,"publisher":"Oxford University Press (OUP)","issue":"12","funder":[{"DOI":"10.13039\/100008954","name":"Maine Medical Center","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100008954","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Emergency departments (EDs) continue to pursue optimal patient flow without sacrificing quality of care. The speed with which a healthcare provider receives pertinent information, such as results from clinical orders, can impact flow. We seek to determine if clinical ordering behavior can be predicted at triage during an ED visit.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>Using data available during triage, we trained multilabel machine learning classifiers to predict clinical orders placed during an ED visit. We benchmarked 4 classifiers with 2 multilabel learning frameworks that predict orders independently (binary relevance) or simultaneously (random k-labelsets). We evaluated algorithm performance, calculated variable importance, and conducted a simple simulation study to examine the effects of algorithm implementation on length of stay and cost.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Aggregate performance across orders was highest when predicting orders independently with a multilayer perceptron (median F1 score = 0.56), but prediction frameworks that simultaneously predict orders for a visit enhanced predictive performance for correlated orders. Visit acuity was the most important predictor for most orders. Simulation results indicated that direct implementation of the model would increase ordering costs (from $21 to $45 per visit) but reduce length of stay (from 158 minutes to 151 minutes) over all visits.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>Simulated implementations of the predictive algorithm decreased length of stay but increased ordering costs. Optimal implementation of these predictions to reduce patient length of stay without incurring additional costs requires more exploration.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>It is possible to predict common clinical orders placed during an ED visit with data available at triage.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocz171","type":"journal-article","created":{"date-parts":[[2019,8,30]],"date-time":"2019-08-30T19:27:46Z","timestamp":1567193266000},"page":"1427-1436","source":"Crossref","is-referenced-by-count":30,"title":["Predicting emergency department orders with multilabel machine learning techniques and simulating effects on length of stay"],"prefix":"10.1093","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5853-3668","authenticated-orcid":false,"given":"Haley S","family":"Hunter-Zinck","sequence":"first","affiliation":[{"name":"Department of Emergency Services, VA Boston Healthcare System, Boston, Massachusetts, USA"}]},{"given":"Jordan S","family":"Peck","sequence":"additional","affiliation":[{"name":"Center for Performance Improvement, MaineHealth, Portland, Maine, USA"},{"name":"Department of Emergency Medicine, Tufts University School of Medicine, Medford, Massachusetts, USA"}]},{"given":"Tania D","family":"Strout","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Tufts University School of Medicine, Medford, Massachusetts, USA"},{"name":"Department of Emergency Medicine, Maine Medical Center, Portland, Maine, USA"}]},{"given":"Stephan A","family":"Gaehde","sequence":"additional","affiliation":[{"name":"Department of Emergency Services, VA Boston Healthcare System, Boston, Massachusetts, USA"},{"name":"School of Medicine, Boston University, Boston, Massachusetts, USA"}]}],"member":"286","published-online":{"date-parts":[[2019,10,3]]},"reference":[{"issue":"5","key":"2020110612451589300_ocz171-B1","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1136\/emj.20.5.402","article-title":"Emergency department overcrowding in the United States","volume":"20","author":"Trzeciak","year":"2003","journal-title":"Emerg Med J"},{"issue":"1","key":"2020110612451589300_ocz171-B2","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1111\/acem.13103","article-title":"The impact of emergency department census on the decision to admit","volume":"24","author":"Gorski","year":"2017","journal-title":"Acad Emerg Med"},{"issue":"3","key":"2020110612451589300_ocz171-B3","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1016\/j.ajem.2010.12.035","article-title":"The role of full capacity protocols on mitigating overcrowding in EDs","volume":"30","author":"Villa-Roel","year":"2012","journal-title":"Am J Emerg Med"},{"issue":"12","key":"2020110612451589300_ocz171-B4","doi-asserted-by":"crossref","first-page":"1324","DOI":"10.1111\/j.1553-2712.2011.01236.x","article-title":"The association between length of emergency department boarding and mortality","volume":"18","author":"Singer","year":"2011","journal-title":"Acad Emerg Med"},{"issue":"1","key":"2020110612451589300_ocz171-B5","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1111\/acem.13520","article-title":"Priorities to overcome barriers impacting data science application in emergency care research","volume":"26","author":"Puskarich","year":"2018","journal-title":"Acad Emerg Med"},{"issue":"6","key":"2020110612451589300_ocz171-B6","doi-asserted-by":"crossref","first-page":"870","DOI":"10.1111\/1742-6723.13145","article-title":"Artificial intelligence and machine learning in emergency medicine","volume":"30","author":"Stewart","year":"2018","journal-title":"Emerg Med Australas"},{"key":"2020110612451589300_ocz171-B7","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1093\/jamia\/ocw151","article-title":"The use of computerized clinical decision support systems in emergency care: a substantive review of the literature","volume":"24","author":"Bennett","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020110612451589300_ocz171-B8","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1016\/j.annemergmed.2017.08.005","article-title":"Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index","volume":"71","author":"Levin","year":"2018","journal-title":"Ann Emerg Med"},{"issue":"3","key":"2020110612451589300_ocz171-B9","doi-asserted-by":"crossref","first-page":"e0194085","DOI":"10.1371\/journal.pone.0194085","article-title":"Predicting urinary tract infections in the emergency department with machine learning","volume":"13","author":"Taylor","year":"2018","journal-title":"PLoS One"},{"issue":"4","key":"2020110612451589300_ocz171-B10","doi-asserted-by":"crossref","first-page":"e0174708","DOI":"10.1371\/journal.pone.0174708","article-title":"Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning","volume":"12","author":"Horng","year":"2017","journal-title":"PLoS One"},{"issue":"5","key":"2020110612451589300_ocz171-B11","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1111\/1742-6723.12964","article-title":"Why do \u2018fast track\u2019 patients stay more than four hours in the emergency department? An investigation of factors that predict length of stay","volume":"30","author":"Gill","year":"2018","journal-title":"Emerg Med Australas"},{"issue":"7","key":"2020110612451589300_ocz171-B12","doi-asserted-by":"crossref","first-page":"e0201016.","DOI":"10.1371\/journal.pone.0201016","article-title":"Predicting hospital admission at emergency department triage using machine learning","volume":"13","author":"Hong","year":"2018","journal-title":"PLoS One"},{"key":"2020110612451589300_ocz171-B13","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1111\/j.1553-2712.2012.01435.x","article-title":"Predicting emergency department inpatient admissions to improve same-day patient flow","volume":"19","author":"Peck","year":"2012","journal-title":"Acad Emerg Med"},{"issue":"11","key":"2020110612451589300_ocz171-B14","doi-asserted-by":"crossref","first-page":"1156","DOI":"10.1111\/acem.12244","article-title":"Generalizability of a simple approach for predicting hospital admission from an emergency department","volume":"20","author":"Peck","year":"2013","journal-title":"Acad Emerg Med"},{"issue":"3","key":"2020110612451589300_ocz171-B15","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1016\/j.ajem.2017.08.049","article-title":"Predicting 72-hour emergency department revisits","volume":"36","author":"Pellerin","year":"2018","journal-title":"Am J Emerg Med"},{"issue":"3","key":"2020110612451589300_ocz171-B16","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1080\/19488300.2014.930765","article-title":"Characterizing the value of predictive analytics in facilitating hospital patient flow","volume":"4","author":"Peck","year":"2014","journal-title":"IIE Trans Healthc Syst Eng"},{"issue":"2","key":"2020110612451589300_ocz171-B17","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.1553-2712.2010.00984.x","article-title":"The role of triage liaison physicians on mitigating overcrowding in emergency departments: a systematic review","volume":"18","author":"Rowe","year":"2011","journal-title":"Acad Emerg Med"},{"issue":"e2","key":"2020110612451589300_ocz171-B18","doi-asserted-by":"crossref","first-page":"e304","DOI":"10.1136\/amiajnl-2013-002316","article-title":"Paving the COWpath: data-driven design of pediatric order sets","volume":"21","author":"Zhang","year":"2014","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2020110612451589300_ocz171-B19","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1093\/jamia\/ocv091","article-title":"OrderRex: clinical order decision support and outcome predictions by data-mining electronic medical records","volume":"23","author":"Chen","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612451589300_ocz171-B20","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.jbi.2013.12.005","article-title":"Decision support from local data: creating adaptive order menus from past clinician behavior","volume":"48","author":"Klann","year":"2014","journal-title":"J Biomed Inform"},{"key":"2020110612451589300_ocz171-B21","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/j.jbi.2018.09.005","article-title":"An evaluation of clinical order patterns machine-learned from clinician cohorts stratified by patient mortality outcomes","volume":"86","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"issue":"7","key":"2020110612451589300_ocz171-B22","doi-asserted-by":"crossref","first-page":"1079","DOI":"10.1109\/TKDE.2010.164","article-title":"Random k-Labelsets for multi-label classification","volume":"23","author":"Tsoumakas","year":"2011","journal-title":"IEEE Trans Knowl Data Eng"},{"issue":"7","key":"2020110612451589300_ocz171-B23","doi-asserted-by":"crossref","first-page":"1203","DOI":"10.1377\/hlthaff.2014.0054","article-title":"Insights from advanced analytics at the veterans health administration","volume":"33","author":"Fihn","year":"2014","journal-title":"Health Aff (Millwood)"},{"key":"2020110612451589300_ocz171-B24","article-title":"Emergency Severity Index (ESI): A Triage Tool for Emergency Department Care, Version 4. Implementation Handbook","author":"Gilboy","year":"2011","journal-title":"Agency for Healthcare Research and Quality"},{"issue":"1","key":"2020110612451589300_ocz171-B25","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1111\/j.1553-2712.2004.tb01371.x","article-title":"Reliability and validity of scores on the emergency severity index version 3","volume":"11","author":"Tanabe","year":"2004","journal-title":"Acad Emerg Med"},{"issue":"6","key":"2020110612451589300_ocz171-B26","doi-asserted-by":"crossref","first-page":"716","DOI":"10.1016\/j.annemergmed.2012.07.010","article-title":"Understanding and interpreting the national hospital ambulatory medical care survey: key questions and answers","volume":"60","author":"McCaig","year":"2012","journal-title":"Ann Emerg Med"},{"key":"2020110612451589300_ocz171-B27","author":"Centers for Medicare and Medicaid Services"},{"issue":"database issue","key":"2020110612451589300_ocz171-B28","doi-asserted-by":"crossref","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"issue":"3","key":"2020110612451589300_ocz171-B29","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1197\/jamia.M0913","article-title":"Mapping abbreviations to full forms in biomedical articles","volume":"9","author":"Yu","year":"2002","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2020110612451589300_ocz171-B30","doi-asserted-by":"crossref","first-page":"22","DOI":"10.32614\/RJ-2011-014","article-title":"Watch your spelling!","volume":"3","author":"Hornik","year":"2011","journal-title":"R J"},{"issue":"3","key":"2020110612451589300_ocz171-B31","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1197\/jamia.M1474","article-title":"A frequency-based technique to improve the spelling suggestion rank in medical queries","volume":"11","author":"Crowell","year":"2004","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"2020110612451589300_ocz171-B32","doi-asserted-by":"crossref","first-page":"1819","DOI":"10.1109\/TKDE.2013.39","article-title":"A review on multi-label learning algorithms","volume":"26","author":"Zhang","year":"2014","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"2020110612451589300_ocz171-B33","author":"A Informatics and Computing Infrastructure (VINCI)"},{"issue":"5","key":"2020110612451589300_ocz171-B34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v028.i05","article-title":"Building predictive models in R using the caret package","volume":"28","author":"Kuhn","year":"2008","journal-title":"J Stat Softw"},{"issue":"1","key":"2020110612451589300_ocz171-B35","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A Coefficient of Agreement for Nominal Scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ Psychol Meas"},{"key":"2020110612451589300_ocz171-B36","author":"U.S. Centers for Medicare and Medicaid Services"},{"issue":"7","key":"2020110612451589300_ocz171-B37","doi-asserted-by":"crossref","first-page":"899","DOI":"10.1016\/j.ajem.2015.03.062","article-title":"Emergency medicine in the Veterans Health Administration-results from a nationwide survey","volume":"33","author":"Ward","year":"2015","journal-title":"Am J Emerg Med"},{"issue":"12","key":"2020110612451589300_ocz171-B38","doi-asserted-by":"crossref","first-page":"1833","DOI":"10.1001\/jamainternmed.2017.5152","article-title":"Evidence-based guidelines to eliminate repetitive laboratory testing","volume":"177","author":"Eaton","year":"2017","journal-title":"JAMA Intern Med"},{"issue":"5","key":"2020110612451589300_ocz171-B39","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1016\/j.jacr.2018.11.011","article-title":"Increasing utilization of chest imaging in US emergency departments from 1994 to 2015","volume":"16","author":"Chung","year":"2019","journal-title":"J Am Coll Radiol"},{"issue":"1","key":"2020110612451589300_ocz171-B40","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1136\/amiajnl-2011-000376","article-title":"Validation of a common data model for active safety surveillance research","volume":"19","author":"Overhage","year":"2012","journal-title":"J Am Med Inform Assoc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1427\/34152287\/ocz171.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1427\/34152287\/ocz171.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,19]],"date-time":"2023-09-19T17:54:40Z","timestamp":1695146080000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/12\/1427\/5580383"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,3]]},"references-count":40,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2019,10,3]]},"published-print":{"date-parts":[[2019,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz171","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,12]]},"published":{"date-parts":[[2019,10,3]]}}}