{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T10:42:53Z","timestamp":1784544173418,"version":"3.55.0"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2025,1,16]],"date-time":"2025-01-16T00:00:00Z","timestamp":1736985600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003550","name":"Queensland Government","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003550","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Advanced Queensland Industry Research"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>The degree to which deployed artificial intelligence-based deterioration prediction algorithms (AI-DPA) differ in their development, the reasons for these differences, and how this may impact their performance remains unclear. Our primary objective was to identify design factors and associated decisions related to the development of AI-DPA and highlight deficits that require further research.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Based on a systematic review of 14 deployed AI-DPA and an updated systematic search, we identified studies of 12 eligible AI-DPA from which data were extracted independently by 2 investigators on all design factors, decisions, and justifications pertaining to 6 machine learning development stages: (1) model requirements, (2) data collection, (3) data cleaning, (4) data labeling, (5) feature engineering, and (6) model training.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We found 13 design factors and 315 decision alternatives likely to impact AI-DPA performance, all of which varied, together with their rationales, between all included AI-DPA. Variable selection, data imputation methods, training data exclusions, training sample definitions, length of lookback periods, and definition of outcome labels were key design factors accounting for most variation. In justifying decisions, most studies made no reference to prior research or compared with other state-of-the-art algorithms.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Algorithm design decisions regarding factors impacting AI-DPA performance have little supporting evidence, are inconsistent, do not learn from prior work, and lack reference standards.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Several deficits in AI-DPA development that prevent implementers selecting the most accurate algorithm have been identified, and future research needs to address these deficits as a priority.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae321","type":"journal-article","created":{"date-parts":[[2025,2,18]],"date-time":"2025-02-18T11:40:23Z","timestamp":1739878823000},"page":"492-509","source":"Crossref","is-referenced-by-count":9,"title":["Factors underpinning the performance of implemented artificial intelligence-based patient deterioration prediction systems: reasons for selection and implications for hospitals and researchers"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5642-5188","authenticated-orcid":false,"given":"Anton H","family":"van der Vegt","sequence":"first","affiliation":[{"name":"Centre for Health Services Research, The University of Queensland , Brisbane, QLD 4102,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Victoria","family":"Campbell","sequence":"additional","affiliation":[{"name":"Intensive Care Unit, Sunshine Coast Hospital and Health Service , Birtinya, QLD 4575,","place":["Australia"]},{"name":"Patient Safety and Quality, Clinical Excellence Queensland , Brisbane, QLD 4001,","place":["Australia"]},{"name":"School of Medicine, Griffith University, Sunshine Coast University Hospital , Birtinya, QLD 4575,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuyi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer Science, The University of Queensland , St Lucia, QLD 4067,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Malycha","sequence":"additional","affiliation":[{"name":"Intensive Care Department, Royal Adelaide Hospital , Adelaide, SA 5000,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ian A","family":"Scott","sequence":"additional","affiliation":[{"name":"Centre for Health Services Research, The University of Queensland , Brisbane, QLD 4102,","place":["Australia"]},{"name":"Metro South Digital Health and Informatics, Princess Alexandra Hospital , Brisbane, QLD 4102,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,1,16]]},"reference":[{"key":"2025021811400466100_ocae321-B1","doi-asserted-by":"publisher","first-page":"e0677","DOI":"10.1097\/cce.0000000000000677","article-title":"Defining physiological decompensation: an expert consensus and retrospective outcome validation","volume":"4","author":"Mitchell","year":"2022","journal-title":"Crit Care Explor"},{"key":"2025021811400466100_ocae321-B2","volume-title":"Trends in Inpatient Hospital Deaths: National Hospital Discharge Survey, 2000-2010","author":"Hall","year":"2013"},{"key":"2025021811400466100_ocae321-B3","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1093\/jamia\/ocad220","article-title":"Systematic review and longitudinal analysis of implementing artificial intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain","volume":"31","author":"Van Der Vegt","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025021811400466100_ocae321-B4","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1164\/rccm.201406-1022OC","article-title":"Multicenter development and validation of a risk stratification tool for ward patients","volume":"190","author":"Churpek","year":"2014","journal-title":"Am J Respir Crit Care Med"},{"key":"2025021811400466100_ocae321-B5","doi-asserted-by":"publisher","first-page":"343","DOI":"10.3390\/jcm9020343","article-title":"MEWS++: enhancing the prediction of clinical deterioration in admitted patients through a machine learning model","volume":"9","author":"Kia","year":"2020","journal-title":"J Clin Med"},{"key":"2025021811400466100_ocae321-B6","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1164\/rccm.202007-2700OC","article-title":"Detecting deteriorating patients in the hospital: development and validation of a novel scoring system","volume":"204","author":"Pimentel","year":"2021","journal-title":"Am J Respir Crit Care Med"},{"key":"2025021811400466100_ocae321-B7","author":"Royal College of Physicians","year":"2017"},{"key":"2025021811400466100_ocae321-B8","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat Mach Intell"},{"key":"2025021811400466100_ocae321-B9","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI210847","volume-title":"Studies in Health Technology and Informatics","author":"Harrison","year":"2022"},{"key":"2025021811400466100_ocae321-B10","doi-asserted-by":"publisher","first-page":"866","DOI":"10.7326\/M18-1990","article-title":"Ensuring fairness in machine learning to advance health equity","volume":"169","author":"Rajkomar","year":"2018","journal-title":"Ann Intern Med"},{"key":"2025021811400466100_ocae321-B11","doi-asserted-by":"publisher","first-page":"172","DOI":"10.5694\/mja2.52195","article-title":"Why clinical artificial intelligence is (almost) non-existent in Australian hospitals and how to fix it","volume":"220","author":"Van Der Vegt","year":"2024","journal-title":"Med J Aust"},{"key":"2025021811400466100_ocae321-B12","doi-asserted-by":"publisher","first-page":"e25187","DOI":"10.2196\/25187","article-title":"Machine learning\u2013based early warning systems for clinical deterioration: systematic scoping review","volume":"23","author":"Muralitharan","year":"2021","journal-title":"J Med Internet Res"},{"key":"2025021811400466100_ocae321-B13","doi-asserted-by":"publisher","first-page":"25","DOI":"10.3390\/informatics7030025","article-title":"Clinical implementation of predictive models embedded within electronic health record systems: a systematic review","volume":"7","author":"Lee","year":"2020","journal-title":"Informatics"},{"key":"2025021811400466100_ocae321-B14","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1016\/j.jclinepi.2019.02.004","article-title":"A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models","volume":"110","author":"Christodoulou","year":"2019","journal-title":"J Clin Epidemiol"},{"key":"2025021811400466100_ocae321-B15","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1002\/jhm.2009","article-title":"A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards","volume":"8","author":"Bailey","year":"2013","journal-title":"J Hosp Med"},{"key":"2025021811400466100_ocae321-B16","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1016\/j.jbi.2016.09.013","article-title":"Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU","volume":"64","author":"Kipnis","year":"2016","journal-title":"J Biomed Inform"},{"key":"2025021811400466100_ocae321-B17","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1109\/ICSE-SEIP.2019.00042","author":"Amershi","year":"2019"},{"key":"2025021811400466100_ocae321-B18","doi-asserted-by":"publisher","first-page":"e078378","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"Collins","year":"2024","journal-title":"BMJ"},{"key":"2025021811400466100_ocae321-B19","doi-asserted-by":"publisher","first-page":"e961","DOI":"10.1097\/CCM.0000000000005064","article-title":"A trend-based early warning score can be implemented in a hospital electronic medical record to effectively predict inpatient deterioration","volume":"49","author":"Bell","year":"2021","journal-title":"Crit Care Med"},{"key":"2025021811400466100_ocae321-B20","doi-asserted-by":"publisher","first-page":"e1920733","DOI":"10.1001\/jamanetworkopen.2019.20733","article-title":"Prospective and external evaluation of a machine learning model to predict in-hospital mortality of adults at time of admission","volume":"3","author":"Brajer","year":"2020","journal-title":"JAMA Netw Open"},{"key":"2025021811400466100_ocae321-B21","doi-asserted-by":"publisher","first-page":"e008678","DOI":"10.1161\/JAHA.118.008678","article-title":"An algorithm based on deep learning for predicting in-hospital cardiac arrest","volume":"7","author":"Kwon","year":"2018","journal-title":"J Am Heart Assoc"},{"key":"2025021811400466100_ocae321-B22","doi-asserted-by":"publisher","first-page":"286","DOI":"10.1186\/s13054-018-2194-7","article-title":"Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital","volume":"22","author":"Dziadzko","year":"2018","journal-title":"Crit Care"},{"key":"2025021811400466100_ocae321-B23","doi-asserted-by":"publisher","first-page":"2381468319899663","DOI":"10.1177\/2381468319899663","article-title":"Development, implementation, and evaluation of an in-hospital optimized early warning score for patient deterioration","volume":"5","author":"O'Brien","year":"2020","journal-title":"MDM Policy Pract"},{"key":"2025021811400466100_ocae321-B24","doi-asserted-by":"publisher","first-page":"932123","DOI":"10.3389\/fdgth.2022.932123","article-title":"From compute to care: lessons learned from deploying an early warning system into clinical practice","volume":"4","author":"Pou-Prom","year":"2022","journal-title":"Front Digit Health"},{"key":"2025021811400466100_ocae321-B25","doi-asserted-by":"publisher","first-page":"1207","DOI":"10.1093\/jamia\/ocaa347","article-title":"Using machine learning to improve the accuracy of patient deterioration predictions: Mayo Clinic Early Warning Score (MC-EWS)","volume":"28","author":"Romero-Brufau","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2025021811400466100_ocae321-B26","doi-asserted-by":"publisher","DOI":"10.1101\/2024.06.04.24308436","article-title":"Multisite pragmatic cluster-randomized controlled trial of the concern early warning system","volume":"4","author":"Rossetti","year":"2024","journal-title":"medRxiv"},{"key":"2025021811400466100_ocae321-B27","doi-asserted-by":"publisher","first-page":"1242","DOI":"10.1093\/jamia\/ocab006","article-title":"Healthcare Process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM-ExpertSignals): development and evaluation of a conceptual framework","volume":"28","author":"Rossetti","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2025021811400466100_ocae321-B28","author":"Churpek","year":"2024"},{"key":"2025021811400466100_ocae321-B29","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1097\/TA.0000000000003431","article-title":"Electronic health record machine learning model predicts trauma inpatient mortality in real time: a validation study","volume":"92","author":"Mou","year":"2022","journal-title":"J Trauma Acute Care Surg"},{"key":"2025021811400466100_ocae321-B30","first-page":"3243","article-title":"BiovitalsTM: a personalized multivariate physiology analytics using continuous mobile biosensors","volume":"2019","author":"Chen","year":"2019","journal-title":"Annu Int Conf IEEE Eng Med Biol Soc"},{"key":"2025021811400466100_ocae321-B31","doi-asserted-by":"publisher","first-page":"e13719","DOI":"10.2196\/13719","article-title":"A real-time early warning system for monitoring inpatient mortality risk: prospective study using electronic medical record data","volume":"21","author":"Ye","year":"2019","journal-title":"J Med Internet Res"},{"key":"2025021811400466100_ocae321-B32","doi-asserted-by":"publisher","first-page":"065004","DOI":"10.1088\/1361-6579\/ad4e90","article-title":"Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards","volume":"45","author":"Keim-Malpass","year":"2024","journal-title":"Physiol Meas"},{"key":"2025021811400466100_ocae321-B33","doi-asserted-by":"publisher","first-page":"797","DOI":"10.1007\/s10877-019-00361-5","article-title":"Towards development of alert thresholds for clinical deterioration using continuous predictive analytics monitoring","volume":"34.4","author":"Keim-Malpass","year":"2020","journal-title":"J Clin Monit Comput"},{"key":"2025021811400466100_ocae321-B34","volume-title":"National Early Warning Score (NEWS): Standardising the Assessment of Acute-Illness Severity in the NHS\u2014Report of a Working Party","author":"Royal College of Physicians London","year":"2012"},{"key":"2025021811400466100_ocae321-B35","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1093\/qjmed\/94.10.521","article-title":"Validation of a modified Early Warning Score in medical admissions","volume":"94","author":"Subbe","year":"2001","journal-title":"QJM"},{"key":"2025021811400466100_ocae321-B36","doi-asserted-by":"publisher","first-page":"564","DOI":"10.1016\/j.resuscitation.2012.09.024","article-title":"Predicting clinical deterioration in the hospital: the impact of outcome selection","volume":"84","author":"Churpek","year":"2013","journal-title":"Resuscitation"},{"key":"2025021811400466100_ocae321-B37","doi-asserted-by":"publisher","first-page":"e439","DOI":"10.1097\/CCM.0000000000006333","article-title":"Beyond unplanned ICU transfers: linking a revised definition of deterioration to patient outcomes","volume":"52","author":"Byrd","year":"2024","journal-title":"Crit Care Med"},{"key":"2025021811400466100_ocae321-B38","doi-asserted-by":"publisher","first-page":"1280","DOI":"10.1097\/CCM.0000000000005568","article-title":"A call for a consensus approach to the design, implementation, and evaluation of early warning systems*","volume":"50","author":"Bhavani","year":"2022","journal-title":"Crit Care Med"},{"key":"2025021811400466100_ocae321-B39","doi-asserted-by":"publisher","first-page":"1198","DOI":"10.1097\/CCM.0000000000005554","article-title":"The impact of health information technology for early detection of patient deterioration on mortality and length of stay in the hospital acute care setting: systematic review and meta-analysis*","volume":"50","author":"Herasevich","year":"2022","journal-title":"Crit Care Med"},{"key":"2025021811400466100_ocae321-B40","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1136\/bmjqs-2021-014527","article-title":"A scoping review of real-time automated clinical deterioration alerts and evidence of impacts on hospitalised patient outcomes","volume":"31","author":"Blythe","year":"2022","journal-title":"BMJ Qual Saf"},{"key":"2025021811400466100_ocae321-B41","doi-asserted-by":"publisher","first-page":"e045849","DOI":"10.1136\/bmjopen-2020-045849","article-title":"Performance of universal early warning scores in different patient subgroups and clinical settings: a systematic review","volume":"11","author":"Alhmoud","year":"2021","journal-title":"BMJ Open"},{"key":"2025021811400466100_ocae321-B42","doi-asserted-by":"publisher","first-page":"e2438986","DOI":"10.1001\/jamanetworkopen.2024.38986","article-title":"Early warning scores with and without artificial intelligence","volume":"7","author":"Edelson","year":"2024","journal-title":"JAMA Netw Open"},{"key":"2025021811400466100_ocae321-B43","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1038\/s41597-019-0103-9","article-title":"Multitask learning and benchmarking with clinical time series data","volume":"6","author":"Harutyunyan","year":"2019","journal-title":"Sci Data"},{"key":"2025021811400466100_ocae321-B44","doi-asserted-by":"publisher","first-page":"e0235424","DOI":"10.1371\/journal.pone.0235424","volume-title":"PlOS One","author":"Sheikhalishahi","year":"2020"},{"key":"2025021811400466100_ocae321-B45","doi-asserted-by":"publisher","first-page":"m1501","DOI":"10.1136\/bmj.m1501","article-title":"Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology","volume":"369","author":"Gerry","year":"2020","journal-title":"BMJ"},{"key":"2025021811400466100_ocae321-B46","doi-asserted-by":"publisher","DOI":"10.1101\/2024.10.20.24315403","article-title":"Improved sensitivity for detection of clinical deterioration when diagnostic pathology and patient trends are included in machine learning models","volume-title":"medRxiv","author":"Greenberg","year":"2024"},{"key":"2025021811400466100_ocae321-B47","doi-asserted-by":"publisher","first-page":"e497","DOI":"10.1016\/S2589-7500(22)00067-X","article-title":"Clinical assessment as a part of an early warning score\u2014a Danish cluster-randomised, multicentre study of an individual early warning score","volume":"4","author":"Nielsen","year":"2022","journal-title":"Lancet Digit Health"},{"key":"2025021811400466100_ocae321-B48","article-title":"The effects of data quality on machine learning performance","volume-title":"arXiv preprint","author":"Budach","year":"2022"},{"key":"2025021811400466100_ocae321-B49","doi-asserted-by":"publisher","first-page":"764","DOI":"10.1093\/aje\/kwt312","article-title":"Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study","volume":"179","author":"Shah","year":"2014","journal-title":"Am J Epidemiol"},{"key":"2025021811400466100_ocae321-B50","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1109\/KSE.2018.8573344","author":"Duy Le","year":"2018"},{"key":"2025021811400466100_ocae321-B51","doi-asserted-by":"publisher","first-page":"20869","DOI":"10.1038\/s41598-021-00220-x","article-title":"The impact of recency and adequacy of historical information on sepsis predictions using machine learning","volume":"11","author":"Zargoush","year":"2021","journal-title":"Sci Rep"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/3\/492\/61459658\/ocae321.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/3\/492\/61459658\/ocae321.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,18]],"date-time":"2025-02-18T11:40:29Z","timestamp":1739878829000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/3\/492\/7958421"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,16]]},"references-count":51,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,1,16]]},"published-print":{"date-parts":[[2025,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae321","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,3]]},"published":{"date-parts":[[2025,1,16]]}}}