{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T04:50:00Z","timestamp":1768971000712,"version":"3.49.0"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,2,23]],"date-time":"2023-02-23T00:00:00Z","timestamp":1677110400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,2,23]],"date-time":"2023-02-23T00:00:00Z","timestamp":1677110400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In the field of emergency medicine (EM), the use of decision support tools based on artificial intelligence has increased markedly in recent years. In some cases, data are omitted deliberately and thus constitute \u201cdata not purposely collected\u201d (DNPC). This accepted information bias can be managed in various ways: dropping patients with missing data, imputing with the mean, or using automatic techniques (e.g., machine learning) to handle or impute the data. Here, we systematically reviewed the methods used to handle missing data in EM research. A systematic review was performed after searching PubMed with the query \u201c(emergency medicine OR emergency service) AND (artificial intelligence OR machine learning)\u201d. Seventy-two studies were included in the review. The trained models variously predicted diagnosis in 25 (35%) publications, mortality in 21 (29%) publications, and probability of admission in 21 (29%) publications. Eight publications (11%) predicted two outcomes. Only 15 (21%) publications described their missing data. DNPC constitute the \u201cmissing data\u201d in EM machine learning studies. Although DNPC have been described more rigorously since 2020, the descriptions in the literature are not exhaustive, systematic or homogeneous. Imputation appears to be the best strategy but requires more time and computational resources. To increase the quality and the comparability of studies, we recommend inclusion of the TRIPOD checklist in each new publication, summarizing the machine learning process in an explicit methodological diagram, and always publishing the area under the receiver operating characteristics curve\u2014even when it is not the primary outcome.<\/jats:p>","DOI":"10.1038\/s41746-023-00770-6","type":"journal-article","created":{"date-parts":[[2023,2,23]],"date-time":"2023-02-23T06:56:10Z","timestamp":1677135370000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Predictive models in emergency medicine and their missing data strategies: a systematic review"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6971-009X","authenticated-orcid":false,"given":"Emilien","family":"Arnaud","sequence":"first","affiliation":[]},{"given":"Mahmoud","family":"Elbattah","sequence":"additional","affiliation":[]},{"given":"Christine","family":"Ammirati","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7497-1182","authenticated-orcid":false,"given":"Gilles","family":"Dequen","sequence":"additional","affiliation":[]},{"given":"Daniel Aiham","family":"Ghazali","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,2,23]]},"reference":[{"key":"770_CR1","doi-asserted-by":"publisher","first-page":"101762","DOI":"10.1016\/j.artmed.2019.101762","volume":"102","author":"M Fernandes","year":"2020","unstructured":"Fernandes, M. et al. Clinical decision support systems for triage in the emergency department using intelligent systems: a review. Artif. Intell. Med. 102, 101762 (2020).","journal-title":"Artif. Intell. Med."},{"key":"770_CR2","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1093\/biomet\/63.3.581","volume":"63","author":"DB Rubin","year":"1976","unstructured":"Rubin, D. B. Inference and missing data. Biometrika 63, 581\u2013592 (1976).","journal-title":"Biometrika"},{"key":"770_CR3","doi-asserted-by":"crossref","unstructured":"van Buuren, S. Flexible Imputation of Missing Data 2nd edn (CRC Press, 2018).","DOI":"10.1201\/9780429492259"},{"key":"770_CR4","doi-asserted-by":"crossref","unstructured":"Little, R. J. A. & Rubin, D. B. Statistical Analysis with Missing Data (John Wiley & Sons, 2019).","DOI":"10.1002\/9781119482260"},{"key":"770_CR5","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-020-01080-1","volume":"20","author":"S Hong","year":"2020","unstructured":"Hong, S. & Lynn, H. S. Accuracy of random-forest-based imputation of missing data in the presence of non-normality, non-linearity, and interaction. BMC Med. Res. Methodol. 20, 199 (2020).","journal-title":"BMC Med. Res. Methodol."},{"key":"770_CR6","doi-asserted-by":"publisher","first-page":"106907","DOI":"10.1016\/j.csda.2019.106907","volume":"145","author":"W Jiang","year":"2020","unstructured":"Jiang, W., Josse, J. & Lavielle, M. Logistic regression with missing covariates\u2014parameter estimation, model selection and prediction within a joint-modeling framework. Comput. Stat. Data Anal. 145, 106907 (2020).","journal-title":"Comput. Stat. Data Anal."},{"key":"770_CR7","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-021-01660-1","volume":"21","author":"A Naemi","year":"2021","unstructured":"Naemi, A., Schmidt, T., Mansourvar, M., Ebrahimi, A. & Wiil, U. K. Quantifying the impact of addressing data challenges in prediction of length of stay. BMC Med. Inform. Decis. Mak. 21, 298 (2021).","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"770_CR8","doi-asserted-by":"publisher","first-page":"314","DOI":"10.1080\/01621459.2015.1136826","volume":"112","author":"RJ Little","year":"2017","unstructured":"Little, R. J., Rubin, D. B. & Zangeneh, S. Z. Conditions for ignoring the missing-data mechanism in likelihood inferences for parameter subsets. J. Am. Stat. Assoc. 112, 314\u2013320 (2017).","journal-title":"J. Am. Stat. Assoc."},{"key":"770_CR9","doi-asserted-by":"crossref","unstructured":"Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. The TRIPOD Group. Circulation. 131, 211\u2013219 (2015).","DOI":"10.1161\/CIRCULATIONAHA.114.014508"},{"key":"770_CR10","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.jclinepi.2013.08.013","volume":"67","author":"T Li","year":"2014","unstructured":"Li, T. et al. Standards should be applied in the prevention and handling of missing data for patient-centered outcomes research: a systematic review and expert consensus. J. Clin. Epidemiol. 67, 15\u201332 (2014).","journal-title":"J. Clin. Epidemiol."},{"key":"770_CR11","doi-asserted-by":"publisher","first-page":"2048","DOI":"10.1093\/ajhp\/zxz245","volume":"76","author":"SW Narayan","year":"2019","unstructured":"Narayan, S. W. et al. Missing data reporting in clinical pharmacy research. Am. J. Health Syst. Pharm. 76, 2048\u20132052 (2019).","journal-title":"Am. J. Health Syst. Pharm."},{"key":"770_CR12","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1186\/s12955-019-1181-2","volume":"17","author":"OF Ayilara","year":"2019","unstructured":"Ayilara, O. F. et al. Impact of missing data on bias and precision when estimating change in patient-reported outcomes from a clinical registry. Health Qual. Life Outcomes 17, 106 (2019).","journal-title":"Health Qual. Life Outcomes"},{"key":"770_CR13","unstructured":"Josse, J., Prost, N., Scornet, E. & Varoquaux, G. On the consistency of supervised learning with missing values. Preprint at http:\/\/arxiv.org\/abs\/1902.06931 (2020)."},{"key":"770_CR14","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1002\/sim.4067","volume":"30","author":"IR White","year":"2011","unstructured":"White, I. R., Royston, P. & Wood, A. M. Multiple imputation using chained equations: Issues and guidance for practice. Stat. Med. 30, 377\u2013399 (2011).","journal-title":"Stat. Med."},{"key":"770_CR15","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1016\/S0895-4356(01)00433-4","volume":"55","author":"PD Faris","year":"2002","unstructured":"Faris, P. D. et al. Multiple imputation versus data enhancement for dealing with missing data in observational health care outcome analyses. J. Clin. Epidemiol. 55, 184\u2013191 (2002).","journal-title":"J. Clin. Epidemiol."},{"key":"770_CR16","doi-asserted-by":"publisher","first-page":"1020","DOI":"10.1016\/j.csda.2004.07.002","volume":"49","author":"E Kuhn","year":"2005","unstructured":"Kuhn, E. & Lavielle, M. Maximum likelihood estimation in nonlinear mixed effects models. Comput. Stat. Data Anal. 49, 1020\u20131038 (2005).","journal-title":"Comput. Stat. Data Anal."},{"key":"770_CR17","doi-asserted-by":"publisher","DOI":"10.1093\/gigascience\/giac013","volume":"11","author":"A Perez-Lebel","year":"2022","unstructured":"Perez-Lebel, A., Varoquaux, G., Le Morvan, M., Josse, J. & Poline, J.-B. Benchmarking missing-values approaches for predictive models on health databases. GigaScience 11, giac013 (2022).","journal-title":"GigaScience"},{"key":"770_CR18","doi-asserted-by":"publisher","first-page":"1322","DOI":"10.1016\/j.cjca.2020.11.010","volume":"37","author":"PC Austin","year":"2021","unstructured":"Austin, P. C., White, I. R., Lee, D. S. & van Buuren, S. Missing data in clinical research: a tutorial on multiple imputation. Can. J. Cardiol. 37, 1322\u20131331 (2021).","journal-title":"Can. J. Cardiol."},{"key":"770_CR19","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1016\/j.annemergmed.2020.05.026","volume":"76","author":"DA Martinez","year":"2020","unstructured":"Martinez, D. A. et al. Early prediction of acute kidney injury in the emergency department with machine-learning methods applied to electronic health record data. Ann. Emerg. Med. 76, 501\u2013514 (2020).","journal-title":"Ann. Emerg. Med."},{"key":"770_CR20","first-page":"11","volume":"13","author":"M Gong","year":"2021","unstructured":"Gong, M. A novel performance measure for machine learning classification. Int. J. Manag. Inf. Technol. 13, 11\u201319 (2021).","journal-title":"Int. J. Manag. Inf. Technol."},{"key":"770_CR21","doi-asserted-by":"publisher","first-page":"e1001744","DOI":"10.1371\/journal.pmed.1001744","volume":"11","author":"KGM Moons","year":"2014","unstructured":"Moons, K. G. M. et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: The CHARMS Checklist. PLoS Med. 11, e1001744 (2014).","journal-title":"PLoS Med."},{"key":"770_CR22","doi-asserted-by":"publisher","unstructured":"Page, M. J. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, n71 https:\/\/doi.org\/10.1136\/bmj.n71 (2021).","DOI":"10.1136\/bmj.n71"},{"key":"770_CR23","doi-asserted-by":"publisher","first-page":"51","DOI":"10.7326\/M18-1376","volume":"170","author":"RF Wolff","year":"2019","unstructured":"Wolff, R. F. et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann. Intern. Med. 170, 51 (2019).","journal-title":"Ann. Intern. Med."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00770-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00770-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00770-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,23]],"date-time":"2023-02-23T06:58:31Z","timestamp":1677135511000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00770-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,23]]},"references-count":23,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["770"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00770-6","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,23]]},"assertion":[{"value":"6 May 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 February 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 February 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"28"}}