{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T01:55:44Z","timestamp":1791942944326,"version":"4.3.4"},"reference-count":40,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2021,9,15]],"date-time":"2021-09-15T00:00:00Z","timestamp":1631664000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000272","name":"National Institute of Health Research","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Imperial College Junior Research Fellowship and a Sir Henry Dale Fellowship funded by the Royal Society and Wellcome Trust","award":["213494\/Z\/18\/Z"],"award-info":[{"award-number":["213494\/Z\/18\/Z"]}]},{"DOI":"10.13039\/501100000867","name":"Commonwealth Scholarship Commission","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000867","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Simon Fraser University Postdoctoral Fellowship funded by Genome Canada"},{"name":"UK Medical Research Council and Department for International Development","award":["MR\/R015600\/1"],"award-info":[{"award-number":["MR\/R015600\/1"]}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:sec id=\"sec001\"><jats:title>Background<\/jats:title><jats:p>Machine learning (ML) algorithms are now increasingly used in infectious disease epidemiology. Epidemiologists should understand how ML algorithms behave within the context of outbreak data where missingness of data is almost ubiquitous.<\/jats:p><\/jats:sec><jats:sec id=\"sec002\"><jats:title>Methods<\/jats:title><jats:p>Using simulated data, we use a ML algorithmic framework to evaluate data imputation performance and the resulting case fatality ratio (CFR) estimates, focusing on the scale and type of data missingness (i.e., missing completely at random\u2014MCAR, missing at random\u2014MAR, or missing not at random\u2014MNAR).<\/jats:p><\/jats:sec><jats:sec id=\"sec003\"><jats:title>Results<\/jats:title><jats:p>Across ML methods, dataset sizes and proportions of training data used, the area under the receiver operating characteristic curve decreased by 7% (median, range: 1%\u201316%) when missingness was increased from 10% to 40%. Overall reduction in CFR bias for MAR across methods, proportion of missingness, outbreak size and proportion of training data was 0.5% (median, range: 0%\u201311%).<\/jats:p><\/jats:sec><jats:sec id=\"sec004\"><jats:title>Conclusion<\/jats:title><jats:p>ML methods could reduce bias and increase the precision in CFR estimates at low levels of missingness. However, no method is robust to high percentages of missingness. Thus, a datacentric approach is recommended in outbreak settings\u2014patient survival outcome data should be prioritised for collection and random-sample follow-ups should be implemented to ascertain missing outcomes.<\/jats:p><\/jats:sec>","DOI":"10.1371\/journal.pone.0257005","type":"journal-article","created":{"date-parts":[[2021,9,15]],"date-time":"2021-09-15T17:45:39Z","timestamp":1631727939000},"page":"e0257005","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":3,"title":["Comparison of machine learning methods for estimating case fatality ratios: An Ebola outbreak simulation study"],"prefix":"10.1371","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4485-8511","authenticated-orcid":true,"given":"Alpha","family":"Forna","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ilaria","family":"Dorigatti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pierre","family":"Nouvellet","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christl A.","family":"Donnelly","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2021,9,15]]},"reference":[{"issue":"12","key":"pone.0257005.ref001","first-page":"2222","article-title":"What is Machine Learning? A Primer for the Epidemiologist","volume":"188","author":"Q Bi","year":"2019","journal-title":"American Journal of Epidemiology"},{"issue":"suppl_2","key":"pone.0257005.ref002","doi-asserted-by":"crossref","first-page":"S99","DOI":"10.1093\/infdis\/jiy421","article-title":"Statistical Learning Methods to Determine Immune Correlates of Herpes Zoster in Vaccine Efficacy Trials","volume":"218","author":"PB Gilbert","year":"2018","journal-title":"The Journal of Infectious Diseases"},{"issue":"4","key":"pone.0257005.ref003","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1093\/infdis\/jiy192","article-title":"Human Gut Microbiota Predicts Susceptibility to Vibrio cholerae Infection","volume":"218","author":"FS Midani","year":"2018","journal-title":"The Journal of Infectious Diseases"},{"issue":"12","key":"pone.0257005.ref004","doi-asserted-by":"crossref","first-page":"1466","DOI":"10.1093\/jamia\/ocz106","article-title":"Assessing clinical heterogeneity in sepsis through treatment patterns and machine learning","volume":"26","author":"AE Fohner","year":"2019","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"12","key":"pone.0257005.ref005","doi-asserted-by":"crossref","first-page":"2476","DOI":"10.1093\/cid\/ciz678","article-title":"Case Fatality Ratio Estimates for the 2013\u20132016 West African Ebola Epidemic: Application of Boosted Regression Trees for Imputation","volume":"70","author":"A Forna","year":"2019","journal-title":"Clinical Infectious Diseases"},{"issue":"1721","key":"pone.0257005.ref006","doi-asserted-by":"crossref","first-page":"20160308","DOI":"10.1098\/rstb.2016.0308","article-title":"Heterogeneities in the case fatality ratio in the West African Ebola outbreak 2013\u20132016","volume":"372","author":"T Garske","year":"2017","journal-title":"Philosophical Transactions of the Royal Society B: Biological Sciences"},{"key":"pone.0257005.ref007","article-title":"Efficient and robust automated machine learning","author":"M Feurer","year":"2015","journal-title":"Advances in neural information processing systems"},{"issue":"4","key":"pone.0257005.ref008","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1111\/j.1365-2656.2008.01390.x","article-title":"A working guide to boosted regression trees","volume":"77","author":"J Elith","year":"2008","journal-title":"Journal of Animal Ecology"},{"issue":"1","key":"pone.0257005.ref009","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1093\/cid\/cix731","article-title":"Machine learning for healthcare: on the verge of a major shift in healthcare epidemiology","volume":"66","author":"J Wiens","year":"2018","journal-title":"Clinical Infectious Diseases"},{"issue":"11","key":"pone.0257005.ref010","doi-asserted-by":"crossref","first-page":"2954","DOI":"10.1029\/2018JF004640","article-title":"Using Machine Learning to Predict Geomorphic Disturbance: The Effects of Sample Size, Sample Prevalence, and Sampling Strategy","volume":"123","author":"GLW Perry","year":"2018","journal-title":"Journal of Geophysical Research: Earth Surface"},{"issue":"1","key":"pone.0257005.ref011","first-page":"826","article-title":"Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA","volume":"18","author":"L Kotthoff","year":"2017","journal-title":"The Journal of Machine Learning Research"},{"key":"pone.0257005.ref012","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/j.wpi.2018.07.002","article-title":"The state-of-the-art on Intellectual Property Analytics (IPA): A literature review on artificial intelligence, machine learning and deep learning methods for analysing intellectual property (IP) data","volume":"55","author":"L Aristodemou","year":"2018","journal-title":"World Patent Information"},{"issue":"371","key":"pone.0257005.ref013","doi-asserted-by":"crossref","first-page":"1481","DOI":"10.1056\/NEJMoa1411100","article-title":"Ebola virus disease in West Africa\u2014the first 9 months of the epidemic and forward projections","volume":"2014","author":"WER Team","year":"2014","journal-title":"N Engl J Med"},{"key":"pone.0257005.ref014","unstructured":"Ishwaran H, Kogalur UB, Kogalur MUB. Package \u2018randomForestSRC\u2019. 2019."},{"issue":"2013","key":"pone.0257005.ref015","first-page":"40","article-title":"Package \u2018gbm\u2019","volume":"10","author":"G Ridgeway","year":"2013","journal-title":"Viitattu"},{"key":"pone.0257005.ref016","unstructured":"Kapelner A, Bleich J. bartMachine: Bayesian Additive Regression Trees. R package version; 2016."},{"key":"pone.0257005.ref017","first-page":"3","article-title":"Package \u2018nnet\u2019","volume":"7","author":"B Ripley","year":"2016","journal-title":"R package version"},{"issue":"1","key":"pone.0257005.ref018","first-page":"98","article-title":"Missing data analysis using multiple imputation: getting to the heart of the matter","volume":"3","author":"Y He","year":"2010","journal-title":"Circulation: Cardiovascular Quality and Outcomes"},{"key":"pone.0257005.ref019","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4614-7138-7","volume-title":"An introduction to statistical learning","author":"G James","year":"2013"},{"key":"pone.0257005.ref020","first-page":"1189","article-title":"Greedy function approximation: a gradient boosting machine","author":"JH Friedman","year":"2001","journal-title":"Annals of statistics"},{"key":"pone.0257005.ref021","volume-title":"The elements of statistical learning","author":"J Friedman","year":"2001"},{"issue":"4","key":"pone.0257005.ref022","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1214\/aos\/1176345513","article-title":"Logistic Regression Diagnostics","volume":"9","author":"D Pregibon","year":"1981","journal-title":"Ann Statist"},{"issue":"13","key":"pone.0257005.ref023","doi-asserted-by":"crossref","first-page":"2786","DOI":"10.1017\/S095026881500014X","article-title":"Use of random forest to estimate population attributable fractions from a case-control study of Salmonella enterica serotype Enteritidis infections","volume":"143","author":"W Gu","year":"2015","journal-title":"Epidemiology & Infection"},{"issue":"1","key":"pone.0257005.ref024","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1214\/09-AOAS285","article-title":"BART: Bayesian additive regression trees","volume":"4","author":"HA Chipman","year":"2010","journal-title":"The Annals of Applied Statistics"},{"issue":"6","key":"pone.0257005.ref025","doi-asserted-by":"crossref","first-page":"588","DOI":"10.1061\/(ASCE)0733-9429(2002)128:6(588)","article-title":"Prediction of sediment load concentration in rivers using artificial neural network model","volume":"128","author":"H Nagy","year":"2002","journal-title":"Journal of Hydraulic Engineering"},{"issue":"1","key":"pone.0257005.ref026","first-page":"5938","article-title":"mlr: Machine Learning in R","volume":"17","author":"B Bischl","year":"2016","journal-title":"The Journal of Machine Learning Research"},{"key":"pone.0257005.ref027","unstructured":"Sergeant E. Package \u2018 RSurveillance \u2018. 2016."},{"key":"pone.0257005.ref028","unstructured":"Freeman ME. Package \u2018 PresenceAbsence \u2018. 2015."},{"issue":"8","key":"pone.0257005.ref029","doi-asserted-by":"crossref","first-page":"826","DOI":"10.1016\/j.jclinepi.2009.11.020","article-title":"Propensity score estimation: neural networks, support vector machines, decision trees (CART), and meta-classifiers as alternatives to logistic regression","volume":"63","author":"D Westreich","year":"2010","journal-title":"Journal of clinical epidemiology"},{"issue":"3","key":"pone.0257005.ref030","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1007\/s11060-016-2055-7","article-title":"Differentiation between treatment-related changes and progressive disease in patients with high grade brain tumors using support vector machine classification based on DCE MRI","volume":"127","author":"M Artzi","year":"2016","journal-title":"Journal of neuro-oncology"},{"issue":"4","key":"pone.0257005.ref031","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1136\/amiajnl-2011-000101","article-title":"The application of naive Bayes model averaging to predict Alzheimer\u2019s disease from genome-wide data","volume":"18","author":"W Wei","year":"2011","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"1","key":"pone.0257005.ref032","doi-asserted-by":"crossref","first-page":"3644","DOI":"10.1038\/s41467-018-06006-6","article-title":"Refined efficacy estimates of the Sanofi Pasteur dengue vaccine CYD-TDV using machine learning","volume":"9","author":"I Dorigatti","year":"2018","journal-title":"Nature Communications"},{"issue":"2","key":"pone.0257005.ref033","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1016\/j.gene.2016.01.023","article-title":"Artificial neural network-based exploration of gene-nutrient interactions in folate and xenobiotic metabolic pathways that modulate susceptibility to breast cancer","volume":"580","author":"SM Naushad","year":"2016","journal-title":"Gene"},{"issue":"2","key":"pone.0257005.ref034","first-page":"128","article-title":"Logistic regression with missing data: a comparisson of handling methods and effects of percent missing values","volume":"4","author":"S Meeyai","year":"2016","journal-title":"Journal of Traffic and Logistics Engineering"},{"key":"pone.0257005.ref035","doi-asserted-by":"crossref","unstructured":"Zhang Q, Rahman A, D\u2019este C, editors. Impute vs. ignore: Missing values for prediction. The 2013 International Joint Conference on Neural Networks (IJCNN); 2013: IEEE.","DOI":"10.1109\/IJCNN.2013.6707014"},{"issue":"1","key":"pone.0257005.ref036","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1016\/j.jsp.2009.10.001","article-title":"An introduction to modern missing data analyses","volume":"48","author":"AN Baraldi","year":"2010","journal-title":"Journal of school psychology"},{"issue":"1","key":"pone.0257005.ref037","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1093\/swr\/30.1.19","article-title":"Imputing missing data: A comparison of methods for social work researchers","volume":"30","author":"JA Saunders","year":"2006","journal-title":"Social work research"},{"issue":"4","key":"pone.0257005.ref038","doi-asserted-by":"crossref","first-page":"795","DOI":"10.1177\/1536867X1301300407","article-title":"Little\u2019s test of missing completely at random","volume":"13","author":"C Li","year":"2013","journal-title":"The Stata Journal"},{"key":"pone.0257005.ref039","doi-asserted-by":"crossref","unstructured":"Chen T, Guestrin C. XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; San Francisco, California, USA: Association for Computing Machinery; 2016. p. 785\u201394.","DOI":"10.1145\/2939672.2939785"},{"key":"pone.0257005.ref040","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1007\/978-3-030-05318-5_8","volume-title":"Automated Machine Learning: Methods, Systems, Challenges","author":"RS Olson","year":"2019"}],"updated-by":[{"DOI":"10.1371\/journal.pone.0316602","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2024,12,26]],"date-time":"2024-12-26T00:00:00Z","timestamp":1735171200000}}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0257005","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,9]],"date-time":"2023-01-09T08:54:57Z","timestamp":1673254497000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0257005"}},"subtitle":[],"editor":[{"given":"Khanh N.Q.","family":"Le","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2021,9,15]]},"references-count":40,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,9,15]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0257005","relation":{"correction":[{"id-type":"doi","id":"10.1371\/journal.pone.0316602","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,15]]}}}