{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T17:57:54Z","timestamp":1779904674115,"version":"3.53.1"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T00:00:00Z","timestamp":1658793600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T00:00:00Z","timestamp":1658793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["LM010098"],"award-info":[{"award-number":["LM010098"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["LM010098"],"award-info":[{"award-number":["LM010098"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BioData Mining"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objectives<\/jats:title><jats:p>Ascertain and compare the performances of Automated Machine Learning (AutoML) tools on large, highly imbalanced healthcare datasets.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>We generated a large dataset using historical de-identified administrative claims including demographic information and flags for disease codes in four different time windows prior to 2019. We then trained three AutoML tools on this dataset to predict six different disease outcomes in 2019 and evaluated model performances on several metrics.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The AutoML tools showed improvement from the baseline random forest model but did not differ significantly from each other. All models recorded low area under the precision-recall curve and failed to predict true positives while keeping the true negative rate high. Model performance was not directly related to prevalence. We provide a specific use-case to illustrate how to select a threshold that gives the best balance between true and false positive rates, as this is an important consideration in medical applications.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>Healthcare datasets present several challenges for AutoML tools, including large sample size, high imbalance, and limitations in the available features. Improvements in scalability, combinations of imbalance-learning resampling and ensemble approaches, and curated feature selection are possible next steps to achieve better performance.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Among the three explored, no AutoML tool consistently outperforms the rest in terms of predictive performance. The performances of the models in this study suggest that there may be room for improvement in handling medical claims data. Finally, selection of the optimal prediction threshold should be guided by the specific practical application.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s13040-022-00300-2","type":"journal-article","created":{"date-parts":[[2022,7,27]],"date-time":"2022-07-27T13:48:54Z","timestamp":1658929734000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":37,"title":["Benchmarking AutoML frameworks for disease prediction using medical claims"],"prefix":"10.1186","volume":"15","author":[{"given":"Roland Albert","family":"A. Romero","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mariefel Nicole","family":"Y. Deypalan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suchit","family":"Mehrotra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John Titus","family":"Jungao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natalie E.","family":"Sheils","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elisabetta","family":"Manduchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5015-1099","authenticated-orcid":false,"given":"Jason H.","family":"Moore","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,7,26]]},"reference":[{"key":"300_CR1","doi-asserted-by":"publisher","unstructured":"Mustafa A, Rahimi Azghadi M. Automated machine learning for healthcare and clinical notes analysis. Computers. 2021; 10(2). https:\/\/doi.org\/10.3390\/computers10020024.","DOI":"10.3390\/computers10020024"},{"key":"300_CR2","doi-asserted-by":"publisher","unstructured":"Chen M, Hao Y, Hwang K, Wang L, Wang L. Disease prediction by machine learning over big data from healthcare communities: IEEE Access; 2017, pp. 1\u20131. https:\/\/doi.org\/10.1109\/ACCESS.2017.2694446.","DOI":"10.1109\/ACCESS.2017.2694446"},{"issue":"8","key":"300_CR3","doi-asserted-by":"publisher","first-page":"175","DOI":"10.2196\/resprot.7757","volume":"6","author":"G Luo","year":"2017","unstructured":"Luo G, Stone BL, Johnson MD, Tarczy-Hornoch P, Wilcox AB, Mooney SD, Sheng X, Haug PJ, Nkoy FL. Automating construction of machine learning models with clinical big data: Proposal rationale and methods. JMIR Res Protoc. 2017; 6(8):175. https:\/\/doi.org\/10.2196\/resprot.7757.","journal-title":"JMIR Res Protoc"},{"issue":"1","key":"300_CR4","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1038\/s41746-020-00354-8","volume":"3","author":"I Osawa","year":"2020","unstructured":"Osawa I, Goto T, Yamamoto Y, Tsugawa Y. Machine-learning-based prediction models for high-need high-cost patients using nationwide clinical and claims data. NPJ Dig Med. 2020; 3(1):148. https:\/\/doi.org\/10.1038\/s41746-020-00354-8.","journal-title":"NPJ Dig Med"},{"key":"300_CR5","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1109\/MITP.2013.55","volume":"15","author":"U Srinivasan","year":"2013","unstructured":"Srinivasan U, Arunasalam B. Leveraging big data analytics to reduce healthcare costs. IT Prof. 2013; 15:21\u201328. https:\/\/doi.org\/10.1109\/MITP.2013.55.","journal-title":"IT Prof"},{"key":"300_CR6","volume-title":"2018 IEEE International Conference on Healthcare Informatics (ICHI)","author":"T Christensen","year":"2018","unstructured":"Christensen T, Frandsen A, Glazier S, Humpherys J, Kartchner D. Machine learning methods for disease prediction with claims data. In: 2018 IEEE International Conference on Healthcare Informatics (ICHI). New York: IEEE Press: 2018. p. 467\u20134674."},{"key":"300_CR7","doi-asserted-by":"publisher","first-page":"101822","DOI":"10.1016\/j.artmed.2020.101822","volume":"104","author":"J Waring","year":"2020","unstructured":"Waring J, Lindvall C, Umeton R. Automated machine learning: Review of the state-of-the-art and opportunities for healthcare. Artif Intell Med. 2020; 104:101822. https:\/\/doi.org\/10.1016\/j.artmed.2020.101822.","journal-title":"Artif Intell Med"},{"key":"300_CR8","doi-asserted-by":"publisher","unstructured":"Popescu M, Khalilia M. Improving disease prediction using ICD-9 ontological features. In: 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2011). IEEE: 2011. https:\/\/doi.org\/10.1109\/fuzzy.2011.6007410. https:\/\/doi.org\/10.1109%2Ffuzzy.2011.6007410.","DOI":"10.1109\/fuzzy.2011.6007410"},{"key":"300_CR9","doi-asserted-by":"crossref","unstructured":"Shimabukuro DW, Barton CW, Feldman MD, Mataraso SJ, Das R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res. 2017; 4(1). https:\/\/doi.org\/10.1136\/bmjresp-2017-000234. http:\/\/arxiv.org\/abs\/https:\/\/bmjopenrespres.bmj.com\/content\/4\/1\/e000234.full.pdf.","DOI":"10.1136\/bmjresp-2017-000234"},{"issue":"3","key":"300_CR10","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1111\/acem.12876","volume":"23","author":"RA Taylor","year":"2016","unstructured":"Taylor RA, Pare JR, Venkatesh AK, Mowafi H, Melnick ER, Fleischman W, Hall MK. Prediction of in-hospital mortality in emergency department patients with sepsis: A local big data\u2013driven, machine learning approach. Acad Emerg Med. 2016; 23(3):269\u201378. https:\/\/doi.org\/10.1111\/acem.12876. http:\/\/arxiv.org\/abs\/https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/acem.12876.","journal-title":"Acad Emerg Med"},{"key":"300_CR11","volume-title":"Pacific Symposium on Biocomputing 2017","author":"K Shameer","year":"2017","unstructured":"Shameer K, Johnson KW, Yahi A, Miotto R, Li L, Ricks D, Jebakaran J, Kovatch P, Sengupta PP, Gelijns S, et al. Predictive modeling of hospital readmission rates using electronic medical record-wide machine learning: a case-study using mount sinai heart failure cohort. In: Pacific Symposium on Biocomputing 2017. Hackensack: World Scientific: 2017. p. 276\u201387."},{"key":"300_CR12","doi-asserted-by":"publisher","first-page":"8869","DOI":"10.1109\/ACCESS.2017.2694446","volume":"5","author":"M Chen","year":"2017","unstructured":"Chen M, Hao Y, Hwang K, Wang L, Wang L. Disease prediction by machine learning over big data from healthcare communities. IEEE Access. 2017; 5:8869\u201379. https:\/\/doi.org\/10.1109\/ACCESS.2017.2694446.","journal-title":"IEEE Access"},{"issue":"1","key":"300_CR13","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1109\/4235.585893","volume":"1","author":"DH Wolpert","year":"1997","unstructured":"Wolpert DH, Macready WG. No free lunch theorems for optimization. IEEE Trans Evol Comput. 1997; 1(1):67\u201382.","journal-title":"IEEE Trans Evol Comput"},{"key":"300_CR14","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1613\/jair.1.11854","volume":"70","author":"M-A Z\u00f6ller","year":"2021","unstructured":"Z\u00f6ller M-A, Huber MF. Benchmark and survey of automated machine learning frameworks. J Artif Intell Res. 2021; 70:409\u201372.","journal-title":"J Artif Intell Res"},{"key":"300_CR15","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05318-5","volume-title":"Automated Machine Learning: Methods, Systems, Challenges","author":"F Hutter","year":"2019","unstructured":"Hutter F, Kotthoff L, Vanschoren J. Automated Machine Learning: Methods, Systems, Challenges. New York: Springer; 2019."},{"key":"300_CR16","unstructured":"Yao Q, Wang M, Chen Y, Dai W, Li Y-F, Tu W-W, Yang Q, Yu Y. Taking human out of learning applications: A survey on automated machine learning. arXiv preprint arXiv:1810.13306. 2018."},{"issue":"2","key":"300_CR17","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1109\/MM.2020.2974843","volume":"40","author":"P Mattson","year":"2020","unstructured":"Mattson P, Reddi VJ, Cheng C, Coleman C, Diamos G, Kanter D, Micikevicius P, Patterson D, Schmuelling G, Tang H, et al. Mlperf: An industry standard benchmark suite for machine learning performance. IEEE Micro. 2020; 40(2):8\u201316.","journal-title":"IEEE Micro"},{"issue":"1","key":"300_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13040-017-0154-4","volume":"10","author":"RS Olson","year":"2017","unstructured":"Olson RS, La Cava W, Orzechowski P, Urbanowicz RJ, Moore JH. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Min. 2017; 10(1):1\u201313.","journal-title":"BioData Min"},{"key":"300_CR19","unstructured":"Milutinovic M, Schoenfeld B, Martinez-Garcia D, Ray S, Shah S, Yan D. On evaluation of automl systems. In: Proceedings of the ICML Workshop on Automatic Machine Learning, vol. 2020. Vienna: 2020."},{"key":"300_CR20","doi-asserted-by":"crossref","unstructured":"Hanussek M, Blohm M, Kintz M. Can AutoML outperform humans? An evaluation on popular OpenML datasets using AutoML Benchmark. 2020. http:\/\/arxiv.org\/abs\/2009.01564. Accessed 15 Dec 2020.","DOI":"10.1145\/3448326.3448353"},{"key":"300_CR21","unstructured":"Balaji A, Allen A. Benchmarking Automatic Machine Learning Frameworks. 2018. http:\/\/arxiv.org\/abs\/1808.06492. Accessed 15 Dec 2020."},{"key":"300_CR22","unstructured":"Gijsbers P, LeDell E, Thomas J, Poirier S, Bischl B, Vanschoren J. An open source automl benchmark. arXiv preprint arXiv:1907.00909. 2019."},{"issue":"1","key":"300_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s13721-016-0125-6","volume":"5","author":"G Luo","year":"2016","unstructured":"Luo G. A review of automatic selection methods for machine learning algorithms and hyper-parameter values. Netw Model Anal Health Inform Bioinforma. 2016; 5(1):1\u201316.","journal-title":"Netw Model Anal Health Inform Bioinforma"},{"issue":"1","key":"300_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1472-6947-11-51","volume":"11","author":"M Khalilia","year":"2011","unstructured":"Khalilia M, Chakraborty S, Popescu M. Predicting disease risks from highly imbalanced data using random forest. BMC Med Inform Decis Making. 2011; 11(1):1\u201313.","journal-title":"BMC Med Inform Decis Making"},{"key":"300_CR25","volume-title":"Automated Machine Learning","author":"M Feurer","year":"2019","unstructured":"Feurer M, Klein A, Eggensperger K, Springenberg JT, Blum M, Hutter F. Auto-sklearn: efficient and robust automated machine learning. In: Automated Machine Learning. Vienna: Springer: 2019. p. 113\u201334."},{"key":"300_CR26","unstructured":"LeDell E, Poirier S. H2o automl: Scalable automatic machine learning. In: Proceedings of the AutoML Workshop at ICML, vol. 2020. Vienna: 2020."},{"key":"300_CR27","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05318-5_8","volume-title":"TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning","year":"2019","unstructured":"Olson RS, Moore JH. In: Hutter F, Kotthoff L, Vanschoren J, (eds).TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning. Cham: Springer; 2019, pp. 151\u201360. https:\/\/doi.org\/10.1007\/978-3-030-05318-5_8. https:\/\/doi.org\/10.1007\/978-3-030-05318-5_8."},{"issue":"1","key":"300_CR28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-017-03011-5","volume":"7","author":"M Schubach","year":"2017","unstructured":"Schubach M, Re M, Robinson PN, Valentini G. Imbalance-aware machine learning for predicting rare and common disease-associated non-coding variants. Sci Rep. 2017; 7(1):1\u201312.","journal-title":"Sci Rep"}],"container-title":["BioData Mining"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13040-022-00300-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13040-022-00300-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13040-022-00300-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,12]],"date-time":"2023-02-12T18:49:11Z","timestamp":1676227751000},"score":1,"resource":{"primary":{"URL":"https:\/\/biodatamining.biomedcentral.com\/articles\/10.1186\/s13040-022-00300-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,26]]},"references-count":28,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["300"],"URL":"https:\/\/doi.org\/10.1186\/s13040-022-00300-2","relation":{},"ISSN":["1756-0381"],"issn-type":[{"value":"1756-0381","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,26]]},"assertion":[{"value":"7 December 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 June 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"Roland Albert Romero, Mariefel Nicole Deypalan, Suchit Mehrotra, John Titus Jungao, and Natalie Sheils are employees of OptumLabs part of UnitedHealth Group. Natalie Sheils owns stock in the company. The other authors have no conflicts of interest to disclose.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"15"}}