{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,20]],"date-time":"2026-11-20T17:32:53Z","timestamp":1795195973059,"version":"4.10.3"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T00:00:00Z","timestamp":1747267200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T00:00:00Z","timestamp":1747267200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Res Methodol"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Supervised machine learning is increasingly being used to estimate clinical predictive models. Several supervised machine learning models involve hyper-parameters, whose values must be judiciously specified to ensure adequate predictive performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To compare several (nine) hyper-parameter optimization (HPO) methods, for tuning the hyper-parameters of an extreme gradient boosting model, with application to predicting high-need high-cost health care users.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Extreme gradient boosting models were estimated using a randomly sampled training dataset. Models were separately trained using nine different HPO methods: 1) random sampling, 2) simulated annealing, 3) quasi-Monte Carlo sampling, 4-5) two variations of Bayesian hyper-parameter optimization via tree-Parzen estimation, 6-7) two implementations of Bayesian hyper-parameter optimization via Gaussian processes, 8) Bayesian hyper-parameter optimization via random forests, and 9) the covariance matrix adaptation evolutionary strategy. For each HPO method, we estimated 100 extreme gradient boosting models at different hyper-parameter configurations; and evaluated model performance using an AUC metric on a randomly sampled validation dataset. Using the best model identified by each HPO method, we evaluated generalization performance in terms of discrimination and calibration metrics on a randomly sampled held-out test dataset (internal validation) and a temporally independent dataset (external validation).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The extreme gradient boosting model estimated using default hyper-parameter settings had reasonable discrimination (AUC=0.82) but was not well calibrated. Hyper-parameter tuning using any HPO algorithm\/sampler improved model discrimination (AUC=0.84), resulted in models with near perfect calibration, and consistently identified features predictive of high-need high-cost health care users.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>In our study, all HPO algorithms resulted in similar gains in model performance relative to baseline models. This finding likely relates to our study dataset having a large sample size, a relatively small number of features, and a strong signal to noise ratio; and would likely apply to other datasets with similar characteristics.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12874-025-02561-x","type":"journal-article","created":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T07:42:31Z","timestamp":1747294951000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Comparison of methods for tuning machine learning model hyper-parameters: with application to predicting high-need high-cost health care users"],"prefix":"10.1186","volume":"25","author":[{"given":"Christopher","family":"Meaney","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuesong","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jun","family":"Guan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Therese A.","family":"Stukel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,5,15]]},"reference":[{"key":"2561_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12874-021-01469-6","volume":"22","author":"CL Andaur Navarro","year":"2022","unstructured":"Andaur Navarro CL, Damen JAA, Takada T, Nijman SWJ, Dhiman P, Ma J, Collins GS, Bajpai R, Riley RD, Moons KGM, et al. Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review. BMC Med Res Methodol. 2022;22:1\u201313.","journal-title":"BMC Med Res Methodol."},{"key":"2561_CR2","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1002\/sim.5941","volume":"33","author":"P Austin","year":"2014","unstructured":"Austin P, Steyerberg E. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med. 2014;33:517\u201335.","journal-title":"Stat Med."},{"key":"2561_CR3","doi-asserted-by":"publisher","first-page":"4051","DOI":"10.1002\/sim.8281","volume":"38","author":"P Austin","year":"2019","unstructured":"Austin P, Steyerberg E. The integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Stat Med. 2019;38:4051\u201365.","journal-title":"Stat Med."},{"key":"2561_CR4","unstructured":"Bergstra J, Bardenet R, Bengio Y, K\u00e9gl B. Algorithms for hyper-parameter optimization. Adv Neural Inf Process Syst. 2011;24."},{"key":"2561_CR5","first-page":"281","volume":"13","author":"J Bergstra","year":"2012","unstructured":"Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res. 2012;13:281\u2013305.","journal-title":"J Mach Learn Res."},{"key":"2561_CR6","doi-asserted-by":"crossref","unstructured":"Bergstra J, Yamins D, Cox D. Hyperopt: a python library for optimizing hyper-parameters of machine learning algorithms. Proceedings of the Python in Science Conference. 2013:13-20.","DOI":"10.25080\/Majora-8b375195-003"},{"key":"2561_CR7","unstructured":"Bergstra J, Yamins D, Cox D. Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. International Conference on Machine Learning. 2013. P. 115\u2013123."},{"key":"2561_CR8","unstructured":"Bergstra J, Komer B, Eliasmith C, Warde-Farley D. Preliminary evaluation of Hyperopt algorithms on\u00a0HPOlib. In: ICML Workshop on AutoML. 2014. p. 1\u20137."},{"issue":"2","key":"2561_CR9","doi-asserted-by":"publisher","first-page":"e1484","DOI":"10.1002\/widm.1484","volume":"13","author":"B Bischl","year":"2023","unstructured":"Bischl B, Binder M, Lang M, Pielok T, Richter J, Coors S, Thomas J, Ullmann T, Becker M, Boulesteix A-L, et al. Hyperparameter optimization: foundations, algorithms, best practices, and open challenges. Data Mining Knowl Discov. 2023;13(2):e1484.","journal-title":"Data Mining Knowl Discov"},{"key":"2561_CR10","doi-asserted-by":"crossref","unstructured":"Chen T, Guestrin C. XGBoost: a scalable tree boosting system. Knowledge Discovery and Databases. 2016. p. 785-794.","DOI":"10.1145\/2939672.2939785"},{"issue":"2","key":"2561_CR11","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1161\/CIRCULATIONAHA.114.014508","volume":"131","author":"GS Collins","year":"2015","unstructured":"Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Circulation. 2015;131(2):211\u20139.","journal-title":"Circulation."},{"key":"2561_CR12","doi-asserted-by":"publisher","first-page":"e078378","DOI":"10.1136\/bmj-2023-078378","volume":"385","author":"GS Collins","year":"2024","unstructured":"Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, Ghassemi M, Liu X, Reitsma JB, Van Smeden M, et al. TRIPOD+ AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. Br Med J. 2024;385:e078378.","journal-title":"Br Med J."},{"issue":"6","key":"2561_CR13","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1002\/sim.9932","volume":"43","author":"ZS Dunias","year":"2024","unstructured":"Dunias ZS, Van Calster B, Timmerman D, Boulesteix A-L, van Smeden M. A comparison of hyperparameter tuning procedures for clinical prediction models: a simulation study. Stat Med. 2024;43(6):1119\u201334.","journal-title":"Stat Med."},{"issue":"5","key":"2561_CR14","first-page":"1189","volume":"19","author":"J Friedman","year":"2001","unstructured":"Friedman J. Greedy function approximation: a gradient boosting machine. Ann Stat. 2001;19(5):1189\u2013232.","journal-title":"Ann Stat."},{"issue":"2","key":"2561_CR15","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1162\/106365601750190398","volume":"9","author":"N Hansen","year":"2001","unstructured":"Hansen N, Ostermeier A. Completely derandomized self-adaptation in evolution strategies. Evol Comput. 2001;9(2):159\u201395.","journal-title":"Evol Comput."},{"key":"2561_CR16","unstructured":"Hansen N. The CMA evolution strategy: a tutorial. arXiv preprint arXiv:1604.00772. 2016."},{"key":"2561_CR17","doi-asserted-by":"crossref","unstructured":"Hutter F, Kotthoff L, Vanschoren J. Automated Machine Learning. Springer. 2019.","DOI":"10.1007\/978-3-030-05318-5"},{"key":"2561_CR18","unstructured":"Kegl B. A systematic study comparing hyperparameter optimization engines on tabular data. arXiv preprint arXiv:2311.15854. 2023."},{"issue":"4598","key":"2561_CR19","doi-asserted-by":"publisher","first-page":"671","DOI":"10.1126\/science.220.4598.671","volume":"220","author":"S Kirkpatrick","year":"1983","unstructured":"Kirkpatrick S, Gelatt CD Jr, Vecchi MP. Optimization by simulated annealing. Science. 1983;220(4598):671\u201380.","journal-title":"Science."},{"issue":"54","key":"2561_CR20","first-page":"1","volume":"23","author":"M Lindauer","year":"2022","unstructured":"Lindauer M, Eggensperger K, Feurer M, Biedenkapp A, Deng D, Benjamins C, Ruhkopf T, Sass R, Hutter F. SMAC3: a versatile Bayesian optimization package for hyperparameter optimization. J Mach Learn Res. 2022;23(54):1\u20139.","journal-title":"J Mach Learn Res."},{"key":"2561_CR21","doi-asserted-by":"crossref","unstructured":"Putatunda S, Rama K. A comparative analysis of Hyperopt as against other approaches for hyper-parameter optimization of XGBoost. In: International Conference on Signal Processing and Machine Learning. 2018. p. 6\u201310.","DOI":"10.1145\/3297067.3297080"},{"key":"2561_CR22","unstructured":"Shekhar S, Bansode A, Salim A. A comparative study of hyper-parameter optimization tools. Arxiv. 2022. Available from: https:\/\/arxiv.org\/abs\/2201.06433."},{"key":"2561_CR23","unstructured":"Snoek J, Larochelle H, Adams RP. Practical Bayesian optimization of machine learning algorithms. Adv Neural Inf Process Syst. 2012;25."},{"issue":"1","key":"2561_CR24","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1097\/EDE.0b013e3181c30fb2","volume":"23","author":"E Steyerberg","year":"2010","unstructured":"Steyerberg E, Vickers A, Cook N, Gerds T, Gonen M, Obuchowski N, Pencina M, Kattan M. Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiology. 2010;23(1):128\u201338.","journal-title":"Epidemiology."},{"issue":"1","key":"2561_CR25","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1016\/j.aci.2018.08.003","volume":"17","author":"A Tharwat","year":"2021","unstructured":"Tharwat A. Classification assessment methods. Appl Comput Inform. 2021;17(1):168\u201392.","journal-title":"Appl Comput Inform."},{"issue":"1","key":"2561_CR26","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","volume":"17","author":"B Van Calster","year":"2019","unstructured":"Van Calster B, McLernon DJ, Van Smeden M, Wynants L, Steyerberg EW, the STRATOS Initiative Topic Group \u2018Evaluating Diagnostic Tests and Prediction Models.\u2019 Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230.","journal-title":"BMC Med."},{"key":"2561_CR27","unstructured":"Yu T, Zhu H. Hyper-parameter optimization: a review of algorithms and applications. Arxiv. 2020. Available from: https:\/\/arxiv.org\/abs\/2003.05689."}],"container-title":["BMC Medical Research Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-025-02561-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12874-025-02561-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-025-02561-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T07:42:36Z","timestamp":1747294956000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedresmethodol.biomedcentral.com\/articles\/10.1186\/s12874-025-02561-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,15]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["2561"],"URL":"https:\/\/doi.org\/10.1186\/s12874-025-02561-x","relation":{},"ISSN":["1471-2288"],"issn-type":[{"value":"1471-2288","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,15]]},"assertion":[{"value":"20 April 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 April 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 May 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"ICES is an independent, non-profit research institute whose legal status under Ontario\u2019s health information privacy law allows it to collect and analyse healthcare and demographic data, without consent, for health system evaluation and improvement. The use of the data in this project is authorized under section 45 of Ontario\u2019s Personal Health Information Protection Act (PHIPA) and does not require review by a research ethics board.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"NA.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"134"}}