{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,30]],"date-time":"2025-08-30T00:06:06Z","timestamp":1756512366806,"version":"3.44.0"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2025,2]]},"abstract":"<jats:p>The development of hyperparameter optimization (HPO) algorithms is an important topic within both the machine learning and data management domains. While numerous strategies employing early stopping mechanisms have been proposed to bolster HPO efficiency, there remains a notable deficiency in understanding how the selection of early stopping metrics influences the reliability of early stopping decisions and, by extension, the broader HPO outcomes. This paper undertakes a systematic exploration of the impact of metric selection on the effectiveness of early stopping-based HPO. Specifically, we introduce a set of metrics that incorporate uncertainty and highlight their practical significance in enhancing the reliability of early stopping decisions. Our empirical experiments on HPO and NAS benchmarks show that using training loss as an early stopping metric in the early training stages improves HPO outcomes by up to 24.76% compared to the more widely accepted validation loss. Furthermore, integrating uncertainty into the metric yields an additional improvement of up to 4% under budget constraints, translating into meaningful resource savings and scalability benefits in large-scale HPO scenarios. These findings demonstrate the critical role of metric selection while shedding light on the potential implications of integrating uncertainty as a metric. This research provides empirical insights that serve as a compass for the selection and formulation of metrics, thereby contributing to a more profound comprehension of mechanisms underpinning early stopping-based HPO.<\/jats:p>","DOI":"10.14778\/3725688.3725689","type":"journal-article","created":{"date-parts":[[2025,8,29]],"date-time":"2025-08-29T14:19:21Z","timestamp":1756477161000},"page":"1551-1564","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["A Systematic Study on Early Stopping Metrics in HPO and the Implications of Uncertainty"],"prefix":"10.14778","volume":"18","author":[{"given":"Jiawei","family":"Guan","sequence":"first","affiliation":[{"name":"Renmin University of China"}]},{"given":"Feng","family":"Zhang","sequence":"additional","affiliation":[{"name":"Renmin University of China"}]},{"given":"Jiesong","family":"Liu","sequence":"additional","affiliation":[{"name":"North Carolina State University"}]},{"given":"Xiaoyong","family":"Du","sequence":"additional","affiliation":[{"name":"Renmin University of China"}]},{"given":"Xipeng","family":"Shen","sequence":"additional","affiliation":[{"name":"North Carolina State University"}]}],"member":"320","published-online":{"date-parts":[[2025,8,29]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"crossref","unstructured":"Moloud Abdar Farhad Pourpanah Sadiq Hussain Dana Rezazadegan Li Liu Mohammad Ghavamzadeh Paul Fieguth Xiaochun Cao Abbas Khosravi U Rajendra Acharya et al. 2021. A review of uncertainty quantification in deep learning: Techniques applications and challenges. Information fusion 76 (2021) 243\u2013297.","DOI":"10.1016\/j.inffus.2021.05.008"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330701"},{"key":"e_1_2_1_3_1","volume-title":"Dehb: Evolutionary hyperband for scalable, robust and efficient hyperparameter optimization. arXiv preprint arXiv:2105.09821","author":"Awad Noor","year":"2021","unstructured":"Noor Awad, Neeratyoy Mallik, and Frank Hutter. 2021. Dehb: Evolutionary hyperband for scalable, robust and efficient hyperparameter optimization. arXiv preprint arXiv:2105.09821 (2021)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/543613.543615"},{"key":"e_1_2_1_5_1","first-page":"38788","article-title":"Jahs-bench-201: A foundation for research on joint architecture and hyperparameter search","volume":"35","author":"Bansal Archit","year":"2022","unstructured":"Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, and Frank Hutter. 2022. Jahs-bench-201: A foundation for research on joint architecture and hyperparameter search. Advances in Neural Information Processing Systems 35 (2022), 38788\u201338802.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1484"},{"key":"e_1_2_1_7_1","unstructured":"Matthias Boehm Iulian Antonov Sebastian Baunsgaard Mark Dokter Robert Ginth\u00f6r Kevin Innerebner Florijan Klezin Stefanie Lindstaedt Arnab Phani Benjamin Rath et al. 2019. SystemDS: A declarative machine learning system for the end-to-end data science lifecycle. arXiv preprint arXiv:1909.02976 (2019)."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3517882"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 33rd international conference on Very large data bases. 3\u201314","author":"Chaudhuri Surajit","year":"2007","unstructured":"Surajit Chaudhuri and Vivek Narasayya. 2007. Self-tuning database systems: a decade of progress. In Proceedings of the 33rd international conference on Very large data bases. 3\u201314."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3399579.3399870"},{"key":"e_1_2_1_11_1","volume-title":"Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326","author":"Dong Xuanyi","year":"2020","unstructured":"Xuanyi Dong and Yi Yang. 2020. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 (2020)."},{"key":"e_1_2_1_12_1","unstructured":"D. Dua and C. Graff. [n.d.]. UCI machine learning repository. https:\/\/archive.ics.uci.edu\/. Accessed: 2024-09-12."},{"key":"e_1_2_1_13_1","volume-title":"HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. arXiv preprint arXiv:2109.06716","author":"Eggensperger Katharina","year":"2021","unstructured":"Katharina Eggensperger, Philipp M\u00fcller, Neeratyoy Mallik, Matthias Feurer, Ren\u00e9 Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. 2021. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. arXiv preprint arXiv:2109.06716 (2021)."},{"key":"e_1_2_1_14_1","volume-title":"Automated machine learning: State-of-the-art and open challenges. arXiv preprint arXiv:1906.02287","author":"Elshawi Radwa","year":"2019","unstructured":"Radwa Elshawi, Mohamed Maher, and Sherif Sakr. 2019. Automated machine learning: State-of-the-art and open challenges. arXiv preprint arXiv:1906.02287 (2019)."},{"key":"e_1_2_1_15_1","volume-title":"International conference on machine learning. PMLR, 1437\u20131446","author":"Falkner Stefan","year":"2018","unstructured":"Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. In International conference on machine learning. PMLR, 1437\u20131446."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the royal society a: mathematical, physical and engineering sciences 463","author":"Forrester Alexander IJ","year":"2007","unstructured":"Alexander IJ Forrester, Andr\u00e1s S\u00f3bester, and Andy J Keane. 2007. Multi-fidelity optimization via surrogate modelling. Proceedings of the royal society a: mathematical, physical and engineering sciences 463, 2088 (2007), 3251\u20133269."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3469440"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626246.3654751"},{"key":"e_1_2_1_19_1","unstructured":"Google. [n.d.]. Google Vertex AI. https:\/\/cloud.google.com\/vertex-ai"},{"key":"e_1_2_1_20_1","first-page":"26578","article-title":"Trec: Transient redundancy elimination-based convolution","volume":"35","author":"Guan Jiawei","year":"2022","unstructured":"Jiawei Guan, Feng Zhang, Jiesong Liu, Hsin-Hsuan Sung, Ruofan Wu, Xiaoyong Du, and Xipeng Shen. 2022. Trec: Transient redundancy elimination-based convolution. Advances in Neural Information Processing Systems 35 (2022), 26578\u201326589.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626765"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-024-40065-x"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-0865-5_26"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-021-05946-3"},{"key":"e_1_2_1_25_1","unstructured":"Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic best arm identification and hyperparameter optimization. In Artificial intelligence and statistics. PMLR 240\u2013248."},{"key":"e_1_2_1_26_1","volume-title":"International Conference on Machine Learning. PMLR, 1799\u20131808","author":"Kandasamy Kirthevasan","year":"2017","unstructured":"Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnab\u00e1s P\u00f3czos. 2017. Multi-fidelity bayesian optimisation with continuous approximations. In International Conference on Machine Learning. PMLR, 1799\u20131808."},{"key":"e_1_2_1_27_1","unstructured":"Aaron Klein Stefan Falkner Simon Bartels Philipp Hennig and Frank Hutter. 2017. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics. PMLR 528\u2013536."},{"key":"e_1_2_1_28_1","volume-title":"Companion of the 2024 International Conference on Management of Data","volume":"4","author":"Lao Jiale","year":"2024","unstructured":"Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Yuanchun Zhou, Mingjie Tang, and Jianguo Wang. 2024. A Demonstration of GPTuner: A GPT-Based Manual-Reading Database Tuning System. In Companion of the 2024 International Conference on Management of Data (Santiago, Chile)(SIGMOD'24). Association for Computing Machinery, New York, NY, USA, Vol. 4."},{"key":"e_1_2_1_29_1","volume-title":"Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165","author":"Lee Jaehoon","year":"2017","unstructured":"Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. 2017. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 (2017)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352129"},{"key":"e_1_2_1_31_1","volume-title":"Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research 18, 1","author":"Li Lisha","year":"2017","unstructured":"Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017. Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research 18, 1 (2017), 6765\u20136816."},{"key":"e_1_2_1_32_1","volume-title":"Hyper-tune: Towards efficient hyper-parameter tuning at scale. arXiv preprint arXiv:2201.06834","author":"Li Yang","year":"2022","unstructured":"Yang Li, Yu Shen, Huaijun Jiang, Wentao Zhang, Jixiang Li, Ji Liu, Ce Zhang, and Bin Cui. 2022. Hyper-tune: Towards efficient hyper-parameter tuning at scale. arXiv preprint arXiv:2201.06834 (2022)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-022-00752-2"},{"key":"e_1_2_1_34_1","volume-title":"UQ-Guided Hyperparameter Optimization for Iterative Learners. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https:\/\/openreview.net\/forum?id=k9uZfaeerK","author":"Liu Jiesong","year":"2024","unstructured":"Jiesong Liu, Feng Zhang, Jiawei Guan, and Xipeng Shen. 2024. UQ-Guided Hyperparameter Optimization for Iterative Learners. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https:\/\/openreview.net\/forum?id=k9uZfaeerK"},{"key":"e_1_2_1_35_1","volume-title":"International Conference on Automated Machine Learning. PMLR, 7\u20131.","author":"Makarova Anastasia","year":"2022","unstructured":"Anastasia Makarova, Huibin Shen, Valerio Perrone, Aaron Klein, Jean Baptiste Faddoul, Andreas Krause, Matthias Seeger, and Cedric Archambeau. 2022. Automatic termination for hyperparameter optimization. In International Conference on Automated Machine Learning. PMLR, 7\u20131."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2021.110361"},{"key":"e_1_2_1_37_1","first-page":"9361","article-title":"Bayesian optimization for iterative learning","volume":"33","author":"Nguyen Vu","year":"2020","unstructured":"Vu Nguyen, Sebastian Schulze, and Michael Osborne. 2020. Bayesian optimization for iterative learning. Advances in Neural Information Processing Systems 33 (2020), 9361\u20139371.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_38_1","volume-title":"Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems 32","author":"Ovadia Yaniv","year":"2019","unstructured":"Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems 32 (2019)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611540.3611566"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2022.111902"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157797"},{"key":"e_1_2_1_42_1","volume-title":"Holoclean: Holistic data repairs with probabilistic inference. arXiv preprint arXiv:1702.00820","author":"Rekatsinas Theodoros","year":"2017","unstructured":"Theodoros Rekatsinas, Xu Chu, Ihab F Ilyas, and Christopher R\u00e9. 2017. Holoclean: Holistic data repairs with probabilistic inference. arXiv preprint arXiv:1702.00820 (2017)."},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 2019 international conference on management of data. 1171\u20131188","author":"Shang Zeyuan","year":"2019","unstructured":"Zeyuan Shang, Emanuel Zgraggen, Benedetto Buratti, Ferdinand Kossmann, Philipp Eichmann, Yeounoh Chung, Carsten Binnig, Eli Upfal, and Tim Kraska. 2019. Democratizing data science through interactive curation of ml pipelines. In Proceedings of the 2019 international conference on management of data. 1171\u20131188."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sigpro.2019.107299"},{"key":"e_1_2_1_45_1","volume-title":"Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge","author":"Turner Ryan","year":"2020","unstructured":"Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon. 2021. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In NeurIPS 2020 Competition and Demonstration Track. PMLR, 3\u201326."},{"key":"e_1_2_1_46_1","first-page":"434","article-title":"Flaml: A fast and lightweight automl library","volume":"3","author":"Wang Chi","year":"2021","unstructured":"Chi Wang, Qingyun Wu, Markus Weimer, and Erkang Zhu. 2021. Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems 3 (2021), 434\u2013447.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_47_1","volume-title":"Teck Khim Ng, and Beng Chin Ooi","author":"Wang Wei","year":"2018","unstructured":"Wei Wang, Sheng Wang, Jinyang Gao, Meihui Zhang, Gang Chen, Teck Khim Ng, and Beng Chin Ooi. 2018. Rafiki: Machine learning as an analytics service system. arXiv preprint arXiv:1804.06087 (2018)."},{"key":"e_1_2_1_48_1","first-page":"13470","article-title":"Supervising the multi-fidelity race of hyperparameter configurations","volume":"35","author":"Wistuba Martin","year":"2022","unstructured":"Martin Wistuba, Arlind Kadra, and Josif Grabocka. 2022. Supervising the multi-fidelity race of hyperparameter configurations. Advances in Neural Information Processing Systems 35 (2022), 13470\u201313484.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_49_1","volume-title":"Wilcoxon signed-rank test","author":"Woolson Robert F","year":"2007","unstructured":"Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials (2007), 1\u20133."},{"key":"e_1_2_1_50_1","first-page":"26","article-title":"Hyperparameter optimization for machine learning models based on Bayesian optimization","volume":"17","author":"Wu Jia","year":"2019","unstructured":"Jia Wu, Xiu-Yun Chen, Hao Zhang, Li-Dong Xiong, Hang Lei, and Si-Hao Deng. 2019. Hyperparameter optimization for machine learning models based on Bayesian optimization. Journal of Electronic Science and Technology 17, 1 (2019), 26\u201340.","journal-title":"Journal of Electronic Science and Technology"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-024-40555-y"},{"key":"e_1_2_1_52_1","first-page":"22534","article-title":"Nas-bench-x11 and the power of learning curves","volume":"34","author":"Yan Shen","year":"2021","unstructured":"Shen Yan, Colin White, Yash Savani, and Frank Hutter. 2021. Nas-bench-x11 and the power of learning curves. Advances in Neural Information Processing Systems 34 (2021), 22534\u201322549.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.07.061"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-024-3937-z"},{"key":"e_1_2_1_55_1","volume-title":"POCLib: A high-performance framework for enabling near orthogonal processing on compression","author":"Zhang Feng","year":"2021","unstructured":"Feng Zhang, Jidong Zhai, Xipeng Shen, Onur Mutlu, and Xiaoyong Du. 2021. POCLib: A high-performance framework for enabling near orthogonal processing on compression. IEEE transactions on Parallel and Distributed Systems 33, 2 (2021), 459\u2013475."},{"key":"e_1_2_1_56_1","volume-title":"Facilitating database tuning with hyper-parameter optimization: a comprehensive experimental evaluation. arXiv preprint arXiv:2110.12654","author":"Zhang Xinyi","year":"2021","unstructured":"Xinyi Zhang, Zhuo Chang, Yang Li, Hong Wu, Jian Tan, Feifei Li, and Bin Cui. 2021. Facilitating database tuning with hyper-parameter optimization: a comprehensive experimental evaluation. arXiv preprint arXiv:2110.12654 (2021)."},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 2021 international conference on management of data. 2102\u20132114","author":"Zhang Xinyi","year":"2021","unstructured":"Xinyi Zhang, Hong Wu, Zhuo Chang, Shuowei Jin, Jian Tan, Feifei Li, Tieying Zhang, and Bin Cui. 2021. Restune: Resource oriented tuning boosted by metalearning for cloud databases. In Proceedings of the 2021 international conference on management of data. 2102\u20132114."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3526176"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2021.10.119"},{"key":"e_1_2_1_60_1","doi-asserted-by":"crossref","first-page":"3079","DOI":"10.1109\/TPAMI.2021.3067763","article-title":"Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL","volume":"43","author":"Zimmer Lucas","year":"2021","unstructured":"Lucas Zimmer, Marius Lindauer, and Frank Hutter. 2021. Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 9 (2021), 3079\u20133090.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3725688.3725689","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,29]],"date-time":"2025-08-29T14:21:08Z","timestamp":1756477268000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3725688.3725689"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2]]},"references-count":60,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,2]]}},"alternative-id":["10.14778\/3725688.3725689"],"URL":"https:\/\/doi.org\/10.14778\/3725688.3725689","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2025,2]]},"assertion":[{"value":"2025-08-29","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}