{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T22:55:35Z","timestamp":1777676135639,"version":"3.51.4"},"reference-count":41,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T00:00:00Z","timestamp":1753228800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/100006132","name":"Office of Science","doi-asserted-by":"publisher","award":["DE-AC02-05CH11231"],"award-info":[{"award-number":["DE-AC02-05CH11231"]}],"id":[{"id":"10.13039\/100006132","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006132","name":"Office of Science","doi-asserted-by":"publisher","award":["17-SC-20-SC"],"award-info":[{"award-number":["17-SC-20-SC"]}],"id":[{"id":"10.13039\/100006132","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:p>In the exascale computing era, tuning High-Performance Computing (HPC) applications has become a significant computational challenge. Although Bayesian optimization (BO) has emerged as a promising tool for HPC performance tuning, the BO workflow is inherently sequential (i.e., one function evaluation at a time) and cannot leverage the huge amount of parallel resources present in modern supercomputers, resulting in a considerable underutilization of their computational capabilities. This paper explores the trade-off between search quality and parallelism in BO, investigating a diverse set of methods. Building upon both previous approaches from the literature and novel methodologies introduced in this work, our study provides a deep analysis to accelerate BO performance tuning. By examining a set of synthetic functions and practical HPC applications, our exploration analyzes the interaction among various BO methods for parallelization, the quantity of parallel resources, the runtime distribution of target HPC applications, and the costs associated with different search orchestration mechanisms that have been overlooked in previous studies. Compared to sequential BO, our novel methodology achieves comparable quality while demonstrating robust scalability in search time as the amount of parallel resources increases; it also outperforms a state-of-the-art tuner, which supports parallelization, achieving up to 3.67x faster search time. We provide high-value insights for practitioners seeking to leverage the power of parallel computing for efficient HPC application tuning. Additionally, to further assist researchers in accelerating the performance tuning of their HPC applications, we provide an extension of an existing open-source tuning framework that incorporates our methods.<\/jats:p>","DOI":"10.1177\/10943420251362001","type":"journal-article","created":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T08:53:12Z","timestamp":1753260792000},"page":"631-654","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Parallelizing autotuning for HPC applications: Unveiling the potential of the speculation strategy in Bayesian optimization"],"prefix":"10.1177","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7168-9050","authenticated-orcid":false,"given":"Adrian P","family":"Dieguez","sequence":"first","affiliation":[{"name":"Lawrence Berkeley National Laboratory, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5692-9359","authenticated-orcid":false,"given":"Seth","family":"Ockerman","sequence":"additional","affiliation":[{"name":"University of Wisconsin\u2013Madison, WI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tristan","family":"Aikman","sequence":"additional","affiliation":[{"name":"New Mexico State University, NM, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6774-7668","authenticated-orcid":false,"given":"Younghyun","family":"Cho","sequence":"additional","affiliation":[{"name":"Santa Clara University, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3750-1178","authenticated-orcid":false,"given":"Yang","family":"Liu","sequence":"additional","affiliation":[{"name":"Lawrence Berkeley National Laboratory, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5362-3612","authenticated-orcid":false,"given":"Khaled Z","family":"Ibrahim","sequence":"additional","affiliation":[{"name":"Lawrence Berkeley National Laboratory, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2025,7,23]]},"reference":[{"key":"e_1_3_4_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2018.2841200"},{"key":"e_1_3_4_3_1","first-page":"253","volume-title":"ACM International Conference on Supercomputing 25th Anniversary Volume","author":"Bilmes J","year":"1997","unstructured":"Bilmes J, Asanovic K, Chin CW, et al. (1997) Optimizing matrix multiply using phipac: a portable, high-performance, ansi c coding methodology. ACM International Conference on Supercomputing 25th Anniversary Volume. New York: ACM, 253\u2013260."},{"key":"e_1_3_4_4_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719642"},{"key":"e_1_3_4_5_1","unstructured":"Cauwet ML Couprie C Dehos J et al. (2020) Fully parallel hyperparameter search: reshaped space-filling. JMLR.org: ICML\u201920."},{"key":"e_1_3_4_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-44973-4_7"},{"key":"e_1_3_4_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS54959.2023.00069"},{"key":"e_1_3_4_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-8191(00)00087-9"},{"key":"e_1_3_4_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11227-018-2676-z"},{"key":"e_1_3_4_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/PMBS56514.2022.00006"},{"key":"e_1_3_4_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW63119.2024.00143"},{"key":"e_1_3_4_12_1","first-page":"381","volume-title":"2022 IEEE International Conference on Cluster Computing","author":"Egele R","year":"2022","unstructured":"Egele R, Balaprakash P, et al. (2022) Hpc storage service autotuning using variational- autoencoder -guided asynchronous bayesian optimization. 2022 IEEE International Conference on Cluster Computing. Washington, DC: IEEE Computer Society, 381\u2013393."},{"key":"e_1_3_4_13_1","unstructured":"Egele R Chang T Sun Y et al (2023a) Parallel multi-objective hyperparameter optimization with uniform normalization and bounded objectives. https:\/\/arxiv.org\/abs\/2309.14936"},{"key":"e_1_3_4_14_1","first-page":"1","volume-title":"Asynchronous decentralized bayesian optimization for large scale hyperparameter optimization 2023","author":"Egel\u00e9 R","year":"2023","unstructured":"Egel\u00e9 R, Guyon I, Vishwanath V, et al (2023b) Asynchronous decentralized bayesian optimization for large scale hyperparameter optimization 2023 IEEE 19th International Conference on E-Science (E-Science). Los Alamitos, CA: IEEE Computer Society, pp. 1\u201310."},{"key":"e_1_3_4_15_1","article-title":"A multi-points criterion for deterministic parallel global optimization based on kriging","volume":"07","author":"Ginsbourger D","year":"2008","unstructured":"Ginsbourger D, Le Riche R, Carraro L (2008) A multi-points criterion for deterministic parallel global optimization based on kriging. International Conference on Nonconvex Programming NCP07. Available at: https:\/\/hal.science\/hal-00294130.","journal-title":"International Conference on Nonconvex Programming"},{"key":"e_1_3_4_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-10701-6_6"},{"key":"e_1_3_4_17_1","unstructured":"Jaderberg M Dalibard V Osindero S et al. (2017) Population based training of neural networks. https:\/\/arxiv.org\/abs\/1711.09846"},{"key":"e_1_3_4_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01174-1_1"},{"key":"e_1_3_4_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/779359.779361"},{"key":"e_1_3_4_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3577197"},{"key":"e_1_3_4_21_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1912.06059"},{"key":"e_1_3_4_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437801.3441621"},{"key":"e_1_3_4_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS47924.2020.00090"},{"key":"e_1_3_4_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00940509"},{"key":"e_1_3_4_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-020-05899-z"},{"key":"e_1_3_4_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2015.106"},{"key":"e_1_3_4_27_1","volume-title":"Systems of Extremal Control","author":"Rastrigin LA","year":"1974","unstructured":"Rastrigin LA (1974) Systems of Extremal Control. Nauka."},{"key":"e_1_3_4_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377929.3390017"},{"key":"e_1_3_4_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454109"},{"key":"e_1_3_4_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2015.2494218"},{"key":"e_1_3_4_31_1","volume-title":"Advances in Neural Information Processing Systems","author":"Snoek J","year":"2012","unstructured":"Snoek J, Larochelle H, Adams RP (2012) Practical bayesian optimization of machine learning algorithms. In: Pereira F, Burges C, Bottou L, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc. Vol. 25. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2012\/file\/05311655a15b75fab86956663e1819cd-Paper.pdf","edition":"25"},{"key":"e_1_3_4_32_1","unstructured":"Surjanovic S Bingham D (2013) Branin function. https:\/\/www.sfu.ca\/\\textasciitildessurjano\/branin.html"},{"key":"e_1_3_4_33_1","volume-title":"Advances in Neural Information Processing Systems","author":"Swersky K","year":"2013","unstructured":"Swersky K, Snoek J, Adams RP (2013) Multi-task bayesian optimization. In: Advances in Neural Information Processing Systems. Curran Associates, Inc."},{"key":"e_1_3_4_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2009.5161054"},{"key":"e_1_3_4_35_1","doi-asserted-by":"publisher","DOI":"10.1115\/DETC2021-70828"},{"key":"e_1_3_4_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2018.08.004"},{"key":"e_1_3_4_37_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094342004041293"},{"key":"e_1_3_4_38_1","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/16\/1\/071"},{"key":"e_1_3_4_39_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.2019.1966"},{"key":"e_1_3_4_40_1","article-title":"The reparameterization trick for acquisition functions","author":"Wilson JT","year":"2017","unstructured":"Wilson JT, Moriconi R, Hutter F, et al. (2017) The reparameterization trick for acquisition functions. ArXiv. https:\/\/arxiv.org\/abs\/1712.00424","journal-title":"ArXiv"},{"key":"e_1_3_4_41_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.8322"},{"key":"e_1_3_4_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-89740-8_24"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420251362001","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10943420251362001","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420251362001","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T08:17:47Z","timestamp":1777450667000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10943420251362001"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,23]]},"references-count":41,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["10.1177\/10943420251362001"],"URL":"https:\/\/doi.org\/10.1177\/10943420251362001","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,23]]}}}