{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:06:42Z","timestamp":1782846402863,"version":"3.54.5"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["SI 2171\/3-2 (699378)"],"award-info":[{"award-number":["SI 2171\/3-2 (699378)"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Knut och Alice Wallenbergs Stiftelse","doi-asserted-by":"publisher","award":["Wallenberg AI, Autonomous Systems and Software Program (WASP)"],"award-info":[{"award-number":["Wallenberg AI, Autonomous Systems and Software Program (WASP)"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["389792660"],"award-info":[{"award-number":["389792660"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>\n                    Modern software usually exposes a large number of configuration options to the user, giving rise to enormous configuration spaces in practice. Appropriate choices for these options dramatically influence the performance of the software (throughput, memory consumption, execution time, etc.). However, due to the sheer size of the configuration space, systematically identifying the worst- or best-performing configurations is computationally infeasible through exhaustive exploration. Instead, practitioners rely on budgeted sampling strategies, such as uniform random sampling or statistical recursive search, to explore the configuration space under fixed measurement budgets in an attempt to find the worst- or best-performing configuration. Even worse, a fundamental limitation of existing sampling strategies is the lack of quantifiable guarantees that the selected configuration truly reflects worst-case (or best-case) performance. In this paper, we define the basic concepts of\n                    <jats:italic toggle=\"yes\">posterior risk<\/jats:italic>\n                    and\n                    <jats:italic toggle=\"yes\">posterior confidence<\/jats:italic>\n                    and present a probabilistic framework to evaluate how well sampling strategies identify the worst- or best-performing configuration of a software system. We evaluate our framework by comparing five representative sampling strategies on seven real-world configurable software systems. We find that statistical recursive search yields consistently tighter best-case guarantees\u2014higher posterior confidence and lower posterior risk\u2014than the alternatives at the same budget. Our results demonstrate the applicability of our framework as a principled basis for reporting, comparing, and refining sampling strategies, and as a tool for practitioners to select strategies and budgets with quantified guarantees across systems and sample sizes.\n                  <\/jats:p>","DOI":"10.1145\/3808147","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"3162-3185","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating Risk and Confidence in Performance Bounds of Configuration Sampling Strategies"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-9120-1670","authenticated-orcid":false,"given":"Kallistos","family":"Weis","sequence":"first","affiliation":[{"name":"Saarland University, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1143-1127","authenticated-orcid":false,"given":"Martina","family":"Maggio","sequence":"additional","affiliation":[{"name":"Saarland University, Saarbr\u00fccken, Germany"},{"name":"Lund University, Lund, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7741-7777","authenticated-orcid":false,"given":"Norbert","family":"Siegmund","sequence":"additional","affiliation":[{"name":"ScaDS.AI, Dresden \/ Leipzig, Germany"},{"name":"Leipzig University, Leipzig, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3687-2233","authenticated-orcid":false,"given":"Sven","family":"Apel","sequence":"additional","affiliation":[{"name":"Saarland University, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3020795"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/Q-SE59154.2023.00012"},{"key":"e_1_2_1_3_1","volume-title":"Feature-Oriented Software Product Lines","author":"Apel Sven","unstructured":"Sven Apel, Don Batory, Christian K\u00e4stner, and Gunter Saake. 2016. Feature-Oriented Software Product Lines. Springer."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2024.103209"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSR66718.2025.00012"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2006.875041"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10250-2"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.3850\/978-981-11-2724-3_0618-cd"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180248"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","unstructured":"Omid Gheibi Christian K\u00e4stner and Pooyan Jamshidi. 2025. Hardness Structural Knowledge and Opportunity: An Analytical Framework for Modular Performance Modeling. doi:10.48550\/arXiv.2509.11000 10.48550\/arXiv.2509.11000","DOI":"10.48550\/arXiv.2509.11000"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616334"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3491945"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.4057"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ase.2013.6693089"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9573-6"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642971"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","unstructured":"Huong Ha Zongwen Fan and Hongyu Zhang. 2022. Uncertainty-Aware Performance Prediction for Highly Configurable Software Systems via Bayesian Neural Networks. doi:10.48550\/arXiv.2212.13359 10.48550\/arXiv.2212.13359","DOI":"10.48550\/arXiv.2212.13359"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00113"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/icsme.2019.00080"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.69"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3382025.3414951"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00091"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3634713.3634716"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3171404"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CDC57313.2025.11312114"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2362536.2362547"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3715340.3715442"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.2987024"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/icse.2019.00112"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10338-3"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1201\/9780203740774"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-41705-5"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3712193"},{"key":"e_1_2_1_35_1","volume-title":"Essentials of Metaheuristics","author":"Luke Sean","unstructured":"Sean Luke. 2013. Essentials of Metaheuristics (second ed.). Lulu."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3688836"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409685"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491627.2491646"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884793"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336294.3336297"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106238"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-017-0225-2"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2870895"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611663"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106273"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336294.3342359"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15579-6_14"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2014.12.041"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3358960.3379137"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111044"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-011-9160-9"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.2307\/2332195"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2019.00032"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1002\/9780470725184"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ase.2015.45"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2013.6693104"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3382494.3410677"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786845"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-59762-7_10"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-53288-8_22"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.09790"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","unstructured":"Kallistos Weis Martina Maggio Norbert Siegmund and Sven Apel. 2026. Risk and Confidence in Performance Bounds of Configuration Sampling Strategies -Sources and Data. doi:10.5281\/zenodo.19682296 10.5281\/zenodo.19682296","DOI":"10.5281\/zenodo.19682296"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1347375.1347389"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510053"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-022-00348-8"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3808147","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:16:03Z","timestamp":1782843363000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3808147"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":66,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3808147"],"URL":"https:\/\/doi.org\/10.1145\/3808147","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}