{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,9]],"date-time":"2026-11-09T23:22:21Z","timestamp":1794266541686,"version":"4.9.0"},"reference-count":73,"publisher":"Association for Computing Machinery (ACM)","issue":"10","license":[{"start":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T00:00:00Z","timestamp":1789516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Science Council","id":[{"id":"https:\/\/ror.org\/04acy3777","id-type":"ROR","asserted-by":"crossref"}]},{"name":"\u201cAutomated Boundary Testing for AI\/ML models\u201d and from WASP for the project \u201cBound-Miner.\u201d"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,10,31]]},"abstract":"<jats:p>In empirical software engineering (SE) research, researchers have considerable freedom to decide how to process data, what operationalizations to use, and which statistical model to fit. Gelman and Loken refer to this freedom as leading to a \u201cgarden of forking paths.\u201d Although this freedom is often seen as an advantage, it also poses a threat to robustness and replicability: variations in analytical decisions, even when justifiable, can lead to divergent conclusions.<\/jats:p>\n                  <jats:p>\n                    To better understand this risk, we conducted a so-called\n                    <jats:italic toggle=\"yes\">multiverse analysis<\/jats:italic>\n                    on a published empirical SE paper. The paper we picked is a Mining Software Repositories study, as MSR studies commonly use non-trivial statistical models to analyze post hoc, observational data. In the study, we identified nine pivotal analytical decisions\u2014each with at least one equally defensible alternative\u2014and systematically reran all the 3,072 resulting analysis pipelines on the original dataset. Interestingly, only six of these universes (&lt;0.2%) reproduced the published results; the overwhelming majority produced qualitatively different, and sometimes even opposite, findings.\n                  <\/jats:p>\n                  <jats:p>This case study of a data analytical method commonly applied to empirical SE data reveals how methodological choices can exert a more profound influence on outcomes than is often acknowledged. We therefore advocate that SE researchers complement standard reporting with robustness checks across plausible analysis variants or, at least, explicitly justify each analytical decision. We propose a structured classification model to help classify and improve justification for methodological choices. Secondly, we show how the multiverse analysis is a practical tool in the methodological arsenal of SE researchers, one that can help produce more reliable, reproducible science.<\/jats:p>","DOI":"10.1145\/3799711","type":"journal-article","created":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T13:10:57Z","timestamp":1772457057000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6511-918X","authenticated-orcid":false,"given":"Nathan","family":"Cassee","sequence":"first","affiliation":[{"name":"University of Victoria, Victoria, British Columbia, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5179-4205","authenticated-orcid":false,"given":"Robert","family":"Feldt","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,9,16]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.10.006"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3558948"},{"key":"e_1_3_3_5_2","unstructured":"Samuel J. Bell Onno P. Kampman Jesse Dodge and Neil D. Lawrence. 2022. Modeling the machine learning multiverse. arXiv:2206.05985. Retrieved from https:\/\/arxiv.org\/abs\/2206.05985"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2009.5069475"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2203150119"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER48275.2020.9054818"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctvc779w5"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-75959-3"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.07.002"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300295"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2005.08.009"},{"issue":"2","key":"e_1_3_3_15_2","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1007\/s10664-022-10277-5","article-title":"Registered reports in software engineering","volume":"28","author":"Ernst Neil A.","year":"2023","unstructured":"Neil A. Ernst and Maria Teresa Baldassarre. 2023. Registered reports in software engineering. Empirical Software Engineering 28, 2 (2023), 55.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_3_16_2","unstructured":"Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri et al. 2020. ACM SIGSOFT Empirical Standards. arXiv:2010.03525. Retrieved from https:\/\/arxiv.org\/abs\/2010.03525"},{"key":"e_1_3_3_17_2","unstructured":"Robert Feldt and Ana Magazinius. 2010. Validity threats in empirical software engineering research\u2014An initial survey. In Proceedings of the International Conference on Software Engineering and Knowledge Engineering 374\u2013379."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3527957"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643664.3648211"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2935974"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611667"},{"key":"e_1_3_3_22_2","first-page":"1","article-title":"The garden of forking paths: Why multiple comparisons can be a problem, even when there is no \u201cfishing expedition\u201d or \u201cp-hacking\u201d and the research hypothesis was posited ahead of time","volume":"348","author":"Gelman Andrew","year":"2013","unstructured":"Andrew Gelman and Eric Loken. 2013. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no \u201cfishing expedition\u201d or \u201cp-hacking\u201d and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University 348 (2013), 1\u201317.","journal-title":"Department of Statistics, Columbia University"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245920954925"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.245"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.14443"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691620917678"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-resource-121517-033306"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3527960"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10370-3"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300432"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9393-5"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00060"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00054"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9437-5"},{"key":"e_1_3_3_35_2","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1145\/3674805.3686691","volume-title":"Proceedings of the 18th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement","author":"Lago Patricia","year":"2024","unstructured":"Patricia Lago, Per Runeson, Qunying Song, and Roberto Verdecchia. 2024. Threats to validity in software engineering\u2013hypocritical paper section or essential analysis? In Proceedings of the 18th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement, 314\u2013324."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3224053"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3527963"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2020.3028985"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER48275.2020.9054792"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10285-5"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1201\/9781315372495"},{"issue":"2019","key":"e_1_3_3_42_2","doi-asserted-by":"crossref","first-page":"1057","DOI":"10.1007\/s10664-019-09712-x","article-title":"The open science initiative of the empirical software engineering journal","volume":"24","author":"M\u00e9ndez Fern\u00e1ndez Daniel","year":"2019","unstructured":"Daniel M\u00e9ndez Fern\u00e1ndez, Martin Monperrus, Robert Feldt, and Thomas Zimmermann. 2019. The open science initiative of the empirical software engineering journal. Empirical Software Engineering 24 (2019), 1057\u20131060.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.04.005"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528528"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00058"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9512-6"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2900308"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180161"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649598"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447580"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.envsoft.2019.01.012"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09907-7"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR59073.2023.00043"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1111\/jpim.12738"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2021.02.003"},{"issue":"2018","key":"e_1_3_3_56_2","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.infsof.2018.01.006","article-title":"The role and value of replication in empirical software engineering results","volume":"99","author":"Shepperd Martin","year":"2018","unstructured":"Martin Shepperd, Nemitari Ajienka, and Steve Counsell. 2018. The role and value of replication in empirical software engineering results. Information and Software Technology 99 (2018), 120\u2013132.","journal-title":"Information and Software Technology"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2322358"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245917747646"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-020-0912-z"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691616658637"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2014.11.010"},{"key":"e_1_3_3_62_2","unstructured":"Margaret-Anne Storey Rashina Hoda Alessandra Maciel Paz Milani and Maria Teresa Baldassarre. 2025. Guiding principles for using mixed methods research in software engineering. arXiv:2404.06011. Retrieved from https:\/\/arxiv.org\/abs\/2404.06011"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1080\/09515089.2018.1490707"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP58684.2023.00010"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180209"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2023.107329"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387465"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00011"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/S10664-022-10130-9"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2_8"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643664.3648203"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643664.3648213"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155575"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2019.00011"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3799711","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T21:57:25Z","timestamp":1789595845000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3799711"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,16]]},"references-count":73,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2026,10,31]]}},"alternative-id":["10.1145\/3799711"],"URL":"https:\/\/doi.org\/10.1145\/3799711","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,16]]},"assertion":[{"value":"2025-06-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-09-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}