{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T05:57:39Z","timestamp":1792475859020,"version":"4.5.0"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,3,7]],"date-time":"2022-03-07T00:00:00Z","timestamp":1646611200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100004359","name":"Swedish Research Council","doi-asserted-by":"crossref","award":["2018-05973"],"award-info":[{"award-number":["2018-05973"]}],"id":[{"id":"10.13039\/501100004359","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,7,31]]},"abstract":"<jats:p>\n                    Statistical analysis is the tool of choice to turn data into information and then information into empirical knowledge. However, the process that goes from data to knowledge is long, uncertain, and riddled with pitfalls. To be valid, it should be supported by detailed, rigorous guidelines that help ferret out issues with the data or model and lead to qualified results that strike a reasonable balance between generality and practical relevance. Such guidelines are being developed by statisticians to support the latest techniques for\n                    <jats:italic>Bayesian<\/jats:italic>\n                    data analysis. In this article, we frame these guidelines in a way that is apt to empirical research in software engineering.\n                  <\/jats:p>\n                  <jats:p>\n                    To demonstrate the guidelines in practice, we apply them to reanalyze a GitHub dataset about code quality in different programming languages. The dataset\u2019s original analysis\u00a0[Ray et\u00a0al.\u00a0\n                    <jats:xref ref-type=\"bibr\">55<\/jats:xref>\n                    ] and a critical reanalysis\u00a0[Berger et\u00a0al.\u00a0\n                    <jats:xref ref-type=\"bibr\">6<\/jats:xref>\n                    ] have attracted considerable attention\u2014in no small part because they target a topic (the impact of different programming languages) on which strong opinions abound. The goals of our reanalysis are largely orthogonal to this previous work, as we are concerned with demonstrating, on data in an interesting domain, how to build a principled Bayesian data analysis and to showcase its benefits. In the process, we will also shed light on some critical aspects of the analyzed data and of the relationship between programming languages and code quality\u2014such as the impact of project-specific characteristics other than the used programming language.\n                  <\/jats:p>\n                  <jats:p>The high-level conclusions of our exercise will be that Bayesian statistical techniques can be applied to analyze software engineering data in a way that is principled, flexible, and leads to convincing results that inform the state-of-the-art while highlighting the boundaries of its validity. The guidelines can support building solid statistical analyses and connecting their results. Thus, they can help buttress continued progress in empirical software engineering research.<\/jats:p>","DOI":"10.1145\/3490953","type":"journal-article","created":{"date-parts":[[2022,3,7]],"date-time":"2022-03-07T05:24:54Z","timestamp":1646630694000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":31,"title":["Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1040-3201","authenticated-orcid":false,"given":"Carlo A.","family":"Furia","sequence":"first","affiliation":[{"name":"Software Institute, USI Universit\u00e0 della Svizzera italiana, Lugano, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0118-8143","authenticated-orcid":false,"given":"Richard","family":"Torkar","sequence":"additional","affiliation":[{"name":"University of Gothenburg, Sweden and Stellenbosch Institute for Advanced Study (STIAS), South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5179-4205","authenticated-orcid":false,"given":"Robert","family":"Feldt","sequence":"additional","affiliation":[{"name":"Chalmers and University of Gothenburg, G\u00f6teborg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,3,7]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-019-0807-z"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-019-00857-9"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601299"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1161\/STROKEAHA.106.474080"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340571"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2015.7321213"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.05.004"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1976.10480949"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-438150-6.50018-2"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1201\/b10905"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v080.i01"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v076.i01"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cortex.2015.03.022"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-012-9227-7"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.07.002"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09791-w"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1177\/1471082X15588398"},{"key":"e_1_3_2_20_2","unstructured":"Robert Feldt and Ana Magazinius. 2010. Validity threats in empirical software engineering research \u2014 an initial survey. In Proceedings of the 22nd International Conference on Software Engineering and Knowledge Engineering (SEKE\u20192010) Redwood City San Francisco Bay CA USA July 1\u2013July 3 2010 . Knowledge Systems Institute Graduate School 374\u2013379."},{"key":"e_1_3_2_21_2","volume-title":"Statistical Methods for Research Workers (1 ed.)","author":"Fisher R. A.","year":"1925","unstructured":"R. A. Fisher. 1925. Statistical Methods for Research Workers (1 ed.). Oliver and Boyd."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2935974"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4472963"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssa.12378"},{"key":"e_1_3_2_25_2","first-page":"1183","volume-title":"Proceedings of the 34th International Conference on Machine Learning (ICML\u201917)","author":"Gal Yarin","year":"2017","unstructured":"Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning (ICML\u201917). JMLR.org, 1183\u20131192."},{"key":"e_1_3_2_26_2","first-page":"1682","volume-title":"International Conference on Artificial Intelligence and Statistics (AISTATS\u201918)","author":"Ge Hong","year":"2018","unstructured":"Hong Ge, Kai Xu, and Zoubin Ghahramani. 2018. Turing: A language for flexible probabilistic inference. In International Conference on Artificial Intelligence and Statistics (AISTATS\u201918). MLResearchPress, Canary Islands, Spain, 1682\u20131690."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1198\/106186004X11435"},{"key":"e_1_3_2_28_2","unstructured":"Andrew Gelman. 2016. Bayesian statistics: What\u2019s it all about? Retrieved December 29 2021 fromhttps:\/\/statmodeling.stat.columbia.edu\/2016\/12\/13\/bayesian-statistics-whats\/."},{"key":"e_1_3_2_29_2","volume-title":"Data Analysis Using Regression and Multilevel\/Hierarchical Models","author":"Gelman Andrew","year":"2007","unstructured":"Andrew Gelman and Jennifer Hill. 2007. Data Analysis Using Regression and Multilevel\/Hierarchical Models. Vol. Analytical methods for social research. Cambridge University Press, Cambridge, UK."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1017\/9781139161879"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.3390\/e19100555"},{"key":"e_1_3_2_32_2","unstructured":"Andrew Gelman Aki Vehtari Daniel Simpson Charles C. Margossian Bob Carpenter Yuling Yao Lauren Kennedy Jonah Gabry Paul-Christian B\u00fcrkner and Martin Modr\u00e1k. 2020. Bayesian workflow. arxiv:stat.ME\/2011.01808."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9513-5"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2966.2011.18877.x"},{"issue":"1","key":"e_1_3_2_35_2","first-page":"1","article-title":"Misinterpretations of significance: A problem students share with their teachers?","volume":"7","author":"Haller Heiko","year":"2002","unstructured":"Heiko Haller and Stefan Kraus. 2002. Misinterpretations of significance: A problem students share with their teachers?Methods of Psychological Research 7, 1 (2002), 1\u201320.","journal-title":"Methods of Psychological Research"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-013-0572-3"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxn041"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511790423"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1214\/088342304000000026"},{"key":"e_1_3_2_40_2","unstructured":"JuliaStan. (n.d.) Stan.jl \u2014 The Julia interface to Stan. Retrieved December 29 2021 from https:\/\/mc-stan.org\/users\/interfaces\/julia-stan."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09837-4"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1002\/sim.3680"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1201\/9780429029608"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.256941"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.04.005"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4613-8122-8"},{"key":"e_1_3_2_47_2","unstructured":"Salman Mohamadi and Hamidreza Amindavar. 2020. Deep Bayesian Active Learning A Brief Survey on Recent Advances. arxiv:cs.LG\/2012.08044."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-0745-0"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511803161"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1214\/20-EJS1711"},{"key":"e_1_3_2_51_2","volume-title":"Proceedings of the 3rd International Workshop on Distributed Statistical Computing: Workshop on Distributed Statistical Computing","author":"Plummer Martin","year":"2003","unstructured":"Martin Plummer. 2003. JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd International Workshop on Distributed Statistical Computing: Workshop on Distributed Statistical Computing. Achim Zeileis, Vienna, Austria, 10."},{"key":"e_1_3_2_52_2","volume-title":"Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC)","author":"Plummer Martyn","year":"2003","unstructured":"Martyn Plummer. 2003. JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC)."},{"key":"e_1_3_2_53_2","unstructured":"PyStan. 2019. Release v3.3.0. Retrieved December 29 2021 from https:\/\/pystan.readthedocs.io\/en\/latest\/."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3210459.3210461"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3126905"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635922"},{"key":"e_1_3_2_57_2","series-title":"Proceedings of the 35th International Conference on Machine Learning (ICML\u201918),","first-page":"4381","volume":"80","author":"Ru Bin Xin","year":"2018","unstructured":"Bin Xin Ru, Mark McLeod, Diego Granziol, and Michael A. Osborne. 2018. Fast information-theoretic Bayesian optimisation. In Proceedings of the 35th International Conference on Machine Learning (ICML\u201918),Proceedings of Machine Learning Research, Jennifer G. Dy and Andreas Krause (Eds.), Vol. 80. PMLR, Stockholm, Sweden, 4381\u20134389. http:\/\/proceedings.mlr.press\/v80\/ru18a.html"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000275"},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","unstructured":"Maximilian Scholz and Richard Torkar. 2020. An empirical study of linespots: A novel past-fault algorithm. arxiv:cs.SE\/2007.09394.","DOI":"10.1002\/stvr.1787"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1201\/9780429347436"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2322358"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2731308"},{"key":"e_1_3_2_63_2","article-title":"Validating Bayesian inference algorithms with simulation-based calibration","author":"Talts Sean","year":"2018","unstructured":"Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. 2018. Validating Bayesian inference algorithms with simulation-based calibration. arXiv:1804.06788.","journal-title":"arXiv:1804.06788"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2553030"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-32489-6_11"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3048991"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1038\/s43586-020-00001-2"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-016-9696-4"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1958.10501419"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2606428"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2016.1154108"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1953045"},{"key":"e_1_3_2_73_2","volume-title":"Data Mining: Practical Machine Learning Tools and Techniques (4 ed.)","author":"Witten Ian H.","year":"2016","unstructured":"Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal. 2016. Data Mining: Practical Machine Learning Tools and Techniques (4 ed.). Morgan Kaufmann."},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110862"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3031401"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490953","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3490953","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T15:30:30Z","timestamp":1750174230000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490953"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,7]]},"references-count":74,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,7,31]]}},"alternative-id":["10.1145\/3490953"],"URL":"https:\/\/doi.org\/10.1145\/3490953","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,7]]},"assertion":[{"value":"2021-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}