{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:53:51Z","timestamp":1791460431349,"version":"4.3.4"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,11,24]],"date-time":"2023-11-24T00:00:00Z","timestamp":1700784000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,1,31]]},"abstract":"<jats:p>\n            There is abundant observational data in the software engineering domain, whereas running large-scale controlled experiments is often practically impossible. Thus, most empirical studies can only report statistical\n            <jats:italic>correlations<\/jats:italic>\n            \u2014instead of potentially more insightful and robust\n            <jats:italic>causal<\/jats:italic>\n            relations.\n          <\/jats:p>\n          <jats:p>\n            To support analyzing purely observational data for causal relations and to assess any differences between purely predictive and causal models of the same data, this article discusses some novel techniques based on structural causal models (such as directed acyclic graphs of causal Bayesian networks). Using these techniques, one can rigorously express, and partially validate, causal hypotheses and then use the causal information to guide the construction of a statistical model that captures genuine causal relations\u2014such that correlation\n            <jats:italic>does imply<\/jats:italic>\n            causation.\n          <\/jats:p>\n          <jats:p>We apply these ideas to analyzing public data about programmer performance in Code Jam, a large world-wide coding contest organized by Google every year. Specifically, we look at the impact of different programming languages on a participant\u2019s performance in the contest. While the overall effect associated with programming languages is weak compared to other variables\u2014regardless of whether we consider correlational or causal links\u2014we found considerable differences between a purely associational and a causal analysis of the very same data.<\/jats:p>\n          <jats:p>The takeaway message is that even an imperfect causal analysis of observational data can help answer the salient research questions more precisely and more robustly than with just purely predictive techniques\u2014where genuine causal effects may be confounded.<\/jats:p>","DOI":"10.1145\/3611667","type":"journal-article","created":{"date-parts":[[2023,8,19]],"date-time":"2023-08-19T09:52:38Z","timestamp":1692438758000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1040-3201","authenticated-orcid":false,"given":"Carlo A.","family":"Furia","sequence":"first","affiliation":[{"name":"Software Institute, USI Universit\u00e0 della Svizzera italiana, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0118-8143","authenticated-orcid":false,"given":"Richard","family":"Torkar","sequence":"additional","affiliation":[{"name":"University of Gothenburg, Sweden, and Stellenbosch Institute for Advanced Study (STIAS), South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5179-4205","authenticated-orcid":false,"given":"Robert","family":"Feldt","sequence":"additional","affiliation":[{"name":"Chalmers and University of Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,11,24]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"25","article-title":"Software Engineering Body of Knowledge","author":"Abran Alain","year":"2004","unstructured":"Alain Abran, James W. Moore, Pierre Bourque, Robert Dupuis, and L. Tripp. 2004. Software Engineering Body of Knowledge. IEEE Computer Society, Angela Burgess, 25.","journal-title":"IEEE Computer Society, Angela Burgess"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-019-00857-9"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/1831708.1831717"},{"key":"e_1_3_2_5_2","volume-title":"Comparing Programming Languages in Google Code Jam","author":"Back Alexandra","year":"2017","unstructured":"Alexandra Back and Emma Westman. 2017. Comparing Programming Languages in Google Code Jam. Master\u2019s thesis. Chalmers University of Technology. https:\/\/publications.lib.chalmers.se\/records\/fulltext\/250672\/250672.pdf"},{"key":"e_1_3_2_6_2","unstructured":"Daniel J. Benjamin James O. Berger Magnus Johannesson Brian A. Nosek E.-J. Wagenmakers Richard Berk Kenneth A. Bollen Bj\u00f6rn Brembs Lawrence Brown Colin Camerer David Cesarini Christopher D. Chambers Merlise Clyde Thomas D. Cook Paul De Boeck Zoltan Dienes Anna Dreber Kenny Easwaran Charles Efferson Ernst Fehr Fiona Fidler Andy P. Field Malcolm Forster Edward I. George Richard Gonzalez Steven Goodman Edwin Green Donald P. Green Anthony G. Greenwald Jarrod D. Hadfield Larry V. Hedges Leonhard Held Teck Hua Ho Herbert Hoijtink Daniel J. Hruschka Kosuke Imai Guido Imbens John P. A. Ioannidis Minjeong Jeon James Holland Jones Michael Kirchler David Laibson John List Roderick Little Arthur Lupia Edouard Machery Scott E. Maxwell Michael McCarthy Don A. Moore Stephen L. Morgan Marcus Munaf\u00f3 Shinichi Nakagawa Brendan Nyhan Timothy H. Parker Luis Pericchi Marco Perugini Jeff Rouder Judith Rousseau Victoria Savalei Felix D. Sch\u00f6nbrodt Thomas Sellke Betsy Sinclair Dustin Tingley Trisha Van Zandt Simine Vazire Duncan J. Watts Christopher Winship Robert L. Wolpert Yu Xie Cristobal Young Jonathan Zinman and Valen E. Johnson. 2018. Redefine statistical significance. Nature Human Behaviour 2 6-10 (2018)."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340571"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0120125"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1177\/00491241221099552"},{"key":"e_1_3_2_10_2","article-title":"Testing causality in scientific modelling software","author":"Clark Andrew G.","year":"2022","unstructured":"Andrew G. Clark, Michael Foster, Benedikt Prifling, Neil Walkinshaw, Robert M. Hierons, Volker Schmidt, and Robert D. Turner. 2022. Testing causality in scientific modelling software. arXiv preprint arXiv:2209.00357 (2022).","journal-title":"arXiv preprint arXiv:2209.00357"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.49.12.997"},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"Clemens Dubslaff Kallistos Weis Christel Baier and Sven Apel. 2022. Causality in configurable software systems. In Proceedings of the 44th International Conference on Software Engineering (ICSE\u201922) ACM.","DOI":"10.1145\/3510003.3510200"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"Hongbo Fang Hemank Lamba James Herbsleb and Bogdan Vasilescu. 2022. \u201cThis is damn slick!\u201d Estimating the impact of tweets on open source project popularity and new contributors. In Proceedings of the 44th International Conference on Software Engineering (ICSE\u201922) ACM.","DOI":"10.1145\/3510003.3510121"},{"issue":"9","key":"e_1_3_2_14_2","first-page":"1786","article-title":"Bayesian data analysis in empirical software engineering research","volume":"47","author":"Furia Carlo A.","year":"2021","unstructured":"Carlo A. Furia, Robert Feldt, and Richard Torkar. 2021. Bayesian data analysis in empirical software engineering research. IEEE Transactions on Software Engineering 47, 9 (September 2021), 1786\u20131810.","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"3","key":"e_1_3_2_15_2","first-page":"40:1\u201340:38","article-title":"Applying Bayesian analysis guidelines to empirical software engineering data: The case of programming languages and code quality","volume":"31","author":"Furia Carlo A.","year":"2022","unstructured":"Carlo A. Furia, Richard Torkar, and Robert Feldt. 2022. Applying Bayesian analysis guidelines to empirical software engineering data: The case of programming languages and code quality. ACM Transactions on Software Engineering and Methodology 31, 3 (2022), 40:1\u201340:38.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7541480"},{"key":"e_1_3_2_17_2","article-title":"The problems with P-values are not just with P-values","author":"Gelman Andrew","year":"2016","unstructured":"Andrew Gelman. 2016. The problems with P-values are not just with P-values. American Statistician 70 (2016). Online discussion: http:\/\/www.stat.columbia.edu\/gelman\/research\/published\/asa_pvalues.pdf","journal-title":"American Statistician"},{"key":"e_1_3_2_18_2","unstructured":"Andrew Gelman. 2016. Why I Prefer 50% Rather Than 95% Intervals. https:\/\/statmodeling.stat.columbia.edu\/2016\/11\/05\/why-i-prefer-50-to-95-intervals\/ From the blog Statistical Modeling Causal Inference and Social Science ."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1017\/9781139161879"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1511\/2009.79.310"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-130-12-199906150-00008"},{"key":"e_1_3_2_22_2","first-page":"12","volume-title":"Olympiads in Informatics","author":"Hakulinen Lasse","year":"2011","unstructured":"Lasse Hakulinen. 2011. Survey on informatics competitions: Developing tasks. In Olympiads in Informatics, Vol. 5. IOI, 12\u201325."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.5555\/2832581.2832671"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1145\/1869459.1869462","volume-title":"Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA\u201910)","author":"Hanenberg Stefan","year":"2010","unstructured":"Stefan Hanenberg. 2010. An experiment about static and dynamic type systems: Doubts about the positive impact of static type systems on development time. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA\u201910). ACM, New York, NY, 22\u201335."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12451"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.p1135"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3522664.3528615"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1257\/jel.20191597"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511790423"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3231711"},{"key":"e_1_3_2_31_2","article-title":"Causal program dependence analysis","author":"Lee Seongmin","year":"2021","unstructured":"Seongmin Lee, Dave Binkley, Robert Feldt, Nicolas Gold, and Shin Yoo. 2021. Causal program dependence analysis. arXiv preprint arXiv:2104.09107 (2021).","journal-title":"arXiv preprint arXiv:2104.09107"},{"key":"e_1_3_2_32_2","first-page":"353","volume-title":"2019 IFIP\/IEEE Symposium on Integrated Network and Service Management (IM\u201919)","author":"Li Xing","year":"2019","unstructured":"Xing Li, Yinbo Yu, Kai Bu, Yan Chen, Jianfeng Yang, and Ruijie Quan. 2019. Thinking inside the box: Differential fault localization for SDN control plane. In 2019 IFIP\/IEEE Symposium on Integrated Network and Service Management (IM\u201919). IEEE, 353\u2013359."},{"key":"e_1_3_2_33_2","article-title":"Bayesian causal inference in automotive software engineering and online evaluation","author":"Liu Yuchu","year":"2022","unstructured":"Yuchu Liu, David Issa Mattos, Jan Bosch, Helena Holmstr\u00f6m Olsson, and Jonn Lantz. 2022. Bayesian causal inference in automotive software engineering and online evaluation. arXiv preprint arXiv:2207.00222 (2022).","journal-title":"arXiv preprint arXiv:2207.00222"},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","DOI":"10.1201\/9780429029608","volume-title":"Statistical Rethinking: A Bayesian Course with Examples in R and Stan (2nd Ed.)","author":"McElreath Richard","year":"2020","unstructured":"Richard McElreath. 2020. Statistical Rethinking: A Bayesian Course with Examples in R and Stan (2nd Ed.). CRC Press."},{"issue":"1","key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1080\/00031305.2018.1527253","article-title":"Abandon statistical significance","volume":"73","author":"McShane Blakeley B.","year":"2019","unstructured":"Blakeley B. McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L. Tackett. 2019. Abandon statistical significance. American Statistician 73, S1 (2019), 235\u2013245.","journal-title":"American Statistician"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.04.005"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2509136.2509515"},{"key":"e_1_3_2_38_2","unstructured":"Sebastian Nanz and Carlo A. Furia. 2014. A Comparative Study of Programming Languages in Rosetta Code. http:\/\/arxiv.org\/abs\/1409.0252"},{"key":"e_1_3_2_39_2","first-page":"778","volume-title":"Proceedings of the 37th International Conference on Software Engineering (ICSE\u201915)","author":"Nanz Sebastian","year":"2015","unstructured":"Sebastian Nanz and Carlo A. Furia. 2015. A comparative study of programming languages in Rosetta code. In Proceedings of the 37th International Conference on Software Engineering (ICSE\u201915), Antonia Bertolino, Gerardo Canfora, and Sebastian Elbaum (Eds.). ACM, 778\u2013788."},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1109\/ESEM.2011.41","volume-title":"Proceedings of the 2011 International Symposium on Empirical Software Engineering and Measurement (ESEM\u201911)","author":"Nanz Sebastian","year":"2011","unstructured":"Sebastian Nanz, Faraz Torshizi, Michela Pedroni, and Bertrand Meyer. 2011. Design of an empirical study for comparing the usability of concurrent programming languages. In Proceedings of the 2011 International Symposium on Empirical Software Engineering and Measurement (ESEM\u201911). IEEE Computer Society, Washington, DC, 325\u2013334."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511803161"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511803161"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/2020408.2020416"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3241036"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1214\/09-SS057"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.5555\/3238230"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.5555\/3202377"},{"issue":"10","key":"e_1_3_2_48_2","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1109\/2.876288","article-title":"An empirical comparison of seven programming languages","volume":"33","author":"Prechelt Lutz","year":"2000","unstructured":"Lutz Prechelt. 2000. An empirical comparison of seven programming languages. IEEE Computer 33, 10 (Oct. 2000), 23\u201329.","journal-title":"IEEE Computer"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635922"},{"issue":"1","key":"e_1_3_2_50_2","first-page":"1","article-title":"Improving the accuracy of medical diagnosis with causal machine learning","volume":"11","author":"Richens Jonathan G.","year":"2020","unstructured":"Jonathan G. Richens, Ciar\u00e1n M. Lee, and Saurabh Johri. 2020. Improving the accuracy of medical diagnosis with causal machine learning. Nature Communications 11, 1 (2020), 1\u20139.","journal-title":"Nature Communications"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/1693453.1693462"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0037350"},{"key":"e_1_3_2_53_2","unstructured":"Maximilian Scholz and Paul-Christian B\u00fcrkner. 2022. Prediction Can be Safely Used as a Proxy for Explanation in Causally Consistent Bayesian Generalized Linear Models. https:\/\/arxiv.org\/abs\/2210.06927"},{"issue":"8","key":"e_1_3_2_54_2","doi-asserted-by":"crossref","first-page":"e1787","DOI":"10.1002\/stvr.1787","article-title":"An empirical study of linespots: A novel past-fault algorithm","volume":"31","author":"Scholz Maximilian","year":"2021","unstructured":"Maximilian Scholz and Richard Torkar. 2021. An empirical study of linespots: A novel past-fault algorithm. Software Testing, Verification and Reliability 31, 8 (2021), e1787.","journal-title":"Software Testing, Verification and Reliability"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568255"},{"key":"e_1_3_2_56_2","article-title":"Applications of statistical causal inference in software engineering","author":"Siebert Julian","year":"2022","unstructured":"Julian Siebert. 2022. Applications of statistical causal inference in software engineering. arXiv preprint arXiv:2211.11482 (2022).","journal-title":"arXiv preprint arXiv:2211.11482"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611417632"},{"key":"e_1_3_2_58_2","first-page":"1","volume-title":"Applied informatics","author":"Spirtes Peter","year":"2016","unstructured":"Peter Spirtes and Kun Zhang. 2016. Causal discovery and inference: Concepts and recent methodological advances. In Applied informatics, Vol. 3. SpringerOpen, 1\u201328."},{"key":"e_1_3_2_59_2","article-title":"We Should Be Cautious about Associations of Patient Characteristics with COVID-19 Outcomes That Are Identified in Hospitalised Patients","author":"Sterne Jonathan","year":"2020","unstructured":"Jonathan Sterne. 2020. We Should Be Cautious about Associations of Patient Characteristics with COVID-19 Outcomes That Are Identified in Hospitalised Patients. Health Data Research UK \u2013 https:\/\/www.hdruk.ac.uk\/news\/we-should-be-cautious-about-associations-of-patient-characteristics-with-covid-19-outcomes-that-are-identified-in-hospitalised-patients\/","journal-title":"Health Data Research UK"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3048991"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-016-9696-4"},{"key":"e_1_3_2_62_2","volume-title":"Future World: Educating for the 21st Century","author":"Verhoeff Tom","year":"1997","unstructured":"Tom Verhoeff. 1997. The role of competitions in education. In Future World: Educating for the 21st Century. IOI."},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2016.1154108"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3611667","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3611667","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:08Z","timestamp":1750178228000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3611667"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,24]]},"references-count":62,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1,31]]}},"alternative-id":["10.1145\/3611667"],"URL":"https:\/\/doi.org\/10.1145\/3611667","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,24]]},"assertion":[{"value":"2023-01-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-07","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}