{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T18:46:13Z","timestamp":1792089973233,"version":"4.4.0"},"reference-count":78,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2024,6,13]],"date-time":"2024-06-13T00:00:00Z","timestamp":1718236800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,6,13]],"date-time":"2024-06-13T00:00:00Z","timestamp":1718236800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerische Nationalfonds","doi-asserted-by":"crossref","award":["200021-182060 (Hi-Fi)"],"award-info":[{"award-number":["200021-182060 (Hi-Fi)"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2024,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Despite its massive popularity as a programming language, especially in novel domains like data science programs, there is comparatively little research about fault localization that targets Python. Even though it is plausible that several findings about programming languages like C\/C++ and Java\u2014the most common choices for fault localization research\u2014carry over to other languages, whether the dynamic nature of Python and how the language is used in practice affect the capabilities of classic fault localization approaches remain open questions to investigate. This paper is the first multi-family large-scale empirical study of fault localization on real-world Python programs and faults. Using Zou et al.\u2019s recent large-scale empirical study of fault localization in Java\u00a0(Zou et al. 2021) as the basis of our study, we investigated the effectiveness (i.e., localization accuracy), efficiency (i.e., runtime performance), and other features (e.g., different entity granularities) of seven well-known fault-localization techniques in four families (spectrum-based, mutation-based, predicate switching, and stack-trace based) on 135 faults from 13 open-source Python projects from the\n                    <jats:sc>BugsInPy<\/jats:sc>\n                    \u00a0curated collection\u00a0(Widyasari et al. 2020). The results replicate for Python several results known about Java, and shed light on whether Python\u2019s peculiarities affect the capabilities of fault localization. The replication package that accompanies this paper includes detailed data about our experiments, as well as the tool\n                    <jats:sc>FauxPy<\/jats:sc>\n                    \u00a0that we implemented to conduct the study.\n                  <\/jats:p>","DOI":"10.1007\/s10664-024-10475-3","type":"journal-article","created":{"date-parts":[[2024,6,13]],"date-time":"2024-06-13T05:01:56Z","timestamp":1718254916000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["An empirical study of fault localization in Python programs"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4522-7279","authenticated-orcid":false,"given":"Mohammad","family":"Rezaalipour","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carlo A.","family":"Furia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,6,13]]},"reference":[{"key":"10475_CR1","doi-asserted-by":"crossref","unstructured":"Abreu R, Zoeteweij P, van Gemund AJC (2007) On the accuracy of spectrum-based fault localization. In: Proceedings of the testing: academic and industrial conference practice and research techniques - MUTATION, pp 89\u201398","DOI":"10.1109\/TAICPART.2007.4344104"},{"key":"10475_CR2","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1038\/d41586-019-00857-9","volume":"567","author":"V Amrehin","year":"2019","unstructured":"Amrehin V, Greenland S, McShane B (2019) Scientists rise up against statistical significance. Nature 567:305\u2013307","journal-title":"Nature"},{"key":"10475_CR3","doi-asserted-by":"publisher","unstructured":"B\u00a0Le T-D, Lo D, Le\u00a0Goues C, Grunske L (2016) A learning-to-rank based fault localization approach using likely invariants. In: Proceedings of the 25th international symposium on software testing and analysis, pp 177\u2014188. https:\/\/doi.org\/10.1145\/2931037.2931049","DOI":"10.1145\/2931037.2931049"},{"key":"10475_CR4","unstructured":"Batchelder N (2023) Coverage.py. https:\/\/coverage.readthedocs.io\/. [Online; Accessed 6 April 2023]"},{"key":"10475_CR5","doi-asserted-by":"publisher","unstructured":"Bettenburg N, Just S, Schr\u00f6ter A, Weiss C, Premraj R, Zimmermann T (2008) What makes a good bug report? In: Proceedings of the 16th ACM SIGSOFT international symposium on foundations of software engineering, pp 308\u2013318. https:\/\/doi.org\/10.1145\/1453101.1453146","DOI":"10.1145\/1453101.1453146"},{"key":"10475_CR6","doi-asserted-by":"publisher","unstructured":"Chen Z, Chen L, Zhou Y, Xu Z, Chu WC, Xu B (2014) Dynamic slicing of Python programs. In: 2014 IEEE 38th annual computer software and applications conference, pp 219\u2013228. https:\/\/doi.org\/10.1109\/COMPSAC.2014.30","DOI":"10.1109\/COMPSAC.2014.30"},{"key":"10475_CR7","doi-asserted-by":"publisher","unstructured":"Cleve H, Zeller A (2005) Locating causes of program failures. In: Proceedings of the 27th international conference on software engineering, pp 342\u2013351. https:\/\/doi.org\/10.1145\/1062455.1062522","DOI":"10.1145\/1062455.1062522"},{"key":"10475_CR8","unstructured":"Cosmic Ray (2019) Cosmic Ray: mutation testing for Python. https:\/\/cosmic-ray.readthedocs.io\/. [Online; Accessed 6 April 2023]"},{"key":"10475_CR9","volume-title":"Biostatistics: A Foundation for Analysis in the Health Sciences","author":"WW Daniel","year":"1999","unstructured":"Daniel WW (1999) Biostatistics: A Foundation for Analysis in the Health Sciences, 7th edn. Wiley","edition":"7"},{"key":"10475_CR10","doi-asserted-by":"publisher","unstructured":"Debroy V, Wong WE, Xu X, Choi B (2010) A grouping-based strategy to improve the effectiveness of fault localization techniques. In: 2010 10th International conference on quality software, pp 13\u201322. https:\/\/doi.org\/10.1109\/QSIC.2010.80","DOI":"10.1109\/QSIC.2010.80"},{"key":"10475_CR11","doi-asserted-by":"crossref","unstructured":"Derezi\u0144ska A, Ha\u0142as K (2014) Analysis of mutation operators for the Python language. In: Zamojski W, Mazurkiewicz J, Sugier J, Walkowiak T, Kacprzyk J (eds) Proceedings of the 9th international conference on dependability and complex systems, pp 155\u2013164","DOI":"10.1007\/978-3-319-07013-1_15"},{"issue":"4","key":"10475_CR12","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1007\/s10664-005-3861-2","volume":"10","author":"H Do","year":"2005","unstructured":"Do H, Elbaum S, Rothermel G (2005) Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empir Softw Eng 10(4):405\u2013435. https:\/\/doi.org\/10.1007\/s10664-005-3861-2","journal-title":"Empir Softw Eng"},{"key":"10475_CR13","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1007\/978-3-030-16722-6_10","volume-title":"Fundamental approaches to software engineering","author":"HF Eniser","year":"2019","unstructured":"Eniser HF, Gerasimou S, Sen A (2019) DeepFault: Fault localization for deep neural networks. In: H\u00e4hnle R, van der Aalst W (eds) Fundamental approaches to software engineering. Springer International Publishing, pp 171\u2013191"},{"issue":"9","key":"10475_CR14","first-page":"1786","volume":"47","author":"CA Furia","year":"2021","unstructured":"Furia CA, Robert Feldt, Richard Torkar (2021) Bayesian data analysis in empirical software engineering research. IEEE Trans Softw Eng 47(9):1786\u20131810","journal-title":"IEEE Trans Softw Eng"},{"key":"10475_CR15","doi-asserted-by":"crossref","unstructured":"Furia CA, Torkar R, Feldt R (2022) Applying Bayesian analysis guidelines to empirical software engineering data: The case of programming languages and code quality. ACM Trans Softw Eng Methodol 31(3):40:1-40:38","DOI":"10.1145\/3490953"},{"key":"10475_CR16","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/TSE.2017.2755013","volume":"45","author":"L Gazzola","year":"2019","unstructured":"Gazzola L, Micucci D, Mariani L (2019) Automatic software repair: A survey. IEEE Trans Softw Eng 45:34\u201367. https:\/\/doi.org\/10.1109\/TSE.2017.2755013","journal-title":"IEEE Trans Softw Eng"},{"key":"10475_CR17","doi-asserted-by":"publisher","unstructured":"Guo Q, Xie X,\u00a0Li Y, Zhang X, Liu Y, Li X, Shen C (2020) Audee: Automated testing for deep learning frameworks. In: Proceedings of the 35th IEEE\/ACM international conference on automated software engineering, pp 486\u2013498. https:\/\/doi.org\/10.1145\/3324884.3416571","DOI":"10.1145\/3324884.3416571"},{"key":"10475_CR18","unstructured":"Hammacher C (2008) Design and implementation of an efficient dynamic slicer for Java. Bachelor\u2019s Thesis"},{"key":"10475_CR19","doi-asserted-by":"publisher","unstructured":"Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: ICSE \u201920: 42nd International conference on software engineering, Seoul, South Korea, 27 June - 19 July, 2020, pp 1110\u20131121. ACM. https:\/\/doi.org\/10.1145\/3377811.3380395. https:\/\/doi.org\/10.1145\/3377811.3380395","DOI":"10.1145\/3377811.3380395"},{"key":"10475_CR20","doi-asserted-by":"publisher","unstructured":"Hutchins M,\u00a0Foster H,\u00a0Goradia T,\u00a0Ostrand T (1994) Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria. In: Proceedings of 16th international conference on software engineering, pp 191\u2013200. https:\/\/doi.org\/10.1109\/ICSE.1994.296778","DOI":"10.1109\/ICSE.1994.296778"},{"key":"10475_CR21","doi-asserted-by":"publisher","unstructured":"Sarhan QI, Szatm\u00e1ri A, T\u00f3th R, Besz\u00e9des \u00c1 (2021) CharmFL: A fault localization tool for Python. In: IEEE 21st International working conference on source code analysis and manipulation (SCAM), pp 114\u2013119. https:\/\/doi.org\/10.1109\/SCAM52516.2021.00022","DOI":"10.1109\/SCAM52516.2021.00022"},{"key":"10475_CR22","doi-asserted-by":"publisher","unstructured":"Islam Md\u00a0J, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: Proceedings of the 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 510\u2013520. https:\/\/doi.org\/10.1145\/3338906.3338955","DOI":"10.1145\/3338906.3338955"},{"key":"10475_CR23","doi-asserted-by":"publisher","unstructured":"Islam Md\u00a0J, Pan R, Nguyen G, Rajan H (2020) Repairing deep neural networks: Fix patterns and challenges. In: Proceedings of the ACM\/IEEE 42nd international conference on software engineering, pp 1135\u20131146. https:\/\/doi.org\/10.1145\/3377811.3380378","DOI":"10.1145\/3377811.3380378"},{"key":"10475_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TIM.2022.3220286","volume":"71","author":"MdJ Islam","year":"2022","unstructured":"Islam MdJ, Ahmad S, Haque F, Reaz MBI, Bhuiyan MAS, Islam MdR (2022) Application of min-max normalization on subject-invariant emg pattern recognition. IEEE Trans Instrum Meas 71:1\u201312. https:\/\/doi.org\/10.1109\/TIM.2022.3220286","journal-title":"IEEE Trans Instrum Meas"},{"issue":"5","key":"10475_CR25","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1109\/TSE.2010.62","volume":"37","author":"Jia Yue","year":"2011","unstructured":"Yue Jia, Mark Harman (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37(5):649\u2013678. https:\/\/doi.org\/10.1109\/TSE.2010.62","journal-title":"IEEE Trans Softw Eng"},{"key":"10475_CR26","doi-asserted-by":"publisher","unstructured":"Jones JA, Harrold MJ (2005) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE\/ACM international conference on automated software engineering, pp 273\u2013282. https:\/\/doi.org\/10.1145\/1101908.1101949","DOI":"10.1145\/1101908.1101949"},{"key":"10475_CR27","doi-asserted-by":"publisher","unstructured":"Just R (2014) The major mutation framework: Efficient and scalable mutation analysis for java. In: Proceedings of the international symposium on software testing and analysis, pp 433\u2013436. https:\/\/doi.org\/10.1145\/2610384.2628053","DOI":"10.1145\/2610384.2628053"},{"key":"10475_CR28","doi-asserted-by":"publisher","unstructured":"Just R, Jalali D, Ernst MD (2014) Defects4j: A database of existing faults to enable controlled testing studies for java programs. In: Proceedings of the international symposium on software testing and analysis, pp 437\u2013440. https:\/\/doi.org\/10.1145\/2610384.2628055","DOI":"10.1145\/2610384.2628055"},{"key":"10475_CR29","unstructured":"Just R, Jalali D et al (2023) Defects4J contributors. Defects4J repository. https:\/\/github.com\/rjust\/defects4j#export-version-specific-properties"},{"key":"10475_CR30","doi-asserted-by":"publisher","unstructured":"Juzgado NJ, G\u00f3mez OS (2010) Replication of software engineering experiments. In: Empirical software engineering and verification \u2013 international summer schools, LASER 2008-2010, Elba Island, Italy, Revised Tutorial Lectures, vol 7007 of Lecture Notes in Computer Science, pp 60\u201388. Springer. https:\/\/doi.org\/10.1007\/978-3-642-25231-0_2","DOI":"10.1007\/978-3-642-25231-0_2"},{"key":"10475_CR31","doi-asserted-by":"publisher","unstructured":"Ko AJ, Myers BA (2008) Debugging reinvented: asking and answering why and why not questions about program behavior. In: Sch\u00e4fer W, Dwyer MB, Gruhn V (eds) 30th International conference on software engineering (ICSE 2008), Leipzig, Germany, May 10-18, 2008, pp 301\u2013310. ACM. https:\/\/doi.org\/10.1145\/1368088.1368130. https:\/\/doi.org\/10.1145\/1368088.1368130","DOI":"10.1145\/1368088.1368130"},{"key":"10475_CR32","unstructured":"Lam P, Bodden E, Lhot\u00e1k O, Hendren L (2011) The Soot framework for Java program analysis: a retrospective. In: Cetus users and compiler infrastructure workshop (CETUS 2011). https:\/\/www.bodden.de\/pubs\/lblh11soot.pdf"},{"key":"10475_CR33","doi-asserted-by":"publisher","unstructured":"B. Le T-D, Thung F, Lo D (2013) Theory and practice, do they match? a case with spectrum-based fault localization. In: IEEE international conference on software maintenance, pp 380\u2013383. https:\/\/doi.org\/10.1109\/ICSM.2013.52","DOI":"10.1109\/ICSM.2013.52"},{"key":"10475_CR34","unstructured":"Li L, Wang J, Quan H (2022) Scalpel: The Python static analysis framework. Preprint arXiv:2202.11840"},{"key":"10475_CR35","doi-asserted-by":"publisher","unstructured":"Li X, Zhang L (2017) Transforming programs and tests in tandem for fault localization. Proc. ACM Program. Lang., 1 (OOPSLA), pp 1\u201330. https:\/\/doi.org\/10.1145\/3133916. Replication package: https:\/\/github.com\/deeprl4fl2021icse\/deeprl4fl-2021-icse","DOI":"10.1145\/3133916"},{"key":"10475_CR36","doi-asserted-by":"publisher","unstructured":"Li X, Li W, Zhang Y, Zhang L (2019) DeepFL: Integrating multiple fault diagnosis dimensions for deep fault localization. In: Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, pp 169\u2013180. https:\/\/doi.org\/10.1145\/3293882.3330574","DOI":"10.1145\/3293882.3330574"},{"key":"10475_CR37","doi-asserted-by":"publisher","unstructured":"Li Y, Wang S, Nguyen TN (2021) Fault localization with code coverage representation learning. In: Proceedings of the 43rd international conference on software engineering, pp 661\u2013673. https:\/\/doi.org\/10.1109\/ICSE43902.2021.00067","DOI":"10.1109\/ICSE43902.2021.00067"},{"key":"10475_CR38","doi-asserted-by":"publisher","unstructured":"Lou Y, Zhu Q, Dong J, Li X, Sun Z, Hao D,\u00a0Zhang L, Zhang L (2021) Boosting coverage-based fault localization via graph-based representation learning. In: Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 664\u2013676. https:\/\/doi.org\/10.1145\/3468264.3468580","DOI":"10.1145\/3468264.3468580"},{"issue":"2","key":"10475_CR39","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1002\/stvr.308","volume":"15","author":"Y-S Ma","year":"2005","unstructured":"Ma Y-S, Offutt J, Kwon YR (2005) MuJava: an automated class mutation system. Softw Test Verif Reliab 15(2):97\u2013133. https:\/\/doi.org\/10.1002\/stvr.308","journal-title":"Softw Test Verif Reliab"},{"key":"10475_CR40","doi-asserted-by":"publisher","unstructured":"MacIver D, Hatfield-Dodds Z, Contributors M (2019) Hypothesis: A new approach to property-based testing. J Open Source Softw 4(43):1891\u20131893. https:\/\/doi.org\/10.21105\/joss.01891https:\/\/doi.org\/10.21105\/joss.01891","DOI":"10.21105\/joss.01891"},{"key":"10475_CR41","unstructured":"McConnell S (2004) Code Complete, 2nd edn. Microsoft Press"},{"key":"10475_CR42","doi-asserted-by":"publisher","unstructured":"Moon S, Kim Y, Kim M, Yoo S (2014) Ask the mutants: Mutating faulty programs for fault localization. In: IEEE seventh international conference on software testing, verification and validation, pp 153\u2013162. https:\/\/doi.org\/10.1109\/ICST.2014.28","DOI":"10.1109\/ICST.2014.28"},{"key":"10475_CR43","doi-asserted-by":"publisher","unstructured":"Mukherjee S, Almanza A, Rubio-Gonz\u00e1lez C (2021) Fixing dependency errors for Python build reproducibility. In: Proceedings of the 30th ACM SIGSOFT international symposium on software testing and analysis, pp 439\u2013451. https:\/\/doi.org\/10.1145\/3460319.3464797","DOI":"10.1145\/3460319.3464797"},{"key":"10475_CR44","doi-asserted-by":"publisher","unstructured":"Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C (1996) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol 5(2):99\u2013118. https:\/\/doi.org\/10.1145\/227607.227610. ISSN 1049-331X","DOI":"10.1145\/227607.227610"},{"issue":"5\u20137","key":"10475_CR45","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1002\/stvr.1509","volume":"25","author":"Papadakis Mike","year":"2015","unstructured":"Mike Papadakis, Yves Le Traon (2015) Metallaxis-fl: Mutation-based fault localization. Softw Test Verif Reliab 25(5\u20137):605\u2013628. https:\/\/doi.org\/10.1002\/stvr.1509","journal-title":"Softw Test Verif Reliab"},{"key":"10475_CR46","doi-asserted-by":"publisher","unstructured":"Papadakis M, Kintis M, Zhang J, Jia Y, Traon YL, Harman M (2019) Chapter six - mutation testing advances: An analysis and survey. Vol 112 of Advances in Computers, pp 275\u2013378. https:\/\/doi.org\/10.1016\/bs.adcom.2018.03.015","DOI":"10.1016\/bs.adcom.2018.03.015"},{"key":"10475_CR47","doi-asserted-by":"publisher","unstructured":"Parnin C, Orso A (2011a) Are automated debugging techniques actually helping programmers? In: Proceedings of the 2011 international symposium on software testing and analysis, pp 199\u2013209. https:\/\/doi.org\/10.1145\/2001420.2001445","DOI":"10.1145\/2001420.2001445"},{"key":"10475_CR48","doi-asserted-by":"publisher","unstructured":"Parnin C, Orso A (2011b) Are automated debugging techniques actually helping programmers? In: Dwyer MB, Tip F (eds) Proceedings of the 20th international symposium on software testing and analysis, ISSTA 2011, Toronto, ON, Canada, July 17-21, 2011, pp 199\u2013209. ACM. https:\/\/doi.org\/10.1145\/2001420.2001445. https:\/\/doi.org\/10.1145\/2001420.2001445","DOI":"10.1145\/2001420.2001445"},{"key":"10475_CR49","doi-asserted-by":"publisher","unstructured":"Pearson S, Campos J, Just R, Fraser G, Abreu R, Ernst MD, Pang D, Keller B (2017) Evaluating and improving fault localization. In: Proceedings of the 39th International conference on software engineering, pp 609\u2013620. https:\/\/doi.org\/10.1109\/ICSE.2017.62","DOI":"10.1109\/ICSE.2017.62"},{"key":"10475_CR50","doi-asserted-by":"publisher","unstructured":"Rahman F, Posnett D, Hindle A, Barr E, Devanbu P (2011) Bugcache for inspections: Hit or miss? In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on foundations of software engineering, pp 322\u2013331. https:\/\/doi.org\/10.1145\/2025113.2025157","DOI":"10.1145\/2025113.2025157"},{"key":"10475_CR51","doi-asserted-by":"publisher","unstructured":"Renieres M, Reiss SP (2003) Fault localization with nearest neighbor queries. In: Proceedings of 18th IEEE international conference on automated software engineering, pp 30\u201339. https:\/\/doi.org\/10.1109\/ASE.2003.1240292","DOI":"10.1109\/ASE.2003.1240292"},{"key":"10475_CR52","doi-asserted-by":"publisher","unstructured":"Reps T, Ball T, Das M, Larus J (1997) The use of program profiling for software maintenance with applications to the year 2000 problem. In: Proceedings of the 6th European software engineering conference held jointly with the 5th ACM SIGSOFT international symposium on foundations of software engineering, pp 432\u2013449. https:\/\/doi.org\/10.1145\/267895.267925","DOI":"10.1145\/267895.267925"},{"key":"10475_CR53","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111669","volume":"201","author":"M Rezaalipour","year":"2023","unstructured":"Rezaalipour M, Furia CA (2023) An annotation-based approach for finding bugs in neural network programs. J Syst Softw 201:111669","journal-title":"J Syst Softw"},{"key":"10475_CR54","unstructured":"Romano J, Kromrey JD,\u00a0Coraggio J,\u00a0Skowronek J (2006) Should we really be using $$t$$-test and Cohen\u2019s $$d$$ for evaluating group differences on the NSSE and other surveys? In: Annual meeting of the Florida Association of Institutional Research"},{"key":"10475_CR55","doi-asserted-by":"publisher","unstructured":"Saha RK, Lease M, Khurshid S, and Perry DE (2013) Improving bug localization using structured information retrieval. In: 28th IEEE\/ACM international conference on automated software engineering (ASE), pp 345\u2013355. https:\/\/doi.org\/10.1109\/ASE.2013.6693093","DOI":"10.1109\/ASE.2013.6693093"},{"key":"10475_CR56","doi-asserted-by":"publisher","unstructured":"Salis V, Sotiropoulos T, Louridas P, Spinellis D, Mitropoulos D (2021) PyCG: Practical call graph generation in Python. In: Proceedings of the 43rd international conference on software engineering, pp 1646\u20131657. https:\/\/doi.org\/10.1109\/ICSE43902.2021.00146","DOI":"10.1109\/ICSE43902.2021.00146"},{"key":"10475_CR57","doi-asserted-by":"publisher","first-page":"10618","DOI":"10.1109\/ACCESS.2022.3144079","volume":"10","author":"Qusay Idrees Sarhan and \u00c1rp\u00e1d Besz\u00e9des","year":"2022","unstructured":"Qusay Idrees Sarhan and \u00c1rp\u00e1d Besz\u00e9des (2022) A survey of challenges in spectrum-based software fault localization. IEEE Access 10:10618\u201310639. https:\/\/doi.org\/10.1109\/ACCESS.2022.3144079","journal-title":"IEEE Access"},{"key":"10475_CR58","doi-asserted-by":"publisher","unstructured":"Schoop E, Huang F, Hartmann B (2021) Umlaut: Debugging deep learning programs using program structure and model behavior. In: Proceedings of the 2021 CHI conference on human factors in computing systems. https:\/\/doi.org\/10.1145\/3411764.3445538","DOI":"10.1145\/3411764.3445538"},{"key":"10475_CR59","doi-asserted-by":"publisher","unstructured":"Schroter A, Schr\u00f6ter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs? In: 2010 7th IEEE working conference on mining software repositories (MSR 2010), pp 118\u2013121. https:\/\/doi.org\/10.1109\/MSR.2010.5463280","DOI":"10.1109\/MSR.2010.5463280"},{"key":"10475_CR60","doi-asserted-by":"publisher","unstructured":"Sobreira V, Durieux T, Madeiral F, Monperrus M, de\u00a0Almeida\u00a0Maia M (2018) Dissection of a bug dataset: Anatomy of 395 patches from Defects4J. In: Oliveto R,\u00a0Di Penta M, Shepherd DC (eds) 25th International Conference on Software Analysis, Evolution and Reengineering, SANER 2018, Campobasso, Italy, March 20-23, 2018, pp 130\u2013140. IEEE Computer Society, 2018. https:\/\/doi.org\/10.1109\/SANER.2018.8330203. https:\/\/doi.org\/10.1109\/SANER.2018.8330203","DOI":"10.1109\/SANER.2018.8330203"},{"key":"10475_CR61","doi-asserted-by":"publisher","unstructured":"Sohn J, Yoo S (2017) FLUCCS: Using code and change metrics to improve fault localization. In: Proceedings of the 26th ACM SIGSOFT international symposium on software testing and analysis, pp 273\u2013283. https:\/\/doi.org\/10.1145\/3092703.3092717","DOI":"10.1145\/3092703.3092717"},{"key":"10475_CR62","doi-asserted-by":"publisher","unstructured":"Steimann F, Frenkel M, Abreu R (2013) Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators. In: Proceedings of the international symposium on software testing and analysis, pp 314\u2013324. https:\/\/doi.org\/10.1145\/2483760.2483767","DOI":"10.1145\/2483760.2483767"},{"key":"10475_CR63","doi-asserted-by":"publisher","unstructured":"Szatm\u00e1ri A, Sarhan QI, Besz\u00e9des \u00c1 (2022) Interactive fault localization for Python with CharmFL. In: Proceedings of the 13th international workshop on automating test case design, selection and evaluation, pp 33\u201336. https:\/\/doi.org\/10.1145\/3548659.3561312","DOI":"10.1145\/3548659.3561312"},{"key":"10475_CR64","unstructured":"Vall\u00e9e-Rai R, Co P, Gagnon E, Hendren LJ, Lam P, Sundaresan V (1999) Soot \u2013 a Java bytecode optimization framework. In: MacKay SA, Johnson JH (eds) Proceedings of the 1999 conference of the Centre for Advanced Studies on Collaborative Research, November 8-11, 1999, Mississauga, Ontario, Canada, pp\u00a013. IBM. https:\/\/dl.acm.org\/citation.cfm?id=782008"},{"key":"10475_CR65","doi-asserted-by":"crossref","unstructured":"Wardat M, Le W, Rajan H (2021) Deeplocalize: Fault localization for deep neural networks. In: ICSE\u201921: The 43nd international conference on software engineering","DOI":"10.1109\/ICSE43902.2021.00034"},{"key":"10475_CR66","doi-asserted-by":"crossref","unstructured":"Wasserstein RL, Lazar NA (2016) The ASA statement on $$p$$-values: Context, process, and purpose. Am Stat 70(2):129\u2013133. https:\/\/www.amstat.org\/asa\/files\/pdfs\/P-ValueStatement.pdf","DOI":"10.1080\/00031305.2016.1154108"},{"key":"10475_CR67","doi-asserted-by":"publisher","unstructured":"Widyasari R, Sim SQ, Lok C, Qi H, Phan J, Tay Q, Tan C, Wee F, Tan JE, Yieh Y, Goh B, Thung F, Kang HJ, Hoang T, Lo D, Ouh EL (2020) BugsInPy: A database of existing bugs in Python programs to enable controlled testing and debugging studies. In: Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 1556\u20131560. https:\/\/doi.org\/10.1145\/3368089.3417943","DOI":"10.1145\/3368089.3417943"},{"key":"10475_CR68","doi-asserted-by":"publisher","unstructured":"Widyasari R, Azriadi Prana GA, Haryono SA, Wang S, Lo D (2022) Real world projects, real faults: Evaluating spectrum based fault localization techniques on Python projects. Empir Softw Eng 27(6). https:\/\/doi.org\/10.1007\/s10664-022-10189-4","DOI":"10.1007\/s10664-022-10189-4"},{"key":"10475_CR69","doi-asserted-by":"publisher","unstructured":"Wong E, Wei T,\u00a0Qi Y, Zhao L (2008) A crostab-based statistical method for effective fault localization. In: 1st International conference on software testing, verification, and validation, pp 42\u201351. https:\/\/doi.org\/10.1109\/ICST.2008.65","DOI":"10.1109\/ICST.2008.65"},{"issue":"1","key":"10475_CR70","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1109\/TR.2013.2285319","volume":"63","author":"WE Wong","year":"2014","unstructured":"Wong WE, Debroy V, Gao R, Li Y (2014) The DStar method for effective software fault localization. IEEE Trans Reliab 63(1):290\u2013308. https:\/\/doi.org\/10.1109\/TR.2013.2285319","journal-title":"IEEE Trans Reliab"},{"issue":"8","key":"10475_CR71","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1109\/TSE.2016.2521368","volume":"42","author":"WE Wong","year":"2016","unstructured":"Wong WE, Gao R, Li Y, Abreu R, Wotawa F (2016) A survey on software fault localization. IEEE Trans Softw Eng 42(8):707\u2013740. https:\/\/doi.org\/10.1109\/TSE.2016.2521368","journal-title":"IEEE Trans Softw Eng"},{"key":"10475_CR72","doi-asserted-by":"publisher","unstructured":"Xuan J, Monperrus M (2014) Learning to combine multiple ranking metrics for fault localization. In: 2014 IEEE international conference on software maintenance and evolution, pp 191\u2013200. https:\/\/doi.org\/10.1109\/ICSME.2014.41","DOI":"10.1109\/ICSME.2014.41"},{"key":"10475_CR73","unstructured":"Zeller A (2009) Why Programs Fail \u2013 A Guide to Systematic Debugging, 2nd edn. Academic Press. http:\/\/store.elsevier.com\/product.jsp?isbn=9780123745156 &pagename=search. ISBN 978-0-12-374515-6"},{"key":"10475_CR74","doi-asserted-by":"publisher","unstructured":"Zhang X,\u00a0Gupta R,\u00a0Gupta N (2006) Locating faults through automated predicate switching. In: Software engineering, international conference on, pp 272\u2013281. https:\/\/doi.org\/10.1145\/1134285.1134324","DOI":"10.1145\/1134285.1134324"},{"key":"10475_CR75","doi-asserted-by":"publisher","unstructured":"Zhang X, Zhai J, Ma S, Shen C (2021) Autotrainer: An automatic dnn training problem detection and repair system. In: 2021 IEEE\/ACM 43rd international conference on software engineering (ICSE), pp 359\u2013371. https:\/\/doi.org\/10.1109\/ICSE43902.2021.00043","DOI":"10.1109\/ICSE43902.2021.00043"},{"key":"10475_CR76","doi-asserted-by":"publisher","unstructured":"Zhang Y, Ren L, Chen L, Xiong Y, Cheung S-C, Xie T (2020) Detecting numerical bugs in neural network architectures. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 826\u2013837. https:\/\/doi.org\/10.1145\/3368089.3409720","DOI":"10.1145\/3368089.3409720"},{"key":"10475_CR77","doi-asserted-by":"publisher","unstructured":"Zhou J, Zhang H, Lo D (2012) Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports. In: 34th International conference on software engineering (ICSE), pp 14\u201324. https:\/\/doi.org\/10.1109\/ICSE.2012.6227210","DOI":"10.1109\/ICSE.2012.6227210"},{"issue":"2","key":"10475_CR78","doi-asserted-by":"publisher","first-page":"332","DOI":"10.1109\/TSE.2019.2892102","volume":"47","author":"D Zou","year":"2021","unstructured":"Zou D, Liang J, Xiong Y, Ernst MD, Zhang L (2021) An empirical study of fault localization families and their combinations. IEEE Trans Softw Eng 47(2):332\u2013347. https:\/\/doi.org\/10.1109\/TSE.2019.2892102","journal-title":"IEEE Trans Softw Eng"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10475-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-024-10475-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10475-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,5]],"date-time":"2024-07-05T11:19:51Z","timestamp":1720178391000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-024-10475-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,13]]},"references-count":78,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,7]]}},"alternative-id":["10475"],"URL":"https:\/\/doi.org\/10.1007\/s10664-024-10475-3","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,13]]},"assertion":[{"value":"11 March 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 June 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that there are no conflicts of interest regarding the publication of this paper","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of Interest"}},{"value":"The authors declare that they have no competing interests that are related to the work described in this paper.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interest"}}],"article-number":"92"}}