{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T21:43:56Z","timestamp":1774129436834,"version":"3.50.1"},"reference-count":130,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,4,27]],"date-time":"2023-04-27T00:00:00Z","timestamp":1682553600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61872445 and 62272072"],"award-info":[{"award-number":["61872445 and 62272072"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Major Key Project of PCL","award":["PCL2021A06"],"award-info":[{"award-number":["PCL2021A06"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>Evaluation is the foundation of automated program repair (APR), as it provides empirical evidence on strengths and weaknesses of APR techniques. However, the reliability of such evaluation is often threatened by various introduced biases. Consequently, bias exploration, which uncovers biases in the APR evaluation, has become a pivotal activity and performed since the early years when pioneer APR techniques were proposed. Unfortunately, there is still no methodology to support a systematic comprehension and discovery of evaluation biases in APR, which impedes the mitigation of such biases and threatens the evaluation of APR techniques.<\/jats:p>\n          <jats:p>\n            In this work, we propose to systematically understand existing evaluation biases by rigorously conducting the first systematic literature review on existing known biases and systematically uncover new biases by building a taxonomy that categorizes evaluation biases. As a result, we identify 17 investigated biases and uncover a new bias in the usage of patch validation strategies. To validate this new bias, we devise and implement an executable framework\n            <jats:sc>APRConfig<\/jats:sc>\n            , based on which we evaluate three typical patch validation strategies with four representative heuristic-based and constraint-based APR techniques on three bug datasets. Overall, this article distills 13 findings for bias understanding, discovery, and validation. The systematic exploration we performed and the open source executable framework we proposed in this article provide new insights as well as an infrastructure for future exploration and mitigation of biases in APR evaluation.\n          <\/jats:p>","DOI":"10.1145\/3561382","type":"journal-article","created":{"date-parts":[[2022,9,4]],"date-time":"2022-09-04T10:45:42Z","timestamp":1662288342000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Seeing the Whole Elephant: Systematically Understanding and Uncovering Evaluation Biases in Automated Program Repair"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8383-1939","authenticated-orcid":false,"given":"Deheng","family":"Yang","sequence":"first","affiliation":[{"name":"National University of Defense Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4504-6806","authenticated-orcid":false,"given":"Yan","family":"Lei","sequence":"additional","affiliation":[{"name":"Chongqing University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4204-7424","authenticated-orcid":false,"given":"Xiaoguang","family":"Mao","sequence":"additional","affiliation":[{"name":"National University of Defense Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1246-3833","authenticated-orcid":false,"given":"Yuhua","family":"Qi","sequence":"additional","affiliation":[{"name":"Beijing Institute of Tracking and Communication Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4511-1495","authenticated-orcid":false,"given":"Xin","family":"Yi","sequence":"additional","affiliation":[{"name":"National University of Defense Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,4,27]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2006.18"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.5555\/1308173.1308264"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360585"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635898"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338952"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00035"},{"key":"e_1_3_2_9_2","doi-asserted-by":"crossref","unstructured":"Zhiqiang Bian Aymeric Blot and Justyna Petke. 2021. Refining fitness functions for search-based program repair. In Proceedings of the IEEE\/ACM International Workshop on Automated Program Repair (APR\u201921) .","DOI":"10.1109\/APR52552.2021.00008"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/1595696.1595716"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2628058"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155642"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2970009"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780190463908.001.0001"},{"key":"e_1_3_2_15_2","unstructured":"Maxime Cordy Renaud Rwemalika Mike Papadakis and Mark Harman. 2019. Flakime: Laboratory-controlled test flakiness impact assessment. a case study on mutation testing and program repair. CoRR abs\/1912.03197 (2019). http:\/\/arxiv.org\/abs\/1912.03197."},{"key":"e_1_3_2_16_2","unstructured":"Benoit Cornu Thomas Durieux Lionel Seinturier and Martin Monperrus. 2015. Npefix: Automatic runtime repair of null pointer exceptions in java. CoRR abs\/1512.07423 (2015). http:\/\/arxiv.org\/abs\/1512.07423."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.5555\/1792734.1792766"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3131151.3131170"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2017.7884635"},{"key":"e_1_3_2_20_2","unstructured":"Thomas Durieux Benjamin Danglot Zhongxing Yu Matias Martinez Simon Urli and Martin Monperrus. 2017. The patches of the nopol automatic repair system on the bugs of defects4j version 1.1. 0. Research Report. hal-01480084. Universit\u00e9 Lille 1 - Sciences et Technologies."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338911"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2896921.2896931"},{"key":"e_1_3_2_23_2","unstructured":"Thomas Durieux and Martin Monperrus. 2016. IntroClassJava: A benchmark of 297 small and buggy Java programs. Research Report. hal-01272126. Universite Lille 1. https:\/\/hal.archives-ouvertes.fr\/hal-01272126\/file\/main.pdf."},{"key":"e_1_3_2_24_2","unstructured":"Marvin Fleischmann Miglena Amirpur Alexander Benlian and Thomas Hess. 2014. Cognitive biases in information systems research: A scientometric analysis. In 22st European Conference on Information Systems ECIS 2014 Tel Aviv Israel June 9-11 2014 Michel Avital Jan Marco Leimeister and Ulrike Schultze (Eds.). http:\/\/aisel.aisnet.org\/ecis2014\/proceedings\/track02\/5."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/1463788.1463807"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"Paul Galdas. 2017. Revisiting bias in qualitative research: Reflections on its relationship with funding and impact. International Journal of Qualitative Methods 16 1 (2017) 1\u20132.","DOI":"10.1177\/1609406917748992"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2755013"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330559"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3318162"},{"key":"e_1_3_2_30_2","volume-title":"Proceedings of the IEEE 1st International Workshop on Intelligent Bug Fixing (IBF\u201919)","unstructured":"He, Ye, Matias, Martinez, Thomas, Durieux, Martin, and Monperrus. 2019. A comprehensive study of automatic program repair on the QuixBugs benchmark. In Proceedings of the IEEE 1st International Workshop on Intelligent Bug Fixing (IBF\u201919)."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110825"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180245"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00033"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213871"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00107"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2628055"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3067156"},{"key":"e_1_3_2_39_2","volume-title":"Guidelines for Performing Systematic Literature Reviews in Software Engineering","author":"Keele Staffs","year":"2007","unstructured":"Staffs Keele et\u00a0al. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical Report. Citeseer."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606626"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09742-5"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2010.03.006"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642997"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2018.2865733"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2015.7381813"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09780-z"},{"key":"e_1_3_2_48_2","unstructured":"Barbara H. Kwasnik. 1999. The role of classification in knowledge representation and discovery. Libr. Trends 48 1 (1999). http:\/\/alexia.lis.uiuc.edu\/puboff\/catalog\/trends\/48_1abs.html#kwasnik."},{"key":"e_1_3_2_49_2","unstructured":"Ryan Lawler. 2012. How do you hire great engineers? Give them a challenge. https:\/\/gigaom.com\/2012\/01\/19\/quixey-challenge\/."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00064"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2016.76"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227211"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2454513"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.104"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.04.001"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380345"},{"issue":"3","key":"e_1_3_2_57_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3505247","article-title":"Context-aware code change embedding for better patch correctness assessment","volume":"31","author":"Lin Bo","year":"2022","unstructured":"Bo Lin, Shangwen Wang, Ming Wen, and Xiaoguang Mao. 2022. Context-aware code change embedding for better patch correctness assessment. ACM Trans. Softw. Eng. Methodol. 31, 3 (2022), 1\u201329.","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC51365.2020.00015"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3135932.3135941"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2019.00020"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8667970"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330577"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2018.00085"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110817"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380338"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2018.8330202"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09715-8"},{"key":"e_1_3_2_68_2","unstructured":"Giuliano Lorenzoni Paulo Alencar Nathalia Nascimento and Donald Cowan. 2021. Machine learning model development from a software engineering perspective: A systematic literature review. CoRR abs\/2102.07574 (2021). https:\/\/arxiv.org\/abs\/2102.07574."},{"key":"e_1_3_2_69_2","unstructured":"Yiling Lou Samuel Benton Dan Hao Lu Zhang and Lingming Zhang. 2021. How does regression test selection affect program repair? An extensive study on 2 million patches. CoRR abs\/2105.07311 (2021). https:\/\/arxiv.org\/abs\/2105.07311."},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397369"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8667991"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cola.2019.03.006"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00039"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9282-8"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2948705"},{"key":"e_1_3_2_76_2","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/978-3-319-99241-9_3","volume-title":"International Symposium on Search Based Software Engineering","author":"Martinez Matias","year":"2018","unstructured":"Matias Martinez and Martin Monperrus. 2018. Ultra-large repair search space with automatically mined templates: The cardumen mode of astor. In International Symposium on Search Based Software Engineering. Springer, 65\u201386."},{"key":"e_1_3_2_77_2","volume-title":"The Pyramid Principle: Logic in Writing and Thinking","author":"Minto Barbara","year":"2009","unstructured":"Barbara Minto. 2009. The Pyramid Principle: Logic in Writing and Thinking. Pearson Education."},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2877759"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568324"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3105906"},{"key":"e_1_3_2_81_2","unstructured":"Martin Monperrus. 2020. The living review on automated program repair. Technical Report. hal-01956501. HAL Archives Ouvertes. https:\/\/hal.archives-ouvertes.fr\/hal-01956501v4\/file\/repair-living-review.pdf."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3349589"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9550-0"},{"key":"e_1_3_2_84_2","volume-title":"IEEE Standard Taxonomy for Software Engineering Standards","author":"Electrical Institute of","year":"1987","unstructured":"Institute of Electrical and Electronics Engineers. 1987. IEEE Standard Taxonomy for Software Engineering Standards."},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.62"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICISCE.2018.00059"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568254"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/2483760.2483785"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771791"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER50967.2021.00035"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-019-9947-6"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196473"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183440.3183487"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00020"},{"issue":"1","key":"e_1_3_2_95_2","first-page":"1","article-title":"Automated selection and quality assessment of primary studies: A systematic literature review","volume":"12","author":"Shakeel Yusra","year":"2019","unstructured":"Yusra Shakeel, Jacob Kr\u00fcger, Ivonne Von Nostitz-Wallwitz, Gunter Saake, and Thomas Leich. 2019. Automated selection and quality assessment of primary studies: A systematic literature review. J. Data Inf. Qual. 12, 1 (2019), 1\u201326.","journal-title":"J. Data Inf. Qual."},{"key":"e_1_3_2_96_2","unstructured":"Andr\u00e9 Silva Matias Martinez Benjamin Danglot Davide Ginelli and Martin Monperrus. 2021. FLACOCO: Fault localization for Java based on industry-grade coverage. CoRR abs\/2111.12513 (2021). https:\/\/arxiv.org\/abs\/2111.12513."},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-012-9217-9"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786825"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1136\/eb-2014-101946"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/203241.203256"},{"key":"e_1_3_2_101_2","first-page":"180","volume-title":"Proceedings of the IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C\u201917)","author":"Tan Shin Hwei","year":"2017","unstructured":"Shin Hwei Tan, Jooyong Yi, Sergey Mechtaev, Abhik Roychoudhury, et\u00a0al. 2017. Codeflaws: A programming competition benchmark for evaluating automated program repair tools. In Proceedings of the IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C\u201917). IEEE, 180\u2013182."},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635873"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/2523088"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.01.006"},{"issue":"2","key":"e_1_3_2_105_2","first-page":"101","article-title":"A critique and improvement of the CL common language effect size statistics of McGraw and Wong","volume":"25","author":"Vargha Andr\u00e1s","year":"2000","unstructured":"Andr\u00e1s Vargha and Harold D. Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J. Educ. Behav. Stat. 25, 2 (2000), 101\u2013132.","journal-title":"J. Educ. Behav. Stat."},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416590"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.1145\/3319008.3319349"},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070536"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180233"},{"key":"e_1_3_2_110_2","unstructured":"George R. Wheaton. 1968. Development of a taxonomy of human performance: A review of classificatory systems relating to tasks and performance. https:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.973.125&rep=rep1&type=pdf."},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4380-9_16"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2521368"},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155644"},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.45"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2987862"},{"key":"e_1_3_2_117_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00063"},{"key":"e_1_3_2_118_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2560811"},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2978819"},{"key":"e_1_3_2_120_2","unstructured":"Deheng Yang. 2022. Artifact Page of Our Study. Retrieved from https:\/\/github.com\/DehengYang\/APRConfig 2021."},{"key":"e_1_3_2_121_2","unstructured":"Deheng Yang. 2022. An Extended Description of the 17 Known Biases. Retrieved from https:\/\/github.com\/DehengYang\/APRConfig\/blob\/master\/doc\/RQ1.3_bias_mitigation\/detailed_explanation_of_the_17_known_biases.md."},{"key":"e_1_3_2_122_2","unstructured":"Deheng Yang. 2022. The Guideline on How to Extend APRConfig. Retrieved from https:\/\/github.com\/DehengYang\/APRConfig\/blob\/master\/How_to_extend.md."},{"key":"e_1_3_2_123_2","unstructured":"Deheng Yang. 2022. The Results of Our Investigation on Known Bias Mitigation. Retrieved from https:\/\/github.com\/DehengYang\/APRConfig\/blob\/master\/doc\/RQ1.3_bias_mitigation\/results_of_investigation_on_known_bias_mitigation.md."},{"key":"e_1_3_2_124_2","unstructured":"Deheng Yang. 2022. The Results of Quality Assessment. Retrieved from https:\/\/github.com\/DehengYang\/APRConfig\/blob\/master\/doc\/SLR_results\/results_of_quality_assessment.md."},{"key":"e_1_3_2_125_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER50967.2021.00018"},{"key":"e_1_3_2_126_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09920-w"},{"key":"e_1_3_2_127_2","unstructured":"He Ye Matias Martinez and Martin Monperrus. 2021. Neural program repair with execution-based backpropagation. CoRR abs\/2105.04123 (2021). https:\/\/arxiv.org\/abs\/2105.04123."},{"key":"e_1_3_2_128_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510222"},{"key":"e_1_3_2_129_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2874648"},{"key":"e_1_3_2_130_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360004"},{"key":"e_1_3_2_131_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00129"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3561382","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3561382","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:35Z","timestamp":1750186835000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3561382"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,27]]},"references-count":130,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3561382"],"URL":"https:\/\/doi.org\/10.1145\/3561382","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,27]]},"assertion":[{"value":"2021-12-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-08-26","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}