{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T16:27:11Z","timestamp":1781022431933,"version":"3.54.1"},"reference-count":93,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2022,8,22]],"date-time":"2022-08-22T00:00:00Z","timestamp":1661126400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Research Council"},{"name":"European Union\u2019s Horizon 2020","award":["949014"],"award-info":[{"award-number":["949014"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62172214"],"award-info":[{"award-number":["62172214"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"crossref","award":["BK20210279"],"award-info":[{"award-number":["BK20210279"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing","award":["2020A06"],"award-info":[{"award-number":["2020A06"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,10,31]]},"abstract":"<jats:p>\n            How do we know a generated patch is correct? This is a key challenging question that automated program repair (APR) systems struggle to address given the incompleteness of available test suites. Our intuition is that we can triage correct patches by checking whether each generated patch implements code changes (i.e., behavior) that are relevant to the bug it addresses. Such a bug is commonly specified by a failing test case. Towards predicting patch correctness in APR, we propose a novel yet simple hypothesis on how the link between the patch behavior and failing test specifications can be drawn:\n            <jats:italic>similar failing test cases should require similar patches<\/jats:italic>\n            . We then propose\n            <jats:sc>BATS<\/jats:sc>\n            , an unsupervised learning-based approach to predict patch correctness by checking patch\n            <jats:bold>B<\/jats:bold>\n            ehavior\n            <jats:bold>A<\/jats:bold>\n            gainst failing\n            <jats:bold>T<\/jats:bold>\n            est\n            <jats:bold>S<\/jats:bold>\n            pecification.\n            <jats:sc>BATS<\/jats:sc>\n            exploits deep representation learning models for code and patches: For a given failing test case, the yielded embedding is used to compute similarity metrics in the search for historical similar test cases to identify the associated applied patches, which are then used as a proxy for assessing the correctness of the APR-generated patches. Experimentally, we first validate our hypothesis by assessing whether ground-truth developer patches cluster together in the same way that their associated failing test cases are clustered. Then, after collecting a large dataset of 1,278 plausible patches (written by developers or generated by 32 APR tools), we use\n            <jats:sc>BATS<\/jats:sc>\n            to predict correct patches:\n            <jats:sc>BATS<\/jats:sc>\n            achieves AUC between 0.557 to 0.718 and recall between 0.562 and 0.854 in identifying correct patches. Our approach outperforms state-of-the-art techniques for identifying correct patches without the need for large labeled patch datasets\u2014as is the case with machine learning-based approaches. While\n            <jats:sc>BATS<\/jats:sc>\n            is constrained by the availability of similar test cases, we show that it can still be complementary to existing approaches: When combined with a recent approach that relies on supervised learning,\n            <jats:sc>BATS<\/jats:sc>\n            improves the overall recall in detecting correct patches. We finally show that\n            <jats:sc>BATS<\/jats:sc>\n            is complementary to the state-of-the-art PATCH-SIM dynamic approach for identifying correct patches generated by APR tools.\n          <\/jats:p>","DOI":"10.1145\/3511096","type":"journal-article","created":{"date-parts":[[2022,4,20]],"date-time":"2022-04-20T12:09:55Z","timestamp":1650456595000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":31,"title":["Predicting Patch Correctness Based on the Similarity of Failing Test Cases"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8049-3997","authenticated-orcid":false,"given":"Haoye","family":"Tian","sequence":"first","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yinghua","family":"Li","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weiguo","family":"Pian","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3151-9433","authenticated-orcid":false,"given":"Abdoul Kader","family":"Kabor\u00e9","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kui","family":"Liu","sequence":"additional","affiliation":[{"name":"Software Engineering Application Technology Lab, Huawei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5857-1864","authenticated-orcid":false,"given":"Andrew","family":"Habib","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacques","family":"Klein","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7270-9869","authenticated-orcid":false,"given":"Tegawend\u00e9 F.","family":"Bissyand\u00e9","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,8,22]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"code2seq: Generating sequences from structured representations of code","author":"Alon Uri","year":"2018","unstructured":"Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 (2018).","journal-title":"arXiv preprint arXiv:1808.01400"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290353"},{"key":"e_1_3_2_4_2","volume-title":"k-means++: The Advantages of Careful Seeding","author":"Arthur David","year":"2006","unstructured":"David Arthur and Sergei Vassilvitskii. 2006. k-means++: The Advantages of Careful Seeding. Technical Report. Stanford University."},{"key":"e_1_3_2_5_2","volume-title":"Automated Regression Testing and Verification of Complex Code Changes","author":"Boehme Marcel","year":"2014","unstructured":"Marcel Boehme. 2014. Automated Regression Testing and Verification of Complex Code Changes. Ph.D. Dissertation. National University of Singapore."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00036"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00104"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2017.8115674"},{"key":"e_1_3_2_9_2","article-title":"Sequencer: Sequence-to-sequence learning for end-to-end program repair","author":"Chen Zimin","year":"2019","unstructured":"Zimin Chen, Steve James Kommrusch, Michele Tufano, Louis-No\u00ebl Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2019. Sequencer: Sequence-to-sequence learning for end-to-end program repair. IEEE Trans. Softw. Eng. (2019).","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_3_2_10_2","article-title":"The remarkable role of similarity in redundancy-based program repair","author":"Chen Zimin","year":"2018","unstructured":"Zimin Chen and Martin Monperrus. 2018. The remarkable role of similarity in redundancy-based program repair. arXiv preprint arXiv:1811.05703 (2018).","journal-title":"arXiv preprint arXiv:1811.05703"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/IBF50092.2020.9034714"},{"key":"e_1_3_2_12_2","first-page":"4171","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171\u20134186."},{"key":"e_1_3_2_13_2","article-title":"Automatic repair of real bugs: An experience report on the Defects4J dataset","volume":"1505","author":"Durieux Thomas","year":"2015","unstructured":"Thomas Durieux, Matias Martinez, Martin Monperrus, Romain Sommerard, and Jifeng Xuan. 2015. Automatic repair of real bugs: An experience report on the Defects4J dataset. CoRR abs\/1505.07002 (2015).","journal-title":"CoRR"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/2896921.2896931"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397362"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978370"},{"key":"e_1_3_2_17_2","article-title":"Codebert: A pre-trained model for programming and natural languages","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et\u00a0al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020).","journal-title":"arXiv preprint arXiv:2002.08155"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3275524"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330558"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3392179"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2755013"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330559"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3318162"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.10742"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3446371"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380361"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180245"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER48275.2020.9054830"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931062"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00033"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213871"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00107"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2628055"},{"key":"e_1_3_2_34_2","volume-title":"Proceedings of the Text Mining Workshop at KDD2000","author":"Karypis Michael Steinbach, George","year":"2000","unstructured":"Michael Steinbach, George Karypis, Vipin Kumar, and Michael Steinbach. 2000. A comparison of document clustering techniques. In Proceedings of the Text Mining Workshop at KDD2000."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09742-5"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180187"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09780-z"},{"key":"e_1_3_2_38_2","first-page":"1188","volume-title":"Proceedings of the 31st International Conference on Machine Learning","author":"Le Quoc","year":"2014","unstructured":"Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning. JMLR.org, 1188\u20131196."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00064"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2016.66"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2016.76"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.104"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380345"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3480027"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2884955"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2019.00020"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8667970"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330577"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2018.00085"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110817"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380338"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2018.8330202"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884872"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397369"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2948705"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-99241-9_3"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380407"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884807"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3105906"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00099"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606623"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366428.3380768"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2013.29"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771791"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.962562"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2017.8115675"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00020"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3276531"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454051"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786825"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2018.8330203"},{"key":"e_1_3_2_72_2","first-page":"97","article-title":"Probabilistic grammar-based test generation","volume":"310","author":"Soremekun Ezekiel","year":"2021","unstructured":"Ezekiel Soremekun, Esteban Pavese, Nikolas Havrikov, Lars Grunske, and Andreas Zeller. 2021. Probabilistic grammar-based test generation. Softw. Eng. 2021 P-310 (2021), 97\u201398.","journal-title":"Softw. Eng. 2021"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416532"},{"key":"e_1_3_2_74_2","article-title":"Learning to encode and classify test executions","author":"Tsimpourlas Foivos","year":"2020","unstructured":"Foivos Tsimpourlas, Ajitha Rajan, and Miltiadis Allamanis. 2020. Learning to encode and classify test executions. arXiv preprint arXiv:2001.02444 (2020).","journal-title":"arXiv preprint arXiv:2001.02444"},{"key":"e_1_3_2_75_2","article-title":"Dynamic neural program embedding for program repair","author":"Wang Ke","year":"2017","unstructured":"Ke Wang, Rishabh Singh, and Zhendong Su. 2017. Dynamic neural program embedding for program repair. arXiv preprint arXiv:1711.07163 (2017).","journal-title":"arXiv preprint arXiv:1711.07163"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416590"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070536"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180233"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668043"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2017.8115676"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180182"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.45"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134018"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00063"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2560811"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/IBF50092.2020.9034821"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106274"},{"key":"e_1_3_2_88_2","article-title":"Automated classification of overfitting patches with statically extracted code features","author":"Ye He","year":"2021","unstructured":"He Ye, Jian Gu, Matias Martinez, Thomas Durieux, and Martin Monperrus. 2021. Automated classification of overfitting patches with statically extracted code features. IEEE Trans. Softw. Eng. (2021).","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09920-w"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9552-y"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i01.5466"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2874648"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236068"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00137"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511096","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3511096","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:11:58Z","timestamp":1750191118000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511096"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,22]]},"references-count":93,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,10,31]]}},"alternative-id":["10.1145\/3511096"],"URL":"https:\/\/doi.org\/10.1145\/3511096","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,22]]},"assertion":[{"value":"2021-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-08-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}