{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T03:06:40Z","timestamp":1775790400432,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":78,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Beijing Outstanding Young Scientist Program","award":["BJJWZYJH01201910001004"],"award-info":[{"award-number":["BJJWZYJH01201910001004"]}]},{"name":"ERC Advanced Grant","award":["741278"],"award-info":[{"award-number":["741278"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,7,12]]},"DOI":"10.1145\/3597926.3598060","type":"proceedings-article","created":{"date-parts":[[2023,7,13]],"date-time":"2023-07-13T20:12:53Z","timestamp":1689279173000},"page":"334-346","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Who Judges the Judge: An Empirical Study on Online Judge Tests"],"prefix":"10.1145","author":[{"given":"Kaibo","family":"Liu","sequence":"first","affiliation":[{"name":"Peking University, China"}]},{"given":"Yudong","family":"Han","sequence":"additional","affiliation":[{"name":"Peking University, China"}]},{"given":"Jie M.","family":"Zhang","sequence":"additional","affiliation":[{"name":"King\u2019s College London, UK"}]},{"given":"Zhenpeng","family":"Chen","sequence":"additional","affiliation":[{"name":"University College London, UK"}]},{"given":"Federica","family":"Sarro","sequence":"additional","affiliation":[{"name":"University College London, UK"}]},{"given":"Mark","family":"Harman","sequence":"additional","affiliation":[{"name":"University College London, UK"}]},{"given":"Gang","family":"Huang","sequence":"additional","affiliation":[{"name":"Peking University, China \/ National Key Laboratory of Data Space Technology and System, China"}]},{"given":"Yun","family":"Ma","sequence":"additional","affiliation":[{"name":"Peking University, China"}]}],"member":"320","published-online":{"date-parts":[[2023,7,13]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"[n. d.]. Github Copilot. https:\/\/github.com\/features\/copilot \t\t\t\t  [n. d.]. Github Copilot. https:\/\/github.com\/features\/copilot"},{"key":"e_1_3_2_1_2_1","unstructured":"[n. d.]. Tabnine. https:\/\/www.tabnine.com\/ \t\t\t\t  [n. d.]. Tabnine. https:\/\/www.tabnine.com\/"},{"key":"e_1_3_2_1_3_1","unstructured":"2015. LeetCode - The world\u2019s leading online programming learning platform. https:\/\/leetcode.com \t\t\t\t  2015. LeetCode - The world\u2019s leading online programming learning platform. https:\/\/leetcode.com"},{"key":"e_1_3_2_1_4_1","unstructured":"2022. Top website designers developers freelancers for your next project. https:\/\/topcoder.com\/ \t\t\t\t  2022. Top website designers developers freelancers for your next project. https:\/\/topcoder.com\/"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2006.83"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcs.2005.12.014"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00036"},{"key":"e_1_3_2_1_8_1","volume-title":"2014 9th International Conference on Computer Science & Education. 149\u2013152","author":"Bez Jean Luca","year":"2014","unstructured":"Jean Luca Bez , Neilor A Tonin , and Paulo R Rodegheri . 2014 . URI Online Judge Academic: A tool for algorithms and programming classes . In 2014 9th International Conference on Computer Science & Education. 149\u2013152 . Jean Luca Bez, Neilor A Tonin, and Paulo R Rodegheri. 2014. URI Online Judge Academic: A tool for algorithms and programming classes. In 2014 9th International Conference on Computer Science & Education. 149\u2013152."},{"key":"e_1_3_2_1_9_1","unstructured":"boxed. 2020. mutmut. https:\/\/github.com\/boxed\/mutmut \t\t\t\t  boxed. 2020. mutmut. https:\/\/github.com\/boxed\/mutmut"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2021.100027"},{"key":"e_1_3_2_1_11_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021 . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374."},{"key":"e_1_3_2_1_12_1","volume-title":"Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE","author":"Chen Zhenpeng","year":"2020","unstructured":"Zhenpeng Chen , Yanbin Cao , Yuanqiang Liu , Haoyu Wang , Tao Xie , and Xuanzhe Liu . 2020 . A comprehensive study on challenges in deploying deep learning based software . In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2020. 750\u2013762. Zhenpeng Chen, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, Tao Xie, and Xuanzhe Liu. 2020. A comprehensive study on challenges in deploying deep learning based software. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2020. 750\u2013762."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2948707"},{"key":"e_1_3_2_1_14_1","volume-title":"Supported Mutation Operators. https:\/\/mull.readthedocs.io\/en\/latest\/SupportedMutations.html [Online","author":"Mull","year":"2022","unstructured":"Mull contributors. 2022. Supported Mutation Operators. https:\/\/mull.readthedocs.io\/en\/latest\/SupportedMutations.html [Online ; accessed November- 2022 ] Mull contributors. 2022. Supported Mutation Operators. https:\/\/mull.readthedocs.io\/en\/latest\/SupportedMutations.html [Online; accessed November-2022]"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2018.00021"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/C-M.1978.218136"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2018.00024"},{"key":"e_1_3_2_1_18_1","volume-title":"International Conference on Database Systems for Advanced Applications. 785\u2013789","author":"Dong Yu","year":"2020","unstructured":"Yu Dong , Jingyang Hou , and Xuesong Lu . 2020 . An intelligent online judge system for programming training . In International Conference on Database Systems for Advanced Applications. 785\u2013789 . Yu Dong, Jingyang Hou, and Xuesong Lu. 2020. An intelligent online judge system for programming training. In International Conference on Database Systems for Advanced Applications. 785\u2013789."},{"key":"e_1_3_2_1_19_1","volume-title":"The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers. 549\u2013552","author":"Evans Robert B","year":"2007","unstructured":"Robert B Evans and Alberto Savoia . 2007 . Differential testing: a new approach to change detection . In The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers. 549\u2013552 . Robert B Evans and Alberto Savoia. 2007. Differential testing: a new approach to change detection. In The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers. 549\u2013552."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/1149707.1149713"},{"key":"e_1_3_2_1_21_1","unstructured":"Rodrigo Elias Francisco and Ana Paula Ambrosio. 2015. Mining an Online Judge System to Support Introductory Computer Programming Teaching.. In EDM (Workshops). \t\t\t\t  Rodrigo Elias Francisco and Ana Paula Ambrosio. 2015. Mining an Online Judge System to Support Introductory Computer Programming Teaching.. In EDM (Workshops)."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER52604.2021.00023"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Daya Guo Shuai Lu Nan Duan Yanlin Wang Ming Zhou and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. arXiv preprint arXiv:2203.03850. \t\t\t\t  Daya Guo Shuai Lu Nan Duan Yanlin Wang Ming Zhou and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. arXiv preprint arXiv:2203.03850.","DOI":"10.18653\/v1\/2022.acl-long.499"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2019.00021"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464825"},{"key":"e_1_3_2_1_26_1","unstructured":"AtCoder Inc.. 2012. AtCoder. https:\/\/atcoder.jp \t\t\t\t  AtCoder Inc.. 2012. AtCoder. https:\/\/atcoder.jp"},{"key":"e_1_3_2_1_27_1","unstructured":"AtCoder Inc.. 2016. Atcoder testcases. https:\/\/atcoder.jp\/posts\/21 \t\t\t\t  AtCoder Inc.. 2016. Atcoder testcases. https:\/\/atcoder.jp\/posts\/21"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568271"},{"key":"e_1_3_2_1_29_1","unstructured":"jacoco. 2022. jacoco. https:\/\/github.com\/jacoco\/jacoco \t\t\t\t  jacoco. 2022. jacoco. https:\/\/github.com\/jacoco\/jacoco"},{"key":"e_1_3_2_1_30_1","unstructured":"JaroslavHavrda. [n. d.]. Mull corrupted LLVM module \u00b7 issue #1004 \u00b7 mull-project\/mull. https:\/\/github.com\/mull-project\/mull\/issues\/1004 \t\t\t\t  JaroslavHavrda. [n. d.]. Mull corrupted LLVM module \u00b7 issue #1004 \u00b7 mull-project\/mull. https:\/\/github.com\/mull-project\/mull\/issues\/1004"},{"key":"e_1_3_2_1_31_1","volume-title":"An analysis and survey of the development of mutation testing","author":"Jia Yue","year":"2010","unstructured":"Yue Jia and Mark Harman . 2010. An analysis and survey of the development of mutation testing . IEEE transactions on software engineering, 37, 5 ( 2010 ), 649\u2013678. Yue Jia and Mark Harman. 2010. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering, 37, 5 (2010), 649\u2013678."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510187"},{"key":"e_1_3_2_1_33_1","volume-title":"International Conference on Web-Based Learning. 343\u2013354","author":"Kosowski Adrian","year":"2007","unstructured":"Adrian Kosowski , Micha\u0142 Ma\u0142 afiejski, and Tomasz Noi\u0144ski . 2007 . Application of an online judge & contester system in academic tuition . In International Conference on Web-Based Learning. 343\u2013354 . Adrian Kosowski, Micha\u0142 Ma\u0142 afiejski, and Tomasz Noi\u0144ski. 2007. Application of an online judge & contester system in academic tuition. In International Conference on Web-Based Learning. 343\u2013354."},{"key":"e_1_3_2_1_34_1","volume-title":"LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation.","author":"Lattner Chris","year":"2004","unstructured":"Chris Lattner and Vikram Adve . 2004 . LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation. San Jose, CA, USA . 75\u201388. Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation. San Jose, CA, USA. 75\u201388."},{"key":"e_1_3_2_1_35_1","unstructured":"Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser R\u00e9mi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d\u2019Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. arXiv preprint arXiv:2203.07814. \t\t\t\t  Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser R\u00e9mi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d\u2019Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. arXiv preprint arXiv:2203.07814."},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7977256"},{"key":"e_1_3_2_1_37_1","volume-title":"Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE","author":"Lou Yiling","year":"2020","unstructured":"Yiling Lou , Zhenpeng Chen , Yanbin Cao , Dan Hao , and Lu Zhang . 2020 . Understanding build issue resolution in practice: symptoms and fix patterns . In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2020. 617\u2013628. Yiling Lou, Zhenpeng Chen, Yanbin Cao, Dan Hao, and Lu Zhang. 2020. Understanding build issue resolution in practice: symptoms and fix patterns. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2020. 617\u2013628."},{"key":"e_1_3_2_1_38_1","unstructured":"luogu dev. 2019. Cyaron. https:\/\/github.com\/luogu-dev\/cyaron \t\t\t\t  luogu dev. 2019. Cyaron. https:\/\/github.com\/luogu-dev\/cyaron"},{"key":"e_1_3_2_1_39_1","volume-title":"Proceedings of the 19th International Conference on Mining Software Repositories. 524\u2013536","author":"Ma Wei","year":"2022","unstructured":"Wei Ma , Mengjie Zhao , Ezekiel Soremekun , Qiang Hu , Jie M Zhang , Mike Papadakis , Maxime Cordy , Xiaofei Xie , and Yves Le Traon . 2022 . GraphCode2Vec: generic code embedding via lexical and program dependence analyses . In Proceedings of the 19th International Conference on Mining Software Repositories. 524\u2013536 . Wei Ma, Mengjie Zhao, Ezekiel Soremekun, Qiang Hu, Jie M Zhang, Mike Papadakis, Maxime Cordy, Xiaofei Xie, and Yves Le Traon. 2022. GraphCode2Vec: generic code embedding via lexical and program dependence analyses. In Proceedings of the 19th International Conference on Mining Software Repositories. 524\u2013536."},{"key":"e_1_3_2_1_40_1","first-page":"100","article-title":"Differential Testing for Software","volume":"10","author":"McKeeman William M.","year":"1998","unstructured":"William M. McKeeman . 1998 . Differential Testing for Software . Digit. Tech. J. , 10 , 1 (1998), 100 \u2013 107 . http:\/\/www.hpl.hp.com\/hpjournal\/dtj\/vol10num1\/vol10num1art9.pdf William M. McKeeman. 1998. Differential Testing for Software. Digit. Tech. J., 10, 1 (1998), 100\u2013107. http:\/\/www.hpl.hp.com\/hpjournal\/dtj\/vol10num1\/vol10num1art9.pdf","journal-title":"Digit. Tech. J."},{"key":"e_1_3_2_1_41_1","volume-title":"Software Testing: Testing Across the Entire Software Development Life Cycle (1 ed.)","author":"McLeod Raymond","year":"2007","unstructured":"Raymond McLeod Jr and Gerald D Everett . 2007 . Software Testing: Testing Across the Entire Software Development Life Cycle (1 ed.) . John Wiley , Hoboken . isbn:047179371X;9780471793717;9780470146347;0470146346; Raymond McLeod Jr and Gerald D Everett. 2007. Software Testing: Testing Across the Entire Software Development Life Cycle (1 ed.). John Wiley, Hoboken. isbn:047179371X;9780471793717;9780470146347;0470146346;"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/366246.366248"},{"key":"e_1_3_2_1_43_1","unstructured":"Mikhail Mirzayanov. 2009. Codeforces. https:\/\/codeforces.com\/ \t\t\t\t  Mikhail Mirzayanov. 2009. Codeforces. https:\/\/codeforces.com\/"},{"key":"e_1_3_2_1_44_1","unstructured":"Mikhail Mirzayanov. 2011. Codeforces contest rules. https:\/\/codeforces.com\/blog\/entry\/4088 \t\t\t\t  Mikhail Mirzayanov. 2011. Codeforces contest rules. https:\/\/codeforces.com\/blog\/entry\/4088"},{"key":"e_1_3_2_1_45_1","unstructured":"Mull-project. 2022. mull. https:\/\/github.com\/mull-project\/mull \t\t\t\t  Mull-project. 2022. mull. https:\/\/github.com\/mull-project\/mull"},{"key":"e_1_3_2_1_46_1","unstructured":"Kenko Nakamura. 2015. Atcoder problems models. https:\/\/kenkoooo.com\/atcoder\/resources\/problem-models.json \t\t\t\t  Kenko Nakamura. 2015. Atcoder problems models. https:\/\/kenkoooo.com\/atcoder\/resources\/problem-models.json"},{"key":"e_1_3_2_1_47_1","unstructured":"Nedbat. 2022. coveragepy. https:\/\/github.com\/nedbat\/coveragepy \t\t\t\t  Nedbat. 2022. coveragepy. https:\/\/github.com\/nedbat\/coveragepy"},{"key":"e_1_3_2_1_48_1","unstructured":"OpenAI. 2022. ChatGPT. https:\/\/chat.openai.com\/ \t\t\t\t  OpenAI. 2022. ChatGPT. https:\/\/chat.openai.com\/"},{"key":"e_1_3_2_1_49_1","volume-title":"Yves Le Traon, and Mark Harman","author":"Papadakis Mike","year":"2019","unstructured":"Mike Papadakis , Marinos Kintis , Jie M. Zhang , Yue Jia , Yves Le Traon, and Mark Harman . 2019 . Mutation testing advances: an analysis and survey. In Advances in Computers. 112, Elsevier , 275\u2013378. Mike Papadakis, Marinos Kintis, Jie M. Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: an analysis and survey. In Advances in Computers. 112, Elsevier, 275\u2013378."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3314221.3314629"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00087"},{"key":"e_1_3_2_1_52_1","volume-title":"Mutation operators. https:\/\/pitest.org\/quickstart\/mutators [Online","author":"Test","year":"2022","unstructured":"PI Test contributors. 2022. Mutation operators. https:\/\/pitest.org\/quickstart\/mutators [Online ; accessed November- 2022 ] PITest contributors. 2022. Mutation operators. https:\/\/pitest.org\/quickstart\/mutators [Online; accessed November-2022]"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-69926-4_25"},{"key":"e_1_3_2_1_54_1","volume-title":"Project codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655, 1035","author":"Puri Ruchir","year":"2021","unstructured":"Ruchir Puri , David S Kung , Geert Janssen , Wei Zhang , Giacomo Domeniconi , Vladmir Zolotov , Julian Dolby , Jie Chen , Mihir Choudhury , and Lindsey Decker . 2021. Project codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655, 1035 ( 2021 ). Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, and Lindsey Decker. 2021. Project codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655, 1035 (2021)."},{"key":"e_1_3_2_1_55_1","volume-title":"Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation.","author":"Radicek Ivan","year":"2018","unstructured":"Ivan Radicek , Sumit Gulwani , and Florian Zuleger . 2018 . Automated clustering and program repair for introductory programming assignments . In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation. Ivan Radicek, Sumit Gulwani, and Florian Zuleger. 2018. Automated clustering and program repair for introductory programming assignments. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation."},{"key":"e_1_3_2_1_56_1","first-page":"131","article-title":"Competitive learning in informatics: The UVa online judge experience","volume":"2","author":"Revilla Miguel A","year":"2008","unstructured":"Miguel A Revilla , Shahriar Manzoor , and Rujia Liu . 2008 . Competitive learning in informatics: The UVa online judge experience . Olympiads in Informatics , 2 , 10 (2008), 131 \u2013 148 . Miguel A Revilla, Shahriar Manzoor, and Rujia Liu. 2008. Competitive learning in informatics: The UVa online judge experience. Olympiads in Informatics, 2, 10 (2008), 131\u2013148.","journal-title":"Olympiads in Informatics"},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10177-8"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1213\/ANE.0000000000002864"},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468598"},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3020266"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_1_62_1","volume-title":"Proceedings of the 44th International Conference on Software Engineering. 1181\u20131192","author":"Sun Zeyu","year":"2022","unstructured":"Zeyu Sun , Jie M Zhang , Yingfei Xiong , Mark Harman , Mike Papadakis , and Lu Zhang . 2022 . Improving machine translation systems via isotopic replacement . In Proceedings of the 44th International Conference on Software Engineering. 1181\u20131192 . Zeyu Sun, Jie M Zhang, Yingfei Xiong, Mark Harman, Mike Papadakis, and Lu Zhang. 2022. Improving machine translation systems via isotopic replacement. In Proceedings of the 44th International Conference on Software Engineering. 1181\u20131192."},{"key":"e_1_3_2_1_63_1","volume-title":"2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 180\u2013182","author":"Tan Shin Hwei","year":"2017","unstructured":"Shin Hwei Tan , Jooyong Yi , Sergey Mechtaev , and Abhik Roychoudhury . 2017 . Codeflaws: a programming competition benchmark for evaluating automated program repair tools . In 2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 180\u2013182 . Shin Hwei Tan, Jooyong Yi, Sergey Mechtaev, and Abhik Roychoudhury. 2017. Codeflaws: a programming competition benchmark for evaluating automated program repair tools. In 2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 180\u2013182."},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3240474"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3143560"},{"key":"e_1_3_2_1_66_1","volume-title":"2016 International Symposium on Educational Technology (ISET). 57\u201360","author":"Wu Huiting","year":"2016","unstructured":"Huiting Wu , Yanshen Liu , Lin Qiu , and Yi Liu . 2016 . Online judge system and its applications in c language teaching . In 2016 International Symposium on Educational Technology (ISET). 57\u201360 . Huiting Wu, Yanshen Liu, Lin Qiu, and Yi Liu. 2016. Online judge system and its applications in c language teaching. In 2016 International Symposium on Educational Technology (ISET). 57\u201360."},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300864"},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/1138929.1138949"},{"key":"e_1_3_2_1_69_1","unstructured":"Fuchen Ying Pengcheng Xu and Di Xie. 2003. Welcome to PKU Judgeonline. http:\/\/poj.org\/ \t\t\t\t  Fuchen Ying Pengcheng Xu and Di Xie. 2003. Welcome to PKU Judgeonline. http:\/\/poj.org\/"},{"key":"e_1_3_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2861392"},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TSE.2019.2962027","article-title":"Machine learning testing: Survey, landscapes and horizons","volume":"48","author":"Zhang Jie M.","year":"2020","unstructured":"Jie M. Zhang , Mark Harman , Lei Ma , and Yang Liu . 2020 . Machine learning testing: Survey, landscapes and horizons . IEEE Transactions on Software Engineering , 48 , 1 (2020), 1 \u2013 36 . Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering, 48, 1 (2020), 1\u201336.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2961897"},{"key":"e_1_3_2_1_73_1","volume-title":"2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). 252\u2013263","author":"Zhang Jie M.","year":"2019","unstructured":"Jie M. Zhang , Lingming Zhang , Dan Hao , Meng Wang , and Lu Zhang . 2019 . Do pseudo test suites lead to inflated correlation in measuring test effectiveness? In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). 252\u2013263 . Jie M. Zhang, Lingming Zhang, Dan Hao, Meng Wang, and Lu Zhang. 2019. Do pseudo test suites lead to inflated correlation in measuring test effectiveness? In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). 252\u2013263."},{"key":"e_1_3_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2019.00020"},{"key":"e_1_3_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786858"},{"key":"e_1_3_2_1_76_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3158670","article-title":"Automatically learning topics and difficulty levels of problems in online judge systems","volume":"36","author":"Zhao Wayne Xin","year":"2018","unstructured":"Wayne Xin Zhao , Wenhui Zhang , Yulan He , Xing Xie , and Ji-Rong Wen . 2018 . Automatically learning topics and difficulty levels of problems in online judge systems . ACM Transactions on Information Systems (TOIS) , 36 , 3 (2018), 1 \u2013 33 . Wayne Xin Zhao, Wenhui Zhang, Yulan He, Xing Xie, and Ji-Rong Wen. 2018. Automatically learning topics and difficulty levels of problems in online judge systems. ACM Transactions on Information Systems (TOIS), 36, 3 (2018), 1\u201333.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_3_2_1_77_1","volume-title":"2020 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 1302\u20131306","author":"Zhu Chenqian","year":"2020","unstructured":"Chenqian Zhu , Weisong Sun , Qin Liu , Yangyang Yuan , Chunrong Fang , and Yong Huang . 2020 . HomoTR: online test recommendation system based on homologous code matching . In 2020 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 1302\u20131306 . Chenqian Zhu, Weisong Sun, Qin Liu, Yangyang Yuan, Chunrong Fang, and Yong Huang. 2020. HomoTR: online test recommendation system based on homologous code matching. In 2020 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 1302\u20131306."},{"key":"e_1_3_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/1840\/1\/012029"}],"event":{"name":"ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Seattle WA USA","acronym":"ISSTA '23","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","AITO"]},"container-title":["Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598060","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597926.3598060","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:42Z","timestamp":1750182522000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598060"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,12]]},"references-count":78,"alternative-id":["10.1145\/3597926.3598060","10.1145\/3597926"],"URL":"https:\/\/doi.org\/10.1145\/3597926.3598060","relation":{},"subject":[],"published":{"date-parts":[[2023,7,12]]},"assertion":[{"value":"2023-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}