{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T16:37:24Z","timestamp":1784997444813,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":86,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,6,23]]},"DOI":"10.1145\/3696630.3734199","type":"proceedings-article","created":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T19:08:09Z","timestamp":1753729689000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5864-4488","authenticated-orcid":false,"given":"Mark","family":"Harman","sequence":"first","affiliation":[{"name":"Meta platforms and UCL, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0910-5350","authenticated-orcid":false,"given":"Peter","family":"O'Hearn","sequence":"additional","affiliation":[{"name":"Meta platforms and UCL, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4204-5185","authenticated-orcid":false,"given":"Shubho","family":"Sengupta","sequence":"additional","affiliation":[{"name":"Independent consultant, San Francisco, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,7,28]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Kinga Bojarczuk, Elena Dulskyte, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Ralf Laemmel, Erik Meijer, Silvia Sapora, and Justin Spahr-Summers.","author":"Ahlgren John","year":"2020","unstructured":"John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk, Elena Dulskyte, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Ralf Laemmel, Erik Meijer, Silvia Sapora, and Justin Spahr-Summers. 2020. WES: Agent-based User Interaction Simulation on Real Infrastructure (keynote paper). In 8th Genetic improvement workshop (GI at ICSE 2020), Shin Yoo, Justyna Petke, Westley Weimer, and Bobby R. Bruce (Eds.). ACM, 276\u2013284."},{"key":"e_1_3_2_1_2_1","volume-title":"International Conference on Software Engineering (ICSE) Software Engineering in Practice (SEIP) track. Virtual.","author":"Ahlgren John","year":"2021","unstructured":"John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk, Elena Dulskyte, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, Erik Meijer, Silvia Sapora, and Justin Spahr-Summers. 2021. Testing Web Enabled Simulation at Scale Using Metamorphic Testing. In International Conference on Software Engineering (ICSE) Software Engineering in Practice (SEIP) track. Virtual."},{"key":"e_1_3_2_1_3_1","volume-title":"25th International Conference on Evaluation and Assessment in Software Engineering (EASE","author":"Ahlgren John","year":"2021","unstructured":"John Ahlgren, Kinga Bojarczuk, Sophia Drossopoulou, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, Simon Lucas, Erik Meijer, Steve Omohundro, Rubmary Rojas, Silvia Sapora, Jie M. Zhang, and Norm Zhou. 2021. Facebook's Cyber-Cyber and Cyber-Physical Digital Twins (keynote paper). In 25th International Conference on Evaluation and Assessment in Software Engineering (EASE 2021). Virtual."},{"key":"e_1_3_2_1_4_1","volume-title":"Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice. 1\u201312","author":"Alshahwan Nadia","year":"2024","unstructured":"Nadia Alshahwan, Arianna Blasi, Kinga Bojarczuk, Andrea Ciancone, Natalija Gucevska, Mark Harman, Michal Krolikowski, Rubmary Rojas, Dragos Martac, Simon Schellaert, et al. 2024. Enhancing Testing at Meta with Rich-State Simulated Populations. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice. 1\u201312."},{"key":"e_1_3_2_1_5_1","volume-title":"ACM International Conference on the Foundations of Software Engineering (FSE 2024)","author":"Alshahwan Nadia","year":"2024","unstructured":"Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Mark Harman, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated unit test improvement using Large Language Models at Meta. In ACM International Conference on the Foundations of Software Engineering (FSE 2024) (Porto de Galinhas, Brazil, Brazil)."},{"key":"e_1_3_2_1_6_1","volume-title":"10th International Symposium on Search Based Software Engineering (SSBSE","author":"Alshahwan Nadia","year":"2018","unstructured":"Nadia Alshahwan, Xinbo Gao, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, Taijin Tei, and Ilya Zorin. 2018. Deploying Search Based Software Engineering with Sapienz at Facebook (keynote paper). In 10th International Symposium on Search Based Software Engineering (SSBSE 2018). Montpellier, France, 3\u201345. Springer LNCS 11036."},{"key":"e_1_3_2_1_7_1","volume-title":"2023 IEEE Conference on Software Testing, Verification and Validation (ICST","author":"Alshahwan Nadia","year":"2023","unstructured":"Nadia Alshahwan, Mark Harman, and Alexandru Marginean. 2023. Software Testing Research Challenges: An Industrial Perspective (keynote paper). In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST 2023). IEEE, 1\u201310."},{"key":"e_1_3_2_1_8_1","volume-title":"ICSE workshop on Interoperability and Robustness of Neural Software Engineering (InteNSE)","author":"Alshahwan Nadia","year":"2024","unstructured":"Nadia Alshahwan, Mark Harman, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Assured LLM-Based Software Engineering (keynote paper). In 2nd. ICSE workshop on Interoperability and Robustness of Neural Software Engineering (InteNSE) (Lisbon, Portugal)."},{"key":"e_1_3_2_1_9_1","volume-title":"Foundations of Software Engineering (FSE","author":"Alshahwan Nadia","year":"2024","unstructured":"Nadia Alshahwan, Mark Harman, Alexandru Marginean, and Eddy Wang. 2024. Observation-based unit test generation at Meta. In Foundations of Software Engineering (FSE 2024)."},{"key":"e_1_3_2_1_10_1","volume-title":"Code Analysis and Manipulation Workshop (SCAM). 2001. SCAM Charter. https:\/\/www.ieee-scam.org\/ The cited quotation has remained part of the workshop's credo since its foundation in 2001","author":"Source","unstructured":"Source Code Analysis and Manipulation Workshop (SCAM). 2001. SCAM Charter. https:\/\/www.ieee-scam.org\/ The cited quotation has remained part of the workshop's credo since its foundation in 2001."},{"key":"e_1_3_2_1_11_1","first-page":"8","article-title":"An orchestrated survey of methodologies for automated software test case generation","volume":"86","author":"Anand Saswat","year":"2013","unstructured":"Saswat Anand, Antonia Bertolino, Edmund Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Jenny Li, Phil McMinn, and Hong Zhu. 2013. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86, 8 (August 2013), 1978\u20132001.","journal-title":"Journal of Systems and Software"},{"key":"e_1_3_2_1_12_1","volume-title":"36th International Conference on Software Engineering (ICSE","author":"Androutsopoulos Kelly","year":"2014","unstructured":"Kelly Androutsopoulos, David Clark, Haitao Dan, Mark Harman, and Robert Hierons. 2014. An Analysis of the Relationship between Conditional Entropy and Failed Error Propagation in Software Testing. In 36th International Conference on Software Engineering (ICSE 2014). Hyderabad, India, 573\u2013583."},{"key":"e_1_3_2_1_13_1","first-page":"5","article-title":"The Oracle Problem in Software Testing: A Survey","volume":"41","author":"Barr Earl T.","year":"2015","unstructured":"Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5 (May 2015), 507\u2013525.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1007\/s10664-021-10094-2","article-title":"Developer-centric test amplification: The interplay between automatic generation human exploration","volume":"27","author":"Brandt Carolin","year":"2022","unstructured":"Carolin Brandt and Andy Zaidman. 2022. Developer-centric test amplification: The interplay between automatic generation human exploration. Empirical Software Engineering 27, 4 (2022), 96.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_1_15_1","first-page":"209","article-title":"Klee: unassisted and automatic generation of high-coverage tests for complex systems programs","volume":"8","author":"Cadar Cristian","year":"2008","unstructured":"Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.. In OSDI, Vol. 8. 209\u2013224.","journal-title":"OSDI"},{"key":"e_1_3_2_1_16_1","volume-title":"33rd International Conference on Software Engineering (ICSE'11)","author":"Cadar Cristian","year":"2011","unstructured":"Cristian Cadar, Patrice Godefroid, Sarfraz Khurshid, Corina S. P\u0103s\u0103reanu, Koushik Sen, Nikolai Tillmann, and Willem Visser. 2011. Symbolic execution for software testing in practice: preliminary assessment. In 33rd International Conference on Software Engineering (ICSE'11) (Waikiki, Honolulu, HI, USA). ACM, New York, NY, USA, 1066\u20131071."},{"key":"e_1_3_2_1_17_1","first-page":"2","article-title":"Symbolic Execution for Software Testing","volume":"56","author":"Cadar Cristian","year":"2013","unstructured":"Cristian Cadar and Koushik Sen. 2013. Symbolic Execution for Software Testing: Three Decades Later. Commun. ACM 56, 2 (Feb. 2013), 82\u201390.","journal-title":"Three Decades Later. Commun. ACM"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"crossref","unstructured":"C. Calcagno D. Distefano J. Dubreil D. Gabi P. Hooimeijer M. Luca P. W. O'Hearn I. Papakonstantinou J. Purbrick and D. Rodriguez. 2015. Moving Fast with Software Verification. In NASA Formal Methods - 7th International Symposium. 3\u201311.","DOI":"10.1007\/978-3-319-17524-9_1"},{"key":"e_1_3_2_1_19_1","volume-title":"Proceedings of the 39th International Conference on Software Engineering, ICSE 2017","author":"Chekam Thierry Titcheu","year":"2017","unstructured":"Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, and Mark Harman. 2017. An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20\u201328, 2017. 597\u2013608."},{"key":"e_1_3_2_1_20_1","volume-title":"FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. In 44th IEEE\/ACM 44th International Conference on Software Engineering, ICSE 2022","author":"Cordy Maxime","year":"2022","unstructured":"Maxime Cordy, Renaud Rwemalika, Adriano Franci, Mike Papadakis, and Mark Harman. 2022. FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. In 44th IEEE\/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25\u201327, 2022. ACM, 982\u2013994."},{"key":"e_1_3_2_1_21_1","first-page":"5","article-title":"Social Processes and Proofs of Theorems and","volume":"22","author":"De Millo Richard A.","year":"1979","unstructured":"Richard A. De Millo, Richard J. Lipton, and Alan J. Perlis. 1979. Social Processes and Proofs of Theorems and Programs. Commun. ACM 22, 5 (May 1979), 271\u2013280. An earlier version appeared in ACM Symposium on Principles of Programming Languages (POPL), Los Angeles, California, 1977 pp. 206\u2013214.","journal-title":"Programs. Commun. ACM"},{"key":"e_1_3_2_1_22_1","first-page":"14","article-title":"On a Political Pamphlet from the Middle Ages (A response to the paper 'Social Processes and Proofs of Theorems and Programs' by DeMillo, Lipton, and Perlis). ACM SIGSOFT","volume":"3","author":"Dijkstra Edsger W.","year":"1978","unstructured":"Edsger W. Dijkstra. 1978. On a Political Pamphlet from the Middle Ages (A response to the paper 'Social Processes and Proofs of Theorems and Programs' by DeMillo, Lipton, and Perlis). ACM SIGSOFT, Software Engineering Notes 3, 2 (1978), 14\u201317.","journal-title":"Software Engineering Notes"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1145\/3338112","article-title":"Scaling static analyses at","volume":"62","author":"Distefano Dino","year":"2019","unstructured":"Dino Distefano, Manuel F\u00e4hndrich, Francesco Logozzo, and Peter W O'Hearn. 2019. Scaling static analyses at Facebook. Commun. ACM 62, 8 (2019), 62\u201370.","journal-title":"Facebook. Commun. ACM"},{"key":"e_1_3_2_1_24_1","unstructured":"Rajendra T Dodhiawala NS Sridharan Peter Raulefs and Cynthia Pickering. 1989. Real-Time AI Systems: A Definition and An Architecture.. In IJCAI. Citeseer 256\u2013264."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/TSE.2008.103","article-title":"Carving and Replaying Differential Unit Test Cases from System Test Cases","volume":"35","author":"Elbaum Sebastian G.","year":"2009","unstructured":"Sebastian G. Elbaum, Hui Nee Chin, Matthew B. Dwyer, and Matthew Jorde. 2009. Carving and Replaying Differential Unit Test Cases from System Test Cases. IEEE Transactions on Software Engineering 35, 1 (2009), 29\u201345.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_1_26_1","volume-title":"ICSE Future of Software Engineering (FoSE","author":"Fan Angela","year":"2023","unstructured":"Angela Fan, Beliz Gokkaya, Mitya Lyubarskiy, Mark Harman, Shubho Sengupta, Shin Yoo, and Jie Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In ICSE Future of Software Engineering (FoSE 2023)."},{"key":"e_1_3_2_1_27_1","volume-title":"Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 628\u2013640","author":"Fan Zhiyu","year":"2024","unstructured":"Zhiyu Fan, Haifeng Ruan, Sergey Mechtaev, and Abhik Roychoudhury. 2024. Oracle-guided Program Selection from Large Language Models. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 628\u2013640."},{"key":"e_1_3_2_1_28_1","volume-title":"Computer Aided Verification: 8th International Conference, CAV'96 New Brunswick, NJ, USA, July 31\u2013August 3, 1996 Proceedings 8. Springer, 348\u2013359","author":"Fernandez Jean Claude","year":"1996","unstructured":"Jean Claude Fernandez, Claude Jard, Thierry J\u00e9ron, and C\u00e9sar Viho. 1996. Using on-the-fly verification techniques for the generation of test suites. In Computer Aided Verification: 8th International Conference, CAV'96 New Brunswick, NJ, USA, July 31\u2013August 3, 1996 Proceedings 8. Springer, 348\u2013359."},{"key":"e_1_3_2_1_29_1","volume-title":"2025 ACM Conference on Foundations of Software Engineering (FSE","author":"Foster Christopher","year":"2025","unstructured":"Christopher Foster, Abhishek Gulati, Mark Harman, Inna Harper, Ke Mao, Jillian Ritchey, Herv\u00e9 Robert, and Shubho Sengupta. 2025. Mutation-Guided LLM-based Test Generation at Meta. In 2025 ACM Conference on Foundations of Software Engineering (FSE 2025). ACM. Also available as arXiv preprint arXiv:2501.12862."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1016\/S0164-1212(96)00154-9","article-title":"All-Uses vs Mutation Testing: An Experimental Comparison of Effectiveness","volume":"38","author":"Frankl Phyllis G.","year":"1997","unstructured":"Phyllis G. Frankl, Stewart N. Weiss, and Cang Hu. 1997. All-Uses vs Mutation Testing: An Experimental Comparison of Effectiveness. Journal of Systems Software 38 (1997), 235\u2013253.","journal-title":"Journal of Systems Software"},{"key":"e_1_3_2_1_31_1","volume-title":"8th European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC\/FSE '11)","author":"Fraser Gordon","year":"2011","unstructured":"Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In 8th European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC\/FSE '11). ACM, 416\u2013419."},{"key":"e_1_3_2_1_32_1","volume-title":"Programming Language Design and Implementation (PLDI","author":"Godefroid Patrice","year":"2005","unstructured":"Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed automated random testing. In Programming Language Design and Implementation (PLDI 2005), Vivek Sarkar and Mary W. Hall (Eds.). ACM, 213\u2013223."},{"key":"e_1_3_2_1_33_1","volume-title":"Proceedings of the 36th international conference on software engineering. 72\u201382","author":"Gopinath Rahul","year":"2014","unstructured":"Rahul Gopinath, Carlos Jensen, and Alex Groce. 2014. Code coverage for suite evaluation by developers. In Proceedings of the 36th international conference on software engineering. 72\u201382."},{"key":"e_1_3_2_1_34_1","volume-title":"10th IEEE International Working Conference on Source Code Analysis and Manipulation","author":"Harman Mark","year":"2010","unstructured":"Mark Harman. 2010. Why Source Code Analysis and Manipulation Will Always Be Important (Keynote Paper). In 10th IEEE International Working Conference on Source Code Analysis and Manipulation. Timisoara, Romania."},{"key":"e_1_3_2_1_35_1","volume-title":"Strong Higher Order Mutation-Based Test Data Generation. In 8th European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC\/FSE '11)","author":"Harman Mark","unstructured":"Mark Harman, Yue Jia, and William B. Langdon. 2011. Strong Higher Order Mutation-Based Test Data Generation. In 8th European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC\/FSE '11) (Szeged, Hungary). ACM, New York, NY, USA, 212\u2013222."},{"key":"e_1_3_2_1_36_1","volume-title":"8th IEEE International Conference on Software Testing, Verification and Validation (ICST","author":"Harman Mark","year":"2015","unstructured":"Mark Harman, Yue Jia, and Yuanyuan Zhang. 2015. Achievements, open problems and challenges for search based software testing (keynote Paper). In 8th IEEE International Conference on Software Testing, Verification and Validation (ICST 2015). Graz, Austria."},{"key":"e_1_3_2_1_37_1","first-page":"14","article-title":"Search Based Software Engineering","volume":"43","author":"Harman Mark","year":"2001","unstructured":"Mark Harman and Bryan F. Jones. 2001. Search Based Software Engineering. Information and Software Technology 43, 14 (Dec. 2001), 833\u2013839.","journal-title":"Information and Software Technology"},{"key":"e_1_3_2_1_38_1","first-page":"1","article-title":"Search Based Software Engineering","volume":"45","author":"Harman Mark","year":"2012","unstructured":"Mark Harman, Afshin Mansouri, and Yuanyuan Zhang. 2012. Search Based Software Engineering: Trends, Techniques and Applications. Comput. Surveys 45, 1 (November 2012), 11:1\u201311:61.","journal-title":"Trends, Techniques and Applications. Comput. Surveys"},{"key":"e_1_3_2_1_39_1","volume-title":"18th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM","author":"Harman Mark","year":"2018","unstructured":"Mark Harman and Peter O'Hearn. 2018. From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis (keynote paper). In 18th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2018). Madrid, Spain, 1\u201323."},{"key":"e_1_3_2_1_40_1","volume-title":"9th annual conference on Genetic and evolutionary computation (GECCO","author":"Harman Mark","year":"2007","unstructured":"Mark Harman and Laurence Tratt. 2007. Pareto Optimal Search-based Refactoring at the Design Level. In 9th annual conference on Genetic and evolutionary computation (GECCO 2007). ACM Press, London, UK, 1106 \u2013 1113."},{"key":"e_1_3_2_1_41_1","volume-title":"36th International Conference on Software Engineering (ICSE","author":"Harman Mark","year":"2014","unstructured":"Mark Harman, Xiangjuan Yao, and Yue Jia. 2014. A Study of Equivalent and Stubborn Mutation Operators Using Human Analysis of Equivalence. In 36th International Conference on Software Engineering (ICSE 2014). Hyderabad, India, 919\u2013930."},{"key":"e_1_3_2_1_42_1","volume-title":"2023 IEEE\/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 1\u201311","author":"Hasan Kazi Amit","year":"2023","unstructured":"Kazi Amit Hasan, Marcos Macedo, Yuan Tian, Bram Adams, and Steven Ding. 2023. Understanding the time to first response in GitHub pull requests. In 2023 IEEE\/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 1\u201311."},{"key":"e_1_3_2_1_43_1","first-page":"2","article-title":"Using Formal Methods to Support","volume":"41","author":"Hierons Rob","year":"2009","unstructured":"Rob Hierons, Kirill Bogdanov, Jonathan Bowen, Rance Cleaveland, John Derrick, Jeremy Dick, Marian Gheorghe, Mark Harman, Kalpesh Kapoor, Paul Krause, Gerald Luettgen, Tony Simons, Sergiy Vilkomir, Martin Woodward, and Hussein Zedan. 2009. Using Formal Methods to Support Testing. Comput. Surveys 41, 2 (Feb. 2009). Article 9.","journal-title":"Testing. Comput. Surveys"},{"key":"e_1_3_2_1_44_1","volume-title":"FME '96: Industrial Benefit and Advances in Formal Methods: Third International Symposium of Formal Methods Europe (LNCS, 1051)","author":"Richard Hoare Charles Anthony","year":"1996","unstructured":"Charles Anthony Richard Hoare. 1996. How did software get so reliable without proof?. In FME '96: Industrial Benefit and Advances in Formal Methods: Third International Symposium of Formal Methods Europe (LNCS, 1051). Springer-Verlag, 1\u201317."},{"key":"e_1_3_2_1_45_1","volume-title":"IEEE International Conference on Software Engineering (ICSE'96)","author":"Richard Hoare Charles Anthony","year":"1996","unstructured":"Charles Anthony Richard Hoare. 1996. How did software get so reliable without proof?. In IEEE International Conference on Software Engineering (ICSE'96). IEEE Computer Society Press, Los Alamitos, California, USA. Keynote talk and extended abstract."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.neucom.2021.04.112","article-title":"Online learning: A comprehensive survey","volume":"459","author":"Hoi Steven CH","year":"2021","unstructured":"Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. 2021. Online learning: A comprehensive survey. Neurocomputing 459 (2021), 249\u2013289.","journal-title":"Neurocomputing"},{"key":"e_1_3_2_1_47_1","volume-title":"TOGLL: Correct and strong test oracle generation with LLMs. arXiv preprint arXiv:2405.03786","author":"Hossain Soneya Binta","year":"2024","unstructured":"Soneya Binta Hossain and Matthew Dwyer. 2024. TOGLL: Correct and strong test oracle generation with LLMs. arXiv preprint arXiv:2405.03786 (2024)."},{"key":"e_1_3_2_1_48_1","volume-title":"Article arXiv:1909.04847 (Sep","author":"Ie Eugene","year":"2019","unstructured":"Eugene Ie, Chih-wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. 2019. RecSim: A Configurable Simulation Platform for Recommender Systems. arXiv e-prints, Article arXiv:1909.04847 (Sep 2019). arXiv:1909.04847 [cs.LG]"},{"key":"e_1_3_2_1_49_1","volume-title":"Proceedings of the 36th international conference on software engineering. 435\u2013445","author":"Inozemtseva Laura","year":"2014","unstructured":"Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th international conference on software engineering. 435\u2013445."},{"key":"e_1_3_2_1_50_1","volume-title":"TestGenEval: A real world unit test generation and test completion benchmark. arXiv preprint arXiv:2410.00752","author":"Jain Kush","year":"2024","unstructured":"Kush Jain, Gabriel Synnaeve, and Baptiste Rozi\u00e8re. 2024. TestGenEval: A real world unit test generation and test completion benchmark. arXiv preprint arXiv:2410.00752 (2024)."},{"key":"e_1_3_2_1_51_1","first-page":"5","article-title":"An Analysis and Survey of the Development of Mutation Testing","volume":"37","author":"Jia Yue","year":"2011","unstructured":"Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (September\u2013October 2011), 649 \u2013 678.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"crossref","unstructured":"Ren\u00e9 Just Darioush Jalali Laura Inozemtseva Michael D. Ernst Reid Holmes and Gordon Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing? Technical Report UW-CSE-14-02-02. University of Washington.","DOI":"10.1145\/2635868.2635929"},{"key":"e_1_3_2_1_53_1","first-page":"97","article-title":"Literate","volume":"27","author":"Knuth Donald E.","year":"1984","unstructured":"Donald E. Knuth. 1984. Literate Programming. Comput. J. 27, 2 (1984), 97\u2013111.","journal-title":"Programming. Comput. J."},{"key":"e_1_3_2_1_54_1","volume-title":"Do LLMs generate test oracles that capture the actual or the expected program behaviour? arXiv preprint arXiv:2410.21136","author":"Konstantinou Michael","year":"2024","unstructured":"Michael Konstantinou, Renzo Degiovanni, and Mike Papadakis. 2024. Do LLMs generate test oracles that capture the actual or the expected program behaviour? arXiv preprint arXiv:2410.21136 (2024)."},{"key":"e_1_3_2_1_55_1","volume-title":"Realtime systems: Design principles for distributed embedded applications","author":"Kopetz Hermann","unstructured":"Hermann Kopetz and Wilfried Steiner. 2022. Real-Time Communication. In Realtime systems: Design principles for distributed embedded applications. Springer, 177\u2013200."},{"key":"e_1_3_2_1_56_1","first-page":"1","article-title":"AUSTIN: An Open Source Tool for Search Based Software Testing of C Programs","volume":"55","author":"Lakhotia Kiran","year":"2013","unstructured":"Kiran Lakhotia, Mark Harman, and Hamilton Gross. 2013. AUSTIN: An Open Source Tool for Search Based Software Testing of C Programs. Journal of Information and Software Technology 55, 1 (January 2013), 112\u2013125.","journal-title":"Journal of Information and Software Technology"},{"key":"e_1_3_2_1_57_1","volume-title":"22nd International Symposium on Foundations of Software Engineering (FSE","author":"Luo Qingzhou","year":"2014","unstructured":"Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In 22nd International Symposium on Foundations of Software Engineering (FSE 2014), Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne Storey (Eds.). ACM, Hong Kong, China, 643\u2013653."},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1109\/TSE.2013.44","article-title":"Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation","volume":"40","author":"Madeyski Lech","year":"2013","unstructured":"Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2013. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE Transactions on Software Engineering 40, 1 (2013), 23\u201342.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_1_59_1","volume-title":"Manuel Egele, Edward J. Schwartz, and Maverick Woo.","author":"Man\u00e8s Valentin J. M.","year":"2018","unstructured":"Valentin J. M. Man\u00e8s, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2018. The Art, Science, and Engineering of Fuzzing: A Survey. CoRR abs\/1812.00140 (2018). arXiv:1812.00140"},{"key":"e_1_3_2_1_60_1","volume-title":"Sapienz: Multi-objective Automated Testing for Android Applications. In International Symposium on Software Testing and Analysis (ISSTA","author":"Mao Ke","year":"2016","unstructured":"Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective Automated Testing for Android Applications. In International Symposium on Software Testing and Analysis (ISSTA 2016). 94\u2013105."},{"key":"e_1_3_2_1_61_1","volume-title":"15th IEEE Conference on Software Testing, Verification and Validation, ICST 2022","author":"Mao Ke","year":"2022","unstructured":"Ke Mao, Timotej Kapus, Lambros Petrou, \u00c1kos Hajdu, Matteo Marescotti, Andreas L\u00f6scher, Mark Harman, and Dino Distefano. 2022. FAUSTA: Scaling Dynamic Analysis with Traffic Generation at WhatsApp. In 15th IEEE Conference on Software Testing, Verification and Validation, ICST 2022, Valencia, Spain, April 4\u201314, 2022. IEEE, 267\u2013278."},{"key":"e_1_3_2_1_62_1","unstructured":"James McGill. 2025. Time to First Review. https:\/\/docs.velocity.codeclimate.com\/en\/articles\/2913584-time-to-first-review"},{"key":"e_1_3_2_1_63_1","first-page":"2","article-title":"Search-based Software Test Data Generation: A Survey","volume":"14","author":"McMinn Phil","year":"2004","unstructured":"Phil McMinn. 2004. Search-based Software Test Data Generation: A Survey. Software Testing, Verification and Reliability 14, 2 (June 2004), 105\u2013156.","journal-title":"Software Testing, Verification and Reliability"},{"key":"e_1_3_2_1_64_1","volume-title":"Test Oracle Automation in the era of LLMs. ACM Transactions on Software Engineering and Methodology","author":"Molina Facundo","year":"2024","unstructured":"Facundo Molina, Alessandra Gorla, and Marcelo d'Amorim. 2024. Test Oracle Automation in the era of LLMs. ACM Transactions on Software Engineering and Methodology (2024)."},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1016\/j.infsof.2014.09.009","article-title":"Testing robot controllers using constraint programming and continuous integration","volume":"57","author":"Mossige Morten","year":"2015","unstructured":"Morten Mossige, Arnaud Gotlieb, and Hein Meling. 2015. Testing robot controllers using constraint programming and continuous integration. Information and Software Technology 57 (2015), 169\u2013185.","journal-title":"Information and Software Technology"},{"key":"e_1_3_2_1_66_1","volume-title":"Foundations of Software Engineering (FSE","author":"Murali Vijayaraghavan","year":"2024","unstructured":"Vijayaraghavan Murali, Chandra Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, Nachiappan Nagappan, and Peter C Rigby. 2024. CodeCompose: A Large-Scale Industrial Deployment of AI-assisted Code Authoring. In Foundations of Software Engineering (FSE 2024). Porto de Galinhas, Brazil. Earlier version available as arXiv:2305.12050."},{"key":"e_1_3_2_1_67_1","volume-title":"Proceedings of the ACM on Programming Languages 4, POPL","author":"O'Hearn Peter W","year":"2019","unstructured":"Peter W O'Hearn. 2019. Incorrectness logic. Proceedings of the ACM on Programming Languages 4, POPL (2019), 1\u201332."},{"key":"e_1_3_2_1_68_1","volume-title":"Fast and Effective Equivalent Mutant Detection Technique. In 37th International Conference on Software Engineering (ICSE","author":"Papadakis Mike","year":"2015","unstructured":"Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In 37th International Conference on Software Engineering (ICSE 2015). Florence, Italy, 936\u2013946."},{"key":"e_1_3_2_1_69_1","first-page":"3","article-title":"Genetic Improvement of Software: a Comprehensive Survey","volume":"22","author":"Petke Justyna","year":"2018","unstructured":"Justyna Petke, Saemundur O. Haraldsson, Mark Harman, William B. Langdon, David R. White, and John R. Woodward. 2018. Genetic Improvement of Software: a Comprehensive Survey. IEEE Transactions on Evolutionary Computation 22, 3 (June 2018), 415\u2013432.","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"e_1_3_2_1_70_1","volume-title":"Coverup: Coverage-guided LLM-based test generation. arXiv preprint arXiv:2403.16218","author":"Pizzorno Juan Altmayer","year":"2024","unstructured":"Juan Altmayer Pizzorno and Emery D Berger. 2024. Coverup: Coverage-guided LLM-based test generation. arXiv preprint arXiv:2403.16218 (2024)."},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1016\/j.cosrev.2010.06.001","article-title":"A survey on Search-Based Software Design","volume":"4","author":"R\u00e4ih\u00e4 Outi","year":"2010","unstructured":"Outi R\u00e4ih\u00e4. 2010. A survey on Search-Based Software Design. Computer Science Review 4, 4 (2010), 203\u2013249.","journal-title":"Computer Science Review"},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1109\/TSE.2018.2803055","article-title":"A systematic review of interaction in search-based software engineering","volume":"45","author":"Ramirez Aurora","year":"2018","unstructured":"Aurora Ramirez, Jose Raul Romero, and Christopher L Simons. 2018. A systematic review of interaction in search-based software engineering. IEEE Transactions on Software Engineering 45, 8 (2018), 760\u2013781.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_1_73_1","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1016\/j.jss.2018.12.015","article-title":"A survey of many-objective optimisation in search-based software engineering","volume":"149","author":"Ramirez Aurora","year":"2019","unstructured":"Aurora Ramirez, Jos\u00e9 Ra\u00fal Romero, and Sebastian Ventura. 2019. A survey of many-objective optimisation in search-based software engineering. Journal of Systems and Software 149 (2019), 382\u2013395.","journal-title":"Journal of Systems and Software"},{"key":"e_1_3_2_1_74_1","volume-title":"7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC\/FSE","author":"Schuler David","year":"2009","unstructured":"David Schuler and Andreas Zeller. 2009. Javalanche: efficient mutation testing for Java. In 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC\/FSE 2009). 297\u2013298."},{"key":"e_1_3_2_1_75_1","volume-title":"10th European Software Engineering Conference and 13th ACM International Symposium on Foundations of Software Engineering (ESEC\/FSE '05)","author":"Sen Koushik","year":"2005","unstructured":"Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: a concolic unit testing engine for C. In 10th European Software Engineering Conference and 13th ACM International Symposium on Foundations of Software Engineering (ESEC\/FSE '05), Michel Wermelinger and Harald Gall (Eds.). ACM, 263\u2013272."},{"key":"e_1_3_2_1_76_1","doi-asserted-by":"crossref","first-page":"798","DOI":"10.1109\/TSE.2010.34","article-title":"Interactive, Evolutionary Search in Upstream Object-Oriented Class Design","volume":"36","author":"Simons Christopher L.","year":"2010","unstructured":"Christopher L. Simons, Ian C. Parmee, and Rhys Gwynllyw. 2010. Interactive, Evolutionary Search in Upstream Object-Oriented Class Design. IEEE Transactions on Software Engineering 36, 6 (2010), 798\u2013816.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_1_77_1","volume-title":"Proceedings of the 2000 ACM SIGSOFT international symposium on Software Testing and Analysis (ISSTA","author":"Steven John","year":"2000","unstructured":"John Steven, Pravir Chandra, Bob Fleck, and Andy Podgurski. 2000. jRapture: A capture\/replay tool for observation-based testing. In Proceedings of the 2000 ACM SIGSOFT international symposium on Software Testing and Analysis (ISSTA 2000). 158\u2013167."},{"key":"e_1_3_2_1_78_1","doi-asserted-by":"crossref","unstructured":"Deepika Tiwari Martin Monperrus and Benoit Baudry. 2023. Mimicking Production Behavior with Generated Mocks. arXiv:2208.01321 [cs.SE]","DOI":"10.1109\/TSE.2024.3458448"},{"key":"e_1_3_2_1_79_1","volume-title":"Simulation-Driven Automated End-to-End Test and Oracle Inference. In 45th IEEE\/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2023","author":"Tuli Shreshth","year":"2023","unstructured":"Shreshth Tuli, Kinga Bojarczuk, Natalija Gucevska, Mark Harman, Xiao-Yu Wang, and Graham Wright. 2023. Simulation-Driven Automated End-to-End Test and Oracle Inference. In 45th IEEE\/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2023, Melbourne, Australia, May 14\u201320, 2023. IEEE, 122\u2013133."},{"key":"e_1_3_2_1_80_1","volume-title":"2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 7\u201312","author":"van Hijfte Lars","year":"2021","unstructured":"Lars van Hijfte and Ana Oprescu. 2021. Mutantbench: an equivalent mutant problem comparison framework. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 7\u201312."},{"key":"e_1_3_2_1_81_1","volume-title":"On-the-fly testing of reactive systems. Submitted for publication","author":"Veanes Margus","year":"2005","unstructured":"Margus Veanes, Colin Campbell, Wolfram Schulte, Pushmeet Kohli, N Tillmann, and W Grieskamp. 2005. On-the-fly testing of reactive systems. Submitted for publication (2005)."},{"key":"e_1_3_2_1_82_1","volume-title":"Software testing with large language model: Survey, landscape, and vision. arXiv preprint arXiv:2307.07221","author":"Wang Junjie","year":"2023","unstructured":"Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2023. Software testing with large language model: Survey, landscape, and vision. arXiv preprint arXiv:2307.07221 (2023)."},{"key":"e_1_3_2_1_83_1","volume-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1258\u20131268","author":"Wang Zejun","year":"2024","unstructured":"Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1258\u20131268."},{"key":"e_1_3_2_1_84_1","unstructured":"Michal Zalewski. Accessed March 27th 2025. American fuzzy lop. http:\/\/lcamtuf.coredump.cx\/afl\/"},{"key":"e_1_3_2_1_85_1","unstructured":"Shuyin Zhao. 2023. GitHub Copilot now has a better AI model and new capabilities. https:\/\/github.blog\/2023-02-14-github-copilot-now-has-a-better-ai-model-and-new-capabilities\/"},{"key":"e_1_3_2_1_86_1","volume-title":"Automatic code summarization: A systematic literature review. arXiv preprint arXiv:1909.04352","author":"Zhu Yuxiang","year":"2019","unstructured":"Yuxiang Zhu and Minxue Pan. 2019. Automatic code summarization: A systematic literature review. arXiv preprint arXiv:1909.04352 (2019)."}],"event":{"name":"FSE Companion '25: 33rd ACM International Conference on the Foundations of Software Engineering","location":"Clarion Hotel Trondheim Trondheim Norway","acronym":"FSE Companion '25","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3696630.3734199","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T19:13:06Z","timestamp":1753729986000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3696630.3734199"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,23]]},"references-count":86,"alternative-id":["10.1145\/3696630.3734199","10.1145\/3696630"],"URL":"https:\/\/doi.org\/10.1145\/3696630.3734199","relation":{},"subject":[],"published":{"date-parts":[[2025,6,23]]},"assertion":[{"value":"2025-07-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}