{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,19]],"date-time":"2026-11-19T03:22:40Z","timestamp":1795058560655,"version":"4.10.2"},"publisher-location":"New York, NY, USA","reference-count":86,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CCF-2131943,CCF-2141474"],"award-info":[{"award-number":["CCF-2131943,CCF-2141474"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,7,12]]},"DOI":"10.1145\/3597926.3598067","type":"proceedings-article","created":{"date-parts":[[2023,7,13]],"date-time":"2023-07-13T16:12:53Z","timestamp":1689264773000},"page":"423-435","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":354,"title":["Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models"],"prefix":"10.1145","author":[{"given":"Yinlin","family":"Deng","sequence":"first","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunqiu Steven","family":"Xia","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haoran","family":"Peng","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenyuan","family":"Yang","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lingming","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,7,13]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2022. Beta distribution. https:\/\/en.wikipedia.org\/wiki\/Beta_distribution \t\t\t\t  2022. Beta distribution. https:\/\/en.wikipedia.org\/wiki\/Beta_distribution"},{"key":"e_1_3_2_1_2_1","unstructured":"2022. Codex Documentation - Best Practices. https:\/\/beta.openai.com\/docs\/guides\/code\/best-practices \t\t\t\t  2022. Codex Documentation - Best Practices. https:\/\/beta.openai.com\/docs\/guides\/code\/best-practices"},{"key":"e_1_3_2_1_3_1","unstructured":"2022. Coverage.py. https:\/\/github.com\/nedbat\/coveragepy \t\t\t\t  2022. Coverage.py. https:\/\/github.com\/nedbat\/coveragepy"},{"key":"e_1_3_2_1_4_1","unstructured":"2022. Gamma function. https:\/\/en.wikipedia.org\/wiki\/Gamma_function \t\t\t\t  2022. Gamma function. https:\/\/en.wikipedia.org\/wiki\/Gamma_function"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Wasi Uddin Ahmad Saikat Chakraborty Baishakhi Ray and Kai-Wei Chang. 2021. Unified Pre-training for Program Understanding and Generation. arxiv:2103.06333. \t\t\t\t  Wasi Uddin Ahmad Saikat Chakraborty Baishakhi Ray and Kai-Wei Chang. 2021. Unified Pre-training for Program Understanding and Generation. arxiv:2103.06333.","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"e_1_3_2_1_6_1","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arxiv:2108.07732 \t\t\t\t  Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arxiv:2108.07732"},{"key":"e_1_3_2_1_7_1","unstructured":"Rahul Banerjee. 2021. Writing Better Tests with AI and GitHub Copilot. Codecov https:\/\/about.codecov.io\/blog\/writing-better-tests-with-ai-and-github-copilot\/ \t\t\t\t  Rahul Banerjee. 2021. Writing Better Tests with AI and GitHub Copilot. Codecov https:\/\/about.codecov.io\/blog\/writing-better-tests-with-ai-and-github-copilot\/"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.3016773"},{"key":"e_1_3_2_1_9_1","volume-title":"Hospitals turn to artificial intelligence to help with an age-old problem: Doctors","author":"Brown Dalvin","year":"2021","unstructured":"Dalvin Brown . 2021. Hospitals turn to artificial intelligence to help with an age-old problem: Doctors \u2019 poor bedside manners. The Washington Post , https:\/\/www.washingtonpost.com\/technology\/ 2021 \/02\/16\/virtual-ai-hospital-patients\/ Dalvin Brown. 2021. Hospitals turn to artificial intelligence to help with an age-old problem: Doctors\u2019 poor bedside manners. The Washington Post, https:\/\/www.washingtonpost.com\/technology\/2021\/02\/16\/virtual-ai-hospital-patients\/"},{"key":"e_1_3_2_1_10_1","unstructured":"Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 \t\t\t\t  Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165"},{"key":"e_1_3_2_1_11_1","volume-title":"Advances in Neural Information Processing Systems","author":"Chapelle Olivier","year":"2011","unstructured":"Olivier Chapelle and Lihong Li. 2011. An Empirical Evaluation of Thompson Sampling . In Advances in Neural Information Processing Systems , J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.). 24, Curran Associates, Inc. . https:\/\/proceedings.neurips.cc\/paper\/ 2011 \/file\/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf Olivier Chapelle and Lihong Li. 2011. An Empirical Evaluation of Thompson Sampling. In Advances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.). 24, Curran Associates, Inc.. https:\/\/proceedings.neurips.cc\/paper\/2011\/file\/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf"},{"key":"e_1_3_2_1_12_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021 . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1179"},{"key":"e_1_3_2_1_14_1","unstructured":"Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann Parker Schuh Kensen Shi Sasha Tsvyashchenko Joshua Maynez Abhishek Rao Parker Barnes Yi Tay Noam Shazeer Vinodkumar Prabhakaran Emily Reif Nan Du Ben Hutchinson Reiner Pope James Bradbury Jacob Austin Michael Isard Guy Gur-Ari Pengcheng Yin Toju Duke Anselm Levskaya Sanjay Ghemawat Sunipa Dev Henryk Michalewski Xavier Garcia Vedant Misra Kevin Robinson Liam Fedus Denny Zhou Daphne Ippolito David Luan Hyeontaek Lim Barret Zoph Alexander Spiridonov Ryan Sepassi David Dohan Shivani Agrawal Mark Omernick Andrew M. Dai Thanumalayan Sankaranarayana Pillai Marie Pellat Aitor Lewkowycz Erica Moreira Rewon Child Oleksandr Polozov Katherine Lee Zongwei Zhou Xuezhi Wang Brennan Saeta Mark Diaz Orhan Firat Michele Catasta Jason Wei Kathy Meier-Hellstern Douglas Eck Jeff Dean Slav Petrov and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. arxiv:2204.02311. \t\t\t\t  Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann Parker Schuh Kensen Shi Sasha Tsvyashchenko Joshua Maynez Abhishek Rao Parker Barnes Yi Tay Noam Shazeer Vinodkumar Prabhakaran Emily Reif Nan Du Ben Hutchinson Reiner Pope James Bradbury Jacob Austin Michael Isard Guy Gur-Ari Pengcheng Yin Toju Duke Anselm Levskaya Sanjay Ghemawat Sunipa Dev Henryk Michalewski Xavier Garcia Vedant Misra Kevin Robinson Liam Fedus Denny Zhou Daphne Ippolito David Luan Hyeontaek Lim Barret Zoph Alexander Spiridonov Ryan Sepassi David Dohan Shivani Agrawal Mark Omernick Andrew M. Dai Thanumalayan Sankaranarayana Pillai Marie Pellat Aitor Lewkowycz Erica Moreira Rewon Child Oleksandr Polozov Katherine Lee Zongwei Zhou Xuezhi Wang Brennan Saeta Mark Diaz Orhan Firat Michele Catasta Jason Wei Kathy Meier-Hellstern Douglas Eck Jeff Dean Slav Petrov and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. arxiv:2204.02311."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213848"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549085"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"crossref","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155 \t\t\t\t  Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the 14th USENIX Conference on Offensive Technologies (WOOT\u201920)","author":"Fioraldi Andrea","year":"2020","unstructured":"Andrea Fioraldi , Dominik Maier , Heiko Ei\u00df feldt, and Marc Heuse . 2020 . AFL++: Combining Incremental Steps of Fuzzing Research . In Proceedings of the 14th USENIX Conference on Offensive Technologies (WOOT\u201920) . USENIX Association, USA. Article 10, 1 pages. Andrea Fioraldi, Dominik Maier, Heiko Ei\u00df feldt, and Marc Heuse. 2020. AFL++: Combining Incremental Steps of Fuzzing Research. In Proceedings of the 14th USENIX Conference on Offensive Technologies (WOOT\u201920). USENIX Association, USA. Article 10, 1 pages."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.14"},{"key":"e_1_3_2_1_21_1","volume-title":"Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999.","author":"Fried Daniel","year":"2022","unstructured":"Daniel Fried , Armen Aghajanyan , Jessy Lin , Sida Wang , Eric Wallace , Freda Shi , Ruiqi Zhong , Wen-tau Yih, Luke Zettlemoyer , and Mike Lewis . 2022 . Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999."},{"key":"e_1_3_2_1_22_1","unstructured":"Josiah Willard Gibbs. 1902. Elementary principles in statistical mechanics: developed with especial reference to the rational foundations of thermodynamics. C. Scribner\u2019s sons. \t\t\t\t  Josiah Willard Gibbs. 1902. Elementary principles in statistical mechanics: developed with especial reference to the rational foundations of thermodynamics. C. Scribner\u2019s sons."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2017.8115618"},{"key":"e_1_3_2_1_24_1","volume-title":"Proceedings of the 27th International Conference on Neural Information Processing Systems -","volume":"2","author":"Goodfellow Ian J.","year":"2014","unstructured":"Ian J. Goodfellow , Jean Pouget-Abadie , Mehdi Mirza , Bing Xu , David Warde-Farley , Sherjil Ozair , Aaron Courville , and Yoshua Bengio . 2014 . Generative Adversarial Nets . In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (NIPS\u201914). MIT Press, Cambridge, MA, USA. 2672\u20132680. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (NIPS\u201914). MIT Press, Cambridge, MA, USA. 2672\u20132680."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510092"},{"key":"e_1_3_2_1_26_1","volume-title":"Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou.","author":"Guo Daya","year":"2021","unstructured":"Daya Guo , Shuo Ren , Shuai Lu , Zhangyin Feng , Duyu Tang , Shujie Liu , Long Zhou , Nan Duan , Alexey Svyatkovskiy , Shengyu Fu , Michele Tufano , Shao Kun Deng , Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021 . GraphCodeBERT: Pre-training Code Representations with Data Flow . arxiv:2009.08366. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Representations with Data Flow. arxiv:2009.08366."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416571"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337322"},{"key":"e_1_3_2_1_29_1","volume-title":"Long Short-term Memory. Neural computation, 9","author":"Hochreiter Sepp","year":"1997","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber . 1997. Long Short-term Memory. Neural computation, 9 ( 1997 ), 12, 1735\u201380. Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long Short-term Memory. Neural computation, 9 (1997), 12, 1735\u201380."},{"key":"e_1_3_2_1_30_1","volume-title":"USENIX Security Symposium. 445\u2013458","author":"Holler Christian","year":"2012","unstructured":"Christian Holler , Kim Herzig , and Andreas Zeller . 2012 . Fuzzing with Code Fragments .. In USENIX Security Symposium. 445\u2013458 . Christian Holler, Kim Herzig, and Andreas Zeller. 2012. Fuzzing with Code Fragments.. In USENIX Security Symposium. 445\u2013458."},{"key":"e_1_3_2_1_31_1","unstructured":"Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. arXiv:1904.09751 \t\t\t\t  Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. arXiv:1904.09751"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3203217.3203241"},{"key":"e_1_3_2_1_33_1","unstructured":"2022. Hugging Face.  https:\/\/huggingface.co \t\t\t\t  2022. Hugging Face.  https:\/\/huggingface.co"},{"key":"e_1_3_2_1_34_1","volume-title":"Ng","author":"Huval Brody","year":"2015","unstructured":"Brody Huval , Tao Wang , Sameep Tandon , Jeff Kiske , Will Song , Joel Pazhayampallil , Mykhaylo Andriluka , Pranav Rajpurkar , Toki Migimatsu , Royce Cheng-Yue , Fernando Mujica , Adam Coates , and Andrew Y . Ng . 2015 . An Empirical Evaluation of Deep Learning on Highway Driving . arxiv:1504.01716. Brody Huval, Tao Wang, Sameep Tandon, Jeff Kiske, Will Song, Joel Pazhayampallil, Mykhaylo Andriluka, Pranav Rajpurkar, Toki Migimatsu, Royce Cheng-Yue, Fernando Mujica, Adam Coates, and Andrew Y. Ng. 2015. An Empirical Evaluation of Deep Learning on Highway Driving. arxiv:1504.01716."},{"key":"e_1_3_2_1_35_1","unstructured":"2015. libFuzzer a library for coverage-guided fuzz testing.  https:\/\/llvm.org\/docs\/LibFuzzer.html \t\t\t\t  2015. libFuzzer a library for coverage-guided fuzz testing.  https:\/\/llvm.org\/docs\/LibFuzzer.html"},{"key":"e_1_3_2_1_36_1","unstructured":"2020. Keras.  https:\/\/keras.io\/ \t\t\t\t  2020. Keras.  https:\/\/keras.io\/"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360607"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594334"},{"key":"e_1_3_2_1_40_1","volume-title":"29th USENIX Security Symposium (USENIX Security 20)","author":"Lee Suyoung","year":"2020","unstructured":"Suyoung Lee , HyungSeok Han , Sang Kil Cha , and Sooel Son . 2020 . Montage: A Neural Network Language $Model-Guided$$JavaScript$ Engine Fuzzer . In 29th USENIX Security Symposium (USENIX Security 20) . 2613\u20132630. Suyoung Lee, HyungSeok Han, Sang Kil Cha, and Sooel Son. 2020. Montage: A Neural Network Language $Model-Guided$$JavaScript$ Engine Fuzzer. In 29th USENIX Security Symposium (USENIX Security 20). 2613\u20132630."},{"key":"e_1_3_2_1_41_1","volume-title":"CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In 45th International Conference on Software Engineering.","author":"Lemieux Caroline","year":"2023","unstructured":"Caroline Lemieux , Jeevana Priya Inala , Shuvendu K Lahiri , and Siddhartha Sen . 2023 . CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In 45th International Conference on Software Engineering. Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In 45th International Conference on Software Engineering."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238176"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"e_1_3_2_1_44_1","unstructured":"Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2023. NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers. In ASPLOS. 530\u2013543. \t\t\t\t  Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2023. NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers. In ASPLOS. 530\u2013543."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3527317"},{"key":"e_1_3_2_1_46_1","volume-title":"Yuyao Wang, and Lingming Zhang.","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu , Chunqiu Steven Xia , Yuyao Wang, and Lingming Zhang. 2023 . Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3560815"},{"key":"e_1_3_2_1_48_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence. 33","author":"Liu Xiao","year":"2019","unstructured":"Xiao Liu , Xiaoting Li , Rupesh Prajapati , and Dinghao Wu . 2019 . Deepfuzz: Automatic generation of syntax valid c programs for fuzz testing . In Proceedings of the AAAI Conference on Artificial Intelligence. 33 , 1044\u20131051. Xiao Liu, Xiaoting Li, Rupesh Prajapati, and Dinghao Wu. 2019. Deepfuzz: Automatic generation of syntax valid c programs for fuzz testing. In Proceedings of the AAAI Conference on Artificial Intelligence. 33, 1044\u20131051."},{"key":"e_1_3_2_1_49_1","unstructured":"Yang Liu. 2019. Fine-tune BERT for Extractive Summarization. arXiv:1903.10318 \t\t\t\t  Yang Liu. 2019. Fine-tune BERT for Extractive Summarization. arXiv:1903.10318"},{"key":"e_1_3_2_1_50_1","unstructured":"2016. American Fuzzy Lop - Whitepaper.  https:\/\/lcamtuf.coredump.cx\/afl\/technical_details.txt \t\t\t\t  2016. American Fuzzy Lop - Whitepaper.  https:\/\/lcamtuf.coredump.cx\/afl\/technical_details.txt"},{"key":"e_1_3_2_1_51_1","volume-title":"Waldinger","author":"Manna Zohar","year":"1971","unstructured":"Zohar Manna and Richard J . Waldinger . 1971 . Toward Automatic Program Synthesis. Commun. ACM , 14, 3 (1971), mar, 151\u2013165. Zohar Manna and Richard J. Waldinger. 1971. Toward Automatic Program Synthesis. Commun. ACM, 14, 3 (1971), mar, 151\u2013165."},{"key":"e_1_3_2_1_52_1","volume-title":"Learning Deep Semantics for Test Completion. In 45th International Conference on Software Engineering.","author":"Nie Pengyu","year":"2023","unstructured":"Pengyu Nie , Rahul Banerjee , Junyi Jessy Li , Raymond J. Mooney , and Milos Gligoric . 2023 . Learning Deep Semantics for Test Completion. In 45th International Conference on Software Engineering. Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, and Milos Gligoric. 2023. Learning Deep Semantics for Test Completion. In 45th International Conference on Software Engineering."},{"key":"e_1_3_2_1_53_1","volume-title":"Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena.","author":"Nye Maxwell","year":"2021","unstructured":"Maxwell Nye , Anders Johan Andreassen , Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021 . Show Your Work: Scratchpads for Intermediate Computation with Language Models . https:\/\/doi.org\/10.48550\/ARXIV.2112.00114 10.48550\/ARXIV.2112.00114 Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. https:\/\/doi.org\/10.48550\/ARXIV.2112.00114"},{"key":"e_1_3_2_1_54_1","volume-title":"BUSTLE: Bottom-Up program synthesis through learning-guided exploration. arXiv preprint arXiv:2007.14381.","author":"Odena Augustus","year":"2020","unstructured":"Augustus Odena , Kensen Shi , David Bieber , Rishabh Singh , Charles Sutton , and Hanjun Dai . 2020 . BUSTLE: Bottom-Up program synthesis through learning-guided exploration. arXiv preprint arXiv:2007.14381. Augustus Odena, Kensen Shi, David Bieber, Rishabh Singh, Charles Sutton, and Hanjun Dai. 2020. BUSTLE: Bottom-Up program synthesis through learning-guided exploration. arXiv preprint arXiv:2007.14381."},{"key":"e_1_3_2_1_55_1","volume-title":"Proceedings of the ACM on Programming Languages, 5, OOPSLA","author":"Park Jiwon","year":"2021","unstructured":"Jiwon Park , Dominik Winterer , Chengyu Zhang , and Zhendong Su . 2021 . Generative type-aware mutation for testing SMT solvers . Proceedings of the ACM on Programming Languages, 5, OOPSLA (2021), 1\u201319. Jiwon Park, Dominik Winterer, Chengyu Zhang, and Zhendong Su. 2021. Generative type-aware mutation for testing SMT solvers. Proceedings of the ACM on Programming Languages, 5, OOPSLA (2021), 1\u201319."},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_3_2_1_57_1","unstructured":"2018. PyTorch.  http:\/\/pytorch.org \t\t\t\t  2018. PyTorch.  http:\/\/pytorch.org"},{"key":"e_1_3_2_1_58_1","volume-title":"Language models are unsupervised multitask learners. OpenAI blog, 1, 8","author":"Radford Alec","year":"2019","unstructured":"Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9."},{"key":"e_1_3_2_1_59_1","volume-title":"Liu","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J . Liu . 2020 . Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res ., jan. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., jan."},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"crossref","unstructured":"Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. arXiv:2102.07350 \t\t\t\t  Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. arXiv:2102.07350","DOI":"10.1145\/3411763.3451760"},{"key":"e_1_3_2_1_61_1","volume-title":"Abbas Kazerouni, Ian Osband, and Zheng Wen.","author":"Russo Daniel J","year":"2018","unstructured":"Daniel J Russo , Benjamin Van Roy , Abbas Kazerouni, Ian Osband, and Zheng Wen. 2018 . A tutorial on thompson sampling. Foundations and Trends\u00ae in Machine Learning , 11, 1 (2018), 1\u201396. Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. 2018. A tutorial on thompson sampling. Foundations and Trends\u00ae in Machine Learning, 11, 1 (2018), 1\u201396."},{"key":"e_1_3_2_1_62_1","unstructured":"John Schulman Barret Zoph Jacob Hilton Christina Kim Jacob Menick Jiayi Weng Juan Felipe Ceron Uribe Liam Fedus Luke Metz Michael Pokorny Rapha Gontijo Lopes Shengjia Zhao Arun Vijayvergiya Eric Sigler Adam Perelman Chelsea Voss Mike Heaton Joel Parish Dave Cummings Rajeev Nayak Valerie Balcom David Schnurr Tomer Kaftan Chris Hallacy Nicholas Turley Noah Deutsch Vik Goel Jonathan Ward Aris Konstantinidis Wojciech Zaremba Long Ouyang Leonard Bogdonoff Joshua Gross David Medina Sarah Yoo Teddy Lee Ryan Lowe Dan Mossing Joost Huizinga Roger Jiang Carroll Wainwright Diogo Almeida Steph Lin Marvin Zhang Kai Xiao Katarina Slama Steven Bills Alex Gray Jan Leike Jakub Pachocki Phil Tillet Shantanu Jain Greg Brockman and Nick Ryder. 2022. ChatGPT: Optimizing Language Models for Dialogue.  https:\/\/openai.com\/blog\/chatgpt\/ \t\t\t\t  John Schulman Barret Zoph Jacob Hilton Christina Kim Jacob Menick Jiayi Weng Juan Felipe Ceron Uribe Liam Fedus Luke Metz Michael Pokorny Rapha Gontijo Lopes Shengjia Zhao Arun Vijayvergiya Eric Sigler Adam Perelman Chelsea Voss Mike Heaton Joel Parish Dave Cummings Rajeev Nayak Valerie Balcom David Schnurr Tomer Kaftan Chris Hallacy Nicholas Turley Noah Deutsch Vik Goel Jonathan Ward Aris Konstantinidis Wojciech Zaremba Long Ouyang Leonard Bogdonoff Joshua Gross David Medina Sarah Yoo Teddy Lee Ryan Lowe Dan Mossing Joost Huizinga Roger Jiang Carroll Wainwright Diogo Almeida Steph Lin Marvin Zhang Kai Xiao Katarina Slama Steven Bills Alex Gray Jan Leike Jakub Pachocki Phil Tillet Shantanu Jain Greg Brockman and Nick Ryder. 2022. ChatGPT: Optimizing Language Models for Dialogue.  https:\/\/openai.com\/blog\/chatgpt\/"},{"key":"e_1_3_2_1_63_1","unstructured":"Max Sch\u00e4fer Sarah Nadi Aryaz Eghbali and Frank Tip. 2023. Adaptive Test Generation Using a Large Language Model. arxiv:2302.06527. \t\t\t\t  Max Sch\u00e4fer Sarah Nadi Aryaz Eghbali and Frank Tip. 2023. Adaptive Test Generation Using a Large Language Model. arxiv:2302.06527."},{"key":"e_1_3_2_1_64_1","volume-title":"Program synthesis by sketching","author":"Solar-Lezama Armando","unstructured":"Armando Solar-Lezama . 2008. Program synthesis by sketching . University of California , Berkeley. Armando Solar-Lezama. 2008. Program synthesis by sketching. University of California, Berkeley."},{"key":"e_1_3_2_1_65_1","volume-title":"Fuzzing: Brute Force Vulnerability Discovery","author":"Sutton Michael","year":"2007","unstructured":"Michael Sutton , Adam Greene , and Pedram Amini . 2007 . Fuzzing: Brute Force Vulnerability Discovery . Addison-Wesley Professional . Michael Sutton, Adam Greene, and Pedram Amini. 2007. Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley Professional."},{"key":"e_1_3_2_1_66_1","unstructured":"2020. TensorFlow.  https:\/\/www.tensorflow.org \t\t\t\t  2020. TensorFlow.  https:\/\/www.tensorflow.org"},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/25.3-4.285"},{"key":"e_1_3_2_1_68_1","unstructured":"Alina Tugend. 2021. A Smarter App Is Watching Your Wallet. The New York Times https:\/\/www.nytimes.com\/2021\/03\/09\/business\/apps-personal-finance-budget.html \t\t\t\t  Alina Tugend. 2021. A Smarter App Is Watching Your Wallet. The New York Times https:\/\/www.nytimes.com\/2021\/03\/09\/business\/apps-personal-finance-budget.html"},{"key":"e_1_3_2_1_69_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 \t\t\t\t  Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762"},{"key":"e_1_3_2_1_70_1","volume-title":"Hung Viet Pham, and Lin Tan","author":"Wang Jiannan","year":"2022","unstructured":"Jiannan Wang , Thibaud Lutellier , Shangshu Qian , Hung Viet Pham, and Lin Tan . 2022 . EAGLE : Creating Equivalent Graphs to Test Deep Learning Libraries . Jiannan Wang, Thibaud Lutellier, Shangshu Qian, Hung Viet Pham, and Lin Tan. 2022. EAGLE: Creating Equivalent Graphs to Test Deep Learning Libraries."},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409761"},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510041"},{"key":"e_1_3_2_1_73_1","volume-title":"Chi, Quoc Le, and Denny Zhou","author":"Wei Jason","year":"2022","unstructured":"Jason Wei , Xuezhi Wang , Dale Schuurmans , Maarten Bosma , Ed Chi, Quoc Le, and Denny Zhou . 2022 . Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903."},{"key":"e_1_3_2_1_74_1","volume-title":"Proceedings of the 45th International Conference on Software Engineering (ICSE","author":"Xia Chunqiu Steven","year":"2023","unstructured":"Chunqiu Steven Xia , Yuxiang Wei , and Lingming Zhang . 2023 . Automated program repair in the era of large pre-trained language models . In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023)."},{"key":"e_1_3_2_1_75_1","unstructured":"Chunqiu Steven Xia and Lingming Zhang. 2022. Less Training More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning. arXiv:2207.08281 \t\t\t\t  Chunqiu Steven Xia and Lingming Zhang. 2022. Less Training More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning. arXiv:2207.08281"},{"key":"e_1_3_2_1_76_1","unstructured":"Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385. \t\t\t\t  Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385."},{"key":"e_1_3_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534220"},{"key":"e_1_3_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_3_2_1_79_1","volume-title":"Fuzzing Automatic Differentiation in Deep-Learning Libraries. In International Conference on Software Engineering (ICSE).","author":"Yang Chenyuan","year":"2023","unstructured":"Chenyuan Yang , Yinlin Deng , Jiayi Yao , Yuxing Tu , Hanchi Li , and Lingming Zhang . 2023 . Fuzzing Automatic Differentiation in Deep-Learning Libraries. In International Conference on Software Engineering (ICSE). to appear Chenyuan Yang, Yinlin Deng, Jiayi Yao, Yuxing Tu, Hanchi Li, and Lingming Zhang. 2023. Fuzzing Automatic Differentiation in Deep-Learning Libraries. In International Conference on Software Engineering (ICSE). to appear"},{"key":"e_1_3_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993498.1993532"},{"key":"e_1_3_2_1_81_1","volume-title":"Le","author":"Yang Zhilin","year":"2020","unstructured":"Zhilin Yang , Zihang Dai , Yiming Yang , Jaime Carbonell , Ruslan Salakhutdinov , and Quoc V . Le . 2020 . XLNet: Generalized Autoregressive Pretraining for Language Understanding . arXiv:1906.08237 Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv:1906.08237"},{"key":"e_1_3_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454054"},{"key":"e_1_3_2_1_83_1","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021.","author":"Yue Wang Shafiq Joty","unstructured":"Shafiq Joty Yue Wang , Weishi Wang and Steven C.H. Hoi . 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021. Shafiq Joty Yue Wang, Weishi Wang and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021."},{"key":"e_1_3_2_1_84_1","unstructured":"Andreas Zeller Rahul Gopinath Marcel B\u00f6hme Gordon Fraser and Christian Holler. 2019. The fuzzing book. \t\t\t\t  Andreas Zeller Rahul Gopinath Marcel B\u00f6hme Gordon Fraser and Christian Holler. 2019. The fuzzing book."},{"key":"e_1_3_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2019.00016"}],"event":{"name":"ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Seattle WA USA","acronym":"ISSTA '23","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","AITO"]},"container-title":["Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598067","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597926.3598067","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T13:48:42Z","timestamp":1750168122000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598067"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,12]]},"references-count":86,"alternative-id":["10.1145\/3597926.3598067","10.1145\/3597926"],"URL":"https:\/\/doi.org\/10.1145\/3597926.3598067","relation":{},"subject":[],"published":{"date-parts":[[2023,7,12]]},"assertion":[{"value":"2023-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}