{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T07:17:26Z","timestamp":1783408646897,"version":"3.54.6"},"publisher-location":"New York, NY, USA","reference-count":27,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T00:00:00Z","timestamp":1775952000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,12]]},"DOI":"10.1145\/3786181.3788709","type":"proceedings-article","created":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T06:28:24Z","timestamp":1783405704000},"page":"192-196","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["LLM-Powered On-Demand Test Suites in Self-Graded Student Programming Assignments"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6721-1959","authenticated-orcid":false,"given":"Chang","family":"Liu","sequence":"first","affiliation":[{"name":"Ohio University, Athens, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,6]]},"reference":[{"key":"e_1_3_3_1_2_2","doi-asserted-by":"crossref","unstructured":"Juan Altmayer\u00a0Pizzorno and Emery\u00a0D Berger. 2025. CoverUp: Effective High Coverage Test Generation for Python. Proceedings of the ACM on Software Engineering 2 FSE (2025) 2897\u20132919.","DOI":"10.1145\/3729398"},{"key":"e_1_3_3_1_3_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2108.07732 (2021)."},{"key":"e_1_3_3_1_4_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique\u00a0Ponde de Oliveira\u00a0Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe\u00a0Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William\u00a0Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew\u00a0N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2107.03374 (2021)."},{"key":"e_1_3_3_1_5_2","unstructured":"Neil Chowdhury James Aung Chan\u00a0Jun Shern Oliver Jaffe Dane Sherburn Giulio Starace Evan Mays Rachel Dias Marwan Aljubeh Mia Glaese Carlos\u00a0E. Jimenez John Yang Leyton Ho Tejal Patwardhan Kevin Liu and Aleksander Madry. 2024. Introducing SWE-bench Verified. https:\/\/openai.com\/index\/introducing-swe-bench-verified\/."},{"key":"e_1_3_3_1_6_2","unstructured":"Fazeleh Dehghaniashkezari. 2024. Investigating the usefulness of LLMs in Test Case Generation. Master\u2019s Thesis. Stockholm University Faculty of Social Sciences Department of Computer and Systems Sciences. https:\/\/su.diva-portal.org\/smash\/record.jsf?pid=diva2:1955685 Available from: 2025-04-30."},{"key":"e_1_3_3_1_7_2","unstructured":"Xiang Deng Jeff Da Edwin Pan Yannis\u00a0Yiming He Charles Ide Kanak Garg Niklas Lauffer Andrew Park Nitin Pasari Chetan Rane et\u00a0al. 2025. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2509.16941 (2025)."},{"key":"e_1_3_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623343"},{"key":"e_1_3_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"e_1_3_3_1_10_2","unstructured":"Paul Gauthier. 2024. Aider: AI Pair Programming in Your Terminal. arXiv preprint (2024)."},{"key":"e_1_3_3_1_11_2","unstructured":"Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora Ethan Guo Collin Burns Samir Puranik Horace He Dawn Song and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2105.09938 (2021)."},{"key":"e_1_3_3_1_12_2","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. arxiv:https:\/\/arXiv.org\/abs\/2308.10620\u00a0[cs.SE]"},{"key":"e_1_3_3_1_13_2","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33 8 (2024) 1\u201379."},{"key":"e_1_3_3_1_14_2","unstructured":"Naman Jain King Han Alex Gu Wen-Ding Li Fanjia Yan Tianjun Zhang Sida Wang Armando Solar-Lezama Koushik Sen and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2403.07974 (2024)."},{"key":"e_1_3_3_1_15_2","volume-title":"ICLR","author":"Jimenez Carlos\u00a0E.","year":"2023","unstructured":"Carlos\u00a0E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In ICLR."},{"key":"e_1_3_3_1_16_2","unstructured":"Yuhang Lai Chengxi Li Yiming Wang Tianyi Zhang Ruiqi Zhong Luke Zettlemoyer Wen tau Yih Daniel Fried Sida Wang and Tao Yu. 2022. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2211.11501 (2022)."},{"key":"e_1_3_3_1_17_2","doi-asserted-by":"crossref","unstructured":"Jiawei Liu Chunqiu\u00a0Steven Xia Yuyao Wang and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2023) 21558\u201321572.","DOI":"10.52202\/075280-0943"},{"key":"e_1_3_3_1_18_2","unstructured":"Satoshi Masuda Satoshi Kouzawa Kyousuke Sezai Hidetoshi Suhara Yasuaki Hiruta and Kunihiro Kudou. 2025. Generating High-Level Test Cases from Requirements using LLM: An Industry Study. arxiv:https:\/\/arXiv.org\/abs\/2510.03641\u00a0[cs.SE] https:\/\/arxiv.org\/abs\/2510.03641"},{"key":"e_1_3_3_1_19_2","doi-asserted-by":"crossref","unstructured":"Christoph Treude and Margaret-Anne Storey. 2025. Generative AI and Empirical Software Engineering: A Paradigm Shift. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2502.08108 (2025).","DOI":"10.1109\/AIware69974.2025.00033"},{"key":"e_1_3_3_1_20_2","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan\u00a0N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_3_3_1_21_2","doi-asserted-by":"crossref","unstructured":"Junjie Wang Yuchao Huang Chunyang Chen Zhe Liu Song Wang and Qing Wang. 2024. Software testing with large language models: Survey landscape and vision. IEEE Transactions on Software Engineering 50 4 (2024) 911\u2013936.","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-naacl.197"},{"key":"e_1_3_3_1_23_2","volume-title":"The Thirteenth International Conference on Learning Representations","author":"White Colin","year":"2025","unstructured":"Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep\u00a0Singh Sandha, Siddartha\u00a0Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. LiveBench: A Challenging, Contamination-Free LLM Benchmark. In The Thirteenth International Conference on Learning Representations."},{"key":"e_1_3_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680323"},{"key":"e_1_3_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.411"},{"key":"e_1_3_3_1_26_2","unstructured":"Linghao Zhang Shilin He Chaoyun Zhang Yu Kang Bowen Li Chengxing Xie Junhao Wang Maoquan Wang Yufan Huang Shengyu Fu Elsie Nallipogu Qingwei Lin Yingnong Dang Saravan Rajmohan and Dongmei Zhang. 2025. SWE-bench Goes Live!arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2505.23419 (2025)."},{"key":"e_1_3_3_1_27_2","unstructured":"Quanjun Zhang Chunrong Fang Yang Xie Yaxin Zhang Yun Yang Weisong Sun Shengcheng Yu and Zhenyu Chen. 2023. A survey on large language models for software engineering. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2312.15223 (2023)."},{"key":"e_1_3_3_1_28_2","unstructured":"Tianyu Zhang Luchen Tan Zihao Wu Jing Jiang and Min Yang. 2023. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2306.03091 (2023)."}],"event":{"name":"LLM4Code '26: 3rd International Workshop on Large Language Models For Code","location":"Rio de Janeiro Brazil","acronym":"LLM4Code '26","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS","Faculty of Engineering of University of Porto"]},"container-title":["Proceedings of the 3rd International Workshop on Large Language Models For Code"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3786181.3788709","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T06:38:29Z","timestamp":1783406309000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3786181.3788709"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,12]]},"references-count":27,"alternative-id":["10.1145\/3786181.3788709","10.1145\/3786181"],"URL":"https:\/\/doi.org\/10.1145\/3786181.3788709","relation":{},"subject":[],"published":{"date-parts":[[2026,4,12]]},"assertion":[{"value":"2026-07-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}