{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T17:40:05Z","timestamp":1791913205645,"version":"4.3.4"},"reference-count":110,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"name":"Science and Technology Development Fund of Macau, Macau SAR","award":["0021\/2023\/RIA1"],"award-info":[{"award-number":["0021\/2023\/RIA1"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,3,31]]},"abstract":"<jats:p>\n                    Testing web forms is an essential activity for ensuring the quality of web applications. It typically involves evaluating the interactions between users and forms. Automated test-case generation remains a challenge for web-form testing: Due to the complex, multi-level structure of Web pages, it can be difficult to automatically capture their inherent contextual information for inclusion in the tests.\n                    <jats:italic toggle=\"yes\">Large Language Models<\/jats:italic>\n                    (LLMs) have shown great potential for contextual text generation. This motivated us to explore how they could generate automated tests for web forms, making use of the contextual information within form elements. To the best of our knowledge, no comparative study examining different LLMs has yet been reported for web-form-test generation. To address this gap in the literature, we conducted a comprehensive empirical study investigating the effectiveness of 11 LLMs on 146 web forms from 30 open source Java web applications. In addition, we propose three HTML-structure-pruning methods to extract key contextual information. The experimental results show that different LLMs can achieve different testing effectiveness, with the GPT-4, GLM-4, and Baichuan2 LLMs generating the best web-form tests. Compared with GPT-4, the other LLMs had difficulty generating appropriate tests for the web forms: Their\n                    <jats:italic toggle=\"yes\">Successfully Submitted Rates<\/jats:italic>\n                    (SSRs)\u2014the proportions of the LLMs-generated web-form tests that could be successfully inserted into the web forms and submitted\u2014decreased by 9.10% to 74.15%. Our findings also show that, for all LLMs, when the designed prompts include complete and clear contextual information about the web forms, more effective web-form tests were generated. Specifically, when using Parser-Processed HTML for Task Prompt (PH-P), the SSR averaged 70.63%, higher than the 60.21% for Raw HTML for Task Prompt (RH-P) and 50.27% for LLM-Processed HTML for Task Prompt (LH-P). With RH-P, GPT-4\u2019s SSR was 98.86%, outperforming models like LLaMa2 (7B) with 34.47% and GLM-4V with 0%. Similarly, with PH-P, GPT-4 reached an SSR of 99.54%, the highest among all models and prompt types. Finally, this article also highlights strategies for selecting LLMs based on performance metrics, and for optimizing the prompt design to improve the quality of the web-form tests.\n                  <\/jats:p>","DOI":"10.1145\/3735553","type":"journal-article","created":{"date-parts":[[2025,5,13]],"date-time":"2025-05-13T12:23:51Z","timestamp":1747139031000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Large Language Models for Automated Web-Form-Test Generation: An Empirical Study"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-7413-9692","authenticated-orcid":false,"given":"Tao","family":"Li","sequence":"first","affiliation":[{"name":"School of Computer Science and Engineering, Macau University of Science and Technology, Taipa, Macao"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-8746-316X","authenticated-orcid":false,"given":"Chenhui","family":"Cui","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Macau University of Science and Technology, Taipa, Macao"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1769-6126","authenticated-orcid":false,"given":"Rubing","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Macau University of Science and Technology, Taipa, Macao and Macau University of Science and Technology Zhuhai MUST Science and Technology Research Institute, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0877-4353","authenticated-orcid":false,"given":"Dave","family":"Towey","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Nottingham Ningbo China, Ningbo, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8621-2420","authenticated-orcid":false,"given":"Lei","family":"Ma","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Tokyo, Bunkyo-ku, Japan and Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,2,13]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"crossref","unstructured":"Parsa Alian Noor Nashid Mobina Shahbandeh and Ali Mesbah. 2024. Semantic constraint inference for web form test generation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201924) 932\u2013944.","DOI":"10.1145\/3650212.3680332"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tcs.2007.07.029"},{"issue":"1","key":"e_1_3_3_4_2","doi-asserted-by":"crossref","first-page":"348","DOI":"10.1109\/TSE.2022.3150618","article-title":"ARTE: Automated generation of realistic test inputs for web APIs","volume":"49","author":"Alonso Juan C.","year":"2022","unstructured":"Juan C. Alonso, Alberto Martin-Lopez, Sergio Segura, Jose Maria Garcia, and Antonio Ruiz-Cortes. 2022. ARTE: Automated generation of realistic test inputs for web APIs. IEEE Transactions on Software Engineering 49, 1 (2022), 348\u2013363.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.354"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.nlp.2024.100065"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-023-00407-8"},{"key":"e_1_3_3_9_2","first-page":"1","volume-title":"Advances in Human-Computer Interaction","author":"Bargas-Avila Javier A.","year":"2011","unstructured":"Javier A. Bargas-Avila, Olivia Brenzikofer, Alexandre N. Tuch, Sandra P. Roth, and Klaus Opwis. 2011. Working towards usable forms on the world wide web: Optimizing multiple selection interface elements. Advances in Human-Computer Interaction 2011 (2011), 1\u20136."},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3593230"},{"key":"e_1_3_3_11_2","first-page":"52","volume-title":"Proceedings of the 9th International Symposium on End User Development (IS-EUD \u201923)","author":"Cal\u00f2 Tommaso","year":"2023","unstructured":"Tommaso Cal\u00f2 and Luigi De Russis. 2023. Leveraging large language models for end-user website generation. In Proceedings of the 9th International Symposium on End User Development (IS-EUD \u201923). Springer, 52\u201361."},{"issue":"3","key":"e_1_3_3_12_2","first-page":"39:1","article-title":"2024. A survey on evaluation of large language models","volume":"15","author":"Chang Yupeng","year":"2024","unstructured":"Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 39:1\u201339:45.","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"key":"e_1_3_3_13_2","first-page":"540","volume-title":"Proceedings of the 3rd International Conference on Integrated Intelligent Computing Communication & Security (ICIIC \u201921)","author":"Sam Cheah Chee","year":"2021","unstructured":"Chee Sam Cheah and Vinesha Selvarajah. 2021. A review of common web application breaching techniques (SQLi, XSS, CSRF). In Proceedings of the 3rd International Conference on Integrated Intelligent Computing Communication & Security (ICIIC \u201921). Atlantis Press, 540\u2013547."},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2023.111165"},{"key":"e_1_3_3_15_2","first-page":"572","volume-title":"Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE \u201924)","author":"Chen Yinghao","year":"2024","unstructured":"Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A framework for LLM-based test generation. In Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE \u201924), 572\u2013576."},{"key":"e_1_3_3_16_2","first-page":"217","volume-title":"Proceedings of the 24th International Conference on Artificial Intelligence in Education (AIED \u201923)","author":"Cochran Keith","year":"2023","unstructured":"Keith Cochran, Clayton Cohn, Jean Francois Rouet, and Peter Hastings. 2023. Improving automated evaluation of student text responses using GPT-3.5 for text data augmentation. In Proceedings of the 24th International Conference on Artificial Intelligence in Education (AIED \u201923), 217\u2013228."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.37.5.553"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2017.2782678"},{"key":"e_1_3_3_19_2","unstructured":"Chenhui Cui Tao Li Junjie Wang Chunyang Chen Dave Towey and Rubing Huang. 2024. Large language models for mobile GUI text input generation: An empirical study. arXiv:2404.08948. Retrieved from https:\/\/arxiv.org\/abs\/2404.08948"},{"issue":"2","key":"e_1_3_3_20_2","first-page":"119","article-title":"Morpheus web testing: A tool for generating test cases for widget based web applications","volume":"21","author":"Neves Romulo de Almeida","year":"2022","unstructured":"Romulo de Almeida Neves, Willian Massami Watanabe, and Rafael Oliveira. 2022. Morpheus web testing: A tool for generating test cases for widget based web applications. Journal of Web Engineering 21, 2 (2022), 119\u2013144.","journal-title":"Journal of Web Engineering"},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","first-page":"111932","DOI":"10.1016\/j.jss.2023.111932","article-title":"Investigating the robustness of locators in template-based Web application testing using a GUI change classification model","volume":"210","author":"De Luca Marco","year":"2024","unstructured":"Marco De Luca, Anna Rita Fasolino, and Porfirio Tramontana. 2024. Investigating the robustness of locators in template-based Web application testing using a GUI change classification model. Journal of Systems and Software 210 (2024), 111932.","journal-title":"Journal of Systems and Software"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.02.005"},{"key":"e_1_3_3_23_2","first-page":"12","volume-title":"Proceedings of the 2023 10th International Conference on Dependable Systems and Their Applications (DSA \u201923). IEEE","author":"Fan Yujia","year":"2023","unstructured":"Yujia Fan, Siyi Wang, Sinan Wang, Yepang Liu, Guoyao Wen, and Qi Rong. 2023. A comprehensive evaluation of Q-learning based automatic web GUI testing. In Proceedings of the 2023 10th International Conference on Dependable Systems and Their Applications (DSA \u201923). IEEE, 12\u201323."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3201209"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-020-09548-1"},{"key":"e_1_3_3_26_2","first-page":"235","volume-title":"Proceedings of the 15th International Conference on Risks and Security of Internet and Systems (CRiSIS \u201920)","author":"Fredj Ouissem Ben","year":"2021","unstructured":"Ouissem Ben Fredj, Omar Cheikhrouhou, Moez Krichen, Habib Hamam, and Abdelouahid Derhab. 2021. An OWASP top ten driven survey on web application protection methods. In Proceedings of the 15th International Conference on Risks and Security of Internet and Systems (CRiSIS \u201920). Springer, 235\u2013252."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-013-0323-0"},{"issue":"5","key":"e_1_3_3_28_2","first-page":"1","article-title":"Automated driver management for Selenium WebDriver","volume":"26","author":"Garc\u00eda Boni","year":"2021","unstructured":"Boni Garc\u00eda, Mario Munoz-Organero, Carlos Alario-Hoyos, and Carlos Delgado Kloos. 2021. Automated driver management for Selenium WebDriver. Empirical Software Engineering 26, 5 (2021), 1\u201351.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_3_29_2","volume-title":"Dynamic HTML: The Definitive Reference","author":"Goodman Danny","year":"2002","unstructured":"Danny Goodman. 2002. Dynamic HTML: The Definitive Reference. O\u2019Reilly Media, Inc."},{"key":"e_1_3_3_30_2","first-page":"1","volume-title":"Proceedings of the 2022 IEEE 2nd International Conference on Mobile Networks and Wireless Communications (ICMNWC)","author":"Goto Hitomi","year":"2022","unstructured":"Hitomi Goto and Vinesha Selvarajah. 2022. Design and implementation of web application penetration testing with cross-site scripting (XSS). In Proceedings of the 2022 IEEE 2nd International Conference on Mobile Networks and Wireless Communications (ICMNWC), 1\u20135."},{"key":"e_1_3_3_31_2","volume-title":"Selenium WebDriver 3 Practical Guide: End-to-End Automation Testing for Web and Mobile Browsers with Selenium WebDriver","author":"Gundecha Unmesh","year":"2018","unstructured":"Unmesh Gundecha and Satya Avasarala. 2018. Selenium WebDriver 3 Practical Guide: End-to-End Automation Testing for Web and Mobile Browsers with Selenium WebDriver (2nd. ed.). Packt Publishing Ltd.","edition":"2"},{"key":"e_1_3_3_32_2","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1007\/978-3-031-44693-1_30","volume-title":"Proceedings of the 12th National CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC \u201923)","author":"Huang Hui","year":"2023","unstructured":"Hui Huang, Shuangzhi Wu, Xinnian Liang, Bing Wang, Yanrui Shi, Peihao Wu, Muyun Yang, and Tiejun Zhao. 2023. Towards making the most of LLM for translation quality estimation. In Proceedings of the 12th National CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC \u201923), 375\u2013386."},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3379592"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2022.3218602"},{"key":"e_1_3_3_35_2","unstructured":"iFlytek Spark Cognitive Large Model. 2024. Retrieved from https:\/\/xinghuo.xfyun.cn\/sparkapi"},{"key":"e_1_3_3_36_2","unstructured":"Internet Archive. 2024. Retrieved from https:\/\/archive.org\/web\/"},{"key":"e_1_3_3_37_2","volume-title":"Forms That Work: Designing Web Forms for Usability","author":"Jarrett Caroline","year":"2009","unstructured":"Caroline Jarrett and Gerry Gaffney. 2009. Forms That Work: Designing Web Forms for Usability. Morgan Kaufmann."},{"key":"e_1_3_3_38_2","unstructured":"James Garrett Jesse. 2011. The Elements of User Experience: User-Centered Design for the Web and Beyond. New Riders Publishing."},{"issue":"3","key":"e_1_3_3_39_2","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1148\/radiographics.16.3.8897632","article-title":"Structured entry of radiology reports using world wide web technology","volume":"16","author":"Kahn Charles E.","year":"1996","unstructured":"Charles E. Kahn Jr, Kun Wang, and Douglas S. Bell. 1996. Structured entry of radiology reports using world wide web technology. Radiographics 16, 3 (1996), 683\u2013691.","journal-title":"Radiographics"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-024-10888-y"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00085"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/bs.adcom.2015.11.007"},{"key":"e_1_3_3_43_2","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1109\/ICST57152.2023.00039","volume-title":"Proceedings of the 16th IEEE Conference on Software Testing, Verification and Validation (ICST \u201923)","author":"Leotta Maurizio","year":"2023","unstructured":"Maurizio Leotta, Boni Garc\u00eda, Filippo Ricca, and Jim Whitehead. 2023. Challenges of end-to-end testing with Selenium WebDriver and how to face them: A survey. In Proceedings of the 16th IEEE Conference on Software Testing, Verification and Validation (ICST \u201923), 339\u2013350."},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649449"},{"issue":"8","key":"e_1_3_3_45_2","first-page":"204:1","article-title":"AceCoder: An effective prompting technique specialized in code generation","volume":"33","author":"Li Jia","year":"2024","unstructured":"Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2024. AceCoder: An effective prompting technique specialized in code generation. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 204:1\u2013204:26.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549099"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2014.02.001"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549081"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00119"},{"key":"e_1_3_3_50_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering (ICSE \u201924)","author":"Liu Zhe","year":"2024","unstructured":"Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2024. Make LLM a testing expert: Bringing human-like interaction to mobile GUI testing via functionality-aware decisions. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering (ICSE \u201924), 1\u201313."},{"key":"e_1_3_3_51_2","first-page":"4011","volume-title":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI \u201916)","author":"Lukanov Kristiyan","year":"2016","unstructured":"Kristiyan Lukanov, Horia A. Maior, and Max L. Wilson. 2016. Using fNIRS in usability testing: Understanding the effect of web form layout on mental workload. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI \u201916), 4011\u20134016."},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.5555\/863219"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-021-09575-w"},{"key":"e_1_3_3_54_2","first-page":"141","volume-title":"Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation (ICST \u201912)","author":"McMinn Phil","year":"2012","unstructured":"Phil McMinn, Muzammil Shahbaz, and Mark Stevenson. 2012. Search-based test input generation for string data types using the results of web queries. In Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation (ICST \u201912), 141\u2013150."},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.28"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCS.2007.30"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/96267.96279"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3605943"},{"key":"e_1_3_3_59_2","first-page":"388","volume-title":"Proceedings of the XXXVII Brazilian Symposium on Software Engineering (SBES \u201923)","author":"Santos de Moura Thiago","year":"2023","unstructured":"Thiago Santos de Moura, Everton L. G. Alves, Hugo Feitosa de Figueir\u00eado, and Cl\u00e1udio de Souza Baptista. 2023. Cytestion: Automated GUI testing for web applications. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering (SBES \u201923), 388\u2013397."},{"key":"e_1_3_3_60_2","unstructured":"Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes and Ajmal Mian. 2023. A comprehensive overview of large language models. arXiv:2307.06435. Retrieved from https:\/\/arxiv.org\/abs\/2307.06435"},{"issue":"3","key":"e_1_3_3_61_2","doi-asserted-by":"crossref","first-page":"e1760","DOI":"10.1002\/stvr.1760","article-title":"Generating and selecting resilient and maintainable locators for Web automated testing","volume":"31","author":"Nguyen Vu","year":"2021","unstructured":"Vu Nguyen, Thanh To, and Gia-Han Diep. 2021. Generating and selecting resilient and maintainable locators for Web automated testing. Software Testing, Verification and Reliability 31, 3 (2021), e1760.","journal-title":"Software Testing, Verification and Reliability"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3697010"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2023.3248401"},{"key":"e_1_3_3_64_2","doi-asserted-by":"crossref","first-page":"2339","DOI":"10.1109\/SP46215.2023.10179324","volume-title":"Proceedings of the 44th IEEE Symposium on Security and Privacy (SP \u201923)","author":"Pearce Hammond","year":"2023","unstructured":"Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language models. In Proceedings of the 44th IEEE Symposium on Security and Privacy (SP \u201923), 2339\u20132356."},{"key":"e_1_3_3_65_2","unstructured":"Laura Plein and Tegawend\u00e9 F. Bissyand\u00e9. 2023. Can LLMs demystify bug reports? arXiv:2310.06310. Retrieved from https:\/\/arxiv.org\/abs\/2310.06310"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.03.016"},{"issue":"8","key":"e_1_3_3_67_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.5555\/381473.381476"},{"key":"e_1_3_3_69_2","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1109\/ICSE.1998.671118","volume-title":"Proceedings of the 20th International Conference on Software Engineering (ICSE \u201998)","author":"Rothermel Gregg","year":"1998","unstructured":"Gregg Rothermel, Lixin Li, Christopher DuPuis, and Margaret Burnett. 1998. What you see is what you test: A methodology for testing form-based visual programs. In Proceedings of the 20th International Conference on Software Engineering (ICSE \u201998), 198\u2013207."},{"key":"e_1_3_3_70_2","first-page":"7","volume-title":"Proceedings of the 2023 IEEE International Conference on Data and Software Engineering (ICoDSE \u201923)","author":"Rukmono Satrio Adi","year":"2023","unstructured":"Satrio Adi Rukmono, Lina Ochoa, and Michel R. V. Chaudron. 2023. Achieving high-level software component summarization via hierarchical chain-of-thought prompting and static code analysis. In Proceedings of the 2023 IEEE International Conference on Data and Software Engineering (ICoDSE \u201923), 7\u201312."},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-023-00816-5"},{"key":"e_1_3_3_72_2","first-page":"217","volume-title":"Proceedings of the IEEE 30th International Symposium on Software Reliability Engineering (ISSRE \u201919)","author":"Santiago Dionny","year":"2019","unstructured":"Dionny Santiago, Justin Phillips, Patrick Alt, Brian Muras, Tariq M. King, and Peter J. Clarke. 2019. Machine learning and constraint solving for automated form testing. In Proceedings of the IEEE 30th International Symposium on Software Reliability Engineering (ISSRE \u201919), 217\u2013227."},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2018.06.004"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3334955"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.799955"},{"key":"e_1_3_3_76_2","doi-asserted-by":"crossref","unstructured":"Mirjam Seckler Silvia Heinz Javier A. Bargas-Avila Klaus Opwis and Alexandre N. Tuch. 2014. Designing usable web forms: Empirical evaluation of web form improvement guidelines. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI \u201914) 1275\u20131284.","DOI":"10.1145\/2556288.2557265"},{"key":"e_1_3_3_77_2","unstructured":"Selenium. 2024. Retrieved from https:\/\/www.selenium.dev\/"},{"key":"e_1_3_3_78_2","unstructured":"Selenium WebDriver. 2024. Retrieved from https:\/\/www.selenium.dev\/documentation\/webdriver\/"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2014.04.008"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3624724"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.1957.10501091"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_3_83_2","doi-asserted-by":"crossref","unstructured":"Paolo Tonella. 2004. Evolutionary testing of classes. In Proceedings of the 13th International Symposium on Software Testing and Analysis (ISSTA \u201904) 119\u2013128.","DOI":"10.1145\/1007512.1007528"},{"key":"e_1_3_3_84_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524481.3527220"},{"issue":"2","key":"e_1_3_3_86_2","first-page":"101","article-title":"A critique and improvement of the CL common language effect size statistics of McGraw and Wong","volume":"25","author":"Vargha Andr\u00e1s","year":"2000","unstructured":"Andr\u00e1s Vargha and Harold D. Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101\u2013132.","journal-title":"Journal of Educational and Behavioral Statistics"},{"key":"e_1_3_3_87_2","first-page":"5998","volume-title":"Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NeurIPS \u201917)","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NeurIPS \u201917), 5998\u20136008."},{"key":"e_1_3_3_88_2","first-page":"144","volume-title":"Proceedings of the International Conference on Advances in Image Processing (ICAIP \u201917)","author":"Vila Elior","year":"2017","unstructured":"Elior Vila, Galia Novakova, and Diana Todorova. 2017. Automation testing framework for web applications with Selenium WebDriver: Opportunities and threats. In Proceedings of the International Conference on Advances in Image Processing (ICAIP \u201917), 144\u2013150."},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_3_90_2","first-page":"125","volume-title":"Proceedings of the 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME \u201924)","author":"Wang Siyi","year":"2024","unstructured":"Siyi Wang, Sinan Wang, Yujia Fan, Xiaolei Li, and Yepang Liu. 2024. Leveraging large vision-language model for better automatic web GUI testing. In Proceedings of the 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME \u201924), 125\u2013137."},{"key":"e_1_3_3_91_2","first-page":"22964","volume-title":"Proceedings of the 39th International Conference on Machine Learning (ICML \u201922)","author":"Wang Thomas","unstructured":"Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022. What language model architecture and pretraining objective works best for zero-shot generalization? In Proceedings of the 39th International Conference on Machine Learning (ICML \u201922), 22964\u201322984."},{"issue":"5","key":"e_1_3_3_92_2","doi-asserted-by":"crossref","first-page":"1063","DOI":"10.1109\/TKDE.2013.11","article-title":"A probabilistic approach to string transformation","volume":"26","author":"Wang Ziqi","year":"2013","unstructured":"Ziqi Wang, Gu Xu, Hang Li, and Ming Zhang. 2013. A probabilistic approach to string transformation. IEEE Transactions on Knowledge and Data Engineering 26, 5 (2013), 1063\u20131075.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"e_1_3_3_93_2","unstructured":"Web-Element Location Strategies. 2024. Retrieved from https:\/\/www.selenium.dev\/documentation\/webdriver\/elements\/locators\/"},{"key":"e_1_3_3_94_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 24824\u201324837.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_95_2","doi-asserted-by":"publisher","DOI":"10.2307\/3001968"},{"key":"e_1_3_3_96_2","unstructured":"Fangzhou Wu Ning Zhang Somesh Jha Patrick McDaniel and Chaowei Xiao. 2024. A new era in LLM security: Exploring security concerns in real-world LLM-based systems. arXiv:2402.18649. Retrieved from https:\/\/arxiv.org\/abs\/2402.18649"},{"key":"e_1_3_3_97_2","unstructured":"Danning Xie Byungwoo Yoo Nan Jiang Mijung Kim Lin Tan Xiangyu Zhang and Judy S. Lee. 2023. Impact of large language models on generating software specifications. arXiv:2306.03324. Retrieved from https:\/\/arxiv.org\/abs\/2306.03324"},{"key":"e_1_3_3_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3140544"},{"key":"e_1_3_3_99_2","unstructured":"Aiyuan Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian Chao Yin Chenxu Lv Da Pan Dian Wang Dong Yan et al. 2023. Baichuan 2: Open large-scale language models. arXiv:2309.10305. Retrieved from https:\/\/arxiv.org\/abs\/2309.10305"},{"key":"e_1_3_3_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649506"},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.hcc.2024.100211"},{"key":"e_1_3_3_102_2","first-page":"206","volume-title":"Proceedings of the IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS \u201923)","author":"Yu Shengcheng","year":"2023","unstructured":"Shengcheng Yu, Chunrong Fang, Yuchen Ling, Chentian Wu, and Zhenyu Chen. 2023. LLM for test script generation and migration: Challenges, capabilities, and opportunities. In Proceedings of the IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS \u201923), 206\u2013217."},{"key":"e_1_3_3_103_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Zeng Aohan","year":"2023","unstructured":"Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2023. GLM-130B: An open bilingual pre-trained model. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)."},{"key":"e_1_3_3_104_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations (ICLR \u201924)","author":"Zhang Biao","year":"2024","unstructured":"Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2024. When scaling meets LLM finetuning: The effect of data, model and finetuning method. In Proceedings of the 12th International Conference on Learning Representations (ICLR \u201924)."},{"key":"e_1_3_3_105_2","first-page":"769","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL \u201923)","author":"Zhang Kechi","year":"2023","unstructured":"Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL \u201923), 769\u2013787."},{"key":"e_1_3_3_106_2","unstructured":"Ting Zhang Ivana Clairine Irsan Ferdian Thung and David Lo. 2023. CUPID: Leveraging ChatGPT for more accurate duplicate bug report detection. arXiv:2308.10022. Retrieved from https:\/\/arxiv.org\/abs\/2308.10022"},{"key":"e_1_3_3_107_2","unstructured":"Ying Zhang Wenjia Song Zhengjie Ji Danfeng Yao and Na Meng. 2023. How well does LLM generate security tests? arXiv:2310.00710. Retrieved from https:\/\/arxiv.org\/abs\/2310.00710"},{"key":"e_1_3_3_108_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC57700.2023.00275"},{"issue":"2","key":"e_1_3_3_109_2","first-page":"20:1","article-title":"Explainability for large language models: A survey","volume":"15","author":"Zhao Haiyan","year":"2024","unstructured":"Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology 15, 2 (2024), 20:1\u201320:38.","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"issue":"3","key":"e_1_3_3_110_2","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1002\/stvr.414","article-title":"Automatic string test data generation for detecting domain errors","volume":"20","author":"Zhao Ruilian","year":"2010","unstructured":"Ruilian Zhao, Michael R. Lyu, and Yinghua Min. 2010. Automatic string test data generation for detecting domain errors. Software Testing, Verification and Reliability 20, 3 (2010), 209\u2013236.","journal-title":"Software Testing, Verification and Reliability"},{"key":"e_1_3_3_111_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et al. 2023. A survey of large language models. arXiv:2303.18223. Retrieved from https:\/\/arxiv.org\/abs\/2303.18223"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3735553","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T14:38:30Z","timestamp":1770993510000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3735553"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,13]]},"references-count":110,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,3,31]]}},"alternative-id":["10.1145\/3735553"],"URL":"https:\/\/doi.org\/10.1145\/3735553","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,13]]},"assertion":[{"value":"2024-09-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-06","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}