{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T10:26:36Z","timestamp":1790936796763,"version":"4.1.0"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:00:00Z","timestamp":1750291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/publication-rights-and-licensing-policy"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"<jats:p>Exploratory testing (ET) harnesses tester\u2019s knowledge, creativity, and experience to create varying tests that uncover unexpected bugs from the end-user\u2019s perspective. Although ET has proven effective in system-level testing of interactive systems, the need for manual execution has hindered large-scale adoption. In this work, we explore the feasibility, challenges and road ahead of automated scenario-based ET (a.k.a soap opera testing). We conduct a formative study, identifying key insights for effective manual soap opera testing and challenges in automating the process. We then develop a multi-agent system leveraging LLMs and a Scenario Knowledge Graph (SKG) to automate soap opera testing. The system consists of three multi-modal agents, Planner, Player, and Detector that collaborate to execute tests and identify potential bugs. Experimental results demonstrate the potential of automated soap opera testing, but there remains a significant gap compared to manual execution, especially under-explored scenario boundaries and incorrectly identified bugs. Based on the observation, we envision road ahead for the future of automated soap opera testing, focusing on three key aspects: the synergy of neural and symbolic approaches, human-AI co-learning, and the integration of soap opera testing with broader software engineering practices. These insights aim to guide and inspire the future research.<\/jats:p>","DOI":"10.1145\/3715752","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T11:15:34Z","timestamp":1750331734000},"page":"757-778","source":"Crossref","is-referenced-by-count":5,"title":["Automated Soap Opera Testing Directed by LLMs and Scenario Knowledge: Feasibility, Challenges, and Road Ahead"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2410-3229","authenticated-orcid":false,"given":"Yanqi","family":"Su","sequence":"first","affiliation":[{"name":"Australian National University, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7663-1421","authenticated-orcid":false,"given":"Zhenchang","family":"Xing","sequence":"additional","affiliation":[{"name":"CSIRO's Data61, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1424-6290","authenticated-orcid":false,"given":"Chong","family":"Wang","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2011-9618","authenticated-orcid":false,"given":"Chunyang","family":"Chen","sequence":"additional","affiliation":[{"name":"TU Munich, Heilbronn, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2273-1862","authenticated-orcid":false,"given":"Sherry (Xiwei)","family":"Xu","sequence":"additional","affiliation":[{"name":"CSIRO's Data61, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9466-1672","authenticated-orcid":false,"given":"Qinghua","family":"Lu","sequence":"additional","affiliation":[{"name":"CSIRO, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5839-3765","authenticated-orcid":false,"given":"Liming","family":"Zhu","sequence":"additional","affiliation":[{"name":"CSIRO's Data61, Sydney, Australia"},{"name":"UNSW, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9301-4"},{"issue":"6","key":"e_1_3_1_3_2","first-page":"32","article-title":"Session-based test management","volume":"2","author":"Bach Jonathan","year":"2000","unstructured":"Jonathan Bach. 2000. Session-based test management. Software Testing and Quality Engineering Magazine 2, 6 (2000), 32\u201337.","journal-title":"Software Testing and Quality Engineering Magazine"},{"key":"e_1_3_1_4_2","unstructured":"James Bach. 2003. Exploratory testing explained. Online: http:\/\/www.satisfice.com\/articles\/et-article.pdf (2003) 1\u201310."},{"key":"e_1_3_1_5_2","unstructured":"James Bach. 2004. Exploratory testing. The testing practitioner (2004) 253\u2013265."},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380328"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2018.2799957"},{"issue":"2","key":"e_1_3_1_9_2","first-page":"30","article-title":"Soap opera testing","volume":"6","author":"Buwalda Hans","year":"2004","unstructured":"Hans Buwalda. 2004. Soap opera testing. Better Software 6, 2 (2004), 30\u201337.","journal-title":"Better Software"},{"key":"e_1_3_1_10_2","unstructured":"JD Cem Kaner. 2013. An introduction to scenario testing. Florida Institute of Technology Melbourne (2013) 1\u201313."},{"key":"e_1_3_1_11_2","unstructured":"JD Cem Kaner and James Bach. 2006. The nature of exploratory testing. (2006)."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213869"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00084"},{"key":"e_1_3_1_14_2","unstructured":"Google. 2024. Android Debug Bridge (adb). https:\/\/developer.android.com\/studio\/command-line\/adb."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510454.3517063"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion52605.2021.00037"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510454.3522684"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9266-8"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.55"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISESE.2005.1541817"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510203"},{"key":"e_1_3_1_22_2","unstructured":"Cem Kaner. 2008. A tutorial in exploratory testing. Tutorial presented at QUEST2008.(Available online at: http:\/\/www.kaner.com\/pdfs\/QAIExploring.pdf accessed: 26 Jan 2014) (2008)."},{"key":"e_1_3_1_23_2","volume-title":"Testing computer software","author":"Kaner Cem","year":"1999","unstructured":"Cem Kaner, Jack Falk, and Hung Q Nguyen. 1999. Testing computer software. John Wiley & Sons."},{"key":"e_1_3_1_24_2","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459\u20139474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_25_2","first-page":"278","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: Companion Proceedings","author":"Li Ziqiang","year":"2020","unstructured":"Ziqiang Li and Shin Hwei Tan. 2020. Bugine: a bug report recommendation system for Android apps. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: Companion Proceedings. 278\u2013279."},{"key":"e_1_3_1_26_2","unstructured":"Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology (1932)."},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00119"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639118"},{"key":"e_1_3_1_29_2","unstructured":"James Lyndsay and Neil Van Eeden. 2003. Adventures in session-based testing. Workroom Productions Ltd. May 27 (2003)."},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110890"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00205"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP46215.2023.10179324"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/2652524.2652531"},{"key":"e_1_3_1_34_2","unstructured":"Advait Sarkar Andrew D Gordon Carina Negreanu Christian Poelitz Sruti Srinivasa Ragavan and Ben Zorn. 2022. What is it like to program with artificial intelligence? arXiv preprint arXiv:2208.06213 (2022)."},{"key":"e_1_3_1_35_2","volume-title":"Proceedings of the 46th International Conference on Software Engineering (ICSE\u201924)","author":"Sidong Feng","year":"2024","unstructured":"Feng Sidong and Chen Chunyang. 2024. Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models. In Proceedings of the 46th International Conference on Software Engineering (ICSE\u201924)."},{"key":"e_1_3_1_36_2","volume-title":"Elements of survey sampling","author":"Singh Ravindra","year":"2013","unstructured":"Ravindra Singh and Naurang Singh Mangat. 2013. Elements of survey sampling. Vol. 15. Springer Science & Business Media."},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556967"},{"key":"e_1_3_1_38_2","first-page":"51","volume-title":"2023 IEEE\/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)","author":"Su Yanqi","year":"2023","unstructured":"Yanqi Su, Zheming Han, Zhenchang Xing, Xiwei Xu, Liming Zhu, and Qinghua Lu. 2023. SoapOperaTG: A Tool for System Knowledge Graph Based Soap Opera Test Generation. In 2023 IEEE\/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 51\u201354."},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639157"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642777"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380349"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.5555\/644741.644744"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580895"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1800"},{"key":"e_1_3_1_45_2","unstructured":"Hao Wen Yuanchun Li Guohong Liu Shanhui Zhao Tao Yu Toby Jia-Jun Li Shiqi Jiang Yunhao Liu Yaqin Zhang and Yunxin Liu. 2023. Empowering llm to use smartphone for intelligent task automation. arXiv e\u2013prints (2023) arXiv-2308."},{"key":"e_1_3_1_46_2","volume-title":"Exploratory software testing: tips, tricks, tours, and techniques to guide test design","author":"Whittaker James A","year":"2009","unstructured":"James A Whittaker. 2009. Exploratory software testing: tips, tricks, tours, and techniques to guide test design. Pearson Education."},{"issue":"5","key":"e_1_3_1_47_2","first-page":"90","article-title":"Applying session-based testing to medical software","volume":"25","author":"Wood Bill","year":"2003","unstructured":"Bill Wood and David James. 2003. Applying session-based testing to medical software. Medical Device and Diagnostic Industry 25, 5 (2003), 90\u2013103.","journal-title":"Medical Device and Diagnostic Industry"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598138"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","unstructured":"Shengcheng Yu Chunrong Fang Mingzhe Du Zimin Ding Zhenyu Chen and Zhendong Su. 2024. Practical Automated Scenario-based Mobile App Testing. IEEE Transactions on Software Engineering (2024). doi:10.1109\/TSE.2024.3414672","DOI":"10.1109\/TSE.2024.3414672"},{"key":"e_1_3_1_50_2","unstructured":"Zhizheng Zhang Xiaoyi Zhang Wenxuan Xie and Yan Lu. 2023. Responsible task automation: Empowering large language models as responsible task automators. arXiv preprint arXiv:2306.01242 (2023)."},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00030"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715752","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T11:01:07Z","timestamp":1787482867000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715752"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":50,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3715752"],"URL":"https:\/\/doi.org\/10.1145\/3715752","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}