{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T04:05:53Z","timestamp":1790222753665,"version":"4.1.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T00:00:00Z","timestamp":1787270400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100004359","name":"Vetenskapsr\u00e5det","doi-asserted-by":"crossref","award":["Grant No. 2020-05272"],"award-info":[{"award-number":["Grant No. 2020-05272"]}],"id":[{"id":"10.13039\/501100004359","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>Large Language Models (LLMs) and Multi-Agent LLMs (MALLMs) introduce non-determinism, unlike traditional or machine learning software, where models typically produce deterministic outputs for given inputs. LLM-based software requires new approaches to verifying correctness that account for variability within individual test cases, not just across different inputs. This article presents a taxonomy for LLM test case design, informed by research literature and our experience. Each facet is exemplified, and we conduct an LLM-assisted analysis of six open source testing frameworks, perform a sensitivity study of an agent-based system across different model configurations, and provide working examples contrasting atomic and aggregated test cases. We identify key variation points that impact test correctness and highlight open challenges that the research, industry, and open source communities must address as LLMs become integral to software systems. Our taxonomy defines four facets of LLM test case design, addressing ambiguity in both inputs and outputs while establishing best practices. It distinguishes variability in goals, the system under test, and inputs, and introduces two key oracle types: atomic and aggregated. Our findings reveal that current tools treat test executions as isolated events, lack explicit aggregation mechanisms, and inadequately capture variability across model versions, configurations, and repeated runs. This highlights the need for viewing correctness as a distribution of outcomes rather than a binary property, requiring closer collaboration between academia and practitioners to establish mature, variability-aware testing methodologies.<\/jats:p>","DOI":"10.1145\/3806396","type":"journal-article","created":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T13:34:27Z","timestamp":1775223267000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Challenges in Testing Large Language Model-Based Software: A Faceted Taxonomy"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9372-3416","authenticated-orcid":false,"given":"Felix","family":"Dobslaw","sequence":"first","affiliation":[{"name":"Department of Communication, Quality Management and Information Systems, Mid Sweden University, \u00d6stersund, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5179-4205","authenticated-orcid":false,"given":"Robert","family":"Feldt","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2706-1156","authenticated-orcid":false,"given":"Juyeon","family":"Yoon","sequence":"additional","affiliation":[{"name":"School of Computing, KAIST, Daejeon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0836-6993","authenticated-orcid":false,"given":"Shin","family":"Yoo","sequence":"additional","affiliation":[{"name":"School of Computing, KAIST, Daejeon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,21]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Berk Atil Alexa Chittams Liseng Fu Ferhan Ture Lixinyu Xu and Breck Baldwin. 2024. Non-Determinism of \u201cDeterministic\u201d LLM Settings. arXiv:2408.04667. Retrieved from https:\/\/arxiv.org\/abs\/2408.04667"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","first-page":"110542","DOI":"10.1016\/j.jss.2020.110542","article-title":"On testing machine learning programs","volume":"164","author":"Braiek Houssem Ben","year":"2020","unstructured":"Houssem Ben Braiek and Foutse Khomh. 2020. On testing machine learning programs. Journal of Systems and Software 164 (2020), 110542.","journal-title":"Journal of Systems and Software"},{"issue":"3","key":"e_1_3_2_5_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3641289","article-title":"A survey on evaluation of large language models","volume":"15","author":"Chang Yupeng","year":"2024","unstructured":"Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 1\u201345.","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"key":"e_1_3_2_6_2","first-page":"66305","volume-title":"Advances in Neural Information Processing Systems","volume":"37","author":"Chen Shuhao","year":"2024","unstructured":"Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. 2024. RouterDC: Query-based router by dual contrastive learning for assembling large language models. In Advances in Neural Information Processing Systems, Vol. 37, 66305\u201366328."},{"key":"e_1_3_2_7_2","first-page":"94","volume-title":"Proceedings of the International Workshop on Software Technology and Engineering Practice (STEP \u201903)","author":"Chen T. Y.","year":"2004","unstructured":"T. Y. Chen, F.-C. Kuo, T. H. Tse, and Zhi Quan Zhou. 2004. Metamorphic testing and beyond. In Proceedings of the International Workshop on Software Technology and Engineering Practice (STEP \u201903), 94\u2013100."},{"key":"e_1_3_2_8_2","first-page":"11346","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Cho Young Min","year":"2023","unstructured":"Young Min Cho, Sunny Rai, Lyle Ungar, Jo\u00e3o Sedoc, and Sharath Guntuku. 2023. An integrative survey on mental health conversational agents to bridge computer science and medical perspectives. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Houda Bouamor, Juan Pino, and Kalika Bali (Eds.), Association for Computational Linguistics, 11346\u201311369."},{"issue":"4","key":"e_1_3_2_9_2","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1145\/1988042.1988046","article-title":"QuickCheck: A lightweight tool for random testing of Haskell programs","volume":"46","author":"Claessen Koen","year":"2011","unstructured":"Koen Claessen and John Hughes. 2011. QuickCheck: A lightweight tool for random testing of Haskell programs. ACM SIGPLAN Notices 46, 4 (2011), 53\u201364.","journal-title":"ACM SIGPLAN Notices"},{"key":"e_1_3_2_10_2","unstructured":"Comet. 2025. Opik: Open-source LLM Evaluation Framework. Retrieved February 27 2025 from https:\/\/github.com\/comet-ml\/opik"},{"key":"e_1_3_2_11_2","first-page":"78","volume-title":"Proceedings of the 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","author":"Dobslaw Felix","year":"2023","unstructured":"Felix Dobslaw and Robert Feldt. 2023. Similarities of testing programmed and learnt software. In Proceedings of the 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 78\u201381."},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"Felix Dobslaw Robert Feldt Juyeon Yoon and Shin Yoo. 2025. Challenges in testing large language model based software: A faceted taxonomy. arXiv:2503.00481. Retrieved from https:\/\/arxiv.org\/abs\/2503.00481","DOI":"10.1145\/3806396"},{"key":"e_1_3_2_13_2","first-page":"1688","volume-title":"Proceedings of the 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Feldt Robert","year":"2023","unstructured":"Robert Feldt, Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Towards autonomous testing agents via conversational large language models. In Proceedings of the 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1688\u20131693."},{"key":"e_1_3_2_14_2","unstructured":"Giskard. 2025. Giskard. Retrieved October 19 2025 from https:\/\/docs.giskard.ai\/hub\/ui\/index.html"},{"key":"e_1_3_2_15_2","unstructured":"Giskard Team. 2024. Giskard: Secure Your LLM Agents. Retrieved February 27 2025 from https:\/\/www.giskard.aiaccessed"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368094"},{"key":"e_1_3_2_17_2","first-page":"183","volume-title":"Proceedings of 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering","author":"Grunske Lars","year":"2009","unstructured":"Lars Grunske and Pengcheng Zhang. 2009. Monitoring probabilistic properties. In Proceedings of 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, 183\u2013192."},{"key":"e_1_3_2_18_2","unstructured":"Jiawei Gu Xuhui Jiang Zhichao Shi Hexiang Tan Xuehao Zhai Chengjin Xu Wei Li Yinghan Shen Shengjie Ma Honghao Liu et al. 2024. A survey on LLM-as-a-judge. arXiv:2411.15594. Retrieved from https:\/\/arxiv.org\/abs\/2411.15594"},{"key":"e_1_3_2_19_2","first-page":"1","volume-title":"ACM Transactions on Software Engineering and Methodology","volume":"34","author":"He Junda","year":"2025","unstructured":"Junda He, Christoph Treude, and David Lo. 2025. LLM-based multi-agent systems for software engineering: Literature review, vision and the road ahead. ACM Transactions on Software Engineering and Methodology 34, 5 (2025), 1\u201330."},{"key":"e_1_3_2_20_2","unstructured":"Sinclair Hudson Sophia Jit Boyue Caroline Hu and Marsha Chechik. 2024. A software engineering perspective on testing large language models: Research practice tools and benchmarks. arXiv:2406.08216. Retrieved from https:\/\/arxiv.org\/abs\/2406.08216"},{"issue":"3","key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"e12047","DOI":"10.1002\/jeo2.12047","article-title":"ChatGPT can yield valuable responses in the context of orthopaedic trauma surgery","volume":"11","author":"Kaarre Janina","year":"2024","unstructured":"Janina Kaarre, Robert Feldt, B\u00e1lint Zsidai, Eric Hamrin Senorski, Emilia M\u00f6ller Rydberg, Olof Wolf, Sebastian Mukka, Michael M\u00f6ller, and Kristian Samuelsson. 2024. ChatGPT can yield valuable responses in the context of orthopaedic trauma surgery. Journal of Experimental Orthopaedics 11, 3 (2024), e12047.","journal-title":"Journal of Experimental Orthopaedics"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660771"},{"issue":"45","key":"e_1_3_2_23_2","first-page":"1","article-title":"Explainable automated debugging via large language model-driven scientific debugging","volume":"30","author":"Kang Sungmin","year":"2025","unstructured":"Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. 2025. Explainable automated debugging via large language model-driven scientific debugging. Empirical Software Engineering 30, 45 (2025), 1\u201328.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3450837"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00194"},{"key":"e_1_3_2_26_2","first-page":"30","volume-title":"Proceedings of IFIP International Conference on Testing Software and Systems","author":"Gholamzadeh Khoee Arsham","year":"2024","unstructured":"Arsham Gholamzadeh Khoee, Yinan Yu, Robert Feldt, Andris Freimanis, Patrick Andersson Rhodin, and Dhasarathy Parthasarathy. 2024. GoNoGo: An efficient LLM-based multi-agent system for streamlining automotive software release decision-making. In Proceedings of IFIP International Conference on Testing Software and Systems. Springer, 30\u201345."},{"key":"e_1_3_2_27_2","unstructured":"Muhammad Laiq and Felix Dobslaw. 2025. Automatic techniques for issue report classification: A systematic mapping study. arXiv:2505.01469. Retrieved from https:\/\/arxiv.org\/abs\/2505.01469"},{"key":"e_1_3_2_28_2","unstructured":"LangChain. 2025. LangSmith. Retrieved February 27 2025 from https:\/\/www.langchain.com\/langsmith"},{"key":"e_1_3_2_29_2","first-page":"1964","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1: Long Papers","author":"Lu Keming","year":"2024","unstructured":"Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1: Long Papers. Kevin Duh, Helena Gomez, and Steven Bethard (Eds.), Association for Computational Linguistics, 1964\u20131974."},{"key":"e_1_3_2_30_2","first-page":"1","volume-title":"ACM Transactions on Software Engineering and Methodology","volume":"34","author":"Molina Facundo","year":"2025","unstructured":"Facundo Molina, Alessandra Gorla, and Marcelo d\u2019Amorim. 2025. Test oracle automation in the era of LLMs. ACM Transactions on Software Engineering and Methodology 34, 5 (2025), 1\u201324."},{"key":"e_1_3_2_31_2","unstructured":"OWASP. 2025. OWASP Top 10 Vulnerabilities. Retrieved February 27 2025 from https:\/\/owasp.org\/www-project-top-ten\/"},{"key":"e_1_3_2_32_2","unstructured":"Parea AI. 2025. Parea AI: Test and Evaluate your AI Systems. Retrieved February 27 2025 from https:\/\/www.parea.ai\/"},{"key":"e_1_3_2_33_2","first-page":"2984","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201925)","author":"Pei Jiahuan","year":"2025","unstructured":"Jiahuan Pei, Fanghua Ye, Xin Sun, Wentao Deng, Koen Hindriks, and Junxiao Wang. 2025. Conversational education at scale: A multi-LLM agent workflow for procedural learning and pedagogic quality assessment. In Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201925). Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.), Association for Computational Linguistics, 2984\u20132997."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09881-0"},{"key":"e_1_3_2_35_2","unstructured":"Confident AI Team. 2024. DeepEval: An open-source framework for evaluating AI models. Retrieved November 27 2024 from https:\/\/github.com\/confident-ai\/deepeval"},{"key":"e_1_3_2_36_2","unstructured":"PromptFoo Team. 2024. PromptFoo: The AI Prompt Testing Tool. Retrieved November 27 2024 from https:\/\/www.promptfoo.dev\/"},{"key":"e_1_3_2_37_2","unstructured":"Trulens Team. 2025. TruLens: Evaluate and Track LLM Applications. Retrieved February 27 2025 from https:\/\/www.trulens.org\/"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.01.006"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177731118"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730197"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-024-40231-1"},{"key":"e_1_3_2_42_2","volume-title":"Proceedings of the 2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE)","author":"Wang Shuai","year":"2025","unstructured":"Shuai Wang, Yinan Yu, Robert Feldt, and Dhasarathy Parthasarathy. 2025. Automating a complete software test process using LLMs: An automotive case study. In Proceedings of the 2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_43_2","unstructured":"Weights & Biases. 2025. Weights & Biases. Retrieved February 27 2025 from https:\/\/www.wandb.ai"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10393-w"},{"key":"e_1_3_2_45_2","unstructured":"Mingxuan Xiao Yan Xiao Shunhui Ji Hanbo Cai Lei Xue and Pengcheng Zhang. 2024. Assessing the Robustness of LLM-based NLP Software via Automated Testing. arXiv:2412.21016. Retrieved from https:\/\/arxiv.org\/abs\/2412.21016"},{"key":"e_1_3_2_46_2","volume-title":"Proceedings of the International Conference on Learning Representation (ICLR \u201923)","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In Proceedings of the International Conference on Learning Representation (ICLR \u201923)."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST60714.2024.00020"},{"key":"e_1_3_2_48_2","volume-title":"Proceedings of 12th International Conference on Learning Representations","author":"Yue Murong","year":"2023","unstructured":"Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. 2023. Large language model Cascades with mixture of thought representations for cost-efficient reasoning. In Proceedings of 12th International Conference on Learning Representations."},{"key":"e_1_3_2_49_2","first-page":"1592","volume-title":"Proceedings of 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201924)","author":"Zhang Yuntong","year":"2024","unstructured":"Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous program improvement. In Proceedings of 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201924), 1592\u20131604."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2020"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3806396","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:30:34Z","timestamp":1787329834000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3806396"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,21]]},"references-count":49,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3806396"],"URL":"https:\/\/doi.org\/10.1145\/3806396","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,21]]},"assertion":[{"value":"2025-10-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-27","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}