{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:52:20Z","timestamp":1791460340378,"version":"4.3.4"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["RS-2023-00208998"],"award-info":[{"award-number":["RS-2023-00208998"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["RS-2021-NR060080"],"award-info":[{"award-number":["RS-2021-NR060080"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Institute of Information & Communications Technology Planning & Evaluation","award":["RS-2022-II220995"],"award-info":[{"award-number":["RS-2022-II220995"]}]},{"name":"Swedish Science Council","award":["2020-05272"],"award-info":[{"award-number":["2020-05272"]}]},{"DOI":"10.13039\/501100003725","name":"Chalmers Foundation","doi-asserted-by":"publisher","award":["CAEP-26-Feldt"],"award-info":[{"award-number":["CAEP-26-Feldt"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003725","name":"Wallenberg AI, Autonomous Systems and Software Programme","doi-asserted-by":"publisher","award":["BoundMiner"],"award-info":[{"award-number":["BoundMiner"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Software increasingly relies on the emergent capabilities of Large Language Models (LLMs), from natural language understanding to program analysis and generation. Yet testing them on specific tasks remains difficult and costly: many prompts lack ground truths, forcing reliance on human judgments, while existing test adequacy measures typically rely on output uncertainty and thus are only available after full inference. A key challenge is to assess how useful a test input is in a way that reflects the demands of the task, ideally before even generating any output. We introduce Clotho, a task-specific, pre-generation test adequacy measure that estimates input difficulty directly from LLM hidden states. Given a large pool of unlabelled inputs for a specific task, Clotho uses a Gaussian Mixture Model (GMM) to adaptively sample the most informative cases for human labelling. Based on this reference set the GMM can then rank unseen inputs by their likelihood of failure. In our empirical evaluation across eight benchmark tasks and three open-weight LLMs, Clotho can predict failures with a ROC-AUC of 0.716, after labelling reference sets that are on average only 5.4% of inputs. It does so without generating any outputs, thereby significantly reducing LLM execution costs compared to output-based uncertainty or confidence measures. Comparison of Clotho and these post-generation adequacy measures shows that the two approaches complement each other. Crucially, we show that adequacy scores learnt from open-weight LLMs transfer effectively to proprietary models, extending the applicability of the approach. When prioritising test inputs for proprietary models, Clotho increases the average number of failing inputs from 18.7 to 42.5 out of 100, compared to random prioritisation.<\/jats:p>","DOI":"10.1145\/3797114","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"1910-1932","source":"Crossref","is-referenced-by-count":3,"title":["Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2706-1156","authenticated-orcid":false,"given":"Juyeon","family":"Yoon","sequence":"first","affiliation":[{"name":"Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8917-7302","authenticated-orcid":false,"given":"Somin","family":"Kim","sequence":"additional","affiliation":[{"name":"Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5179-4205","authenticated-orcid":false,"given":"Robert","family":"Feldt","sequence":"additional","affiliation":[{"name":"Chalmers University of Technology, Computer Science and Engineering, Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0836-6993","authenticated-orcid":false,"given":"Shin","family":"Yoo","sequence":"additional","affiliation":[{"name":"Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Principal component analysis","author":"Abdi Herv\u00e9","year":"2010","unstructured":"Herv\u00e9 Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statistics 2, 4 (2010), 433-459."},{"key":"e_1_2_1_2_1","unstructured":"DAIR AI. 2025. Prompt Engieering Guide: https:\/\/www.promptingguide.ai."},{"key":"e_1_2_1_3_1","unstructured":"Giskard AI. 2024. Giskard. https:\/\/github.com\/Giskard-AI\/giskard."},{"key":"e_1_2_1_4_1","unstructured":"AI@Meta. 2024. Llama 3 Model Card. (2024). https:\/\/github.com\/meta-llama\/llama3\/blob\/main\/MODEL_CARD.md"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.68"},{"key":"e_1_2_1_6_1","volume-title":"2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188-2200","author":"Bouzenia Islem","year":"2025","unstructured":"Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188-2200."},{"key":"e_1_2_1_7_1","volume-title":"INSIDE: LLMs' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744","author":"Chen Chao","year":"2024","unstructured":"Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. INSIDE: LLMs' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744 (2024)."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622737.1622744"},{"key":"e_1_2_1_10_1","unstructured":"CShorten. 2022. arXiv Dataset: Hugging Face Dataset Card. https:\/\/huggingface.co\/datasets\/CShorten\/Last-Week-on- ML-ArXiv. Accessed: 2025-09-11."},{"key":"e_1_2_1_11_1","volume-title":"JSON Repair: How to fix JSON and validate it with ease. https:\/\/jsoneditor","author":"de Jong Jos","year":"2025","unstructured":"Jos de Jong. 2022. JSON Repair: How to fix JSON and validate it with ease. https:\/\/jsoneditoronline.org\/indepth\/parse\/fixjson\/. Accessed: 2025-09-11."},{"key":"e_1_2_1_12_1","volume-title":"The mahalanobis distance. Chemometrics and intelligent laboratory systems 50, 1","author":"Maesschalck Roy De","year":"2000","unstructured":"Roy De Maesschalck, Delphine Jouan-Rimbaud, and D\u00e9sir\u00e9 L Massart. 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems 50, 1 (2000), 1-18."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of International Symposium on Software Testing and Analysis. 102-112","author":"Elbaum Sebastian G.","year":"2000","unstructured":"Sebastian G. Elbaum, Alexey G. Malishevsky, and Gregg Rothermel. 2000. Prioritizing test cases for regression testing. In Proceedings of International Symposium on Software Testing and Analysis. 102-112."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.99"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-024-07421-0"},{"key":"e_1_2_1_16_1","volume-title":"WordNet: An electronic lexical database","author":"Fellbaum Christiane","unstructured":"Christiane Fellbaum. 1998. WordNet: An electronic lexical database. MIT press."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-1809.1936.tb02137.x"},{"key":"e_1_2_1_18_1","volume-title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 [cs.CL] https:\/\/arxiv.org\/abs\/2101.00027","author":"Gao Leo","year":"2020","unstructured":"Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 [cs.CL] https:\/\/arxiv.org\/abs\/2101.00027"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-025-09422-z"},{"key":"e_1_2_1_20_1","volume-title":"GitHub typo corpus: A large-scale multilingual dataset of misspellings and grammatical errors. arXiv preprint arXiv:1911.12893","author":"Hagiwara Masato","year":"2019","unstructured":"Masato Hagiwara and Masato Mita. 2019. GitHub typo corpus: A large-scale multilingual dataset of misspellings and grammatical errors. arXiv preprint arXiv:1911.12893 (2019)."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"4128","author":"Havtorn Jakob D.","year":"2021","unstructured":"Jakob D. Havtorn, Jes Frellsen, S\u00f8ren Hauberg, and Lars Maal\u00f8e. 2021. Hierarchical VAEs Know What They Don't Know. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 4117-4128. https:\/\/proceedings.mlr.press\/v139\/havtorn21a.html"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.608"},{"key":"e_1_2_1_23_1","volume-title":"Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654","author":"He Pengcheng","year":"2020","unstructured":"Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020)."},{"key":"e_1_2_1_24_1","volume-title":"Look before you leap: An exploratory study of uncertainty analysis for large language models","author":"Huang Yuheng","year":"2025","unstructured":"Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2025. Look before you leap: An exploratory study of uncertainty analysis for large language models. IEEE Transactions on Software Engineering (2025)."},{"key":"e_1_2_1_25_1","unstructured":"Albert Q. Jiang Alexandre Sablayrolles Arthur Mensch Chris Bamford Devendra Singh Chaplot Diego de las Casas Florian Bressand Gianna Lengyel Guillaume Lample Lucile Saulnier L\u00e9lio Renard Lavaud Marie-Anne Lachaux Pierre Stock Teven Le Scao Thibaut Lavril Thomas Wang Timoth\u00e9e Lacroix and William El Sayed. 2023. Mistral 7B. arXiv:2310.06825 [cs.CL] https:\/\/arxiv.org\/abs\/2310.06825"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3660771"},{"key":"e_1_2_1_27_1","first-page":"1039","volume-title":"Proceedings of the 41th International Conference on Software Engineering (ICSE","author":"Kim Jinhan","year":"2019","unstructured":"Jinhan Kim, Robert Feldt, and Shin Yoo. 2019. Guiding Deep Learning System Testing using Surprise Adequacy. In Proceedings of the 41th International Conference on Software Engineering (ICSE 2019). IEEE Press, 1039-1049."},{"key":"e_1_2_1_28_1","first-page":"2","article-title":"Evaluating Surprise Adequacy for Deep Learning System Testing","volume":"32","author":"Kim Jinhan","year":"2022","unstructured":"Jinhan Kim, Robert Feldt, and Shin Yoo. 2022. Evaluating Surprise Adequacy for Deep Learning System Testing. ACM Transactions on Software Engineering and Methodology 32, 2 (June 2022), 1-29.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_2_1_29_1","first-page":"1466","volume-title":"Proceedings of ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE Industry Track) (ESEC\/FSE","author":"Kim Jinhan","year":"2020","unstructured":"Jinhan Kim, Jeongil Ju, Robert Feldt, and Shin Yoo. 2020. Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving. In Proceedings of ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE Industry Track) (ESEC\/FSE 2020). 1466-1476."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpurol.2023.05.018"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 2nd ACM\/IEEE International Conference on Automated Software Testing (AST","author":"Kim Seah","year":"2021","unstructured":"Seah Kim and Shin Yoo. 2021. Multimodal Surprise Adequacy Analysis of Inputs for Natural Language Processing DNN Models. In Proceedings of the 2nd ACM\/IEEE International Conference on Automated Software Testing (AST 2021)."},{"key":"e_1_2_1_32_1","volume-title":"Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927","author":"Kossen Jannik","year":"2024","unstructured":"Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927 (2024)."},{"key":"e_1_2_1_33_1","unstructured":"LangChain. 2024. LangChain Prompt Template. https:\/\/python.langchain.com\/docs\/concepts\/prompt_templates\/."},{"key":"e_1_2_1_34_1","unstructured":"LangChain. 2024. LangSmith. https:\/\/docs.smith.langchain.com\/evaluation\/concepts."},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Hugo Lauren\u00e7on Lucile Saulnier Thomas Wang Christopher Akiki Albert Villanova del Moral Teven Le Scao Leandro Von Werra Chenghao Mou Eduardo Gonz\u00e1lez Ponferrada Huu Nguyen J\u00f6rg Frohberg Mario \u0160a\u0161ko Quentin Lhoest Angelina McMillan-Major Gerard Dupont Stella Biderman Anna Rogers Loubna Ben allal Francesco De Toni Giada Pistilli Olivier Nguyen Somaieh Nikpoor Maraim Masoud Pierre Colombo Javier de la Rosa Paulo Villegas Tristan Thrush Shayne Longpre Sebastian Nagel Leon Weber Manuel Mu\u00f1oz Jian Zhu Daniel Van Strien Zaid Alyafeai Khalid Almubarak Minh Chien Vu Itziar Gonzalez-Dios Aitor Soroa Kyle Lo Manan Dey Pedro Ortiz Suarez Aaron Gokaslan Shamik Bose David Adelani Long Phan Hieu Tran Ian Yu Suhas Pai Jenny Chim Violette Lepercq Suzana Ilic Margaret Mitchell Sasha Alexandra Luccioni and Yacine Jernite. 2023. The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. arXiv:2303.03915 [cs.CL] https:\/\/arxiv.org\/abs\/2303.03915","DOI":"10.52202\/068431-2306"},{"key":"e_1_2_1_36_1","volume-title":"Bug in the code stack: Can llms find bugs in large python code stacks. arXiv preprint arXiv:2406.15325","author":"Lee Hokyung","year":"2024","unstructured":"Hokyung Lee, Sumanyu Sharma, and Bing Hu. 2024. Bug in the code stack: Can llms find bugs in large python code stacks. arXiv preprint arXiv:2406.15325 (2024)."},{"key":"e_1_2_1_37_1","volume-title":"Advances in Neural Information Processing Systems (NeurIPS","author":"Lee Kimin","year":"2018","unstructured":"Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A Simple Unified Framework for Detecting Outof-Distribution Samples and Adversarial Attacks. In Advances in Neural Information Processing Systems (NeurIPS 2018). https:\/\/papers.neurips.cc\/paper\/7947-a-simple-unified-framework-for-detecting-out-of-distribution-samplesand-adversarial-attacks.pdf Also available as arXiv:1807.03888."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_2_1_39_1","first-page":"2579","article-title":"Visualizing data using t-SNE","author":"van der Maaten Laurens","year":"2008","unstructured":"Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, Nov (2008), 2579-2605.","journal-title":"Journal of machine learning research 9"},{"key":"e_1_2_1_40_1","volume-title":"Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896","author":"Manakul Potsawee","year":"2023","unstructured":"Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896 (2023)."},{"key":"e_1_2_1_41_1","unstructured":"Roger Mitton et al. 1980. Birkbeck spelling error corpus. Oxford Text Archive Legacy Collection (1980)."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_2_1_44_1","unstructured":"PromptFoo. 2024. PromptFoo. https:\/\/github.com\/promptfoo\/promptfoo."},{"key":"e_1_2_1_45_1","volume-title":"A survey of deep active learning. ACM computing surveys (CSUR) 54, 9","author":"Ren Pengzhen","year":"2021","unstructured":"Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. 2021. A survey of deep active learning. ACM computing surveys (CSUR) 54, 9 (2021), 1-40."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP66354.2025.00038"},{"key":"e_1_2_1_47_1","volume-title":"Symbolic prompt program search: A structure-aware approach to efficient compile-time prompt optimization. arXiv preprint arXiv:2404.02319","author":"Schnabel Tobias","year":"2024","unstructured":"Tobias Schnabel and Jennifer Neville. 2024. Symbolic prompt program search: A structure-aware approach to efficient compile-time prompt optimization. arXiv preprint arXiv:2404.02319 (2024)."},{"key":"e_1_2_1_48_1","volume-title":"Shraddha Barke, and Benjamin Zorn.","author":"Sharma Reshabh K","year":"2025","unstructured":"Reshabh K Sharma, Jonathan De Halleux, Shraddha Barke, and Benjamin Zorn. 2025. PromptPex: Automatic Test Generation for Language Model Prompts. arXiv preprint arXiv:2503.05070 (2025)."},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)","author":"Silveira Natalia","unstructured":"Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A Gold Standard Dependency Corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637528.3671796"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.854"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.34740\/KAGGLE\/M\/3301"},{"key":"e_1_2_1_53_1","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1145\/3180155.3180220","volume-title":"Proceedings of the 40th International Conference on Software Engineering (ICSE '18)","author":"Tian Yuchi","year":"2018","unstructured":"Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated Testing of Deep-neural-networkdriven Autonomous Cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE '18). ACM, New York, NY, USA, 303-314."},{"key":"e_1_2_1_54_1","volume-title":"Attention is all you need. Advances in neural information processing systems 30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_2_1_55_1","volume-title":"Aakanksha Chowdhery, and Denny Zhou.","author":"Wang Xuezhi","year":"2023","unstructured":"Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. CoRR abs\/2203.11171 (2023)."},{"key":"e_1_2_1_56_1","doi-asserted-by":"crossref","unstructured":"John Yang Carlos E. Jimenez Alexander Wettig Kilian Lieret Shunyu Yao Karthik Narasimhan and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.","DOI":"10.52202\/079017-1601"},{"key":"e_1_2_1_57_1","first-page":"129","volume-title":"Proceedings of the 16th IEEE International Conference on Software Testing, Verification and Validation (ICST","author":"Yoon Juyeon","year":"2024","unstructured":"Juyeon Yoon, Robert Feldt, and Shin Yoo. 2024. Intent-Driven Mobile GUI Testing with Autonomous Large Language Model Agents. In Proceedings of the 16th IEEE International Conference on Software Testing, Verification and Validation (ICST 2024). 129-139."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","unstructured":"Juyeon Yoon Somin Kim Robert Feldt and Shin Yoo. 2026. Artifact for Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs. doi:10.5281\/zenodo.19693236 10.5281\/zenodo.19693236","DOI":"10.5281\/zenodo.19693236"},{"key":"e_1_2_1_59_1","first-page":"649","volume-title":"Proceedings of the 29th International Conference on Neural Information Processing Systems -Volume 1","author":"Zhang Xiang","year":"2015","unstructured":"Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the 29th International Conference on Neural Information Processing Systems -Volume 1 (Montreal, Canada) (NIPS'15). MIT Press, Cambridge, MA, USA, 649-657."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680384"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680384"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE55347.2025.00209"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3797114","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:32:00Z","timestamp":1782844320000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3797114"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":62,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3797114"],"URL":"https:\/\/doi.org\/10.1145\/3797114","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}