{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T08:13:03Z","timestamp":1769847183903,"version":"3.49.0"},"reference-count":122,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T00:00:00Z","timestamp":1727395200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["CNS1764039, CNS1956180, CCF2119184, CNS1952050, CCF1823032, CNS2313190"],"award-info":[{"award-number":["CNS1764039, CNS1956180, CCF2119184, CNS1952050, CCF1823032, CNS2313190"]}]},{"name":"ARO","award":["W911NF 1920321"],"award-info":[{"award-number":["W911NF 1920321"]}]},{"name":"DOE Early Career Award","award":["DESC0014195 0003"],"award-info":[{"award-number":["DESC0014195 0003"]}]},{"name":"CERES Center for Unstoppable Computing, UChicago Marian and Stuart Rice Research Award"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>The increasing number of software applications incorporating machine learning (ML) solutions has led to the need for testing techniques. However, testing ML software requires tremendous human effort to design realistic and relevant test inputs and to judge software output correctness according to human common sense. Even when misbehavior is exposed, it is often unclear whether the defect is inside ML API or the surrounding code and how to fix the implementation. This article tackles these challenges by proposing Keeper, an automated testing and fixing tool for ML software. The core idea of Keeper is designing pseudo-inverse functions that semantically reverse the corresponding ML task in an empirical way and proxy common human judgment of real-world data. It incorporates these functions into a symbolic execution engine to generate tests. Keeper also detects code smells that degrade software performance. Once misbehavior is exposed, Keeper attempts to change how ML APIs are used to alleviate the misbehavior.<\/jats:p>\n          <jats:p>Our evaluation on a variety of applications shows that Keeper greatly improves branch coverage, while identifying 74 previously unknown failures and 19 code smells from 56 out of 104 applications. Our user studies show that 78% of end-users and 95% of developers agree with Keeper\u2019s detection and fixing results.<\/jats:p>","DOI":"10.1145\/3672451","type":"journal-article","created":{"date-parts":[[2024,6,13]],"date-time":"2024-06-13T17:56:59Z","timestamp":1718301419000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Keeper: Automated Testing and Fixing of Machine Learning Software"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9162-9688","authenticated-orcid":false,"given":"Chengcheng","family":"Wan","sequence":"first","affiliation":[{"name":"East China Normal University, Shanghai, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2697-1916","authenticated-orcid":false,"given":"Shicheng","family":"Liu","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5484-5234","authenticated-orcid":false,"given":"Sophie","family":"Xie","sequence":"additional","affiliation":[{"name":"University of California, Berkeley, CA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-5957-5071","authenticated-orcid":false,"given":"Yuhan","family":"Liu","sequence":"additional","affiliation":[{"name":"University of Chicago, Chicago, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0816-8150","authenticated-orcid":false,"given":"Henry","family":"Hoffmann","sequence":"additional","affiliation":[{"name":"University of Chicago, Chicago, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9778-6673","authenticated-orcid":false,"given":"Michael","family":"Maire","sequence":"additional","affiliation":[{"name":"University of Chicago, Chicago, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0757-4600","authenticated-orcid":false,"given":"Shan","family":"Lu","sequence":"additional","affiliation":[{"name":"Microsoft Research and University of Chicago, Chicago, IL, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,9,27]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180160"},{"key":"e_1_3_4_3_2","unstructured":"Amazon. 2022. Amazon Artificial Intelligence Service. Retrieved from https:\/\/aws.amazon.com\/machine-learning\/ai-services"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264590"},{"key":"e_1_3_4_7_2","unstructured":"Animal-Analysis. 2019. An Image Sharing Platform. Retrieved from https:\/\/github.com\/OkapalDominic\/animal_analysis"},{"key":"e_1_3_4_8_2","first-page":"23","volume-title":"Software Fault Tolerance","author":"Avizienis Algirdas","year":"1995","unstructured":"Algirdas Avizienis. 1995. The methodology of n-version programming. Software Fault Tolerance 3 (1995), 23\u201346."},{"key":"e_1_3_4_9_2","unstructured":"Soheil Bahrampour Naveen Ramakrishnan Lukas Schott and Mohak Shah. 2015. Comparative study of deep learning software frameworks. arXiv:1511.06435. Retrieved from https:\/\/arxiv.org\/abs\/1511.06435"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-22110-1_14"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416609"},{"key":"e_1_3_4_12_2","unstructured":"Natesh M Bhat. 2023. pyttsx3: Text-to-Speech Library for Python. Retrieved from https:\/\/pypi.org\/project\/pyttsx3\/"},{"key":"e_1_3_4_13_2","unstructured":"Microsoft Bing. 2023. Bing Image Search. Retrieved from https:\/\/www.bing.com\/images\/trending?FORM=ILPTRD"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00014"},{"key":"e_1_3_4_15_2","unstructured":"Lars Buitinck Gilles Louppe Mathieu Blondel Fabian Pedregosa Andreas Mueller Olivier Grisel Vlad Niculae Peter Prettenhofer Alexandre Gramfort Jaques Grobler Robert Layton Jake Vanderplas Arnaud Joly Brian Holt and Ga\u00ebl Varoquaux. 2013. API design for machine learning software: Experiences from the scikit-learn project. arXiv:1309.0238. Retrieved from https:\/\/arxiv.org\/abs\/1309.0238"},{"key":"e_1_3_4_16_2","unstructured":"Dave Chaffey. 2024. Search Engine Marketing Statistics 2020. Retrieved from https:\/\/www.smartinsights.com\/search-engine-marketing\/search-engine-statistics\/"},{"key":"e_1_3_4_17_2","volume-title":"Proceedings of Machine Learning Research (PMLR)","author":"Chen Lingjiao","year":"2022","unstructured":"Lingjiao Chen, Matei Zaharia, and James Zou. 2022. FrugalMCT: Efficient online ML API selection for multi-label classification tasks. In Proceedings of Machine Learning Research (PMLR)."},{"key":"e_1_3_4_18_2","unstructured":"Lingjiao Chen Matei Zaharia and James Y Zou. 2020. FrugalML: How to use ML prediction APIs more accurately and cheaply. In Advances in Neural Information Processing Systems H. Larochelle M. Ranzato R. Hadsell M. F. Balcan and H. Lin (Eds.) Vol. 33 Curran Associates Inc 10685\u201310696. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/789ba2ae4d335e8a2ad283a3f7effced-Paper.pdf"},{"key":"e_1_3_4_19_2","unstructured":"Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang Yao Liu Hieu Pham Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu and Quoc V. Le. 2023. Symbolic discovery of optimization algorithms. arXiv:2302.06675. Retrieved from https:\/\/arxiv.org\/abs\/2302.06675"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2018.00044"},{"key":"e_1_3_4_21_2","unstructured":"Jeffrey A. Clark. 2023. Pillow: Python Imaging Library. Retrieved from https:\/\/pypi.org\/project\/Pillow\/"},{"key":"e_1_3_4_22_2","unstructured":"darude brainstorm. 2022. A Clothes Checking Application. Retrieved from https:\/\/github.com\/paulgan98\/darude-brainstorm"},{"issue":"2","key":"e_1_3_4_23_2","first-page":"1301","article-title":"A survey on machine learning: concept, algorithms and applications","volume":"5","author":"Das Kajaree","year":"2017","unstructured":"Kajaree Das and Rabi Narayan Behera. 2017. A survey on machine learning: concept, algorithms and applications. International Journal of Innovative Research in Computer and Communication Engineering 5, 2 (2017), 1301\u20131309.","journal-title":"International Journal of Innovative Research in Computer and Communication Engineering"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397366"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213858"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-17108-6_11"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-53288-8_3"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_4_30_2","unstructured":"Fuzzit.dev. 2023. Pythonfuzz: Coverage-Guided Fuzz Testing for Python. Retrieved from https:\/\/gitlab.com\/gitlab-org\/security-products\/analyzers\/fuzzers\/pythonfuzz"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510232"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380391"},{"key":"e_1_3_4_34_2","unstructured":"Alec Go Richa Bhayani and Lei Huang. 2009. Twitter Sentiment Classification Using Distant Supervision. CS224N Project Report Stanford 1\u201312."},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397374"},{"key":"e_1_3_4_36_2","unstructured":"Google. 2022. Google Cloud AI. Retrieved from https:\/\/cloud.google.com\/products\/ai."},{"key":"e_1_3_4_37_2","first-page":"134","volume-title":"Computational Materials Science","author":"Gossett Eric","year":"2018","unstructured":"Eric Gossett, Cormac Toher, Corey Oses, Olexandr Isayev, Fleur Legrain, Frisco Rose, Eva Zurek, Jes\u00fas Carrete, Natalio Mingo, Alexander Tropsha, and Stefano Curtarolo. 2018. AFLOW-ML: A RESTful API for machine-learning predictions of materials properties. Computational Materials Science 152 (2018), 134\u2013145."},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416571"},{"key":"e_1_3_4_39_2","unstructured":"Dave Halter. 2022. Jedi: An Awesome Auto-Completion Static Analysis and Refactoring Library for Python. Retrieved from https:\/\/jedi.readthedocs.io."},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_3_4_41_2","unstructured":"HeapSortCypher. 2019. A Garbage Classification Application. Retrieved from https:\/\/github.com\/matthew-chu\/heapsortcypher"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.2334-5837.2016.00179.x"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2016.7739680"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00152"},{"key":"e_1_3_4_45_2","unstructured":"IBM. 2022. IBM Watson. Retrieved from https:\/\/www.ibm.com\/watson"},{"key":"e_1_3_4_46_2","first-page":"26","article-title":"Deconstructing dynamic symbolic execution","volume":"40","author":"Irlbeck M.","year":"2015","unstructured":"M. Irlbeck, D. Peled, and A. Pretschner. 2015. Deconstructing dynamic symbolic execution. Dependable Software Systems Engineering 40 (2015), 26.","journal-title":"Dependable Software Systems Engineering"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380378"},{"key":"e_1_3_4_48_2","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering (ICSE)","author":"Jahangirova Gunel","year":"2020","unstructured":"Gunel Jahangirova, Nargiz Humbatova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering (ICSE)."},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSESS.2015.7339246"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS-C.2018.00026"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3128473.3128484"},{"key":"e_1_3_4_52_2","unstructured":"Kaggle. 2019. Twitter US Airline Sentiment. Retrieved from https:\/\/www.kaggle.com\/crowdflower\/twitter-airline-sentiment"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3243127.3243129"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884783"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2754374"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-020-01316-z"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397346"},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2018.1455307"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409696"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/MET.2017.6"},{"key":"e_1_3_4_61_2","unstructured":"Lingle. 2018. A News Summary Application. Retrieved from https:\/\/github.com\/AllegraChen\/uofthacks6"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29509-7_5"},{"key":"e_1_3_4_63_2","unstructured":"Lisa-Assistant. 2021. A Voice Assistant Application. Retrieved from https:\/\/github.com\/AlexNguyen27\/lisa-assistant-gcp"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510231"},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/64"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106291"},{"key":"e_1_3_4_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236082"},{"key":"e_1_3_4_69_2","unstructured":"David Marby and Nijiko Yonskai. 2021. Pyan3: Offline Call Graph Generator for Python 3. Retrieved from https:\/\/github.com\/davidfraser\/pyan"},{"key":"e_1_3_4_70_2","unstructured":"David Marby and Nijiko Yonskai. 2022. Lorem Picsum. Retrieved from https:\/\/picsum.photos"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.94"},{"key":"e_1_3_4_72_2","unstructured":"Microsoft. 2022. Microsoft Azure Cognitive Services. Retrieved from https:\/\/azure.microsoft.com\/en-us\/services\/cognitive-services"},{"key":"e_1_3_4_73_2","unstructured":"Microsoft. 2022. Visual Studio Code. Retrieved from https:\/\/code.visualstudio.com\/"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00078"},{"key":"e_1_3_4_75_2","volume-title":"Proceedings of the 36th International Conference on Machine Learning (ICML)","author":"Odena Augustus","year":"2019","unstructured":"Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. 2019. TensorFuzz: Debugging neural networks with coverage-guided fuzzing. In Proceedings of the 36th International Conference on Machine Learning (ICML)."},{"key":"e_1_3_4_76_2","unstructured":"Omakase. 2022. A Food Manager Application. Retrieved from https:\/\/github.com\/Seangottarun\/Omakase"},{"key":"e_1_3_4_77_2","unstructured":"Tejit Pabari. 2019. A Prescription Identification Application. Retrieved from https:\/\/github.com\/gitika-bose\/ResearchSpring2019"},{"key":"e_1_3_4_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380337"},{"key":"e_1_3_4_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_4_80_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46295-0_10"},{"key":"e_1_3_4_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_3_4_82_2","unstructured":"Phoenix. 2020. A Fire-Detection Application. Retrieved from https:\/\/github.com\/yunusemreemik\/Phoenix"},{"key":"e_1_3_4_83_2","unstructured":"Prolific. 2023. An Online Platform for Recruiting Participants. Retrieved from https:\/\/www.prolific.co"},{"key":"e_1_3_4_84_2","unstructured":"Python. 2023. ast\u2014Abstract Syntax Trees. Retrieved from https:\/\/docs.python.org\/3\/library\/ast.html"},{"issue":"8","key":"e_1_3_4_85_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.","journal-title":"OpenAI blog"},{"key":"e_1_3_4_86_2","unstructured":"RainorShine. 2020. A Weather Application. Retrieved from https:\/\/github.com\/DanialKhan6312\/RainorShine"},{"key":"e_1_3_4_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409730"},{"key":"e_1_3_4_88_2","unstructured":"RoomR. 2020. A Property Management Application. Retrieved from https:\/\/github.com\/rodrigoHM\/RoomR-Server"},{"key":"e_1_3_4_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00118"},{"key":"e_1_3_4_90_2","unstructured":"scikit learn. 2023. scikit-learn: Machine Learning in Python. Retrieved from https:\/\/scikit-learn.org\/stable\/"},{"key":"e_1_3_4_91_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2014.04.008"},{"key":"e_1_3_4_92_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397352"},{"key":"e_1_3_4_93_2","unstructured":"SIVG. 2021. A Video Generator Application. Retrieved from https:\/\/github.com\/aarenstade\/speech-img-vid-generator"},{"key":"e_1_3_4_94_2","unstructured":"Soap. 2018. A Public Opinion Analysis Application. Retrieved from https:\/\/github.com\/jcavejr\/soap"},{"key":"e_1_3_4_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454064"},{"key":"e_1_3_4_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2013.158"},{"key":"e_1_3_4_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/3193977.3193981"},{"key":"e_1_3_4_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510080"},{"key":"e_1_3_4_99_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_4_100_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00067"},{"key":"e_1_3_4_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_4_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3404540"},{"key":"e_1_3_4_103_2","volume-title":"Proceedings of the 43th International Conference on Software Engineering (ICSE \u201921)","author":"Wan Chengcheng","year":"2021","unstructured":"Chengcheng Wan, Shicheng Liu, Henry Hoffmann, Michael Maire, and Shan Lu. 2021. Are machine learning cloud APIs used correctly? In Proceedings of the 43th International Conference on Software Engineering (ICSE \u201921)."},{"key":"e_1_3_4_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510068"},{"key":"e_1_3_4_105_2","unstructured":"WanderStub. 2019. An Exchange Conversion Application. Retrieved from https:\/\/github.com\/richardjpark26\/WanderStub"},{"key":"e_1_3_4_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510165"},{"key":"e_1_3_4_107_2","unstructured":"Wikidata. 2022. A Free and Open Knowledge Base. Retrieved from https:\/\/www.wikidata.org\/"},{"key":"e_1_3_4_108_2","unstructured":"Wikipedia. 2023. A Free Encyclopedia. Retrieved from https:\/\/en.m.wikipedia.org\/"},{"key":"e_1_3_4_109_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM48880.2022.9796701"},{"key":"e_1_3_4_110_2","unstructured":"Xiaofei Xie Lei Ma Felix Juefei-Xu Hongxu Chen Minhui Xue Bo Li Yang Liu Jianjun Zhao Jianxiong Yin and Simon See. 2018. Deephunter: Hunting deep neural network defects via coverage-guided fuzzing. arXiv:1809.01266. Retrieved from https:\/\/arxiv.org\/abs\/1809.01266"},{"key":"e_1_3_4_111_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_4_112_2","unstructured":"Kaidi Xu Huan Zhang Shiqi Wang Yihan Wang Suman Jana Xue Lin and Cho-Jui Hsieh. 2020. Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. arXiv:2011.13824. Retrieved from https:\/\/arxiv.org\/abs\/2011.13824"},{"key":"e_1_3_4_113_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409671"},{"key":"e_1_3_4_114_2","doi-asserted-by":"crossref","unstructured":"Heiga Zen Viet Dang Rob Clark Yu Zhang Ron J Weiss Ye Jia Zhifeng Chen and Yonghui Wu. 2019. Libritts: A corpus derived from librispeech for text-to-speech. arXiv:1904.02882. Retrieved from https:\/\/arxiv.org\/abs\/1904.02882","DOI":"10.21437\/Interspeech.2019-2441"},{"key":"e_1_3_4_115_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409750"},{"key":"e_1_3_4_116_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00043"},{"key":"e_1_3_4_117_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3560425"},{"key":"e_1_3_4_118_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485517"},{"key":"e_1_3_4_119_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_4_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380368"},{"key":"e_1_3_4_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409720"},{"key":"e_1_3_4_122_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS-C.2018.00017"},{"key":"e_1_3_4_123_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.00018"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672451","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3672451","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:01Z","timestamp":1750294681000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672451"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,27]]},"references-count":122,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3672451"],"URL":"https:\/\/doi.org\/10.1145\/3672451","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,27]]},"assertion":[{"value":"2023-05-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}