{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:38:51Z","timestamp":1791459531905,"version":"4.3.4"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,3,29]],"date-time":"2023-03-29T00:00:00Z","timestamp":1680048000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"crossref","award":["NRF-2018R1A5A1059921"],"award-info":[{"award-number":["NRF-2018R1A5A1059921"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Institute of Information & communications Technology Planning & Evaluation","award":["2018-0-00769"],"award-info":[{"award-number":["2018-0-00769"]}]},{"name":"Swedish Scientific Council","award":["2015-04913, 2020-05272"],"award-info":[{"award-number":["2015-04913, 2020-05272"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,4,30]]},"abstract":"<jats:p>\n            The rapid adoption of Deep Learning (DL) systems in safety critical domains such as medical imaging and autonomous driving urgently calls for ways to test their correctness and robustness. Borrowing from the concept of test adequacy in traditional software testing, existing work on testing of DL systems initially investigated DL systems from structural point of view, leading to a number of coverage metrics. Our lack of understanding of the internal mechanism of Deep Neural Networks (DNNs), however, means that coverage metrics defined on the Boolean dichotomy of coverage are hard to intuitively interpret and understand. We propose the degree of out-of-distribution-ness of a given input as its adequacy for testing: the more\n            <jats:italic>surprising<\/jats:italic>\n            a given input is to the DNN under test, the more likely the system will show unexpected behavior for the input. We develop the concept of surprise into a test adequacy criterion, called Surprise Adequacy (SA). Intuitively, SA measures the difference in the behavior of the DNN for the given input and its behavior for the training data. We posit that a good test input should be sufficiently, but not overtly, surprising compared to the training dataset. This article evaluates SA using a range of DL systems from simple image classifiers to autonomous driving car platforms, as well as both small and large data benchmarks ranging from MNIST to ImageNet. The results show that the SA value of an input can be a reliable predictor of the correctness of the mode behavior. We also show that SA can be used to detect adversarial examples, and also be efficiently computed against large training dataset such as ImageNet using sampling.\n          <\/jats:p>","DOI":"10.1145\/3546947","type":"journal-article","created":{"date-parts":[[2022,7,6]],"date-time":"2022-07-06T09:33:41Z","timestamp":1657100021000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["Evaluating Surprise Adequacy for Deep Learning System Testing"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0140-7908","authenticated-orcid":false,"given":"Jinhan","family":"Kim","sequence":"first","affiliation":[{"name":"KAIST, Yuseong-gu, Daejeon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5179-4205","authenticated-orcid":false,"given":"Robert","family":"Feldt","sequence":"additional","affiliation":[{"name":"Chalmers University of Technology, Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0836-6993","authenticated-orcid":false,"given":"Shin","family":"Yoo","sequence":"additional","affiliation":[{"name":"KAIST, Yuseong-gu, Daejeon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,3,29]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Udacity. 2016. Autonomous driving model: Chauffeur. Retrieved from https:\/\/github.com\/udacity\/self-driving-car\/tree\/master\/steering-models\/community-models\/chauffeur."},{"key":"e_1_3_2_3_2","unstructured":"Udacity. 2016. The Udacity open source self-driving car project. Retrieved from https:\/\/github.com\/udacity\/self-driving-car."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.5555\/3133461"},{"key":"e_1_3_2_5_2","unstructured":"Yoshua Bengio Gr\u00e9goire Mesnil Yann Dauphin and Salah Rifai. 2013. Better mixing via deep representations. Proceedings of the 30th International Conference on Machine Learning 28 1 (2013) 552\u2013560."},{"key":"e_1_3_2_6_2","unstructured":"Mariusz Bojarski Davide Del Testa Daniel Dworakowski Bernhard Firner Beat Flepp Prasoon Goyal Lawrence D. Jackel Mathew Monfort Urs Muller Jiakai Zhang Xin Zhang and Jake Zhao. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3128572.3140444"},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Nicholas Carlini and David A. Wagner. 2016. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (S&P) . 39\u201357.","DOI":"10.1109\/SP.2017.49"},{"key":"e_1_3_2_9_2","first-page":"2722","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Chen Chenyi","year":"2015","unstructured":"Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. 2015. Deepdriving: Learning affordance for direct perception in autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision. 2722\u20132730."},{"key":"e_1_3_2_10_2","unstructured":"Junjie Chen Ming Yan Zan Wang Yuning Kang and Zhuo Wu. 2020. Deep neural network test coverage: How far are we? arXiv:2010.04946. Retrieved from https:\/\/arxiv.org\/abs\/2010.04946."},{"key":"e_1_3_2_11_2","first-page":"94","volume-title":"Proceedings of the International Workshop on Software Technology and Engineering Practice (STEP\u201903)","author":"Chen T. Y.","year":"2004","unstructured":"T. Y. Chen, F.-C. Kuo, T. H. Tse, and Zhi Quan Zhou. 2004. Metamorphic testing and beyond. In Proceedings of the International Workshop on Software Technology and Engineering Practice (STEP\u201903). 94\u2013100."},{"key":"e_1_3_2_12_2","unstructured":"Cody Coleman Christopher Yeh Stephen Mussmann Baharan Mirzasoleiman Peter Bailis Percy Liang Jure Leskovec and Matei Zaharia. 2019. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations (ICLR) ."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2018.2822680"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/tpami.2012.231"},{"key":"e_1_3_2_16_2","unstructured":"Reuben Feinman Ryan R. Curtin Saurabh Shintre and Andrew B. Gardner. 2017. Detecting adversarial samples from artifacts. arXiv:1703.00410. Retrieved from https:\/\/arxiv.org\/abs\/1703.00410."},{"key":"e_1_3_2_17_2","first-page":"223","volume-title":"Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation (ICST\u201916)","author":"Feldt Robert","year":"2016","unstructured":"Robert Feldt, Simon Poulding, David Clark, and Shin Yoo. 2016. Test set diameter: Quantifying the diversity of sets of test cases. In Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation (ICST\u201916). 223\u2013233."},{"key":"e_1_3_2_18_2","first-page":"1050","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Gal Yarin","year":"2016","unstructured":"Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning. 1050\u20131059."},{"key":"e_1_3_2_19_2","first-page":"1183","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Gal Yarin","year":"2017","unstructured":"Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep bayesian active learning with image data. In Proceedings of the International Conference on Machine Learning. PMLR, 1183\u20131192."},{"key":"e_1_3_2_20_2","unstructured":"Jakob Gawlikowski Cedrique Rovile Njieutcheu Tassi Mohsin Ali Jongseok Lee Matthias Humt Jianxiang Feng Anna Kruspe Rudolph Triebel Peter Jung Ribana Roscher et\u00a0al. 2021. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342."},{"key":"e_1_3_2_21_2","first-page":"702","volume-title":"Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE)","author":"Gerasimou Simos","year":"2020","unstructured":"Simos Gerasimou, Hasan Ferit Eniser, Alper Sen, and Alper Cakan. 2020. Importance-driven deep learning system testing. In Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 702\u2013713."},{"key":"e_1_3_2_22_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Goodfellow Ian","year":"2015","unstructured":"Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"348","DOI":"10.1109\/ICSE43902.2021.00042","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE)","author":"Guerriero Antonio","year":"2021","unstructured":"Antonio Guerriero, Roberto Pietrantuono, and Stefano Russo. 2021. Operation is the hardest teacher: Estimating DNN accuracy looking for mispredictions. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 348\u2013358."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","unstructured":"Geoffrey Hinton Li Deng Dong Yu George E. Dahl Abdel-Rahman Mohamed Navdeep Jaitly Andrew Senior Vincent Vanhoucke Patrick Nguyen Tara N. Sainath and Brian Kingsbury. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine 29 6 (2012) 82\u201397.","DOI":"10.1109\/MSP.2012.2205597"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-319-63387-9_1","volume-title":"Computer Aided Verification","author":"Huang Xiaowei","year":"2017","unstructured":"Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. 2017. Safety verification of deep neural networks. In Computer Aided Verification, Rupak Majumdar and Viktor Kun\u010dak (Eds.), Springer International Publishing, 3\u201329."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1001"},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1145\/3387940.3391456","volume-title":"Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops (SBST\u201920)","author":"Kang Sungmin","year":"2020","unstructured":"Sungmin Kang, Robert Feldt, and Shin Yoo. 2020. SINVAD: Search-based image space navigation for DNN image classifier test input generation. In Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops (SBST\u201920). 521\u2013528."},{"key":"e_1_3_2_30_2","first-page":"1039","volume-title":"Proceedings of the 41th International Conference on Software Engineering (ICSE\u201919)","author":"Kim Jinhan","year":"2019","unstructured":"Jinhan Kim, Robert Feldt, and Shin Yoo. 2019. Guiding deep learning system testing using surprise adequacy. In Proceedings of the 41th International Conference on Software Engineering (ICSE\u201919). IEEE Press, 1039\u20131049."},{"key":"e_1_3_2_31_2","first-page":"1466","volume-title":"Proceedings of ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE Industry Track) (ESEC\/FSE\u201920)","author":"Kim Jinhan","year":"2020","unstructured":"Jinhan Kim, Jeongil Ju, Robert Feldt, and Shin Yoo. 2020. Reducing DNN labelling cost using surprise adequacy: An industrial case study for autonomous driving. In Proceedings of ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE Industry Track) (ESEC\/FSE\u201920). 1466\u20131476."},{"key":"e_1_3_2_32_2","first-page":"80","volume-title":"Proceedings of the 2021 IEEE\/ACM International Conference on Automation of Software Test (AST)","author":"Kim Seah","year":"2021","unstructured":"Seah Kim and Shin Yoo. 2021. Multimodal surprise adequacy analysis of inputs for natural language processing DNN models. In Proceedings of the 2021 IEEE\/ACM International Conference on Automation of Software Test (AST). IEEE, 80\u201389."},{"key":"e_1_3_2_33_2","volume-title":"Learning Multiple Layers of Features from Tiny Images","author":"Krizhevsky Alex","year":"2009","unstructured":"Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report. University of Toronto."},{"key":"e_1_3_2_34_2","unstructured":"Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2014. The CIFAR-10 dataset. Retrieved from http:\/\/www.cs.toronto.edu\/kriz\/cifar.html. (2014)."},{"key":"e_1_3_2_35_2","first-page":"1097","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems. 1097\u20131105."},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","unstructured":"Alexey Kurakin Ian J. Goodfellow and Samy Bengio. 2016. Adversarial examples in the physical world. Artificial Intelligence Safety and Security . Chapman and Hall\/CRC 99\u2013112.","DOI":"10.1201\/9781351251389-8"},{"key":"e_1_3_2_37_2","unstructured":"Balaji Lakshminarayanan Alexander Pritzel and Charles Blundell. 2016. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems 30."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_3_2_39_2","article-title":"MNIST handwritten digit database","volume":"2","author":"LeCun Yann","year":"2010","unstructured":"Yann LeCun, Corinna Cortes, and CJ Burges. 2010. MNIST handwritten digit database. AT & T Labs [Online]. Retrieved from http:\/\/yann.lecun.com\/exdb\/mnist. 2 (2010).","journal-title":"AT & T Labs"},{"key":"e_1_3_2_40_2","unstructured":"Yu Li Min Li Qiuxia Lai Yannan Liu and Qiang Xu. 2021. TestRank: Bringing order into unlabeled test instances for deep learning tasks. Advances in Neural Information Processing Systems 34 (2021) 20874\u201320886."},{"issue":"2","key":"e_1_3_2_41_2","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.artmed.2013.11.007","article-title":"Detecting rare events using extreme value statistics applied to epileptic convulsions in children","volume":"60","author":"Luca Stijn","year":"2014","unstructured":"Stijn Luca, Peter Karsmakers, Kris Cuppens, Tom Croonenborghs, Anouk Van de Vel, Berten Ceulemans, Lieven Lagae, Sabine Van Huffel, and Bart Vanrumste. 2014. Detecting rare events using extreme value statistics applied to epileptic convulsions in children. Artificial Intelligence in Medicine 60, 2 (2014), 89\u201396.","journal-title":"Artificial Intelligence in Medicine"},{"key":"e_1_3_2_42_2","doi-asserted-by":"crossref","unstructured":"Lei Ma Felix Juefei-Xu Jiyuan Sun Chunyang Chen Ting Su Fuyuan Zhang Minhui Xue Bo Li Li Li Yang Liu Jianjun Zhao and Yadong Wang. 2018. DeepGauge: Comprehensive and multi-granularity testing criteria for gauging the robustness of deep learning systems. In Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering . 120\u2013131.","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","unstructured":"Lei Ma Fuyuan Zhang Jiyuan Sun Minhui Xue Bo Li Felix Juefei-Xu Chao Xie Li Li Yang Liu Jianjun Zhao et\u00a0al. 2018. DeepMutation: Mutation testing of deep learning systems. In IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) . 100\u2013111.","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_44_2","unstructured":"Lei Ma Fuyuan Zhang Minhui Xue Bo Li Yang Liu Jianjun Zhao and Yadong Wang. 2018. Combinatorial testing for deep learning systems. arXiv preprint arXiv:1806.07723."},{"key":"e_1_3_2_45_2","unstructured":"Xingjun Ma Bo Li Yisen Wang Sarah M. Erfani Sudanthi Wijewickrema Michael E. Houle Grant Schoenebeck Dawn Song and James Bailey. 2018. Characterizing adversarial subspaces using local intrinsic dimensionality. In Proceedings of the 6th International Conference on Learning Representations . ICLR 1\u201315."},{"issue":"1","key":"e_1_3_2_46_2","first-page":"1","article-title":"Reprint of: Mahalanobis, P.C. (1936) \u201cOn the generalised distance in statistics\u201d","volume":"80","author":"Mahalanobis Prasanta Chandra","year":"2018","unstructured":"Prasanta Chandra Mahalanobis. 2018. Reprint of: Mahalanobis, P.C. (1936) \u201cOn the generalised distance in statistics\u201d. Sankhya A 80, 1 (2018), 1\u20137.","journal-title":"Sankhya A"},{"key":"e_1_3_2_47_2","first-page":"189","volume-title":"Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA\u201909)","author":"Murphy Christian","year":"2009","unstructured":"Christian Murphy, Kuang Shen, and Gail Kaiser. 2009. Automatic system testing of programs without test oracles. In Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA\u201909). ACM Press, 189\u2013200."},{"key":"e_1_3_2_48_2","volume-title":"Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning","author":"Netzer Yuval","year":"2011","unstructured":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning."},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/AITEST52744.2021.00016","volume-title":"Proceedings of the 2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","author":"Ouyang Tinghui","year":"2021","unstructured":"Tinghui Ouyang, Yoshinao Isobe, Vicent Sanz Marco, Jun Ogata, Yoshiki Seo, and Yutaka Oiwa. 2021. AI robustness analysis with consideration of corner cases. In Proceedings of the 2021 IEEE International Conference on Artificial Intelligence Testing (AITest). IEEE, 29\u201336."},{"key":"e_1_3_2_50_2","unstructured":"Nicolas Papernot Fartash Faghri Nicholas Carlini Ian Goodfellow Reuben Feinman Alexey Kurakin Cihang Xie Yash Sharma Tom Brown Aurko Roy Alexander Matyasko Vahid Behzadan Karen Hambardzumyan Zhishuai Zhang Yi-Lin Juang Zhi Li Ryan Sheatsley Abhibhav Garg Jonathan Uesato Willi Gierke Yinpeng Dong David Berthelot Paul Hendricks Jonas Rauber and Rujun Long. 2018. Technical report on the cleverhans v2.1.0 adversarial examples library. arXiv preprint arXiv:1610.00768."},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Nicolas Papernot Patrick D. McDaniel Somesh Jha Matt Fredrikson Z. Berkay Celik and Ananthram Swami. 2015. The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy (EuroS&P\u201916) . IEEE 372\u2013387.","DOI":"10.1109\/EuroSP.2016.36"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_53_2","first-page":"81","volume-title":"Proceedings of the Software Testing, Verification and Validation Workshops (ICSTW\u201917), IEEE International Conference on","author":"Poulding Simon","year":"2017","unstructured":"Simon Poulding and Robert Feldt. 2017. Generating controllably invalid and atypical inputs for robustness testing. In Proceedings of the Software Testing, Verification and Validation Workshops (ICSTW\u201917), IEEE International Conference on. IEEE, 81\u201384."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472291"},{"key":"e_1_3_2_55_2","doi-asserted-by":"crossref","unstructured":"Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein Alexander C. Berg and Li Fei-Fei. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115 3 (2015) 211\u2013252.","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2021.3060483"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1002\/9781118575574"},{"key":"e_1_3_2_58_2","first-page":"3104","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Sutskever Ilya","year":"2014","unstructured":"Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the Advances in Neural Information Processing Systems. 3104\u20133112."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_3_2_60_2","first-page":"2818","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Szegedy Christian","year":"2016","unstructured":"Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2818\u20132826."},{"key":"e_1_3_2_61_2","doi-asserted-by":"crossref","unstructured":"L. Tarassenko A. Hann A. Patterson E. Braithwaite K. Davidson V. Barber and D. Young. 2005. Biosign\u2122: Multi-parameter monitoring for early warning of patient deterioration. In The 3rd IEE International Seminar on Medical Applications of Signal Processing 2005 (Ref. No. 2005-1119). 71\u201376.","DOI":"10.1049\/ic:20050334"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1201\/b14876"},{"key":"e_1_3_2_64_2","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1109\/ICSE43902.2021.00046","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE)","author":"Wang Zan","year":"2021","unstructured":"Zan Wang, Hanmo You, Junjie Chen, Yingyi Zhang, Xuyuan Dong, and Wenbin Zhang. 2021. Prioritizing test inputs for deep neural networks via mutation analysis. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 397\u2013409."},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/DeepTest52559.2021.00009"},{"key":"e_1_3_2_66_2","unstructured":"Wikipedia. 2021. Computational complexity of mathematical operations. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Computational_complexity_of_mathematical_operations. Online; accessed 22-October-2021."},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409671"},{"key":"e_1_3_2_69_2","first-page":"192","volume-title":"Proceedings of the 3rd International Workshop on Search-Based Software Testing (SBST\u201910)","author":"Yoo Shin","year":"2010","unstructured":"Shin Yoo. 2010. Metamorphic testing of stochastic optimisation. In Proceedings of the 3rd International Workshop on Search-Based Software Testing (SBST\u201910). 192\u2013201."},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/SBST.2019.000-2"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1002\/stv.430"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/267580.267590"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3546947","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3546947","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:42Z","timestamp":1750186842000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3546947"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,29]]},"references-count":71,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,4,30]]}},"alternative-id":["10.1145\/3546947"],"URL":"https:\/\/doi.org\/10.1145\/3546947","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,29]]},"assertion":[{"value":"2021-11-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-06-11","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}