{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,10]],"date-time":"2026-11-10T04:24:42Z","timestamp":1794284682751,"version":"4.9.0"},"reference-count":133,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2020,9,15]],"date-time":"2020-09-15T00:00:00Z","timestamp":1600128000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,9,15]],"date-time":"2020-09-15T00:00:00Z","timestamp":1600128000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"ERC-2017-ADG","award":["787703"],"award-info":[{"award-number":["787703"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2020,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Context:<\/jats:title>\n                    <jats:p>A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective:<\/jats:title>\n                    <jats:p>To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Method:<\/jats:title>\n                    <jats:p>We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used\/proposed adequacy criteria, the algorithms for test input generation, and the test oracles.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions:<\/jats:title>\n                    <jats:p>The most active research areas in MLS testing address automated scenario\/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s10664-020-09881-0","type":"journal-article","created":{"date-parts":[[2020,9,15]],"date-time":"2020-09-15T05:02:39Z","timestamp":1600146159000},"page":"5193-5254","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":256,"title":["Testing machine learning based systems: a systematic mapping"],"prefix":"10.1007","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6229-8231","authenticated-orcid":false,"given":"Vincenzo","family":"Riccio","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gunel","family":"Jahangirova","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Stocco","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nargiz","family":"Humbatova","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Weiss","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paolo","family":"Tonella","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,9,15]]},"reference":[{"key":"9881_CR1","doi-asserted-by":"crossref","unstructured":"Abdessalem RB, Nejati S, Briand LC, Stifter T (2016) Testing advanced driver assistance systems using multi-objective search and neural networks. In: Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering. ASE, pp 63\u201374","DOI":"10.1145\/2970276.2970311"},{"key":"9881_CR2","doi-asserted-by":"publisher","unstructured":"Abdessalem RB, Nejati S, Briand LC, Stifter T (2018a) Testing vision-based control systems using learnable evolutionary algorithms. In: Proceedings of the 40th International Conference on Software Engineering, ICSE \u201918. ACM, New York. https:\/\/doi.org\/10.1145\/3180155.3180160, pp 1016\u20131026","DOI":"10.1145\/3180155.3180160"},{"key":"9881_CR3","doi-asserted-by":"publisher","unstructured":"Abdessalem RB, Panichella A, Nejati S, Briand LC, Stifter T (2018b) Testing autonomous cars for feature interaction failures using many-objective search. In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, ASE 2018. ACM, New York, pp 143\u2013154. https:\/\/doi.org\/10.1145\/3238147.3238192","DOI":"10.1145\/3238147.3238192"},{"key":"9881_CR4","doi-asserted-by":"publisher","unstructured":"Abeysirigoonawardena Y, Shkurti F, Dudek G (2019) Generating adversarial driving scenarios in high-fidelity simulators. In: International conference on robotics and automation, ICRA 2019, montreal, QC, Canada, May 20-24, 2019, pp 8271\u20138277. https:\/\/doi.org\/10.1109\/ICRA.2019.8793740","DOI":"10.1109\/ICRA.2019.8793740"},{"key":"9881_CR5","doi-asserted-by":"publisher","unstructured":"Ali NB, Petersen K (2014) Evaluating strategies for study selection in systematic literature studies. In: Proceedings of the 8th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM \u201914. ACM, New York, pp 45:1\u201345:4. https:\/\/doi.org\/10.1145\/2652524.2652557","DOI":"10.1145\/2652524.2652557"},{"issue":"3","key":"9881_CR6","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1080\/00031305.1992.10475879","volume":"46","author":"NS Altman","year":"1992","unstructured":"Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Amer Stat 46(3):175\u2013185. https:\/\/doi.org\/10.1080\/00031305.1992.10475879","journal-title":"Amer Stat"},{"key":"9881_CR7","doi-asserted-by":"publisher","unstructured":"Aniculaesei A, Grieser J, Rausch A, Rehfeldt K, Warnecke T (2018) Towards a holistic software systems engineering approach for dependable autonomous systems. In: Proceedings of the 1st International Workshop on Software Engineering for AI in Autonomous Systems - SEFAIS. ACM Press. https:\/\/doi.org\/10.1145\/3194085.3194091","DOI":"10.1145\/3194085.3194091"},{"key":"9881_CR8","doi-asserted-by":"crossref","unstructured":"Arp D, Spreitzenbarth M, Hubner M, Gascon H, Rieck K, Siemens C (2014) Drebin: Effective and explainable detection of android malware in your pocket. In: Ndss, vol 14, pp 23\u201326","DOI":"10.14722\/ndss.2014.23247"},{"key":"9881_CR9","unstructured":"Arthur D, Vassilvitskii S (2007) K-means++: The advantages of careful seeding. In: Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2007. Society for Industrial and Applied Mathematics, pp 1027\u20131035. http:\/\/dl.acm.org\/citation.cfm?id=1283383.1283494"},{"key":"9881_CR10","doi-asserted-by":"crossref","unstructured":"Beglerovic H, Stolz M, Horn M (2017) Testing of autonomous vehicles using surrogate models and stochastic optimization. In: 20Th IEEE international conference on intelligent transportation systems, ITSC \u201917, pp 1\u20136","DOI":"10.1109\/ITSC.2017.8317768"},{"key":"9881_CR11","unstructured":"Bishop CM (2006) Pattern recognition and machine learning. Springer Science+ Business Media"},{"key":"9881_CR12","doi-asserted-by":"publisher","unstructured":"Bolte J, Bar A, Lipinski D, Fingscheidt T (2019) Towards corner case detection for autonomous driving. In: 2019 IEEE Intelligent vehicles symposium (IV), pp 438\u2013445. https:\/\/doi.org\/10.1109\/IVS.2019.8813817","DOI":"10.1109\/IVS.2019.8813817"},{"key":"9881_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2991\/jase.d.190131.001","volume":"1","author":"M Borg","year":"2019","unstructured":"Borg M, Englund C, Wnuk K, Duran B, Levandowski C, Gao S, Tan Y, Kaijser H, L\u00f6nn H, T\u00f6rnqvist J (2019) Safely entering the deep: a review of verification and validation for machine learning and a challenge elicitation in the automotive industry. J Autom Softw Eng 1:1\u201319. https:\/\/doi.org\/10.2991\/jase.d.190131.001","journal-title":"J Autom Softw Eng"},{"key":"9881_CR14","unstructured":"Braiek HB, Khomh F (2018) On testing machine learning programs"},{"issue":"1","key":"9881_CR15","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1023\/A:1009783721306","volume":"3","author":"LC Briand","year":"1998","unstructured":"Briand LC, Daly JW, Wu\u0307st J (1998) A unified framework for cohesion measurement in object-oriented systems. Empir Softw Eng 3(1):65\u2013117","journal-title":"Empir Softw Eng"},{"issue":"1","key":"9881_CR16","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1109\/32.748920","volume":"25","author":"LC Briand","year":"1999","unstructured":"Briand LC, Daly JW, Wu\u0307st J (1999) A unified framework for coupling measurement in object-oriented systems. IEEE Trans Softw Eng 25 (1):91\u2013121","journal-title":"IEEE Trans Softw Eng"},{"key":"9881_CR17","doi-asserted-by":"crossref","unstructured":"B\u00fchler O, Wegener J (2004) Automatic testing of an autonomous parking system using evolutionary computation. Technical report, SAE Technical Paper","DOI":"10.4271\/2004-01-0459"},{"key":"9881_CR18","doi-asserted-by":"crossref","unstructured":"Byun T, Sharma V, Vijayakumar A, Rayadurgam S, Cofer D (2019) Input prioritization for testing neural networks","DOI":"10.1109\/AITest.2019.000-6"},{"issue":"4","key":"9881_CR19","doi-asserted-by":"publisher","first-page":"891","DOI":"10.1007\/s10618-015-0444-8","volume":"30","author":"GO Campos","year":"2016","unstructured":"Campos GO, Zimek A, Sander J, Campello RJ, Micenkov\u00e1 B, Schubert E, Assent I, Houle ME (2016) On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study. Data Min Knowl Discov 30 (4):891\u2013927. https:\/\/doi.org\/10.1007\/s10618-015-0444-8","journal-title":"Data Min Knowl Discov"},{"issue":"2","key":"9881_CR20","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1145\/3177753","volume":"61","author":"VG Cerf","year":"2018","unstructured":"Cerf VG (2018) A comprehensive self-driving car test. Commun ACM 61(2):7\u20137. https:\/\/doi.org\/10.1145\/3177753","journal-title":"Commun ACM"},{"key":"9881_CR21","doi-asserted-by":"crossref","unstructured":"Cheng D, Cao C, Xu C, Ma X (2018a) Manifesting bugs in machine learning code: an explorative study with mutation testing. In: 2018 IEEE International conference on software quality, reliability and security (QRS). IEEE, pp 313\u2013324","DOI":"10.1109\/QRS.2018.00044"},{"key":"9881_CR22","doi-asserted-by":"crossref","unstructured":"Cheng CH, Huang CH, Yasuoka H (2018b) Quantitative projection coverage for testing ml-enabled autonomous systems. In: Lahiri SK, Wang C (eds) Automated technology for verification and analysis. Springer International Publishing, Cham, pp 126\u2013142","DOI":"10.1007\/978-3-030-01090-4_8"},{"key":"9881_CR23","doi-asserted-by":"crossref","unstructured":"Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3213\u20133223","DOI":"10.1109\/CVPR.2016.350"},{"key":"9881_CR24","doi-asserted-by":"publisher","unstructured":"Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273\u2013297. https:\/\/doi.org\/10.1023\/A:1022627411411","DOI":"10.1023\/A:1022627411411"},{"key":"9881_CR25","doi-asserted-by":"crossref","unstructured":"Davis MD, Weyuker EJ (1981) Pseudo-oracles for non-testable programs. In: Proceedings of the ACM\u201981 Conference. ACM, pp 254\u2013257","DOI":"10.1145\/800175.809889"},{"issue":"5\/6","key":"9881_CR26","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1504\/IJES.2016.080388","volume":"8","author":"V de Oliveira Neves","year":"2016","unstructured":"de Oliveira Neves V, Delamaro ME, Masiero PC (2016) Combination and mutation strategies to support test data generation in the context of autonomous vehicles. IJES 8(5\/6):464\u2013482. https:\/\/doi.org\/10.1504\/IJES.2016.10001345","journal-title":"IJES"},{"key":"9881_CR27","doi-asserted-by":"publisher","unstructured":"Ding J, Kang X, Hu X (2017) Validating a deep learning framework by metamorphic testing. In: 2017 IEEE\/ACM 2Nd international workshop on metamorphic testing (MET), pp 28\u201334. https:\/\/doi.org\/10.1109\/MET.2017.2","DOI":"10.1109\/MET.2017.2"},{"issue":"4","key":"9881_CR28","first-page":"405","volume":"10","author":"H Do","year":"2005","unstructured":"Do H, Elbaum S, Rothermel G (2005) Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. EMSE 10(4):405\u2013435","journal-title":"EMSE"},{"key":"9881_CR29","unstructured":"Dosovitskiy A, Ros G, Codevilla F, Lopez A, Koltun V (2017) CARLA: An open urban driving simulator. In: Levine S, Vanhoucke V, Goldberg K. In: Proceedings of the 1st Annual Conference on Robot Learning Proceedings of Machine Learning Research (eds). PMLR. http:\/\/proceedings.mlr.press\/v78\/dosovitskiy17a.html, vol 78, pp 1\u201316"},{"key":"9881_CR30","doi-asserted-by":"crossref","unstructured":"Du X, Xie X, Li Y, Ma L, Liu Y, Zhao J (2019) Deepstellar: Model-based quantitative analysis of stateful deep learning systems. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, pp 477\u2013487","DOI":"10.1145\/3338906.3338954"},{"key":"9881_CR31","unstructured":"Dua D, Graff C (2017) UCI machine learning repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"9881_CR32","doi-asserted-by":"publisher","unstructured":"Dwarakanath A, Ahuja M, Sikand S, Rao RM, Bose RPJC, Dubash N, Podder S (2018) Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018. ACM, New York, pp 118\u2013128. https:\/\/doi.org\/10.1145\/3213846.3213858","DOI":"10.1145\/3213846.3213858"},{"key":"9881_CR33","doi-asserted-by":"crossref","unstructured":"Eniser HF, Gerasimou S, Sen A (2019) Deepfault: Fault localization for deep neural networks. In: H\u00e4hnle R, van der Aalst W (eds) Fundamental approaches to software engineering. Springer International Publishing, Cham, pp 171\u2013191","DOI":"10.1007\/978-3-030-16722-6_10"},{"key":"9881_CR34","doi-asserted-by":"crossref","unstructured":"Fraser G, Arcuri A (2012) Sound empirical evidence in software testing. In: Proceedings of ICSE \u201912, pp 178\u2013188","DOI":"10.1109\/ICSE.2012.6227195"},{"key":"9881_CR35","doi-asserted-by":"publisher","unstructured":"Fremont DJ, Dreossi T, Ghosh S, Yue X, Sangiovanni-Vincentelli AL, Seshia SA (2019) Scenic: a language for scenario specification and scene generation. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019. ACM, New York, pp 63\u201378. https:\/\/doi.org\/10.1145\/3314221.3314633","DOI":"10.1145\/3314221.3314633"},{"key":"9881_CR36","unstructured":"Gal Y (2016) Uncertainty in deep learning. PhD thesis, University of Cambridge"},{"key":"9881_CR37","doi-asserted-by":"crossref","unstructured":"Garousi V, Felderer M, M\u00e4ntyl\u00e4 MV, Rainer A (2019) Benefitting from the grey literature in software engineering research","DOI":"10.1007\/978-3-030-32489-6_14"},{"key":"9881_CR38","unstructured":"Gopinath D, Wang K, Zhang M, Pasareanu CS, Khurshid S (2018) Symbolic execution for deep neural networks. CoRR arXiv:1807.10439"},{"issue":"3","key":"9881_CR39","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1109\/TSE.2013.59","volume":"40","author":"A Groce","year":"2014","unstructured":"Groce A, Kulesza T, Zhang C, Shamasunder S, Burnett M, Wong W, Stumpf S, Das S, Shinsel A, Bice F, McIntosh K (2014) You are the only possible oracle: Effective test selection for end users of interactive machine learning systems. IEEE Trans Softw Eng 40(3):307\u2013323. https:\/\/doi.org\/10.1109\/TSE.2013.59","journal-title":"IEEE Trans Softw Eng"},{"key":"9881_CR40","unstructured":"Gross P, Boulanger A, Arias M, Waltz D, Long PM, Lawson C, Anderson R, Koenig M, Mastrocinque M, Fairechio W, Johnson JA, Lee S, Doherty F, Kressner A (2006) Predicting electricity distribution feeder failures using machine learning susceptibility analysis. In: Proceedings of the 18th Conference on Innovative Applications of Artificial Intelligence - Volume 2, IAAI\u201906. AAAI Press, pp 1705\u20131711. http:\/\/dl.acm.org\/citation.cfm?id=1597122.1597127"},{"key":"9881_CR41","doi-asserted-by":"crossref","unstructured":"Guo J, Jiang Y, Zhao Y, Chen Q, Sun J (2018) Dlfuzz: differential fuzzing testing of deep learning systems. In: Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/SIGSOFT FSE, pp 739\u2013743","DOI":"10.1145\/3236024.3264835"},{"key":"9881_CR42","doi-asserted-by":"crossref","unstructured":"Gyimesi P, Vancsics B, Stocco A, Mazinanian D, Beszedes A, Ferenc R, Mesbah A (2019) BugsJS: a benchmark of JavaScript bugs. In: Proceedings of the International Conference on Software Testing, Verification, and Validation (ICST). IEEE Computer Society","DOI":"10.1109\/ICST.2019.00019"},{"key":"9881_CR43","doi-asserted-by":"publisher","unstructured":"Hains G, Jakobsson A, Khmelevsky Y (2018) Towards formal methods and software engineering for deep learning: security, safety and productivity for dl systems development. In: 2018 Annual IEEE international systems conference (syscon), pp 1\u20135. https:\/\/doi.org\/10.1109\/SYSCON.2018.8369576","DOI":"10.1109\/SYSCON.2018.8369576"},{"key":"9881_CR44","unstructured":"Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference and prediction, 2 edn. Springer. http:\/\/www-stat.stanford.edu\/~tibs\/ElemStatLearn\/"},{"key":"9881_CR45","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"9881_CR46","doi-asserted-by":"publisher","unstructured":"Henriksson J, Berger C, Borg M, Tornberg L, Englund C, Sathyamoorthy SR, Ursing S (2019) Towards structured evaluation of deep neural network supervisors. In: 2019 IEEE International conference on artificial intelligence testing (AITest). IEEE. https:\/\/doi.org\/10.1109\/aitest.2019.00-12","DOI":"10.1109\/aitest.2019.00-12"},{"key":"9881_CR47","unstructured":"Huang X, Kroening D, Ruan W, Sharp J, Sun Y, Thamo E, Wu M, Yi X (2018) A survey of safety and trustworthiness of deep neural networks"},{"key":"9881_CR48","doi-asserted-by":"crossref","unstructured":"Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: Proceedings of 42nd International Conference on Software Engineering, ICSE \u201920. ACM, pp 12","DOI":"10.1145\/3377811.3380395"},{"key":"9881_CR49","doi-asserted-by":"publisher","unstructured":"IEEE (1990) Ieee standard glossary of software engineering terminology. IEEE Std 610.12-1990 Reaffirmed 12-9-2002, pp 1\u201384. https:\/\/doi.org\/10.1109\/IEEESTD.1990.101064","DOI":"10.1109\/IEEESTD.1990.101064"},{"key":"9881_CR50","unstructured":"International T (2017) Prescan simulation of adas and active safety. https:\/\/www.tassinternational.com\/prescan"},{"key":"9881_CR51","doi-asserted-by":"publisher","unstructured":"Islam MJ, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2019. ACM, New York, pp 510\u2013520. https:\/\/doi.org\/10.1145\/3338906.3338955","DOI":"10.1145\/3338906.3338955"},{"key":"9881_CR52","doi-asserted-by":"crossref","unstructured":"Jahangirova G, Tonella P (2020) An empirical evaluation of mutation operators for deep learning systems. In: Proceedings of the International Conference on Software Testing, Verification, and Validation (ICST). IEEE Computer Society","DOI":"10.1109\/ICST46399.2020.00018"},{"key":"9881_CR53","doi-asserted-by":"crossref","unstructured":"James G, Witten D, Hastie T, Tibshirani R (2013) An introduction to statistical learning, vol 112. Springer","DOI":"10.1007\/978-1-4614-7138-7"},{"key":"9881_CR54","doi-asserted-by":"crossref","unstructured":"Just R, Jalali D, Ernst MD (2014) Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In: Proceedings of the ISSTA \u201914, pp 437\u2013440","DOI":"10.1145\/2610384.2628055"},{"key":"9881_CR55","unstructured":"Kaufman L, Rousseeuw P (1987) Clustering by means of medoids. Statistical Data Analysis Based on the L1-Norm and Related Methods, pp North\u2013Holland"},{"key":"9881_CR56","doi-asserted-by":"crossref","unstructured":"Kaufman L, Rousseeuw PJ (1990) Finding groups in data: an introduction to cluster analysis. Wiley series in Probability and Mathematical Statistics. Wiley. A Wiley-Interscience publication","DOI":"10.1002\/9780470316801"},{"key":"9881_CR57","doi-asserted-by":"crossref","unstructured":"Kim J, Feldt R, Yoo S (2019) Guiding deep learning system testing using surprise adequacy. In: Proceedings of the 41st International Conference on Software Engineering, ICSE, pp 1039\u20131049","DOI":"10.1109\/ICSE.2019.00108"},{"key":"9881_CR58","unstructured":"Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering"},{"issue":"5","key":"9881_CR59","doi-asserted-by":"publisher","first-page":"316","DOI":"10.1109\/TSE.2007.1001","volume":"33","author":"BA Kitchenham","year":"2007","unstructured":"Kitchenham BA, Mendes E, Travassos GH (2007) Cross versus within-company cost estimation studies: a systematic review. IEEE Trans Softw Eng 33 (5):316\u2013329. https:\/\/doi.org\/10.1109\/TSE.2007.1001","journal-title":"IEEE Trans Softw Eng"},{"key":"9881_CR60","doi-asserted-by":"publisher","unstructured":"Kitchenham B, Brereton OP, Budgen D, Turner M, Bailey J, Linkman S (2009) Systematic literature reviews in software engineering \u2013 a systematic literature review. Inf Softw Technol 51(1):7\u201315. https:\/\/doi.org\/10.1016\/j.infsof.2008.09.009, http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950584908001390. Special Section - Most Cited Articles in 2002 and Regular Research Papers","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"9881_CR61","doi-asserted-by":"publisher","unstructured":"Klueck F, Li Y, Nica M, Tao J, Wotawa F (2018) Using ontologies for test suites generation for automated and autonomous driving functions. In: 2018 IEEE International symposium on software reliability engineering workshops (ISSREW), pp 118\u2013123. https:\/\/doi.org\/10.1109\/ISSREW.2018.00-20","DOI":"10.1109\/ISSREW.2018.00-20"},{"key":"9881_CR62","unstructured":"Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images. Technical report, Citeseer"},{"key":"9881_CR63","unstructured":"LeCun Y, Cortes C (2010) MNIST handwritten digit database. http:\/\/yann.lecun.com\/exdb\/mnist\/"},{"issue":"11","key":"9881_CR64","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y LeCun","year":"1998","unstructured":"LeCun Y, Bottou L, Bengio Y, Haffner P et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278\u20132324","journal-title":"Proc IEEE"},{"issue":"2","key":"9881_CR65","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1109\/TIV.2016.2608003","volume":"1","author":"L Li","year":"2016","unstructured":"Li L, Huang W, Liu Y, Zheng N, Wang F (2016) Intelligence testing for autonomous vehicles: a new approach. IEEE Trans Intell Veh 1(2):158\u2013166. https:\/\/doi.org\/10.1109\/TIV.2016.2608003","journal-title":"IEEE Trans Intell Veh"},{"key":"9881_CR66","doi-asserted-by":"publisher","unstructured":"Li G, Pattabiraman K, DeBardeleben N (2018) Tensorfi: a configurable fault injector for tensorflow applications. In: 2018 IEEE International symposium on software reliability engineering workshops (ISSREW), pp 313\u2013320. https:\/\/doi.org\/10.1109\/ISSREW.2018.00024","DOI":"10.1109\/ISSREW.2018.00024"},{"key":"9881_CR67","doi-asserted-by":"crossref","unstructured":"Long PM, Servedio RA (2005) Martingale boosting. In: Auer P, Meir R. In: Learning theory (eds). Springer, Berlin, pp 79\u201394","DOI":"10.1007\/11503415_6"},{"key":"9881_CR68","unstructured":"Ma L, Juefei-Xu F, Xue M, Hu Q, Chen S, Li B, Liu Y, Zhao J, Yin J, See S (2018a) Secure deep learning engineering: A software quality assurance perspective"},{"key":"9881_CR69","doi-asserted-by":"publisher","unstructured":"Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, Chen C, Su T, Li L, Liu Y, Zhao J, Wang Y (2018b) Deepgauge: Multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, ASE 2018. ACM, New York, pp 120\u2013131. https:\/\/doi.org\/10.1145\/3238147.3238202","DOI":"10.1145\/3238147.3238202"},{"key":"9881_CR70","doi-asserted-by":"publisher","unstructured":"Ma S, Liu Y, Lee WC, Zhang X, Grama A (2018c) Mode: Automated neural network model debugging via state differential analysis and input selection. In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2018. ACM, New York, pp 175\u2013186. https:\/\/doi.org\/10.1145\/3236024.3236082","DOI":"10.1145\/3236024.3236082"},{"key":"9881_CR71","doi-asserted-by":"crossref","unstructured":"Ma L, Zhang F, Sun J, Xue M, Li B, Juefei-Xu F, Xie C, Li L, Liu Y, Zhao J et al (2018d) Deepmutation: Mutation testing of deep learning systems. In: 2018 IEEE 29Th international symposium on software reliability engineering (ISSRE). IEEE, pp 100\u2013111","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"9881_CR72","doi-asserted-by":"crossref","unstructured":"Ma L, Juefei-Xu F, Xue M, Li B, Li L, Liu Y, Zhao J (2019) Deepct: Tomographic combinatorial testing for deep learning systems. In: 2019 IEEE 26Th international conference on software analysis, evolution and reengineering (SANER). IEEE, pp 614\u2013618","DOI":"10.1109\/SANER.2019.8668044"},{"key":"9881_CR73","unstructured":"Majumdar R, Mathur A, Pirron M, Stegner L, Zufferey D (2019) Paracosm: A language and tool for testing autonomous driving systems"},{"key":"9881_CR74","doi-asserted-by":"crossref","unstructured":"Manning CD, Raghavan P, Sch\u00fctze H (2008) Introduction to information retrieval. Cambridge University Press, New York","DOI":"10.1017\/CBO9780511809071"},{"key":"9881_CR75","doi-asserted-by":"publisher","unstructured":"Masuda S, Ono K, Yasue T, Hosokawa N (2018) A survey of software quality for machine learning applications. In: 2018 IEEE International conference on software testing, verification and validation workshops (ICSTW), pp 279\u2013284. https:\/\/doi.org\/10.1109\/ICSTW.2018.00061","DOI":"10.1109\/ICSTW.2018.00061"},{"key":"9881_CR76","doi-asserted-by":"crossref","unstructured":"McLachlan GJ, Basford KE (1988) Mixture models: Inference and applications to clustering, vol 38. M Dekker, New York","DOI":"10.2307\/2348072"},{"key":"9881_CR77","doi-asserted-by":"publisher","unstructured":"Mullins G, Stankiewicz P, Hawthorne R, Gupta S (2018) Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles. J Syst Softw 137:197\u2013215. https:\/\/doi.org\/10.1016\/j.jss.2017.10.031. https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85037521968","DOI":"10.1016\/j.jss.2017.10.031"},{"key":"9881_CR78","first-page":"167","volume":"2007","author":"C Murphy","year":"2007","unstructured":"Murphy C, Kaiser G, Arias M (2007a) An approach to software testing of machine learning applications. SEKE 2007:167","journal-title":"SEKE"},{"key":"9881_CR79","doi-asserted-by":"crossref","unstructured":"Murphy C, Kaiser G, Arias M (2007b) Parameterizing random test data according to equivalence classes. In: Proceedings of the 2nd international workshop on Random testing: co-located with the 22nd IEEE\/ACM International Conference on Automated Software Engineering (ASE 2007). ACM, pp 38\u201341","DOI":"10.1145\/1292414.1292425"},{"key":"9881_CR80","unstructured":"Murphy C, Kaiser G (2008) Improving the dependability of machine learning applications"},{"key":"9881_CR81","first-page":"867","volume":"2008","author":"C Murphy","year":"2008","unstructured":"Murphy C, Kaiser G, Hu L, Wu L (2008) Properties of machine learning applications for use in metamorphic testing. SEKE 2008:867","journal-title":"SEKE"},{"key":"9881_CR82","doi-asserted-by":"crossref","unstructured":"Murphy C, Shen K, Kaiser G (2009) Automatic system testing of programs without test oracles. In: Proceedings of the eighteenth international symposium on Software testing and analysis. ACM, pp 189\u2013200","DOI":"10.1145\/1572272.1572295"},{"key":"9881_CR83","doi-asserted-by":"crossref","unstructured":"Nakajima S (2018) Generalized oracle for testing machine learning computer programs. In: Cerone A, Roveri M (eds) Software engineering and formal methods. Springer International Publishing, Cham, pp 174\u2013179","DOI":"10.1007\/978-3-319-74781-1_13"},{"key":"9881_CR84","doi-asserted-by":"crossref","unstructured":"Nakajima S (2019) Dataset diversity for metamorphic testing of machine learning software. In: Structured object-oriented formal language and method. Springer international publishing, pp 21\u201338","DOI":"10.1007\/978-3-030-13651-2_2"},{"key":"9881_CR85","doi-asserted-by":"publisher","unstructured":"Nakajima S, Bui H.N (2016) Dataset coverage for testing machine learning computer programs. In: Proceedings of the 23rd Asia-Pacific Software Engineering Conference, APSEC \u201916, pp 297\u2013304. https:\/\/doi.org\/10.1109\/APSEC.2016.049","DOI":"10.1109\/APSEC.2016.049"},{"key":"9881_CR86","unstructured":"Odena A, Olsson C, Andersen D, Goodfellow I (2019) TensorFuzz: Debugging neural networks with coverage-guided fuzzing. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research. http:\/\/proceedings.mlr.press\/v97\/odena19a.html, vol 97. PMLR, Long Beach, pp 4901\u20134911."},{"key":"9881_CR87","doi-asserted-by":"publisher","unstructured":"Patel N, Saridena AN, Choromanska A, Krishnamurthy P, Khorrami F (2018) Adversarial learning-based on-line anomaly monitoring for assured autonomy. In: 2018 IEEE\/RSJ International conference on intelligent robots and systems, IROS 2018, Madrid, pp 6149\u20136154. https:\/\/doi.org\/10.1109\/IROS.2018.8593375","DOI":"10.1109\/IROS.2018.8593375"},{"key":"9881_CR88","doi-asserted-by":"crossref","unstructured":"Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: Automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles, pp 1\u201318","DOI":"10.1145\/3132747.3132785"},{"key":"9881_CR89","doi-asserted-by":"crossref","unstructured":"Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: Ease, vol 8, pp 68\u201377","DOI":"10.14236\/ewic\/EASE2008.8"},{"key":"9881_CR90","doi-asserted-by":"publisher","unstructured":"Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: An update. Inf Softw Technol 64:1\u201318. https:\/\/doi.org\/10.1016\/j.infsof.2015.03.007. http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950584915000646","DOI":"10.1016\/j.infsof.2015.03.007"},{"key":"9881_CR91","doi-asserted-by":"crossref","unstructured":"Qin Y, Wang H, Xu C, Ma X, Lu J (2018) Syneva: Evaluating ml programs by mirror program synthesis. In: 2018 IEEE International conference on software quality, reliability and security (QRS). IEEE, pp 171\u2013182","DOI":"10.1109\/QRS.2018.00031"},{"key":"9881_CR92","doi-asserted-by":"crossref","unstructured":"Riccio V, Tonella P (2020) Model-based exploration of the frontier of behaviours for deep learning system testing. In: Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). ACM","DOI":"10.1145\/3368089.3409730"},{"key":"9881_CR93","unstructured":"Riccio V, Jahangirova G, Stocco A, Humbatova N, Weiss M, Tonella P (2019) Replication package. https:\/\/github.com\/testingautomated\/deepthoughts"},{"key":"9881_CR94","doi-asserted-by":"publisher","DOI":"10.1142\/9097","volume-title":"Data mining with decision trees: Theory and applications","author":"L Rokach","year":"2014","unstructured":"Rokach L, Maimon O (2014) Data mining with decision trees: Theory and applications, 2nd edn. World Scientific Publishing Co., Inc., River Edge","edition":"2nd edn."},{"key":"9881_CR95","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","volume":"20","author":"PJ Rousseeuw","year":"1987","unstructured":"Rousseeuw PJ (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math 20:53\u201365","journal-title":"J Comput Appl Math"},{"key":"9881_CR96","doi-asserted-by":"publisher","unstructured":"Rubaiyat AHM, Qin Y, Alemzadeh H (2018) Experimental resilience assessment of an open-source driving agent. In: 2018 IEEE 23Rd pacific rim international symposium on dependable computing (PRDC), pp 54\u201363. https:\/\/doi.org\/10.1109\/PRDC.2018.00016","DOI":"10.1109\/PRDC.2018.00016"},{"key":"9881_CR97","doi-asserted-by":"publisher","unstructured":"Saha P, Kanewala U (2019) Fault detection effectiveness of metamorphic relations developed for testing supervised classifiers. In: 2019 IEEE International conference on artificial intelligence testing (AITest), pp 157\u2013164. https:\/\/doi.org\/10.1109\/AITest.2019.00019","DOI":"10.1109\/AITest.2019.00019"},{"key":"9881_CR98","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","volume":"61","author":"J Schmidhuber","year":"2015","unstructured":"Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85\u2013117","journal-title":"Neural Netw"},{"key":"9881_CR99","doi-asserted-by":"publisher","unstructured":"Sekhon J, Fleming C (2019) Towards improved testing for deep learning. In: 2019 IEEE\/ACM 41St international conference on software engineering: New ideas and emerging results (ICSE-NIER), pp 85\u201388. https:\/\/doi.org\/10.1109\/ICSE-NIER.2019.00030","DOI":"10.1109\/ICSE-NIER.2019.00030"},{"key":"9881_CR100","doi-asserted-by":"crossref","unstructured":"Shen W, Wan J, Chen Z (2018) Munn: Mutation analysis of neural networks. In: 2018 IEEE International conference on software quality, reliability and security companion (QRS-c). IEEE, pp 108\u2013115","DOI":"10.1109\/QRS-C.2018.00032"},{"key":"9881_CR101","unstructured":"Sherin S, khan MU, Iqbal MZ (2019) A systematic mapping study on testing of machine learning programs"},{"key":"9881_CR102","unstructured":"Shi Q, Wan J, Feng Y, Fang C, Chen Z. (2019) Deepgini: Prioritizing massive tests to reduce labeling cost"},{"key":"9881_CR103","unstructured":"Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition"},{"key":"9881_CR104","doi-asserted-by":"publisher","unstructured":"Speed T (2010) Statistical models: Theory and practice, revised edition by david a. freedman. Int Stat Rev 78(3):457\u2013458. https:\/\/doi.org\/10.1111\/j.1751-5823.2010.00122_11.x","DOI":"10.1111\/j.1751-5823.2010.00122_11.x"},{"key":"9881_CR105","unstructured":"Spieker H, Gotlieb A (2019) Towards testing of deep learning systems with training set reduction. CoRR arXiv:1901.04169"},{"key":"9881_CR106","doi-asserted-by":"crossref","unstructured":"Stocco A, Weiss M, Calzana M, Tonella P (2020) Misbehaviour prediction for autonomous driving systems. In: Proceedings of 42nd International Conference on Software Engineering, ICSE \u201920. ACM, pp 12","DOI":"10.1145\/3377811.3380353"},{"key":"9881_CR107","doi-asserted-by":"publisher","unstructured":"Strickland M, Fainekos G, Ben amor H (2018) Deep predictive models for collision risk assessment in autonomous driving. In: 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Proceedings - IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc, pp 4685\u20134692. https:\/\/doi.org\/10.1109\/ICRA.2018.8461160","DOI":"10.1109\/ICRA.2018.8461160"},{"key":"9881_CR108","unstructured":"Stuart R, Peter N (2016) Artificial intelligence - a modern approach, 3rd ed, Berkeley"},{"key":"9881_CR109","doi-asserted-by":"crossref","unstructured":"Sun Y, Huang X, Kroening D, Sharp J, Hill M, Ashmore R (2018a) Testing deep neural networks","DOI":"10.1145\/3238147.3238172"},{"key":"9881_CR110","doi-asserted-by":"publisher","unstructured":"Sun Y, Wu M, Ruan W, Huang X, Kwiatkowska M, Kroening D (2018b) Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, ASE \u201918. ACM, New York, pp 109\u2013119. https:\/\/doi.org\/10.1145\/3238147.3238172","DOI":"10.1145\/3238147.3238172"},{"key":"9881_CR111","doi-asserted-by":"publisher","unstructured":"Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering, ICSE \u201918. ACM, New York, pp 303\u2013314. https:\/\/doi.org\/10.1145\/3180155.3180220","DOI":"10.1145\/3180155.3180220"},{"key":"9881_CR112","unstructured":"Tuncali CE, Fainekos G (2019) Rapidly-exploring random trees-based test generation for autonomous vehicles. CoRR arXiv:1903.10629"},{"key":"9881_CR113","doi-asserted-by":"crossref","unstructured":"Tuncali CE, Fainekos G, Ito H, Kapinski J (2018) Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: 2018 IEEE Intelligent vehicles symposium (IV). IEEE, pp 1555\u20131562","DOI":"10.1109\/IVS.2018.8500421"},{"key":"9881_CR114","doi-asserted-by":"crossref","unstructured":"Turing AM (2009) Computing machinery and intelligence. In: Parsing the turing test. Springer, pp 23\u201365","DOI":"10.1007\/978-1-4020-6710-5_3"},{"key":"9881_CR115","doi-asserted-by":"crossref","unstructured":"Udeshi S, Chattopadhyay S (2019) Grammar based directed testing of machine learning systems","DOI":"10.1109\/TSE.2019.2953066"},{"key":"9881_CR116","doi-asserted-by":"publisher","unstructured":"Udeshi S, Arora P, Chattopadhyay S (2018) Automated directed fairness testing. In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, ASE 2018. ACM, New York, pp 98\u2013108. https:\/\/doi.org\/10.1145\/3238147.3238165","DOI":"10.1145\/3238147.3238165"},{"key":"9881_CR117","unstructured":"Uesato J, Kumar A, Szepesva\u0307ri C, Erez T, Ruderman A, Anderson K, Dvijotham KD, Heess N, Kohli P (2019) Rigorous agent evaluation: an adversarial approach to uncover catastrophic failures. In: 7Th international conference on learning representations, ICLR 2019, New Orleans"},{"key":"9881_CR118","doi-asserted-by":"publisher","unstructured":"Wang J, Dong G, Sun J, Wang X, Zhang P (2019) Adversarial sample detection for deep neural network through model mutation testing. In: Proceedings of the 41st International Conference on Software Engineering, ICSE \u201919. IEEE Press, Piscataway, pp 1245\u20131256. https:\/\/doi.org\/10.1109\/ICSE.2019.00126","DOI":"10.1109\/ICSE.2019.00126"},{"issue":"1","key":"9881_CR119","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1007\/s00766-005-0021-6","volume":"11","author":"R Wieringa","year":"2006","unstructured":"Wieringa R, Maiden N, Mead N, Rolland C (2006) Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir Eng 11(1):102\u2013107","journal-title":"Requir Eng"},{"key":"9881_CR120","doi-asserted-by":"crossref","unstructured":"Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of EASE \u201914, pp 1\u201310","DOI":"10.1145\/2601248.2601268"},{"key":"9881_CR121","doi-asserted-by":"publisher","unstructured":"Wolschke C, Kuhn T, Rombach D, Liggesmeyer P (2017) Observation based creation of minimal test suites for autonomous vehicles. In: 2017 IEEE International symposium on software reliability engineering workshops (ISSREW), pp 294\u2013301. https:\/\/doi.org\/10.1109\/ISSREW.2017.46","DOI":"10.1109\/ISSREW.2017.46"},{"key":"9881_CR122","doi-asserted-by":"crossref","unstructured":"Wolschke C, Rombach D, Liggesmeyer P, Kuhn T (2018) Mining test inputs for autonomous vehicles. In: Berns K, Dressler K, Fleischmann P, G\u00f6rges D, Kalmar R, Sauer B, Stephan N, Teutsch R, Thul M (eds) Commercial vehicle technology 2018. Springer Fachmedien, Wiesbaden, pp 102\u2013113","DOI":"10.1007\/978-3-658-21300-8_9"},{"issue":"4","key":"9881_CR123","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1016\/j.jss.2010.11.920","volume":"84","author":"X Xie","year":"2011","unstructured":"Xie X, Ho JW, Murphy C, Kaiser G, Xu B, Chen TY (2011) Testing and validating machine learning classifiers by metamorphic testing. J Syst Softw 84(4):544\u2013558","journal-title":"J Syst Softw"},{"key":"9881_CR124","unstructured":"Xie X, Zhang Z, Yueh Chen T, Liu Y, Poon PL, Xu B (2018) Mettle: A metamorphic testing approach to validating unsupervised machine learning methods. arXiv:1807.10453"},{"key":"9881_CR125","doi-asserted-by":"publisher","unstructured":"Xie X, Ma L, Juefei-Xu F, Xue M, Chen H, Liu Y, Zhao J, Li B, Yin J, See S (2019) Deephunter: a coverage-guided fuzz testing framework for deep neural networks. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA \u201919. ACM, New York, pp 146\u2013157. https:\/\/doi.org\/10.1145\/3293882.3330579","DOI":"10.1145\/3293882.3330579"},{"key":"9881_CR126","unstructured":"Young M, Pezz\u00e8 M (2005) Software Testing and Analysis: process, Principles and Techniques. Wiley, USA"},{"key":"9881_CR127","doi-asserted-by":"publisher","unstructured":"Zhang J, Jing X, Zhang W, Wang H, Dong Y (2016) Improve the quality of arc systems based on the metamorphic testing. In: 2016 International symposium on system and software reliability (ISSSR), pp 137\u2013141. https:\/\/doi.org\/10.1109\/ISSSR.2016.029","DOI":"10.1109\/ISSSR.2016.029"},{"key":"9881_CR128","doi-asserted-by":"publisher","unstructured":"Zhang Y, Chen Y, Cheung SC, Xiong Y, Zhang L (2018a) An empirical study on tensorflow program bugs. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018. ACM, New York, pp 129\u2013140. https:\/\/doi.org\/10.1145\/3213846.3213866","DOI":"10.1145\/3213846.3213866"},{"key":"9881_CR129","doi-asserted-by":"crossref","unstructured":"Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018b) Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, ASE, pp 132\u2013142","DOI":"10.1145\/3238147.3238187"},{"key":"9881_CR130","unstructured":"Zhang L, Sun X, Li Y, Zhang Z (2019) A noise-sensitivity-analysis-based test prioritization technique for deep neural networks. CoRR aRxiv:1901.00054"},{"key":"9881_CR131","doi-asserted-by":"crossref","unstructured":"Zhang JM, Harman M, Ma L, Liu Y (2020) Machine learning testing: survey, landscapes and horizons. IEEE Trans Softw Eng:1\u20131","DOI":"10.1109\/TSE.2019.2962027"},{"key":"9881_CR132","doi-asserted-by":"crossref","unstructured":"Zhao X, Gao X (2018) An ai software test method based on scene deductive approach. In: 2018 IEEE International conference on software quality, reliability and security companion (QRS-c). IEEE, pp 14\u201320","DOI":"10.1109\/QRS-C.2018.00017"},{"key":"9881_CR133","doi-asserted-by":"crossref","unstructured":"Zheng W, Wang W, Liu D, Zhang C, Zeng Q, Deng Y, Yang W, He P, Xie T (2019) Testing untestable neural machine translation: an industrial case. In: Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings. IEEE Press, pp 314\u2013315","DOI":"10.1109\/ICSE-Companion.2019.00131"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09881-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-020-09881-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09881-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,14]],"date-time":"2021-09-14T20:58:18Z","timestamp":1631653098000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-020-09881-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,15]]},"references-count":133,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2020,11]]}},"alternative-id":["9881"],"URL":"https:\/\/doi.org\/10.1007\/s10664-020-09881-0","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,15]]},"assertion":[{"value":"15 September 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}