{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T04:04:29Z","timestamp":1792123469407,"version":"4.4.0"},"publisher-location":"New York, NY, USA","reference-count":40,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,7,11]],"date-time":"2021-07-11T00:00:00Z","timestamp":1625961600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["787703"],"award-info":[{"award-number":["787703"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,11]]},"DOI":"10.1145\/3460319.3464825","type":"proceedings-article","created":{"date-parts":[[2021,7,8]],"date-time":"2021-07-08T22:18:43Z","timestamp":1625782723000},"page":"67-78","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":127,"title":["DeepCrime: mutation testing of deep learning systems based on real faults"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3037-8368","authenticated-orcid":false,"given":"Nargiz","family":"Humbatova","sequence":"first","affiliation":[{"name":"USI Lugano, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1423-1083","authenticated-orcid":false,"given":"Gunel","family":"Jahangirova","sequence":"additional","affiliation":[{"name":"USI Lugano, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3088-0339","authenticated-orcid":false,"given":"Paolo","family":"Tonella","sequence":"additional","affiliation":[{"name":"USI Lugano, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,7,11]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2013. DiffMerge: an application to visually compare and merge files on Windows OS X and Linux. https:\/\/sourcegear.com\/diffmerge\/  2013. DiffMerge: an application to visually compare and merge files on Windows OS X and Linux. https:\/\/sourcegear.com\/diffmerge\/"},{"key":"e_1_3_2_1_2_1","unstructured":"2019. FrameworkData. https:\/\/towardsdatascience.com\/deep-learning-framework-power-scores-2018-23607ddf297a  2019. FrameworkData. https:\/\/towardsdatascience.com\/deep-learning-framework-power-scores-2018-23607ddf297a"},{"key":"e_1_3_2_1_3_1","unstructured":"2020. DeepCrime Replication Package. https:\/\/zenodo.org\/record\/4772465  2020. DeepCrime Replication Package. https:\/\/zenodo.org\/record\/4772465"},{"key":"e_1_3_2_1_4_1","unstructured":"2020. An implementation of a multimodal CNN for appearance-based gaze estimation. https:\/\/github.com\/dlsuroviki\/UnityEyesModel  2020. An implementation of a multimodal CNN for appearance-based gaze estimation. https:\/\/github.com\/dlsuroviki\/UnityEyesModel"},{"key":"e_1_3_2_1_5_1","unstructured":"2020. Keras Code Examples. Available at https:\/\/keras.io\/examples\/  2020. Keras Code Examples. Available at https:\/\/keras.io\/examples\/"},{"key":"e_1_3_2_1_6_1","unstructured":"2020. Keras MNIST CNN Model. Available at https:\/\/keras.io\/examples\/vision\/mnist_convnet\/  2020. Keras MNIST CNN Model. Available at https:\/\/keras.io\/examples\/vision\/mnist_convnet\/"},{"key":"e_1_3_2_1_7_1","unstructured":"2020. Keras Movie Recommender Model. Available at https:\/\/keras.io\/examples\/structured_data\/collaborative_filtering_movielens\/  2020. Keras Movie Recommender Model. Available at https:\/\/keras.io\/examples\/structured_data\/collaborative_filtering_movielens\/"},{"key":"e_1_3_2_1_8_1","unstructured":"2020. Movie Recommender Dataset. Available at http:\/\/files.grouplens.org\/datasets\/movielens\/ml-latest-small.zip  2020. Movie Recommender Dataset. Available at http:\/\/files.grouplens.org\/datasets\/movielens\/ml-latest-small.zip"},{"key":"e_1_3_2_1_9_1","unstructured":"2020. Speaker Recognition Dataset. Available at https:\/\/www.kaggle.com\/kongaevans\/speaker-recognition-dataset  2020. Speaker Recognition Dataset. Available at https:\/\/www.kaggle.com\/kongaevans\/speaker-recognition-dataset"},{"key":"e_1_3_2_1_10_1","unstructured":"2020. Speaker Recognition Model. Available at https:\/\/keras.io\/examples\/audio\/speaker_recognition_using_cnn\/  2020. Speaker Recognition Model. Available at https:\/\/keras.io\/examples\/audio\/speaker_recognition_using_cnn\/"},{"key":"e_1_3_2_1_11_1","volume-title":"Software System Testing and Quality Assurance","author":"Beizer Boris","unstructured":"Boris Beizer . 1984. Software System Testing and Quality Assurance . Van Nostrand Reinhold Co. , New York, NY, USA . isbn:0-442-21306-9 Boris Beizer. 1984. Software System Testing and Quality Assurance. Van Nostrand Reinhold Co., New York, NY, USA. isbn:0-442-21306-9"},{"key":"e_1_3_2_1_12_1","volume-title":"Davide Del Testa","author":"Bojarski Mariusz","year":"2016","unstructured":"Mariusz Bojarski , Davide Del Testa , Daniel Dworakowski, Bernhard Firner , Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. 2016 . End to End Learning for Self-Driving Cars.. CoRR , abs\/1604.07316 (2016), arxiv:1604.07316 Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. 2016. End to End Learning for Self-Driving Cars.. CoRR, abs\/1604.07316 (2016), arxiv:1604.07316"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.000-6"},{"key":"e_1_3_2_1_14_1","unstructured":"2020. DeepCrime. https:\/\/github.com\/dlfaults\/deepcrime  2020. DeepCrime. https:\/\/github.com\/dlfaults\/deepcrime"},{"key":"e_1_3_2_1_15_1","volume-title":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems. 1019\u20131027","author":"Gal Yarin","year":"2016","unstructured":"Yarin Gal and Zoubin Ghahramani . 2016 . A Theoretically Grounded Application of Dropout in Recurrent Neural Networks . In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems. 1019\u20131027 . Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems. 1019\u20131027."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00029"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00030"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00126"},{"key":"e_1_3_2_1_19_1","unstructured":"Nargiz Humbatova Gunel Jahangirova Gabriele Bavota Vincenzo Riccio Andrea Stocco and Paolo Tonella. [n.d.]. Dataset of Real Faults in Deep Learning Systems. https:\/\/zenodo.org\/record\/3667541#.Xzmily2B3zs  Nargiz Humbatova Gunel Jahangirova Gabriele Bavota Vincenzo Riccio Andrea Stocco and Paolo Tonella. [n.d.]. Dataset of Real Faults in Deep Learning Systems. https:\/\/zenodo.org\/record\/3667541#.Xzmily2B3zs"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00018"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0028086"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2014.20"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950322"},{"key":"e_1_3_2_1_27_1","unstructured":"Yann LeCun. 1998. The MNIST Database of Handwritten Digits. Available at http:\/\/yann. lecun. com\/exdb\/mnist\/  Yann LeCun. 1998. The MNIST Database of Handwritten Digits. Available at http:\/\/yann. lecun. com\/exdb\/mnist\/"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.2307\/2344614"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00016"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/QRS-C.2018.00032"},{"key":"e_1_3_2_1_34_1","unstructured":"Jeongju Sohn Sungmin Kang and Shin Yoo. 2019. Search Based Repair of Deep Neural Networks. arXiv preprint arXiv:1912.12463 arxiv:1912.12463  Jeongju Sohn Sungmin Kang and Shin Yoo. 2019. Search Based Repair of Deep Neural Networks. arXiv preprint arXiv:1912.12463 arxiv:1912.12463"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00046"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1927.10502953"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2857491.2857492"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0019-9958(65)90241-X"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"}],"event":{"name":"ISSTA '21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Virtual Denmark","acronym":"ISSTA '21","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3460319.3464825","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3460319.3464825","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:32Z","timestamp":1750193312000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3460319.3464825"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,11]]},"references-count":40,"alternative-id":["10.1145\/3460319.3464825","10.1145\/3460319"],"URL":"https:\/\/doi.org\/10.1145\/3460319.3464825","relation":{},"subject":[],"published":{"date-parts":[[2021,7,11]]},"assertion":[{"value":"2021-07-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}