{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T23:41:21Z","timestamp":1768520481010,"version":"3.49.0"},"reference-count":99,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T00:00:00Z","timestamp":1685059200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Fonds de Recherche du Quebec"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,10,31]]},"abstract":"<jats:p>At present, we are witnessing an increasing effort to improve the performance and trustworthiness of Deep Neural Networks (DNNs), with the aim to enable their adoption in safety critical systems such as self-driving cars or aircraft collision-avoidance systems. Multiple testing techniques are proposed to generate test cases that can expose inconsistencies in the behavior of DNN models. These techniques assume implicitly that the training program is bug-free and appropriately configured. However, satisfying this assumption for a novel problem requires significant engineering work to prepare the data, design the DNN, implement the training program, and tune the hyperparameters to produce the model for which current automated test data generators search for corner-case behaviors. All these model training steps can be error prone. Therefore, it is crucial to detect and correct errors throughout all the engineering steps of DNN-based software systems and not only on the resulting DNN model.<\/jats:p>\n          <jats:p>\n            In this article, we gather a catalog of training issues and based on their symptoms and their effects on the behavior of the training program, we propose practical verification routines to detect the aforementioned issues, automatically, by continuously validating that some important properties of the learning dynamics hold during the training. Then, we design\n            <jats:italic>TheDeepChecker<\/jats:italic>\n            , an end-to-end property-based debugging approach for DNN training programs and implement it as a TensorFlow-based library. As an empirical evaluation, we conduct a case study to assess the effectiveness of\n            <jats:italic>TheDeepChecker<\/jats:italic>\n            on synthetic and real-world buggy DL programs and compare its performance to that of the Amazon SageMaker Debugger (\n            <jats:italic>SMD<\/jats:italic>\n            ). Results show that\n            <jats:italic>TheDeepChecker<\/jats:italic>\n            \u2019s on-execution validation of DNN-based program\u2019s properties through three sequential phases (pre-, on-, and post-fitting) succeeds in revealing several coding bugs and system misconfigurations errors early on and at a low cost. Moreover, our property-based approach outperforms the\n            <jats:italic>SMD<\/jats:italic>\n            \u2019s offline rules verification on training logs in terms of detection accuracy for unstable learning issues and coverage of additional DL bugs.\n          <\/jats:p>","DOI":"10.1145\/3529318","type":"journal-article","created":{"date-parts":[[2022,5,10]],"date-time":"2022-05-10T11:17:26Z","timestamp":1652181446000},"page":"1-61","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Testing Feedforward Neural Networks Training Programs"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0542-9140","authenticated-orcid":false,"given":"Houssem","family":"Ben Braiek","sequence":"first","affiliation":[{"name":"SWAT Lab., Polytechnique Montr\u00e9al, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5704-4173","authenticated-orcid":false,"given":"Foutse","family":"Khomh","sequence":"additional","affiliation":[{"name":"SWAT Lab., Polytechnique Montr\u00e9al, Canada"}]}],"member":"320","published-online":{"date-parts":[[2023,5,26]]},"reference":[{"key":"e_1_3_1_2_2","article-title":"TensorFlow: Large-scale machine learning on heterogeneous distributed systems","volume":"1603","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal J\u00f3zefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Man\u00e9, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Vi\u00e9gas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: Large-scale machine learning on heterogeneous distributed systems. CoRR abs\/1603.04467 (2016).","journal-title":"CoRR"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338937"},{"key":"e_1_3_1_4_2","volume-title":"Visualization: How to Visualize, Monitor and Debug Neural Network Learning. Retrieved from","author":"Ng Kian Katanforoosh Andrew","year":"2020","unstructured":"Kian Katanforoosh Andrew Ng and Younes Bensouda Mourri. 2020. Visualization: How to Visualize, Monitor and Debug Neural Network Learning. Retrieved fromhttps:\/\/www.coursera.org\/learn\/deep-neural-network."},{"key":"e_1_3_1_5_2","volume-title":"Correlation and Regression Analysis: A Historian\u2019s Guide","author":"Archdeacon Thomas J.","year":"1994","unstructured":"Thomas J. Archdeacon. 1994. Correlation and Regression Analysis: A Historian\u2019s Guide. University of Wisconsin Press, Madison, WI."},{"key":"e_1_3_1_6_2","article-title":"Towards principled methods for training generative adversarial networks","author":"Arjovsky Martin","year":"2017","unstructured":"Martin Arjovsky and L\u00e9on Bottou. 2017. Towards principled methods for training generative adversarial networks. arXiv:1701.04862. Retrieved from https:\/\/arxiv.org\/abs\/1701.04862.","journal-title":"arXiv:1701.04862"},{"key":"e_1_3_1_7_2","article-title":"Layer normalization","author":"Ba Jimmy Lei","year":"2016","unstructured":"Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv:1607.06450. Retrieved from https:\/\/arxiv.org\/abs\/1607.06450.","journal-title":"arXiv:1607.06450"},{"key":"e_1_3_1_8_2","article-title":"Security and privacy issues in deep learning","author":"Bae Ho","year":"2018","unstructured":"Ho Bae, Jaehee Jang, Dahuin Jung, Hyemi Jang, Heonseok Ha, and Sungroh Yoon. 2018. Security and privacy issues in deep learning. arXiv:1807.11655. Retrieved from https:\/\/arxiv.org\/abs\/1807.11655.","journal-title":"arXiv:1807.11655"},{"key":"e_1_3_1_9_2","first-page":"2814","volume-title":"Advances in Neural Information Processing Systems","author":"Baldi Pierre","year":"2013","unstructured":"Pierre Baldi and Peter J. Sadowski. 2013. Understanding dropout. In Advances in Neural Information Processing Systems. 2814\u20132822."},{"key":"e_1_3_1_10_2","volume-title":"Pitfalls of Batch Norm in TensorFlow and Sanity Checks for Training Networks","author":"Bansal Dishank","year":"2020","unstructured":"Dishank Bansal. 2020. Pitfalls of Batch Norm in TensorFlow and Sanity Checks for Training Networks. Retrieved from https:\/\/towardsdatascience.com\/pitfalls-of-batch-norm-in-tensorflow-and-sanity-checks-for-training-networks-e86c207548c8."},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2019.00078"},{"key":"e_1_3_1_13_2","article-title":"On testing machine learning programs","author":"Braiek Houssem Ben","year":"2018","unstructured":"Houssem Ben Braiek and Houssem Khomh. 2018. On testing machine learning programs. arXiv:1812.02257. Retrieved from https:\/\/arxiv.org\/abs\/1812.02257.","journal-title":"arXiv:1812.02257"},{"key":"e_1_3_1_14_2","first-page":"7694","volume-title":"Advances in Neural Information Processing Systems","author":"Bjorck Nils","year":"2018","unstructured":"Nils Bjorck, Carla P. Gomes, Bart Selman, and Kilian Q. Weinberger. 2018. Understanding batch normalization. In Advances in Neural Information Processing Systems. 7694\u20137705."},{"key":"e_1_3_1_15_2","volume-title":"Multilayer Neural Networks","author":"Bottou Leon","year":"2015","unstructured":"Leon Bottou. 2015. Multilayer Neural Networks. Retrieved from http:\/\/videolectures.net\/site\/normal_dl\/tag=983658\/deeplearning2015_bottou_neural_networks_01.pdf."},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196445"},{"key":"e_1_3_1_17_2","volume-title":"Proceedings of the Reliable Machine Learning in the Wild\u2014NIPS\u201916 Workshop","author":"Cai Shanqing","year":"2016","unstructured":"Shanqing Cai, Eric Breck, Eric Nielsen, Michael Salib, and D. Sculley. 2016. Tensorflow debugger: Debugging dataflow graphs for machine learning. In Proceedings of the Reliable Machine Learning in the Wild\u2014NIPS\u201916 Workshop."},{"key":"e_1_3_1_18_2","article-title":"An analysis of deep neural network models for practical applications","author":"Canziani Alfredo","year":"2016","unstructured":"Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. 2016. An analysis of deep neural network models for practical applications. arXiv:1605.07678. Retrieved from https:\/\/arxiv.org\/abs\/1605.07678.","journal-title":"arXiv:1605.07678"},{"key":"e_1_3_1_19_2","volume-title":"My Neural Network Isn\u2019t Working! What Should I Do?","author":"community Reddit D. L.","year":"2018","unstructured":"Reddit D. L. community. 2018. My Neural Network Isn\u2019t Working! What Should I Do? Retrieved from https:\/\/www.reddit.com\/r\/MachineLearning\/comments\/6xvnwo\/d_my_neural_network_isnt_working_what_should_i_do\/."},{"key":"e_1_3_1_20_2","article-title":"Underspecification presents challenges for credibility in modern machine learning","author":"D\u2019Amour Alexander","year":"2020","unstructured":"Alexander D\u2019Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, et\u00a0al. 2020. Underspecification presents challenges for credibility in modern machine learning. arXiv:2011.03395. Retrieved from https:\/\/arxiv.org\/abs\/2011.03395.","journal-title":"arXiv:2011.03395"},{"key":"e_1_3_1_21_2","volume-title":"Troubleshooting Neural Net Training","year":"2019","unstructured":"Deeplearning4j. 2019. Troubleshooting Neural Net Training. Retrieved from https:\/\/deeplearning4j.konduit.ai\/tuning-and-training\/troubleshooting-training."},{"key":"e_1_3_1_22_2","unstructured":"Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. Retrieved from http:\/\/archive.ics.uci.edu\/ml."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213858"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213858"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.908957"},{"key":"e_1_3_1_26_2","article-title":"Detecting dead weights and units in neural networks","author":"Evci Utku","year":"2018","unstructured":"Utku Evci. 2018. Detecting dead weights and units in neural networks. arXiv:1806.06068. Retrieved from https:\/\/arxiv.org\/abs\/1806.06068.","journal-title":"arXiv:1806.06068"},{"key":"e_1_3_1_27_2","volume-title":"MutPy 0.4.0","author":"Foundation Python Software","year":"2013","unstructured":"Python Software Foundation. 2013. MutPy 0.4.0. Retrieved from https:\/\/pypi.python.org\/pypi\/."},{"key":"e_1_3_1_28_2","unstructured":"Yarin Gal and Zoubin Ghahramani. 2015. Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv:1506.02158. Retrieved from https:\/\/arxiv.org\/abs\/1506.02158. arXiv:1506.02158"},{"key":"e_1_3_1_29_2","first-page":"1","article-title":"Dropout vs. batch normalization: an empirical study of their impact to deep learning","author":"Garbin Christian","year":"2020","unstructured":"Christian Garbin, Xingquan Zhu, and Oge Marques. 2020. Dropout vs. batch normalization: an empirical study of their impact to deep learning. Multimedia Tools Appl. (2020), 1\u201339.","journal-title":"Multimedia Tools Appl."},{"key":"e_1_3_1_30_2","first-page":"249","volume-title":"Proceedings of the 13th International Conference on Artificial Intelligence and Statistics","author":"Glorot Xavier","year":"2010","unstructured":"Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. 249\u2013256."},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.5555\/3086952"},{"key":"e_1_3_1_32_2","volume-title":"Basic Regression: Predict Fuel Efficiency","year":"2017","unstructured":"Google. 2017. Basic Regression: Predict Fuel Efficiency. Retrieved from https:\/\/www.tensorflow.org\/tutorials\/keras\/regression."},{"key":"e_1_3_1_33_2","volume-title":"Google Machine Learning Crash Course","year":"2020","unstructured":"Google. 2020. Google Machine Learning Crash Course. Retrieved from https:\/\/developers.google.com\/machine-learning\/crash-course."},{"key":"e_1_3_1_34_2","article-title":"Lecture 15: Exploding and vanishing gradients","author":"Grosse Roger","year":"2017","unstructured":"Roger Grosse. 2017. Lecture 15: Exploding and vanishing gradients. University of Toronto Computer Science.","journal-title":"University of Toronto Computer Science"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/1656274.1656278"},{"key":"e_1_3_1_37_2","article-title":"Towards principled design of deep convolutional networks: Introducing SimpNet","author":"Hasanpour Seyyed Hossein","year":"2018","unstructured":"Seyyed Hossein Hasanpour, Mohammad Rouhani, Mohsen Fayyaz, Mohammad Sabokrou, and Ehsan Adeli. 2018. Towards principled design of deep convolutional networks: Introducing SimpNet. arXiv:1802.06205. Retrieved from https:\/\/arxiv.org\/abs\/1802.06205.","journal-title":"arXiv:1802.06205"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.123"},{"key":"e_1_3_1_39_2","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1007\/978-3-642-35289-8_32","volume-title":"Neural Networks: Tricks of the Trade","author":"Hinton Geoffrey E.","year":"2012","unstructured":"Geoffrey E. Hinton. 2012. A practical guide to training restricted Boltzmann machines. In Neural Networks: Tricks of the Trade. Springer, 599\u2013619."},{"key":"e_1_3_1_40_2","article-title":"Visual analytics in deep learning: An interrogative survey for the next frontiers","author":"Hohman Fred Matthew","year":"2018","unstructured":"Fred Matthew Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE Trans. Vis. Comput. Graph. (2018).","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_3_1_42_2","article-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","author":"Ioffe Sergey","year":"2015","unstructured":"Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167. Retrieved from https:\/\/arxiv.org\/abs\/1502.03167.","journal-title":"arXiv:1502.03167"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380378"},{"key":"e_1_3_1_46_2","volume-title":"37 Reasons Why Your Neural Network Is Not Working","author":"Ivanov Slav","year":"2017","unstructured":"Slav Ivanov. 2017. 37 Reasons Why Your Neural Network Is Not Working. Retrieved from https:\/\/blog.slavv.com\/37-reasons-why-your-neural-network-is-not-working-4020854bd607."},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_1_48_2","volume-title":"Visualization: How to Visualize, Monitor and Debug Neural Network Learning. Retrieved from","author":"Loh Paul Dubs Jing Zhi","year":"2020","unstructured":"Paul Dubs Jing Zhi Loh and Shams U. I. Azeem. 2020. Visualization: How to Visualize, Monitor and Debug Neural Network Learning. Retrieved fromhttps:\/\/deeplearning4j.konduit.ai\/tuning-and-training\/visualization."},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744718"},{"key":"e_1_3_1_50_2","volume-title":"Most Common Neural Net Mistakes (tweet)","author":"Karpathy Andrej","year":"2018","unstructured":"Andrej Karpathy. 2018. Most Common Neural Net Mistakes (tweet). Retrieved from https:\/\/twitter.com\/karpathy\/status\/1013244313327681536?lang=en."},{"key":"e_1_3_1_51_2","volume-title":"Neural Networks Part 3: Learning and Evaluation","author":"Karpathy Andrej","year":"2018","unstructured":"Andrej Karpathy. 2018. Neural Networks Part 3: Learning and Evaluation. Retrieved from http:\/\/cs231n.github.io\/neural-networks-3\/."},{"key":"e_1_3_1_52_2","volume-title":"A Recipe for Training Neural Networks","author":"Karpathy Andrej","year":"2020","unstructured":"Andrej Karpathy. 2020. A Recipe for Training Neural Networks. Retrieved from http:\/\/karpathy.github.io\/2019\/04\/25\/recipe\/."},{"key":"e_1_3_1_53_2","article-title":"Progressive growing of gans for improved quality, stability, and variation","author":"Karras Tero","year":"2017","unstructured":"Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv:1710.10196. Retrieved from https:\/\/arxiv.org\/abs\/1710.10196.","journal-title":"arXiv:1710.10196"},{"key":"e_1_3_1_54_2","article-title":"Towards proving the adversarial robustness of deep neural networks","author":"Katz Guy","year":"2017","unstructured":"Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. 2017. Towards proving the adversarial robustness of deep neural networks. arXiv:1709.02802. Retrieved from https:\/\/arxiv.org\/abs\/1709.02802.","journal-title":"arXiv:1709.02802"},{"key":"e_1_3_1_55_2","article-title":"Similarity of neural network representations revisited","author":"Kornblith Simon","year":"2019","unstructured":"Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. arXiv:1905.00414. Retrieved from https:\/\/arxiv.org\/abs\/1905.00414.","journal-title":"arXiv:1905.00414"},{"key":"e_1_3_1_56_2","unstructured":"Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2014. The cifar-10 Dataset. Retrieved from http:\/\/www.cs.toronto.edu\/kriz\/cifar.html."},{"key":"e_1_3_1_57_2","unstructured":"Yann LeCun. 1998. The MNIST Database of Handwritten Digits. Retrieved from http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"e_1_3_1_58_2","unstructured":"Yann LeCun et\u00a0al. 2015. LeNet-5 Convolutional Neural Networks. Retrieved from http:\/\/yann.lecun.com\/exdb\/lenet."},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_1_60_2","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/978-3-642-35289-8_3","volume-title":"Neural Networks: Tricks of the Trade","author":"LeCun Yann A.","year":"2012","unstructured":"Yann A. LeCun, L\u00e9on Bottou, Genevieve B. Orr, and Klaus-Robert M\u00fcller. 2012. Efficient backprop. In Neural Networks: Tricks of the Trade. Springer, 9\u201348."},{"key":"e_1_3_1_61_2","first-page":"6389","volume-title":"Advances in Neural Information Processing Systems","author":"Li Hao","year":"2018","unstructured":"Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems. 6389\u20136399."},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00279"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2018.8802509"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598831"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2016.12.038"},{"key":"e_1_3_1_66_2","article-title":"Dying relu and initialization: Theory and numerical examples","author":"Lu Lu","year":"2019","unstructured":"Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis. 2019. Dying relu and initialization: Theory and numerical examples. arXiv:1903.06733. Retrieved from https:\/\/arxiv.org\/abs\/1903.06733.","journal-title":"arXiv:1903.06733"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_1_68_2","article-title":"DeepMutation: Mutation testing of deep learning systems","author":"Ma Lei","year":"2018","unstructured":"Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, et\u00a0al. 2018. DeepMutation: Mutation testing of deep learning systems. arXiv:1805.05206. Retrieved from https:\/\/arxiv.org\/abs\/1805.05206.","journal-title":"arXiv:1805.05206"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.5555\/1077303.1077304"},{"key":"e_1_3_1_70_2","volume-title":"CS231n: Convolutional Neural Networks for Visual Recognition","year":"2020","unstructured":"MIT. 2020. CS231n: Convolutional Neural Networks for Visual Recognition. Retrieved from https:\/\/cs231n.github.io\/."},{"key":"e_1_3_1_71_2","volume-title":"MIT Deep Learning Lectures","year":"2020","unstructured":"MIT. 2020. MIT Deep Learning Lectures. Retrieved from https:\/\/deeplearning.mit.edu\/."},{"key":"e_1_3_1_72_2","article-title":"Survey of resampling techniques for improving classification performance in unbalanced datasets","author":"More Ajinkya","year":"2016","unstructured":"Ajinkya More. 2016. Survey of resampling techniques for improving classification performance in unbalanced datasets. arXiv:1608.06048. Retrieved from https:\/\/arxiv.org\/abs\/1608.06048.","journal-title":"arXiv:1608.06048"},{"key":"e_1_3_1_73_2","volume-title":"Specialization in Deep Learning","author":"Ng Andrew","year":"2020","unstructured":"Andrew Ng, Kian Katanforoosh, and Younes Bensouda Mourri. 2020. Specialization in Deep Learning. Retrieved from https:\/\/www.coursera.org\/specializations\/deep-learning."},{"key":"e_1_3_1_74_2","article-title":"TensorFuzz: Debugging neural networks with coverage-guided fuzzing","author":"Odena Augustus","year":"2018","unstructured":"Augustus Odena and Ian Goodfellow. 2018. TensorFuzz: Debugging neural networks with coverage-guided fuzzing. arXiv:1807.10875. Retrieved from https:\/\/arxiv.org\/abs\/1807.10875.","journal-title":"arXiv:1807.10875"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-22102-1_22"},{"key":"e_1_3_1_76_2","article-title":"Gradient-coherent strong regularization for deep neural networks","author":"Park Dae Hoon","year":"2018","unstructured":"Dae Hoon Park, Chiu Man Ho, Yi Chang, and Huaqing Zhang. 2018. Gradient-coherent strong regularization for deep neural networks. arXiv:1811.08056. Retrieved from https:\/\/arxiv.org\/abs\/1811.08056.","journal-title":"arXiv:1811.08056"},{"key":"e_1_3_1_77_2","first-page":"1310","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Pascanu Razvan","year":"2013","unstructured":"Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1310\u20131318."},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_1_79_2","article-title":"The effectiveness of data augmentation in image classification using deep learning","author":"Perez Luis","year":"2017","unstructured":"Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621. Retrieved from https:\/\/arxiv.org\/abs\/1712.04621.","journal-title":"arXiv:1712.04621"},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744358"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI.2015.202"},{"key":"e_1_3_1_82_2","article-title":"Amazon sagemaker debugger: A system for real-time insights into machine learning model training","volume":"3","author":"Rauschmayr Nathalie","year":"2021","unstructured":"Nathalie Rauschmayr, Vikas Kumar, Rahul Huilgol, Andrea Olgiati, Satadal Bhattacharjee, Nihal Harish, Vandana Kannan, Amol Lele, Anirudh Acharya, Jared Nielsen, et\u00a0al. 2021. Amazon sagemaker debugger: A system for real-time insights into machine learning model training. Proceedings of Machine Learning and Systems 3 (2021).","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_1_83_2","volume-title":"How to Unit Test Machine Learning Code. Retrieved from","author":"Roberts Chase","year":"2018","unstructured":"Chase Roberts. 2018. How to Unit Test Machine Learning Code. Retrieved fromhttps:\/\/medium.com\/@keeper6928\/how-to-unit-test-machine-learning-code-57cf6fd81765."},{"key":"e_1_3_1_84_2","article-title":"Stabilizing training of generative adversarial networks through regularization","author":"Roth Kevin","year":"2017","unstructured":"Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. 2017. Stabilizing training of generative adversarial networks through regularization. arXiv:1705.09367. Retrieved from https:\/\/arxiv.org\/abs\/1705.09367.","journal-title":"arXiv:1705.09367"},{"key":"e_1_3_1_85_2","article-title":"Weight normalization: A simple reparameterization to accelerate training of deep neural networks","author":"Salimans Tim","year":"2016","unstructured":"Tim Salimans and Diederik P. Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv:1602.07868. Retrieved from https:\/\/arxiv.org\/abs\/1602.07868.","journal-title":"arXiv:1602.07868"},{"key":"e_1_3_1_86_2","first-page":"2483","volume-title":"Advances in Neural Information Processing Systems","author":"Santurkar Shibani","year":"2018","unstructured":"Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. 2018. How does batch normalization help optimization? In Advances in Neural Information Processing Systems. 2483\u20132493."},{"key":"e_1_3_1_87_2","article-title":"Developing bug-free machine learning systems with formal mathematics","author":"Selsam Daniel","year":"2017","unstructured":"Daniel Selsam, Percy Liang, and David L. Dill. 2017. Developing bug-free machine learning systems with formal mathematics. arXiv:1706.08605. Retrieved from https:\/\/arxiv.org\/abs\/1706.08605.","journal-title":"arXiv:1706.08605"},{"key":"e_1_3_1_88_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb01338.x"},{"key":"e_1_3_1_89_2","volume-title":"Checklist for Debugging Neural Networks","author":"Shao Cecelia","year":"2019","unstructured":"Cecelia Shao. 2019. Checklist for Debugging Neural Networks. Retrieved from https:\/\/towardsdatascience.com\/checklist-for-debugging-neural-networks-d8b2a9434f21."},{"key":"e_1_3_1_90_2","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https:\/\/arxiv.org\/abs\/1409.1556.","journal-title":"arXiv:1409.1556"},{"key":"e_1_3_1_91_2","article-title":"Deep convolutional neural network design patterns","author":"Smith Leslie N.","year":"2016","unstructured":"Leslie N. Smith and Nicholay Topin. 2016. Deep convolutional neural network design patterns. arXiv:1611.00847. Retrieved from https:\/\/arxiv.org\/abs\/1611.00847.","journal-title":"arXiv:1611.00847"},{"key":"e_1_3_1_92_2","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2670313"},{"key":"e_1_3_1_93_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_3_1_94_2","unstructured":"AWS Team."},{"key":"e_1_3_1_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_1_96_2","first-page":"228","article-title":"A theory of learning with corrupted labels.","volume":"18","author":"Rooyen Brendan Van","year":"2017","unstructured":"Brendan Van Rooyen and Robert C. Williamson. 2017. A theory of learning with corrupted labels. J. Mach. Learn. Res. 18 (2017), 228\u20131.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_1_97_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_1_98_2","article-title":"Coverage-guided fuzzing for deep neural networks","author":"Xie Xiaofei","year":"2018","unstructured":"Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Hongxu Chen, Minhui Xue, Bo Li, Yang Liu, Jianjun Zhao, Jianxiong Yin, and Simon See. 2018. Coverage-guided fuzzing for deep neural networks. arXiv:1809.01266. Retrieved from https:\/\/arxiv.org\/abs\/1809.01266.","journal-title":"arXiv:1809.01266"},{"key":"e_1_3_1_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864499"},{"key":"e_1_3_1_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3529318","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3529318","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:38Z","timestamp":1750188638000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3529318"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,26]]},"references-count":99,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,10,31]]}},"alternative-id":["10.1145\/3529318"],"URL":"https:\/\/doi.org\/10.1145\/3529318","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,26]]},"assertion":[{"value":"2020-08-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-28","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-05-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}