{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T03:37:56Z","timestamp":1776310676658,"version":"3.50.1"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T00:00:00Z","timestamp":1695945600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62002256, 62232001, and 62202324"],"award-info":[{"award-number":["62002256, 62232001, and 62202324"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,11,30]]},"abstract":"<jats:p>\n            DL frameworks are the basis of constructing all DL programs and models, and thus their bugs could lead to the unexpected behaviors of any DL program or model relying on them. Such a wide effect demonstrates the necessity and importance of guaranteeing DL frameworks\u2019 quality. Understanding the characteristics of DL framework bugs is a fundamental step for this quality assurance task, facilitating designing effective bug detection and debugging approaches. Hence, in this work, we conduct the most large-scale study on 1,000 bugs from four popular and diverse DL frameworks (i.e., TensorFlow, PyTorch, MXNet, and DL4J). By analyzing the root causes and symptoms of DL framework bugs associated with five components decomposed from DL frameworks, as well as measuring test coverage achieved by three state-of-the-art testing techniques, we obtain 12 major findings for the comprehensive understanding of DL framework bugs and the current status of existing DL framework testing practice, and then provide a series of actionable guidelines for better DL framework bug detection and debugging. Finally, based on the guidelines, we design and implement a prototype DL-framework testing tool, called\n            <jats:sc>TenFuzz<\/jats:sc>\n            , which is evaluated to be effective and finds three unknown bugs on the latest TensorFlow framework in a preliminary study, indicating the significance of our guidelines.\n          <\/jats:p>","DOI":"10.1145\/3587155","type":"journal-article","created":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T12:14:42Z","timestamp":1678968882000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":68,"title":["Toward Understanding Deep Learning Framework Bugs"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3056-9962","authenticated-orcid":false,"given":"Junjie","family":"Chen","sequence":"first","affiliation":[{"name":"Tianjin University"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9621-0878","authenticated-orcid":false,"given":"Yihua","family":"Liang","sequence":"additional","affiliation":[{"name":"Tianjin University"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6128-2123","authenticated-orcid":false,"given":"Qingchao","family":"Shen","sequence":"additional","affiliation":[{"name":"Tianjin University"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1983-6572","authenticated-orcid":false,"given":"Jiajun","family":"Jiang","sequence":"additional","affiliation":[{"name":"Tianjin University"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3409-8275","authenticated-orcid":false,"given":"Shuochuan","family":"Li","sequence":"additional","affiliation":[{"name":"Tianjin University"}]}],"member":"320","published-online":{"date-parts":[[2023,9,29]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Accessed: April 4th 2022. Coverage.py. Retrieved from https:\/\/coverage.readthedocs.io\/."},{"key":"e_1_3_2_3_2","unstructured":"Accessed: April 4th 2022. Deeplearning4J. Retrieved from https:\/\/deeplearning4j.org\/."},{"key":"e_1_3_2_4_2","unstructured":"Accessed: April 4th 2022. Gcov. Retrieved from https:\/\/gcc.gnu.org\/onlinedocs\/gcc\/Gcov.html."},{"key":"e_1_3_2_5_2","unstructured":"Accessed: April 4th 2022. Gradle. Retrieved from https:\/\/gradle.org\/."},{"key":"e_1_3_2_6_2","unstructured":"Accessed: April 4th 2022. MXNet. Retrieved from https:\/\/mxnet.apache.org."},{"key":"e_1_3_2_7_2","unstructured":"Accessed: April 4th 2022. News. https:\/\/www.vice.com\/en\/article\/9kga85\/uber-is-giving-up-on-self-driving-cars-in-california-after-deadly-crash."},{"key":"e_1_3_2_8_2","unstructured":"Accessed: April 4th 2022. News. https:\/\/www.newsweek.com\/autonomous-tesla-crashes-parked-fire-truck-california-freeway-789177."},{"key":"e_1_3_2_9_2","unstructured":"Accessed: April 4th 2022. PyTorch. Retrieved from https:\/\/pytorch.org."},{"key":"e_1_3_2_10_2","unstructured":"Accessed: April 4th 2022. TensorFlow. Retrieved from https:\/\/www.tensorflow.org."},{"key":"e_1_3_2_11_2","unstructured":"Accessed: April 4th 2022. Bazel. Retrieved from https:\/\/bazel.build\/."},{"key":"e_1_3_2_12_2","unstructured":"Accessed: April 4th 2022. Caffe. Retrieved from https:\/\/github.com\/intel\/caffe."},{"key":"e_1_3_2_13_2","unstructured":"Accessed: April 4th 2022. Keras. Retrieved from https:\/\/github.com\/keras-team\/keras."},{"key":"e_1_3_2_14_2","unstructured":"Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Gregory S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian J. Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal J\u00f3zefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dan Man\u00e9 Rajat Monga Sherry Moore Derek Gordon Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul A. Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda B. Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2016. TensorFlow: Large-scale machine learning on heterogeneous distributed systems. CoRR abs\/1603.04467 (2016). arXiv:1603.04467. http:\/\/arxiv.org\/abs\/1603.04467."},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","first-page":"464","DOI":"10.1145\/2901739.2903506","volume-title":"Proceedings of the 13th International Conference on Mining Software Repositories","author":"Amann Sven","year":"2016","unstructured":"Sven Amann, Sarah Nadi, Hoan A. Nguyen, Tien N. Nguyen, and Mira Mezini. 2016. MUBench: A benchmark for API-misuse detectors. In Proceedings of the 13th International Conference on Mining Software Repositories. 464\u2013467."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2827384"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.312"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/2627508.2627515"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394112"},{"key":"e_1_3_2_20_2","first-page":"509","volume-title":"Proceedings of 32nd IEEE\/ACM International Conference on Automated Software Engineering","author":"Franco Anthony Di","year":"2017","unstructured":"Anthony Di Franco, Hui Guo, and Cindy Rubio-Gonz\u00e1lez. 2017. A comprehensive study of real-world numerical bug characteristics. In Proceedings of 32nd IEEE\/ACM International Conference on Automated Software Engineering. 509\u2013519."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2020.3000681"},{"key":"e_1_3_2_22_2","unstructured":"Fabio Ferreira Luciana Lourdes Silva and Marco Tulio Valente. 2019. Software engineering meets deep learning: A literature review. arXiv:1909.11436."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380397"},{"key":"e_1_3_2_24_2","volume-title":"Proceedings of the 3rd International Conference on Learning Representations","author":"Goodfellow Ian J.","year":"2015","unstructured":"Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations."},{"key":"e_1_3_2_25_2","unstructured":"Jiazhen Gu Xuchuan Luo Yangfan Zhou and Xin Wang. 2022. Muffin: Testing deep learning libraries via neural architecture fuzzing. In Proceedings of the 44th International Conference on Software Engineering (ICSE\u201922) . 1418\u20131430."},{"key":"e_1_3_2_26_2","first-page":"486","volume-title":"Proceedings of the 2020 35th IEEE\/ACM International Conference on Automated Software Engineering","author":"Guo Qianyu","year":"2020","unstructured":"Qianyu Guo, Xiaofei Xie, Yi Li, Xiaoyu Zhang, Yang Liu, Xiaohong Li, and Chao Shen. 2020. Audee: Automated testing for deep learning frameworks. In Proceedings of the 2020 35th IEEE\/ACM International Conference on Automated Software Engineering. 486\u2013498."},{"key":"e_1_3_2_27_2","first-page":"868","volume-title":"Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution","author":"Han Junxiao","year":"2020","unstructured":"Junxiao Han, Shuiguang Deng, David Lo, Chen Zhi, Jianwei Yin, and Xin Xia. 2020. An empirical study of the dependency networks of deep learning libraries. In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution. IEEE, 868\u2013878."},{"key":"e_1_3_2_28_2","first-page":"23:1\u201323:10","volume-title":"Proceedings of the 10th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement","author":"Han Xue","year":"2016","unstructured":"Xue Han and Tingting Yu. 2016. An empirical study on performance bugs for highly configurable software systems. In Proceedings of the 10th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement. 23:1\u201323:10."},{"key":"e_1_3_2_29_2","volume-title":"Building Machine Learning Pipelines","author":"Hapke Hannes","year":"2020","unstructured":"Hannes Hapke and Catherine Nelson. 2020. Building Machine Learning Pipelines. O\u2019Reilly Media."},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","first-page":"1078","DOI":"10.1145\/3180155.3180181","volume-title":"Proceedings of the 2018 IEEE\/ACM 40th International Conference on Software Engineering","author":"Hassan Foyzul","year":"2018","unstructured":"Foyzul Hassan and Xiaoyin Wang. 2018. Hirebuild: An automatic approach to history-driven repair of build scripts. In Proceedings of the 2018 IEEE\/ACM 40th International Conference on Software Engineering. IEEE, 1078\u20131089."},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1109\/MSR.2019.00016","volume-title":"Proceedings of the 2019 IEEE\/ACM 16th International Conference on Mining Software Repositories","author":"Hoang Thong","year":"2019","unstructured":"Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi. 2019. DeepJIT: An end-to-end deep learning framework for just-in-time defect prediction. In Proceedings of the 2019 IEEE\/ACM 16th International Conference on Mining Software Repositories. IEEE, 34\u201345."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1007\/978-3-030-59410-7_40","volume-title":"Proceedings of the International Conference on Database Systems for Advanced Applications","author":"Jia Li","year":"2020","unstructured":"Li Jia, Hao Zhong, Xiaoyin Wang, Linpeng Huang, and Xuansheng Lu. 2020. An empirical study on bugs inside tensorflow. In Proceedings of the International Conference on Database Systems for Advanced Applications. 604\u2013620."},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"Li Jia Hao Zhong Xiaoyin Wang Linpeng Huang and Xuansheng Lu. 2021. The symptoms causes and repairs of bugs inside a deep learning library. Journal of Systems and Software 177 (2021) 110935.","DOI":"10.1016\/j.jss.2021.110935"},{"key":"e_1_3_2_36_2","first-page":"1","volume-title":"Proceedings of the 2016 IEEE\/AIAA 35th Digital Avionics Systems Conference","author":"Julian Kyle D.","year":"2016","unstructured":"Kyle D. Julian, Jessica Lopez, Jeffrey S. Brush, Michael P. Owen, and Mykel J. Kochenderfer. 2016. Policy compression for aircraft collision avoidance systems. In Proceedings of the 2016 IEEE\/AIAA 35th Digital Avionics Systems Conference. 1\u201310."},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","first-page":"3425","DOI":"10.18653\/v1\/2021.emnlp-main.275","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Kang Yuning","year":"2021","unstructured":"Yuning Kang, Zan Wang, Hongyu Zhang, Junjie Chen, and Hanmo You. 2021. Apirecx: Cross-library api recommendation via pre-trained language model. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3425\u20133436."},{"key":"e_1_3_2_38_2","first-page":"2009","article-title":"Open coding","volume":"23","author":"Khandkar Shahedul Huq","year":"2009","unstructured":"Shahedul Huq Khandkar. 2009. Open coding. University of Calgary 23 (2009), 2009.","journal-title":"University of Calgary"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_40_2","volume-title":"Proceedings of the 5th International Conference on Learning Representations","author":"Kurakin Alexey","year":"2017","unstructured":"Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2017. Adversarial examples in the physical world. In Proceedings of the 5th International Conference on Learning Representations."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00034"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330578"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/1346281.1346323"},{"key":"e_1_3_2_44_2","volume-title":"Qualitative Research Methods for the Social Sciences","author":"Lune Howard","year":"2017","unstructured":"Howard Lune and Bruce L. Berg. 2017. Qualitative Research Methods for the Social Sciences. Pearson."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_47_2","unstructured":"Lei Ma Fuyuan Zhang Minhui Xue Bo Li Yang Liu Jianjun Zhao and Yadong Wang. 2018. Combinatorial Testing for Deep Learning Systems. CoRR abs\/1806.07723 (2018). arXiv:1806.07723. Retrieved from https:\/\/arxiv.org\/abs\/1806.07723."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00078"},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1109\/ESEM.2013.18","volume-title":"Proceedings of the 2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement","author":"Ocariza Frolin","year":"2013","unstructured":"Frolin Ocariza, Kartik Bajaj, Karthik Pattabiraman, and Ali Mesbah. 2013. An empirical study of client-side JavaScript bugs. In Proceedings of the 2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement. 55\u201364."},{"key":"e_1_3_2_50_2","first-page":"12","volume-title":"Proceedings of the 2017 IEEE International Conference on Software Testing, Verification and Validation","author":"Perez Alexandre","year":"2017","unstructured":"Alexandre Perez, Rui Abreu, and Marcelo D\u2019Amorim. 2017. Prevalence of single-fault fixes and its impact on fault localization. In Proceedings of the 2017 IEEE International Conference on Software Testing, Verification and Validation. 12\u201322."},{"key":"e_1_3_2_51_2","first-page":"1027","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering","author":"Pham Hung Viet","year":"2019","unstructured":"Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. 2019. CRADLE: Cross-backend validation to detect and localize bugs in deep learning libraries. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering. 1027\u20131038."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556953"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468591"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSC53577.2021.00059"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9258-8"},{"key":"e_1_3_2_57_2","first-page":"271","volume-title":"Proceedings of 23rd International Symposium on Software Reliability Engineering","author":"Thung Ferdian","year":"2012","unstructured":"Ferdian Thung, Shaowei Wang, David Lo, and Lingxiao Jiang. 2012. An empirical study of bugs in machine learning systems. In Proceedings of 23rd International Symposium on Software Reliability Engineering. 271\u2013280."},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556949"},{"key":"e_1_3_2_60_2","first-page":"1","volume-title":"Proceedings of International Conference on Fuzzy Systems","author":"Vieira Susana M.","year":"2010","unstructured":"Susana M. Vieira, Uzay Kaymak, and Jo\u00e3o MC Sousa. 2010. Cohen\u2019s kappa coefficient as a performance measure for feature selection. In Proceedings of International Conference on Fuzzy Systems. 1\u20138."},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.153381"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3559561"},{"key":"e_1_3_2_63_2","unstructured":"Jiannan Wang Thibaud Lutellier Shangshu Qian Hung Viet Pham and Lin Tan. 2022. EAGLE: Creating Equivalent Graphs to Test Deep Learning Libraries. In Proceedings of the 44th International Conference on Software Engineering (ICSE\u201922) . Association for Computing Machinery New York NY 798\u2013810."},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387464"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409761"},{"key":"e_1_3_2_66_2","first-page":"397","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering","author":"Wang Zan","year":"2021","unstructured":"Zan Wang, Hanmo You, Junjie Chen, Yingyi Zhang, Xuyuan Dong, and Wenbin Zhang. 2021. Prioritizing test inputs for deep neural networks via mutation analysis. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering. IEEE, 397\u2013409."},{"key":"e_1_3_2_67_2","first-page":"251","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering","author":"Wardat Mohammad","year":"2021","unstructured":"Mohammad Wardat, Wei Le, and Hridesh Rajan. 2021. DeepLocalize: Fault localization for deep neural networks. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering. 251\u2013262."},{"key":"e_1_3_2_68_2","unstructured":"Anjiang Wei Yinlin Deng Chenyuan Yang and Lingming Zhang. 2022. Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh Pennsylvania) (ICSE\u201922) . Association for Computing Machinery New York NY 995\u20131007."},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534220"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-022-3580-5"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468612"},{"key":"e_1_3_2_73_2","first-page":"1448","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering","author":"Yang Lin","year":"2021","unstructured":"Lin Yang, Junjie Chen, Zan Wang, Weijing Wang, Jiajun Jiang, Xuyuan Dong, and Wenbin Zhang. 2021. Semi-supervised log-based anomaly detection via probabilistic label estimation. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering. IEEE, 1448\u20131460."},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00019"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1002\/0470011815.b2a15150"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.988498"},{"key":"e_1_3_2_77_2","doi-asserted-by":"crossref","first-page":"886","DOI":"10.1145\/3180155.3180260","volume-title":"Proceedings of the 40th IEEE\/ACM International Conference on Software Engineering","author":"Zhang Tianyi","year":"2018","unstructured":"Tianyi Zhang, Ganesha Upadhyaya, Anastasia Reinhardt, Hridesh Rajan, and Miryung Kim. 2018. Are code examples on an online Q&A forum reliable?: A study of API misuse on stack overflow. In Proceedings of the 40th IEEE\/ACM International Conference on Software Engineering. 886\u2013896."},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2021.3107165"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464843"},{"key":"e_1_3_2_80_2","first-page":"359","volume-title":"Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering","author":"Zhang Xiaoyu","year":"2021","unstructured":"Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen. 2021. AUTOTRAINER: An automatic DNN training problem detection and repair system. In Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering. 359\u2013371."},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"},{"key":"e_1_3_2_82_2","first-page":"104:1\u2013104:13","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering","author":"Zhang Yingyi","year":"2022","unstructured":"Yingyi Zhang, Zan Wang, Jiajun Jiang, Hanmo You, and Junjie Chen. 2022. Toward improving the robustness of deep learning models via model transformation. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering. ACM, 104:1\u2013104:13."},{"key":"e_1_3_2_83_2","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1007\/978-3-030-71500-7_16","volume-title":"Proceedings of the International Conference on Fundamental Approaches to Software Engineering","author":"Zhong Ziyuan","year":"2021","unstructured":"Ziyuan Zhong, Yuchi Tian, and Baishakhi Ray. 2021. Understanding local robustness of deep neural networks under natural variations. In Proceedings of the International Conference on Fundamental Approaches to Software Engineering. Springer, Cham, 313\u2013337."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587155","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3587155","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:47:15Z","timestamp":1750178835000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587155"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,29]]},"references-count":82,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,11,30]]}},"alternative-id":["10.1145\/3587155"],"URL":"https:\/\/doi.org\/10.1145\/3587155","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,29]]},"assertion":[{"value":"2022-04-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-02-07","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}