{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T05:05:19Z","timestamp":1750309519046,"version":"3.41.0"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2025,4,28]],"date-time":"2025-04-28T00:00:00Z","timestamp":1745798400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JSPS for the KAKENHI","award":["JP20H04167, JP21H04877, JP22K17874, JP22K18630, JP23H03372, JP24K02920, JP24K02921, and JP24KJ1784"],"award-info":[{"award-number":["JP20H04167, JP21H04877, JP22K17874, JP22K18630, JP23H03372, JP24K02920, JP24K02921, and JP24KJ1784"]}]},{"name":"Bilateral Program","award":["JPJSBP120239929"],"award-info":[{"award-number":["JPJSBP120239929"]}]},{"DOI":"10.13039\/501100020959","name":"JST-Mirai Program","doi-asserted-by":"crossref","award":["JPMJMI20B8"],"award-info":[{"award-number":["JPMJMI20B8"]}],"id":[{"id":"10.13039\/501100020959","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,5,31]]},"abstract":"<jats:p>With the increasing prevalence of software incorporating deep neural networks (DNNs), quality assurance for these software systems has become a crucial concern. To this end, various methods have been proposed to repair the misbehavior of DNNs by modifying their weights. However, these repair methods may not meet the developer\u2019s needs for a given dataset and model. In this study, we build prediction models for repair outcomes (i.e., repairs and breaks) to help determine whether the repair method is likely to work. By using our prediction models, developers and operators of DNNs can decide whether or not to apply a repair method, and if so, which method to use. Our prediction models utilize four metrics as explanatory metrics that represent the confidence or ambiguity in the DNN predictions. We experimented with four repair methods and 10 datasets. The experimental results demonstrate that our prediction models successfully select a repair method that meets developers\u2019 needs in 16 out of 24 cases, resulting in an average time saving of 16.29% compared to the naive method. Based on these results, our prediction models can reduce costs for developers and operators when deciding whether to employ repair methods for real-world applications of DNNs.<\/jats:p>","DOI":"10.1145\/3702983","type":"journal-article","created":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T16:29:38Z","timestamp":1730824178000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Repairs and Breaks Prediction for Deep Neural Networks"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-2606-5701","authenticated-orcid":false,"given":"Yuta","family":"Ishimoto","sequence":"first","affiliation":[{"name":"Kyushu University, Fukuoka, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6317-7001","authenticated-orcid":false,"given":"Masanari","family":"Kondo","sequence":"additional","affiliation":[{"name":"Kyushu University, Fukuoka, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8621-2420","authenticated-orcid":false,"given":"Lei","family":"Ma","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Bunkyo-ku, Japan and University of Alberta, Edmonton, Alberta, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1578-7195","authenticated-orcid":false,"given":"Naoyasu","family":"Ubayashi","sequence":"additional","affiliation":[{"name":"Kyushu University, Fukuoka, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7058-1045","authenticated-orcid":false,"given":"Yasutaka","family":"Kamei","sequence":"additional","affiliation":[{"name":"Kyushu University, Fukuoka, Japan"}]}],"member":"320","published-online":{"date-parts":[[2025,4,28]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Taiwan News. 2020. Video Shows Tesla on Autopilot Slam into Truck on Taiwan Highway. Retrieved from https:\/\/www.taiwannews.com.tw\/en\/news\/3943199"},{"key":"e_1_3_2_3_2","unstructured":"NBC News. 2021. 2 Dead in Tesla Crash After Car \u2018No One Was Driving\u2019 Hits Tree Authorities Say. Retrieved from https:\/\/www.nbcnews.com\/news\/us-news\/2-dead-tesla-crash-after-car-no-one-was-driving-n1264470"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/591"},{"key":"e_1_3_2_5_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Balog Matej","year":"2017","unstructured":"Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to write programs. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST57152.2023.00017"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510099"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.49"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.312"},{"key":"e_1_3_2_12_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2107.03374"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.114.3.494"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.13643"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-16722-6_10"},{"key":"e_1_3_2_16_2","first-page":"1437","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Falkner Stefan","year":"2018","unstructured":"Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of the International Conference on Machine Learning, 1437\u20131446."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2020.2972974"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_19_2","first-page":"1289","article-title":"An extensive empirical study of feature selection metrics for text classification","volume":"3","author":"Forman George","year":"2003","unstructured":"George Forman. 2003. An extensive empirical study of feature selection metrics for text classification. Journal of Machine Learning Research 3, Mar (2003), 1289\u20131305.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_20_2","first-page":"1050","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Gal Yarin","year":"2016","unstructured":"Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning, 1050\u20131059."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00171"},{"key":"e_1_3_2_22_2","unstructured":"Ian J. Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv:1412.6572. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1412.6572"},{"key":"e_1_3_2_23_2","unstructured":"Gurobi Optimization LLC. 2024. Gurobi Optimizer Reference Manual. Retrieved from https:\/\/www.gurobi.com"},{"key":"e_1_3_2_24_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-019-00227-x"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","unstructured":"Hans Hofmann. 1994. Statlog (German Credit Data). UCI Machine Learning Repository. DOI: 10.24432\/C5NC77","DOI":"10.24432\/C5NC77"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.107117"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/CAIN58948.2023.00017"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.2514\/1.G003724"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-63387-9_5"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICNN.1995.488968"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST57152.2023.00030"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","unstructured":"Ron Kohavi. 1996. Census Income. UCI Machine Learning Repository. DOI: 10.24432\/C5GP7S","DOI":"10.24432\/C5GP7S"},{"key":"e_1_3_2_34_2","volume-title":"Learning multiple layers of features from tiny images","author":"Krizhevsky Alex","year":"2009","unstructured":"Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. Master\u2019s Thesis, University of Tront (2009)."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583131.3590477"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236082"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.5555\/2002472.2002491"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730491"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","unstructured":"S. Moro P. Rita and P. Cortez. 2012. Bank Marketing. UCI Machine Learning Repository. DOI: 10.24432\/C5K306","DOI":"10.24432\/C5K306"},{"key":"e_1_3_2_41_2","volume-title":"Machine Learning: A Probabilistic Perspective","author":"Murphy Kevin P.","year":"2012","unstructured":"Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. MIT press."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER56733.2023.00090"},{"key":"e_1_3_2_43_2","volume-title":"Feature Extraction and Image Processing for Computer Vision","author":"Nixon Mark","year":"2019","unstructured":"Mark Nixon and Alberto Aguado. 2019. Feature Extraction and Image Processing for Computer Vision. Academic press."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.3115\/1219840.1219855"},{"key":"e_1_3_2_45_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311\u2013318."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3585005"},{"key":"e_1_3_2_47_2","first-page":"1","volume-title":"Proceedings of the Annual meeting of the Florida Association of Institutional Research","author":"Romano J.","year":"2006","unstructured":"J. Romano, J. D. Kromrey, J. Coraggio, and J. Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen\u2019s d for evaluating group differences on the NSSE and other surveys. In Proceedings of the Annual meeting of the Florida Association of Institutional Research, 1\u201333."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPEC58863.2023.10363447"},{"key":"e_1_3_2_49_2","volume-title":"The Mathematical Theory of Communication","author":"Shannon Claude E.","year":"1949","unstructured":"Claude E. Shannon and Warren Weaver. 1949. The Mathematical Theory of Communication. University of Illinois Press."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3563210"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion58688.2023.00033"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2011.6033395"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008202821328"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13735-021-00218-1"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510080"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3591238"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00128"},{"key":"e_1_3_2_58_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00046"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510071"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00034"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534375"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00129"},{"key":"e_1_3_2_64_2","unstructured":"Han Xiao Kashif Rasul and Roland Vollgraf. 2017. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1708.07747"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01407"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2021.3096332"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2983149"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00043"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380331"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380368"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00043"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.485"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3702983","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3702983","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:04Z","timestamp":1750295884000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3702983"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,28]]},"references-count":71,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,5,31]]}},"alternative-id":["10.1145\/3702983"],"URL":"https:\/\/doi.org\/10.1145\/3702983","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"type":"print","value":"1049-331X"},{"type":"electronic","value":"1557-7392"}],"subject":[],"published":{"date-parts":[[2025,4,28]]},"assertion":[{"value":"2024-02-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-10-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}