{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T13:22:33Z","timestamp":1773840153616,"version":"3.50.1"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,12,23]],"date-time":"2023-12-23T00:00:00Z","timestamp":1703289600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61932012, 62141215, 62372228"],"award-info":[{"award-number":["61932012, 62141215, 62372228"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"CCF-Huawei Populus Grove Fund","award":["CCF-HuaweiSY202306"],"award-info":[{"award-number":["CCF-HuaweiSY202306"]}]},{"name":"Science, Technology and Innovation Commission of Shenzhen Municipality","award":["CJGJZD20200617103001003"],"award-info":[{"award-number":["CJGJZD20200617103001003"]}]},{"name":"Canada CIFAR AI Chairs Program, the Natural Sciences and Engineering Research Council of Canada","award":["RGPIN-2021-02549, RGPAS-2021-00034, DGECR-2021-00019"],"award-info":[{"award-number":["RGPIN-2021-02549, RGPAS-2021-00034, DGECR-2021-00019"]}]},{"DOI":"10.13039\/501100020959","name":"JST-Mirai Program","doi-asserted-by":"crossref","award":["JPMJMI20B8"],"award-info":[{"award-number":["JPMJMI20B8"]}],"id":[{"id":"10.13039\/501100020959","id-type":"DOI","asserted-by":"crossref"}]},{"name":"JSPS KAKENHI","award":["JP21H04877, JP23H03372"],"award-info":[{"award-number":["JP21H04877, JP23H03372"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>\n            Deep learning (DL) libraries have become the key component in developing and deploying DL-based software nowadays. With the growing popularity of applying DL models in both academia and industry across various domains, any bugs inherent in the DL libraries can potentially cause unexpected server outcomes. As such, there is an urgent demand for improving the software quality of DL libraries. Although there are some existing approaches specifically designed for testing DL libraries, their focus is usually limited to one specific domain, such as computer vision (CV). It is still not very clear how the existing approaches perform in detecting bugs of different DL libraries regarding different task domains and to what extent. To bridge this gap, we first conduct an empirical study on four representative and state-of-the-art DL library testing approaches. Our empirical study results reveal that it is hard for existing approaches to generalize to other task domains. We also find that the test inputs generated by these approaches usually lack diversity, with only a few types of bugs. What is worse, the false-positive rate of existing approaches is also high (\n            <jats:bold>up to 58%<\/jats:bold>\n            ). To address these issues, we propose a\n            <jats:italic>guided differential fuzzing approach based on generation<\/jats:italic>\n            , namely,\n            <jats:italic>Gandalf<\/jats:italic>\n            . To generate testing inputs across diverse task domains effectively,\n            <jats:italic>Gandalf<\/jats:italic>\n            adopts the context-free grammar to ensure validity and utilizes a\n            <jats:italic>Deep Q-Network<\/jats:italic>\n            to maximize the diversity.\n            <jats:italic>Gandalf<\/jats:italic>\n            also includes 15 metamorphic relations to make it possible for the generated test cases to generalize across different DL libraries. Such a design can decrease the false positives because of the semantic difference for different APIs. We evaluate the effectiveness of\n            <jats:italic>Gandalf<\/jats:italic>\n            on nine versions of three representative DL libraries, covering 309 operators from computer vision, natural language processing, and automated speech recognition. The evaluation results demonstrate that\n            <jats:italic>Gandalf<\/jats:italic>\n            can effectively and efficiently generate diverse test inputs. Meanwhile,\n            <jats:italic>Gandalf<\/jats:italic>\n            successfully detects five categories of bugs with\n            <jats:bold>only 3.1%<\/jats:bold>\n            false-positive rates. We report all 49 new unique bugs found during the evaluation to the DL libraries\u2019 developers, and most of these bugs have been confirmed. Details about our empirical study and evaluation results are available on our project website.\n            <jats:xref ref-type=\"fn\">\n              <jats:sup>1<\/jats:sup>\n            <\/jats:xref>\n          <\/jats:p>","DOI":"10.1145\/3628159","type":"journal-article","created":{"date-parts":[[2023,10,18]],"date-time":"2023-10-18T21:37:52Z","timestamp":1697665072000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Generation-based Differential Fuzzing for Deep Learning Libraries"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4930-9637","authenticated-orcid":false,"given":"Jiawei","family":"Liu","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3666-4020","authenticated-orcid":false,"given":"Yuheng","family":"Huang","sequence":"additional","affiliation":[{"name":"University of Alberta, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4559-5426","authenticated-orcid":false,"given":"Zhijie","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Alberta, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8621-2420","authenticated-orcid":false,"given":"Lei","family":"Ma","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Japan and University of Alberta, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9930-7111","authenticated-orcid":false,"given":"Chunrong","family":"Fang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5141-8923","authenticated-orcid":false,"given":"Mingzheng","family":"Gu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7284-1931","authenticated-orcid":false,"given":"Xufan","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9592-7022","authenticated-orcid":false,"given":"Zhenyu","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]}],"member":"320","published-online":{"date-parts":[[2023,12,23]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Reddit. 2021. Using PyTorch + NumPy? A bug that plagues thousands of open-source projects. Retrieved from https:\/\/www.reddit.com\/r\/MachineLearning\/comments\/mocpgj\/p_using_pytorch_numpy_a_bug_that_plagues\/"},{"key":"e_1_3_2_3_2","first-page":"265","volume-title":"12th USENIX Symposium on Operating Systems Design and Implementation","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, Kimberly Keeton and Timothy Roscoe (Eds.). USENIX Association, 265\u2013283. Retrieved from https:\/\/www.usenix.org\/conference\/osdi16\/technical-sessions\/presentation\/abadi"},{"key":"e_1_3_2_4_2","unstructured":"The Theano Development Team. 2016. Theano: A Python Framework for Fast Computation of Mathematical Expressions . CoRR abs\/1605.02688 (2016)."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3062341.3062349"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338970"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/SPW.2018.00026"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.413"},{"key":"e_1_3_2_9_2","article-title":"Toward understanding deep learning framework bugs","volume":"2203","author":"Chen Junjie","year":"2022","unstructured":"Junjie Chen, Yihua Liang, Qingchao Shen, and Jiajun Jiang. 2022. Toward understanding deep learning framework bugs. CoRR abs\/2203.04026 (2022).","journal-title":"CoRR"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3363562"},{"key":"e_1_3_2_11_2","article-title":"MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems","volume":"1512","author":"Chen Tianqi","year":"2015","unstructured":"Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. CoRR abs\/1512.01274 (2015).","journal-title":"CoRR"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2022.102444"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194733.3194744"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642963"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3133917"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.5555\/3322706.3361996"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2016.33"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409679"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2976002.2976017"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510092"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00016"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416571"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2430536.2430540"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102602"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-020-3097-4"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME52107.2021.00011"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-59410-7_40"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654889"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-35413-2_16"},{"key":"e_1_3_2_33_2","unstructured":"Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images . Technical Report."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2814270.2814319"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i11.21527"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6360"},{"key":"e_1_3_2_38_2","first-page":"142","volume-title":"49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","author":"Maas Andrew L.","year":"2011","unstructured":"Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (Eds.). The Association for Computer Linguistics, 142\u2013150. Retrieved from https:\/\/aclanthology.org\/P11-1015\/"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2946563"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3314221.3314651"},{"issue":"1","key":"e_1_3_2_41_2","first-page":"100","article-title":"Differential testing for software","volume":"10","author":"McKeeman William M.","year":"1998","unstructured":"William M. McKeeman. 1998. Differential testing for software. Digit. Tech. J. 10, 1 (1998), 100\u2013107. Retrieved from http:\/\/www.hpl.hp.com\/hpjournal\/dtj\/vol10num1\/vol10num1art9.pdf","journal-title":"Digit. Tech. J."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.28"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/96267.96279"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/APCCAS.2016.7804063"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3418930"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.2979670"},{"key":"e_1_3_2_48_2","first-page":"8024","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K\u00f6pf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Annual Conference on Neural Information Processing Systems, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d\u2019Alch\u00e9-Buc, Emily B. Fox, and Roman Garnett (Eds.). 8024\u20138035. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2019\/hash\/bdbca288fee7f92f2bfa9f7012727740-Abstract.html"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2945397"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3560429"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884879"},{"key":"e_1_3_2_54_2","volume-title":"Fuzzing: Brute Force Vulnerability Discovery","author":"Sutton Michael","year":"2007","unstructured":"Michael Sutton, Adam Greene, and Pedram Amini. 2007. Fuzzing: Brute Force Vulnerability Discovery. Pearson Education."},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2010.39"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_57_2","first-page":"2741","volume-title":"30th USENIX Security Symposium","author":"Wang Daimeng","year":"2021","unstructured":"Daimeng Wang, Zheng Zhang, Hang Zhang, Zhiyun Qian, Srikanth V. Krishnamurthy, and Nael B. Abu-Ghazaleh. 2021. SyzVegas: Beating kernel fuzzing odds with reinforcement learning. In 30th USENIX Security Symposium, Michael Bailey and Rachel Greenstadt (Eds.). USENIX Association, 2741\u20132758. Retrieved from https:\/\/www.usenix.org\/conference\/usenixsecurity21\/presentation\/wang-daimeng"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510165"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409761"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510041"},{"key":"e_1_3_2_61_2","article-title":"Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms","volume":"1708","author":"Xiao Han","year":"2017","unstructured":"Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. CoRR abs\/1708.07747 (2017).","journal-title":"CoRR"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2021.3107165"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464843"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2876433"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3628159","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3628159","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:49:01Z","timestamp":1750286941000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3628159"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,23]]},"references-count":64,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3628159"],"URL":"https:\/\/doi.org\/10.1145\/3628159","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,23]]},"assertion":[{"value":"2022-12-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-20","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}