{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T13:22:08Z","timestamp":1773840128289,"version":"3.50.1"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2023,7,21]],"date-time":"2023-07-21T00:00:00Z","timestamp":1689897600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61932021, 62002125"],"award-info":[{"award-number":["61932021, 62002125"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"National Key Research and Development Program of China","award":["2019YFE0198100"],"award-info":[{"award-number":["2019YFE0198100"]}]},{"name":"Hong Kong RGC\/GRF","award":["16205722"],"award-info":[{"award-number":["16205722"]}]},{"name":"Hong Kong ITF","award":["MHP\/055\/19"],"award-info":[{"award-number":["MHP\/055\/19"]}]},{"name":"MSRA Collaborative Research Grant"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,9,30]]},"abstract":"<jats:p>\n            Recent\n            <jats:bold>deep learning (DL)<\/jats:bold>\n            applications are mostly built on top of DL libraries. The quality assurance of these libraries is critical to the dependable deployment of DL applications. Techniques have been proposed to generate various DL models and apply them to test these libraries. However, their test effectiveness is constrained by the diversity of layer API calls in their generated DL models. Our study reveals that these techniques can cover at most 34.1% layer inputs, 25.9% layer parameter values, and 15.6% layer sequences. As a result, we find that many bugs arising from specific layer API calls (i.e., specific layer inputs, parameter values, or layer sequences) can be missed by existing techniques.\n          <\/jats:p>\n          <jats:p>Because of this limitation, we propose COMET to effectively generate DL models with diverse layer API calls for DL library testing. COMET: (1) designs a set of mutation operators and a coverage-based search algorithm to diversify layer inputs, layer parameter values, and layer sequences in DL models. (2) proposes a model synthesis method to boost the test efficiency without compromising the layer API call diversity. Our evaluation result shows that COMET outperforms baselines by covering twice as many layer inputs (69.7% vs. 34.1%), layer parameter values (50.2% vs. 25.9%), and layer sequences (39.0% vs. 15.6%) as those by the state-of-the-art. Moreover, COMET covers 3.4% more library branches than those by existing techniques. Finally, COMET detects 32 new bugs in the latest version of eight popular DL libraries, including TensorFlow and MXNet, with 21 of them confirmed by DL library developers and seven of those confirmed bugs have been fixed by developers.<\/jats:p>","DOI":"10.1145\/3583566","type":"journal-article","created":{"date-parts":[[2023,2,8]],"date-time":"2023-02-08T13:28:49Z","timestamp":1675862929000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["COMET: Coverage-guided Model Generation For Deep Learning Library Testing"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5947-4030","authenticated-orcid":false,"given":"Meiziniu","family":"Li","sequence":"first","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4892-6294","authenticated-orcid":false,"given":"Jialun","family":"Cao","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, China and Guangzhou HKUST Fok Ying Tung Research Institute, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1644-2965","authenticated-orcid":false,"given":"Yongqiang","family":"Tian","sequence":"additional","affiliation":[{"name":"University of Waterloo, Canada and The Hong Kong University of Science and Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8031-4947","authenticated-orcid":false,"given":"Tsz On","family":"Li","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5588-9618","authenticated-orcid":false,"given":"Ming","family":"Wen","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3508-7172","authenticated-orcid":false,"given":"Shing-Chi","family":"Cheung","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, China and Guangzhou HKUST Fok Ying Tung Research Institute, China"}]}],"member":"320","published-online":{"date-parts":[[2023,7,21]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"265","volume-title":"Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI\u201916)","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI\u201916). USENIX Association, USA, 265\u2013283."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.5555\/1355340"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2020.103795"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scs.2020.102589"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.312"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2203.04026"},{"key":"e_1_3_2_8_2","unstructured":"Tianqi Chen Mu Li Yutian Li Min Lin Naiyan Wang Minjie Wang Tianjun Xiao Bing Xu Chiyuan Zhang and Zheng Zhang. 2015. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. (2015). arxiv:cs.DC\/1512.01274."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2980983.2908095"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/2908080.2908095"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786835"},{"key":"e_1_3_2_12_2","unstructured":"Fran\u00e7ois Chollet et\u00a0al. 2015. Keras. (2015)."},{"key":"e_1_3_2_13_2","unstructured":"Operator Fusion For Conv2D and Mul Operators. Accessed: 2022. https:\/\/github.com\/microsoft\/onnxruntime\/blob\/main\/onnxruntime\/core\/optimizer\/conv_mul_fusion.cc."},{"key":"e_1_3_2_14_2","unstructured":"Bug Report For Conv3DTranspose. Accessed: 2022. https:\/\/github.com\/keras-team\/keras\/issues\/16933."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICC.2018.8422832"},{"key":"e_1_3_2_16_2","unstructured":"Bug Report For DepthwiseConv2D. Accessed: 2022. https:\/\/github.com\/keras-team\/keras\/issues\/16314."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCOMM.2018.2878025"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510092"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416571"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46478-7_39"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464825"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME52107.2021.00011"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1007\/978-3-030-59410-7_40","volume-title":"Database Systems for Advanced Applications","author":"Jia Li","year":"2020","unstructured":"Li Jia, Hao Zhong, Xiaoyin Wang, Linpeng Huang, and Xuansheng Lu. 2020. An empirical study on bugs inside TensorFlow. In Database Systems for Advanced Applications, Yunmook Nah, Bin Cui, Sang-Won Lee, Jeffrey Xu Yu, Yang-Sae Moon, and Steven Euijong Whang (Eds.). Springer International Publishing, Cham, 604\u2013620."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.110935"},{"key":"e_1_3_2_26_2","unstructured":"How Kernel Size Influences Padding In Keras. Accessed: 2022. https:\/\/github.com\/keras-team\/keras\/blob\/db667716ac6be59a8495b58e0d2e246e9c031599\/keras\/layers\/convolutional\/base_conv.py#L389."},{"key":"e_1_3_2_27_2","unstructured":"Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. (2009) 32\u201333 pages. https:\/\/www.cs.toronto.edu\/kriz\/learning-features-2009-TR.pdf."},{"key":"e_1_3_2_28_2","first-page":"1097","article-title":"ImageNet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 (2012), 1097\u20131105.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2004.24"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00037"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_34_2","article-title":"keras-apache-mxnet","author":"MXNet Keras Apache","year":"2022","unstructured":"Keras Apache MXNet. Accessed: 2022. keras-apache-mxnet. https:\/\/github.com\/awslabs\/keras-apache-mxnet.","journal-title":"https:\/\/github.com\/awslabs\/keras-apache-mxnet."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00078"},{"key":"e_1_3_2_36_2","first-page":"4901","volume-title":"Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research)","volume":"97","author":"Odena Augustus","year":"2019","unstructured":"Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. 2019. TensorFuzz: Debugging neural networks with coverage-guided fuzzing. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.), Vol. 97. PMLR, Long Beach, California, 4901\u20134911. https:\/\/proceedings.mlr.press\/v97\/odena19a.html."},{"key":"e_1_3_2_37_2","article-title":"Open standard for machine learning interoperability","year":"2022","unstructured":"Onnx. Accessed: 2022. Open standard for machine learning interoperability. https:\/\/github.com\/onnx\/onnx.","journal-title":"https:\/\/github.com\/onnx\/onnx."},{"key":"e_1_3_2_38_2","article-title":"tf2onnx - Convert TensorFlow, Keras, Tensorflow.js and Tflite models to ONNX","author":"ONNX TensorFlow","year":"2022","unstructured":"TensorFlow ONNX. Accessed: 2022. tf2onnx - Convert TensorFlow, Keras, Tensorflow.js and Tflite models to ONNX. https:\/\/github.com\/onnx\/tensorflow-onnx.","journal-title":"https:\/\/github.com\/onnx\/tensorflow-onnx."},{"key":"e_1_3_2_39_2","article-title":"ONNX to PyTorch","year":"2022","unstructured":"Onnx2pytorch. Accessed: 2022. ONNX to PyTorch. https:\/\/github.com\/ToriML\/onnx2pytorch.","journal-title":"https:\/\/github.com\/ToriML\/onnx2pytorch."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/1297846.1297902"},{"key":"e_1_3_2_41_2","unstructured":"Bug Report For Inconsistent Padding. Accessed: 2022. https:\/\/github.com\/ToriML\/onnx2pytorch\/pull\/41."},{"key":"e_1_3_2_42_2","volume-title":"PyTorch: An Imperative Style, High-Performance Deep Learning Library","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K\u00f6pf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Curran Associates Inc., Red Hook, NY, USA."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3361566"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_3_2_45_2","article-title":"Optimize and Accelerate Machine Learning Inferencing and Training","author":"Runtime ONNX","year":"2022","unstructured":"ONNX Runtime. Accessed: 2022. Optimize and Accelerate Machine Learning Inferencing and Training. https:\/\/onnxruntime.ai\/.","journal-title":"https:\/\/onnxruntime.ai\/."},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","unstructured":"Ahmad El Sallab Mohammed Abdou E. Perot and S. Yogamani. 2017. Deep Reinforcement Learning framework for Autonomous Driving. (2017).","DOI":"10.2352\/ISSN.2470-1173.2017.19.AVM-023"},{"key":"e_1_3_2_47_2","unstructured":"S. Shalev-Shwartz Shaked Shammah and A. Shashua. 2016. Safe Multi-Agent Reinforcement Learning for Autonomous Driving. (2016)."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468591"},{"key":"e_1_3_2_49_2","unstructured":"TensorFlow Padding Strategy. Accessed: 2022. https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/nn."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2112.13314"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950355"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00138"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409761"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00034"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510041"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534220"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.107004"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464843"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409720"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583566","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3583566","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:54Z","timestamp":1750178274000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583566"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,21]]},"references-count":61,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,9,30]]}},"alternative-id":["10.1145\/3583566"],"URL":"https:\/\/doi.org\/10.1145\/3583566","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,21]]},"assertion":[{"value":"2022-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-10","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}