{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T15:00:15Z","timestamp":1784300415385,"version":"3.55.0"},"reference-count":102,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,2,24]],"date-time":"2022-02-24T00:00:00Z","timestamp":1645660800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"RGC ECS grant","award":["26206520"],"award-info":[{"award-number":["26206520"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Meas. Anal. Comput. Syst."],"published-print":{"date-parts":[[2022,2,24]]},"abstract":"<jats:p>The prosperous trend of deploying deep neural network (DNN) models to diverse hardware platforms has boosted the development of deep learning (DL) compilers. DL compilers take the high-level DNN model specifications as input and generate optimized DNN executables for diverse hardware architectures like CPUs, GPUs, and various hardware accelerators. Compiling DNN models into high-efficiency executables is not easy: the compilation procedure often involves converting high-level model specifications into several different intermediate representations (IR), e.g., graph IR and operator IR, and performing rule-based or learning-based optimizations from both platform-independent and platform-dependent perspectives. Despite the prosperous adoption of DL compilers in real-world scenarios, principled and systematic understanding toward the correctness of DL compilers does not yet exist. To fill this critical gap, this paper introduces MT-DLComp, a metamorphic testing framework specifically designed for DL compilers to effectively uncover erroneous compilations. Our approach leverages deliberately-designed metamorphic relations (MRs) to launch semantics-preserving mutations toward DNN models to generate their variants. This way, DL compilers can be automatically examined for compilation correctness utilizing DNN models and their variants without requiring manual intervention. We also develop a set of practical techniques to realize an effective workflow and localize identified error-revealing inputs. Real-world DL compilers exhibit a high level of engineering quality. Nevertheless, we detected over 435 inputs that can result in erroneous compilations in four popular DL compilers, all of which are industry-strength products maintained by Amazon, Facebook, Microsoft, and Google. While the discovered error-triggering inputs do not cause the DL compilers to crash directly, they can lead to the generation of incorrect DNN executables. With substantial manual effort and help from the DL compiler developers, we uncovered four bugs in these DL compilers by debugging them using the error-triggering inputs. Our proposed testing frameworks and findings can be used to guide developers in their efforts to improve DL compilers.<\/jats:p>","DOI":"10.1145\/3508035","type":"journal-article","created":{"date-parts":[[2022,2,28]],"date-time":"2022-02-28T23:44:29Z","timestamp":1646091869000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":50,"title":["Metamorphic Testing of Deep Learning Compilers"],"prefix":"10.1145","volume":"6","author":[{"given":"Dongwei","family":"Xiao","sequence":"first","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhibo","family":"LIU","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuanyuan","family":"Yuan","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qi","family":"Pang","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuai","family":"Wang","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,2,28]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"et almbox","author":"Abadi Mart'in","year":"2016","unstructured":"Mart'in Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , et almbox . 2016 . Tensorflow : A system for large-scale machine learning. In 12th $$USENIX$$ symposium on operating systems design and implementation ($$OSDI$$ 16) . 265--283. Mart'in Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et almbox. 2016. Tensorflow: A system for large-scale machine learning. In 12th $$USENIX$$ symposium on operating systems design and implementation ($$OSDI$$ 16) . 265--283."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3322967"},{"key":"e_1_2_1_3_1","unstructured":"Amazon. 2021. Amazon SageMaker Neo uses Apache TVM for performance improvement on hardware target . https:\/\/aws.amazon.com\/sagemaker\/neo\/. Amazon. 2021. Amazon SageMaker Neo uses Apache TVM for performance improvement on hardware target . https:\/\/aws.amazon.com\/sagemaker\/neo\/."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/3314872.3314896"},{"key":"e_1_2_1_5_1","unstructured":"Jialun Cao Meiziniu Li Yeting Li Ming Wen and Shing-Chi Cheung. 2020. SemMT: A Semantic-based Testing Approach for Machine Translation Systems. arXiv preprint arXiv:2012.01815 (2020). Jialun Cao Meiziniu Li Yeting Li Ming Wen and Shing-Chi Cheung. 2020. SemMT: A Semantic-based Testing Approach for Machine Translation Systems. arXiv preprint arXiv:2012.01815 (2020)."},{"key":"e_1_2_1_6_1","volume-title":"et almbox","author":"Chen Tianqi","year":"2018","unstructured":"Tianqi Chen , Thierry Moreau , Ziheng Jiang , Lianmin Zheng , Eddie Yan , Haichen Shen , Meghan Cowan , Leyuan Wang , Yuwei Hu , Luis Ceze , et almbox . 2018 a. $$TVM$$: An automated end-to-end optimizing compiler for deep learning. In 13th $$USENIX$$ Symposium on Operating Systems Design and Implementation ( $$OSDI$$ 18) . 578--594. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et almbox. 2018a. $$TVM$$: An automated end-to-end optimizing compiler for deep learning. In 13th $$USENIX$$ Symposium on Operating Systems Design and Implementation ($$OSDI$$ 18) . 578--594."},{"key":"e_1_2_1_7_1","first-page":"3389","article-title":"Learning to optimize tensor programs","volume":"31","author":"Chen Tianqi","year":"2018","unstructured":"Tianqi Chen , Lianmin Zheng , Eddie Yan , Ziheng Jiang , Thierry Moreau , Luis Ceze , Carlos Guestrin , and Arvind Krishnamurthy . 2018 b. Learning to optimize tensor programs . Advances in Neural Information Processing Systems , Vol. 31 (2018), 3389 -- 3400 . Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018b. Learning to optimize tensor programs. Advances in Neural Information Processing Systems , Vol. 31 (2018), 3389--3400.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30502-6_23"},{"key":"e_1_2_1_10_1","unstructured":"choi. 2020. TVM Performance Degradation . https:\/\/discuss.tvm.apache.org\/t\/performance-has-been-too-slow-since-the-tvm-update\/5865\/7 . choi. 2020. TVM Performance Degradation . https:\/\/discuss.tvm.apache.org\/t\/performance-has-been-too-slow-since-the-tvm-update\/5865\/7 ."},{"key":"e_1_2_1_11_1","unstructured":"Al Danial. [n. d.]. CLOC . https:\/\/goo.gl\/3KFACB . Al Danial. [n. d.]. CLOC . https:\/\/goo.gl\/3KFACB ."},{"key":"e_1_2_1_12_1","volume-title":"Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv preprint arXiv:1812.05339","author":"Du Xiaoning","year":"2018","unstructured":"Xiaoning Du , Xiaofei Xie , Yi Li , Lei Ma , Jianjun Zhao , and Yang Liu . 2018 . Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv preprint arXiv:1812.05339 (2018). Xiaoning Du, Xiaofei Xie, Yi Li, Lei Ma, Jianjun Zhao, and Yang Liu. 2018. Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv preprint arXiv:1812.05339 (2018)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236057"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338972"},{"key":"e_1_2_1_15_1","volume-title":"Neville Dubash, and Sanjay Podder.","author":"Dwarakanath Anurag","year":"2018","unstructured":"Anurag Dwarakanath , Manish Ahuja , Samarth Sikand , Raghotham M. Rao , R. P. Jagadeesh Chandra Bose , Neville Dubash, and Sanjay Podder. 2018 . Identifying Implementation Bugs in Machine Learning Based Image Classifiers Using Metamorphic Testing. In ISSTA . Anurag Dwarakanath, Manish Ahuja, Samarth Sikand, Raghotham M. Rao, R. P. Jagadeesh Chandra Bose, Neville Dubash, and Sanjay Podder. 2018. Identifying Implementation Bugs in Machine Learning Based Image Classifiers Using Metamorphic Testing. In ISSTA ."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.1972.5009071"},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Sainyam Galhotra Yuriy Brun and Alexandra Meliou. 2017. Fairness testing: testing software for discrimination. In ACM ESEC\/FSE. ACM 498--510. Sainyam Galhotra Yuriy Brun and Alexandra Meliou. 2017. Fairness testing: testing software for discrimination. In ACM ESEC\/FSE. ACM 498--510.","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_2_1_19_1","volume-title":"2019 b. Rnn-test: Adversarial testing framework for recurrent neural network systems. arXiv preprint arXiv:1911.06155","author":"Guo Jianmin","year":"2019","unstructured":"Jianmin Guo , Yue Zhao , Xueying Han , Yu Jiang , and Jiaguang Sun . 2019 b. Rnn-test: Adversarial testing framework for recurrent neural network systems. arXiv preprint arXiv:1911.06155 ( 2019 ). Jianmin Guo, Yue Zhao, Xueying Han, Yu Jiang, and Jiaguang Sun. 2019 b. Rnn-test: Adversarial testing framework for recurrent neural network systems. arXiv preprint arXiv:1911.06155 (2019)."},{"key":"e_1_2_1_20_1","volume-title":"Angel-eye: A complete design flow for mapping cnn onto embedded fpga","author":"Guo Kaiyuan","year":"2017","unstructured":"Kaiyuan Guo , Lingzhi Sui , Jiantao Qiu , Jincheng Yu , Junbin Wang , Song Yao , Song Han , Yu Wang , and Huazhong Yang . 2017 . Angel-eye: A complete design flow for mapping cnn onto embedded fpga . IEEE transactions on computer-aided design of integrated circuits and systems , Vol. 37 , 1 (2017), 35--47. Kaiyuan Guo, Lingzhi Sui, Jiantao Qiu, Jincheng Yu, Junbin Wang, Song Yao, Song Han, Yu Wang, and Huazhong Yang. 2017. Angel-eye: A complete design flow for mapping cnn onto embedded fpga. IEEE transactions on computer-aided design of integrated circuits and systems , Vol. 37, 1 (2017), 35--47."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00080"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416571"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 863--875","author":"Gupta Shashij","year":"2020","unstructured":"Shashij Gupta , Pinjia He , Clara Meister , and Zhendong Su . 2020 . Machine translation testing via pathological invariance . In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 863--875 . Shashij Gupta, Pinjia He, Clara Meister, and Zhendong Su. 2020. Machine translation testing via pathological invariance. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 863--875."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_25_1","unstructured":"Pinjia He Clara Meister and Zhendong Su. 2020. Structure-invariant testing for machine translation. In ICSE . Pinjia He Clara Meister and Zhendong Su. 2020. Structure-invariant testing for machine translation. In ICSE ."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00047"},{"key":"e_1_2_1_27_1","volume-title":"The promise and peril of human evaluation for model interpretability. arXiv preprint arXiv:1711.07414","author":"Herman Bernease","year":"2017","unstructured":"Bernease Herman . 2017. The promise and peril of human evaluation for model interpretability. arXiv preprint arXiv:1711.07414 ( 2017 ), 8. Bernease Herman. 2017. The promise and peril of human evaluation for model interpretability. arXiv preprint arXiv:1711.07414 (2017), 8."},{"key":"e_1_2_1_28_1","unstructured":"SA Hex-Rays. 2014. IDA Pro: a cross-platform multi-processor disassembler and debugger . SA Hex-Rays. 2014. IDA Pro: a cross-platform multi-processor disassembler and debugger ."},{"key":"e_1_2_1_29_1","volume-title":"Long short-term memory. Neural computation","author":"Hochreiter Sepp","year":"1997","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber . 1997. Long short-term memory. Neural computation , Vol. 9 , 8 ( 1997 ), 1735--1780. Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural computation , Vol. 9, 8 (1997), 1735--1780."},{"key":"e_1_2_1_30_1","volume-title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861","author":"Howard Andrew G","year":"2017","unstructured":"Andrew G Howard , Menglong Zhu , Bo Chen , Dmitry Kalenichenko , Weijun Wang , Tobias Weyand , Marco Andreetto , and Hartwig Adam . 2017 . Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017). Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)."},{"key":"e_1_2_1_31_1","volume-title":"testrnn: Coverage-guided testing on recurrent neural networks. arXiv preprint arXiv:1906.08557","author":"Huang Wei","year":"2019","unstructured":"Wei Huang , Youcheng Sun , Xiaowei Huang , and James Sharp . 2019. testrnn: Coverage-guided testing on recurrent neural networks. arXiv preprint arXiv:1906.08557 ( 2019 ). Wei Huang, Youcheng Sun, Xiaowei Huang, and James Sharp. 2019. testrnn: Coverage-guided testing on recurrent neural networks. arXiv preprint arXiv:1906.08557 (2019)."},{"key":"e_1_2_1_32_1","unstructured":"Texas Instruments. 2021. The AM335x microprocessors support TVM . https:\/\/software-dl.ti.com\/processor-sdk-linux\/esd\/docs\/latest\/linux\/Foundational_Components\/Machine_Learning\/tvm.html . Texas Instruments. 2021. The AM335x microprocessors support TVM . https:\/\/software-dl.ti.com\/processor-sdk-linux\/esd\/docs\/latest\/linux\/Foundational_Components\/Machine_Learning\/tvm.html ."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_2_1_34_1","volume-title":"et almbox","author":"Krizhevsky Alex","year":"2009","unstructured":"Alex Krizhevsky , Geoffrey Hinton , et almbox . 2009 . Learning multiple layers of features from tiny images. (2009). Alex Krizhevsky, Geoffrey Hinton, et almbox. 2009. Learning multiple layers of features from tiny images. (2009)."},{"key":"e_1_2_1_35_1","volume-title":"Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , Vol. 25 ( 2012 ), 1097--1105. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , Vol. 25 (2012), 1097--1105."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/977395.977673"},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Vu Le Mehrdad Afshari and Zhendong Su. 2014. Compiler Validation via Equivalence Modulo Inputs. In PLDI . Vu Le Mehrdad Afshari and Zhendong Su. 2014. Compiler Validation via Equivalence Modulo Inputs. In PLDI .","DOI":"10.1145\/2594291.2594334"},{"key":"e_1_2_1_38_1","doi-asserted-by":"crossref","unstructured":"Vu Le Chengnian Sun and Zhendong Su. 2015. Finding Deep Compiler Bugs via Guided Stochastic Program Mutation. In OOPSLA . Vu Le Chengnian Sun and Zhendong Su. 2015. Finding Deep Compiler Bugs via Guided Stochastic Program Mutation. In OOPSLA .","DOI":"10.1145\/2814270.2814319"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2020.3030548"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397370"},{"key":"e_1_2_1_41_1","volume-title":"14th $$USENIX$$ Symposium on Operating Systems Design and Implementation ($$OSDI$$ 20) . 881--897.","author":"Ma Lingxiao","unstructured":"Lingxiao Ma , Zhiqiang Xie , Zhi Yang , Jilong Xue , Youshan Miao , Wei Cui , Wenxiang Hu , Fan Yang , Lintao Zhang , and Lidong Zhou . 2020 b. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks . In 14th $$USENIX$$ Symposium on Operating Systems Design and Implementation ($$OSDI$$ 20) . 881--897. Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020 b. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks. In 14th $$USENIX$$ Symposium on Operating Systems Design and Implementation ($$OSDI$$ 20) . 881--897."},{"key":"e_1_2_1_42_1","unstructured":"Pingchuan Ma and Shuai Wang. 2022. MT-Teql: Evaluating and Augmenting Neural NLIDB on Real-world Linguistic and Schema Variations. In PVLDB . Pingchuan Ma and Shuai Wang. 2022. MT-Teql: Evaluating and Augmenting Neural NLIDB on Real-world Linguistic and Schema Variations. In PVLDB ."},{"key":"e_1_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Pingchuan Ma Shuai Wang and Jin Liu. 2020 a. Metamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models. In IJCAI . 458--465. Pingchuan Ma Shuai Wang and Jin Liu. 2020 a. Metamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models. In IJCAI . 458--465.","DOI":"10.24963\/ijcai.2020\/64"},{"key":"e_1_2_1_44_1","unstructured":"Microsoft. 2020. onnxruntime . https:\/\/github.com\/microsoft\/onnxruntime . Microsoft. 2020. onnxruntime . https:\/\/github.com\/microsoft\/onnxruntime ."},{"key":"e_1_2_1_45_1","unstructured":"Microsoft. 2021. Microsoft Linear Algebra Subprograms . https:\/\/github.com\/microsoft\/onnxruntime\/tree\/master\/onnxruntime\/core\/mlas . Microsoft. 2021. Microsoft Linear Algebra Subprograms . https:\/\/github.com\/microsoft\/onnxruntime\/tree\/master\/onnxruntime\/core\/mlas ."},{"key":"e_1_2_1_46_1","unstructured":"Timothy Prickett Morgan. 2020. INSIDE FACEBOOK'S FUTURE RACK AND MICROSERVER IRON . https:\/\/www.nextplatform.com\/2020\/05\/14\/inside-facebooks-future-rack-and-microserver-iron\/. Timothy Prickett Morgan. 2020. INSIDE FACEBOOK'S FUTURE RACK AND MICROSERVER IRON . https:\/\/www.nextplatform.com\/2020\/05\/14\/inside-facebooks-future-rack-and-microserver-iron\/."},{"key":"e_1_2_1_47_1","unstructured":"MT-DLComp. 2021. MT-DLComp . https:\/\/github.com\/Wilbur-Django\/Testing-DNN-Compilers . MT-DLComp. 2021. MT-DLComp . https:\/\/github.com\/Wilbur-Django\/Testing-DNN-Compilers ."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925952"},{"key":"e_1_2_1_49_1","volume-title":"Akbar Siami Namin, and Craig Snoeyink","author":"Nair Varsha","year":"2020","unstructured":"Varsha Nair , Moitrayee Chatterjee , Neda Tavakoli , Akbar Siami Namin, and Craig Snoeyink . 2020 . Fast Fourier Transformation for Optimizing Convolutional Neural Networks in Object Recognition . arxiv: 2010.04257 [cs.CV] Varsha Nair, Moitrayee Chatterjee, Neda Tavakoli, Akbar Siami Namin, and Craig Snoeyink. 2020. Fast Fourier Transformation for Optimizing Convolutional Neural Networks in Object Recognition. arxiv: 2010.04257 [cs.CV]"},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","unstructured":"Shin Nakajima and Tsong Yueh Chen. 2019. Generating biased dataset for metamorphic testing of machine learning programs. In IFIP-ICTSS . Shin Nakajima and Tsong Yueh Chen. 2019. Generating biased dataset for metamorphic testing of machine learning programs. In IFIP-ICTSS .","DOI":"10.1007\/978-3-030-31280-0_4"},{"key":"e_1_2_1_51_1","unstructured":"Nvidia. 2021. NVVM IR . https:\/\/docs.nvidia.com\/cuda\/nvvm-ir-spec\/index.html . Nvidia. 2021. NVVM IR . https:\/\/docs.nvidia.com\/cuda\/nvvm-ir-spec\/index.html ."},{"key":"e_1_2_1_52_1","unstructured":"NXP. 2020. NXP uses Glow to optimize models for low-power NXP MCUs . https:\/\/www.nxp.com\/company\/blog\/glow-compiler-optimizes-neural-networks-for-low-power-nxp-mcus:BL-OPTIMIZES-NEURAL-NETWORKS . NXP. 2020. NXP uses Glow to optimize models for low-power NXP MCUs . https:\/\/www.nxp.com\/company\/blog\/glow-compiler-optimizes-neural-networks-for-low-power-nxp-mcus:BL-OPTIMIZES-NEURAL-NETWORKS ."},{"key":"e_1_2_1_53_1","unstructured":"OctoML. 2021. OctoML leverages TVM to optimize and deploy models . https:\/\/octoml.ai\/features\/maximize-performance\/. OctoML. 2021. OctoML leverages TVM to optimize and deploy models . https:\/\/octoml.ai\/features\/maximize-performance\/."},{"key":"e_1_2_1_54_1","volume-title":"Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875","author":"Odena Augustus","year":"2018","unstructured":"Augustus Odena and Ian Goodfellow . 2018 . Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875 (2018). Augustus Odena and Ian Goodfellow. 2018. Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875 (2018)."},{"key":"e_1_2_1_55_1","unstructured":"ONNX. 2021. ONNX Zoo: A collection of pre-trained state-of-the-art models in the ONNX format . https:\/\/github.com\/onnx\/models . ONNX. 2021. ONNX Zoo: A collection of pre-trained state-of-the-art models in the ONNX format . https:\/\/github.com\/onnx\/models ."},{"key":"e_1_2_1_56_1","volume-title":"MDPFuzzer: Finding Crash-Triggering State Sequences in Models Solving the Markov Decision Process. arXiv preprint arXiv:2112.02807","author":"Pang Qi","year":"2021","unstructured":"Qi Pang , Yuanyuan Yuan , and Shuai Wang . 2021. MDPFuzzer: Finding Crash-Triggering State Sequences in Models Solving the Markov Decision Process. arXiv preprint arXiv:2112.02807 ( 2021 ). Qi Pang, Yuanyuan Yuan, and Shuai Wang. 2021. MDPFuzzer: Finding Crash-Triggering State Sequences in Models Solving the Markov Decision Process. arXiv preprint arXiv:2112.02807 (2021)."},{"key":"e_1_2_1_57_1","volume-title":"et almbox","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et almbox . 2019 . Pytorch : An imperative style, high-performance deep learning library. In Advances in neural information processing systems. 8026--8037. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et almbox. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems. 8026--8037."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_2_1_60_1","unstructured":"Qualcomm. 2020. Qualcomm contributes Hexagon DSP improvements to the Apache TVM community . https:\/\/developer.qualcomm.com\/blog\/tvm-open-source-compiler-now-includes-initial-support-qualcomm-hexagon-dsp . Qualcomm. 2020. Qualcomm contributes Hexagon DSP improvements to the Apache TVM community . https:\/\/developer.qualcomm.com\/blog\/tvm-open-source-compiler-now-includes-initial-support-qualcomm-hexagon-dsp ."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499370.2462176"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.442"},{"key":"e_1_2_1_63_1","volume-title":"et almbox","author":"Rotem Nadav","year":"2018","unstructured":"Nadav Rotem , Jordan Fix , Saleem Abdulrasool , Garret Catron , Summer Deng , Roman Dzhabarov , Nick Gibson , James Hegeman , Meghan Lele , Roman Levenstein , et almbox . 2018 . Glow : Graph lowering compiler techniques for neural networks. arXiv preprint arXiv:1805.00907 (2018). Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Levenstein, et almbox. 2018. Glow: Graph lowering compiler techniques for neural networks. arXiv preprint arXiv:1805.00907 (2018)."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion52605.2021.00052"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2019.00022"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468591"},{"key":"e_1_2_1_68_1","volume-title":"Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2019.00010"},{"key":"e_1_2_1_70_1","volume-title":"Astraea: Grammar-based Fairness Testing. arXiv preprint arXiv:2010.02542","author":"Soremekun Ezekiel","year":"2020","unstructured":"Ezekiel Soremekun , Sakshi Udeshi , and Sudipta Chattopadhyay . 2020 . Astraea: Grammar-based Fairness Testing. arXiv preprint arXiv:2010.02542 (2020). Ezekiel Soremekun, Sakshi Udeshi, and Sudipta Chattopadhyay. 2020. Astraea: Grammar-based Fairness Testing. arXiv preprint arXiv:2010.02542 (2020)."},{"key":"e_1_2_1_71_1","volume-title":"Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence .","author":"Srisakaokul Siwakorn","year":"2018","unstructured":"Siwakorn Srisakaokul , Zhengkai Wu , Angello Astorga , Oreoluwa Alebiosu , and Tao Xie . 2018 . Multiple-implementation testing of supervised learning software . In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence . Siwakorn Srisakaokul, Zhengkai Wu, Angello Astorga, Oreoluwa Alebiosu, and Tao Xie. 2018. Multiple-implementation testing of supervised learning software. In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence ."},{"key":"e_1_2_1_72_1","unstructured":"Chengnian Sun Vu Le and Zhendong Su. 2016. Finding Compiler Bugs via Live Code Mutation. In OOPSLA . Chengnian Sun Vu Le and Zhendong Su. 2016. Finding Compiler Bugs via Live Code Mutation. In OOPSLA ."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_2_1_75_1","volume-title":"XLA: Optimizing Compiler for TensorFlow . https:\/\/clang.llvm.org\/docs\/UndefinedBehaviorSanitizer.html .","year":"2019","unstructured":"TensorFlow. 2019 . XLA: Optimizing Compiler for TensorFlow . https:\/\/clang.llvm.org\/docs\/UndefinedBehaviorSanitizer.html . TensorFlow. 2019. XLA: Optimizing Compiler for TensorFlow . https:\/\/clang.llvm.org\/docs\/UndefinedBehaviorSanitizer.html ."},{"key":"e_1_2_1_76_1","unstructured":"Tensorflow. 2020. Tensorflow backend for ONNX (Open Neural Network Exchange) . https:\/\/pypi.org\/project\/onnx-tf\/. Tensorflow. 2020. Tensorflow backend for ONNX (Open Neural Network Exchange) . https:\/\/pypi.org\/project\/onnx-tf\/."},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-09985-1"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_2_1_79_1","volume-title":"Evaluating robustness of neural networks with mixed integer programming. arXiv preprint arXiv:1711.07356","author":"Tjeng Vincent","year":"2017","unstructured":"Vincent Tjeng , Kai Xiao , and Russ Tedrake . 2017. Evaluating robustness of neural networks with mixed integer programming. arXiv preprint arXiv:1711.07356 ( 2017 ). Vincent Tjeng, Kai Xiao, and Russ Tedrake. 2017. Evaluating robustness of neural networks with mixed integer programming. arXiv preprint arXiv:1711.07356 (2017)."},{"key":"e_1_2_1_80_1","unstructured":"TVM. 2020. AlterOpLayout . https:\/\/tvm.apache.org\/docs\/api\/python\/relay\/transform.html . TVM. 2020. AlterOpLayout . https:\/\/tvm.apache.org\/docs\/api\/python\/relay\/transform.html ."},{"key":"e_1_2_1_81_1","doi-asserted-by":"crossref","unstructured":"Sakshi Udeshi Pryanshu Arora and Sudipta Chattopadhyay. 2018. Automated Directed Fairness Testing (ASE). Sakshi Udeshi Pryanshu Arora and Sudipta Chattopadhyay. 2018. Automated Directed Fairness Testing (ASE).","DOI":"10.1145\/3238147.3238165"},{"key":"e_1_2_1_82_1","volume-title":"et almbox","author":"Uesato Jonathan","year":"2018","unstructured":"Jonathan Uesato , Ananya Kumar , Csaba Szepesvari , Tom Erez , Avraham Ruderman , Keith Anderson , Nicolas Heess , Pushmeet Kohli , et almbox . 2018 . Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv:1812.01647 (2018). Jonathan Uesato, Ananya Kumar, Csaba Szepesvari, Tom Erez, Avraham Ruderman, Keith Anderson, Nicolas Heess, Pushmeet Kohli, et almbox. 2018. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv:1812.01647 (2018)."},{"key":"e_1_2_1_83_1","volume-title":"Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions. arXiv preprint arXiv:1802.04730","author":"Vasilache Nicolas","year":"2018","unstructured":"Nicolas Vasilache , Oleksandr Zinenko , Theodoros Theodoridis , Priya Goyal , Zachary DeVito , William S Moses , Sven Verdoolaege , Andrew Adams , and Albert Cohen . 2018. Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions. arXiv preprint arXiv:1802.04730 ( 2018 ). Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. 2018. Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions. arXiv preprint arXiv:1802.04730 (2018)."},{"key":"e_1_2_1_84_1","volume-title":"High-Performance Computing on the Intel\u00ae Xeon Phi?","author":"Wang Endong","unstructured":"Endong Wang , Qing Zhang , Bo Shen , Guangyong Zhang , Xiaowei Lu , Qing Wu , and Yajuan Wang . 2014. Intel math kernel library . In High-Performance Computing on the Intel\u00ae Xeon Phi? . Springer , 167--188. Endong Wang, Qing Zhang, Bo Shen, Guangyong Zhang, Xiaowei Lu, Qing Wu, and Yajuan Wang. 2014. Intel math kernel library. In High-Performance Computing on the Intel\u00ae Xeon Phi?. Springer, 167--188."},{"key":"e_1_2_1_85_1","doi-asserted-by":"crossref","unstructured":"Jingyi Wang Guoliang Dong Jun Sun Xinyu Wang and Peixin Zhang. 2019. Adversarial Sample Detection for Deep Neural Network Through Model Mutation Testing (ICSE). Jingyi Wang Guoliang Dong Jun Sun Xinyu Wang and Peixin Zhang. 2019. Adversarial Sample Detection for Deep Neural Network Through Model Mutation Testing (ICSE).","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_2_1_86_1","doi-asserted-by":"crossref","unstructured":"Shuai Wang and Zhendong Su. 2020. Metamorphic Object Insertion for Testing Object Detection Systems. In ASE . Shuai Wang and Zhendong Su. 2020. Metamorphic Object Insertion for Testing Object Detection Systems. In ASE .","DOI":"10.1145\/3324884.3416584"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409761"},{"key":"e_1_2_1_88_1","volume-title":"Google and Nvidia Tie in MLPerf","author":"Ward-Foxton Sally","unstructured":"Sally Ward-Foxton . 2021. Google and Nvidia Tie in MLPerf ; Graphcore and Habana Debut . https:\/\/www.eetimes.com\/google-and-nvidia-tie-in-mlperf-graphcore-and-habana-debut . Sally Ward-Foxton. 2021. Google and Nvidia Tie in MLPerf; Graphcore and Habana Debut . https:\/\/www.eetimes.com\/google-and-nvidia-tie-in-mlperf-graphcore-and-habana-debut ."},{"key":"e_1_2_1_89_1","unstructured":"Xilinx. 2020. Xilinx support TVM on DPU . https:\/\/www.xilinx.com\/html_docs\/xilinx2019_2\/vitis_doc\/deploying_running.html . Xilinx. 2020. Xilinx support TVM on DPU . https:\/\/www.xilinx.com\/html_docs\/xilinx2019_2\/vitis_doc\/deploying_running.html ."},{"key":"e_1_2_1_90_1","volume-title":"2021 a. Enhancing Deep Neural Networks Testing by Traversing Data Manifold. arXiv preprint arXiv:2112.01956","author":"Yuan Yuanyuan","year":"2021","unstructured":"Yuanyuan Yuan , Qi Pang , and Shuai Wang . 2021 a. Enhancing Deep Neural Networks Testing by Traversing Data Manifold. arXiv preprint arXiv:2112.01956 ( 2021 ). Yuanyuan Yuan, Qi Pang, and Shuai Wang. 2021 a. Enhancing Deep Neural Networks Testing by Traversing Data Manifold. arXiv preprint arXiv:2112.01956 (2021)."},{"key":"e_1_2_1_91_1","volume-title":"2021 b. You Can't See the Forest for Its Trees: Assessing Deep Neural Network Testing via NeuraL Coverage. arXiv preprint arXiv:2112.01955","author":"Yuan Yuanyuan","year":"2021","unstructured":"Yuanyuan Yuan , Qi Pang , and Shuai Wang . 2021 b. You Can't See the Forest for Its Trees: Assessing Deep Neural Network Testing via NeuraL Coverage. arXiv preprint arXiv:2112.01955 ( 2021 ). Yuanyuan Yuan, Qi Pang, and Shuai Wang. 2021 b. You Can't See the Forest for Its Trees: Assessing Deep Neural Network Testing via NeuraL Coverage. arXiv preprint arXiv:2112.01955 (2021)."},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01663"},{"key":"e_1_2_1_93_1","volume-title":"International Workshop on Languages and Compilers for Parallel Computing. Springer, 17--31","author":"Yuki Tomofumi","year":"2012","unstructured":"Tomofumi Yuki , Gautam Gupta , DaeGon Kim , Tanveer Pathan , and Sanjay Rajopadhye . 2012 . Alphaz: A system for design space exploration in the polyhedral model . In International Workshop on Languages and Compilers for Parallel Computing. Springer, 17--31 . Tomofumi Yuki, Gautam Gupta, DaeGon Kim, Tanveer Pathan, and Sanjay Rajopadhye. 2012. Alphaz: A system for design space exploration in the polyhedral model. In International Workshop on Languages and Compilers for Parallel Computing. Springer, 17--31."},{"key":"e_1_2_1_94_1","volume-title":"my program worked. Today, it does not. Why? ACM SIGSOFT Software engineering notes","author":"Zeller Andreas","year":"1999","unstructured":"Andreas Zeller . 1999. Yesterday , my program worked. Today, it does not. Why? ACM SIGSOFT Software engineering notes , Vol. 24 , 6 ( 1999 ), 253--267. Andreas Zeller. 1999. Yesterday, my program worked. Today, it does not. Why? ACM SIGSOFT Software engineering notes , Vol. 24, 6 (1999), 253--267."},{"key":"e_1_2_1_95_1","volume-title":"2020 a. Machine learning testing: Survey, landscapes and horizons","author":"Zhang Jie M","year":"2020","unstructured":"Jie M Zhang , Mark Harman , Lei Ma , and Yang Liu . 2020 a. Machine learning testing: Survey, landscapes and horizons . IEEE Transactions on Software Engineering ( 2020 ). Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020 a. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering (2020)."},{"key":"e_1_2_1_96_1","doi-asserted-by":"crossref","unstructured":"Mengshi Zhang Yuqun Zhang Lingming Zhang Cong Liu and Sarfraz Khurshid. 2018b. DeepRoad: GAN-based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. In ASE . Mengshi Zhang Yuqun Zhang Lingming Zhang Cong Liu and Sarfraz Khurshid. 2018b. DeepRoad: GAN-based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. In ASE .","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464843"},{"key":"e_1_2_1_98_1","volume-title":"An Empirical Study on TensorFlow Program Bugs (ISSTA","author":"Zhang Yuhao","year":"2018","unstructured":"Yuhao Zhang , Yifan Chen , Shing-Chi Cheung , Yingfei Xiong , and Lu Zhang . 2018a. An Empirical Study on TensorFlow Program Bugs (ISSTA 2018 ). Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018a. An Empirical Study on TensorFlow Program Bugs (ISSTA 2018)."},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409720"},{"key":"e_1_2_1_100_1","unstructured":"Lianmin Zheng Chengfan Jia Minmin Sun Zhao Wu Cody Hao Yu Ameer Haj-Ali Yida Wang Jun Yang Danyang Zhuo Koushik Sen et almbox. 2020 a. Ansor: Generating high-performance tensor programs for deep learning. In 14th $$USENIX$$ Symposium on Operating Systems Design and Implementation ($$OSDI$$ 20) . 863--879. Lianmin Zheng Chengfan Jia Minmin Sun Zhao Wu Cody Hao Yu Ameer Haj-Ali Yida Wang Jun Yang Danyang Zhuo Koushik Sen et almbox. 2020 a. Ansor: Generating high-performance tensor programs for deep learning. In 14th $$USENIX$$ Symposium on Operating Systems Design and Implementation ($$OSDI$$ 20) . 863--879."},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378508"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380422"},{"key":"e_1_2_1_103_1","volume-title":"Fairness-aware machine learning: a perspective. arXiv preprint arXiv:1708.00754","author":"Zliobaite Indre","year":"2017","unstructured":"Indre Zliobaite . 2017. Fairness-aware machine learning: a perspective. arXiv preprint arXiv:1708.00754 ( 2017 ). Indre Zliobaite. 2017. Fairness-aware machine learning: a perspective. arXiv preprint arXiv:1708.00754 (2017)."}],"container-title":["Proceedings of the ACM on Measurement and Analysis of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3508035","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3508035","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:12:29Z","timestamp":1750191149000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3508035"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,24]]},"references-count":102,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,2,24]]}},"alternative-id":["10.1145\/3508035"],"URL":"https:\/\/doi.org\/10.1145\/3508035","relation":{},"ISSN":["2476-1249"],"issn-type":[{"value":"2476-1249","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,24]]},"assertion":[{"value":"2022-02-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}