{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T15:21:49Z","timestamp":1770996109829,"version":"3.50.1"},"reference-count":32,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62172205"],"award-info":[{"award-number":["62172205"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,3,31]]},"abstract":"<jats:p>\n                    The rapid evolution of machine learning model training has outpaced the development of corresponding measurement and testing tools, leading to two significant challenges. Firstly, developers of deep learning frameworks struggle to identify operators that fail to meet precision criteria, as these issues may only manifest in a few data points. Secondly, model trainers lack effective methods to estimate precision loss caused by operators during training. To address these issues, we introduce a Pythonic framework inspired by common network layer definitions in deep learning. Our framework includes two new layers: the Fuzz Layer and the Check Layer, designed to aid in measurement and testing. The Fuzz Layer introduces minor perturbations to tensor inputs for any deterministic layer under testing (LUT). The Check Layer then measures precision by analyzing the differences before and after the perturbation. This approach estimates a lower bound of the maximal relative error and alerts developers or trainers of potential bugs if the difference exceeds a pre-defined tolerance. Additionally, Check Layers can be used independently to conduct precision tests for specific layers, ensuring the precision of operators during runtime. Despite the additional memory and time requirements, this runtime testing ensures proper training of the original model. We demonstrate the utility of our framework, FuMi, through two experiments. First, we tested 21 torch operators across nine popular machine learning models using PyTorch for various tasks, finding that the\n                    <jats:monospace>conv2d<\/jats:monospace>\n                    and\n                    <jats:monospace>linear<\/jats:monospace>\n                    operators often fail to meet precision requirements. Second, to showcase the generalizability of our framework, we tested the\n                    <jats:monospace>ATTENTION<\/jats:monospace>\n                    operator. By comparing different implementations of state-of-the-art\n                    <jats:monospace>ATTENTION<\/jats:monospace>\n                    operators, we found that the maximum relative error of the\n                    <jats:monospace>ATTENTION<\/jats:monospace>\n                    operator is not less than 1%, which is 13 times larger than that measured by Predoo (a unit test tool). This framework provides a robust solution for identifying precision issues in deep learning models, ensuring reliable and accurate model training.\n                  <\/jats:p>","DOI":"10.1145\/3734866","type":"journal-article","created":{"date-parts":[[2025,5,8]],"date-time":"2025-05-08T11:37:51Z","timestamp":1746704271000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["FuMi: A Runtime Fuzz-based Machine Learning Precision Measurement and Testing Framework"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2902-2249","authenticated-orcid":false,"given":"Peng","family":"Zhang","sequence":"first","affiliation":[{"name":"Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, Esch-sur-Alzette, Luxembourg"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1852-2547","authenticated-orcid":false,"given":"Mike","family":"Papadakis","sequence":"additional","affiliation":[{"name":"Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, Esch-sur-Alzette, Luxembourg"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4645-2526","authenticated-orcid":false,"given":"Yuming","family":"Zhou","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,2,13]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Peng Zhang. 2024. FuMi GitHub repository. Retrieved from https:\/\/github.com\/zhangpengNJU\/FuMi2024"},{"key":"e_1_3_3_3_2","first-page":"265","article-title":"TensorFlow: Large-scale machine learning on heterogeneous systems","author":"Abadi Mart\u00edn","year":"2015","unstructured":"Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI '16). USENIX Association, Savannah, GA, USA, 265\u2013283.","journal-title":"Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI '16)"},{"issue":"24","key":"e_1_3_3_4_2","first-page":"3370","article-title":"Keras: The Python deep learning library","volume":"3","author":"Chollet Fran\u00e7ois","year":"2018","unstructured":"Fran\u00e7ois Chollet. 2018. Keras: The Python deep learning library. Journal of Open Source Software 3, 24 (2018), 3370.","journal-title":"Journal of Open Source Software"},{"key":"e_1_3_3_5_2","unstructured":"NVIDIA Corporation. 2023. NVIDIA Official Website. Retrieved from https:\/\/www.nvidia.com"},{"key":"e_1_3_3_6_2","unstructured":"Daniel Hanchen. 2024. RoPE Loses Precision for Llama\/Gemma + Gemma logits.float(). Retrieved from https:\/\/github.com\/huggingface\/transformers\/pull\/29285"},{"key":"e_1_3_3_7_2","unstructured":"Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra and Christopher R\u00e9. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). Retrieved from https:\/\/arxiv.org\/abs\/2205.14135"},{"key":"e_1_3_3_8_2","unstructured":"Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra and Christopher R\u00e9. 2023. FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08691. Retrieved from https:\/\/arxiv.org\/abs\/2307.08691"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549085"},{"key":"e_1_3_3_10_2","unstructured":"Ian Goodfellow Yoshua Bengio and Aaron Courville. 2016. Deep Learning. MIT Press. Retrieved from https:\/\/www.deeplearningbook.org\/"},{"key":"e_1_3_3_11_2","unstructured":"Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv:1308.0850. Retrieved from https:\/\/arxiv.org\/abs\/1308.0850"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_3_16_2","first-page":"1776","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Hu Qiang","year":"2023","unstructured":"Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Mike Papadakis, Lei Ma, and Yves Le Traon. 2023. Aries: Efficient testing of deep neural networks via labeling-free accuracy estimation. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1776\u20131787."},{"key":"e_1_3_3_17_2","unstructured":"Huawei Technologies Co. Ltd. 2023. Huawei Ascend Official Website. Retrieved from https:\/\/www.hiascend.com\/en\/"},{"key":"e_1_3_3_18_2","unstructured":"Huawei Technologies Co. Ltd. 2024. Dump Mode. Retrieved from https:\/\/www.hiascend.com\/document\/detail\/zh\/canncommercial\/80RC1\/devaids\/auxiliarydevtool\/atlasatc_16_0112.html"},{"key":"e_1_3_3_19_2","unstructured":"Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv:1312.6114. Retrieved from https:\/\/arxiv.org\/abs\/1312.6114"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"},{"key":"e_1_3_3_21_2","unstructured":"Cambricon Technologies Corporation Limited. 2023. Cambricon Official Website. Retrieved from https:\/\/www.cambricon.com\/"},{"key":"e_1_3_3_22_2","doi-asserted-by":"crossref","unstructured":"Jiawei Liu Yuheng Huang Zhijie Wang Lei Ma Chunrong Fang Mingzheng Gu Xufan Zhang and Zhenyu Chen. 2023. Generation-based differential fuzzing for deep learning libraries. ACM Transactions on Software Engineering and Methodology 33 2 (2023) 1\u201328.","DOI":"10.1145\/3628159"},{"key":"e_1_3_3_23_2","unstructured":"NVIDIA Corporation. 2023. NVIDIA cuBLAS Library. Retrieved from https:\/\/docs.nvidia.com\/cuda\/cublas\/index.html"},{"key":"e_1_3_3_24_2","unstructured":"Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga et al. 2019. PyTorch: An imperative style high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019). Curran Associates Inc. 8024\u20138035."},{"key":"e_1_3_3_25_2","unstructured":"PyTorch. 2024. PyTorch Examples Repository. Retrieved December 11 2024 from https:\/\/github.com\/pytorch\/examples"},{"key":"e_1_3_3_26_2","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3560442.3560445"},{"key":"e_1_3_3_28_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_29_2","first-page":"706","volume-title":"Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)","author":"Wang Kairui","year":"2023","unstructured":"Kairui Wang, Yawen Wang, Junjie Wang, and Qing Wang. 2023. Fuzzing with sequence diversity inference for sequential decision-making model testing. In Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 706\u2013717."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510041"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00105"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2021.3107165"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464843"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3734866","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T14:37:15Z","timestamp":1770993435000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3734866"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,13]]},"references-count":32,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,3,31]]}},"alternative-id":["10.1145\/3734866"],"URL":"https:\/\/doi.org\/10.1145\/3734866","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,13]]},"assertion":[{"value":"2024-07-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}