{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:41:10Z","timestamp":1791459670350,"version":"4.3.4"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62232014 and 62372227"],"award-info":[{"award-number":["62232014 and 62372227"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:p>Deep Learning (DL) has already played a significant role in numerous fields, making it crucial to ensure the stability of both training and inference in DL systems. The computation of DL models can be viewed as the execution of a series of DL operators, which are essential components that perform the core numerical computations. Therefore, the numerical stability of these operators is critical for the overall quality of these systems. Operators with numerical instabilities can cause degraded model performance or even numerical exceptions that disrupt the training process. A common practice to improve numerical stability is increasing the precision of specific floating-point computations. However, determining the appropriate data type for internal computations, referred to as precision tuning, remains a significant challenge.<\/jats:p>\n                  <jats:p>\n                    This article presents the first systematic study on precision tuning in the implementation of DL operators. We introduce\n                    <jats:sc>OpScanner<\/jats:sc>\n                    , capable of extracting precision casting operations critical to numerical stability of operators within existing DL frameworks. Using the captured operations, we conduct a comprehensive categorization and comparative analysis between two popular DL frameworks. Our findings can help developers understand precision tuning in operators and implement new operators. Based on the results of our comparative analysis, we conduct precision testing experiments on a set of operators from both frameworks to examine the impact of precision tuning. The test results show that tested TensorFlow operators produce significantly more floating-point errors compared to their PyTorch counterparts due to a lack of precision tuning. One of the most inaccurate operators has been confirmed to have numerical issues by the TensorFlow community. Additionally, our study examines the effect of precision tuning on weight updates during training. We found that larger errors, resulting from a lack of tuning, lead to greater weight deviations during training.\n                  <\/jats:p>","DOI":"10.1145\/3773992","type":"journal-article","created":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T14:55:08Z","timestamp":1761836108000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["A Study of Floating-Point Precision Tuning in Deep Learning Operators Implementations"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-6737-9953","authenticated-orcid":false,"given":"Zhongzhen","family":"Wen","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-9198-2615","authenticated-orcid":false,"given":"Hongyu","family":"Liu","sequence":"additional","affiliation":[{"name":"Software Engineering Application Technology Lab, Huawei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6962-6330","authenticated-orcid":false,"given":"Tingwei","family":"Zhu","sequence":"additional","affiliation":[{"name":"The State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4011-5350","authenticated-orcid":false,"given":"Minxue","family":"Pan","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5777-7759","authenticated-orcid":false,"given":"Shaohua","family":"Wang","sequence":"additional","affiliation":[{"name":"Central University of Finance and Economics, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4069-2550","authenticated-orcid":false,"given":"Yuanyi","family":"Lin","sequence":"additional","affiliation":[{"name":"Software Engineering Application Technology Lab, Huawei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-8452-9662","authenticated-orcid":false,"given":"Kairui","family":"Liu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0104-2731","authenticated-orcid":false,"given":"Tian","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3090-9568","authenticated-orcid":false,"given":"Xuandong","family":"Li","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,17]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/IEEESTD.2019.8766229"},{"key":"e_1_3_2_3_2","unstructured":"NVIDIA Corporation. 2024. CUDA. Retrieved from https:\/\/docs.nvidia.com\/cuda\/"},{"key":"e_1_3_2_4_2","unstructured":"LLVM. 2024. Iterating over Def-Use Chains in LLVM. Retrieved from https:\/\/llvm.org\/docs\/ProgrammersManual.html#iterating-over-def-use-use-def-chains"},{"key":"e_1_3_2_5_2","unstructured":"The Clang Team. 2024. JSON Compilation Database Format Specification. Retrieved from https:\/\/clang.llvm.org\/docs\/JSONCompilationDatabase.html"},{"key":"e_1_3_2_6_2","unstructured":"NVIDIA Corporation. 2024. NVIDIA cuDNN. Retrieved from https:\/\/developer.nvidia.com\/cudnn"},{"key":"e_1_3_2_7_2","unstructured":"NVIDIA Corporation. 2024. NVIDIA Nsight Systems. Retrieved from https:\/\/developer.nvidia.com\/nsight-systems"},{"key":"e_1_3_2_8_2","unstructured":"NVIDIA Corporation. 2024. Nvidia Tensor Cores. Retrieved from https:\/\/www.nvidia.com\/en-us\/data-center\/tensor-cores\/"},{"key":"e_1_3_2_9_2","unstructured":"LLVM Project. 2024. Source Level Debugging with LLVM. Retrieved from https:\/\/llvm.org\/docs\/SourceLevelDebugging.html"},{"key":"e_1_3_2_10_2","unstructured":"Google. 2024. tf.raw_ops. Retrieved from https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/raw_ops"},{"key":"e_1_3_2_11_2","unstructured":"The FPBench Community. 2024. FPBench. Retrieved from https:\/\/fpbench.org\/"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1177\/10943420211003313"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-88806-0_1"},{"key":"e_1_3_2_14_2","unstructured":"David H. Bailey. 2008. Resolving numerical anomalies in scientific computation (2008). Retrieved from https:\/\/www.davidhbailey.com\/dhbpapers\/numerical-bugs.pdf"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","unstructured":"Junjie Chen Yihua Liang Qingchao Shen Jiajun Jiang and Shuochuan Li. 2023. Toward understanding deep learning framework bugs. ACM Transactions on Software Engineering and Methodology 32 6 Article 135 (Sept. 2023) 31 pages. DOI: 10.1145\/3587155","DOI":"10.1145\/3587155"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","unstructured":"Stefano Cherubin and Giovanni Agosta. 2020. Tools for reduced precision computation: A survey. ACM Computing Surveys 53 2 Article 33 (Apr. 2020) 35 pages. DOI: 10.1145\/3381039","DOI":"10.1145\/3381039"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/LES.2019.2913774"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3093333.3009846"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/2555243.2555265"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2019.00363"},{"key":"e_1_3_2_21_2","first-page":"2383","volume-title":"Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923)","author":"Christou Neophytos","year":"2023","unstructured":"Neophytos Christou, Di Jin, Vaggelis Atlidakis, Baishakhi Ray, and Vasileios P. Kemerlis. 2023. IvySyn: Automated vulnerability discovery in deep learning frameworks. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923). Joseph A. Calandrino, and Carmela Troncoso (Eds.), USENIX Association, 2383\u20132400. Retrieved from https:\/\/www.usenix.org\/conference\/usenixsecurity23\/presentation\/christou"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPS.2018.00028"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/2535838.2535874"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598067"},{"key":"e_1_3_2_25_2","first-page":"7393","volume-title":"Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923)","author":"Deng Zizhuang","year":"2023","unstructured":"Zizhuang Deng, Guozhu Meng, Kai Chen, Tong Liu, Lu Xiang, and Chunyang Chen. 2023. Differential testing of cross deep learning framework APIs: Revealing inconsistencies and vulnerabilities. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923). USENIX Association, Anaheim, CA, 7393\u20137410. Retrieved from https:\/\/www.usenix.org\/conference\/usenixsecurity23\/presentation\/deng-zizhuang"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_27_2","first-page":"208","volume-title":"Automated Deduction\u2014CADE-24","author":"Sicun Gao Soonho Kong","year":"2013","unstructured":"Soonho Kong Sicun Gao and Edmund M. Clarke. 2013. dReal: An SMT solver for nonlinear theories over the reals. In Automated Deduction\u2014CADE-24. Maria Paola Bonacina (Ed.), Springer, Berlin, 208\u2013214."},{"key":"e_1_3_2_28_2","doi-asserted-by":"crossref","unstructured":"Amir Gholami Sehoon Kim Zhen Dong Zhewei Yao Michael W. Mahoney and Kurt Keutzer. 2021. A survey of quantization methods for efficient neural network inference. arXiv:2103.13630. Retrieved from https:\/\/arxiv.org\/abs\/2103.13630","DOI":"10.1201\/9781003162810-13"},{"key":"e_1_3_2_29_2","unstructured":"Alicia Golden Samuel Hsia Fei Sun Bilge Acun Basil Hosmer Yejin Lee Zachary DeVito Jeff Johnson Gu-Yeon Wei David Brooks et al. 2024. Is flash attention stable? arXiv:2405.02803. Retrieved from https:\/\/arxiv.org\/abs\/2405.02803"},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1007\/11823230_3","volume-title":"Static Analysis","author":"Goubault Eric","year":"2006","unstructured":"Eric Goubault and Sylvie Putot. 2006. Static analysis of numerical algorithms. In Static Analysis. Kwangkeun Yi (Ed.), Springer, Berlin, 18\u201334."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213862"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0962492922000022"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00140"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMST.2018.2869360"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_2_36_2","unstructured":"Dhiraj Kalamkar Dheevatsa Mudigere Naveen Mellempudi Dipankar Das Kunal Banerjee Sasikanth Avancha Dharma Teja Vooturi Nataraj Jammalamadaka Jianyu Huang Hector Yuen et al. 2019. A study of BFLOAT16 for deep learning training. arXiv:1905.12322. Retrieved from https:\/\/arxiv.org\/abs\/1905.12322"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510095"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-20656-7_12"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/CGO.2004.1281665"},{"key":"e_1_3_2_40_2","unstructured":"Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ili\u0107 Daniel Hesslow Roman Castagn\u00e9 Alexandra Sasha Luccioni Fran\u00e7ois Yvon Matthias Gall\u00e9 et al. 2023. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https:\/\/arxiv.org\/abs\/2211.05100"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649449"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00156"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","unstructured":"Xiaoye S. Li James W. Demmel David H. Bailey Greg Henry Yozo Hida Jimmy Iskandar William Kahan Suh Y. Kang Anil Kapur Michael C. Martin et al. 2002. Design implementation and testing of extended and mixed precision BLAS. ACM Transactions on Mathematical Software 28 2 (June 2002) 152\u2013205. DOI: 10.1145\/567806.567808","DOI":"10.1145\/567806.567808"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","unstructured":"Debasmita Lohar Clothilde Jeangoudoux Anastasia Volkova and Eva Darulova. 2023. Sound mixed fixed-point quantization of neural networks. ACM Transactions on Embedded Computing Systems 22 5s Article 136 (Sept. 2023) 26 pages. DOI: 10.1145\/3609118","DOI":"10.1145\/3609118"},{"key":"e_1_3_2_45_2","first-page":"614","volume-title":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC \u201918)","author":"Menon Harshitha","year":"2018","unstructured":"Harshitha Menon, Michael O. Lam, Daniel Osei-Kuffuor, Markus Schordan, Scott Lloyd, Kathryn Mohror, and Jeffrey Hittinger. 2018. Adapt: Algorithmic differentiation applied to floating-point precision tuning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC \u201918). IEEE, 614\u2013626."},{"key":"e_1_3_2_46_2","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR \u201918)","author":"Micikevicius Paulius","year":"2018","unstructured":"Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Garc\u00eda, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In Proceedings of the 6th International Conference on Learning Representations (ICLR \u201918). Conference Track Proceedings, OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=r1gs9JgRZ"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3361566"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416545"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884850"},{"key":"e_1_3_2_50_2","first-page":"1","volume-title":"Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis","author":"Rubio-Gonz\u00e1lez Cindy","year":"2013","unstructured":"Cindy Rubio-Gonz\u00e1lez, Cuong Nguyen, Hong Diep Nguyen, James Demmel, William Kahan, Koushik Sen, David H. Bailey, Costin Iancu, and David Hough. 2013. Precimonious: Tuning assistant for floating-point precision. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, 1\u201312."},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Shaila Sharmin Subhankar Anwar Hossain Zahid Chiamaka Bhattacharjee Miryung Igwilo and Wei Le Kim. 2025. Automatically detecting numerical instability in machine learning applications via soft assertions. arXiv:2504.15507. Retrieved from https:\/\/arxiv.org\/abs\/2504.15507","DOI":"10.1145\/3729394"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468591"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598088"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","unstructured":"Alexey Solovyev Marek S. Baranowski Ian Briggs Charles Jacobsen Zvonimir Rakamari\u0107 and Ganesh Gopalakrishnan. 2018. Rigorous estimation of floating-point round-off errors with symbolic Taylor expansions. ACM Transactions on Programming Languages and Systems 41 1 Article 2 (Dec. 2018) 39 pages. DOI: 10.1145\/3230733","DOI":"10.1145\/3230733"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3559561"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510071"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639121"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00129"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE62328.2024.00018"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468612"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3505243"},{"key":"e_1_3_2_62_2","unstructured":"Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai Ming Ding Zhuoyi Yang Yifan Xu Wendi Zheng Xiao Xia et al. 2022. GLM-130B: An open bilingual pre-trained model. arXiv:2210.02414. Retrieved from https:\/\/arxiv.org\/abs\/2210.02414"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_64_2","unstructured":"Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen Shuohui Chen Christopher Dewan Mona Diab Xian Li Xi Victoria Lin et al. 2022. OPT: Open pre-trained transformer language models. arXiv:2205.01068. Retrieved from https:\/\/arxiv.org\/abs\/2205.01068"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464843"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409720"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","unstructured":"Daming Zou Muhan Zeng Yingfei Xiong Zhoulai Fu Lu Zhang and Zhendong Su. 2019. Detecting floating-point errors via atomic conditions. Proceedings of the ACM on Programming Languages 4 POPL Article 60 (Dec. 2019) 27 pages. DOI: 10.1145\/3371128","DOI":"10.1145\/3371128"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3773992","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T14:53:06Z","timestamp":1781707986000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3773992"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,17]]},"references-count":67,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7,31]]}},"alternative-id":["10.1145\/3773992"],"URL":"https:\/\/doi.org\/10.1145\/3773992","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,17]]},"assertion":[{"value":"2025-01-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}