{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,8]],"date-time":"2025-12-08T22:46:04Z","timestamp":1765233964613,"version":"3.38.0"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,2,11]],"date-time":"2025-02-11T00:00:00Z","timestamp":1739232000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,2,11]],"date-time":"2025-02-11T00:00:00Z","timestamp":1739232000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010669","name":"H2020 LEIT Information and Communication Technologies","doi-asserted-by":"publisher","award":["826452"],"award-info":[{"award-number":["826452"]}],"id":[{"id":"10.13039\/100010669","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Optim Appl"],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>In the training of neural networks with low-precision computation and fixed-point arithmetic, rounding errors often cause stagnation or are detrimental to the convergence of the optimizers. This study provides insights into the choice of appropriate stochastic rounding strategies to mitigate the adverse impact of roundoff errors on the convergence of the gradient descent method, for problems satisfying the Polyak\u2013\u0141ojasiewicz inequality. Within this context, we show that a biased stochastic rounding strategy may be even beneficial in so far as it eliminates the vanishing gradient problem and forces the expected roundoff error in a descent direction. Furthermore, we obtain a bound on the convergence rate that is stricter than the one achieved by unbiased stochastic rounding. The theoretical analysis is validated by comparing the performances of various rounding strategies when optimizing several examples using low-precision fixed-point arithmetic.<\/jats:p>","DOI":"10.1007\/s10589-025-00656-1","type":"journal-article","created":{"date-parts":[[2025,2,11]],"date-time":"2025-02-11T13:56:49Z","timestamp":1739282209000},"page":"753-799","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["On the convergence of the gradient descent method with stochastic fixed-point rounding errors under the Polyak\u2013\u0141ojasiewicz inequality"],"prefix":"10.1007","volume":"90","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5784-0367","authenticated-orcid":false,"given":"Lu","family":"Xia","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Massei","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michiel E.","family":"Hochstenbach","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,2,11]]},"reference":[{"key":"656_CR1","unstructured":"Wang, N., Choi, J., Brand, D., Chen, C.-Y., Gopalakrishnan, K.: Training deep neural networks with 8-bit floating point numbers. In: Proc. 32nd Conf. Neural Inf. Process. Syst., pp. 7675\u20137684 (2018)"},{"key":"656_CR2","unstructured":"Gupta, S., Agrawal, A., Gopalakrishnan, K., Narayanan, P.: Deep learning with limited numerical precision. In: Proc. 32nd Int. Conf. Mach. Learn., pp. 1737\u20131746 (2015)"},{"key":"656_CR3","doi-asserted-by":"crossref","unstructured":"Chen, X., Hu, X., Zhou, H., Xu, N.: FxpNet: Training a deep convolutional neural network in fixed-point representation. In: 2017 International Joint Conference on Neural Networks, IEEE, pp. 2494\u20132501 (2017)","DOI":"10.1109\/IJCNN.2017.7966159"},{"issue":"3","key":"656_CR4","doi-asserted-by":"publisher","first-page":"1913","DOI":"10.1109\/JIOT.2021.3091643","volume":"9","author":"D Palossi","year":"2021","unstructured":"Palossi, D., Zimmerman, N., Burrello, A., Conti, F., M\u00fcller, H., Gambardella, L.M., Benini, L., Giusti, A., Guzzi, J.: Fully onboard AI-powered human-drone pose estimation on ultralow-power autonomous flying nano-UAVs. IEEE Internet Things J. 9(3), 1913\u20131929 (2021)","journal-title":"IEEE Internet Things J."},{"key":"656_CR5","doi-asserted-by":"crossref","unstructured":"M\u00fcller, H., Palossi, D., Mach, S., Conti, F., Benini, L.: F\u00fcnfiiber-drone: A modular open-platform 18-grams autonomous nano-drone. In: 2021 Design, Automation & Test in Europe Conference & Exhibition, IEEE, pp. 1610\u20131615 (2021)","DOI":"10.23919\/DATE51398.2021.9474262"},{"issue":"4","key":"656_CR6","doi-asserted-by":"publisher","first-page":"548","DOI":"10.1109\/JETCAS.2021.3126259","volume":"11","author":"V Niculescu","year":"2021","unstructured":"Niculescu, V., Lamberti, L., Conti, F., Benini, L., Palossi, D.: Improving autonomous nano-drones performance via automated end-to-end optimization and deployment of DNNs. IEEE J. Emerg. Sel. Top. Circuits Syst. 11(4), 548\u2013562 (2021)","journal-title":"IEEE J. Emerg. Sel. Top. Circuits Syst."},{"key":"656_CR7","volume-title":"Introductory lectures on convex optimization: a basic course","author":"Y Nesterov","year":"2003","unstructured":"Nesterov, Y.: Introductory lectures on convex optimization: a basic course. Springer, New York, US (2003)"},{"issue":"6","key":"656_CR8","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/0041-5553(64)90079-5","volume":"4","author":"BT Polyak","year":"1963","unstructured":"Polyak, B.T.: Gradient methods for solving equations and inequalities. USSR Comput. Math. & Math. Phys. 4(6), 17\u201332 (1963)","journal-title":"USSR Comput. Math. & Math. Phys."},{"key":"656_CR9","first-page":"87","volume":"117","author":"S Lojasiewicz","year":"1963","unstructured":"Lojasiewicz, S.: A topological property of real analytic subsets. Coll. du CNRS, Les \u00e9quations aux d\u00e9riv\u00e9es partielles 117, 87\u201389 (1963)","journal-title":"Coll. du CNRS, Les \u00e9quations aux d\u00e9riv\u00e9es partielles"},{"key":"656_CR10","doi-asserted-by":"crossref","unstructured":"Karimi, H., Nutini, J., Schmidt, M.: Linear convergence of gradient and proximal-gradient methods under the Polyak-Lojasiewicz condition. In: Proc. Mach. Learn. Knowl. Discovery in Databases: Eur. Conf., Springer, pp. 795\u2013811 (2016)","DOI":"10.1007\/978-3-319-46128-1_50"},{"key":"656_CR11","unstructured":"Charles, Z., Papailiopoulos, D.: Stability and generalization of learning algorithms that converge to global optima. In: Int. Conf. Mach. Learn, PMLR, pp. 745\u2013754 (2018)"},{"key":"656_CR12","unstructured":"Nguyen, Q.N., Mondelli, M.: Global convergence of deep networks with one wide layer followed by pyramidal topology. In: Proc. 34th Conf. Neural Inf. Process. Syst., pp. 11961\u201311972 (2020)"},{"key":"656_CR13","unstructured":"Frei, S., Gu, Q.: Proxy convexity: A unified framework for the analysis of neural networks trained by gradient descent. In: Proc. 35th Conf. Neural Inf. Process. Syst., pp. 7937\u20137949 (2021)"},{"key":"656_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.acha.2021.12.009","volume-title":"Loss landscapes and optimization in over-parameterized non-linear systems and neural networks","author":"C Liu","year":"2022","unstructured":"Liu, C., Zhu, L., Belkin, M.: Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Appl. Comput. Harmon, Anal (2022)"},{"key":"656_CR15","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898718027","volume-title":"Accuracy and stability of numerical algorithms","author":"NJ Higham","year":"2002","unstructured":"Higham, N.J.: Accuracy and stability of numerical algorithms. SIAM, Philadelphia, US (2002)"},{"issue":"1","key":"656_CR16","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1137\/20M1334796","volume":"43","author":"MP Connolly","year":"2021","unstructured":"Connolly, M.P., Higham, N.J., Mary, T.: Stochastic rounding and its probabilistic backward error analysis. SIAM J. Sci. Comput. 43(1), 566\u2013585 (2021)","journal-title":"SIAM J. Sci. Comput."},{"key":"656_CR17","doi-asserted-by":"crossref","unstructured":"Na, T., Ko, J.H., Kung, J., Mukhopadhyay, S.: On-chip training of recurrent neural networks with limited numerical precision. In: Proc. Int. Jt. Conf. Neural Netw., IEEE, pp. 3716\u20133723 (2017)","DOI":"10.1109\/IJCNN.2017.7966324"},{"key":"656_CR18","unstructured":"Ortiz, M., Cristal, A., Ayguad\u00e9, E., Casas, M.: Low-precision floating-point schemes for neural network training. arXiv: 1804.05267 (2018)"},{"issue":"2","key":"656_CR19","doi-asserted-by":"publisher","first-page":"634","DOI":"10.1007\/s10957-023-02345-7","volume":"200","author":"L Xia","year":"2024","unstructured":"Xia, L., Massei, S., Hochstenbach, M.E., Koren, B.: On stochastic roundoff errors in gradient descent with low-precision computation. J. Optim. Theory Appl. 200(2), 634\u2013668 (2024)","journal-title":"J. Optim. Theory Appl."},{"key":"656_CR20","unstructured":"Oberstar, E.L.: Fixed-point representation & fractional math. Oberstar Consulting 9 (2007)"},{"key":"656_CR21","doi-asserted-by":"crossref","unstructured":"Santoro, M.R., Bewick, G., Horowitz, M.A.: Rounding algorithms for IEEE multipliers. In: Proc. 9th Symp. Comput. Arithmetic, IEEE, pp. 176\u2013183 (1989)","DOI":"10.1109\/ARITH.1989.72824"},{"issue":"83","key":"656_CR22","first-page":"198","volume":"81","author":"R Yates","year":"2009","unstructured":"Yates, R.: Fixed-point arithmetic: An introduction. Digital Signal Labs 81(83), 198 (2009)","journal-title":"Digital Signal Labs"},{"issue":"2","key":"656_CR23","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1007\/BF01931367","volume":"16","author":"S Linnainmaa","year":"1976","unstructured":"Linnainmaa, S.: Taylor expansion of the accumulated rounding error. BIT Numer. Math. 16(2), 146\u2013160 (1976)","journal-title":"BIT Numer. Math."},{"key":"656_CR24","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511804441","volume-title":"Convex optimization","author":"S Boyd","year":"2004","unstructured":"Boyd, S., Vandenberghe, L.: Convex optimization. Cambridge University Press, Cambridge, UK (2004)"},{"issue":"5","key":"656_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0041-5553(64)90137-5","volume":"4","author":"BT Polyak","year":"1964","unstructured":"Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. & Math. Phys. 4(5), 1\u201317 (1964)","journal-title":"USSR Comput. Math. & Math. Phys."},{"issue":"3","key":"656_CR26","doi-asserted-by":"publisher","first-page":"627","DOI":"10.1137\/S1052623497331063","volume":"10","author":"DP Bertsekas","year":"2000","unstructured":"Bertsekas, D.P., Tsitsiklis, J.N.: Gradient convergence in gradient methods with errors. SIAM J. Optim. 10(3), 627\u2013642 (2000)","journal-title":"SIAM J. Optim."},{"key":"656_CR27","unstructured":"Schmidt, M., Roux, N., Bach, F.: Convergence rates of inexact proximal-gradient methods for convex optimization. In: Proc. of the 24th Neural Inf. Process. Syst. Conf., pp. 1458\u20131466 (2011)"},{"key":"656_CR28","unstructured":"Nguyen, L.M., Nguyen, N.H., Phan, D.T., Kalagnanam, J.R., Scheinberg, K.: When does stochastic gradient algorithm work well? arXiv:1801.06159 (2018)"},{"key":"656_CR29","unstructured":"NVIDIA, Train with Mixed Precision. Available at https:\/\/docs.nvidia.com\/deeplearning\/performance\/pdf\/Training-Mixed-Precision-User-Guide.pdf (2023)"},{"key":"656_CR30","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1016\/j.ins.2019.04.012","volume":"492","author":"E Moulay","year":"2019","unstructured":"Moulay, E., L\u00e9chapp\u00e9, V., Plestan, F.: Properties of the sign gradient descent algorithms. Inf. Sci. 492, 29\u201339 (2019)","journal-title":"Inf. Sci."},{"key":"656_CR31","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07254-8","volume-title":"Elements of Probability and Statistics","author":"F Biagini","year":"2016","unstructured":"Biagini, F., Campanino, M.: Elements of Probability and Statistics. Springer, Cham, Switzerland (2016)"},{"key":"656_CR32","volume-title":"Probability and Conditional Expectation: Fundamentals for the Empirical Sciences","author":"R Steyer","year":"2017","unstructured":"Steyer, R., Nagel, W.: Probability and Conditional Expectation: Fundamentals for the Empirical Sciences. John Wiley & Sons, Chichester, UK (2017)"},{"issue":"3","key":"656_CR33","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1080\/10659360600787700","volume":"17","author":"J Chen","year":"2006","unstructured":"Chen, J., Tsai, C.-A., Moon, H., Ahn, H., Young, J., Chen, C.-H.: Decision threshold adjustment in class prediction. SAR QSAR Environ. Res. 17(3), 337\u2013352 (2006)","journal-title":"SAR QSAR Environ. Res."},{"key":"656_CR34","unstructured":"Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proc. 13th Int. Conf. Artif. Intell. Stat., pp. 249\u2013256 (2010)"}],"container-title":["Computational Optimization and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10589-025-00656-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10589-025-00656-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10589-025-00656-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T16:37:13Z","timestamp":1741883833000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10589-025-00656-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,11]]},"references-count":34,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,4]]}},"alternative-id":["656"],"URL":"https:\/\/doi.org\/10.1007\/s10589-025-00656-1","relation":{},"ISSN":["0926-6003","1573-2894"],"issn-type":[{"type":"print","value":"0926-6003"},{"type":"electronic","value":"1573-2894"}],"subject":[],"published":{"date-parts":[[2025,2,11]]},"assertion":[{"value":"7 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 January 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 February 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This research was funded by the EU ECSEL Joint Undertaking under grant agreement no.\u00a0826452 (project Arrowhead Tools).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Funding"}},{"value":"The authors declare that they have no Conflict of interest.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}