{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T01:39:17Z","timestamp":1783561157951,"version":"3.55.0"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T00:00:00Z","timestamp":1750723200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T00:00:00Z","timestamp":1750723200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000057","name":"National Institute of General Medical Sciences","doi-asserted-by":"publisher","award":["1R35GM146894"],"award-info":[{"award-number":["1R35GM146894"]}],"id":[{"id":"10.13039\/100000057","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Sci Comput"],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Neural network-based approaches have recently shown significant promise in solving partial differential equations (PDEs) in science and engineering, especially in scenarios featuring complex domains or incorporation of empirical data. One advantage of the neural network methods for PDEs lies in its automatic differentiation (AD), which necessitates only the sample points themselves, unlike traditional finite difference (FD) approximations that require nearby local points to compute derivatives. In this paper, we quantitatively demonstrate the advantage of AD in training neural networks. The concept of truncated entropy is introduced to characterize the training property. Specifically, through comprehensive experimental and theoretical analyses conducted on random feature models and two-layer neural networks, we discover that the defined truncated entropy serves as a reliable metric for quantifying the residual loss of random feature models and the training speed of neural networks for both AD and FD methods. Our experimental and theoretical analyses demonstrate that, from a training perspective, AD outperforms FD in solving PDEs.<\/jats:p>","DOI":"10.1007\/s10915-025-02965-3","type":"journal-article","created":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T08:29:50Z","timestamp":1750753790000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations"],"prefix":"10.1007","volume":"104","author":[{"given":"Chuqi","family":"Chen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yahong","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7279-3711","authenticated-orcid":false,"given":"Yang","family":"Xiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenrui","family":"Hao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,6,24]]},"reference":[{"key":"2965_CR1","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/j.jcp.2018.10.045","volume":"378","author":"M Raissi","year":"2019","unstructured":"Raissi, M., Perdikaris, P., Karniadakis, G.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686\u2013707 (2019)","journal-title":"J. Comput. Phys."},{"key":"2965_CR2","doi-asserted-by":"crossref","unstructured":"Yu, B., W, E.: The deep ritz method: a deep learning-based numerical algorithm for solving variational problems, Comm. Math. Stat. 6(1), 1\u201312 (2018)","DOI":"10.1007\/s40304-018-0127-z"},{"key":"2965_CR3","doi-asserted-by":"crossref","unstructured":"Han, J., Jentzen, A., W,\u00a0E.: Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences 115(34), 8505\u20138510 (2018)","DOI":"10.1073\/pnas.1718942115"},{"key":"2965_CR4","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2023.112084","volume":"484","author":"J Siegel","year":"2023","unstructured":"Siegel, J., Hong, Q., Jin, X., Hao, W., Xu, J.: Greedy training algorithms for neural networks and applications to pdes. J. Comput. Phys. 484, 112084 (2023)","journal-title":"J. Comput. Phys."},{"key":"2965_CR5","unstructured":"Hong, Q., Siegel, J., Xu, J.: A priori analysis of stable neural network solutions to numerical pdes, arXiv preprint arXiv:2104.02903 (2021)"},{"key":"2965_CR6","first-page":"24924","volume":"34","author":"S Cao","year":"2021","unstructured":"Cao, S.: Choose a transformer: fourier or galerkin. Adv. Neural. Inf. Process. Syst. 34, 24924\u201324940 (2021)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"2965_CR7","doi-asserted-by":"crossref","unstructured":"Chen, J., Chi, X., Yang, Z., E, W.: Bridging traditional and machine learning-based algorithms for solving pdes: the random feature method, J Mach Learn 1, 268\u201398 (2022)","DOI":"10.4208\/jml.220726"},{"key":"2965_CR8","doi-asserted-by":"publisher","unstructured":"Lan, Y., Li, Z., Sun, J., Xiang, Y.: as a non-black-box pde solver: When deep learning meets operator splitting. J. Comput. Phys. 491, 112343 (2023). https:\/\/doi.org\/10.1016\/j.jcp.2023.112343. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0021999123004382","DOI":"10.1016\/j.jcp.2023.112343"},{"key":"2965_CR9","unstructured":"Chen, C., Yang, Y., Xiang, Y., Hao, W.: Learn sharp interface solution by homotopy dynamics, arXiv preprint arXiv:2502.00488 (2025)"},{"key":"2965_CR10","first-page":"120832","volume":"37","author":"W Hao","year":"2025","unstructured":"Hao, W., Liu, X., Yang, Y.: Newton informed neural operator for solving nonlinear partial differential equations. Adv. Neural. Inf. Process. Syst. 37, 120832\u2013120860 (2025)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"2965_CR11","unstructured":"Hao, W., Li, R., Xi, Y., Xu, T., Yang, Y.: Multiscale neural networks for approximating green\u2019s functions, arXiv preprint arXiv:2410.18439 (2024)"},{"issue":"6","key":"2965_CR12","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1007\/s10444-022-09985-9","volume":"48","author":"T De Ryck","year":"2022","unstructured":"De Ryck, T., Mishra, S.: Error analysis for physics-informed neural networks (pinns) approximating kolmogorov pdes. Advances in Computational Mathematics 48(6), 79 (2022)","journal-title":"Advances in Computational Mathematics"},{"key":"2965_CR13","unstructured":"Yang, Y., He, J.: Deeper or wider: A perspective from optimal generalization error with sobolev loss, International Conference on Machine Learning (2024)"},{"issue":"5","key":"2965_CR14","doi-asserted-by":"publisher","first-page":"5465","DOI":"10.1137\/20M134695X","volume":"53","author":"J Lu","year":"2021","unstructured":"Lu, J., Shen, Z., Yang, H., Zhang, S.: Deep network approximation for smooth functions. SIAM J. Math. Anal. 53(5), 5465\u20135506 (2021)","journal-title":"SIAM J. Math. Anal."},{"issue":"153","key":"2965_CR15","first-page":"1","volume":"18","author":"A Baydin","year":"2018","unstructured":"Baydin, A., Pearlmutter, B., Radul, A., Siskind, J.: Automatic differentiation in machine learning: a survey. J. Mach. Learn. Res. 18(153), 1\u201343 (2018)","journal-title":"J. Mach. Learn. Res."},{"key":"2965_CR16","doi-asserted-by":"publisher","unstructured":"Chiu, P.-H., Wong, J.C., Ooi, C., Dao, M.H., Ong, Y.-S.: Can-pinn: A fast physics-informed neural network based on coupled-automaticnumerical differentiation method. Comput. Methods Appl. Mech. Eng. 395, 114909 (2022). https:\/\/doi.org\/10.1016\/j.cma.2022.114909. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0045782522001906","DOI":"10.1016\/j.cma.2022.114909"},{"key":"2965_CR17","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-71584-9","volume-title":"Numerical treatment of partial differential equations","author":"C Grossmann","year":"2007","unstructured":"Grossmann, C.: Numerical treatment of partial differential equations. Springer, Berlin (2007)"},{"key":"2965_CR18","doi-asserted-by":"publisher","unstructured":"Patel, R.G., Manickam, I., Trask, N.A., Wood, M.A., Lee, M., Tomas, I., Cyr, E.C.: consistent physics-informed neural networks for hyperbolic systems. J. Comput. Phys. 449, 110754 (2022). https:\/\/doi.org\/10.1016\/j.jcp.2021.110754. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0021999121006495","DOI":"10.1016\/j.jcp.2021.110754"},{"key":"2965_CR19","doi-asserted-by":"crossref","unstructured":"Lim, K.\u00a0L., Dutta, R., Rotaru, M.: Physics informed neural network using finite difference method, in: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, 2022, pp. 1828\u20131833","DOI":"10.1109\/SMC53654.2022.9945171"},{"key":"2965_CR20","unstructured":"Praditia, T., Karlbauer, M., Otte, S., Oladyshkin, S., Butz, M.\u00a0V., Nowak, W.: Finite volume neural network: Modeling subsurface contaminant transport, arXiv preprint arXiv:2104.06010 (2021)"},{"key":"2965_CR21","unstructured":"Wandel, N., Weinmann, M., Klein, R.: Learning incompressible fluid dynamics from scratch\u2013towards fast, differentiable fluid models that generalize, arXiv preprint arXiv:2006.08762 (2020)"},{"key":"2965_CR22","doi-asserted-by":"crossref","unstructured":"Wandel, N., Weinmann, M., Klein, R.: Teaching the incompressible navier\u2013stokes equations to fast neural surrogate models in three dimensions, Physics of Fluids 33\u00a0(4) (2021)","DOI":"10.1063\/5.0047428"},{"key":"2965_CR23","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2020.110079","volume":"428","author":"H Gao","year":"2021","unstructured":"Gao, H., Sun, L., Wang, J.-X.: Phygeonet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state pdes on irregular domain. J. Comput. Phys. 428, 110079 (2021)","journal-title":"J. Comput. Phys."},{"key":"2965_CR24","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2022.114909","volume":"395","author":"P-H Chiu","year":"2022","unstructured":"Chiu, P.-H., Wong, J.C., Ooi, C., Dao, M.H., Ong, Y.-S.: Can-pinn: A fast physics-informed neural network based on coupled-automatic-numerical differentiation method. Comput. Methods Appl. Mech. Eng. 395, 114909 (2022)","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"2965_CR25","unstructured":"Roy, O., Vetterli, M., The effective rank: A measure of effective dimensionality, in: 2007 15th European signal processing conference, IEEE, pp. 606\u2013610 (2007)"},{"key":"2965_CR26","volume-title":"Understanding molecular simulation: from algorithms to applications","author":"D Frenkel","year":"2023","unstructured":"Frenkel, D., Smit, B.: Understanding molecular simulation: from algorithms to applications. Elsevier, Amsterdam (2023)"},{"issue":"3","key":"2965_CR27","doi-asserted-by":"publisher","first-page":"930","DOI":"10.1109\/18.256500","volume":"39","author":"A Barron","year":"1993","unstructured":"Barron, A.: Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inf. Theory 39(3), 930\u2013945 (1993)","journal-title":"IEEE Trans. Inf. Theory"},{"issue":"1","key":"2965_CR28","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1162\/neco.1996.8.1.164","volume":"8","author":"H Mhaskar","year":"1996","unstructured":"Mhaskar, H.: Neural networks for optimal approximation of smooth and analytic functions. Neural Comput. 8(1), 164\u2013177 (1996)","journal-title":"Neural Comput."},{"key":"2965_CR29","unstructured":"Lu, J., Lu, Y., Wang, M.: A priori generalization analysis of the Deep Ritz method for solving high dimensional elliptic partial differential equations, in: Conference on Learning Theory, PMLR, pp. 3196\u20133241 (2021)"},{"key":"2965_CR30","first-page":"21721","volume":"36","author":"Y Yang","year":"2024","unstructured":"Yang, Y., Yang, H., Xiang, Y.: Nearly optimal vc-dimension and pseudo-dimension bounds for deep neural network derivatives. Advances in Neural Information Processing Systems 36, 21721 (2024)","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2965_CR31","doi-asserted-by":"crossref","unstructured":"Rahimi, A., Recht, B., Uniform approximation of functions with random bases, in: 2008 46th annual allerton conference on communication, control, and computing, IEEE, pp. 555\u2013561 (2008)","DOI":"10.1109\/ALLERTON.2008.4797607"},{"key":"2965_CR32","unstructured":"Rahimi, A., Recht, B.: Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning, Advances in neural information processing systems 21 (2008)"},{"key":"2965_CR33","doi-asserted-by":"crossref","unstructured":"Morton, K.\u00a0W., Mayers, D.\u00a0F.: Numerical solution of partial differential equations: an introduction, Cambridge university press, (2005)","DOI":"10.1017\/CBO9780511812248"},{"key":"2965_CR34","unstructured":"Ruder, S.: An overview of gradient descent optimization algorithms, arXiv preprint arXiv:1609.04747 (2016)"},{"key":"2965_CR35","unstructured":"Zinkevich, M., Weimer, M., Li, L., Smola, A.: Parallelized stochastic gradient descent, Advances in neural information processing systems 23 (2010)"},{"key":"2965_CR36","unstructured":"Kingma, D., Ba, J.: Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014)"},{"key":"2965_CR37","doi-asserted-by":"crossref","unstructured":"Hao, W., Hong, Q., Jin, X., Wang, Y.: Gauss newton method for solving variational problems of pdes with neural network discretizaitons, arXiv preprint arXiv:2306.08727 (2023)","DOI":"10.1007\/s10915-024-02535-z"},{"issue":"1","key":"2965_CR38","doi-asserted-by":"publisher","first-page":"712","DOI":"10.1137\/23M1576360","volume":"45","author":"H He","year":"2024","unstructured":"He, H., Tang, Z., Zhao, S., Saad, Y., Xi, Y.: nltgcr: A class of nonlinear acceleration procedures based on conjugate residuals. SIAM J. Matrix Anal. Appl. 45(1), 712\u2013743 (2024)","journal-title":"SIAM J. Matrix Anal. Appl."},{"key":"2965_CR39","unstructured":"Yang, Y., Chen, Q., Hao, W.: Homotopy relaxation training algorithms for infinite-width two-layer relu neural networks, arXiv preprint arXiv:2309.15244 (2023)"}],"updated-by":[{"DOI":"10.1007\/s10915-025-03098-3","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2025,11,14]],"date-time":"2025-11-14T00:00:00Z","timestamp":1763078400000}}],"container-title":["Journal of Scientific Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10915-025-02965-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10915-025-02965-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10915-025-02965-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T06:47:32Z","timestamp":1779259652000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10915-025-02965-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,24]]},"references-count":39,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["2965"],"URL":"https:\/\/doi.org\/10.1007\/s10915-025-02965-3","relation":{},"ISSN":["0885-7474","1573-7691"],"issn-type":[{"value":"0885-7474","type":"print"},{"value":"1573-7691","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,24]]},"assertion":[{"value":"25 September 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 March 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 June 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 June 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 January 2026","order":6,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":7,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This article was originally published under the subscription model but it is now published under an Open Access license. Affiliation information for first and second author have been corrected","order":8,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 November 2025","order":9,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":10,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":11,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1007\/s10915-025-03098-3","URL":"https:\/\/doi.org\/10.1007\/s10915-025-03098-3","order":12,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"54"}}