{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T17:39:18Z","timestamp":1783445958813,"version":"3.55.0"},"reference-count":62,"publisher":"MIT Press - Journals","issue":"4","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,3,26]]},"abstract":"<jats:p>A new network with super-approximation power is introduced. This network is built with Floor (\u230ax\u230b) or ReLU (max{0,x}) activation function in each neuron; hence, we call such networks Floor-ReLU networks. For any hyperparameters N\u2208N+ and L\u2208N+, we show that Floor-ReLU networks with width max{d,5N+13} and depth 64dL+3 can uniformly approximate a H\u00f6lder function f on [0,1]d with an approximation error 3\u03bbd\u03b1\/2N-\u03b1L, where \u03b1\u2208(0,1] and \u03bb are the H\u00f6lder order and constant, respectively. More generally for an arbitrary continuous function f on [0,1]d with a modulus of continuity \u03c9f(\u00b7), the constructive approximation rate is \u03c9f(dN-L)+2\u03c9f(d)N-L. As a consequence, this new class of networks overcomes the curse of dimensionality in approximation power when the variation of \u03c9f(r) as r\u21920 is moderate (e.g., \u03c9f(r)\u2272r\u03b1 for H\u00f6lder continuous functions), since the major term to be considered in our approximation rate is essentially d times a function of N and L independent of d within the modulus of continuity.<\/jats:p>","DOI":"10.1162\/neco_a_01364","type":"journal-article","created":{"date-parts":[[2021,1,29]],"date-time":"2021-01-29T23:13:43Z","timestamp":1611962023000},"page":"1005-1036","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":42,"title":["Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth"],"prefix":"10.1162","volume":"33","author":[{"given":"Zuowei","family":"Shen","sequence":"first","affiliation":[{"name":"Department of Mathematics, Purdue University, West Lafayette, IN 47907, USA, haizhao@purdue.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haizhao","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Purdue University, West Lafayette, IN 47907, USA, haizhao@purdue.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shijun","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Purdue University, West Lafayette, IN 47907, USA, zhangshijun@u.nus.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,3,26]]},"reference":[{"key":"2021041320044133400_B1","author":"Allen-Zhu","year":"2019","journal-title":"Learning and generalization in overparameterized neural networks, going beyond two layers"},{"issue":"5","key":"2021041320044133400_B2","first-page":"679","article-title":"On functions of three variables","volume":"114","author":"Arnold","year":"1957","journal-title":"Dokl. Akad. Nauk SSSR"},{"key":"2021041320044133400_B3","article-title":"Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks","author":"Arora","year":"2019","journal-title":"Proceedings of the ICML"},{"key":"2021041320044133400_B4","author":"Bao","year":"2019","journal-title":"Approximation analysis of convolutional neural networks"},{"issue":"3","key":"2021041320044133400_B5","doi-asserted-by":"crossref","first-page":"930","DOI":"10.1109\/18.256500","article-title":"Universal approximation bounds for superpositions of a sigmoidal function","volume":"39","author":"Barron","year":"1993","journal-title":"IEEE Transactions on Information Theory"},{"key":"2021041320044133400_B6","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1162\/089976698300017016","article-title":"Almost linear VC-dimension bounds for piecewise polynomial networks","volume":"10","author":"Bartlett","year":"1998","journal-title":"Neural Computation"},{"key":"2021041320044133400_B7","author":"Bengio","year":"2013","journal-title":"Estimating or propagating gradients through stochastic neurons for conditional computation."},{"key":"2021041320044133400_B8","author":"Berner","year":"2018","journal-title":"Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations."},{"issue":"1","key":"2021041320044133400_B9","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1137\/18M118709X","article-title":"Optimal approximation with sparsely connected deep neural networks","volume":"1","author":"B\u00f6lcskei","year":"2019","journal-title":"SIAM Journal on Mathematics of Data Science"},{"key":"2021041320044133400_B10","author":"Boo","year":"2020","journal-title":"Quantized neural networks: Characterization and holistic optimization"},{"key":"2021041320044133400_B11","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1007\/s00365-009-9054-2","article-title":"On a constructive proof of Kolmogorov's superposition theorem","volume":"30","author":"Braun","year":"2009","journal-title":"Constructive Approximation"},{"key":"2021041320044133400_B12","author":"Cao","year":"2019","journal-title":"Generalization bounds of stochastic gradient descent for wide and deep neural networks."},{"key":"2021041320044133400_B13","author":"Carrillo","year":"2019","journal-title":"A consensus-based global optimization method for high dimensional machine learning problems"},{"issue":"9","key":"2021041320044133400_B14","doi-asserted-by":"crossref","first-page":"3400","DOI":"10.1002\/mma.5575","article-title":"A note on the expressive power of deep rectified linear unit networks in high-dimensional spaces","volume":"42","author":"Chen","year":"2019","journal-title":"Mathematical Methods in the Applied Sciences"},{"key":"2021041320044133400_B15","first-page":"8174","volume-title":"Advances in neural information processing systems, 32","author":"Chen","year":"2019"},{"key":"2021041320044133400_B16","author":"Chen","year":"2019","journal-title":"How much over-parameterization is sufficient to learn deep ReLU networks?"},{"key":"2021041320044133400_B17","doi-asserted-by":"crossref","DOI":"10.3389\/fams.2018.00014","article-title":"Construction of neural networks for realization of localized deep learning","volume":"4","author":"Chui","year":"2018","journal-title":"Frontiers in Applied Mathematics and Statistics"},{"key":"2021041320044133400_B18","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/BF02551274","article-title":"Approximation by superpositions of a sigmoidal function","volume":"2","author":"Cybenko","year":"1989","journal-title":"Mathematics of Control, Signals, and Systems"},{"issue":"4","key":"2021041320044133400_B19","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1007\/BF01171759","article-title":"Optimal nonlinear approximation","volume":"63","author":"Devore","year":"1989","journal-title":"Manuskripta Math"},{"key":"2021041320044133400_B20","author":"Gribonval","year":"2019","journal-title":"Approximation spaces of deep neural networks."},{"key":"2021041320044133400_B21","author":"hring","year":"2019"},{"key":"2021041320044133400_B22","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1016\/j.neucom.2018.07.075","article-title":"Approximation capability of two hidden layer feedforward neural networks with fixed weights","volume":"316","author":"Guliyev","year":"2018","journal-title":"Neurocomputing"},{"key":"2021041320044133400_B23","first-page":"1064","volume-title":"Proceedings of Machine Learning Research","author":"Harvey","year":"2017"},{"issue":"1","key":"2021041320044133400_B24","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1038\/scientificamerican0792-66","article-title":"Genetic algorithms","volume":"267","author":"Holland","year":"1992","journal-title":"Scientific American"},{"issue":"5","key":"2021041320044133400_B25","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/0893-6080(89)90020-8","article-title":"Multilayer feedforward networks are universal approximators","volume":"2","author":"Hornik","year":"1989","journal-title":"Neural Networks"},{"issue":"1","key":"2021041320044133400_B26","first-page":"6869","article-title":"Quantized neural networks: Training neural networks with low precision weights and activations","volume":"18","author":"Hubara","year":"2017","journal-title":"J. Mach. Learn. Res."},{"issue":"4","key":"2021041320044133400_B27","doi-asserted-by":"crossref","first-page":"725","DOI":"10.1109\/TNN.2003.813830","article-title":"Kolmogorov's spline network","volume":"14","author":"Igelnik","year":"2003","journal-title":"IEEE Transactions on Neural Networks"},{"key":"2021041320044133400_B28","author":"Jacot","year":"2018","journal-title":"Neural tangent kernel: Convergence and generalization in neural networks."},{"key":"2021041320044133400_B29","author":"Ji","year":"2020","journal-title":"Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks"},{"key":"2021041320044133400_B30","doi-asserted-by":"crossref","first-page":"1942","DOI":"10.1109\/ICNN.1995.488968","author":"Kennedy","year":"1995","journal-title":"Proceedings of the International Conference on Neural Networks"},{"issue":"4598","key":"2021041320044133400_B31","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1126\/science.220.4598.671","article-title":"Optimization by simulated annealing","volume":"220","author":"Kirkpatrick","year":"1983","journal-title":"Science"},{"key":"2021041320044133400_B32","first-page":"179","article-title":"On the representation of continuous functions of several variables by superposition of continuous functions of a smaller number of variables","volume":"108","author":"Kolmogorov","year":"1956","journal-title":"Dokl. Akad. Nauk SSSR"},{"key":"2021041320044133400_B33","first-page":"953","article-title":"On the representation of continuous functions of several variables by superposition of continuous functions of one variable and addition","volume":"114","author":"Kolmogorov","year":"1957","journal-title":"Dokl. Akad. Nauk SSSR"},{"issue":"3","key":"2021041320044133400_B34","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1016\/0893-6080(92)90012-8","article-title":"Kolmogorov's theorem and multilayer neural networks","volume":"5","author":"K\u016frkov\u00e1","year":"1992","journal-title":"Neural Networks"},{"key":"2021041320044133400_B35","author":"Li","year":"2019","journal-title":"Deep learning via dynamical systems: An approximation perspective."},{"key":"2021041320044133400_B36","author":"Liang","year":"2016","journal-title":"Why deep neural networks?"},{"key":"2021041320044133400_B37","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1109\/ICDMW.2019.00063","article-title":"Optimization strategies in quantized neural networks: A review","author":"Lin","year":"2019","journal-title":"Proceedings of the 2019 International Conference on Data Mining Workshops"},{"key":"2021041320044133400_B38","author":"Lu","year":"2020","journal-title":"Deep network approximation for smooth functions"},{"key":"2021041320044133400_B39","author":"Luo","year":"2020","journal-title":"Two-layer neural networks for partial differential equations: Optimization and generalization theory"},{"issue":"1","key":"2021041320044133400_B40","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/S0925-2312(98)00111-8","article-title":"Lower bounds for approximation by MLP neural networks","volume":"25","author":"Maiorov","year":"1999","journal-title":"Neurocomputing"},{"issue":"1","key":"2021041320044133400_B41","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1137\/18M1189336","article-title":"New error bounds for deep ReLU networks using sparse grids","volume":"1","author":"Montanelli","year":"2019","journal-title":"SIAM Journal on Mathematics of Data Science"},{"key":"2021041320044133400_B42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neunet.2019.12.013","article-title":"Error bounds for deep ReLU networks using the Kolmogorov-Arnold superposition theorem","volume":"129","author":"Montanelli","year":"2020","journal-title":"Neural Networks"},{"key":"2021041320044133400_B43","article-title":"Deep ReLU networks overcome the curse of dimensionality for bandlimited functions","author":"Montanelli","journal-title":"Journal of Computational Mathematics"},{"key":"2021041320044133400_B44","author":"Nakada","year":"2019","journal-title":"Adaptive approximation and estimation of deep neural network with intrinsic dimensionality"},{"key":"2021041320044133400_B45","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1093\/comjnl\/7.4.308","article-title":"A simplex method for function minimization","volume":"7","author":"Nelder","year":"1965","journal-title":"Comput. J."},{"key":"2021041320044133400_B46","author":"Opschoor","year":"2019","journal-title":"Exponential ReLU DNN expression of holomorphic maps in high dimension"},{"key":"2021041320044133400_B47","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1016\/j.neunet.2018.08.019","article-title":"Optimal approximation of piecewise smooth functions using deep ReLU neural networks","volume":"108","author":"Petersen","year":"2018","journal-title":"Neural Networks"},{"issue":"1","key":"2021041320044133400_B48","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1142\/S0218202517400061","article-title":"A consensus-based model for global optimization and its mean-field limit","volume":"27","author":"Pinnau","year":"2017","journal-title":"Mathematical Models and Methods in Applied Sciences"},{"key":"2021041320044133400_B49","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1007\/s11633-017-1054-2","article-title":"Why and when can deep\u2014but not shallow\u2014networks avoid the curse of dimensionality: A review","volume":"14","author":"Poggio","year":"2017","journal-title":"International Journal of Automation and Computing"},{"key":"2021041320044133400_B50","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1016\/j.neunet.2019.07.011","article-title":"Nonlinear approximation via compositions","volume":"119","author":"Shen","year":"2019","journal-title":"Neural Networks"},{"issue":"5","key":"2021041320044133400_B51","doi-asserted-by":"crossref","first-page":"1768","DOI":"10.4208\/cicp.OA-2020-0149","article-title":"Deep network approximation characterized by number of neurons","volume":"28","author":"Shen","year":"2020","journal-title":"Communications in Computational Physics"},{"key":"2021041320044133400_B52","article-title":"Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: Optimal rate and curse of dimensionality","author":"Suzuki","year":"2019","journal-title":"Proceedings of the International Conference on Learning Representations"},{"key":"2021041320044133400_B53","doi-asserted-by":"crossref","first-page":"4376","DOI":"10.1109\/CVPR.2018.00460","article-title":"Two-step quantization for low-bit neural networks","author":"Wang","year":"2018","journal-title":"Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition"},{"issue":"5","key":"2021041320044133400_B54","doi-asserted-by":"crossref","first-page":"1407","DOI":"10.4310\/CMS.2019.v17.n5.a11","article-title":"A priori estimates of the population risk for two-layer neural networks","volume":"17","author":"Weinan","year":"2019","journal-title":"Communications in Mathematical Sciences"},{"key":"2021041320044133400_B55","author":"Weinan","year":"2018","journal-title":"Exponential convergence of the deep neural network approximation for analytic functions."},{"key":"2021041320044133400_B56","author":"Weinan","year":"2020","journal-title":"Representation formulas and pointwise properties for Barron functions"},{"key":"2021041320044133400_B57","author":"Yang","year":"2020","journal-title":"Approximation in shift-invariant spaces with deep ReLU neural networks."},{"key":"2021041320044133400_B58","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.neunet.2017.07.002","article-title":"Error bounds for approximations with deep ReLU networks","volume":"94","author":"Yarotsky","year":"2017","journal-title":"Neural Networks"},{"key":"2021041320044133400_B59","first-page":"639","article-title":"Optimal approximation of continuous functions by very deep ReLU networks","author":"Yarotsky","year":"2018","journal-title":"Proceedings of Machine Learning Research"},{"key":"2021041320044133400_B60","author":"Yarotsky","year":"2019","journal-title":"The phase diagram of approximation rates for deep neural networks."},{"key":"2021041320044133400_B61","author":"Yin","year":"2019","journal-title":"Understanding straight-through estimator in training activation quantized neural nets."},{"issue":"2","key":"2021041320044133400_B62","doi-asserted-by":"crossref","first-page":"787","DOI":"10.1016\/j.acha.2019.06.004","article-title":"Universality of deep convolutional neural networks","volume":"48","author":"Zhou","year":"2020","journal-title":"Applied and Computational Harmonic Analysis"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/neco\/article-pdf\/33\/4\/1005\/1902284\/neco_a_01364.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/neco\/article-pdf\/33\/4\/1005\/1902284\/neco_a_01364.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,4,15]],"date-time":"2021-04-15T00:16:48Z","timestamp":1618445808000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/33\/4\/1005\/97476\/Deep-Network-With-Approximation-Error-Being"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":62,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,3,26]]},"published-print":{"date-parts":[[2021,3,26]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01364","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}