{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T19:29:19Z","timestamp":1783106959194,"version":"3.54.6"},"reference-count":68,"publisher":"IOP Publishing","issue":"1","license":[{"start":{"date-parts":[[2020,12,1]],"date-time":"2020-12-01T00:00:00Z","timestamp":1606780800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,12,1]],"date-time":"2020-12-01T00:00:00Z","timestamp":1606780800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100000121","name":"Division of Mathematical Sciences","doi-asserted-by":"crossref","award":["1818772"],"award-info":[{"award-number":["1818772"]}],"id":[{"id":"10.13039\/100000121","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"crossref","award":["FA9550-19-1-0036"],"award-info":[{"award-number":["FA9550-19-1-0036"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2021,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper introduces a novel algorithmic framework for a deep neural network (DNN), which in a mathematically rigorous manner, allows us to incorporate history (or memory) into the network\u2014it ensures all layers are connected to one another. This DNN, called Fractional-DNN, can be viewed as a time-discretization of a fractional in time non-linear ordinary differential equation (ODE). The learning problem then is a minimization problem subject to that fractional ODE as constraints. We emphasize that an analogy between the existing DNN and ODEs, with standard time derivative, is well-known by now. The focus of our work is the Fractional-DNN. Using the Lagrangian approach, we provide a derivation of the backward propagation and the design equations. We test our network on several datasets for classification problems. Fractional-DNN offers various advantages over the existing DNN. The key benefits are a significant improvement to the vanishing gradient issue due to the memory effect, and better handling of nonsmooth data due to the network\u2019s ability to approximate non-smooth functions.<\/jats:p>","DOI":"10.1088\/2632-2153\/aba8e7","type":"journal-article","created":{"date-parts":[[2020,7,23]],"date-time":"2020-07-23T22:31:28Z","timestamp":1595543488000},"page":"015003","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Fractional deep neural network via constrained optimization"],"prefix":"10.1088","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6641-1449","authenticated-orcid":false,"given":"Harbir","family":"Antil","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0931-4025","authenticated-orcid":false,"given":"Ratna","family":"Khatri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rainald","family":"L\u00f6hner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deepanshu","family":"Verma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2020,12,1]]},"reference":[{"key":"mlstaba8e7bib1","doi-asserted-by":"publisher","first-page":"pp 770","DOI":"10.1109\/CVPR.2016.90","article-title":"Deep residual learning for image recognition","author":"He","year":"2016"},{"key":"mlstaba8e7bib2","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6420\/ab80d7","article-title":"Bilevel optimization, deep learning and fractional Laplacian regularization with applications in tomography","volume":"36","author":"Antil","year":"2020","journal-title":"Inverse Probl."},{"key":"mlstaba8e7bib3","doi-asserted-by":"crossref","first-page":"10437","DOI":"10.1007\/s11042-017-4440-4","article-title":"Deep residual learning for image steganalysis","volume":"77","author":"Wu","year":"2018","journal-title":"Multimed. Tools. Appl."},{"key":"mlstaba8e7bib4","doi-asserted-by":"publisher","first-page":"4509","DOI":"10.1109\/TIP.2017.2713099","article-title":"Deep convolutional neural network for inverse problems in imaging","volume":"26","author":"Jin","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"mlstaba8e7bib5","doi-asserted-by":"publisher","first-page":"1985","DOI":"10.1109\/TBME.2018.2821699","article-title":"Deep residual learning for accelerated mri using magnitude and phase networks","volume":"65","author":"Lee","year":"2018","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"mlstaba8e7bib6","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1016\/j.neuroimage.2017.04.041","article-title":"Voxresnet: Deep voxelwise residual networks for brain segmentation from 3d mr images","volume":"170","author":"Chen","year":"2018","journal-title":"Neuroimage"},{"key":"mlstaba8e7bib7","doi-asserted-by":"publisher","first-page":"3055","DOI":"10.1002\/mrm.26977","article-title":"Learning a variational network for reconstruction of accelerated mri data","volume":"79","author":"Hammernik","year":"2018","journal-title":"Magn. Reson. Med."},{"key":"mlstaba8e7bib8","doi-asserted-by":"publisher","first-page":"pp 2790","DOI":"10.1109\/CVPR.2017.298","article-title":"Image super-resolution via deep recursive residual network","author":"Tai","year":"2017"},{"key":"mlstaba8e7bib9","doi-asserted-by":"publisher","first-page":"4274","DOI":"10.1109\/TGRS.2018.2810208","article-title":"Missing data reconstruction in remote sensing image with a unified spatial\u2013temporal\u2013spectral deep convolutional neural network","volume":"56","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"mlstaba8e7bib10","article-title":"Detection of flooding events in social multimedia and satellite imagery using deep neural networks","author":"Bischke","year":"2017"},{"key":"mlstaba8e7bib11","first-page":"pp 249","article-title":"Understanding the difficulty of training deep feedforward neural networks","volume":"vol 9","author":"Glorot","year":"2010"},{"key":"mlstaba8e7bib12","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1186\/s13634-016-0355-x","article-title":"A survey of machine learning for big data processing","volume":"2016","author":"Qiu","year":"2016","journal-title":"EURASIP J. Adv. Signal Process."},{"key":"mlstaba8e7bib13","author":"Wigderson","year":"2019"},{"key":"mlstaba8e7bib14","doi-asserted-by":"publisher","first-page":"352","DOI":"10.1007\/s10851-019-00903-1","article-title":"Deep neural networks motivated by partial differential equations","volume":"62","author":"Ruthotto","year":"2019","journal-title":"J. Math. Imaging Vis."},{"key":"mlstaba8e7bib15","first-page":"1813","article-title":"Machine learning: Mathematical theory and scientific applications","volume":"66","author":"Weinan","year":"2019","journal-title":"Not. Am. Math. Soc."},{"key":"mlstaba8e7bib16","article-title":"Modelling the influence of data structure on learning in neural networks","author":"Goldt","year":"2019"},{"key":"mlstaba8e7bib17","article-title":"Deep learning by scattering","author":"Mallat","year":"2013"},{"key":"mlstaba8e7bib18","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1109\/72.279181","article-title":"Learning long-term dependencies with gradient descent is difficult","volume":"5","author":"Bengio","year":"1994-03","journal-title":"IEEE Trans. Neural Netw."},{"key":"mlstaba8e7bib19","first-page":"550","article-title":"Residual networks behave like ensembles of relatively shallow networks","author":"Veit","year":"2016"},{"key":"mlstaba8e7bib20","article-title":"Multi-level residual networks from dynamical systems view","author":"Chang","year":"2017"},{"key":"mlstaba8e7bib21","doi-asserted-by":"publisher","first-page":"pp 2261","DOI":"10.1109\/CVPR.2017.243","article-title":"Densely connected convolutional networks","author":"Huang","year":"2017"},{"key":"mlstaba8e7bib22","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2018.00262","article-title":"Residual dense network for image super-resolution","author":"Zhang","year":"2018"},{"key":"mlstaba8e7bib23","article-title":"Training Very Deep Networks","author":"Srivastava","year":"2015"},{"key":"mlstaba8e7bib24","first-page":"pp 874","article-title":"Adanet: Adaptive structural learning of artificial neural networks","author":"Cortes","year":"2016"},{"key":"mlstaba8e7bib25","doi-asserted-by":"publisher","first-page":"3943","DOI":"10.1109\/TSG.2018.2844307","article-title":"Short-term load forecasting with deep residual networks","volume":"10","author":"Chen","year":"2019","journal-title":"IEEE Trans. Smart Grid"},{"key":"mlstaba8e7bib26","first-page":"869","article-title":"Deep neural networks learn non-smooth functions effectively","volume":"vol 89","author":"Imaizumi","year":"2019"},{"key":"mlstaba8e7bib27","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6420\/aa9a90","article-title":"Stable architectures for deep neural networks","volume":"34","author":"Haber","year":"2017","journal-title":"Inverse Probl."},{"key":"mlstaba8e7bib28","first-page":"3276","article-title":"Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations","volume":"80","author":"Lu","year":"2017"},{"key":"mlstaba8e7bib29","article-title":"Research data supporting \u2018deep learning as optimal control problems\u2019","author":"Sch\u00f6enlieb","year":"2019"},{"key":"mlstaba8e7bib30","doi-asserted-by":"publisher","first-page":"171","DOI":"10.3934\/jcd.2019009","article-title":"Deep learning as optimal control problems: Models and numerical methods","volume":"6","author":"Benning","year":"2019","journal-title":"J. Comput. Dyn."},{"key":"mlstaba8e7bib31","doi-asserted-by":"publisher","first-page":"A2603\u2013A2626","DOI":"10.1137\/18M1229845","article-title":"fpinns: Fractional physics-informed neural networks","volume":"41","author":"Pang","year":"2019","journal-title":"SIAM J. Sci. Comput."},{"key":"mlstaba8e7bib32","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/j.jcp.2018.10.045","article-title":"Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations","volume":"378","author":"Raissi","year":"2019","journal-title":"J. Comput. Phys."},{"key":"mlstaba8e7bib33","doi-asserted-by":"publisher","first-page":"A2485\u2013A2509","DOI":"10.1137\/18M1204991","article-title":"Machine learning of space-fractional differential equations","volume":"41","author":"Gulian","year":"2019","journal-title":"SIAM J. Sci. Comput."},{"key":"mlstaba8e7bib34","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1016\/j.chaos.2017.06.030","article-title":"Solving fractional differential equations of variable-order involving operators with mittag-leffler kernel using artificial neural networks","volume":"103","author":"Z\u00fa\u00f1iga-Aguilar","year":"2017","journal-title":"Chaos Solitons Fractals"},{"key":"mlstaba8e7bib35","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1140\/epjp\/i2018-11917-0","article-title":"New numerical approximation for solving fractional delay differential equations of variable order using artificial neural networks","volume":"133","author":"Z\u00fa niga-Aguilar","year":"2018","journal-title":"Eur. Phys. J. Plus"},{"key":"mlstaba8e7bib36","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1016\/S0022-5096(99)00029-0","article-title":"Reformulation of elasticity theory for discontinuities and long-range forces","volume":"48","author":"Silling","year":"2000","journal-title":"J. Mech. Phys. Solids"},{"key":"mlstaba8e7bib37","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1137\/09074807X","article-title":"Peridynamics as an upscaling of molecular dynamics","volume":"8","author":"Seleson","year":"2009","journal-title":"Multiscale Model. Simul."},{"key":"mlstaba8e7bib38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/S0370-1573(00)00070-3","article-title":"The random walk\u2019s guide to anomalous diffusion: a fractional dynamics approach","volume":"339","author":"Metzler","year":"2000","journal-title":"Phys. Rep."},{"key":"mlstaba8e7bib39","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6420\/ab1299","article-title":"External optimal control of nonlocal PDEs","volume":"35","author":"Antil","year":"2019","journal-title":"Inverse Probl."},{"key":"mlstaba8e7bib40","doi-asserted-by":"publisher","DOI":"10.1051\/cocv\/2020005","article-title":"External optimal control of fractional parabolic PDEs","volume":"26","author":"Antil","year":"2020","journal-title":"ESAIM Control Optim. Calc. Var."},{"key":"mlstaba8e7bib41","doi-asserted-by":"publisher","first-page":"688","DOI":"10.1016\/j.physa.2018.03.056","article-title":"Non validity of index law in fractional calculus: A fractional differential operator with markovian and non-markovian properties","volume":"505","author":"Atangana","year":"2018","journal-title":"Physica A"},{"key":"mlstaba8e7bib42","first-page":"1242","article-title":"Fractional operators applied to geophysical electromagnetics","volume":"220","author":"Weiss","year":"2020","journal-title":"Geophys. J. Int."},{"key":"mlstaba8e7bib43","doi-asserted-by":"crossref","DOI":"10.2172\/1569144","article-title":"A fast solver for the fractional helmholtz equation","author":"Antil","year":"2019"},{"key":"mlstaba8e7bib44","doi-asserted-by":"publisher","first-page":"661","DOI":"10.1515\/cmam-2017-0039","article-title":"Spectral approximation of fractional PDEs in image processing and phase field modeling","volume":"17","author":"Antil","year":"2017","journal-title":"J. Comput. Methods Appl. Math."},{"key":"mlstaba8e7bib45","doi-asserted-by":"publisher","first-page":"2479","DOI":"10.1137\/18M1224970","article-title":"Sobolev spaces with non-Muckenhoupt weights, fractional elliptic operators and applications","volume":"51","author":"Antil","year":"2019","journal-title":"SIAM J. Math. Anal."},{"key":"mlstaba8e7bib46","doi-asserted-by":"publisher","first-page":"55","DOI":"10.21042\/AMNS.2018.1.00006","article-title":"Analysis of models for viscoelastic wave propagation","volume":"3","author":"Brown","year":"2018","journal-title":"Appl. Math. Nonlinear Sci."},{"key":"mlstaba8e7bib47","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1137\/19M1247620","article-title":"Layer-parallel training of deep residual neural networks","volume":"2","author":"G\u00fcnther","year":"2020","journal-title":"SIAM J. Math. Data Sci."},{"key":"mlstaba8e7bib48","article-title":"Convergence of solutions of discrete semi-linear space-time fractional evolution equations","author":"Antil","year":"2019"},{"key":"mlstaba8e7bib49","doi-asserted-by":"publisher","first-page":"1295","DOI":"10.1137\/15M1014991","article-title":"A space-time fractional optimal control problem: analysis and discretization","volume":"54","author":"Antil","year":"2016","journal-title":"SIAM J. Control Optim."},{"key":"mlstaba8e7bib50","first-page":"pp 36","article-title":"Fractional optimal economic control problem described by the generalized fractional order derivative","author":"Thiao","year":"2020"},{"key":"mlstaba8e7bib51","doi-asserted-by":"publisher","DOI":"10.1063\/1.5112177","article-title":"A new and efficient numerical method for the fractional modeling and optimal control of diabetes and tuberculosis co-existence","volume":"29","author":"Jajarmi","year":"2019","journal-title":"Chaos"},{"key":"mlstaba8e7bib52","first-page":"pp 437","author":"Bengio","year":"2012"},{"key":"mlstaba8e7bib53","doi-asserted-by":"publisher","DOI":"10.1016\/j.chaos.2019.109444","article-title":"Fractional order neural networks for system identification","volume":"130","author":"Z\u00fa niga-Aguilar","year":"2020","journal-title":"Chaos Solitons Fractals"},{"key":"mlstaba8e7bib54","volume":"vol 204","author":"Kilbas","year":"2006"},{"key":"mlstaba8e7bib55","author":"Podlubny","year":"1999"},{"key":"mlstaba8e7bib56","author":"Nocedal","year":"2006"},{"key":"mlstaba8e7bib57","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1145\/321296.321305","article-title":"Iterative procedures for nonlinear integral equations","volume":"12","author":"Anderson","year":"1965","journal-title":"J. ACM"},{"key":"mlstaba8e7bib58","first-page":"pp 2663","author":"Roux","year":"2012"},{"key":"mlstaba8e7bib59","author":"Samko","year":"1993"},{"key":"mlstaba8e7bib60","author":"Scherer","year":"2020"},{"key":"mlstaba8e7bib61","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1016\/j.neunet.2012.02.023","article-title":"Multi-column deep neural network for traffic sign classification","volume":"32","author":"Cire\u015fan","year":"2012","journal-title":"Neural Netw."},{"key":"mlstaba8e7bib62","doi-asserted-by":"publisher","first-page":"332","DOI":"10.1016\/j.cma.2018.12.011","article-title":"Numerical methods for time-fractional evolution equations with nonsmooth data: a concise overview","volume":"346","author":"Jin","year":"2019","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"mlstaba8e7bib63","first-page":"pp 448","article-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","author":"Ioffe","year":"2015"},{"key":"mlstaba8e7bib64","author":"LeVeque","year":"2007"},{"key":"mlstaba8e7bib65","author":"Kelley","year":"1999"},{"key":"mlstaba8e7bib66","author":"Dua","year":"2017"},{"key":"mlstaba8e7bib67","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1016\/j.asoc.2016.05.030","article-title":"Fuzzy c-means based support vector machines classifier for perfume recognition","volume":"46","author":"Esme","year":"2016","journal-title":"Appl. Soft Comput."},{"key":"mlstaba8e7bib68","doi-asserted-by":"publisher","first-page":"1291","DOI":"10.1002\/jcc.24764","article-title":"Deep learning for computational chemistry","volume":"38","author":"Goh","year":"2017","journal-title":"J. Comput. Chem."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/ampdf","content-type":"application\/pdf","content-version":"am","intended-application":"unspecified"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7","content-type":"text\/html","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,10]],"date-time":"2024-08-10T18:11:13Z","timestamp":1723313473000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba8e7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,1]]},"references-count":68,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12,1]]},"published-print":{"date-parts":[[2021,3,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/aba8e7","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,1]]},"assertion":[{"value":"Fractional deep neural network via constrained optimization","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2020 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2020-04-23","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2020-07-23","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2020-12-01","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}