{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T09:14:59Z","timestamp":1775898899856,"version":"3.50.1"},"reference-count":58,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,13]],"date-time":"2023-06-13T00:00:00Z","timestamp":1686614400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100012338","name":"Alan Turing Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100012338","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Appl. Math. Stat."],"abstract":"<jats:p>We propose a novel framework for the regularized inversion of deep neural networks. The framework is based on the authors' recent work on training feed-forward neural networks without the differentiation of activation functions. The framework lifts the parameter space into a higher dimensional space by introducing auxiliary variables, and penalizes these variables with tailored Bregman distances. We propose a family of variational regularizations based on these Bregman distances, present theoretical results and support their practical application with numerical examples. In particular, we present the first convergence result (to the best of our knowledge) for the regularized inversion of a single-layer perceptron that only assumes that the solution of the inverse problem is in the range of the regularization operator, and that shows that the regularized inverse provably converges to the true inverse if measurement errors converge to zero.<\/jats:p>","DOI":"10.3389\/fams.2023.1176850","type":"journal-article","created":{"date-parts":[[2023,6,13]],"date-time":"2023-06-13T10:57:08Z","timestamp":1686653828000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["A lifted Bregman formulation for the inversion of deep neural networks"],"prefix":"10.3389","volume":"9","author":[{"given":"Xiaoyu","family":"Wang","sequence":"first","affiliation":[]},{"given":"Martin","family":"Benning","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,6,13]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"B2","author":"Goodfellow","year":"2016","journal-title":"Deep Learning"},{"key":"B3","first-page":"1","article-title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","author":"Simonyan","year":"2014","journal-title":"Proceedings of the International Conference on Learning Representations"},{"key":"B4","doi-asserted-by":"publisher","first-page":"3429","DOI":"10.1109\/ICCV.2017.371","article-title":"Interpretable explanations of black boxes by meaningful perturbation","author":"Fong","year":"2017","journal-title":"Proceedings of the IEEE International Conference on Computer Vision"},{"key":"B5","unstructured":"Explaining image classifiers by counterfactual generation\n            ChangCH\n            CreagerE\n            GoldenbergA\n            DuvenaudD\n          arXiv [Preprint]2019"},{"key":"B6","doi-asserted-by":"publisher","first-page":"2950","DOI":"10.1109\/ICCV.2019.00304","article-title":"Understanding deep networks via extremal perturbations and smooth masks","author":"Fong","year":"2019","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B7","doi-asserted-by":"publisher","first-page":"5188","DOI":"10.1109\/CVPR.2015.7299155","article-title":"Understanding deep image representations by inverting them","author":"Mahendran","year":"2015","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B8","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1109\/IJCNN.1989.118277","article-title":"Inversion of multilayer nets","author":"Linden","year":"1989","journal-title":"Proceedings of International Joint Conference on Neural Networks"},{"key":"B9","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1016\/0167-8191(90)90081-J","article-title":"Inversion of neural networks by gradient descent","volume":"14","author":"Kindermann","year":"1990","journal-title":"Parallel Comput"},{"key":"B10","doi-asserted-by":"publisher","first-page":"1536","DOI":"10.1109\/5.784232","article-title":"Inversion of feedforward neural networks: algorithms and applications","volume":"87","author":"Jensen","year":"1999","journal-title":"Proc IEEE"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1271","DOI":"10.1109\/72.809074","article-title":"Inverting feedforward neural networks using linear and nonlinear programming","volume":"10","author":"Lu","year":"1999","journal-title":"IEEE Trans Neural Netw"},{"key":"B12","article-title":"Auto-encoding variational Bayes","author":"Kingma","year":"2013","journal-title":"arXiv preprint arXiv:13126114"},{"key":"B13","first-page":"1530","article-title":"Variational inference with normalizing flows","author":"Rezende","year":"2015","journal-title":"International Conference on Machine Learning"},{"key":"B14","article-title":"Nice: Non-linear independent components estimation","author":"Dinh","year":"2015","journal-title":"International Conference on Learning Representations"},{"key":"B15","doi-asserted-by":"publisher","first-page":"2223","DOI":"10.1109\/ICCV.2017.244","article-title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","author":"Zhu","year":"2017","journal-title":"Proceedings of the IEEE International Conference on Computer Vision"},{"key":"B16","first-page":"2256","article-title":"Deep unsupervised learning using nonequilibrium thermodynamics","author":"Sohl-Dickstein","year":"2015","journal-title":"International Conference on Machine Learning"},{"key":"B17","first-page":"6840","article-title":"Denoising diffusion probabilistic models","volume":"33","author":"Ho","year":"2020","journal-title":"Adv Neural Inform Process Syst"},{"key":"B18","first-page":"573","article-title":"Invertible residual networks","author":"Behrmann","year":"2019","journal-title":"International Conference on Machine Learning"},{"key":"B19","first-page":"1792","article-title":"Understanding and mitigating exploding inverses in invertible neural networks","author":"Behrmann","year":"2021","journal-title":"International Conference on Artificial Intelligence and Statistics"},{"key":"B20","doi-asserted-by":"publisher","first-page":"3121","DOI":"10.1109\/TPAMI.2022.3181070","article-title":"GAN inversion: a survey","volume":"45","author":"Xia","year":"2022","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"B21","article-title":"An image is worth one word: Personalizing text-to-image generation using textual inversion","author":"Gal","year":"2022","journal-title":"arXiv preprint arXiv:220801618"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-009-1740-8","author":"Engl","year":"1996","journal-title":"Regularization of Inverse Problems"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-69277-7","author":"Scherzer","year":"2009","journal-title":"Variational Methods in Imaging"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1017\/S0962492918000016","article-title":"Modern regularization methods for inverse problems","volume":"27","author":"Benning","year":"2018","journal-title":"Acta Numer"},{"key":"B25","article-title":"Adversarial regularizers in inverse problems","author":"Lunz","year":"2018","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1017\/S0962492919000059","article-title":"Solving inverse problems using data-driven models","volume":"28","author":"Arridge","year":"2019","journal-title":"Acta Numer"},{"key":"B27","doi-asserted-by":"publisher","first-page":"025008","DOI":"10.1088\/1361-6420\/aaf14a","article-title":"Deep null space learning for inverse problems: convergence analysis and rates","volume":"35","author":"Schwab","year":"2019","journal-title":"Inverse Probl"},{"key":"B28","doi-asserted-by":"publisher","first-page":"065005","DOI":"10.1088\/1361-6420\/ab6d57","article-title":"NETT: Solving inverse problems with deep neural networks","volume":"36","author":"Li","year":"2020","journal-title":"Inverse Probl"},{"key":"B29","article-title":"Learned convex regularizers for inverse problems","author":"Mukherjee","year":"2021","journal-title":"arXiv preprint arXiv:200802839v2"},{"key":"B30","article-title":"Lifted Bregman training of neural networks","author":"Wang","year":"2022","journal-title":"arXiv preprint arXiv:220808772"},{"key":"B31","unstructured":"Generalised perceptron learning\n            WangX\n            BenningM\n          12th Annual Workshop on Optimization for Machine Learning2020"},{"key":"B32","doi-asserted-by":"publisher","first-page":"200","DOI":"10.1016\/0041-5553(67)90040-7","article-title":"The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming","volume":"7","author":"Bregman","year":"1967","journal-title":"USSR Comput Math Math Phys"},{"key":"B33","doi-asserted-by":"publisher","first-page":"1142","DOI":"10.1137\/S0363012995281742","article-title":"Proximal minimization methods with generalized Bregman functions","volume":"35","author":"Kiwiel","year":"1997","journal-title":"SIAM J Control Optim"},{"key":"B34","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1109\/18.61115","article-title":"Divergence measures based on the Shannon entropy","volume":"37","author":"Lin","year":"1991","journal-title":"IEEE Trans Inform Theory"},{"key":"B35","doi-asserted-by":"publisher","first-page":"961","DOI":"10.1109\/TIT.1982.1056573","article-title":"On the convexity of higher order Jensen differences based on entropy functions (Corresp","volume":"28","author":"Burbea","year":"1982","journal-title":"IEEE Trans Inform Theory"},{"key":"B36","doi-asserted-by":"publisher","first-page":"489","DOI":"10.1109\/TIT.1982.1056497","article-title":"On the convexity of some divergence measures based on entropy functions","volume":"28","author":"Burbea","year":"1982","journal-title":"IEEE Trans Inform Theory"},{"key":"B37","doi-asserted-by":"publisher","first-page":"5455","DOI":"10.1109\/TIT.2011.2159046","article-title":"The burbea-rao and bhattacharyya centroids","volume":"57","author":"Nielsen","year":"2011","journal-title":"IEEE Trans Inform Theory"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611974997","article-title":"First-order methods in optimization","author":"Beck","year":"2017","journal-title":"SIAM"},{"key":"B39","first-page":"77","article-title":"Error estimates for general fidelities","volume":"38","author":"Benning","year":"2011","journal-title":"Electron Trans Numer Anal"},{"key":"B40","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1017\/S096249291600009X","article-title":"An introduction to continuous optimization for imaging","volume":"25","author":"Chambolle","year":"2016","journal-title":"Acta Numer"},{"key":"B41","first-page":"8","article-title":"An efficient primal-dual hybrid gradient algorithm for total variation image restoration","volume":"34","author":"Zhu","year":"2008","journal-title":"Ucla Cam Rep"},{"key":"B42","doi-asserted-by":"publisher","first-page":"1133","DOI":"10.1109\/ICCV.2009.5459348","article-title":"An algorithm for minimizing the Mumford-Shah functional","author":"Pock","year":"2009","journal-title":"2009 IEEE 12th International Conference on Computer Vision"},{"key":"B43","doi-asserted-by":"publisher","first-page":"1015","DOI":"10.1137\/09076934X","article-title":"A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science","volume":"3","author":"Esser","year":"2010","journal-title":"SIAM J Imaging Sci"},{"key":"B44","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1007\/s10851-010-0251-1","article-title":"A first-order primal-dual algorithm for convex problems with applications to imaging","volume":"40","author":"Chambolle","year":"2011","journal-title":"J Math Imaging Vis"},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-03009-4_62-2","article-title":"Bregman methods for large-scale optimization with applications in imaging","author":"Benning","year":"2023","journal-title":"Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging"},{"key":"B46","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1016\/0167-2789(92)90242-F","article-title":"Nonlinear total variation based noise removal algorithms","volume":"60","author":"Rudin","year":"1992","journal-title":"Phys D"},{"key":"B47","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1007\/s002110050258","article-title":"Image recovery via total variation minimization and related problems","volume":"76","author":"Chambolle","year":"1997","journal-title":"Numer Math"},{"key":"B48","doi-asserted-by":"publisher","first-page":"2037","DOI":"10.1137\/120887679","article-title":"On the convergence of block coordinate descent type methods","volume":"23","author":"Beck","year":"2013","journal-title":"SIAM J Optim"},{"key":"B49","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s10107-015-0892-3","article-title":"Coordinate descent algorithms","volume":"151","author":"Wright","year":"2015","journal-title":"Math Programm"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1017\/9781009004282","author":"Wright","year":"2022","journal-title":"Optimization for Data Analysis"},{"key":"B51","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc IEEE"},{"key":"B52","doi-asserted-by":"publisher","first-page":"615","DOI":"10.2307\/2372313","article-title":"An iteration formula for Fredholm integral equations of the first kind","volume":"73","author":"Landweber","year":"1951","journal-title":"Am J Math"},{"key":"B53","author":"Morozov","year":"2012","journal-title":"Methods for Solving Incorrectly Posed Problems"},{"key":"B54","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1023\/B:JMIV.0000011321.19549.88","article-title":"An algorithm for total variation minimization and applications","volume":"20","author":"Chambolle","year":"2004","journal-title":"J Math Imaging Vis"},{"key":"B55","doi-asserted-by":"publisher","first-page":"770","DOI":"10.1109\/CVPR.2016.90","article-title":"Deep residual learning for image recognition","author":"He","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B56","doi-asserted-by":"publisher","first-page":"234","DOI":"10.1007\/978-3-319-24574-4_28","article-title":"U-net: convolutional networks for biomedical image segmentation","author":"Ronneberger","year":"2015","journal-title":"Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference"},{"key":"B57","unstructured":"Convergent data-driven regularizations for CT reconstruction\n            KabriS\n            AurasA\n            RiccioD\n            BauermeisterH\n            BenningM\n            MoellerM\n          36048759arXiv [Preprint]2022"},{"key":"B58","doi-asserted-by":"publisher","first-page":"1190","DOI":"10.1080\/01630563.2020.1740734","article-title":"A data-driven iteratively regularized Landweber iteration","volume":"41","author":"Aspri","year":"2020","journal-title":"Numer Funct Anal Optim"}],"container-title":["Frontiers in Applied Mathematics and Statistics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fams.2023.1176850\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,13]],"date-time":"2023-06-13T10:57:26Z","timestamp":1686653846000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fams.2023.1176850\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,13]]},"references-count":58,"alternative-id":["10.3389\/fams.2023.1176850"],"URL":"https:\/\/doi.org\/10.3389\/fams.2023.1176850","relation":{},"ISSN":["2297-4687"],"issn-type":[{"value":"2297-4687","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,13]]},"article-number":"1176850"}}