{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:29:04Z","timestamp":1787329744028,"version":"build-2736575974"},"reference-count":29,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"3","funder":[{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000005","name":"U.S. Department of Defense","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000005","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM Journal on Mathematics of Data Science"],"published-print":{"date-parts":[[2019,1]]},"abstract":"<jats:p>Deep neural networks (DNNs) typically have enough capacity to fit random data by brute force even when conventional data-dependent regularizations focusing on the geometry of the features are imposed. We find out that the reason for this is the inconsistency between the enforced geometry and the standard softmax cross entropy loss. To resolve this, we propose a new framework for data-dependent DNN regularization, the Geometrically-Regularized-Self-Validating neural Networks (GRSVNet). During training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss consistent with the geometry. We study a particular case of GRSVNet, the Orthogonal-Low-rank Embedding (OLE)-GRSVNet, which is capable of producing highly discriminative features residing in orthogonal low-rank subspaces. Numerical experiments show that OLE-GRSVNet outperforms DNNs with conventional regularization when trained on real data, especially when the training samples are scarce. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses to memorize random data or random labels, suggesting that it only learns intrinsic patterns by reducing the memorizing capacity of the baseline DNN.<\/jats:p>","DOI":"10.1137\/19m1236886","type":"journal-article","created":{"date-parts":[[2019,8,27]],"date-time":"2019-08-27T14:27:27Z","timestamp":1566916047000},"page":"476-496","source":"Crossref","is-referenced-by-count":2,"title":["Stop Memorizing: A Data-Dependent Regularization Framework for Intrinsic Pattern Learning"],"prefix":"10.1137","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9181-5103","authenticated-orcid":true,"given":"Wei","family":"Zhu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Qiu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bao","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6255-5165","authenticated-orcid":true,"given":"Jianfeng","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillermo","family":"Sapiro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ingrid","family":"Daubechies","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2019,8,27]]},"reference":[{"key":"atypb1","first-page":"233","author":"Arpit D.","year":"2017","journal-title":"Australia"},{"key":"atypb2","first-page":"463","volume":"3","author":"Bartlett P. L.","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"atypb3","first-page":"499","volume":"2","author":"Bousquet O.","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"atypb4","first-page":"1335","author":"Cheng D.","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"atypb5","first-page":"1","author":"Chopra S.","year":"2005","journal-title":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR"},{"key":"atypb6","doi-asserted-by":"crossref","unstructured":"J. W. Demmel,\n                      Applied Numerical Linear Algebra\n                      , SIAM, Philadelphia, 1997,https:\/\/doi.org\/10.1137\/1.9781611971446.","DOI":"10.1137\/1.9781611971446"},{"key":"atypb7","first-page":"249","author":"Glorot X.","year":"2010","journal-title":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics"},{"key":"atypb8","unstructured":"G. H. Golub and C. F. Van Loan,\n                      Matrix Computations\n                      , 3rd ed., Johns Hopkins University Press, Baltimore, MD, 2012."},{"key":"atypb9","doi-asserted-by":"crossref","unstructured":"R. Hadsell, S. Chopra, and Y. LeCun,\n                      Dimensionality reduction by learning an invariant mapping\n                      , in Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, 2006, pp. 1735-1742.","DOI":"10.1109\/CVPR.2006.100"},{"key":"atypb10","first-page":"770","author":"He K.","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"atypb11","first-page":"1875","author":"Hu J.","year":"2014","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"atypb12","first-page":"4700","author":"Huang G.","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"atypb13","unstructured":"A. Krizhevsky and G. Hinton,\n                      Learning Multiple Layers of Features from Tiny Images\n                      , Technical Report TR-2009, University of Toronto, Toronto, Canada, 2009."},{"key":"atypb14","first-page":"396","author":"LeCun Y.","year":"1990","journal-title":"San Francisco"},{"key":"atypb15","doi-asserted-by":"crossref","unstructured":"J. Lezama, Q. Qiu, P. Mus\u00e9, and G. Sapiro,\n                      OLE: Orthogonal low-rank embedding, a plug and play geometric loss for deep learning\n                      , in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.","DOI":"10.1109\/CVPR.2018.00846"},{"key":"atypb16","unstructured":"Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng,\n                      Reading digits in natural images with unsupervised feature learning\n                      , in NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011."},{"key":"atypb17","doi-asserted-by":"crossref","unstructured":"T. Papadopoulo and M. I. A. Lourakis,\n                      Estimating the Jacobian of the singular value decomposition: Theory and applications\n                      , in Computer Vision - ECCV 2000, Springer, Berlin, Heidelberg, pp. 554-570.","DOI":"10.1007\/3-540-45054-8_36"},{"key":"atypb18","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1038\/nature02341","volume":"428","author":"Poggio T.","year":"2004","journal-title":"Nature"},{"key":"atypb19","first-page":"187","volume":"16","author":"Qiu Q.","year":"2015","journal-title":"J. Mach. Learn. Res."},{"key":"atypb20","doi-asserted-by":"publisher","DOI":"10.1137\/070697835"},{"key":"atypb21","first-page":"815","author":"Schroff F.","year":"2015","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"atypb22","unstructured":"K. Simonyan and A. Zisserman,\n                      Very Deep Convolutional Networks for Large-Scale Image Recognition\n                      , preprint,https:\/\/arxiv.org\/abs\/1409.1556, 2014."},{"key":"atypb23","unstructured":"Y. Sun, Y. Chen, X. Wang, and X. Tang,\n                      Deep learning face representation by joint identification-verification\n                      , in Proceedings of the 27th International Conference on Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, eds., MIT Press, Cambridge, MA, 2014, pp. 1988-1996."},{"key":"atypb24","unstructured":"V. Vapnik,\n                      Statistical Learning Theory\n                      , Wiley, New York, 1998."},{"key":"atypb25","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1016\/0024-3795(92)90407-2","volume":"170","author":"Watson G. A.","year":"1992","journal-title":"Linear Algebra Appl."},{"key":"atypb26","first-page":"499","author":"Wen Y.","year":"2016","journal-title":"Cham"},{"key":"atypb27","first-page":"639","author":"Weston J.","year":"2012","journal-title":"Heidelberg"},{"key":"atypb28","unstructured":"C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals,\n                      Understanding deep learning requires rethinking generalization\n                      , in Proceedings of the International Conference on Learning Representations, 2017."},{"key":"atypb29","doi-asserted-by":"crossref","unstructured":"W. Zhu, Q. Qiu, J. Huang, R. Calderbank, G. Sapiro, and I. Daubechies,\n                      LDMNet: Low dimensional manifold regularized neural networks\n                      , in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.","DOI":"10.1109\/CVPR.2018.00290"}],"container-title":["SIAM Journal on Mathematics of Data Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/19M1236886","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T15:31:30Z","timestamp":1787326290000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/19M1236886"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1]]},"references-count":29,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,1]]}},"alternative-id":["10.1137\/19M1236886"],"URL":"https:\/\/doi.org\/10.1137\/19m1236886","relation":{},"ISSN":["2577-0187"],"issn-type":[{"value":"2577-0187","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,1]]}}}