{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T21:20:20Z","timestamp":1790284820200,"version":"4.1.0"},"reference-count":43,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,1,27]],"date-time":"2021-01-27T00:00:00Z","timestamp":1611705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Deep Learning (DL) has attracted a lot of attention for its ability to reach state-of-the-art performance in many machine learning tasks. The core principle of DL methods consists of training composite architectures in an end-to-end fashion, where inputs are associated with outputs trained to optimize an objective function. Because of their compositional nature, DL architectures naturally exhibit several intermediate representations of the inputs, which belong to so-called latent spaces. When treated individually, these intermediate representations are most of the time unconstrained during the learning process, as it is unclear which properties should be favored. However, when processing a batch of inputs concurrently, the corresponding set of intermediate representations exhibit relations (what we call a geometry) on which desired properties can be sought. In this work, we show that it is possible to introduce constraints on these latent geometries to address various problems. In more detail, we propose to represent geometries by constructing similarity graphs from the intermediate representations obtained when processing a batch of inputs. By constraining these Latent Geometry Graphs (LGGs), we address the three following problems: (i) reproducing the behavior of a teacher architecture is achieved by mimicking its geometry, (ii) designing efficient embeddings for classification is achieved by targeting specific geometries, and (iii) robustness to deviations on inputs is achieved via enforcing smooth variation of geometry between consecutive latent spaces. Using standard vision benchmarks, we demonstrate the ability of the proposed geometry-based methods in solving the considered problems.<\/jats:p>","DOI":"10.3390\/a14020039","type":"journal-article","created":{"date-parts":[[2021,1,27]],"date-time":"2021-01-27T12:20:26Z","timestamp":1611750026000},"page":"39","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Representing Deep Neural Networks Latent Space Geometries with Graphs"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7754-6656","authenticated-orcid":false,"given":"Carlos","family":"Lassance","sequence":"first","affiliation":[{"name":"Electronics Department, IMT Atlantique, 29280 Brest, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4353-4542","authenticated-orcid":false,"given":"Vincent","family":"Gripon","sequence":"additional","affiliation":[{"name":"Electronics Department, IMT Atlantique, 29280 Brest, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Antonio","family":"Ortega","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90007, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,27]]},"reference":[{"key":"ref_1","unstructured":"Tan, M., and Le, Q.V. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Edunov, S., Ott, M., Auli, M., and Grangier, D. (2018). Understanding back-translation at scale. arXiv.","DOI":"10.18653\/v1\/D18-1045"},{"key":"ref_3","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_5","first-page":"22","article-title":"The Power and Limits of Deep Learning","volume":"61","author":"LeCun","year":"2018","journal-title":"Res.-Technol. Manag."},{"key":"ref_6","unstructured":"Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., and Usunier, N. (2017, January 6\u201311). Parseval networks: Improving robustness to adversarial examples. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_7","unstructured":"Hendrycks, D., and Dietterich, T. (2019, January 6\u20139). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Lassance, C., Gripon, V., and Ortega, A. (2021, January 8). Laplacian Networks: Bounding Indicator Function Smoothness for Neural Networks Robustness. Proceedings of the APSIPA Transactions on Signal and Information Processing, to appear.","DOI":"10.1017\/ATSIP.2021.2"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Park, W., Kim, D., Lu, Y., and Cho, M. (2019, January 16\u201320). Relational Knowledge Distillation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00409"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Lassance, C., Bontonou, M., Hacene, G.B., Gripon, V., Tang, J., and Ortega, A. (2020, January 4\u20138). Deep geometric knowledge distillation with graphs. Proceedings of the ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9053986"},{"key":"ref_11","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2014, January 8\u201313). Distilling the knowledge in a neural network. Proceedings of the Neural Information Processing Systems 2014 Deep Learning Workshop, Montreal, QC, Canada."},{"key":"ref_12","unstructured":"Koratana, A., Kang, D., Bailis, P., and Zaharia, M. (2019, January 9\u201315). LIT: Learned intermediate representation training for model compression. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_13","unstructured":"Hermans, A., Beyer, L., and Leibe, B. (2017). In defense of the triplet loss for person re-identification. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bontonou, M., Lassance, C., Hacene, G.B., Gripon, V., Tang, J., and Ortega, A. (2019, January 2\u20135). Introducing Graph Smoothness Loss for Training Deep Learning Architectures. Proceedings of the 2019 IEEE Data Science Workshop (DSW), Minneapolis, MN, USA.","DOI":"10.1109\/DSW.2019.8755559"},{"key":"ref_15","unstructured":"Svoboda, J., Masci, J., Monti, F., Bronstein, M., and Guibas, L. (2019, January 6\u20139). PeerNets: Exploiting Peer Wisdom Against Adversarial Attacks. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_16","unstructured":"Qian, H., and Wegman, M.N. (2019, January 6\u20139). L2-Nonexpansive Neural Networks. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_18","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in neural information processing systems, Lake Tahoe, NV, USA."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1109\/MSP.2012.2235192","article-title":"The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains","volume":"30","author":"Shuman","year":"2013","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_20","unstructured":"Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv."},{"key":"ref_21","unstructured":"Vialatte, J.C. (2018). On Convolution of Graph Signals and Deep Learning on Graph Domains. [Ph.D. Thesis, IMT Atlantique]."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1109\/MSP.2020.3016143","article-title":"Graphs, Convolutions, and Neural Networks: From Graph Filters to Graph Neural Networks","volume":"37","author":"Gama","year":"2020","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Philip, S.Y. (2020). A comprehensive survey on graph neural networks. IEEE Trans. Neural Networks Learn. Syst.","DOI":"10.1109\/TNNLS.2020.2978386"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Liu, Y., Cao, J., Li, B., Yuan, C., Hu, W., Li, Y., and Duan, Y. (2019, January 16\u201320). Knowledge Distillation via Instance Relationship Graph. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00726"},{"key":"ref_25","unstructured":"Lee, S., and Song, B. (2019). Graph-based knowledge distillation by multi-head attention network. arXiv."},{"key":"ref_26","unstructured":"Anirudh, R., Bremer, P., Sridhar, R., and Thiagarajan, J. (2017). Influential Sample Selection: A Graph Signal Processing Approach, Lawrence Livermore National Lab. (LLNL). Technical Report."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Gripon, V., Ortega, A., and Girault, B. (2018, January 11\u201316). An Inside Look at Deep Neural Networks using Graph Signal Processing. Proceedings of the ITA, San Diego, CA, USA.","DOI":"10.1109\/ITA.2018.8503214"},{"key":"ref_28","unstructured":"Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., and Bengio, Y. (2015, January 7\u20139). Fitnets: Hints for thin deep nets. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA."},{"key":"ref_29","unstructured":"Veli\u010dkovi\u0107, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. (2017). Graph attention networks. arXiv."},{"key":"ref_30","unstructured":"Hu, Y., Gripon, V., and Pateux, S. (2020). Exploiting Unsupervised Inputs for Accurate Few-Shot Classification. arXiv."},{"key":"ref_31","unstructured":"Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1613\/jair.105","article-title":"Solving multiclass learning problems via error-correcting output codes","volume":"2","author":"Dietterich","year":"1994","journal-title":"J. Artif. Intell. Res."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_34","unstructured":"Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv."},{"key":"ref_35","unstructured":"Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (May, January 30). Towards Deep Learning Models Resistant to Adversarial Attacks. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_36","unstructured":"Hacene, G.B. (2019). Processing and Learning Deep Neural Networks on Chip. [Ph.D. Thesis, Ecole Nationale Sup\u00e9rieure Mines-T\u00e9l\u00e9com Atlantique]."},{"key":"ref_37","unstructured":"Frankle, J., and Carbin, M. (2019, January 6\u20139). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1373","DOI":"10.1162\/089976603321780317","article-title":"Laplacian eigenmaps for dimensionality reduction and data representation","volume":"15","author":"Belkin","year":"2003","journal-title":"Neural Comput."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zagoruyko, S., and Komodakis, N. (2016). Wide residual networks. arXiv.","DOI":"10.5244\/C.30.87"},{"key":"ref_40","unstructured":"Krizhevsky, A., and Hinton, G. (2021, January 15). Learning Multiple Layers of Features from Tiny Images. Available online: https:\/\/www.cs.toronto.edu\/~kriz\/learning-features-2009-TR.pdf."},{"key":"ref_41","unstructured":"Kalofolias, V., and Perraudin, N. (2019, January 6\u20139). Large Scale Graph Learning From Smooth Signals. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Shekkizhar, S., and Ortega, A. (2020, January 4\u20138). Graph Construction from Data by Non-Negative Kernel Regression. Proceedings of the ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9054425"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1958","DOI":"10.1109\/TPAMI.2008.128","article-title":"80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition","volume":"30","author":"Torralba","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/2\/39\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:16:05Z","timestamp":1760159765000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/2\/39"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,27]]},"references-count":43,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["a14020039"],"URL":"https:\/\/doi.org\/10.3390\/a14020039","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,27]]}}}