{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T12:50:10Z","timestamp":1773924610970,"version":"3.50.1"},"reference-count":43,"publisher":"American Association for the Advancement of Science (AAAS)","content-domain":{"domain":["spj.science.org"],"crossmark-restriction":true},"short-container-title":["Intell Comput"],"published-print":{"date-parts":[[2026,1,1]]},"abstract":"<jats:p>Deep learning models are known to be vulnerable to privacy attacks that can reconstruct or infer sensitive information from the training data. While many privacy assessment methods exist, they are often computationally prohibitive or overlook the considerable information leakage occurring from the model\u2019s intermediate layers. This paper introduces an efficient, attack-independent framework for quantifying privacy risks by analyzing the sensitivity of these internal representations. We discover that the rank of the Jacobian matrix of an intermediate layer\u2019s output with respect to the model\u2019s input serves as a robust proxy for privacy vulnerability. Specifically, we analyze the dynamics of the Jacobian rank throughout training, proposing 2 metrics, rank recovery (RR) and normalized rank recovery (NRR), to specifically quantify a layer\u2019s dynamic transition from generalization to memorization. Through extensive experiments on a diverse range of computer vision models and datasets, including modern residual networks, we demonstrate a strong and consistent positive correlation between our proposed rank-based metrics and the success rates of membership inference attacks. Our findings reveal that deeper layers, which exhibit considerably higher RR, are the primary sources of privacy leakage. This work provides an actionable methodology for developers to identify high-risk layers in real time, facilitating the development of more secure and privacy-preserving machine learning systems without the need for expensive adversarial simulations.<\/jats:p>","DOI":"10.34133\/icomputing.0270","type":"journal-article","created":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T11:00:17Z","timestamp":1768820417000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark_01","source":"Crossref","is-referenced-by-count":0,"title":["Probing Privacy Risks in Deep Neural Networks: A Jacobian Rank-Based Analysis of Intermediate Layers"],"prefix":"10.34133","volume":"5","author":[{"given":"Sheng","family":"Liu","sequence":"first","affiliation":[{"name":"School of Science and Big Data, \rNanjing Institute of Technology, Nanjing, China."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xuefeng","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Science and Big Data, \rNanjing Institute of Technology, Nanjing, China."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9067-2270","authenticated-orcid":true,"given":"Tao","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Computer and Big Data, \rMinjiang University, Fuzhou, China."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guolong","family":"Zheng","sequence":"additional","affiliation":[{"name":"School of Computer and Big Data, \rMinjiang University, Fuzhou, China."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wencheng","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Mathematics, Physics and Computing, \rUniversity of Southern Queensland, Toowoomba, Queensland, Australia."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ji","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Mathematics, Physics and Computing, \rUniversity of Southern Queensland, Toowoomba, Queensland, Australia."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"221","published-online":{"date-parts":[[2026,3,4]]},"reference":[{"issue":"1","key":"e_1_3_3_2_2","first-page":"7068349","article-title":"Deep learning for computer vision: A brief review","volume":"2018","author":"Voulodimos A","year":"2018","unstructured":"Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E. Deep learning for computer vision: A brief review. Comput Intell Neurosci. 2018;2018(1):7068349.","journal-title":"Comput Intell Neurosci"},{"key":"e_1_3_3_3_2","article-title":"Deep learning in computer vision: A critical review of emerging techniques and application scenarios","volume":"6","author":"Chai J","year":"2021","unstructured":"Chai J, Zeng H, Li A, Ngai EW. Deep learning in computer vision: A critical review of emerging techniques and application scenarios. Mach Learn Appl. 2021;6: Article 100134.","journal-title":"Mach Learn Appl"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2018.01.092"},{"key":"e_1_3_3_5_2","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1016\/j.conbuildmat.2017.09.110","article-title":"Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection","volume":"157","author":"Gopalakrishnan K","year":"2017","unstructured":"Gopalakrishnan K, Khaitan SK, Choudhary A, Agrawal A. Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Constr Build Mater. 2017;157:322\u2013330.","journal-title":"Constr Build Mater"},{"key":"e_1_3_3_6_2","doi-asserted-by":"crossref","unstructured":"Yeom S Giacomelli I Fredrikson M Jha S. Privacy risk in machine learning: Analyzing the connection to overfitting. In: 2018 IEEE 31st computer security foundations symposium (CSF). Piscataway (NJ): IEEE; 2018. p. 268\u2013282.","DOI":"10.1109\/CSF.2018.00027"},{"key":"e_1_3_3_7_2","doi-asserted-by":"crossref","unstructured":"Orekondy T Schiele B Fritz M. Towards a visual privacy advisor: Understanding and predicting privacy risks in images. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). Piscataway (NJ): IEEE; 2017. p. 3686\u20133695.","DOI":"10.1109\/ICCV.2017.398"},{"key":"e_1_3_3_8_2","doi-asserted-by":"crossref","unstructured":"Papernot N McDaniel P Sinha A Wellman MP. Sok: Security and privacy in machine learning. In: 2018 IEEE European symposium on security and privacy (EuroS&P). Piscataway (NJ): IEEE; 2018. p. 399\u2013414.","DOI":"10.1109\/EuroSP.2018.00035"},{"key":"e_1_3_3_9_2","first-page":"7232","article-title":"Evaluating gradient inversion attacks and defenses in federated learning","volume":"34","author":"Huang Y","year":"2021","unstructured":"Huang Y, Gupta S, Song Z, Li K, Arora S. Evaluating gradient inversion attacks and defenses in federated learning. Adv Neural Inf Proces Syst. 2021;34:7232\u20137241.","journal-title":"Adv Neural Inf Proces Syst"},{"key":"e_1_3_3_10_2","first-page":"29898","article-title":"Gradient inversion with generative image prior","volume":"34","author":"Jeon J","year":"2021","unstructured":"Jeon J, Lee K, Oh S, Ok J. Gradient inversion with generative image prior. Adv Neural Inf Proces Syst. 2021;34:29898\u201329908.","journal-title":"Adv Neural Inf Proces Syst"},{"key":"e_1_3_3_11_2","doi-asserted-by":"crossref","unstructured":"Hatamizadeh A Yin H Roth HR Li W Kautz J Xu D Molchanov P. GradViT: Gradient inversion of vision transformers. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway (NJ): IEEE; 2022. p. 10021\u201310030.","DOI":"10.1109\/CVPR52688.2022.00978"},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","unstructured":"Zhang R Guo S Wang J Xie X Tao D. A survey on gradient inversion: Attacks defenses and future directions. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2206.07284","DOI":"10.24963\/ijcai.2022\/791"},{"key":"e_1_3_3_13_2","first-page":"323","volume-title":"European Conference on Computer Vision","author":"Wang Z","year":"2024","unstructured":"Wang Z, Shen L, Guo J, Duan T, Luan S, Liu T, Gao M. Training a secure model against data-free model extraction. In:European Conference on Computer Vision. Cham (Switzerland): Springer; 2024. p. 323\u2013340."},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3523273"},{"key":"e_1_3_3_15_2","doi-asserted-by":"crossref","unstructured":"Shokri R Stronati M Song C Shmatikov V. Membership inference attacks against machine learning models. In: 2017 IEEE symposium on security and privacy (SP). Piscataway (NJ): IEEE; 2017. p. 3\u201318.","DOI":"10.1109\/SP.2017.41"},{"key":"e_1_3_3_16_2","doi-asserted-by":"crossref","unstructured":"Carlini N Chien S Nasr M Song S Terzis A Tramer F. Membership inference attacks from first principles. In: 2022 IEEE Symposium on Security and Privacy (SP). Piscataway (NJ): IEEE; 2022. p. 1897\u20131914.","DOI":"10.1109\/SP46214.2022.9833649"},{"issue":"6","key":"e_1_3_3_17_2","doi-asserted-by":"crossref","first-page":"2073","DOI":"10.1109\/TSC.2019.2897554","article-title":"Demystifying membership inference attacks in machine learning as a service","volume":"14","author":"Truex S","year":"2019","unstructured":"Truex S, Liu L, Gursoy ME, Yu L, Wei W. Demystifying membership inference attacks in machine learning as a service. IEEE Trans Serv Comput. 2019;14(6):2073\u20132089.","journal-title":"IEEE Trans Serv Comput"},{"key":"e_1_3_3_18_2","doi-asserted-by":"crossref","unstructured":"Ye J Maddi A Murakonda SK Bindschaedler V Shokri R. Enhanced membership inference at tacks against machine learning models. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. New York (NY): ACM; 2022. p. 3093\u20133106.","DOI":"10.1145\/3548606.3560675"},{"key":"e_1_3_3_19_2","unstructured":"Choquette-Choo CA Tramer F Carlini N. Label-only membership inference attacks. In: The Eleventh International Conference on Learning Representations (ICLR). Kigali (Rwanda): OpenReview.net; 2023."},{"key":"e_1_3_3_20_2","doi-asserted-by":"crossref","unstructured":"Dwork C. Differential privacy. In: International colloquium on automata languages and programming. Berlin (Germany): Springer; 2006. p. 1\u201312.","DOI":"10.1007\/11787006_1"},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","unstructured":"Dwork C. Differential privacy: A survey of results. In: International conference on theory and applications of models of computation. Berlin (Germany): Springer; 2008. p. 1\u201319.","DOI":"10.1007\/978-3-540-79228-4_1"},{"key":"e_1_3_3_22_2","doi-asserted-by":"crossref","unstructured":"Abadi M Chu A Goodfellow I McMahan HB Mironov I Talwar K Zhang L. Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. New York (NY): ACM; 2016. p. 308\u2013318.","DOI":"10.1145\/2976749.2978318"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2020.2988575"},{"issue":"5","key":"e_1_3_3_24_2","doi-asserted-by":"crossref","first-page":"1551","DOI":"10.1007\/s11280-021-00922-2","article-title":"Data privacy preservation algorithm with k- anonymity","volume":"24","author":"Mahanan W","year":"2021","unstructured":"Mahanan W, Chaovalitwongse WA, Natwichai J. Data privacy preservation algorithm with k- anonymity. World Wide Web. 2021;24(5):1551\u20131561.","journal-title":"World Wide Web"},{"issue":"2","key":"e_1_3_3_25_2","first-page":"81","article-title":"Enhancing data anonymization: A semantic K-anonymity framework with ML and NLP integration","volume":"5","author":"Saxena AK","year":"2022","unstructured":"Saxena AK. Enhancing data anonymization: A semantic K-anonymity framework with ML and NLP integration. Sage Sci Rev Appl Mach Learn. 2022;5(2):81\u201392.","journal-title":"Sage Sci Rev Appl Mach Learn"},{"key":"e_1_3_3_26_2","doi-asserted-by":"crossref","DOI":"10.1016\/j.cose.2021.102488","article-title":"K-anonymity in practice: How generalisation and suppression affect machine learning classifiers","volume":"111","author":"Slijep\u010devi\u0107 D","year":"2021","unstructured":"Slijep\u010devi\u0107 D, Henzl M, Klausner LD, Dam T, Kieseberg P, Zeppelzauer M. K-anonymity in practice: How generalisation and suppression affect machine learning classifiers. Comput Secur. 2021;111: Article 102488.","journal-title":"Comput Secur"},{"issue":"3","key":"e_1_3_3_27_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3456876","article-title":"Anonymization of daily activity data by using l-diversity privacy model","volume":"12","author":"Parameshwarappa P","year":"2021","unstructured":"Parameshwarappa P, Chen Z, Koru G. Anonymization of daily activity data by using l-diversity privacy model. ACM Trans Manag Inf Syst. 2021;12(3):1\u201321.","journal-title":"ACM Trans Manag Inf Syst"},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ins.2020.07.066","article-title":"DI-Mondrian: Distributed improved Mondrian for satisfaction of the L-diversity privacy model using apache spark","volume":"546","author":"Ashkouti F","year":"2021","unstructured":"Ashkouti F, Sheikhahmadi A. DI-Mondrian: Distributed improved Mondrian for satisfaction of the L-diversity privacy model using apache spark. Inf Sci. 2021;546:1\u201324.","journal-title":"Inf Sci"},{"issue":"4","key":"e_1_3_3_29_2","doi-asserted-by":"crossref","first-page":"1423","DOI":"10.1016\/j.jksuci.2019.08.006","article-title":"Improved l-diversity: Scalable anonymization approach for privacy preserving big data publishing","volume":"34","author":"Mehta BB","year":"2022","unstructured":"Mehta BB, Rao UP. Improved l-diversity: Scalable anonymization approach for privacy preserving big data publishing. J King Saud Univ Comput Inf Sci. 2022;34(4):1423\u20131430.","journal-title":"J King Saud Univ Comput Inf Sci"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics10222877"},{"key":"e_1_3_3_31_2","doi-asserted-by":"crossref","unstructured":"Sun J Li A Wang B Yang H Li H Chen Y. Soteria: Provable defense against privacy leakage in federated learning from representation perspective. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway (NJ): IEEE; 2021. p. 9311\u20139319.","DOI":"10.1109\/CVPR46437.2021.00919"},{"key":"e_1_3_3_32_2","unstructured":"Mireshghallah F Taram M Vepakomma P Singh A Raskar R Esmaeilzadeh H. Privacy in deep learning: A survey. arXiv. 2020. https:\/\/doi.org\/10.48550\/arXiv.2004.12254"},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","unstructured":"UcedaVelez T Morana MM. Risk centric threat modeling: Process for attack simulation and threat analysis. Hoboken (NJ): John Wiley & Sons; 2015.","DOI":"10.1002\/9781118988374"},{"key":"e_1_3_3_34_2","doi-asserted-by":"crossref","unstructured":"Johnson P Lagerstr\u00f6m R Ekstedt M. A meta language for threat modeling and attack simulations. In: Proceedings of the 13th international conference on availability reliability and security. New York (NY): ACM; 2018. p. 1\u20138.","DOI":"10.1145\/3230833.3232799"},{"issue":"2","key":"e_1_3_3_35_2","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1109\/TBDATA.2024.3362191","article-title":"Decentralized federated learning: A survey on security and privacy","volume":"10","author":"Hallaji E","year":"2024","unstructured":"Hallaji E, Razavi-Far R, Saif M, Wang B, Yang Q. Decentralized federated learning: A survey on security and privacy. IEEE Trans Big Data. 2024;10(2):194\u2013213.","journal-title":"IEEE Trans Big Data"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2015.06.032"},{"key":"e_1_3_3_37_2","unstructured":"Ancona M Ceolini E \u00d6ztireli C Gross M. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv. 2017. https:\/\/doi.org\/10.48550\/arXiv.1711.06104"},{"key":"e_1_3_3_38_2","unstructured":"Srinivas S Fleuret F. Knowledge transfer with Jacobian matching. In: International Conference on Machine Learning. Cambridge (MA): PMLR; 2018. p. 4723\u20134731."},{"key":"e_1_3_3_39_2","unstructured":"Hoffman J Roberts DA Yaida S. Robust learning with Jacobian regularization. arXiv. 2019. https:\/\/doi.org\/10.48550\/arXiv.1908.02729"},{"key":"e_1_3_3_40_2","doi-asserted-by":"crossref","unstructured":"Zhang H Zhang P Hsieh CJ. RecurJac: An efficient recursive algorithm for bounding jacobian matrix of neural networks and its applications. In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. Palo Alto CA (USA): AAAI Press; 2019. p. 5757\u20135764.","DOI":"10.1609\/aaai.v33i01.33015757"},{"issue":"44","key":"e_1_3_3_41_2","doi-asserted-by":"crossref","first-page":"27162","DOI":"10.1073\/pnas.2005013117","article-title":"Overparameterized neural networks implement associative memory","volume":"117","author":"Radhakrishnan A","year":"2020","unstructured":"Radhakrishnan A, Belkin M, Uhler C. Overparameterized neural networks implement associative memory. Proc Natl Acad Sci USA. 2020;117(44):27162\u201327170.","journal-title":"Proc Natl Acad Sci USA"},{"issue":"2","key":"e_1_3_3_42_2","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1002\/rsa.20218","article-title":"On variants of the Johnson\u2013Lindenstrauss lemma","volume":"33","author":"Matou\u0161ek J","year":"2008","unstructured":"Matou\u0161ek J. On variants of the Johnson\u2013Lindenstrauss lemma. Random Struct Algoritm. 2008;33(2):142\u2013156.","journal-title":"Random Struct Algoritm"},{"key":"e_1_3_3_43_2","doi-asserted-by":"crossref","unstructured":"Larsen KG Nelson J. Optimality of the Johnson-Lindenstrauss lemma. In: 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS). Piscataway (NJ): IEEE; 2017. p. 633\u2013638.","DOI":"10.1109\/FOCS.2017.64"},{"key":"e_1_3_3_44_2","doi-asserted-by":"crossref","unstructured":"Nasr M Shokri R Houmansadr A. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In: 2019 IEEE symposium on security and privacy (SP). Piscataway (NJ): IEEE; p. 2019. p. 739\u2013753.","DOI":"10.1109\/SP.2019.00065"}],"container-title":["Intelligent Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/spj.science.org\/doi\/pdf\/10.34133\/icomputing.0270","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T08:59:48Z","timestamp":1773910788000},"score":1,"resource":{"primary":{"URL":"https:\/\/spj.science.org\/doi\/10.34133\/icomputing.0270"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,1]]},"references-count":43,"alternative-id":["10.34133\/icomputing.0270"],"URL":"https:\/\/doi.org\/10.34133\/icomputing.0270","relation":{},"ISSN":["2771-5892"],"issn-type":[{"value":"2771-5892","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,1]]},"assertion":[{"value":"2025-08-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-14","order":1,"name":"revised","label":"Revised","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"0270"}}