{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T16:03:44Z","timestamp":1774281824443,"version":"3.50.1"},"reference-count":69,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2025,4,18]],"date-time":"2025-04-18T00:00:00Z","timestamp":1744934400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>This review examines the field of machine unlearning in neural networks, an area driven by data privacy regulations such as the General Data Protection Regulation and the California Consumer Privacy Act. By analyzing 37 primary studies of machine unlearning applied to neural networks in both regression and classification tasks, this review thoroughly evaluates the foundational principles, key performance metrics, and methodologies used to assess these techniques. Special attention is given to recent advancements up to December 2023, including emerging approaches and frameworks. By categorizing and detailing these unlearning techniques, this work offers deeper insights into their evolution, effectiveness, efficiency, and broader applicability, thus providing a solid foundation for future research, development, and practical implementations in the realm of data privacy, model management, and compliance with evolving legal standards. Additionally, this review addresses the challenges of selectively removing data contributions at both the client and instance levels, highlighting the balance between computational costs and privacy guarantees.<\/jats:p>","DOI":"10.3390\/computers14040150","type":"journal-article","created":{"date-parts":[[2025,4,18]],"date-time":"2025-04-18T02:25:31Z","timestamp":1744943131000},"page":"150","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["A Systematic Literature Review of Machine Unlearning Techniques in Neural Networks"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-1108-8208","authenticated-orcid":false,"given":"Ivanna Daniela","family":"Cevallos","sequence":"first","affiliation":[{"name":"Artificial Intelligence and Computer Vision Research Laboratory \u201cAlan Turing\u201d, Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n (DICC), Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5275-7262","authenticated-orcid":false,"given":"Marco E.","family":"Benalc\u00e1zar","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Computer Vision Research Laboratory \u201cAlan Turing\u201d, Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n (DICC), Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3502-020X","authenticated-orcid":false,"given":"\u00c1ngel Leonardo","family":"Valdivieso Caraguay","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Computer Vision Research Laboratory \u201cAlan Turing\u201d, Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n (DICC), Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8263-2682","authenticated-orcid":false,"given":"Jonathan A.","family":"Zea","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Computer Vision Research Laboratory \u201cAlan Turing\u201d, Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n (DICC), Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5184-3759","authenticated-orcid":false,"given":"Lorena Isabel","family":"Barona-L\u00f3pez","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Computer Vision Research Laboratory \u201cAlan Turing\u201d, Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n (DICC), Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}]}],"member":"1968","published-online":{"date-parts":[[2025,4,18]]},"reference":[{"key":"ref_1","unstructured":"(2024, July 13). Art. 17 GDPR\u2014Right to Erasure (\u2018Right to Be Forgotten\u2019)\u2014GDPR.eu\u2014Gdpr.eu. Available online: https:\/\/gdpr.eu\/article-17-right-to-be-forgotten\/."},{"key":"ref_2","unstructured":"(2024, July 13). California Consumer Privacy Act (CCPA)\u2014Oag.ca.gov, Available online: https:\/\/oag.ca.gov\/privacy\/ccpa#heading5d."},{"key":"ref_3","unstructured":"Nast, C. (2024, July 13). Now That Machines Can Learn, Can They Unlearn?\u2014Wired.com. Available online: https:\/\/www.wired.com\/story\/machines-can-learn-can-they-unlearn\/."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Shaik, T., Tao, X., Xie, H., Li, L., Zhu, X., and Li, Q. (2024). Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy. arXiv.","DOI":"10.1109\/TNNLS.2024.3486109"},{"key":"ref_5","unstructured":"Li, C. (2024, July 13). OpenAI\u2019s GPT-3 Language Model: A Technical Overview\u2014Lambdalabs.com. Available online: https:\/\/lambdalabs.com\/blog\/demystifying-gpt-3."},{"key":"ref_6","first-page":"1","article-title":"Procedures for performing systematic reviews","volume":"33","author":"Kitchenham","year":"2004","journal-title":"Keele Uk Keele Univ."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Liu, Y., Ma, S., Aafer, Y., Lee, W.C., Zhai, J., Wang, W., and Zhang, X. (2018, January 18\u201321). Trojaning Attack on Neural Networks. Proceedings of the Network and Distributed System Security Symposium, San Diego, CA, USA.","DOI":"10.14722\/ndss.2018.23291"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Felps, D.L., Schwickerath, A.D., Williams, J.D., Vuong, T.N., Briggs, A., Hunt, M., Sakmar, E., Saranchak, D.D., and Shumaker, T. (2020). Class Clown: Data Redaction in Machine Unlearning at Enterprise Scale. arXiv.","DOI":"10.5220\/0010419600070014"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"13046","DOI":"10.1109\/TNNLS.2023.3266233","article-title":"Fast Yet Effective Machine Unlearning","volume":"35","author":"Tarun","year":"2024","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_10","unstructured":"Zhou, Z.H. (2021, January 19\u201327). Learning with Selective Forgetting. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21; International Joint Conferences on Artificial Intelligence Organization, Montreal, QC, Canada."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C.A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. (2020). Machine Unlearning. arXiv.","DOI":"10.1109\/SP40001.2021.00019"},{"key":"ref_12","unstructured":"Gupta, V., Jung, C., Neel, S., Roth, A., Sharifi-Malvajerdi, S., and Waites, C. (2021). Adaptive Machine Unlearning. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Koch, K., and Soll, M. (2023, January 8\u201310). No Matter How You Slice It: Machine Unlearning with SISA Comes at the Expense of Minority Classes. Proceedings of the 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), Raleigh, NC, USA.","DOI":"10.1109\/SaTML54575.2023.00047"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"88137","DOI":"10.1109\/ACCESS.2021.3090019","article-title":"Coded Machine Unlearning","volume":"9","author":"Aldaghri","year":"2021","journal-title":"IEEE Access"},{"key":"ref_15","unstructured":"He, Y., Meng, G., Chen, K., He, J., and Hu, X. (2021). DeepObliviate: A Powerful Charm for Erasing Data Residual Memory in Deep Neural Networks. arXiv."},{"key":"ref_16","unstructured":"Raedt, L.D. (2022, January 23\u201329). ARCANE: An Efficient Architecture for Exact Machine Unlearning. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22; International Joint Conferences on Artificial Intelligence Organization, Vienna, Austria."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Graves, L., Nagisetty, V., and Ganesh, V. (2020). Amnesiac Machine Learning. arXiv.","DOI":"10.1609\/aaai.v35i13.17371"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. (2022). Unrolling SGD: Understanding Factors Influencing Machine Unlearning. arXiv.","DOI":"10.1109\/EuroSP53844.2022.00027"},{"key":"ref_19","unstructured":"Liu, Y., Ma, Z., Liu, X., Liu, J., Jiang, Z., Ma, J., Yu, P., and Ren, K. (2021). Learn to Forget: Machine Unlearning via Neuron Masking. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Liu, Y., Fan, M., Chen, C., Liu, X., Ma, Z., Wang, L., and Ma, J. (2022). Backdoor Defense with Machine Unlearning. arXiv.","DOI":"10.1109\/INFOCOM48880.2022.9796974"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Chundawat, V.S., Tarun, A.K., Mandal, M., and Kankanhalli, M. (2023). Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher. arXiv.","DOI":"10.1609\/aaai.v37i6.25879"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2345","DOI":"10.1109\/TIFS.2023.3265506","article-title":"Zero-Shot Machine Unlearning","volume":"18","author":"Chundawat","year":"2023","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kim, J., and Woo, S.S. (2022, January 19\u201320). Efficient Two-stage Model Retraining for Machine Unlearning. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA.","DOI":"10.1109\/CVPRW56347.2022.00482"},{"key":"ref_24","unstructured":"Kurmanji, M., Triantafillou, P., Hayes, J., and Triantafillou, E. (2023). Towards Unbounded Machine Unlearning. arXiv."},{"key":"ref_25","unstructured":"Chen, K., Wang, Y., and Huang, Y. (2021). Lightweight machine unlearning in neural network. arXiv."},{"key":"ref_26","unstructured":"Tarun, A.K., Chundawat, V.S., Mandal, M., and Kankanhalli, M. (2023). Deep Regression Unlearning. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Golatkar, A., Achille, A., and Soatto, S. (2020). Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks. arXiv.","DOI":"10.1109\/CVPR42600.2020.00932"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Golatkar, A., Achille, A., and Soatto, S. (2020). Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations. arXiv.","DOI":"10.1007\/978-3-030-58526-6_23"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Golatkar, A., Achille, A., Ravichandran, A., Polito, M., and Soatto, S. (2021). Mixed-Privacy Forgetting in Deep Networks. arXiv.","DOI":"10.1109\/CVPR46437.2021.00085"},{"key":"ref_30","unstructured":"Guo, C., Goldstein, T., Hannun, A., and van der Maaten, L. (2023). Certified Data Removal from Machine Learning Models. arXiv."},{"key":"ref_31","unstructured":"Izzo, Z., Smart, M.A., Chaudhuri, K., and Zou, J. (2021). Approximate Data Deletion from Machine Learning Models. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wu, G., Hashemi, M., and Srinivasa, C. (2022). PUMA: Performance Unchanged Model Augmentation for Training Data Removal. arXiv.","DOI":"10.1609\/aaai.v36i8.20846"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Warnecke, A., Pirch, L., Wressnegger, C., and Rieck, K. (2023). Machine Unlearning of Features and Labels. arXiv.","DOI":"10.14722\/ndss.2023.23087"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Liu, G., Ma, X., Yang, Y., Wang, C., and Liu, J. (2021, January 25\u201328). FedEraser: Enabling Efficient Client-Level Data Removal from Federated Learning Models. Proceedings of the 2021 IEEE\/ACM 29th International Symposium on Quality of Service (IWQOS), Tokyo, Japan.","DOI":"10.1109\/IWQOS52092.2021.9521274"},{"key":"ref_35","unstructured":"Wu, C., Zhu, S., and Mitra, P. (2022). Federated Unlearning with Knowledge Distillation. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, J., Guo, S., Xie, X., and Qi, H. (2022). Federated Unlearning via Class-Discriminative Pruning. arXiv.","DOI":"10.1145\/3485447.3512222"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Liu, Y., Xu, L., Yuan, X., Wang, C., and Li, B. (2022, January 2\u20135). The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining. Proceedings of the IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, London, UK.","DOI":"10.1109\/INFOCOM48880.2022.9796721"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Cao, X., Jia, J., Zhang, Z., and Gong, N.Z. (2022). FedRecover: Recovering from Poisoning Attacks in Federated Learning using Historical Information. arXiv.","DOI":"10.1109\/SP46215.2023.10179336"},{"key":"ref_39","unstructured":"Li, G., Shen, L., Sun, Y., Hu, Y., Hu, H., and Tao, D. (2023). Subspace based Federated Unlearning. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Su, N., and Li, B. (2023). Asynchronous Federated Unlearning. arXiv.","DOI":"10.1109\/INFOCOM53939.2023.10229075"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Ye, G., Chen, T., Nguyen, Q.V.H., and Yin, H. (2023). Heterogeneous Decentralized Machine Unlearning with Seed Model Distillation. arXiv.","DOI":"10.21203\/rs.3.rs-2439540\/v1"},{"key":"ref_42","unstructured":"Halimi, A., Kadhe, S., Rawat, A., and Baracaldo, N. (2023). Federated Unlearning: How to Efficiently Erase a Client in FL?. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Dhasade, A., Ding, Y., Guo, S., marie Kermarrec, A., Vos, M.D., and Wu, L. (2023). QuickDrop: Efficient Federated Unlearning by Integrated Dataset Distillation. arXiv.","DOI":"10.1145\/3652892.3700764"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Cao, Y., and Yang, J. (2015, January 17\u201321). Towards Making Systems Forget with Machine Unlearning. Proceedings of the 2015 IEEE Symposium on Security and Privacy, San Jose, CA, USA.","DOI":"10.1109\/SP.2015.35"},{"key":"ref_45","unstructured":"Sekhari, A., Acharya, J., Kamath, G., and Suresh, A.T. (2021). Remember What You Want to Forget: Algorithms for Machine Unlearning. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1561\/0400000042","article-title":"The algorithmic foundations of differential privacy","volume":"9","author":"Dwork","year":"2013","journal-title":"Found. Trends\u00ae Theor. Comput. Sci."},{"key":"ref_47","first-page":"19","article-title":"Federated learning: Applications, challenges and future directions","volume":"18","author":"Bharati","year":"2022","journal-title":"Int. J. Hybrid Intell. Syst."},{"key":"ref_48","unstructured":"Shao, J., Lin, T., Cao, X., and Luo, B. (2024). Federated Unlearning: A Perspective of Stability and Fairness. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1007\/s00769-006-0191-z","article-title":"Understanding the meaning of accuracy, trueness and precision","volume":"12","author":"Menditto","year":"2007","journal-title":"Accredit. Qual. Assur."},{"key":"ref_50","first-page":"16","article-title":"The Streisand Effect and Censorship Backfire","volume":"9","author":"Jansen","year":"2015","journal-title":"Int. J. Commun."},{"key":"ref_51","unstructured":"Serr\u00e0, J., Sur\u00eds, D., Miron, M., and Karatzoglou, A. (2018). Overcoming catastrophic forgetting with hard attention to the task. arXiv."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1109\/MC.2023.3333319","article-title":"Learn to unlearn: Insights into machine unlearning","volume":"57","author":"Qu","year":"2024","journal-title":"IEEE Computer"},{"key":"ref_53","unstructured":"Wang, R. (2025). Redefining Machine Unlearning: A Conformal Prediction-Motivated Approach. arXiv."},{"key":"ref_54","unstructured":"Li, J., Wei, Q., Zhang, C., Qi, G., Du, M., and Chen, Y. (2024). Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models. arXiv."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Shrivastava, A., and Ahirwal, M.K. (2024, January 8\u20139). Refining of Learning in Human Decision Making Models: A Step Towards Machine Unlearning. Proceedings of the 2024 International Conference on Integrated Circuits, Communication, and Computing Systems (ICIC3S), Una, India.","DOI":"10.1109\/ICIC3S61846.2024.10603026"},{"key":"ref_56","unstructured":"Baumhauer, T., Sch\u00f6ttle, P., and Zeppelzauer, M. (2020). Machine Unlearning: Linear Filtration for Logit-based Classifiers. arXiv."},{"key":"ref_57","unstructured":"Chen, R., Yang, J., Xiong, H., Bai, J., Hu, T., Hao, J., Feng, Y., Zhou, J.T., Wu, J., and Liu, Z. (2023). Fast Model Debias with Machine Unlearning. arXiv."},{"key":"ref_58","unstructured":"Jia, J., Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., Sharma, P., and Liu, S. (2024). Model Sparsity Can Simplify Machine Unlearning. arXiv."},{"key":"ref_59","unstructured":"Loog, M., and Viering, T. (2022). A Survey of Learning Curves with Bad Behavior: Or How More Data Need Not Lead to Better Performance. arXiv."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_61","unstructured":"Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2019). MobileNetV2: Inverted Residuals and Linear Bottlenecks. arXiv.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K.Q. (2018). Densely Connected Convolutional Networks. arXiv.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"6255","DOI":"10.1038\/s41467-023-41703-x","article-title":"A Unified Method to Revoke the Private Data of Patients in Intelligent Healthcare with Audit to Forget","volume":"14","author":"Zhou","year":"2023","journal-title":"Nat. Commun."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Lindstrom, C. Evaluation Metrics for Machine Unlearning. Preprints, 2024.","DOI":"10.20944\/preprints202409.1925.v1"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"G\u00fcndo\u011fdu, E., Unal, A., and Unal, G. (2024, January 27\u201331). A Study Regarding Machine Unlearning on Facial Attribute Data. Proceedings of the 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), Istanbul, Turkiye.","DOI":"10.1109\/FG59268.2024.10581972"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1007\/s42979-023-01767-4","article-title":"A Review on Machine Unlearning","volume":"4","author":"Zhang","year":"2023","journal-title":"Sn Comput. Sci."},{"key":"ref_68","unstructured":"Solanki, R., and Creager, E. (2024). Promoting User Data Autonomy During the Dissolution of a Monopolistic Firm. arXiv."},{"key":"ref_69","unstructured":"Li, G. (2024). Theoretically-Grounded Efficient Deep Learning System Design. [Doctoral Dissertation, University of Texas Institutional Repository]."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/4\/150\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:16:56Z","timestamp":1760030216000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/4\/150"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,18]]},"references-count":69,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["computers14040150"],"URL":"https:\/\/doi.org\/10.3390\/computers14040150","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,18]]}}}