{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T22:29:54Z","timestamp":1782772194568,"version":"3.54.5"},"reference-count":121,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,1,13]],"date-time":"2025-01-13T00:00:00Z","timestamp":1736726400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,1,13]],"date-time":"2025-01-13T00:00:00Z","timestamp":1736726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007512","name":"Universitat Rovira i Virgili","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007512","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Artif Intell Rev"],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Large language models (LLMs) have become the state of the art in natural language processing. The massive adoption of generative LLMs and the capabilities they have shown have prompted public concerns regarding their impact on the labor market, privacy, the use of copyrighted work, and how these models align with human ethics and the rule of law. As a response, new regulations are being pushed, which require developers and service providers to evaluate, monitor, and forestall or at least mitigate the risks posed by their models. One mitigation strategy is digital forgetting: given a model with undesirable knowledge or behavior, the goal is to obtain a new model where the detected issues are no longer present. Digital forgetting is usually enforced via machine unlearning techniques, which modify trained machine learning models for them to behave as models trained on a subset of the original training data. In this work, we describe the motivations and desirable properties of digital forgetting when applied to LLMs, and we survey recent works on machine unlearning. Specifically, we propose a taxonomy of unlearning methods based on the reach and depth of the modifications done on the models, we discuss and compare the effectiveness of machine unlearning methods for LLMs proposed so far, and we survey their evaluation. Finally, we describe open problems of machine unlearning applied to LLMs and we put forward recommendations for developers and practitioners.<\/jats:p>","DOI":"10.1007\/s10462-024-11078-6","type":"journal-article","created":{"date-parts":[[2025,1,13]],"date-time":"2025-01-13T06:37:17Z","timestamp":1736750237000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Digital forgetting in large language models: a survey of unlearning methods"],"prefix":"10.1007","volume":"58","author":[{"given":"Alberto","family":"Blanco-Justicia","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Najeeb","family":"Jebreel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benet","family":"Manzanares-Salor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"S\u00e1nchez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Josep","family":"Domingo-Ferrer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillem","family":"Collell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kuan","family":"Eeik Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,1,13]]},"reference":[{"key":"11078_CR1","doi-asserted-by":"crossref","unstructured":"Abadi M, Chu A, Goodfellow I, et al (2016) Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp 308\u2013318","DOI":"10.1145\/2976749.2978318"},{"key":"11078_CR2","doi-asserted-by":"publisher","unstructured":"Amini A, Gabriel S, Lin S, et al (2019) MathQA: Towards interpretable math word problem solving with operation-based formalisms. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp 2357\u20132367, https:\/\/doi.org\/10.18653\/v1\/N19-1245, https:\/\/aclanthology.org\/N19-1245","DOI":"10.18653\/v1\/N19-1245"},{"key":"11078_CR3","unstructured":"Belrose N, Schneider-Joseph D, Ravfogel S, et al (2023) Leace: Perfect linear concept erasure in closed form. Preprint at arXiv:2306.03819"},{"key":"11078_CR4","unstructured":"Biderman S, Schoelkopf H, Anthony QG, et al (2023) Pythia: A suite for analyzing large language models across training and scaling. In: International Conference on Machine Learning, PMLR, pp 2397\u20132430"},{"key":"11078_CR5","doi-asserted-by":"publisher","unstructured":"Bisk Y, Zellers R, Bras RL, et al (2020) PIQA: reasoning about physical commonsense in natural language. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, pp 7432\u2013743https:\/\/doi.org\/10.1609\/AAAI.V34I05.6239,","DOI":"10.1609\/AAAI.V34I05.6239"},{"key":"11078_CR6","unstructured":"Black S, Gao L, Wang P, et al (2022) Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow, 2021. URL: https:\/\/doiorg\/105281\/zenodo5297715"},{"key":"11078_CR7","doi-asserted-by":"publisher","unstructured":"Borkan D, Dixon L, Sorensen J, et al (2019) Nuanced metrics for measuring unintended bias with real data for text classification. In: Companion Proceedings of The 2019 World Wide Web Conference. Association for Computing Machinery, New York, NY, USA, WWW \u201919, p 491\u2013500, https:\/\/doi.org\/10.1145\/3308560.3317593,","DOI":"10.1145\/3308560.3317593"},{"key":"11078_CR8","unstructured":"Borkar J (2023) What can we learn from data leakage and unlearning for law? Preprint at arXiv:2307.10476"},{"key":"11078_CR9","doi-asserted-by":"crossref","unstructured":"Bourtoule L, Chandrasekaran V, Choquette-Choo CA, et al (2021) Machine unlearning. In: 2021 IEEE Symposium on Security and Privacy (SP), IEEE, pp 141\u2013159","DOI":"10.1109\/SP40001.2021.00019"},{"key":"11078_CR10","unstructured":"Carlini N, Tramer F, Wallace E, et al (2021) Extracting training data from large language models. In: 30th USENIX Security Symposium (USENIX Security 21), pp 2633\u20132650"},{"key":"11078_CR11","unstructured":"Cettolo M, Niehues J, St\u00fcker S, et al (2014) Report on the 11th IWSLT evaluation campaign. In: Federico M, St\u00fcker S, Yvon F (eds) Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign@IWSLT 2014, Lake Tahoe, CA, USA, December 4-5, 2014, https:\/\/aclanthology.org\/2014.iwslt-evaluation.1"},{"key":"11078_CR12","doi-asserted-by":"crossref","unstructured":"Chen J, Yang D (2023) Unlearn what you want to forget: Efficient unlearning for llms. Preprint at arXiv:2310.20150","DOI":"10.18653\/v1\/2023.emnlp-main.738"},{"key":"11078_CR13","unstructured":"Clark K, Luong M, Le QV, et al (2020) ELECTRA: pre-training text encoders as discriminators rather than generators. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, https:\/\/openreview.net\/forum?id=r1xMH1BtvB"},{"key":"11078_CR14","unstructured":"Clark P, Cowhey I, Etzioni O, et al (2018) Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR abs\/1803.05457. arXiv:1803.05457"},{"key":"11078_CR15","unstructured":"Computer T (2023) Redpajama: an open dataset for training large language models. https:\/\/github.com\/togethercomputer\/RedPajama-Data"},{"key":"11078_CR16","doi-asserted-by":"crossref","unstructured":"Daheim N, Dziri N, Sachan M, et al (2023) Elastic weight removal for faithful and abstractive dialogue generation. Preprint at arXiv:2303.17574","DOI":"10.18653\/v1\/2024.naacl-long.393"},{"key":"11078_CR17","doi-asserted-by":"publisher","unstructured":"De-Arteaga M, Romanov A, Wallach HM, et al (2019) Bias in bios: A case study of semantic representation bias in a high-stakes setting. In: danah boyd, Morgenstern JH (eds) Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29-31, 2019. ACM, pp 120\u2013128, https:\/\/doi.org\/10.1145\/3287560.3287572, https:\/\/doi.org\/10.1145\/3287560.3287572","DOI":"10.1145\/3287560.3287572"},{"key":"11078_CR18","unstructured":"Devlin J, Chang MW, Lee K, et al (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint at arXiv:1810.04805"},{"key":"11078_CR19","doi-asserted-by":"publisher","unstructured":"Devlin J, Chang M, Lee K, et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp 4171\u2013418https:\/\/doi.org\/10.18653\/V1\/N19-1423","DOI":"10.18653\/V1\/N19-1423"},{"key":"11078_CR20","unstructured":"Dinan E, Roller S, Shuster K, et al (2019) Wizard of wikipedia: Knowledge-powered conversational agents. In: International Conference on Learning Representations"},{"key":"11078_CR21","unstructured":"Eldan R, Russinovich M (2023) Who\u2019s harry potter? approximate unlearning in llms. Preprint at arXiv:2310.02238"},{"key":"11078_CR22","doi-asserted-by":"publisher","unstructured":"Commission European (2019) Ethics guidelines for trustworthy AI. Publications Office of the EU, doi\/. https:\/\/doi.org\/10.2759\/346720","DOI":"10.2759\/346720"},{"key":"11078_CR23","unstructured":"European Parliament (2024) European parliament legislative resolution of 13 march 2024 on the proposal for a regulation of the european parliament and of the council on laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (com(2021)0206 \u2013 c9-0146\/2021 \u2013 2021\/0106(cod)). https:\/\/www.europarl.europa.eu\/doceo\/document\/TA-9-2024-0138_EN.html"},{"key":"#cr-split#-11078_CR24.1","unstructured":"European Parliament, Council of the European Union (2016) Regulation"},{"key":"#cr-split#-11078_CR24.2","unstructured":"(EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation). https:\/\/data.europa.eu\/eli\/reg\/2016\/679\/oj"},{"key":"11078_CR25","doi-asserted-by":"publisher","unstructured":"Fan A, Lewis M, Dauphin YN (2018) Hierarchical neural story generation. In: Gurevych I, Miyao Y (eds) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. Association for Computational Linguistics, pp 889\u201389https:\/\/doi.org\/10.18653\/V1\/P18-1082, https:\/\/aclanthology.org\/P18-1082\/","DOI":"10.18653\/V1\/P18-1082"},{"key":"11078_CR26","unstructured":"Gallegos IO, Rossi RA, Barrow J, et al (2024) Self-debiasing large language models: Zero-shot recognition and reduction of stereotypes. Preprint at arXiv:2402.01981"},{"key":"11078_CR27","unstructured":"Gao L, Biderman S, Black S, et al (2021) The pile: An 800gb dataset of diverse text for language modeling. CoRR abs\/2101.00027. arXiv:2101.00027"},{"key":"11078_CR28","doi-asserted-by":"crossref","unstructured":"Gehman S, Gururangan S, Sap M, et al (2020) Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In: Findings","DOI":"10.18653\/v1\/2020.findings-emnlp.301"},{"key":"11078_CR29","doi-asserted-by":"crossref","unstructured":"Gliwa B, Mochol I, Biesek M, et al (2019) Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. CoRR abs\/1911.12237. arXiv:1911.12237","DOI":"10.18653\/v1\/D19-5409"},{"key":"11078_CR30","unstructured":"Gordon A, Kozareva Z, Roemmele M (2012) SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In: Agirre E, Bos J, Diab M, et al (eds) *SEM 2012: The First Joint Conference on Lexical and Computational Semantics \u2013 Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012). Association for Computational Linguistics, Montr\u00e9al, Canada, pp 394\u2013398, https:\/\/aclanthology.org\/S12-1052"},{"key":"11078_CR31","unstructured":"Group of Seven (G7) (2023) Hiroshima process international guiding principles for organizations developing advanced ai systems. https:\/\/www.mofa.go.jp\/files\/100573471.pdf"},{"key":"11078_CR32","doi-asserted-by":"crossref","unstructured":"Hassan F, S\u00e1nchez D, Soria-Comas J, et al (2019) Automatic anonymization of textual documents: detecting sensitive information via word embeddings. In: 2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications\/13th IEEE International Conference On Big Data Science And Engineering (TrustCom\/BigDataSE), IEEE, pp 358\u2013365","DOI":"10.1109\/TrustCom\/BigDataSE.2019.00055"},{"key":"11078_CR33","doi-asserted-by":"crossref","unstructured":"Hassan F, S\u00e1\u00a1nchez D, Domingo-Ferrer J (2023) Utility-preserving privacy protection of textual documents via word embeddings. IEEE Transactions on Knowledge and Data Engineering 35(1):1058\u20131071","DOI":"10.1109\/TKDE.2021.3076632"},{"key":"11078_CR34","unstructured":"Houlsby N, Giurgiu A, Jastrzebski S, et al (2019) Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning, PMLR, pp 2790\u20132799"},{"key":"11078_CR35","unstructured":"Ilharco G, Ribeiro MT, Wortsman M, et al (2022) Editing models with task arithmetic. Preprint at arXiv:2212.04089"},{"key":"11078_CR36","unstructured":"Jang J, Yoon D, Yang S, et al (2022) Knowledge unlearning for mitigating privacy risks in language models. Preprint at arXiv:2210.01504"},{"key":"11078_CR37","doi-asserted-by":"publisher","unstructured":"Ji J, Liu M, Dai J, et al (2023) Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. CoRR abs\/2307.04657. https:\/\/doi.org\/10.48550\/ARXIV.2307.04657, arXiv:2307.04657","DOI":"10.48550\/ARXIV.2307.04657"},{"key":"11078_CR38","doi-asserted-by":"crossref","unstructured":"Jin Q, Dhingra B, Liu Z, et al. (2019) Pubmedqa: A dataset for biomedical research question answering. In: Inui K, Jiang J, Ng V, et al. (eds) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3\u20137, 2019. Association for Computational Linguistics, pp 2567\u20132577, https:\/\/doi.org\/10.18653\/V1\/D19-1259","DOI":"10.18653\/v1\/D19-1259"},{"key":"11078_CR39","unstructured":"Joseph R. Biden Jr. (2023) Executive order on the safe, secure, and trustworthy development and use of artificial intelligence. https:\/\/www.whitehouse.gov\/briefing-room\/presidential-actions\/2023\/10\/30\/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence\/"},{"key":"11078_CR40","unstructured":"Kadhe S, Halimi A, Rawat A, et al (2023) Fairsisa: Ensemble post-processing to improve fairness of unlearning in llms. In: Socially Responsible Language Modelling Research"},{"key":"11078_CR41","unstructured":"Kandpal N, Wallace E, Raffel C (2022) Deduplicating training data mitigates privacy risks in language models. arXiv:2202.06539"},{"key":"11078_CR42","unstructured":"Klimt B, Yang Y (2004) Introducing the enron corpus. In: CEAS 2004 - First Conference on Email and Anti-Spam, July 30-31, 2004, Mountain View, California, USA, http:\/\/www.ceas.cc\/papers-2004\/168.pdf"},{"key":"11078_CR43","doi-asserted-by":"publisher","unstructured":"Komeili M, Shuster K, Weston J (2022) Internet-augmented dialogue generation. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, pp 8460\u20138478, https:\/\/doi.org\/10.18653\/v1\/2022.acl-long.579, https:\/\/aclanthology.org\/2022.acl-long.579","DOI":"10.18653\/v1\/2022.acl-long.579"},{"key":"11078_CR44","unstructured":"Kumar VB, Gangadharaiah R, Roth D (2022) Privacy adhering machine un-learning in nlp. Preprint at arXiv:2212.09573"},{"key":"11078_CR45","unstructured":"Lan Z, Chen M, Goodman S, et al (2020) ALBERT: A lite BERT for self-supervised learning of language representations. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, https:\/\/openreview.net\/forum?id=H1eA7AEtvS"},{"key":"11078_CR46","doi-asserted-by":"crossref","unstructured":"Leong CT, Cheng Y, Wang J, et al (2023) Self-detoxifying language models via toxification reversal. Preprint at arXiv:2310.09573","DOI":"10.18653\/v1\/2023.emnlp-main.269"},{"key":"11078_CR47","doi-asserted-by":"publisher","unstructured":"Levy O, Seo M, Choi E, et al (2017) Zero-shot relation extraction via reading comprehension. In: Levy R, Specia L (eds) Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics, pp 333\u2013342, https:\/\/doi.org\/10.18653\/V1\/K17-1034","DOI":"10.18653\/V1\/K17-1034"},{"key":"11078_CR48","doi-asserted-by":"crossref","unstructured":"Li J, Cheng X, Zhao X, et al (2023a) Halueval: A large-scale hallucination evaluation benchmark for large language models. In: Bouamor H, Pino J, Bali K (eds) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, pp 6449\u20136464, https:\/\/aclanthology.org\/2023.emnlp-main.397","DOI":"10.18653\/v1\/2023.emnlp-main.397"},{"key":"11078_CR49","unstructured":"Li Y, Bubeck S, Eldan R, et al (2023b) Textbooks are all you need ii: phi-1.5 technical report. Preprint at arXiv:2309.05463"},{"key":"11078_CR50","doi-asserted-by":"crossref","unstructured":"Limisiewicz T, Mare\u010dek D (2022) Don\u2019t forget about pronouns: Removing gender bias in language models without losing factual gender information. Preprint at arXiv:2206.10744","DOI":"10.18653\/v1\/2022.gebnlp-1.3"},{"key":"11078_CR51","doi-asserted-by":"publisher","unstructured":"Lin S, Hilton J, Evans O (2022) Truthfulqa: Measuring how models mimic human falsehoods. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, pp 3214\u20133252, https:\/\/doi.org\/10.18653\/V1\/2022.ACL-LONG.229,","DOI":"10.18653\/V1\/2022.ACL-LONG.229"},{"key":"11078_CR52","doi-asserted-by":"crossref","unstructured":"Lison P, Pil\u00e1n I, S\u00e1nchez D, et al (2021) Anonymisation models for text data: State of the art, challenges and future directions. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp 4188\u20134203","DOI":"10.18653\/v1\/2021.acl-long.323"},{"issue":"9","key":"11078_CR53","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3560815","volume":"55","author":"P Liu","year":"2023","unstructured":"Liu P, Yuan W, Fu J et al (2023) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comp Surv 55(9):1\u201335","journal-title":"ACM Comp Surv"},{"key":"11078_CR54","unstructured":"Liu Y, Ott M, Goyal N, et al (2019) Roberta: A robustly optimized BERT pretraining approach. CoRR abs\/1907.11692. arXiv:1907.11692"},{"key":"11078_CR55","doi-asserted-by":"crossref","unstructured":"Liu Z, Kalinli O (2023) Forgetting private textual sequences in language models via leave-one-out ensemble. Preprint at arXiv:2309.16082","DOI":"10.1109\/ICASSP48485.2024.10446299"},{"key":"11078_CR56","first-page":"27591","volume":"35","author":"X Lu","year":"2022","unstructured":"Lu X, Welleck S, Hessel J et al (2022) Quark: controllable text generation with reinforced unlearning. Adv Neural Inf Process Syst 35:27591\u201327609","journal-title":"Adv Neural Inf Process Syst"},{"key":"11078_CR57","unstructured":"Maas AL, Daly RE, Pham PT, et al (2011) Learning word vectors for sentiment analysis. In: Lin D, Matsumoto Y, Mihalcea R (eds) The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA. The Association for Computer Linguistics, pp 142\u2013150, https:\/\/aclanthology.org\/P11-1015\/"},{"key":"11078_CR58","doi-asserted-by":"crossref","unstructured":"Madaan A, Tandon N, Clark P, et al (2022) Memory-assisted prompt editing to improve gpt-3 after deployment. Preprint at arXiv:2201.06009","DOI":"10.18653\/v1\/2022.emnlp-main.183"},{"key":"11078_CR59","unstructured":"Meng K, Bau D, Andonian A, et al (2022) Locating and editing factual knowledge in GPT. CoRR abs\/2202.05262. arXiv:2202.05262"},{"key":"11078_CR60","unstructured":"Merity S, Xiong C, Bradbury J, et al (2016) Pointer sentinel mixture models. arXiv:1609.07843"},{"key":"11078_CR61","unstructured":"Mitchell E, Lin C, Bosselut A, et al (2022) Memory-based model editing at scale. In: International Conference on Machine Learning, PMLR, pp 15817\u201315831"},{"key":"11078_CR62","doi-asserted-by":"publisher","unstructured":"Nadeem M, Bethke A, Reddy S (2021) StereoSet: Measuring stereotypical bias in pretrained language models. In: Zong C, Xia F, Li W, et al (eds) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, pp 5356\u20135371, https:\/\/doi.org\/10.18653\/v1\/2021.acl-long.416, https:\/\/aclanthology.org\/2021.acl-long.416","DOI":"10.18653\/v1\/2021.acl-long.416"},{"key":"11078_CR63","doi-asserted-by":"publisher","unstructured":"Nangia N, Vania C, Bhalerao R, et al (2020) CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In: Webber B, Cohn T, He Y, et al (eds) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, pp 1953\u20131967, https:\/\/doi.org\/10.18653\/v1\/2020.emnlp-main.154, https:\/\/aclanthology.org\/2020.emnlp-main.154","DOI":"10.18653\/v1\/2020.emnlp-main.154"},{"key":"11078_CR64","unstructured":"Nasr M, Carlini N, Hayase J, et al (2023) Scalable extraction of training data from (production) language models. Preprint at arXiv:2311.17035"},{"key":"11078_CR65","unstructured":"Nguyen TT, Huynh TT, Nguyen PL, et al (2022) A survey of machine unlearning. Preprint at arXiv:2209.02299"},{"key":"11078_CR66","unstructured":"Ni S, Chen D, Li C, et al (2023) Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models. Preprint at arXiv:2311.08011"},{"key":"11078_CR67","unstructured":"Nivre J, de Marneffe M, Ginter F, et al (2020) Universal dependencies v2: An evergrowing multilingual treebank collection. In: Calzolari N, B\u00e9chet F, Blache P, et al (eds) Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020. European Language Resources Association, pp 4034\u20134043, https:\/\/aclanthology.org\/2020.lrec-1.497\/"},{"key":"11078_CR68","doi-asserted-by":"publisher","unstructured":"Paperno D, Kruszewski G, Lazaridou A, et al (2016) The LAMBADA dataset: Word prediction requiring a broad discourse context. In: Erk K, Smith NA (eds) Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pp 1525\u20131534, https:\/\/doi.org\/10.18653\/v1\/P16-1144, https:\/\/aclanthology.org\/P16-1144","DOI":"10.18653\/v1\/P16-1144"},{"key":"11078_CR69","unstructured":"Papernot N, Song S, Mironov I, et al (2018) Scalable private learning with pate. Preprint at arXiv:1802.08908"},{"key":"11078_CR70","unstructured":"Patil V, Hase P, Bansal M (2023) Can sensitive information be deleted from llms? objectives for defending against extraction attacks. Preprint at arXiv:2309.17410"},{"key":"11078_CR71","unstructured":"Pawelczyk M, Neel S, Lakkaraju H (2023) In-context unlearning: Language models as few shot unlearners. Preprint at arXiv:2310.07579"},{"key":"11078_CR72","unstructured":"Pochinkov N, Schoots N (2023) Dissecting large language models. In: Socially Responsible Language Modelling Research"},{"key":"11078_CR73","unstructured":"Qu Y, Yuan X, Ding M, et al (2023) Learn to unlearn: A survey on machine unlearning. Preprint at arXiv:2305.07512"},{"key":"11078_CR74","unstructured":"Radford A, Narasimhan K, Salimans T, et al (2018) Improving language understanding by generative pre-training. https:\/\/openai.com\/index\/language-unsupervised\/"},{"issue":"8","key":"11078_CR75","first-page":"9","volume":"1","author":"A Radford","year":"2019","unstructured":"Radford A, Wu J, Child R et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9","journal-title":"OpenAI blog"},{"issue":"1","key":"11078_CR76","first-page":"5485","volume":"21","author":"C Raffel","year":"2020","unstructured":"Raffel C, Shazeer N, Roberts A et al (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(1):5485\u20135551","journal-title":"J Mach Learn Res"},{"key":"11078_CR77","doi-asserted-by":"publisher","unstructured":"Rashkin H, Smith EM, Li M, et al (2019) Towards empathetic open-domain conversation models: A new benchmark and dataset. In: Korhonen A, Traum D, M\u00e0rquez L (eds) Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, pp 5370\u20135381, https:\/\/doi.org\/10.18653\/v1\/P19-1534, https:\/\/aclanthology.org\/P19-1534","DOI":"10.18653\/v1\/P19-1534"},{"key":"11078_CR78","doi-asserted-by":"crossref","unstructured":"Ravfogel S, Elazar Y, Gonen H, et al (2020) Null it out: Guarding protected attributes by iterative nullspace projection. Preprint at arXiv:2004.07667","DOI":"10.18653\/v1\/2020.acl-main.647"},{"key":"11078_CR79","unstructured":"Rowling JK (2000) Harry Potter and the Sorcerer\u2019s Stone. Bloomsbury"},{"key":"11078_CR80","doi-asserted-by":"publisher","unstructured":"Rudinger R, Naradowsky J, Leonard B, et al (2018) Gender bias in coreference resolution. In: Walker M, Ji H, Stent A (eds) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, New Orleans, Louisiana, pp 8\u201314, https:\/\/doi.org\/10.18653\/v1\/N18-2002, https:\/\/aclanthology.org\/N18-2002","DOI":"10.18653\/v1\/N18-2002"},{"key":"11078_CR81","doi-asserted-by":"publisher","unstructured":"Sakaguchi K, Bras RL, Bhagavatula C, et al (2020) Winogrande: An adversarial winograd schema challenge at scale. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, pp 8732\u20138740, https:\/\/doi.org\/10.1609\/AAAI.V34I05.6399","DOI":"10.1609\/AAAI.V34I05.6399"},{"key":"11078_CR82","doi-asserted-by":"crossref","unstructured":"Salem A, Zhang Y, Humbert M, et al (2018) Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. Preprint at arXiv:1806.01246","DOI":"10.14722\/ndss.2019.23119"},{"issue":"1","key":"11078_CR83","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1002\/asi.23363","volume":"67","author":"D S\u00e1nchez","year":"2016","unstructured":"S\u00e1nchez D, Batet M (2016) C-sanitized: a privacy model for document redaction and sanitization. J Assoc Inf Sci Tech 67(1):148\u2013163","journal-title":"J Assoc Inf Sci Tech"},{"key":"11078_CR84","unstructured":"Sanh V, Debut L, Chaumond J, et al (2019) Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint at arXiv:1910.01108"},{"key":"11078_CR85","unstructured":"Scao TL, Fan A, Akiki C, et al (2022) Bloom: A 176b-parameter open-access multilingual language model. Preprint at arXiv:2211.05100"},{"key":"11078_CR86","doi-asserted-by":"publisher","unstructured":"Sellam T, Das D, Parikh AP (2020) BLEURT: learning robust metrics for text generation. In: Jurafsky D, Chai J, Schluter N, et al (eds) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, pp 7881\u20137892, https:\/\/doi.org\/10.18653\/V1\/2020.ACL-MAIN.704","DOI":"10.18653\/V1\/2020.ACL-MAIN.704"},{"key":"11078_CR87","doi-asserted-by":"crossref","unstructured":"Shaik T, Tao X, Xie H, et al (2023) Exploring the landscape of machine unlearning: A survey and taxonomy. Preprint at arXiv:2305.06360","DOI":"10.1109\/TNNLS.2024.3486109"},{"key":"11078_CR88","unstructured":"Shi W, Ajith A, Xia M, et al (2023) Detecting pretraining data from large language models. Preprint at arXiv:2310.16789"},{"key":"11078_CR89","doi-asserted-by":"crossref","unstructured":"Shokri R, Stronati M, Song C, et al (2017) Membership inference attacks against machine learning models. In: 2017 IEEE symposium on security and privacy (SP), IEEE, pp 3\u201318","DOI":"10.1109\/SP.2017.41"},{"key":"11078_CR90","unstructured":"Silveira N, Dozat T, de Marneffe MC, et al (2014) A gold standard dependency corpus for English. In: Calzolari N, Choukri K, Declerck T, et al (eds) Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914). European Language Resources Association (ELRA), Reykjavik, Iceland, pp 2897\u20132904, http:\/\/www.lrec-conf.org\/proceedings\/lrec2014\/pdf\/1089_Paper.pdf"},{"key":"11078_CR91","doi-asserted-by":"publisher","unstructured":"Smith EM, Williamson M, Shuster K, et al (2020) Can you put it all together: Evaluating conversational agents\u2019 ability to blend skills. In: Jurafsky D, Chai J, Schluter N, et al (eds) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, pp 2021\u20132030, https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.183, https:\/\/aclanthology.org\/2020.acl-main.183","DOI":"10.18653\/v1\/2020.acl-main.183"},{"key":"11078_CR92","unstructured":"Smith V, Shamsabadi AS, Ashurst C, et al (2023) Identifying and mitigating privacy risks stemming from language models: A survey. Preprint at arXiv:2310.01424"},{"key":"11078_CR93","doi-asserted-by":"crossref","unstructured":"Socher R, Perelygin A, Wu J, et al (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In: Yarowsky D, Baldwin T, Korhonen A, et al (eds) Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pp 1631\u20131642, https:\/\/aclanthology.org\/D13-1170","DOI":"10.18653\/v1\/D13-1170"},{"key":"11078_CR94","doi-asserted-by":"publisher","unstructured":"Stanovsky G, Smith NA, Zettlemoyer L (2019) Evaluating gender bias in machine translation. In: Korhonen A, Traum D, M\u00e0rquez L (eds) Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, pp 1679\u20131684, https:\/\/doi.org\/10.18653\/v1\/P19-1164, https:\/\/aclanthology.org\/P19-1164","DOI":"10.18653\/v1\/P19-1164"},{"key":"11078_CR95","unstructured":"Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: International conference on machine learning, PMLR, pp 3319\u20133328"},{"key":"11078_CR96","doi-asserted-by":"crossref","unstructured":"S\u00e1\u00a1nchez D, Batet M, (2017) Toward sensitive document release with privacy guarantees. Engineering Applications of Artificial Intelligence 59:23\u201334","DOI":"10.1016\/j.engappai.2016.12.013"},{"key":"11078_CR97","unstructured":"Taylor R, Kardas M, Cucurull G, et al (2022) Galactica: A large language model for science. Preprint at arXiv:2211.09085"},{"key":"11078_CR98","unstructured":"Touvron H, Lavril T, Izacard G, et al (2023a) Llama: Open and efficient foundation language models. arXiv:2302.13971"},{"key":"11078_CR99","unstructured":"Touvron H, Martin L, Stone K, et al (2023b) Llama 2: Open foundation and fine-tuned chat models. efficient foundation language models. arXiv:2307.09288"},{"key":"11078_CR100","unstructured":"Tuggener D, von D\u00e4niken P, Peetz T, et al (2020) LEDGAR: A large-scale multi-label corpus for text classification of legal provisions in contracts. In: Calzolari N, B\u00e9chet F, Blache P, et al (eds) Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020. European Language Resources Association, pp 1235\u20131241, https:\/\/aclanthology.org\/2020.lrec-1.155\/"},{"key":"11078_CR101","unstructured":"Tunstall L, Von Werra L, Wolf T (2022) Natural language processing with transformers. \" O\u2019Reilly Media, Inc.\""},{"key":"11078_CR102","unstructured":"United Nations (1948) Universal Declaration of Human Rights"},{"key":"11078_CR103","doi-asserted-by":"crossref","unstructured":"\u00dcst\u00fcn A, Aryabumi V, Yong ZX, et al (2024) Aya model: An instruction finetuned open-access multilingual language model. Preprint at arXiv:2402.07827","DOI":"10.18653\/v1\/2024.acl-long.845"},{"key":"11078_CR104","unstructured":"Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. Advances in neural information processing systems 30"},{"key":"11078_CR105","doi-asserted-by":"crossref","unstructured":"Wang L, Chen T, Yuan W, et al (2023) Kga: A general machine unlearning framework based on knowledge gap alignment. Preprint at arXiv:2305.06535","DOI":"10.18653\/v1\/2023.acl-long.740"},{"key":"11078_CR106","unstructured":"Wang L, Zeng X, Guo J, et al (2024) Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. Preprint at arXiv:2402.05813"},{"key":"11078_CR107","doi-asserted-by":"publisher","unstructured":"Webster K, Recasens M, Axelrod V et al (2018) Mind the GAP: a balanced corpus of gendered ambiguous pronouns. Trans Assoc Comput Linguistics 6:605\u2013617 https:\/\/doi.org\/10.1162\/tacl_a_00240, URL  https:\/\/aclanthology.org\/Q18-1042","DOI":"10.1162\/tacl_a_00240"},{"key":"11078_CR108","doi-asserted-by":"crossref","unstructured":"Wu X, Li J, Xu M, et al (2023) Depn: Detecting and editing privacy neurons in pretrained language models. Preprint at arXiv:2310.20138","DOI":"10.18653\/v1\/2023.emnlp-main.174"},{"issue":"1","key":"11078_CR109","first-page":"1","volume":"56","author":"H Xu","year":"2023","unstructured":"Xu H, Zhu T, Zhang L et al (2023) Machine unlearning: a survey. ACM Comp Surv 56(1):1\u201336","journal-title":"ACM Comp Surv"},{"key":"11078_CR110","unstructured":"Yao Y, Xu X, Liu Y (2023) Large language model unlearning. Preprint at arXiv:2310.10683"},{"key":"11078_CR111","doi-asserted-by":"crossref","unstructured":"Yeom S, Giacomelli I, Fredrikson M, et al (2018) Privacy risk in machine learning: Analyzing the connection to overfitting. In: 2018 IEEE 31st computer security foundations symposium (CSF), IEEE, pp 268\u2013282","DOI":"10.1109\/CSF.2018.00027"},{"key":"11078_CR112","doi-asserted-by":"crossref","unstructured":"Yoon D, Jang J, Kim S, et al (2023) Gradient ascent post-training enhances language model generalization. Preprint at arXiv:2306.07052","DOI":"10.18653\/v1\/2023.acl-short.74"},{"key":"11078_CR113","first-page":"6032","volume":"2023","author":"C Yu","year":"2023","unstructured":"Yu C, Jeoung S, Kasi A et al (2023) Unlearning bias in language models by partitioning gradients. Find Assoc Comput Linguistics: ACL 2023:6032\u20136048","journal-title":"Find Assoc Comput Linguistics: ACL"},{"key":"11078_CR114","doi-asserted-by":"crossref","unstructured":"Zellers R, Holtzman A, Bisk Y, et al (2019) Hellaswag: Can a machine really finish your sentence? CoRR abs\/1905.07830. arXiv:1905.07830","DOI":"10.18653\/v1\/P19-1472"},{"key":"11078_CR115","unstructured":"Zhang J, Liu J, He J, et al (2024a) Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems 36"},{"key":"11078_CR116","unstructured":"Zhang N, Yao Y, Tian B, et al (2024b) A comprehensive study of knowledge editing for large language models. Preprint at arXiv:2401.01286"},{"key":"11078_CR117","doi-asserted-by":"publisher","unstructured":"Zhang S, Dinan E, Urbanek J, et al (2018) Personalizing dialogue agents: I have a dog, do you have pets too? In: Gurevych I, Miyao Y (eds) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, pp 2204\u20132213https:\/\/doi.org\/10.18653\/v1\/P18-1205, https:\/\/aclanthology.org\/P18-1205","DOI":"10.18653\/v1\/P18-1205"},{"key":"11078_CR118","unstructured":"Zhang S, Roller S, Goyal N, et al (2022) Opt: Open pre-trained transformer language models. arXiv:2205.01068"},{"key":"11078_CR119","unstructured":"Zhang X, Zhao JJ, LeCun Y (2015) Character-level convolutional networks for text classification. In: Cortes C, Lawrence ND, Lee DD, et al (eds) Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp 649\u2013657, https:\/\/proceedings.neurips.cc\/paper\/2015\/hash\/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html"},{"key":"11078_CR120","doi-asserted-by":"publisher","unstructured":"Zhao J, Wang T, Yatskar M, et al (2018) Gender bias in coreference resolution: Evaluation and debiasing methods. In: Walker M, Ji H, Stent A (eds) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, New Orleans, Louisiana, pp 15\u201320, https:\/\/doi.org\/10.18653\/v1\/N18-2003, https:\/\/aclanthology.org\/N18-2003","DOI":"10.18653\/v1\/N18-2003"}],"container-title":["Artificial Intelligence Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-024-11078-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10462-024-11078-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-024-11078-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,3]],"date-time":"2025-02-03T11:32:49Z","timestamp":1738582369000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10462-024-11078-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,13]]},"references-count":121,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["11078"],"URL":"https:\/\/doi.org\/10.1007\/s10462-024-11078-6","relation":{},"ISSN":["1573-7462"],"issn-type":[{"value":"1573-7462","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,13]]},"assertion":[{"value":"13 December 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 January 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"90"}}