{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T23:09:08Z","timestamp":1778368148365,"version":"3.51.4"},"publisher-location":"Cham","reference-count":68,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783032191045","type":"print"},{"value":"9783032191052","type":"electronic"}],"license":[{"start":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T00:00:00Z","timestamp":1767225600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T00:00:00Z","timestamp":1767225600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026]]},"DOI":"10.1007\/978-3-032-19105-2_41","type":"book-chapter","created":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T22:16:47Z","timestamp":1778365007000},"page":"591-606","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["What Should LLMs Forget? Quantifying Personal Data in\u00a0LLMs for\u00a0Right-to-Be-Forgotten Requests"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-1671-222X","authenticated-orcid":false,"given":"Dimitri","family":"Staufer","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,5,1]]},"reference":[{"key":"41_CR1","doi-asserted-by":"publisher","first-page":"66044","DOI":"10.52202\/075280-2884","volume":"36","author":"N Belrose","year":"2023","unstructured":"Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., Biderman, S.: Leace: perfect linear concept erasure in closed form. Adv. Neural. Inf. Process. Syst. 36, 66044\u201366063 (2023)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"41_CR2","doi-asserted-by":"publisher","unstructured":"Bertram, T., et al.: Five years of the right to be forgotten. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 959\u2013972. ACM, London, UK (2019). https:\/\/doi.org\/10.1145\/3319535.3354208","DOI":"10.1145\/3319535.3354208"},{"key":"41_CR3","first-page":"28072","volume":"36","author":"S Biderman","year":"2023","unstructured":"Biderman, S., et al.: Emergent and predictable memorization in large language models. Adv. Neural. Inf. Process. Syst. 36, 28072\u201328090 (2023)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"41_CR4","unstructured":"Biderman, S., et al.: Pythia: a suite for analyzing large language models across training and scaling. In: International Conference on Machine Learning, pp. 2397\u20132430. PMLR (2023)"},{"issue":"3","key":"41_CR5","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1007\/s10462-024-11078-6","volume":"58","author":"A Blanco-Justicia","year":"2025","unstructured":"Blanco-Justicia, A., et al.: Digital forgetting in large language models: a survey of unlearning methods. Artif. Intell. Rev. 58(3), 90 (2025). https:\/\/doi.org\/10.1007\/s10462-024-11078-6","journal-title":"Artif. Intell. Rev."},{"key":"41_CR6","doi-asserted-by":"crossref","unstructured":"Bourtoule, L., et al.: Machine unlearning. In: 2021 IEEE Symposium on Security and Privacy (SP), pp. 141\u2013159. IEEE (2021)","DOI":"10.1109\/SP40001.2021.00019"},{"issue":"1\u20137","key":"41_CR7","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1016\/S0169-7552(98)00110-X","volume":"30","author":"S Brin","year":"1998","unstructured":"Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Comput. Netw. ISDN Syst. 30(1\u20137), 107\u2013117 (1998)","journal-title":"Comput. Netw. ISDN Syst."},{"key":"41_CR8","doi-asserted-by":"crossref","unstructured":"Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., Zhang, C.: Quantifying memorization across neural language models. In: The Eleventh International Conference on Learning Representations (2022)","DOI":"10.52202\/075280-1708"},{"key":"41_CR9","unstructured":"Carlini, N., Liu, C., Erlingsson, \u00da., Kos, J., Song, D.: The secret sharer: evaluating and testing unintended memorization in neural networks. In: 28th USENIX Security Symposium (USENIX Security 2019), pp. 267\u2013284 (2019)"},{"key":"41_CR10","unstructured":"Carlini, N., et al.: Stealing part of a production language model. arXiv:2403.06634 (2024)"},{"key":"41_CR11","unstructured":"Carlini, N., et al.: Extracting training data from large language models. In: 30th USENIX Security Symposium (USENIX Security 2021), pp. 2633\u20132650 (2021)"},{"key":"41_CR12","doi-asserted-by":"crossref","unstructured":"Chen, J., Yang, D.: Unlearn what you want to forget: efficient unlearning for LLMs. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12041\u201312052 (2023)","DOI":"10.18653\/v1\/2023.emnlp-main.738"},{"key":"41_CR13","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1162\/tacl_a_00644","volume":"12","author":"R Cohen","year":"2024","unstructured":"Cohen, R., Biran, E., Yoran, O., Globerson, A., Geva, M.: Evaluating the ripple effects of knowledge editing in language models. Trans. Assoc. Computat. Linguist. 12, 283\u2013298 (2024)","journal-title":"Trans. Assoc. Computat. Linguist."},{"key":"41_CR14","unstructured":"Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., Wei, F.: Knowledge neurons in pretrained transformers. arXiv:2104.08696 (2021)"},{"key":"41_CR15","unstructured":"Duan, M., et al.: Do membership inference attacks work on large language models? arXiv:2402.07841 (2024)"},{"key":"41_CR16","unstructured":"Eldan, R., Russinovich, M.: Who\u2019s harry potter? Approximate unlearning in LLMs. arXiv:2310.02238 (2023)"},{"key":"41_CR17","unstructured":"European Data Protection Board: Opinion 28\/2024 on certain data protection aspects related to the processing of personal data in the context of ai models (2024). https:\/\/www.edpb.europa.eu\/system\/files\/2024-12\/edpb_opinion_202428_ai-models_en.pdf. Accessed 2 June 2025"},{"key":"41_CR18","unstructured":"Gao, T., Fisch, A., Chen, D.: Making pre-trained language models better few-shot learners. arXiv:2012.15723 (2020)"},{"key":"41_CR19","unstructured":"Ginart, A., Guan, M., Valiant, G., Zou, J.Y.: Making AI forget you: data deletion in machine learning. In: Advances in Neural Information Processing Systems, vol. 32 (2019)"},{"key":"41_CR20","unstructured":"Grattafiori, A., et al.: The llama 3 herd of models. arXiv:2407.21783 (2024)"},{"key":"41_CR21","unstructured":"Hase, P., et al.: Do language models have beliefs? Methods for detecting, updating, and visualizing model beliefs. arXiv:2111.13654 (2021)"},{"key":"41_CR22","doi-asserted-by":"crossref","unstructured":"Hayes, J., et al.: Measuring memorization in language models via probabilistic extraction. In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 9266\u20139291 (2025)","DOI":"10.18653\/v1\/2025.naacl-long.469"},{"key":"41_CR23","doi-asserted-by":"crossref","unstructured":"Huang, J., Shao, H., Chang, K.C.C.: Are large pre-trained language models leaking your personal information? arXiv:2205.12628 (2022)","DOI":"10.18653\/v1\/2022.findings-emnlp.148"},{"key":"41_CR24","unstructured":"Ilharco, G., et al.: Editing models with task arithmetic. arXiv:2212.04089 (2022)"},{"key":"41_CR25","doi-asserted-by":"crossref","unstructured":"Ippolito, D., et al.: Preventing verbatim memorization in language models gives a false sense of privacy. arXiv:2210.17546 (2022)","DOI":"10.18653\/v1\/2023.inlg-main.3"},{"key":"41_CR26","unstructured":"Jin, Z., et al.: Rwku: benchmarking real-world knowledge unlearning for large language models. arXiv:2406.10890 (2024)"},{"key":"41_CR27","doi-asserted-by":"crossref","unstructured":"Kassem, A.M., et al.: Alpaca against vicuna: using LLMs to uncover memorization of LLMs. arXiv:2403.04801 (2024)","DOI":"10.18653\/v1\/2025.naacl-long.421"},{"key":"41_CR28","unstructured":"Khattab, O., et al.: Dspy: compiling declarative language model calls into self-improving pipelines. arXiv:2310.03714 (2023)"},{"key":"41_CR29","first-page":"20750","volume":"36","author":"S Kim","year":"2023","unstructured":"Kim, S., Yun, S., Lee, H., Gubri, M., Yoon, S., Oh, S.J.: Propile: probing privacy leakage in large language models. Adv. Neural. Inf. Process. Syst. 36, 20750\u201320762 (2023)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"41_CR30","doi-asserted-by":"crossref","unstructured":"Lee, K., et al.: Deduplicating training data makes language models better. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424\u20138445 (2022)","DOI":"10.18653\/v1\/2022.acl-long.577"},{"key":"41_CR31","unstructured":"Li, X., Tramer, F., Liang, P., Hashimoto, T.: Large language models can be strong differentially private learners. arXiv:2110.05679 (2021)"},{"key":"41_CR32","first-page":"118198","volume":"37","author":"C Liu","year":"2024","unstructured":"Liu, C., Wang, Y., Flanigan, J., Liu, Y.: Large language model unlearning via embedding-corrupted prompts. Adv. Neural. Inf. Process. Syst. 37, 118198\u2013118266 (2024)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"41_CR33","doi-asserted-by":"publisher","unstructured":"Liu, S., et al.: Rethinking machine unlearning for large language models (2024). https:\/\/doi.org\/10.48550\/arXiv.2402.08787","DOI":"10.48550\/arXiv.2402.08787"},{"key":"41_CR34","doi-asserted-by":"crossref","unstructured":"Liu, Y., et al.: DP-memarc: differential privacy transfer learning for memory efficient language models. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 26317\u201326325 (2025)","DOI":"10.1609\/aaai.v39i25.34830"},{"key":"41_CR35","doi-asserted-by":"crossref","unstructured":"Lu, Y., Bartolo, M., Moore, A., Riedel, S., Stenetorp, P.: Fantastically ordered prompts and where to find them: overcoming few-shot prompt order sensitivity. arXiv:2104.08786 (2021)","DOI":"10.18653\/v1\/2022.acl-long.556"},{"key":"41_CR36","unstructured":"\u0141ucki, J., Wei, B., Huang, Y., Henderson, P., Tramer, F., Rando, J.: An adversarial perspective on machine unlearning for ai safety (2024). https:\/\/arxiv.org\/abs\/2409.18025"},{"key":"41_CR37","unstructured":"Lynch, A., Guo, P., Ewart, A., Casper, S., Hadfield-Menell, D.: Eight methods to evaluate robust unlearning in LLMs. arXiv:2402.16835 (2024)"},{"key":"41_CR38","unstructured":"Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z.C., Kolter, J.Z.: Tofu: a task of fictitious unlearning for LLMs. arXiv:2401.06121 (2024)"},{"key":"41_CR39","unstructured":"Meeus, M., Jain, S., Rei, M., de Montjoye, Y.A.: Did the neurons read your book? Document-level membership inference for large language models. In: 33rd USENIX Security Symposium (USENIX Security 2024), pp. 2369\u20132385 (2024)"},{"key":"41_CR40","first-page":"17359","volume":"35","author":"K Meng","year":"2022","unstructured":"Meng, K., Bau, D., Andonian, A., Belinkov, Y.: Locating and editing factual associations in GPT. Adv. Neural. Inf. Process. Syst. 35, 17359\u201317372 (2022)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"41_CR41","unstructured":"Morris, J.X., Zhao, W., Chiu, J.T., Shmatikov, V., Rush, A.M.: Language model inversion. arXiv:2311.13647 (2023)"},{"key":"41_CR42","unstructured":"Muresanu, A., Thudi, A., Zhang, M.R., Papernot, N.: Unlearnable algorithms for in-context learning. arXiv:2402.00751 (2024)"},{"key":"41_CR43","doi-asserted-by":"crossref","unstructured":"Nakka, K.K., Frikha, A., Mendes, R., Jiang, X., Zhou, X.: PII-compass: guiding LLM training data extraction prompts towards the target PII via grounding. arXiv:2407.02943 (2024)","DOI":"10.18653\/v1\/2024.privatenlp-1.7"},{"key":"41_CR44","unstructured":"Nasr, M., et al.: Scalable extraction of training data from (production) language models. arXiv:2311.17035 (2023)"},{"key":"41_CR45","unstructured":"Nguyen, T.T., et al.: A survey of machine unlearning. arXiv:2209.02299 (2022)"},{"key":"41_CR46","unstructured":"NOYB: Chatgpt provides false information about people, and openai can\u2019t correct it (2024). https:\/\/noyb.eu\/en\/chatgpt-provides-false-information-about-people-and-openai-cant-correct-it. Accessed 02 June 2025"},{"key":"41_CR47","doi-asserted-by":"publisher","unstructured":"Panda, A., Tang, X., Nasr, M., Choquette-Choo, C.A., Mittal, P.: Privacy auditing of large language models (2025). https:\/\/doi.org\/10.48550\/arXiv.2503.06808","DOI":"10.48550\/arXiv.2503.06808"},{"key":"41_CR48","unstructured":"Patil, V., Hase, P., Bansal, M.: Can sensitive information be deleted from LLMs? Objectives for defending against extraction attacks. arXiv:2309.17410 (2023)"},{"key":"41_CR49","unstructured":"Pawelczyk, M., Neel, S., Lakkaraju, H.: In-context unlearning: language models as few shot unlearners. arXiv:2310.07579 (2023)"},{"key":"41_CR50","doi-asserted-by":"crossref","unstructured":"Puerto, H., Gubri, M., Yun, S., Oh, S.J.: Scaling up membership inference: when and how attacks succeed on large language models. arXiv:2411.00154 (2024)","DOI":"10.18653\/v1\/2025.findings-naacl.234"},{"key":"41_CR51","unstructured":"Schlangen, D.: Targeting the benchmark: on methodology in current natural language processing research. arXiv:2007.04792 (2020)"},{"key":"41_CR52","unstructured":"Sekhari, A., Acharya, J., Kamath, G., Suresh, A.T.: Remember what you want to forget: algorithms for machine unlearning. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 18075\u201318086. Curran Associates, Inc. (2021)"},{"key":"41_CR53","unstructured":"Shani, C., Jurafsky, D., LeCun, Y., Shwartz-Ziv, R.: From tokens to thoughts: How LLMs and humans trade compression for meaning. arXiv:2505.17117 (2025)"},{"key":"41_CR54","unstructured":"Shi, W., et al.: Detecting pretraining data from large language models. arXiv:2310.16789 (2023)"},{"key":"41_CR55","doi-asserted-by":"crossref","unstructured":"Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: eliciting knowledge from language models with automatically generated prompts. arXiv:2010.15980 (2020)","DOI":"10.18653\/v1\/2020.emnlp-main.346"},{"key":"41_CR56","unstructured":"Staab, R., Vero, M., Balunovi\u0107, M., Vechev, M.: Beyond memorization: violating privacy via inference with large language models. arXiv:2310.07298 (2023)"},{"key":"41_CR57","unstructured":"Team, G., et al.: Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530 (2024)"},{"key":"41_CR58","unstructured":"Thaker, P., Maurya, Y., Hu, S., Wu, Z.S., Smith, V.: Guardrail baselines for unlearning in LLMs. arXiv:2403.03329 (2024)"},{"key":"41_CR59","unstructured":"Tiwari, T., Suh, G.E.: Sequence-level analysis of leakage risk of training data in large language models. arXiv:2412.11302 (2024)"},{"key":"41_CR60","unstructured":"Ullah, E., Mai, T., Rao, A., Rossi, R.A., Arora, R.: Machine unlearning via algorithmic stability. In: Belkin, M., Kpotufe, S. (eds.) Proceedings of Thirty Fourth Conference on Learning Theory. Proceedings of Machine Learning Research, vol. 134, pp. 4126\u20134142. PMLR (2021)"},{"key":"41_CR61","unstructured":"Ulmer, D.: On uncertainty in natural language processing. arXiv:2410.03446 (2024)"},{"key":"41_CR62","doi-asserted-by":"publisher","unstructured":"Vilella, S., Ruffo, G.: (De)-indexing and the right to be forgotten (2025). https:\/\/doi.org\/10.48550\/arXiv.2501.03989","DOI":"10.48550\/arXiv.2501.03989"},{"key":"41_CR63","doi-asserted-by":"crossref","unstructured":"Wang, L., Zeng, X., Guo, J., Wong, K.F., Gottlob, G.: Selective forgetting: advancing machine unlearning techniques and evaluation in language models. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 843\u2013851 (2025)","DOI":"10.1609\/aaai.v39i1.32068"},{"key":"41_CR64","doi-asserted-by":"crossref","unstructured":"Yao, J., et al.: Machine unlearning of pre-trained large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8403\u20138419 (2024)","DOI":"10.18653\/v1\/2024.acl-long.457"},{"key":"41_CR65","first-page":"105425","volume":"37","author":"Y Yao","year":"2024","unstructured":"Yao, Y., Xu, X., Liu, Y.: Large language model unlearning. Adv. Neural. Inf. Process. Syst. 37, 105425\u2013105475 (2024)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"41_CR66","first-page":"39321","volume":"36","author":"C Zhang","year":"2023","unstructured":"Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tram\u00e8r, F., Carlini, N.: Counterfactual memorization in neural language models. Adv. Neural. Inf. Process. Syst. 36, 39321\u201339362 (2023)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"41_CR67","doi-asserted-by":"crossref","unstructured":"Zhang, D., et al.: Right to be forgotten in the Era of large language models: implications, challenges, and solutions. AI and Ethics (2024)","DOI":"10.1007\/s43681-024-00573-9"},{"key":"41_CR68","doi-asserted-by":"publisher","unstructured":"Zhou, Z., Xiang, J., Chen, C., Su, S.: Quantifying and analyzing entity-level memorization in large language models. In: Proceedings of the AAAI Conference on AI, vol. 38, no. 17, pp. 19741\u201319749 (2024). https:\/\/doi.org\/10.1609\/aaai.v38i17.29948","DOI":"10.1609\/aaai.v38i17.29948"}],"container-title":["Communications in Computer and Information Science","Machine Learning and Principles and Practice of Knowledge Discovery in Databases"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-032-19105-2_41","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T22:16:55Z","timestamp":1778365015000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-032-19105-2_41"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"ISBN":["9783032191045","9783032191052"],"references-count":68,"URL":"https:\/\/doi.org\/10.1007\/978-3-032-19105-2_41","relation":{},"ISSN":["1865-0929","1865-0937"],"issn-type":[{"value":"1865-0929","type":"print"},{"value":"1865-0937","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026]]},"assertion":[{"value":"1 May 2026","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"This work supports transparency and RTBF compliance by measuring factual memorization in LLMs. WikiMem relies on Wikidata, which may reflect cultural or systemic biases, and focuses on English-only prompts. All models were publicly available, and no personal data was introduced beyond existing Wikidata entries.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Considerations"}},{"value":"The author has no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclosure of Interests"}},{"value":"ECML PKDD","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Porto","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Portugal","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2025","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"15 September 2025","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"19 September 2025","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"ecml2025","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/ecmlpkdd.org\/2025\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}