{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T00:48:08Z","timestamp":1774658888182,"version":"3.50.1"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T00:00:00Z","timestamp":1725926400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T00:00:00Z","timestamp":1725926400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000943","name":"Commonwealth Scientific and Industrial Research Organisation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000943","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["AI Ethics"],"published-print":{"date-parts":[[2025,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>The Right to be Forgotten (RTBF) was first established as the result of the ruling of Google Spain SL, Google Inc. v AEPD, Mario Costeja Gonz\u00e1lez, and was later included as the Right to Erasure under the General Data Protection Regulation (GDPR) of European Union to allow individuals the right to request personal data be deleted by organizations. Specifically for search engines, individuals can send requests to organizations to exclude their information from the query results. It was a significant emergent right as the result of the evolution of technology. With the recent development of Large Language Models (LLMs) and their use in chatbots, LLM-enabled software systems have become popular. But they are not excluded from the RTBF. Compared with the indexing approach used by search engines, LLMs store, and process information in a completely different way. This poses new challenges for compliance with the RTBF. In this paper, we explore these challenges and provide our insights on how to implement technical solutions for the RTBF, including the use of differential privacy, machine unlearning, model editing, and guardrails. With the rapid advancement of AI and the increasing need of regulating this powerful technology, learning from the case of RTBF can provide valuable lessons for technical practitioners, legal experts, organizations, and authorities.<\/jats:p>","DOI":"10.1007\/s43681-024-00573-9","type":"journal-article","created":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T16:02:16Z","timestamp":1725984136000},"page":"2445-2454","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Right to be forgotten in the Era of large language models: implications, challenges, and solutions"],"prefix":"10.1007","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0801-475X","authenticated-orcid":false,"given":"Dawen","family":"Zhang","sequence":"first","affiliation":[]},{"given":"Pamela","family":"Finckenberg-Broman","sequence":"additional","affiliation":[]},{"given":"Thong","family":"Hoang","sequence":"additional","affiliation":[]},{"given":"Shidong","family":"Pan","sequence":"additional","affiliation":[]},{"given":"Zhenchang","family":"Xing","sequence":"additional","affiliation":[]},{"given":"Mark","family":"Staples","sequence":"additional","affiliation":[]},{"given":"Xiwei","family":"Xu","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,9,10]]},"reference":[{"key":"573_CR1","unstructured":"TU and RE v Google LLC (2022)"},{"key":"573_CR2","volume-title":"Google Inc","author":"SL Google Spain","year":"2014","unstructured":"Google Spain, S.L.: Google Inc, vol. Agencia. Mario Costeja Gonz\u00e1lez, Espa\u00f1ola de Protecci\u00f3n de Datos (AEPD) (2014)"},{"key":"573_CR3","doi-asserted-by":"crossref","unstructured":"Werro, F.: The right to be forgotten a comparative study of the emergent right\u2019s evolution and application in europe, the americas, and asia (2020)","DOI":"10.1007\/978-3-030-33512-0"},{"key":"573_CR4","doi-asserted-by":"crossref","unstructured":"Bertram, T., Bursztein, E., Caro, S., Chao, H., Feman, R.C., Fleischer, P., Gustafsson, A., Hemerly, J., Hibbert, C., Invernizzi, L., Donnelly, L.K., Ketover, J., Laefer, J., Nicholas, P., Niu, Y., Obhi, H., Price, D., Strait, A., Thomas, K., Verney, A.: Five years of the right to be forgotten. In: Proceedings of the Conference on Computer and Communications Security (2019)","DOI":"10.1145\/3319535.3354208"},{"key":"573_CR5","unstructured":"Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. (2023) arXiv preprint arXiv:2303.18223"},{"key":"573_CR6","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877\u20131901 (2020)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"573_CR7","unstructured":"Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A.M., Pillai, T.S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., Fiedel, N.: PaLM: Scaling Language Modeling with Pathways (2022)"},{"issue":"8","key":"573_CR8","first-page":"9","volume":"1","author":"A Radford","year":"2019","unstructured":"Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)","journal-title":"OpenAI blog"},{"key":"573_CR9","doi-asserted-by":"crossref","unstructured":"Yang, Y., Yuan, S., Cer, D., Kong, S.-y., Constant, N., Pilar, P., Ge, H., Sung, Y.-H., Strope, B., Kurzweil, R.: Learning semantic textual similarity from conversations. (2018) arXiv preprint arXiv:1804.07754","DOI":"10.18653\/v1\/W18-3022"},{"key":"573_CR10","unstructured":"OpenAI: GPT-4 Technical Report (2023)"},{"key":"573_CR11","doi-asserted-by":"crossref","unstructured":"Edunov, S., Baevski, A., Auli, M.: Pre-trained language model representations for language generation. (2019) arXiv preprint arXiv:1903.09722","DOI":"10.18653\/v1\/N19-1409"},{"key":"573_CR12","unstructured":"Wang, A., Cho, K.: Bert has a mouth, and it must speak: Bert as a markov random field language model. (2019) arXiv preprint arXiv:1902.04094"},{"key":"573_CR13","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1007\/s11023-020-09548-1","volume":"30","author":"L Floridi","year":"2020","unstructured":"Floridi, L., Chiriatti, M.: Gpt-3: Its nature, scope, limits, and consequences. Mind. Mach. 30, 681\u2013694 (2020)","journal-title":"Mind. Mach."},{"key":"573_CR14","unstructured":"Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. (2022) arXiv preprint arXiv:2206.07682"},{"key":"573_CR15","first-page":"27730","volume":"35","author":"L Ouyang","year":"2022","unstructured":"Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Adv. Neural. Inf. Process. Syst. 35, 27730\u201327744 (2022)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"573_CR16","unstructured":"Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T.B., Song, D., Erlingsson, U., et\u00a0al.: (2021) Extracting training data from large language models. In: USENIX Security Symposium, vol. 6"},{"key":"573_CR17","unstructured":"Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., Wallace, E.: Extracting training data from diffusion models. In: 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253\u20135270 (2023)"},{"key":"573_CR18","doi-asserted-by":"crossref","unstructured":"Maynez, J., Narayan, S., Bohnet, B., Mcdonald, R.T.: On faithfulness and factuality in abstractive summarization. In: Proceedings of The 58th Annual Meeting of the Association for Computational Linguistics (ACL) (2020)","DOI":"10.18653\/v1\/2020.acl-main.173"},{"key":"573_CR19","doi-asserted-by":"crossref","unstructured":"Liu, N.F., Zhang, T., Liang, P.: Evaluating Verifiability in Generative Search Engines (2023)","DOI":"10.18653\/v1\/2023.findings-emnlp.467"},{"key":"573_CR20","unstructured":"Saffarizadeh, K., Boodraj, M., Alashoor, T.M., et\u00a0al.: Conversational assistants: Investigating privacy concerns, trust, and self-disclosure. In: ICIS (2017)"},{"key":"573_CR21","unstructured":"Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al.: Siren\u2019s song in the ai ocean: a survey on hallucination in large language models. (2023) arXiv preprint arXiv:2309.01219"},{"issue":"2","key":"573_CR22","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1109\/MSEC.2018.2888775","volume":"17","author":"M Al-Rubaie","year":"2019","unstructured":"Al-Rubaie, M., Chang, J.M.: Privacy-preserving machine learning: Threats and solutions. IEEE Security & Privacy 17(2), 49\u201358 (2019)","journal-title":"IEEE Security & Privacy"},{"key":"573_CR23","doi-asserted-by":"publisher","unstructured":"Yue, X., Inan, H., Li, X., Kumar, G., McAnallen, J., Shajari, H., Sun, H., Levitan, D., Sim, R.: Synthetic text generation with differential privacy: A simple and practical recipe. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1321\u20131342. Association for Computational Linguistics, Toronto, Canada (2023). https:\/\/doi.org\/10.18653\/v1\/2023.acl-long.74 . https:\/\/aclanthology.org\/2023.acl-long.74","DOI":"10.18653\/v1\/2023.acl-long.74"},{"key":"573_CR24","doi-asserted-by":"crossref","unstructured":"Cao, Y., Yang, J.: Towards making systems forget with machine unlearning. In: 2015 IEEE Symposium on Security and Privacy, pp. 463\u2013480 (2015). IEEE","DOI":"10.1109\/SP.2015.35"},{"key":"573_CR25","doi-asserted-by":"crossref","unstructured":"Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C.A., Jia, H., Travers, A., Zhang, B., Lie, D., Papernot, N.: Machine unlearning. In: 2021 IEEE Symposium on Security and Privacy (SP), pp. 141\u2013159 (2021). IEEE","DOI":"10.1109\/SP40001.2021.00019"},{"key":"573_CR26","doi-asserted-by":"crossref","unstructured":"Koch, K., Soll, M.: No matter how you slice it: Machine unlearning with SISA comes at the expense of minority classes. In: First IEEE Conference on Secure and Trustworthy Machine Learning (2023). https:\/\/openreview.net\/forum?id=RBX1H-SGdT","DOI":"10.1109\/SaTML54575.2023.00047"},{"key":"573_CR27","doi-asserted-by":"crossref","unstructured":"Zhang, D., Pan, S., Hoang, T., Xing, Z., Staples, M., Xu, X., Yao, L., Lu, Q., Zhu, L.: To be forgotten or to be fair: Unveiling fairness implications of machine unlearning methods. AI and Ethics, 1\u201311 (2024)","DOI":"10.1007\/s43681-023-00398-y"},{"key":"573_CR28","unstructured":"Guo, C., Goldstein, T., Hannun, A., Van Der\u00a0Maaten, L.: Certified data removal from machine learning models. In: Proceedings of the 37th International Conference on Machine Learning, pp. 3832\u20133842 (2020)"},{"key":"573_CR29","doi-asserted-by":"crossref","unstructured":"Golatkar, A., Achille, A., Soatto, S.: Eternal sunshine of the spotless net: Selective forgetting in deep networks. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304\u20139312 (2020)","DOI":"10.1109\/CVPR42600.2020.00932"},{"issue":"13","key":"573_CR30","doi-asserted-by":"publisher","first-page":"11516","DOI":"10.1609\/aaai.v35i13.17371","volume":"35","author":"L Graves","year":"2021","unstructured":"Graves, L., Nagisetty, V., Ganesh, V.: Amnesiac machine learning. Proceedings of the AAAI Conference on Artificial Intelligence 35(13), 11516\u201311524 (2021)","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"573_CR31","doi-asserted-by":"crossref","unstructured":"Hu, H., Wang, S., Chang, J., Zhong, H., Sun, R., Hao, S., Zhu, H., Xue, M.: A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services (2023)","DOI":"10.14722\/ndss.2024.24252"},{"key":"573_CR32","unstructured":"Mitchell, E., Lin, C., Bosselut, A., Finn, C., Manning, C.D.: Fast model editing at scale. (2021) arXiv preprint arXiv:2110.11309"},{"key":"573_CR33","unstructured":"Mitchell, E., Lin, C., Bosselut, A., Manning, C.D., Finn, C.: Memory-based model editing at scale. In: International Conference on Machine Learning, pp. 15817\u201315831 (2022). PMLR"},{"key":"573_CR34","doi-asserted-by":"crossref","unstructured":"Rebedea, T., Dinu, R., Sreedhar, M., Parisien, C., Cohen, J.: Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. (2023) arXiv preprint arXiv:2310.10501","DOI":"10.18653\/v1\/2023.emnlp-demo.40"},{"key":"573_CR35","doi-asserted-by":"crossref","unstructured":"Zeng, Y., Lin, H., Zhang, J., Yang, D., Jia, R., Shi, W.: How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. (2024) arXiv preprint arXiv:2401.06373","DOI":"10.18653\/v1\/2024.acl-long.773"},{"issue":"2","key":"573_CR36","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1016\/j.clsr.2017.08.007","volume":"34","author":"EF Villaronga","year":"2018","unstructured":"Villaronga, E.F., Kieseberg, P., Li, T.: Humans forget, machines remember: Artificial intelligence and the right to be forgotten. Computer Law & Security Review 34(2), 304\u2013313 (2018)","journal-title":"Computer Law & Security Review"},{"key":"573_CR37","doi-asserted-by":"crossref","unstructured":"Dang, Q.-V.: Right to be forgotten in the age of machine learning. In: Advances in Digital Science: ICADS 2021, pp. 403\u2013411 (2021). Springer","DOI":"10.1007\/978-3-030-71782-7_35"},{"issue":"1","key":"573_CR38","doi-asserted-by":"publisher","first-page":"205395171770399","DOI":"10.1177\/2053951717703996","volume":"4","author":"E Esposito","year":"2017","unstructured":"Esposito, E.: Algorithmic memory and the right to be forgotten on the web. Big Data & Society 4(1), 2053951717703996 (2017)","journal-title":"Big Data & Society"},{"issue":"2","key":"573_CR39","doi-asserted-by":"publisher","first-page":"159","DOI":"10.5235\/17577632.6.2.159","volume":"6","author":"D Lindsay","year":"2014","unstructured":"Lindsay, D.: The \u2019right to be forgotten\u2019by search engines under data privacy law: A legal analysis of the costeja ruling. Journal of Media Law 6(2), 159\u2013179 (2014)","journal-title":"Journal of Media Law"}],"container-title":["AI and Ethics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43681-024-00573-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s43681-024-00573-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43681-024-00573-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,24]],"date-time":"2025-05-24T09:11:10Z","timestamp":1748077870000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s43681-024-00573-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,10]]},"references-count":39,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,6]]}},"alternative-id":["573"],"URL":"https:\/\/doi.org\/10.1007\/s43681-024-00573-9","relation":{},"ISSN":["2730-5953","2730-5961"],"issn-type":[{"value":"2730-5953","type":"print"},{"value":"2730-5961","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,10]]},"assertion":[{"value":"10 June 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 August 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 September 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}