{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T18:17:42Z","timestamp":1789064262327,"version":"build-2803163510"},"reference-count":233,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2025,1,9]],"date-time":"2025-01-09T00:00:00Z","timestamp":1736380800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100018537","name":"National Science and Technology Major Project","doi-asserted-by":"crossref","award":["2021ZD0114002"],"award-info":[{"award-number":["2021ZD0114002"]}],"id":[{"id":"10.13039\/501100018537","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Nature Science Foundation of China","doi-asserted-by":"crossref","award":["62477010, and 62307028"],"award-info":[{"award-number":["62477010, and 62307028"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100003399","name":"Science and Technology Commission of Shanghai Municipality","doi-asserted-by":"crossref","award":["22511105901, and 21511100402"],"award-info":[{"award-number":["22511105901, and 21511100402"]}],"id":[{"id":"10.13039\/501100003399","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Shanghai Science and Technology Innovation Action Plan","award":["24YF2710100"],"award-info":[{"award-number":["24YF2710100"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2025,5,31]]},"abstract":"<jats:p>Recently, foundation language models (LMs) have marked significant achievements in the domains of natural language processing and computer vision. Unlike traditional neural network models, foundation LMs obtain a great ability for transfer learning by acquiring rich common sense knowledge through pre-training on extensive unsupervised datasets with a vast number of parameters. Despite these capabilities, LMs still struggle with catastrophic forgetting, hindering their ability to learn continuously like humans. To address this, continual learning (CL) methodologies have been introduced, allowing LMs to adapt to new tasks while retaining learned knowledge. However, a systematic taxonomy of existing approaches and a comparison of their performance are still lacking. In this article, we delve into a comprehensive review, summarization, and classification of the existing literature on CL-based approaches applied to foundation language models, such as pre-trained language models, large language models, and vision-language models. We divide these studies into offline and online CL, which consist of traditional methods, parameter-efficient-based methods, instruction tuning-based methods and continual pre-training methods. Additionally, we outline the typical datasets and metrics employed in CL research and provide a detailed analysis of the challenges and future work for LMs-based continual learning.<\/jats:p>","DOI":"10.1145\/3705725","type":"journal-article","created":{"date-parts":[[2024,12,9]],"date-time":"2024-12-09T05:56:20Z","timestamp":1733723780000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":47,"title":["Recent Advances of Foundation Language Models-based Continual Learning: A Survey"],"prefix":"10.1145","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6446-0426","authenticated-orcid":false,"given":"Yutao","family":"Yang","sequence":"first","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2589-0164","authenticated-orcid":false,"given":"Jie","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2598-2365","authenticated-orcid":false,"given":"Xuanwen","family":"Ding","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-3573-9962","authenticated-orcid":false,"given":"Tianyu","family":"Huai","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-5091-7577","authenticated-orcid":false,"given":"Shunyu","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5602-1877","authenticated-orcid":false,"given":"Qin","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6945-7437","authenticated-orcid":false,"given":"Yuan","family":"Xie","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4723-5486","authenticated-orcid":false,"given":"Liang","family":"He","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,1,9]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et\u00a0al. 2023. Gpt-4 technical report. Retrieved from https:\/\/arXiv:2303.08774"},{"key":"e_1_3_1_3_2","first-page":"18852","volume-title":"ICCV","author":"Hammoud Hasan Abed Al Kader","year":"2023","unstructured":"Hasan Abed Al Kader Hammoud, Ameya Prabhu, Ser-Nam Lim, Philip H. S. Torr, Adel Bibi, and Bernard Ghanem. 2023. Rapid adaptation in online continual learning: Are we evaluating it right?. In ICCV. 18852\u201318861."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01219-9_9"},{"key":"e_1_3_1_5_2","unstructured":"Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen Nova DasSarma Dawn Drain Stanislav Fort Deep Ganguli Tom Henighan et\u00a0al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. Retrieved from https:\/\/arXiv:2204.05862"},{"key":"e_1_3_1_6_2","unstructured":"Shawn Beaulieu Lapo Frati Thomas Miconi Joel Lehman Kenneth O. Stanley Jeff Clune and Nick Cheney. 2020. Learning to continually learn. Retrieved from https:\/\/arXiv:2002.09571"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2020.12.003"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.574"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","unstructured":"Raad Bin Tareaf. 2017. Tweets Dataset - Top 20 most followed users in Twitter social platform. Harvard Dataverse. DOI:10.7910\/DVN\/JBXKFD","DOI":"10.7910\/DVN\/JBXKFD"},{"key":"e_1_3_1_10_2","article-title":"The description length of deep learning models","volume":"31","author":"Blier L\u00e9onard","year":"2018","unstructured":"L\u00e9onard Blier and Yann Ollivier. 2018. The description length of deep learning models. NeurIPS 31 (2018).","journal-title":"NeurIPS"},{"key":"e_1_3_1_11_2","unstructured":"Antoine Bordes Nicolas Usunier Sumit Chopra and Jason Weston. 2015. Large-scale simple question answering with memory networks. Retrieved from https:\/\/arXiv:1506.02075"},{"key":"e_1_3_1_12_2","unstructured":"Tom B. Brown. 2020. Language models are few-shot learners. Retrieved from https:\/\/arXiv:2005.14165"},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","unstructured":"Pawe\u0142 Budzianowski Tsung-Hsien Wen Bo-Hsiang Tseng Inigo Casanueva Stefan Ultes Osman Ramadan and Milica Ga\u0161i\u0107. 2018. MultiWOZ\u2013a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. Retrieved from https:\/\/arXiv:1810.00278","DOI":"10.18653\/v1\/D18-1547"},{"key":"e_1_3_1_14_2","unstructured":"Minwoo Byeon Beomhee Park Haecheon Kim Sungjun Lee Woonhyuk Baek and Saehoon Kim. 2022. COYO-700M: Image-Text Pair Dataset. Retrieved from https:\/\/github.com\/kakaobrain\/coyo-dataset"},{"key":"e_1_3_1_15_2","doi-asserted-by":"crossref","unstructured":"Bill Byrne Karthik Krishnamoorthi Chinnadhurai Sankar Arvind Neelakantan Daniel Duckworth Semih Yavuz Ben Goodrich Amit Dubey Andy Cedilnik and Kyu-Young Kim. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. Retrieved from https:\/\/arXiv:1909.05358","DOI":"10.18653\/v1\/D19-1459"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.sealp-1.5"},{"key":"e_1_3_1_17_2","article-title":"e-SNLI: Natural language inference with natural language explanations","author":"Camburu Oana-Maria","year":"2018","unstructured":"Oana-Maria Camburu, Tim Rockt\u00e4schel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural language inference with natural language explanations. In NeurIPS. 9560\u20139572.","journal-title":"NeurIPS"},{"key":"e_1_3_1_18_2","unstructured":"Xusheng Cao Haori Lu Linlan Huang Xialei Liu and Ming-Ming Cheng. 2024. Generative multi-modal models are good class-incremental learners. Retrieved from https:\/\/arXiv:2403.18383"},{"key":"e_1_3_1_19_2","doi-asserted-by":"crossref","unstructured":"I\u00f1igo Casanueva Tadas Tem\u010dinas Daniela Gerz Matthew Henderson and Ivan Vuli\u0107. 2020. Efficient intent detection with dual sentence encoders. Retrieved from https:\/\/arXiv:2003.04807","DOI":"10.18653\/v1\/2020.nlp4convai-1.5"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-short.106"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00356"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01252-6_33"},{"key":"e_1_3_1_23_2","volume-title":"ICLR","author":"Chaudhry Arslan","year":"2019","unstructured":"Arslan Chaudhry, Marc\u2019Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient lifelong learning with A-GEM. In ICLR."},{"key":"e_1_3_1_24_2","doi-asserted-by":"crossref","unstructured":"Sanyuan Chen Yutai Hou Yiming Cui Wanxiang Che Ting Liu and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. Retrieved from https:\/\/arXiv:2004.12651","DOI":"10.18653\/v1\/2020.emnlp-main.634"},{"key":"e_1_3_1_25_2","first-page":"5383","volume-title":"ICML","author":"Chen Wuyang","year":"2023","unstructured":"Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. 2023. Lifelong language pretraining with distribution-specialized experts. In ICML. PMLR, 5383\u20135395."},{"key":"e_1_3_1_26_2","unstructured":"Daixuan Cheng Shaohan Huang and Furu Wei. 2023. Adapting large language models via reading comprehension. Retrieved from https:\/\/arXiv:2309.09530"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-acl.901"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1241"},{"key":"e_1_3_1_29_2","doi-asserted-by":"crossref","unstructured":"Andrea Cossu Tinne Tuytelaars Antonio Carta Lucia Passaro Vincenzo Lomonaco and Davide Bacciu. 2022. Continual pre-training mitigates forgetting in language and vision. Retrieved from https:\/\/arXiv:2205.09357","DOI":"10.2139\/ssrn.4495233"},{"key":"e_1_3_1_30_2","unstructured":"Alice Coucke Alaa Saade Adrien Ball Th\u00e9odore Bluche Alexandre Caulier David Leroy Cl\u00e9ment Doumouro Thibault Gisselbrecht Francesco Caltagirone Thibaut Lavril et\u00a0al. 2018. Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces. Retrieved from https:\/\/arXiv:1805.10190"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i10.29054"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01621"},{"issue":"7","key":"e_1_3_1_33_2","first-page":"3366","article-title":"A continual learning survey: Defying forgetting in classification tasks","volume":"44","author":"Lange Matthias De","year":"2021","unstructured":"Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale\u0161 Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. TPAMI 44, 7 (2021), 3366\u20133385.","journal-title":"TPAMI"},{"key":"e_1_3_1_34_2","article-title":"Episodic memory in lifelong language learning","author":"D\u2019Autume Cyprien de Masson","year":"2019","unstructured":"Cyprien de Masson D\u2019Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In NeurIPS. 13132\u201313141.","journal-title":"NeurIPS"},{"key":"e_1_3_1_35_2","unstructured":"Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Retrieved from https:\/\/arXiv:1810.04805"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341561"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-emnlp.252"},{"key":"e_1_3_1_38_2","unstructured":"Qingxiu Dong Lei Li Damai Dai Ce Zheng Zhiyong Wu Baobao Chang Xu Sun Jingjing Xu and Zhifang Sui. 2022. A survey on in-context learning. Retrieved from https:\/\/arXiv:2301.00234"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58565-5_6"},{"key":"e_1_3_1_40_2","unstructured":"Mingzhe Du Anh Tuan Luu Bin Ji and See-kiong Ng. 2023. From static to dynamic: A continual learning framework for large language models. Retrieved from https:\/\/arXiv:2310.14248"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2022\/762"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/11787006_1"},{"key":"e_1_3_1_43_2","first-page":"5988","volume-title":"ICML","author":"Ethayarajh Kawin","year":"2022","unstructured":"Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In ICML. 5988\u20136008."},{"key":"e_1_3_1_44_2","article-title":"Datacomp: In search of the next generation of multimodal datasets","author":"Gadre Samir Yitzhak","year":"2024","unstructured":"Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang et\u00a0al. 2024. Datacomp: In search of the next generation of multimodal datasets. In NeurIPS. 27092\u201327112.","journal-title":"NeurIPS"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-short.66"},{"key":"e_1_3_1_46_2","doi-asserted-by":"crossref","unstructured":"Evangelia Gogoulou Timoth\u00e9e Lesort Magnus Boman and Joakim Nivre. 2023. A study of continual learning under language shift. Retrieved from https:\/\/arXiv:2311.01200","DOI":"10.1007\/978-3-031-70563-2_6"},{"key":"e_1_3_1_47_2","unstructured":"Oded Goldreich. 1998. Secure multi-party computation. Manuscript. Preliminary Version 78 110 (1998) 1\u2013108."},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.670"},{"key":"e_1_3_1_49_2","volume-title":"International Workshop OntoImage","author":"Grubinger Michael","year":"2006","unstructured":"Michael Grubinger, Paul Clough, Henning M\u00fcller, and Thomas Deselaers. 2006. The iapr tc-12 benchmark: A new evaluation resource for visual information systems. In International Workshop OntoImage, Vol. 2."},{"key":"e_1_3_1_50_2","doi-asserted-by":"crossref","unstructured":"Sonal Gupta Rushin Shah Mrinal Mohit Anuj Kumar and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. Retrieved from https:\/\/arXiv:1810.07942","DOI":"10.18653\/v1\/D18-1300"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.407"},{"key":"e_1_3_1_52_2","doi-asserted-by":"crossref","unstructured":"Xu Han Hao Zhu Pengfei Yu Ziyun Wang Yuan Yao Zhiyuan Liu and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. Retrieved from https:\/\/arXiv:1810.10147","DOI":"10.18653\/v1\/D18-1514"},{"key":"e_1_3_1_53_2","unstructured":"Zeyu Han Chao Gao Jinyang Liu Sai Qian Zhang et\u00a0al. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. Retrieved from https:\/\/arXiv:2403.14608"},{"issue":"11","key":"e_1_3_1_54_2","first-page":"2908","article-title":"Replay in deep learning: Current approaches and missing biological elements","volume":"33","author":"Hayes Tyler L.","year":"2021","unstructured":"Tyler L. Hayes, Giri P. Krishnan, Maxim Bazhenov, Hava T. Siegelmann, Terrence J. Sejnowski, and Christopher Kanan. 2021. Replay in deep learning: Current approaches and missing biological elements. Neural Comput. 33, 11 (2021), 2908\u20132950.","journal-title":"Neural Comput."},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883037"},{"key":"e_1_3_1_56_2","unstructured":"Nithin Holla Pushkar Mishra Helen Yannakoudakis and Ekaterina Shutova. 2020. Meta-learning with sparse experience replay for lifelong language learning. Retrieved from https:\/\/arXiv:2009.04891"},{"key":"e_1_3_1_57_2","first-page":"2790","volume-title":"ICML","author":"Houlsby Neil","year":"2019","unstructured":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In ICML. 2790\u20132799."},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.3115\/1614049.1614064"},{"issue":"5","key":"e_1_3_1_59_2","first-page":"5684","article-title":"Drinking from a firehose: Continual learning with web-scale natural language","volume":"45","author":"Hu Hexiang","year":"2022","unstructured":"Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. 2022. Drinking from a firehose: Continual learning with web-scale natural language. TPAMI 45, 5 (2022), 5684\u20135696.","journal-title":"TPAMI"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014073"},{"key":"e_1_3_1_61_2","doi-asserted-by":"crossref","unstructured":"Linlan Huang Xusheng Cao Haori Lu and Xialei Liu. 2024. Class-incremental learning with CLIP: Adaptive representation adjustment and parameter fusion. Retrieved from https:\/\/arXiv:2407.14143","DOI":"10.1007\/978-3-031-72949-2_13"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.218"},{"key":"e_1_3_1_63_2","unstructured":"Hamel Husain Ho-Hsiang Wu Tiferet Gazit Miltiadis Allamanis and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. Retrieved from https:\/\/arXiv:1909.09436"},{"key":"e_1_3_1_64_2","unstructured":"Srinivasan Iyer Ioannis Konstas Alvin Cheung and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. Retrieved from https:\/\/arXiv:1808.09588"},{"key":"e_1_3_1_65_2","first-page":"14702","volume-title":"ICML","author":"Jang Joel","year":"2023","unstructured":"Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2023. Exploring the benefits of training expert language models over instruction tuning. In ICML. 14702\u201314729."},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.418"},{"key":"e_1_3_1_67_2","volume-title":"ICLR","author":"Jang Joel","year":"2022","unstructured":"Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2022. Towards continual knowledge learning of language models. In ICLR."},{"key":"e_1_3_1_68_2","article-title":"Meta-learning representations for continual learning","author":"Javed Khurram","year":"2019","unstructured":"Khurram Javed and Martha White. 2019. Meta-learning representations for continual learning. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_1_69_2","unstructured":"Saurav Jha Dong Gong and Lina Yao. 2024. CLAP4CLIP: Continual learning with probabilistic finetuning for vision-language models. Retrieved from https:\/\/arXiv:2403.19137"},{"key":"e_1_3_1_70_2","first-page":"4904","volume-title":"ICML","author":"Jia Chao","year":"2021","unstructured":"Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML. PMLR, 4904\u20134916."},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.62"},{"key":"e_1_3_1_72_2","first-page":"4764","volume-title":"NAACL","author":"Jin Xisen","year":"2022","unstructured":"Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022. Lifelong pretraining: Continually adapting language models to emerging corpora. In NAACL. 4764\u20134780."},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1147"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.217"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.695"},{"key":"e_1_3_1_76_2","unstructured":"Zixuan Ke and Bing Liu. 2022. Continual learning of natural language processing tasks: A survey. Retrieved from https:\/\/arXiv:2211.12701"},{"key":"e_1_3_1_77_2","first-page":"22443","volume-title":"NeurIPS","author":"Ke Zixuan","year":"2021","unstructured":"Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. 2021. Achieving forgetting prevention and knowledge transfer in continual learning. In NeurIPS, Vol. 34. 22443\u201322456."},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.550"},{"key":"e_1_3_1_79_2","volume-title":"ICLR","author":"Ke Zixuan","year":"2023","unstructured":"Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre-training of language models. In ICLR."},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.378"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11651"},{"key":"e_1_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01053"},{"key":"e_1_3_1_83_2","doi-asserted-by":"crossref","unstructured":"Junsu Kim Yunhoe Ku Jihyeon Kim Junuk Cha and Seungryul Baek. 2024. VLM-PL: Advanced pseudo labeling approach class incremental object detection with vision-language model. Retrieved from https:\/\/arXiv:2403.05346","DOI":"10.1109\/CVPRW63382.2024.00420"},{"key":"e_1_3_1_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599516"},{"key":"e_1_3_1_85_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1611835114"},{"key":"e_1_3_1_86_2","volume-title":"ICLR","author":"Koh Hyunseo","year":"2022","unstructured":"Hyunseo Koh, Minhyuk Seo, Jihwan Bang, Hwanjun Song, Deokki Hong, Seulki Park, Jung-Woo Ha, and Jonghyun Choi. 2022. Online boundary-free continual learning by scheduled data prior. In ICLR."},{"key":"e_1_3_1_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2017.0-134"},{"key":"e_1_3_1_88_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-377-6.50048-7"},{"key":"e_1_3_1_89_2","doi-asserted-by":"crossref","unstructured":"Stefan Larson Anish Mahendran Joseph J. Peper Christopher Clarke Andrew Lee Parker Hill Jonathan K. Kummerfeld Kevin Leach Michael A. Laurenzano Lingjia Tang et\u00a0al. 2019. An evaluation dataset for intent classification and out-of-scope prediction. Retrieved from https:\/\/arXiv:1909.02027","DOI":"10.18653\/v1\/D19-1131"},{"key":"e_1_3_1_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV56688.2023.00642"},{"key":"e_1_3_1_91_2","unstructured":"Sungjin Lee. 2017. Toward continual learning for conversational agents. Retrieved from https:\/\/arXiv:1712.09943"},{"key":"e_1_3_1_92_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.12.004"},{"key":"e_1_3_1_93_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.398"},{"key":"e_1_3_1_94_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i11.26534"},{"key":"e_1_3_1_95_2","first-page":"333","volume-title":"COLING","author":"Li Guodun","year":"2022","unstructured":"Guodun Li, Yuchen Zhai, Qianglong Chen, Xing Gao, Ji Zhang, and Yin Zhang. 2022. Continual few-shot intent detection. In COLING. 333\u2013343."},{"key":"e_1_3_1_96_2","unstructured":"Jingyao Li Pengguang Chen Sitong Wu Chuanyang Zheng Hong Xu and Jiaya Jia. 2024. RoboCoder: Robotic learning from basic skills to general tasks with large language models. Retrieved from https:\/\/arXiv:2406.03757"},{"key":"e_1_3_1_97_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_1_98_2","doi-asserted-by":"crossref","unstructured":"Yong Lin Hangyu Lin Wei Xiong Shizhe Diao Jianmeng Liu Jipeng Zhang Rui Pan Haoxiang Wang Wenbin Hu Hanning Zhang Hanze Dong Renjie Pi Han Zhao Nan Jiang Heng Ji Yuan Yao and Tong Zhang. 2024. Mitigating the Alignment Tax of RLHF. Retrieved from https:\/\/arXiv:2309.06256","DOI":"10.18653\/v1\/2024.emnlp-main.35"},{"key":"e_1_3_1_99_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i09.7079"},{"key":"e_1_3_1_100_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i17.17768"},{"key":"e_1_3_1_101_2","doi-asserted-by":"crossref","unstructured":"Bing Liu Sahisnu Mazumder Eric Robertson and Scott Grigsby. 2023. AI autonomy: Self-initiated open-world continual learning and adaptation. AI Magazine 44 2 (2023) 185\u2013199.","DOI":"10.1002\/aaai.12087"},{"key":"e_1_3_1_102_2","unstructured":"Jiaming Liu Chenxuan Li Guanqun Wang Lily Lee Kaichen Zhou Sixiang Chen Chuyan Xiong Jiaxin Ge Renrui Zhang and Shanghang Zhang. 2024. Self-corrected multimodal large language model for end-to-end robot manipulation. Retrieved from https:\/\/arXiv:2405.17418"},{"key":"e_1_3_1_103_2","volume-title":"IJCAI","author":"Liu Qian","year":"2015","unstructured":"Qian Liu, Zhiqiang Gao, Bing Liu, and Yuanlin Zhang. 2015. Automated rule selection for aspect extraction in opinion mining. In IJCAI."},{"key":"e_1_3_1_104_2","unstructured":"Qingbin Liu Xiaoyan Yu Shizhu He Kang Liu and Jun Zhao. 2021. Lifelong intent detection via multi-strategy rebalancing. Retrieved from https:\/\/arXiv:2108.04445"},{"key":"e_1_3_1_105_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1331"},{"key":"e_1_3_1_106_2","unstructured":"Xialei Liu Xusheng Cao Haori Lu Jia-wen Xiao Andrew D. Bagdanov and Ming-Ming Cheng. 2023. Class incremental learning with pre-trained vision-language models. Retrieved from https:\/\/arXiv:2310.20348"},{"key":"e_1_3_1_107_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-15-9323-9_15"},{"key":"e_1_3_1_108_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-short.8"},{"key":"e_1_3_1_109_2","unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Retrieved from https:\/\/arXiv:1907.11692"},{"key":"e_1_3_1_110_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.447"},{"key":"e_1_3_1_111_2","article-title":"Gradient episodic memory for continual learning","author":"Lopez-Paz David","year":"2017","unstructured":"David Lopez-Paz and Marc\u2019Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_1_112_2","unstructured":"Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy Ambrosio Blanco Colin Clement Dawn Drain Daxin Jiang Duyu Tang et\u00a0al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. Retrieved from https:\/\/arXiv:2102.04664"},{"key":"e_1_3_1_113_2","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2017.2776321"},{"key":"e_1_3_1_114_2","unstructured":"Yun Luo Zhen Yang Fandong Meng Yafu Li Jie Zhou and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. Retrieved from https:\/\/arxiv.org\/abs\/2308.08747"},{"key":"e_1_3_1_115_2","unstructured":"Shirong Ma Shen Huang Shulin Huang Xiaobin Wang Yangning Li Hai-Tao Zheng Pengjun Xie Fei Huang and Yong Jiang. 2023. EcomGPT-CT: Continual pre-training of e-commerce large language models with semi-structured data. Retrieved from https:\/\/arXiv:2312.15696"},{"key":"e_1_3_1_116_2","volume-title":"ICLR","author":"Ma Yecheng Jason","year":"2024","unstructured":"Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2024. Eureka: Human-level reward design via coding large language models. In ICLR."},{"key":"e_1_3_1_117_2","first-page":"142","volume-title":"ACL","author":"Maas Andrew L.","year":"2011","unstructured":"Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL. 142\u2013150."},{"key":"e_1_3_1_118_2","doi-asserted-by":"crossref","unstructured":"Andrea Madotto Zhaojiang Lin Zhenpeng Zhou Seungwhan Moon Paul Crook Bing Liu Zhou Yu Eunjoon Cho and Zhiguang Wang. 2020. Continual learning in task-oriented dialogue systems. Retrieved from https:\/\/arXiv:2012.15504","DOI":"10.18653\/v1\/2021.emnlp-main.590"},{"key":"e_1_3_1_119_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.eacl-main.65"},{"key":"e_1_3_1_120_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2021.10.021"},{"key":"e_1_3_1_121_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2022.3213473"},{"key":"e_1_3_1_122_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-48189-5"},{"key":"e_1_3_1_123_2","unstructured":"Bryan McCann Nitish Shirish Keskar Caiming Xiong and Richard Socher. [n. d.]. The natural language decathlon: Multitask learning as question answering. arXiv 2018. Retrieved from https:\/\/arXiv:1806.08730"},{"key":"e_1_3_1_124_2","unstructured":"Sanket Vaibhav Mehta Darshan Patil Sarath Chandar and Emma Strubell. 2023. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research 24 214 (2023) 1\u201350."},{"key":"e_1_3_1_125_2","doi-asserted-by":"crossref","unstructured":"Martin Menabue Emanuele Frascaroli Matteo Boschini Enver Sangineto Lorenzo Bonicelli Angelo Porrello and Simone Calderara. 2024. Semantic residual prompts for continual learning. Retrieved from https:\/\/arXiv:2403.06870","DOI":"10.1007\/978-3-031-73030-6_1"},{"key":"e_1_3_1_126_2","doi-asserted-by":"crossref","unstructured":"Umberto Michieli Pablo Peso Parada and Mete Ozay. 2023. Online continual learning in keyword spotting for low-resource devices via pooling high-order temporal statistics. Retrieved from https:\/\/arXiv:2307.12660","DOI":"10.21437\/Interspeech.2023-90"},{"key":"e_1_3_1_127_2","doi-asserted-by":"publisher","DOI":"10.1145\/3605943"},{"key":"e_1_3_1_128_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.703"},{"key":"e_1_3_1_129_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i15.17600"},{"key":"e_1_3_1_130_2","doi-asserted-by":"publisher","DOI":"10.1145\/3632410.3632463"},{"key":"e_1_3_1_131_2","unstructured":"Siddarth Narasimhan Aaron Hao Tan Daniel Choi and Goldie Nejat. 2024. OLiVia-Nav: An online lifelong vision language approach for mobile robot social navigation. Retrieved from https:\/\/arXiv:2409.13675"},{"key":"e_1_3_1_132_2","first-page":"26129","volume-title":"ICML","author":"Ni Zixuan","year":"2023","unstructured":"Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. 2023. Continual vision-language representation learning with off-diagonal information. In ICML. 26129\u201326149."},{"key":"e_1_3_1_133_2","doi-asserted-by":"crossref","unstructured":"Jekaterina Novikova Ond\u0159ej Du\u0161ek and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. Retrieved from https:\/\/arXiv:1706.09254","DOI":"10.18653\/v1\/W17-5525"},{"key":"e_1_3_1_134_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray et\u00a0al. 2022. Training language models to follow instructions with human feedback. In NeurIPS. 27730\u201327744.","journal-title":"NeurIPS"},{"key":"e_1_3_1_135_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2019.01.012"},{"key":"e_1_3_1_136_2","volume-title":"ICLR","author":"PENG Bohao","year":"2024","unstructured":"Bohao PENG, Zhuotao Tian, Shu Liu, Ming-Chang Yang, and Jiaya Jia. 2024. Scalable language model with generalized continual learning. In ICLR."},{"key":"e_1_3_1_137_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-demo.13"},{"key":"e_1_3_1_138_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00276"},{"key":"e_1_3_1_139_2","unstructured":"Jingyang Qiao Zhizhong Zhang Xin Tan Yanyun Qu Wensheng Zhang and Yuan Xie. 2024. Gradient projection for parameter-efficient continual learning. Retrieved from https:\/\/arXiv:2405.13383"},{"key":"e_1_3_1_140_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.198"},{"key":"e_1_3_1_141_2","volume-title":"ICLR","author":"Qin Chengwei","year":"2022","unstructured":"Chengwei Qin and Shafiq Joty. 2022. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5. In ICLR."},{"key":"e_1_3_1_142_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-acl.723"},{"key":"e_1_3_1_143_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-acl.220"},{"key":"e_1_3_1_144_2","unstructured":"Haoxuan Qu Hossein Rahmani Li Xu Bryan Williams and Jun Liu. 2021. Recent advances of continual learning in computer vision: An overview. Retrieved from https:\/\/arXiv:2109.11369"},{"key":"e_1_3_1_145_2","first-page":"8748","volume-title":"ICML","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark et\u00a0al. 2021. Learning transferable visual models from natural language supervision. In ICML. PMLR, 8748\u20138763."},{"key":"e_1_3_1_146_2","unstructured":"Alec Radford. 2018. Improving language understanding by generative pre-training. (2018). https:\/\/openai.com\/index\/language-unsupervised\/"},{"key":"e_1_3_1_147_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1264"},{"key":"e_1_3_1_148_2","volume-title":"ICLR","author":"Ramasesh Vinay Venkatesh","year":"2022","unstructured":"Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. 2022. Effect of scale on catastrophic forgetting in neural networks. In ICLR."},{"key":"e_1_3_1_149_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6394"},{"key":"e_1_3_1_150_2","volume-title":"ICLR","author":"Razdaibiedina Anastasia","year":"2023","unstructured":"Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. 2023. Progressive prompts: Continual learning for language models. In ICLR."},{"key":"e_1_3_1_151_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.587"},{"key":"e_1_3_1_152_2","unstructured":"Erik F. Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. Retrieved from https:\/\/cs\/0306050"},{"key":"e_1_3_1_153_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.psychres.2021.114135"},{"key":"e_1_3_1_154_2","unstructured":"Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ili\u0107 Daniel Hesslow Roman Castagn\u00e9 Alexandra Sasha Luccioni Fran\u00e7ois Yvon Matthias Gall\u00e9 et\u00a0al. 2022. Bloom: A 176b-parameter open-access multilingual language model. Retrieved from https:\/\/arXiv:2211.05100"},{"key":"e_1_3_1_155_2","first-page":"25278","article-title":"Laion-5b: An open large-scale dataset for training next generation image-text models","author":"Schuhmann Christoph","year":"2022","unstructured":"Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman et\u00a0al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurIPS. 25278\u201325294.","journal-title":"NeurIPS"},{"key":"e_1_3_1_156_2","unstructured":"Christoph Schuhmann Richard Vencu Romain Beaumont Robert Kaczmarczyk Clayton Mullis Aarush Katta Theo Coombes Jenia Jitsev and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. Retrieved from https:\/\/arXiv:2111.02114"},{"key":"e_1_3_1_157_2","doi-asserted-by":"crossref","unstructured":"Sebastian Schuster Sonal Gupta Rushin Shah and Mike Lewis. 2018. Cross-lingual transfer learning for multilingual task oriented dialog. Retrieved from https:\/\/arXiv:1810.13327","DOI":"10.18653\/v1\/N19-1380"},{"key":"e_1_3_1_158_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.410"},{"key":"e_1_3_1_159_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.410"},{"key":"e_1_3_1_160_2","unstructured":"Abigail See Peter J. Liu and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. Retrieved from https:\/\/arXiv:1704.04368"},{"key":"e_1_3_1_161_2","unstructured":"Minhyuk Seo Diganta Misra Seongwon Cho Minjae Lee and Jonghyun Choi. 2024. Just say the name: Online continual learning with category names only via data generation. Retrieved from https:\/\/arXiv:2403.10853"},{"key":"e_1_3_1_162_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10846-022-01603-6"},{"key":"e_1_3_1_163_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1238"},{"key":"e_1_3_1_164_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1126"},{"key":"e_1_3_1_165_2","unstructured":"Haizhou Shi Zihao Xu Hengyi Wang Weiyi Qin Wenyuan Wang Yibin Wang and Hao Wang. 2024. Continual learning of large language models: A comprehensive survey. Retrieved from https:\/\/arXiv:2404.16789"},{"key":"e_1_3_1_166_2","unstructured":"Chenyang Song Xu Han Zheni Zeng Kuai Li Chen Chen Zhiyuan Liu Maosong Sun and Tao Yang. 2023. ConPET: Continual parameter-efficient tuning for large language models. Retrieved from https:\/\/arXiv:2309.14763"},{"key":"e_1_3_1_167_2","volume-title":"ICLR","author":"Sun Fan-Keng","year":"2020","unstructured":"Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020. LAMOL: LAnguage MOdeling for lifelong language learning. In ICLR. OpenReview.net."},{"key":"e_1_3_1_168_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6428"},{"key":"e_1_3_1_169_2","doi-asserted-by":"crossref","unstructured":"Longxiang Tang Zhuotao Tian Kai Li Chunming He Hantao Zhou Hengshuang Zhao Xiu Li and Jiaya Jia. 2024. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. Retrieved from https:\/\/arXiv:2407.05342","DOI":"10.1007\/978-3-031-72764-1_20"},{"key":"e_1_3_1_170_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00020"},{"key":"e_1_3_1_171_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et\u00a0al. 2023. Llama: Open and efficient foundation language models. Retrieved from https:\/\/arXiv:2302.13971"},{"key":"e_1_3_1_172_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340544"},{"key":"e_1_3_1_173_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-022-00568-3"},{"key":"e_1_3_1_174_2","doi-asserted-by":"crossref","unstructured":"Steven Vander Eeckt et\u00a0al. 2023. Rehearsal-free online continual learning for automatic speech recognition. Retrieved from https:\/\/arxiv.org\/abs\/2306.10860","DOI":"10.21437\/Interspeech.2023-788"},{"key":"e_1_3_1_175_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-naacl.84"},{"key":"e_1_3_1_176_2","doi-asserted-by":"crossref","unstructured":"Michael Volske Martin Potthast Shahbaz Syed and Benno Stein. 2017. TL;DR: Mining reddit to learn automatic summarization. In NewSumm. 59\u201363.","DOI":"10.18653\/v1\/W17-4508"},{"key":"e_1_3_1_177_2","article-title":"Superglue: A stickier benchmark for general-purpose language understanding systems","author":"Wang Alex","year":"2019","unstructured":"Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_1_178_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-5446"},{"key":"e_1_3_1_179_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467107"},{"key":"e_1_3_1_180_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN55064.2022.9892670"},{"key":"e_1_3_1_181_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1086"},{"key":"e_1_3_1_182_2","unstructured":"Leyuan Wang Liuyu Xiang Yujie Wei Yunlong Wang and Zhaofeng He. 2024. CLIP model is an efficient online lifelong learner. Retrieved from https:\/\/arXiv:2405.15155"},{"key":"e_1_3_1_183_2","doi-asserted-by":"publisher","unstructured":"Liyuan Wang Xingxing Zhang Hang Su and Jun Zhu. 2024. A comprehensive survey of continual learning: Theory method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 8 (2024) 5362\u20135383. DOI:10.1109\/TPAMI.2024.3367329","DOI":"10.1109\/TPAMI.2024.3367329"},{"key":"e_1_3_1_184_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11633-022-1410-8"},{"key":"e_1_3_1_185_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.715"},{"key":"e_1_3_1_186_2","unstructured":"Xiao Wang Yuansen Zhang Tianze Chen Songyang Gao Senjie Jin Xianjun Yang Zhiheng Xi Rui Zheng Yicheng Zou Tao Gui Qi Zhang and Xuanjing Huang. 2024. TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models."},{"key":"e_1_3_1_187_2","unstructured":"Yabin Wang Zhiwu Huang and Xiaopeng Hong. 2022. S-Prompts learning with pre-trained transformers: An occam\u2019s razor for domain incremental learning. In Advances in Neural Information Processing Systems. Retrieved from https:\/\/openreview.net\/forum?id=ZVe_WeMold"},{"key":"e_1_3_1_188_2","doi-asserted-by":"crossref","unstructured":"Yizhong Wang Swaroop Mishra Pegah Alipoormolabashi Yeganeh Kordi Amirreza Mirzaei Anjana Arunkumar Arjun Ashok Arut Selvan Dhanasekaran Atharva Naik David Stap et\u00a0al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. Retrieved from https:\/\/arXiv:2204.07705","DOI":"10.18653\/v1\/2022.emnlp-main.340"},{"key":"e_1_3_1_189_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.612"},{"key":"e_1_3_1_190_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.39"},{"key":"e_1_3_1_191_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19809-0_36"},{"key":"e_1_3_1_192_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00024"},{"key":"e_1_3_1_193_2","unstructured":"Tsung-Hsien Wen Milica Gasic Nikola Mrksic Pei-Hao Su David Vandyke and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. Retrieved from https:\/\/arXiv:1508.01745"},{"key":"e_1_3_1_194_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-industry.51"},{"key":"e_1_3_1_195_2","unstructured":"Tongtong Wu Linhao Luo Yuan-Fang Li Shirui Pan Thuy-Trang Vu and Gholamreza Haffari. 2024. Continual learning for large language models: A survey. Retrieved from https:\/\/arXiv:2402.01364"},{"key":"e_1_3_1_196_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2022.05.014"},{"key":"e_1_3_1_197_2","unstructured":"Yuhao Wu Tongjun Shi Karthick Sharma Chun Wei Seah and Shuhao Zhang. 2023. Online continual knowledge learning for language models. Retrieved from https:\/\/arXiv:2311.09632"},{"key":"e_1_3_1_198_2","unstructured":"Congying Xia Wenpeng Yin Yihao Feng and Philip Yu. 2021. Incremental few-shot text classification with multi-round new classes: Formulation dataset and system. Retrieved from https:\/\/arXiv:2104.11882"},{"key":"e_1_3_1_199_2","doi-asserted-by":"crossref","unstructured":"Yong Xie Karan Aggarwal and Aitzaz Ahmad. 2023. Efficient continual pre-training for building domain specific large language models. Retrieved from https:\/\/arXiv:2311.08545","DOI":"10.18653\/v1\/2024.findings-acl.606"},{"key":"e_1_3_1_200_2","unstructured":"Hu Xu Bing Liu Lei Shu and Philip S. Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. Retrieved from https:\/\/arXiv:1904.02232"},{"key":"e_1_3_1_201_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2016.12.008"},{"key":"e_1_3_1_202_2","doi-asserted-by":"crossref","unstructured":"Prateek Yadav Qing Sun Hantian Ding Xiaopeng Li Dejiao Zhang Ming Tan Xiaofei Ma Parminder Bhatia Ramesh Nallapati Murali Krishna Ramanathan et\u00a0al. 2023. Exploring continual learning for code generation models. Retrieved from https:\/\/arXiv:2307.02435","DOI":"10.18653\/v1\/2023.acl-short.68"},{"key":"e_1_3_1_203_2","unstructured":"An Yang Junyang Lin Rui Men Chang Zhou Le Jiang Xianyan Jia Ang Wang Jie Zhang Jiamang Wang Yong Li et\u00a0al. 2021. M6-t: Exploring sparse expert models and beyond. Retrieved from https:\/\/arXiv:2105.15082"},{"key":"e_1_3_1_204_2","unstructured":"Huahui Yi Ziyuan Qin Qicheng Lao Wei Xu Zekun Jiang Dequan Wang Shaoting Zhang and Kang Li. 2023. Towards general purpose medical AI: Continual learning medical foundation model. Retrieved from https:\/\/arXiv:2303.06580"},{"key":"e_1_3_1_205_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.218"},{"key":"e_1_3_1_206_2","unstructured":"Dani Yogatama Cyprien de Masson d\u2019Autume Jerome Connor Tomas Kocisky Mike Chrzanowski Lingpeng Kong Angeliki Lazaridou Wang Ling Lei Yu Chris Dyer et\u00a0al. 2019. Learning and evaluating general linguistic intelligence. Retrieved from https:\/\/arXiv:1901.11373"},{"key":"e_1_3_1_207_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00166"},{"key":"e_1_3_1_208_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.02191"},{"key":"e_1_3_1_209_2","doi-asserted-by":"crossref","unstructured":"Yu-Chu Yu Chi-Pin Huang Jr-Jen Chen Kai-Po Chang Yung-Hsuan Lai Fu-En Yang and Yu-Chiang Frank Wang. 2024. Select and distill: Selective dual-teacher knowledge transfer for continual learning on vision-language models. Retrieved from https:\/\/arXiv:2403.09296","DOI":"10.1007\/978-3-031-73347-5_13"},{"key":"e_1_3_1_210_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401156"},{"key":"e_1_3_1_211_2","doi-asserted-by":"publisher","unstructured":"Fajie Yuan Guoxiao Zhang Alexandros Karatzoglou Joemon Jose Beibei Kong and Yudong Li. 2021. One person one model one world: Learning continual user representation without forgetting. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201921). Association for Computing Machinery Virtual Event Canada 696\u2013705. DOI:10.1145\/3404835.3462884","DOI":"10.1145\/3404835.3462884"},{"key":"e_1_3_1_212_2","article-title":"Defending against neural fake news","author":"Zellers Rowan","year":"2019","unstructured":"Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_1_213_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.106775"},{"key":"e_1_3_1_214_2","unstructured":"Han Zhang Lin Gui Yu Lei Yuanzhao Zhai Yehong Zhang Yulan He Hui Wang Yue Yu Kam-Fai Wong Bin Liang et\u00a0al. 2024. COPR: Continual human preference learning via optimal policy regularization. Retrieved from https:\/\/arXiv:2402.14228"},{"key":"e_1_3_1_215_2","unstructured":"Han Zhang Lin Gui Yuanzhao Zhai Hui Wang Yu Lei and Ruifeng Xu. 2023. Copf: Continual learning human preference through optimal policy fitting. Retrieved from https:\/\/arXiv:2310.15694"},{"key":"e_1_3_1_216_2","volume-title":"ICLR","author":"Zhang Han","year":"2024","unstructured":"Han Zhang, Yu Lei, Lin Gui, Min Yang, Yulan He, Hui Wang, and Ruifeng Xu. 2024. CPPO: Continual learning for reinforcement learning with human feedback. In ICLR."},{"key":"e_1_3_1_217_2","unstructured":"Peiyan Zhang and Sunghun Kim. 2023. A survey on incremental update for neural recommender systems. Retrieved from https:\/\/arXiv:2303.02851"},{"key":"e_1_3_1_218_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01831"},{"key":"e_1_3_1_219_2","article-title":"Character-level convolutional networks for text classification","author":"Zhang Xiang","year":"2015","unstructured":"Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_1_220_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.255"},{"key":"e_1_3_1_221_2","unstructured":"Yating Zhang Yexiang Wang Fei Cheng Sadao Kurohashi et\u00a0al. 2023. Reformulating domain adaptation of large language models as adapt-retrieve-revise. Retrieved from https:\/\/arXiv:2310.03328"},{"key":"e_1_3_1_222_2","doi-asserted-by":"crossref","unstructured":"Zihan Zhang Meng Fang Ling Chen and Mohammad-Reza Namazi-Rad. 2023. Citb: A benchmark for continual instruction tuning. Retrieved from https:\/\/arXiv:2310.14510","DOI":"10.18653\/v1\/2023.findings-emnlp.633"},{"key":"e_1_3_1_223_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et\u00a0al. 2023. A survey of large language models. Retrieved from https:\/\/arXiv:2303.18223"},{"key":"e_1_3_1_224_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.766"},{"key":"e_1_3_1_225_2","unstructured":"Junhao Zheng Shengjie Qiu Chengming Shi and Qianli Ma. 2024. Towards lifelong learning of large language models: A survey. Retrieved from https:\/\/arXiv:2406.06391"},{"key":"e_1_3_1_226_2","doi-asserted-by":"publisher","unstructured":"Mengyu Zheng Yehui Tang Zhiwei Hao Kai Han Yunhe Wang and Chang Xu. 2024. Adapt without forgetting: Distill proximity from dual teachers in vision-language models. In Computer Vision \u2013 ECCV 2024: 18th European Conference Milan Italy September 29\u2013October 4 2024 Proceedings Part LIV Springer-Verlag Milan Italy 109\u2013125. DOI:10.1007\/978-3-031-72949-2_7","DOI":"10.1007\/978-3-031-72949-2_7"},{"key":"e_1_3_1_227_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01752"},{"key":"e_1_3_1_228_2","unstructured":"Victor Zhong Caiming Xiong and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. Retrieved from https:\/\/arXiv:1709.00103"},{"key":"e_1_3_1_229_2","unstructured":"Da-Wei Zhou Qi-Wei Wang Zhi-Hong Qi Han-Jia Ye De-Chuan Zhan and Ziwei Liu. 2023. Deep class-incremental learning: A survey. Retrieved from https:\/\/arXiv:2302.03648"},{"key":"e_1_3_1_230_2","unstructured":"Da-Wei Zhou Yuanhan Zhang Jingyi Ning Han-Jia Ye De-Chuan Zhan and Ziwei Liu. 2023. Learning without forgetting for vision-language models. Retrieved from https:\/\/arXiv:2305.19270"},{"key":"e_1_3_1_231_2","first-page":"1","article-title":"ChatGPT: Potential, prospects, and limitations","author":"Zhou Jie","year":"2023","unstructured":"Jie Zhou, Pei Ke, Xipeng Qiu, Minlie Huang, and Junping Zhang. 2023. ChatGPT: Potential, prospects, and limitations. FITEE (2023), 1\u20136.","journal-title":"FITEE"},{"key":"e_1_3_1_232_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.02034"},{"key":"e_1_3_1_233_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.80"},{"key":"e_1_3_1_234_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.11"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3705725","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3705725","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:18:12Z","timestamp":1750281492000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3705725"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,9]]},"references-count":233,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,5,31]]}},"alternative-id":["10.1145\/3705725"],"URL":"https:\/\/doi.org\/10.1145\/3705725","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,9]]},"assertion":[{"value":"2024-05-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-18","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}