{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T05:54:37Z","timestamp":1768802077867,"version":"3.49.0"},"reference-count":64,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,11,23]],"date-time":"2023-11-23T00:00:00Z","timestamp":1700697600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"CSIRO\u2019s Machine Learning and Artificial Intelligence Future Science Platform (MLAI FSP)","award":["DP230101176"],"award-info":[{"award-number":["DP230101176"]}]},{"name":"CSIRO\u2019s Research Plus Postgraduate Scholarship","award":["DP230101176"],"award-info":[{"award-number":["DP230101176"]}]},{"name":"Australian Research Council (ARC)","award":["DP230101176"],"award-info":[{"award-number":["DP230101176"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Lifelong learning portrays learning gradually in nonstationary environments and emulates the process of human learning, which is efficient, robust, and able to learn new concepts incrementally from sequential experience. To equip neural networks with such a capability, one needs to overcome the problem of catastrophic forgetting, the phenomenon of forgetting past knowledge while learning new concepts. In this work, we propose a novel knowledge distillation algorithm that makes use of contrastive learning to help a neural network to preserve its past knowledge while learning from a series of tasks. Our proposed generalized form of contrastive distillation strategy tackles catastrophic forgetting of old knowledge, and minimizes semantic drift by maintaining a similar embedding space, as well as ensures compactness in feature distribution to accommodate novel tasks in a current model. Our comprehensive study shows that our method achieves improved performances in the challenging class-incremental, task-incremental, and domain-incremental learning for supervised scenarios.<\/jats:p>","DOI":"10.3390\/jimaging9120259","type":"journal-article","created":{"date-parts":[[2023,11,23]],"date-time":"2023-11-23T11:31:25Z","timestamp":1700739085000},"page":"259","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["CL3: Generalization of Contrastive Loss for Lifelong Learning"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2727-6704","authenticated-orcid":false,"given":"Kaushik","family":"Roy","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Systems Engineering, Faculty of Engineering, Monash University, Clayton, VIC 3800, Australia"},{"name":"Data61, CSIRO, Brisbane, QLD 4069, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christian","family":"Simon","sequence":"additional","affiliation":[{"name":"School of Engineering, College of Engineering, Computing and Cybernetics, Australian National University, Canberra, ACT 2601, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8169-3560","authenticated-orcid":false,"given":"Peyman","family":"Moghadam","sequence":"additional","affiliation":[{"name":"Data61, CSIRO, Brisbane, QLD 4069, Australia"},{"name":"School of Electrical Engineering and Robotics, Faculty of Engineering, Queensland University of Technology, Brisbane, QLD 4000, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mehrtash","family":"Harandi","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Systems Engineering, Faculty of Engineering, Monash University, Clayton, VIC 3800, Australia"},{"name":"Data61, CSIRO, Brisbane, QLD 4069, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,11,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neunet.2012.09.017","article-title":"Adaptive Resonance Theory: How a brain learns to consciously attend, learn, and recognize a changing world","volume":"37","author":"Grossberg","year":"2013","journal-title":"Neural Netw."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.neunet.2019.01.012","article-title":"Continual lifelong learning with neural networks: A review","volume":"113","author":"Parisi","year":"2019","journal-title":"Neural Netw."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/S0079-7421(08)60536-8","article-title":"Catastrophic interference in connectionist networks: The sequential learning problem","volume":"Volume 24","author":"McCloskey","year":"1989","journal-title":"Psychology of Learning and Motivation"},{"key":"ref_4","unstructured":"Nguyen, C.V., Achille, A., Lam, M., Hassner, T., Mahadevan, V., and Soatto, S. (2019). Toward understanding catastrophic forgetting in continual learning. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1080\/09540099550039318","article-title":"Catastrophic forgetting, rehearsal and pseudorehearsal","volume":"7","author":"Robins","year":"1995","journal-title":"Connect. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Rebuffi, S.A., Kolesnikov, A., Sperl, G., and Lampert, C.H. (2017, January 21\u201326). icarl: Incremental classifier and representation learning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.587"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Doan, H.G., Luong, H.Q., Ha, T.O., and Pham, T.T.T. (2023). An Efficient Strategy for Catastrophic Forgetting Reduction in Incremental Learning. Electronics, 12.","DOI":"10.3390\/electronics12102265"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hou, S., Pan, X., Loy, C.C., Wang, Z., and Lin, D. (2019, January 16\u201320). Learning a unified classifier incrementally via rebalancing. Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00092"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Grossberg, S. (1982). Studies of Mind and Brain, Springer.","DOI":"10.1007\/978-94-009-7758-7"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/S0734-189X(87)80014-2","article-title":"A massively parallel architecture for a self-organizing neural pattern recognition machine","volume":"37","author":"Carpenter","year":"1987","journal-title":"Comput. Vision, Graph. Image Process."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Mermillod, M., Bugaiska, A., and Bonin, P. (2013). The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol., 4.","DOI":"10.3389\/fpsyg.2013.00504"},{"key":"ref_12","unstructured":"Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. (2020). Dark experience for general continual learning: A strong, simple baseline. arXiv."},{"key":"ref_13","unstructured":"Douillard, A., Cord, M., Ollion, C., Robert, T., and Valle, E. Podnet: Pooled outputs distillation for small-tasks incremental learning. Proceedings of the European Conference on Computer Vision."},{"key":"ref_14","unstructured":"Van de Ven, G.M., and Tolias, A.S. (2019). Three scenarios for continual learning. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Roy, K., Moghadam, P., and Harandi, M. (2023, January 8\u201312). L3DMC: Lifelong Learning using Distillation via Mixed-Curvature Space. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Vancouver, BC, Canada.","DOI":"10.1007\/978-3-031-43895-0_12"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/j.neunet.2023.07.047","article-title":"Subspace distillation for continual learning","volume":"167","author":"Roy","year":"2023","journal-title":"Neural Netw."},{"key":"ref_17","unstructured":"Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. arXiv."},{"key":"ref_18","unstructured":"Knights, J., Harwood, B., Ward, D., Vanderkop, A., Mackenzie-Ross, O., and Moghadam, P. (2020, January 10\u201315). Temporally Coherent Embeddings for Self-Supervised Video Representation Learning. Proceedings of the 25th International Conference on Pattern Recognition (ICPR), Milan, Italy."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Fini, E., da Costa, V.G.T., Alameda-Pineda, X., Ricci, E., Alahari, K., and Mairal, J. (2022, January 18\u201324). Self-supervised models are continual learners. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00940"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Cha, H., Lee, J., and Shin, J. (2021, January 11\u201317). Co2l: Contrastive continual learning. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00938"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2935","DOI":"10.1109\/TPAMI.2017.2773081","article-title":"Learning without forgetting","volume":"40","author":"Li","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","unstructured":"Zenke, F., Poole, B., and Ganguli, S. (2017). Continual learning through synaptic intelligence. Proc. Mach. Learn. Res., 70."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3521","DOI":"10.1073\/pnas.1611835114","article-title":"Overcoming catastrophic forgetting in neural networks","volume":"114","author":"Kirkpatrick","year":"2017","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_24","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Simon, C., Koniusz, P., and Harandi, M. (2021, January 20\u201325). On learning the geodesic path for incremental learning. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00164"},{"key":"ref_26","first-page":"350","article-title":"Experience replay for continual learning","volume":"32","author":"Rolnick","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_27","first-page":"11816","article-title":"Gradient based sample selection for online continual learning","volume":"32","author":"Aljundi","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Caccia, M., Lin, M., and Page-Caccia, L. (2019). Online continual learning with maximal interfered retrieval. Adv. Neural Inf. Process. Syst., 11849\u201311860.","DOI":"10.1109\/CVPR.2019.01151"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., and Fu, Y. (2019, January 15\u201320). Large scale incremental learning. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00046"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Liu, X., Wu, C., Menta, M., Herranz, L., Raducanu, B., Bagdanov, A.D., Jui, S., and van de Weijer, J. (2020, January 14\u201319). Generative Feature Replay For Class-Incremental Learning. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00121"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Shen, G., Zhang, S., Chen, X., and Deng, Z.H. (2021, January 18\u201322). Generative feature replay with orthogonal weight modification for continual learning. Proceedings of the IEEE 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.","DOI":"10.1109\/IJCNN52387.2021.9534437"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Pellegrini, L., Graffieti, G., Lomonaco, V., and Maltoni, D. (2020, January 25\u201329). Latent replay for real-time continual learning. Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.","DOI":"10.1109\/IROS45743.2020.9341460"},{"key":"ref_33","first-page":"2990","article-title":"Continual learning with deep generative replay","volume":"30","author":"Shin","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Lesort, T., Caselles-Dupr\u00e9, H., Garcia-Ortiz, M., Stoian, A., and Filliat, D. (2019, January 14\u201319). Generative models from the perspective of continual learning. Proceedings of the IEEE 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.","DOI":"10.1109\/IJCNN.2019.8851986"},{"key":"ref_35","unstructured":"Van de Ven, G.M., and Tolias, A.S. (2018). Generative replay with feedback connections as a general strategy for continual learning. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. (2022, January 18\u201324). Learning to prompt for continual learning. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00024"},{"key":"ref_37","unstructured":"Oord, A.v.d., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv."},{"key":"ref_38","unstructured":"Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. (2020). Supervised contrastive learning. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Caron, M., Touvron, H., Misra, I., J\u00e9gou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021, January 11\u201317). Emerging properties in self-supervised vision transformers. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00951"},{"key":"ref_40","unstructured":"Hadsell, R., Chopra, S., and LeCun, Y. (2006, January 17\u201322). Dimensionality reduction by learning an invariant mapping. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201906), New York, NY, USA."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 14\u201319). Momentum contrast for unsupervised visual representation learning. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00975"},{"key":"ref_42","first-page":"21271","article-title":"Bootstrap your own latent-a new approach to self-supervised learning","volume":"33","author":"Grill","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Xie, E., Ding, J., Wang, W., Zhan, X., Xu, H., Sun, P., Li, Z., and Luo, P. (2021, January 11\u201317). Detco: Unsupervised contrastive learning for object detection. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00828"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Xie, J., Xiang, J., Chen, J., Hou, X., Zhao, X., and Shen, L. (2022, January 18\u201324). C2am: Contrastive learning of class-agnostic activation map for weakly supervised object localization and semantic segmentation. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00106"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Gao, T., Yao, X., and Chen, D. (2021). Simcse: Simple contrastive learning of sentence embeddings. arXiv.","DOI":"10.18653\/v1\/2021.emnlp-main.552"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Alakooz, A.S., and Ammour, N. (2022, January 17\u201322). A contrastive continual learning for the classification of remote sensing imagery. Proceedings of the IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium, Kuala Lumpur, Malaysia.","DOI":"10.1109\/IGARSS46834.2022.9884527"},{"key":"ref_47","unstructured":"Luo, Y., Lin, X., Yang, Z., Meng, F., Zhou, J., and Zhang, Y. (2023). Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion. arXiv."},{"key":"ref_48","unstructured":"Wang, Z., Liu, L., Kong, Y., Guo, J., and Tao, D. Online continual learning with contrastive vision transformer. Proceedings of the European Conference on Computer Vision."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Varshney, V., Patidar, M., Kumar, R., Vig, L., and Shroff, G. (2022). Prompt augmented generative replay via supervised contrastive learning for lifelong intent detection. Find. Assoc. Comput. Linguist. NAACL, 1113\u20131127.","DOI":"10.18653\/v1\/2022.findings-naacl.84"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Mai, Z., Li, R., Kim, H., and Sanner, S. (2021, January 20\u201325). Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPRW53098.2021.00398"},{"key":"ref_51","unstructured":"Chen, T., and Li, L. (2020). Intriguing Properties of Contrastive Losses. arXiv."},{"key":"ref_52","first-page":"513","article-title":"Neighbourhood components analysis","volume":"17","author":"Goldberger","year":"2004","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"ref_53","unstructured":"Smola, A.J., and Sch\u00f6lkopf, B. (1998). Learning with Kernels, Citeseer."},{"key":"ref_54","unstructured":"Wang, T., and Isola, P. (2020, January 13\u201318). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_56","unstructured":"Krizhevsky, A., Hinton, G., and Nair, V. (2023, October 30). Learning Multiple Layers of Features from Tiny Images. Available online: https:\/\/www.cs.toronto.edu\/~kriz\/cifar.html."},{"key":"ref_57","unstructured":"Stanford (2023, October 30). Tiny ImageNet Challenge (CS231n). Available online: http:\/\/cs231n.stanford.edu\/tiny-imagenet-200.zip."},{"key":"ref_58","unstructured":"Lopez-Paz, D., and Ranzato, M. (2017). Gradient episodic memory for continual learning. arXiv."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Deng","year":"2015","journal-title":"Int. J. Comput. Vis. (IJCV)"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_61","unstructured":"Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinska, A., Teh, Y.W., Pascanu, R., and Hadsell, R. (2018, January 10\u201315). Progress & compress: A scalable framework for continual learning. Proceedings of the ICML. PMLR, Stockholm, Sweden."},{"key":"ref_62","unstructured":"Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. (2018). Efficient lifelong learning with a-gem. arXiv."},{"key":"ref_63","unstructured":"Benjamin, A.S., Rolnick, D., and Kording, K. (2019). Measuring and regularizing networks in function space. arXiv."},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Yu, L., Twardowski, B., Liu, X., Herranz, L., Wang, K., Cheng, Y., Jui, S., and Weijer, J.v.d. (2020, January 13\u201319). Semantic drift compensation for class-incremental learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00701"}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/12\/259\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:28:46Z","timestamp":1760131726000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/12\/259"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,23]]},"references-count":64,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["jimaging9120259"],"URL":"https:\/\/doi.org\/10.3390\/jimaging9120259","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,23]]}}}