{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:55:07Z","timestamp":1760151307414,"version":"build-2065373602"},"reference-count":38,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,2,21]],"date-time":"2022-02-21T00:00:00Z","timestamp":1645401600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004242","name":"Princess Nourah bint Abdulrahman University","doi-asserted-by":"publisher","award":["PNURSP2022R40"],"award-info":[{"award-number":["PNURSP2022R40"]}],"id":[{"id":"10.13039\/501100004242","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Human beings tend to incrementally learn from the rapidly changing environment without comprising or forgetting the already learned representations. Although deep learning also has the potential to mimic such human behaviors to some extent, it suffers from catastrophic forgetting due to which its performance on already learned tasks drastically decreases while learning about newer knowledge. Many researchers have proposed promising solutions to eliminate such catastrophic forgetting during the knowledge distillation process. However, to our best knowledge, there is no literature available to date that exploits the complex relationships between these solutions and utilizes them for the effective learning that spans over multiple datasets and even multiple domains. In this paper, we propose a continual learning objective that encompasses mutual distillation loss to understand such complex relationships and allows deep learning models to effectively retain the prior knowledge while adapting to the new classes, new datasets, and even new applications. The proposed objective was rigorously tested on nine publicly available, multi-vendor, and multimodal datasets that span over three applications, and it achieved the top-1 accuracy of 0.9863% and an F1-score of 0.9930.<\/jats:p>","DOI":"10.3390\/s22041667","type":"journal-article","created":{"date-parts":[[2022,2,21]],"date-time":"2022-02-21T20:48:41Z","timestamp":1645476521000},"page":"1667","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Continual Learning Objective for Analyzing Complex Knowledge Representations"],"prefix":"10.3390","volume":"22","author":[{"given":"Asad Mansoor","family":"Khan","sequence":"first","affiliation":[{"name":"Department of Computer and Software Engineering, National University of Sciences and Technology, Islamabad 44000, Pakistan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5896-8677","authenticated-orcid":false,"given":"Taimur","family":"Hassan","sequence":"additional","affiliation":[{"name":"Department of Computer and Software Engineering, National University of Sciences and Technology, Islamabad 44000, Pakistan"},{"name":"Center for Cyber-Physical Systems (C2PS), Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi 127788, United Arab Emirates"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Muhammad Usman","family":"Akram","sequence":"additional","affiliation":[{"name":"Department of Computer and Software Engineering, National University of Sciences and Technology, Islamabad 44000, Pakistan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6421-6001","authenticated-orcid":false,"given":"Norah Saleh","family":"Alghamdi","sequence":"additional","affiliation":[{"name":"Department of Computer Sciences, College of Computer and Information Sciences, Princess Nourah Bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Naoufel","family":"Werghi","sequence":"additional","affiliation":[{"name":"Center for Cyber-Physical Systems (C2PS), Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi 127788, United Arab Emirates"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1109\/JBHI.2020.2982914","article-title":"RAG-FW: A hybrid convolutional framework for the automated extraction of retinal lesions and lesion-influenced grading of human retinal pathology","volume":"25","author":"Hassan","year":"2020","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lieder, I., Segal, M., Avidan, E., Cohen, A., and Hope, T. (2019, January 9\u201312). Learning a Faceted Customer Segmentation for Discovering new Business Opportunities at Intel. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.","DOI":"10.1109\/BigData47090.2019.9006589"},{"key":"ref_3","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Castro, F.M., Mar\u00edn-Jim\u00e9nez, M.J., Guil, N., Schmid, C., and Alahari, K. (2018, January 8\u201314). End-to-End Incremental Learning. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01258-8_15"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2935","DOI":"10.1109\/TPAMI.2017.2773081","article-title":"Learning without Forgetting","volume":"40","author":"Li","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Aljundi, R., Chakravarty, P., and Tuytelaars, T. (2017, January 21\u201326). Expert Gate: Lifelong Learning with a Network of Experts. Proceedings of the IEEE Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.753"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Rebuffi, S.A., Kolesnikov, A., Sperl, G., and Lampert, C.H. (2017, January 21\u201326). iCaRL: Incremental Classifier and Representation Learning. Proceedings of the IEEE Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.587"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.neunet.2019.09.010","article-title":"Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning","volume":"121","author":"Roy","year":"2020","journal-title":"Neural Netw."},{"key":"ref_9","unstructured":"Tian, Y., Krishnan, D., and Isola, P. (2020, January 26\u201330). Contrastive Representation Distillation. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mirzadeh, S.I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadeh, H. (2020, January 7\u201312). Improved Knowledge Distillation via Teacher Assistant. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i04.5963"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Gao, M., Wan, L., Wang, Y., and Chen, J. (2020, January 13\u201315). Stage-by-stage knowledge distillation. Proceedings of the Twelfth International Conference on Graphics and Image Processing (ICGIP 2020), Xi\u2019an, China.","DOI":"10.1117\/12.2589331"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Son, W., Na, J., Choi, J., and Hwang, W. (2021, January 11\u201317). Densely guided knowledge distillation using multiple teacher assistants. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00926"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ahn, S., Hu, S.X., Damianou, A., Lawrence, N.D., and Dai, Z. (2019, January 16\u201320). Variational Information Distillation for Knowledge Transfer. Proceedings of the IEEE Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00938"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Cho, J.H., and Hariharan, B. (2019). On the Efficacy of Knowledge Distillation. arXiv.","DOI":"10.1109\/ICCV.2019.00489"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Yuan, L., Tay, F.E., Li, G., Wang, T., and Feng, J. (2020, January 13\u201319). Revisiting knowledge distillation via label smoothing regularization. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00396"},{"key":"ref_16","unstructured":"Lee, K., Lee, K., Shin, J., and Lee, H. (November, January 27). Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zhang, J., Zhang, J., Ghosh, S., Li, D., Tasci, S., Heck, L., Zhang, H., and Kuo, C.C.J. (2020, January 1\u20135). Class-incremental learning via deep model consolidation. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV 2020), Snowmass Village, CO, USA.","DOI":"10.1109\/WACV45572.2020.9093365"},{"key":"ref_18","unstructured":"Lopez-Paz, D., and Ranzato, M.A. (2017, January 4\u20139). Gradient Episodic Memory for Continual Learning. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_19","unstructured":"Chaudhry, A., Ranzato, M.A., Rohrbach, M., and Elhoseiny, M. (2017, January 4\u20139). Efficient lifelong learning with A-GEM. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hegde, S., Prasad, R., Hebbalaguppe, R., and Kumar, V. (2019). Variational Student: Learning Compact and Sparser Networks in Knowledge Distillation Framework. arXiv.","DOI":"10.1109\/ICASSP40776.2020.9054157"},{"key":"ref_21","unstructured":"Hassan, T., Akcay, S., Bennamoun, M., Khan, S., and Werghi, N. (2019). Cascaded Structure Tensor Framework for Robust Identification of Heavily Occluded Baggage Items from Multi-Vendor X-ray Scans. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ak\u00e7ay, S., Atapour-Abarghouei, A., and Breckon, T.P. (2019). Skip-GANomaly: Skip Connected and Adversarially Trained Encoder-Decoder Anomaly Detection. arXiv.","DOI":"10.1109\/IJCNN.2019.8851808"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Miao, C., Xie, L., Wan, F., Su, C., Liu, H., Jiao, J., and Ye, Q.S. (2019, January 16\u201320). SIXray: A Large-scale Security Inspection X-ray Benchmark for Prohibited Item Discovery in Overlapping Images. Proceedings of the IEEE Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00222"},{"key":"ref_24","unstructured":"Ak\u00e7ay, S., and Breckon, T. (2020). Towards Automatic Threat Detection: A Survey of Advances of Deep Learning within X-ray Security Imaging. arXiv."},{"key":"ref_25","unstructured":"(2020, March 01). Pneumonia. World Health Organization, Available online: https:\/\/www.who.int\/news-room\/fact-sheets\/detail\/pneumonia."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1016\/j.cell.2018.02.010","article-title":"Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning","volume":"172","author":"Kermany","year":"2018","journal-title":"Cell"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hassan, B., Hassan, T., Li, B., Ahmed, R., and Hassan, O. (2019). Deep Ensemble Learning Based Objective Grading of Macular Edema by Extracting Clinically Significant Findings from Fused Retinal Imaging Modalities. Sensors, 19.","DOI":"10.3390\/s19132970"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1016\/j.compbiomed.2018.12.015","article-title":"Deep structure tensor graph search framework for automated extraction and characterization of retinal layers and fluid pathology in retinal SD-OCT scans","volume":"105","author":"Hassan","year":"2019","journal-title":"Comput. Biol. Med."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1959","DOI":"10.1109\/TMI.2019.2898414","article-title":"Attention to Lesion: Lesion-Aware Convolutional Neural Network for Retinal Optical Coherence Tomography Image Classification","volume":"38","author":"Fang","year":"2019","journal-title":"IEEE Trans. Med Imaging"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10921-015-0315-7","article-title":"GDXray: The Database of X-ray Images for Nondestructive Testing","volume":"34","author":"Mery","year":"2015","journal-title":"J. Nondestruct. Eval."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1024","DOI":"10.1109\/TMI.2017.2780115","article-title":"Macular OCT Classification using a Multi-Scale Convolutional Neural Network Ensemble","volume":"37","author":"Rasti","year":"2018","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hassan, T., Akram, M.U., Masood, M.F., and Yasin, U. (2018, January 27\u201329). BIOMISA Retinal Image Database for Macular and Ocular Syndromes. Proceedings of the International Conference on Image Analysis and Recognition, P\u00f3voa de Varzim, Portugal.","DOI":"10.1007\/978-3-319-93000-8_79"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1016\/j.ophtha.2013.07.013","article-title":"Quantitative Classification of Eyes with and without Intermediate Age-related Macular Degeneration Using Optical Coherence Tomography","volume":"121","author":"Farsiu","year":"2014","journal-title":"Ophthalmology"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1172","DOI":"10.1364\/BOE.6.001172","article-title":"Kernel regression based segmentation of optical coherence tomography images with diabetic macular edema","volume":"6","author":"Chiu","year":"2015","journal-title":"Biomed. Opt. Express"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"3568","DOI":"10.1364\/BOE.5.003568","article-title":"Fully automated detection of diabetic macular edema and dry age-related macular degeneration from optical coherence tomography images","volume":"5","author":"Srinivasan","year":"2014","journal-title":"Biomed. Opt. Express"},{"key":"ref_36","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Wey, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_38","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/4\/1667\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:23:48Z","timestamp":1760135028000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/4\/1667"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,21]]},"references-count":38,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["s22041667"],"URL":"https:\/\/doi.org\/10.3390\/s22041667","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,2,21]]}}}