{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T14:21:26Z","timestamp":1783520486443,"version":"3.55.0"},"reference-count":43,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,8,29]],"date-time":"2024-08-29T00:00:00Z","timestamp":1724889600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Big Data"],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Recently, Google introduced Pathways as its next-generation AI architecture. Pathways must address three critical challenges: learning one general model for several continuous tasks, ensuring tasks can leverage each other without forgetting old tasks, and learning from multi-modal data such as images and audio. Additionally, Pathways must maintain sparsity in both learning and deployment. Current lifelong multi-task learning approaches are inadequate in addressing these challenges.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>To address these challenges, we propose SEN, a Sparse and Expandable Network. SEN is designed to handle multiple tasks concurrently by maintaining sparsity and enabling expansion when new tasks are introduced. The network leverages multi-modal data, integrating information from different sources while preventing interference between tasks.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The proposed SEN model demonstrates significant improvements in multi-task learning, successfully managing task interference and forgetting. It effectively integrates data from various modalities and maintains efficiency through sparsity during both the learning and deployment phases.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>SEN offers a straightforward yet effective solution to the limitations of current lifelong multi-task learning methods. By addressing the challenges identified in the Pathways architecture, SEN provides a promising approach for developing AI systems capable of learning and adapting over time without sacrificing performance or efficiency.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fdata.2024.1348030","type":"journal-article","created":{"date-parts":[[2024,8,29]],"date-time":"2024-08-29T04:30:42Z","timestamp":1724905842000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Sparse and Expandable Network for Google's Pathways"],"prefix":"10.3389","volume":"7","author":[{"given":"Charles X.","family":"Ling","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ganyu","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Boyu","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,8,29]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.12533","article-title":"Pathways: asynchronous distributed dataflow for ML","author":"Barham","year":"2022","journal-title":"arXiv"},{"key":"B2","first-page":"221","article-title":"\u201cTowards a unified lifelong learning framework,\u201d","author":"Bohn","year":"2021","journal-title":"NeurIPS 2020 Workshop on Pre-registration in Machine Learning, volume 148 of Proceedings of Machine Learning Research"},{"key":"B3","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1023\/A:1007379606734","article-title":"Multitask learning","volume":"28","author":"Caruana","year":"1997","journal-title":"Mach. Learn"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1801.10112","article-title":"Riemannian walk for incremental learning: understanding forgetting and intransigence","author":"Chaudhry","year":"2018","journal-title":"arXiv"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1902.10486","article-title":"Continual learning with tiny episodic memories","author":"Chaudhry","year":"2019","journal-title":"arXiv"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.10.48550\/arXiv.1702.05373","article-title":"Emnist: an extension of mnist to handwritten letters","author":"Cohen","year":"2017","journal-title":"arXiv"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.08383","article-title":"Continual learning: a comparative study on how to defy forgetting in classification tasks","author":"De Lange","year":"2019","journal-title":"arXiv"},{"key":"B8","author":"Dean","year":"2023","journal-title":"Introducing Pathways: A next-generation AI architecture"},{"key":"B9","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"\u201cImagenet: a large-scale hierarchical image database,\u201d","volume-title":"2009 IEEE conference on computer vision and pattern recognition","author":"Deng","year":"2009"},{"key":"B10","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1109\/MSP.2012.2211477","article-title":"The mnist database of handwritten digit images for machine learning research","volume":"29","author":"Deng","year":"2012","journal-title":"IEEE Signal Process. Mag"},{"key":"B11","first-page":"4171","article-title":"\u201cBERT: pre-training of deep bidirectional transformers for language understanding,\u201d","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"B12","doi-asserted-by":"publisher","first-page":"594","DOI":"10.1109\/TPAMI.2006.79","article-title":"One-shot learning of object categories","volume":"28","author":"Fei-Fei","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B13","first-page":"1","article-title":"\u201cImage and text fusion for upmc food-101 using BERT and CNNs,\u201d","volume-title":"2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ)","author":"Gallo","year":"2020"},{"key":"B14","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.11595","article-title":"\u201cSelective experience replay for lifelong learning,\u201d","volume-title":"Thirty-second AAAI conference on artificial intelligence","author":"Isele","year":"2018"},{"key":"B15","unstructured":"JacksonZ.\n          Spoken Digit Dataset2016"},{"key":"B16","first-page":"10734","article-title":"\u201cForget-free continual learning with winning subnetworks,\u201d","volume-title":"International Conference on Machine Learning","author":"Kang","year":"2022"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.48550\/arxiv.1612.00796","article-title":"Overcoming catastrophic forgetting in neural networks","author":"Kirkpatrick","year":"2016","journal-title":"arXiv"},{"key":"B18","doi-asserted-by":"publisher","first-page":"2935","DOI":"10.1109\/TPAMI.2017.2773081","article-title":"Learning without forgetting","volume":"40","author":"Li","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B19","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"Litjens","year":"2017","journal-title":"Med. Image Anal"},{"key":"B20","first-page":"7765","article-title":"\u201cPacknet: adding multiple tasks to a single network by iterative pruning,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Mallya","year":"2018"},{"key":"B21","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1016\/S0079-7421(08)60536-8","article-title":"Catastrophic interference in connectionist networks: the sequential learning problem","volume":"24","author":"McCloskey","year":"1989","journal-title":"Psychol. Learn. Motiv"},{"key":"B22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/IVCNZ48456.2019.8960960","article-title":"\u201cAre these birds similar: Learning branched networks for fine-grained representations,\u201d","volume-title":"2019 International Conference on Image and Vision Computing New Zealand (IVCNZ)","author":"Nawaz","year":"2019"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2009","journal-title":"IEEE Trans. Knowl. Data Eng"},{"key":"B24","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.neunet.2019.01.012","article-title":"Continual lifelong learning with neural networks: a review","volume":"113","author":"Parisi","year":"2019","journal-title":"Neural Netw"},{"key":"B25","first-page":"28414","article-title":"Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions","volume":"35","author":"Peng","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B26","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"B27","doi-asserted-by":"publisher","first-page":"5533","DOI":"10.1109\/CVPR.2017.587","author":"Rebuffi","year":"2017"},{"key":"B28","first-page":"3738","article-title":"\u201cOnline structured laplace approximations for overcoming catastrophic forgetting,\u201d","author":"Ritter","year":"2018","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.48550\/arxiv.1606.04671","article-title":"Progressive neural networks","author":"Rusu","year":"2016","journal-title":"arXiv"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1801.01423","article-title":"Overcoming catastrophic forgetting with hard attention to the task","author":"Serr\u00e0","year":"2018","journal-title":"arXiv"},{"key":"B31","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1146\/annurev-bioeng-071516-044442","article-title":"Deep learning in medical image analysis","volume":"19","author":"Shen","year":"2017","journal-title":"Annu. Rev. Biomed. Eng"},{"key":"B32","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"B33","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1007\/978-1-4615-5529-2_8","article-title":"\u201cLifelong learning algorithms,\u201d","volume-title":"Learning to Learn","author":"Thrun","year":"1998"},{"key":"B34","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1016\/B978-044482250-5\/50015-3","article-title":"\u201cA lifelong learning perspective for mobile robot control,\u201d","volume-title":"Intelligent Robots and Systems","year":"1995"},{"key":"B35","first-page":"20366","article-title":"SPARCL: sparse continual learning on the edge","volume":"35","author":"Wang","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B36","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2019.00046","article-title":"\u201cLarge scale incremental learning,\u201d","volume-title":"2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Wu","year":"2019"},{"key":"B37","unstructured":"XiaoH.\n            RasulK.\n            VollgrafR.\n          Fashion-MNIST2017"},{"key":"B38","first-page":"899","article-title":"\u201cReinforced continual learning,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Xu","year":"2018"},{"key":"B39","first-page":"5630","article-title":"\u201cData augmented flatness-aware gradient projection for continual learning,\u201d","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Yang","year":"2023"},{"key":"B40","unstructured":"\u201cLifelong learning with dynamically expandable networks,\u201d\n            YoonJ.\n            YangE.\n            LeeJ.\n            HwangS. J.\n          Vancouver, BCICLR (Poster)2018"},{"key":"B41","first-page":"3987","article-title":"\u201cContinual learning through synaptic intelligence,\u201d","volume-title":"ICML, volume 70 of Proceedings of Machine Learning Research","author":"Zenke","year":"2017"},{"key":"B42","first-page":"1131","article-title":"\u201cClass-incremental learning via deep model consolidation,\u201d","volume-title":"IEEE Winter Conference on Applications of Computer Vision","author":"Zhang","year":"2020"},{"key":"B43","first-page":"7103","article-title":"Mixture-of-experts with expert choice routing","volume":"35","author":"Zhou","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"}],"container-title":["Frontiers in Big Data"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2024.1348030\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,29]],"date-time":"2024-08-29T04:31:06Z","timestamp":1724905866000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2024.1348030\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,29]]},"references-count":43,"alternative-id":["10.3389\/fdata.2024.1348030"],"URL":"https:\/\/doi.org\/10.3389\/fdata.2024.1348030","relation":{},"ISSN":["2624-909X"],"issn-type":[{"value":"2624-909X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,29]]},"article-number":"1348030"}}