{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,5]],"date-time":"2026-09-05T15:46:08Z","timestamp":1788623168086,"version":"build-2803163510"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T00:00:00Z","timestamp":1635724800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,11]],"date-time":"2021-11-11T00:00:00Z","timestamp":1636588800000},"content-version":"vor","delay-in-days":10,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100006764","name":"Technische Universit\u00e4t Berlin","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100006764","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Datenbank Spektrum"],"published-print":{"date-parts":[[2021,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Deep Learning (DL) has consistently surpassed other Machine Learning methods and achieved state-of-the-art performance in multiple cases. Several modern applications like financial and recommender systems require models that are constantly updated with fresh data. The prominent approach for keeping a DL model fresh is to trigger full retraining from scratch when enough new data are available. However, retraining large and complex DL models is time-consuming and compute-intensive. This makes full retraining costly, wasteful, and slow. In this paper, we present an approach to continuously train and deploy DL models. First, we enable continuous training through proactive training that combines samples of historical data with new streaming data. Second, we enable continuous deployment through gradient sparsification that allows us to send a small percentage of the model updates per training iteration. Our experimental results with LeNet5 on MNIST and modern DL models on CIFAR-10 show that proactive training keeps models fresh with comparable\u2014if not superior\u2014performance to full retraining at a fraction of the time. Combined with gradient sparsification, sparse proactive training enables very fast updates of a deployed model with arbitrarily large sparsity, reducing communication per iteration up to four orders of magnitude, with minimal\u2014if any\u2014losses in model quality. Sparse training, however, comes at a price; it incurs overhead on the training that depends on the size of the model and increases the training time by factors ranging from 1.25 to 3 in our experiments. Arguably, a small price to pay for successfully enabling the continuous training and deployment of large DL models.<\/jats:p>","DOI":"10.1007\/s13222-021-00386-8","type":"journal-article","created":{"date-parts":[[2021,11,11]],"date-time":"2021-11-11T12:03:32Z","timestamp":1636632212000},"page":"203-212","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":49,"title":["Continuous Training and Deployment of Deep Learning Models"],"prefix":"10.1007","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9111-4112","authenticated-orcid":false,"given":"Ioannis","family":"Prapas","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Behrouz","family":"Derakhshan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alireza Rezaei","family":"Mahdiraji","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Volker","family":"Markl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,11,11]]},"reference":[{"issue":"6","key":"386_CR1","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MSP.2012.2205597","volume":"29","author":"G Hinton","year":"2012","unstructured":"Hinton G, Deng L, Yu D, Dahl GE, Mohamed A, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) IEEE Signal Process Mag 29(6):82","journal-title":"IEEE Signal Process Mag"},{"key":"386_CR2","series-title":"arXiv:1608.06993 1608","volume-title":"arXiv preprint","author":"G Huang","year":"2018","unstructured":"Huang G, Liu Z, Weinberger KQ, van der Maaten L (2018) arXiv preprint. arXiv:1608.06993 1608"},{"key":"386_CR3","series-title":"arXiv:2005.14165","volume-title":"arXiv preprint","author":"TB Brown","year":"2020","unstructured":"Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A et al (2020) arXiv preprint. arXiv:2005.14165"},{"key":"386_CR4","series-title":"preprint arXiv:2003.07631","volume-title":"arXiv","author":"W Samek","year":"2020","unstructured":"Samek W, Montavon G, Lapuschkin S, Anders CJ, M\u00fcller KR (2020) arXiv. preprint arXiv:2003.07631"},{"key":"386_CR5","volume-title":"2007 NIPS workshop on Efficient Learning","author":"Y LeCun","year":"2007","unstructured":"LeCun Y (2007) In:  2007 NIPS workshop on Efficient Learning Vancouver, December. vol 7 (Citeseer)"},{"key":"386_CR6","unstructured":"Amodei D, Hernandez D (2018) https:\/\/blog.openai.com\/ai-and-compute. Accessed: 1 Jun 2021"},{"key":"386_CR7","series-title":"arXiv:1910.07867","volume-title":"arXiv preprint","author":"S Zeuch","year":"2019","unstructured":"Zeuch S, Chaudhary A, Del Monte B, Gavriilidis H, Giouroukis D, Grulich PM, Bre\u00df S, Traub J, Markl V (2019) arXiv preprint. arXiv:1910.07867"},{"key":"386_CR8","first-page":"397","volume-title":"EDBT","author":"B Derakhshan","year":"2019","unstructured":"Derakhshan B, Mahdiraji AR, Rabl T, Markl V (2019) In:  EDBT, pp 397\u2013408"},{"key":"386_CR9","series-title":"arXiv:1910.08475","volume-title":"arXiv preprint","author":"JT Ash","year":"2019","unstructured":"Ash JT, Adams RP (2019) arXiv preprint. arXiv:1910.08475"},{"key":"386_CR10","series-title":"arXiv:1712.01887","volume-title":"arXiv preprint","author":"Y Lin","year":"2017","unstructured":"Lin Y, Han S, Mao H, Wang Y, Dally WJ (2017) arXiv preprint. arXiv:1712.01887"},{"key":"386_CR11","first-page":"4447","volume-title":"Advances in neural information processing systems","author":"SU Stich","year":"2018","unstructured":"Stich SU, Cordonnier JB, Jaggi M (2018) In:  Advances in neural information processing systems, pp 4447\u20134458"},{"key":"386_CR12","series-title":"arXiv:1704.05021","volume-title":"arXiv preprint","author":"AF Aji","year":"2017","unstructured":"Aji AF, Heafield K (2017) arXiv preprint. arXiv:1704.05021"},{"key":"386_CR13","first-page":"177","volume-title":"Proceedings of COMPSTAT\u20192010","author":"L Bottou","year":"2010","unstructured":"Bottou L (2010) In:  Proceedings of COMPSTAT\u20192010. Springer, Berlin, Heidelberg, pp 177\u2013186"},{"key":"386_CR14","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1007\/978-3-642-35289-8_25","volume-title":"Neural networks: tricks of the trade","author":"L Bottou","year":"2012","unstructured":"Bottou L (2012) In:  Neural networks: tricks of the trade. Springer, Berlin, Heidelberg, pp 421\u2013436"},{"key":"386_CR15","first-page":"228","volume-title":"Conference on Probabilistic Graphical Models","author":"P Jaini","year":"2016","unstructured":"Jaini P, Rashwan A, Zhao H, Liu Y, Banijamali E, Chen Z, Poupart P (2016) In:  Conference on Probabilistic Graphical Models, pp 228\u2013239"},{"key":"386_CR16","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1145\/2502081.2502112","volume-title":"Proceedings of the 21st ACM international conference on Multimedia","author":"P Wu","year":"2013","unstructured":"Wu P, Hoi SC, Xia H, Zhao P, Wang D, Miao C (2013) In:  Proceedings of the 21st ACM international conference on Multimedia, pp 153\u2013162"},{"issue":"4","key":"386_CR17","doi-asserted-by":"publisher","first-page":"2923","DOI":"10.1109\/COMST.2018.2844341","volume":"20","author":"M Mohammadi","year":"2018","unstructured":"Mohammadi M, Al-Fuqaha A, Sorour S, Guizani M (2018) IEEE Commun Surv Tutor 20(4):2923","journal-title":"IEEE Commun Surv Tutor"},{"key":"386_CR18","first-page":"51","volume-title":"2019 USENIX Conference on Operational Machine Learning","author":"D Baylor","year":"2019","unstructured":"Baylor D, Haas K, Katsiapis K, Leong S, Liu R, Menwald C, Miao H, Polyzotis N, Trott M, Zinkevich M (2019) In:  2019 USENIX Conference on Operational Machine Learning OpML 19, pp 51\u201353"},{"key":"386_CR19","doi-asserted-by":"publisher","first-page":"1387","DOI":"10.1145\/3097983.3098021","volume-title":"Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"D Baylor","year":"2017","unstructured":"Baylor D, Breck E, Cheng HT, Fiedel N, Foo CY, Haque Z, Haykal S, Ispir M, Jain V, Koc L et al (2017) In:  Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1387\u20131395"},{"key":"386_CR20","doi-asserted-by":"publisher","first-page":"3182","DOI":"10.1145\/3292500.3340408","volume-title":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","author":"K Katsiapis","year":"2019","unstructured":"Katsiapis K, Haas K (2019) In:  Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 3182\u20133182"},{"key":"386_CR21","volume-title":"Continual learning with deep architectures","author":"V Lomonaco","year":"2019","unstructured":"Lomonaco V (2019) Continual learning with deep architectures (Ph.D. thesis, alma)"},{"key":"386_CR22","series-title":"arXiv:1907.03799","volume-title":"arXiv preprint","author":"V Lomonaco","year":"2019","unstructured":"Lomonaco V, Maltoni D, Pellegrini L (2019) arXiv preprint. arXiv:1907.03799, vol 1"},{"key":"386_CR23","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1007\/978-3-030-43883-8_8","volume-title":"Recent trends in learning from data","author":"GI Parisi","year":"2020","unstructured":"Parisi GI, Lomonaco V (2020) In:  Recent trends in learning from data. Springer, Cham, pp 197\u2013221"},{"key":"386_CR24","volume-title":"Neurocomputing","author":"J Pomponi","year":"2020","unstructured":"Pomponi J, Scardapane S, Lomonaco V, Uncini A (2020) Neurocomputing"},{"key":"386_CR25","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1016\/S0079-7421(08)60536-8","volume-title":"Psychology of learning and motivation","author":"M McCloskey","year":"1989","unstructured":"McCloskey M, Cohen NJ (1989) In:  Psychology of learning and motivation, vol 24. Elsevier, Academic Press, pp 109\u2013165. https:\/\/doi.org\/10.1016\/S0079-7421(08)60536-8"},{"key":"386_CR26","first-page":"693","volume-title":"Advances in neural information processing systems","author":"B Recht","year":"2011","unstructured":"Recht B, Re C, Wright S, Niu F (2011) In:  Advances in neural information processing systems, pp 693\u2013701"},{"key":"386_CR27","first-page":"1709","volume-title":"Advances in neural information processing systems","author":"D Alistarh","year":"2017","unstructured":"Alistarh D, Grubic D, Li J, Tomioka R, Vojnovic M (2017) In:  Advances in neural information processing systems, pp 1709\u20131720"},{"key":"386_CR28","series-title":"arXiv:1902.00340","volume-title":"arXiv preprint","author":"A Koloskova","year":"2019","unstructured":"Koloskova A, Stich SU, Jaggi M (2019) arXiv preprint. arXiv:1902.00340"},{"key":"386_CR29","unstructured":"Wikipedia contributors (2021) Independent and identically distributed random variables. https:\/\/en.wikipedia.org\/w\/index.php?title=Independent_and_identically_distributed_random_variables&oldid=1017206855. Accessed 9 Aug 2021"},{"issue":"2","key":"386_CR30","doi-asserted-by":"publisher","first-page":"278","DOI":"10.2307\/2981683","volume":"147","author":"AP Dawid","year":"1984","unstructured":"Dawid AP (1984) J\u00a0Royal Stat Soc Ser\u00a0A 147(2):278","journal-title":"J Royal Stat Soc Ser A"},{"key":"386_CR31","first-page":"8026","volume-title":"Advances in neural information processing systems","author":"A Paszke","year":"2019","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L et al (2019) In:  Advances in neural information processing systems, pp 8026\u20138037"},{"key":"386_CR32","doi-asserted-by":"publisher","first-page":"1485","DOI":"10.1145\/1873951.1874254","volume-title":"Proceedings of the 18th ACM international conference on Multimedia","author":"S Marcel","year":"2010","unstructured":"Marcel S, Rodriguez Y (2010) In:  Proceedings of the 18th ACM international conference on Multimedia, pp 1485\u20131488"},{"key":"386_CR33","unstructured":"LeCun Y (1998) http:\/\/yann.lecun.com\/exdb\/mnist\/. Accessed: 1 Jun 2021"},{"key":"386_CR34","volume-title":"MSc Thesis, University of Toronto","author":"A Krizhevsky","year":"2009","unstructured":"Krizhevsky A (2009) MSc Thesis, University of Toronto"},{"key":"386_CR35","unstructured":"LeCun Y et al (2015) 20(5), 14. http:\/\/yann.lecun.com\/exdb\/lenet. Accessed 1: Jun 2021"},{"key":"386_CR36","first-page":"4510","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"M Sandler","year":"2018","unstructured":"Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) In:  Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510\u20134520"},{"key":"386_CR37","first-page":"770","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"K He","year":"2016","unstructured":"He K, Zhang X, Ren S, Sun J (2016) In:  Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770\u2013778"},{"key":"386_CR38","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","volume-title":"2009 IEEE conference on computer vision and pattern recognition","author":"J Deng","year":"2009","unstructured":"Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) In:  2009 IEEE conference on computer vision and pattern recognition Ieee, pp 248\u2013255"},{"key":"386_CR39","series-title":"arXiv:1412.6980","volume-title":"arXiv preprint","author":"DP Kingma","year":"2014","unstructured":"Kingma DP, Ba J (2014) arXiv preprint. arXiv:1412.6980"},{"key":"386_CR40","unstructured":"kuangliu (2020) Train cifar10 with pytorch. https:\/\/github.com\/kuangliu\/pytorch-cifar. Accessed 1 May 2020"}],"container-title":["Datenbank-Spektrum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13222-021-00386-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13222-021-00386-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13222-021-00386-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T08:30:35Z","timestamp":1637310635000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13222-021-00386-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11]]},"references-count":40,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,11]]}},"alternative-id":["386"],"URL":"https:\/\/doi.org\/10.1007\/s13222-021-00386-8","relation":{},"ISSN":["1618-2162","1610-1995"],"issn-type":[{"value":"1618-2162","type":"print"},{"value":"1610-1995","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11]]},"assertion":[{"value":"31 May 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 August 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}