{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T15:27:43Z","timestamp":1787239663616,"version":"build-2736575974"},"reference-count":68,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T00:00:00Z","timestamp":1613692800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"},{"start":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T00:00:00Z","timestamp":1613692800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2021,3]]},"DOI":"10.1007\/s10664-020-09894-9","type":"journal-article","created":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T18:58:10Z","timestamp":1613761090000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Automated end-to-end management of the modeling lifecycle in deep learning"],"prefix":"10.1007","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0062-4748","authenticated-orcid":false,"given":"Gharib","family":"Gharibi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vijay","family":"Walunj","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Raju","family":"Nekadi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Raj","family":"Marri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yugyung","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,2,19]]},"reference":[{"key":"9894_CR1","unstructured":"Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Man\u00e9 D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Vi\u00e9gas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-scale machine learning on heterogeneous systems. https:\/\/www.tensorflow.org\/, Software available from tensorflow.org"},{"key":"9894_CR2","unstructured":"Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. In: OSDI, vol 16, pp 265\u2013283"},{"key":"9894_CR3","unstructured":"Albishri AA, Shah S JH, Schmiedler A, Kang SS, Lee Y (2019) Automated human claustrum segmentation using deep learning technologies. arXiv:1911.07515"},{"key":"9894_CR4","doi-asserted-by":"crossref","unstructured":"Bergstra J, Breuleux O, Bastien F, Lamblin P, Pascanu R, Desjardins G, Turian J, Warde-Farley D, Bengio Y (2010) Theano: A cpu and gpu math compiler in python. In: Proc. 9th Python in Science Conf, vol 1, pp 3\u201310","DOI":"10.25080\/Majora-92bf1922-003"},{"issue":"7623","key":"9894_CR5","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/538020a","volume":"538","author":"D Castelvecchi","year":"2016","unstructured":"Castelvecchi D (2016) Can we open the black box of ai?. Nat 538 (7623):20","journal-title":"Nat"},{"key":"9894_CR6","unstructured":"Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P (2016) Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In: Advances in neural information processing systems, pp 2172\u20132180"},{"key":"9894_CR7","unstructured":"Chollet F et al (2015) Keras. https:\/\/keras.io"},{"key":"9894_CR8","unstructured":"DeepCognition (2019) One stop for deep learning developers. https:\/\/deepcognition.ai\/"},{"key":"9894_CR9","unstructured":"Documentation P (2019) Abstract syntax trees. https:\/\/docs.python.org\/3\/library\/ast.html"},{"key":"9894_CR10","unstructured":"Facebook (2019) Introducing fblearner flow: Facebook\u2019s ai backbone. https:\/\/conferences.oreilly.com\/strata\/big-data-conference-ny-2015\/public\/schedule\/detail\/42988"},{"key":"9894_CR11","unstructured":"Garcia R, Sreekanti V, Yadwadkar N, Crankshaw D, Gonzalez JE, Hellerstein JM (2018) Context: The missing piece in the machine learning lifecycle. In: KDD CMI Workshop, vol 114"},{"key":"9894_CR12","doi-asserted-by":"crossref","unstructured":"Gharibi G, Walunj V, Alanazi R, Rella S, Lee Y (2019a) Automated management of deep learning experiments. In: Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning, pp 1\u20134","DOI":"10.1145\/3329486.3329495"},{"key":"9894_CR13","doi-asserted-by":"crossref","unstructured":"Gharibi G, Walunj V, Rella S, Lee Y (2019b) Modelkb: towards automated management of the modeling lifecycle in deep learning. In: 2019 IEEE\/ACM 7th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). IEEE, pp 28\u201334","DOI":"10.1109\/RAISE.2019.00013"},{"key":"9894_CR14","unstructured":"Ghezzi C, Jazayeri M, Mandrioli D (2002) Fundamentals of software engineering. Prentice Hall PTR"},{"key":"9894_CR15","unstructured":"Goodfellow IJ, Shlens J, Szegedy C (2014) Explaining and harnessing adversarial examples. arXiv:1412.6572"},{"key":"9894_CR16","unstructured":"Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press. http:\/\/www.deeplearningbook.org"},{"key":"9894_CR17","doi-asserted-by":"crossref","unstructured":"Goodfellow I, McDaniel P, Papernot N (2018) Making machine learning robust against adversarial inputs. Commun ACM 61(7)","DOI":"10.1145\/3134599"},{"key":"9894_CR18","unstructured":"Google (2019) Tensorboard: Visualizing learning. https:\/\/www.tensorflow.org\/guide\/summaries_and_tensorbard"},{"key":"9894_CR19","unstructured":"Goudarzvand S, Gharibi G, Lee Y (2020) Scat: Second chance autoencoder for textual data. arXiv:2005.06632"},{"key":"9894_CR20","unstructured":"Grinberg M (2018) Flask web development: developing web applications with python. O\u2019Reilly Media, Inc."},{"key":"9894_CR21","unstructured":"Hall MA (1999) Correlation-based feature selection for machine learning"},{"key":"9894_CR22","unstructured":"Hannun A, Case C, Casper J, Catanzaro B, Diamos G, Elsen E, Prenger R, Satheesh S, Sengupta S, Coates A et al (2014) Deep speech: Scaling up end-to-end speech recognition. arXiv:1412.5567"},{"key":"9894_CR23","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"9894_CR24","unstructured":"Hellerstein JM, Sreekanti V, Gonzalez JE, Dalton J, Dey A, Nag S, Ramachandran K, Arora S, Bhattacharyya A, Das S et al (2017) Ground: A data context service. In: CIDR"},{"issue":"1","key":"9894_CR25","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1023\/B:JCNS.0000023869.22017.2e","volume":"17","author":"ML Hines","year":"2004","unstructured":"Hines ML, Morse T, Migliore M, Carnevale NT, Shepherd GM (2004) Modeldb: a database to support computational neuroscience. J Comput Neurosci 17(1):7\u201311","journal-title":"J Comput Neurosci"},{"key":"9894_CR26","doi-asserted-by":"crossref","unstructured":"Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on Multimedia. ACM, pp 675\u2013678","DOI":"10.1145\/2647868.2654889"},{"key":"9894_CR27","unstructured":"Jinja (2019) Python template language. https:\/\/jinja.palletsprojects.com\/en\/2.11.x\/"},{"key":"9894_CR28","doi-asserted-by":"crossref","unstructured":"Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp 1725\u20131732","DOI":"10.1109\/CVPR.2014.223"},{"key":"9894_CR29","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097\u20131105"},{"issue":"4","key":"9894_CR30","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1145\/2935694.2935698","volume":"44","author":"A Kumar","year":"2016","unstructured":"Kumar A, McCann R, Naughton J, Patel JM (2016) Model selection management systems: The next frontier of advanced analytics. ACM SIGMOD Record 44(4):17\u201322","journal-title":"ACM SIGMOD Record"},{"key":"9894_CR31","doi-asserted-by":"crossref","unstructured":"Kumar A, Boehm M, Yang J (2017) Data management in machine learning: Challenges, techniques, and systems. In: Proceedings of the 2017 ACM International Conference on Management of Data. ACM, pp 1717\u20131722","DOI":"10.1145\/3035918.3054775"},{"issue":"1","key":"9894_CR32","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1109\/72.554195","volume":"8","author":"S Lawrence","year":"1997","unstructured":"Lawrence S, Giles CL, Tsoi AC, Back AD (1997) Face recognition: A convolutional neural-network approach. IEEE Trans Neural Netw 8 (1):98\u2013113","journal-title":"IEEE Trans Neural Netw"},{"key":"9894_CR33","unstructured":"LeCun Y, Boser BE, Denker JS, Henderson D, Howard RE, Hubbard WE, Jackel LD (1990) Handwritten digit recognition with a back-propagation network. In: Advances in neural information processing systems, pp 396\u2013404"},{"issue":"11","key":"9894_CR34","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y LeCun","year":"1998","unstructured":"LeCun Y, Bottou L, Bengio Y, Haffner P et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278\u20132324","journal-title":"Proc IEEE"},{"issue":"7553","key":"9894_CR35","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436","journal-title":"Nature"},{"key":"9894_CR36","doi-asserted-by":"crossref","unstructured":"Miao H, Li A, Davis LS, Deshpande A (2017a) Modelhub: Deep learning lifecycle management. In: 2017 IEEE 33rd International Conference on Data Engineering (ICDE). IEEE, pp 1393\u20131394","DOI":"10.1109\/ICDE.2017.192"},{"key":"9894_CR37","doi-asserted-by":"crossref","unstructured":"Miao H, Li A, Davis LS, Deshpande A (2017b) Towards unified data and lifecycle management for deep learning. In: 2017 IEEE 33rd International Conference on Data Engineering (ICDE). IEEE, pp 571\u2013582","DOI":"10.1109\/ICDE.2017.112"},{"issue":"4","key":"9894_CR38","first-page":"26","volume":"41","author":"H Miao","year":"2018","unstructured":"Miao H, Deshpande A (2018) Provdb: Provenance-enabled lifecycle management of collaborative data analysis workflows. IEEE Data Eng Bull 41(4):26\u201338","journal-title":"IEEE Data Eng Bull"},{"key":"9894_CR39","unstructured":"Microsoft (2017) Machine learning studio. https:\/\/azure.microsoft.com\/en-us\/services\/machine-learning-studio\/https:\/\/azure.microsoft.com\/en-us\/services\/machine-learning-studio\/"},{"key":"9894_CR40","unstructured":"ModelHubAI (2019) A collection of deep learning models managed by the computational imaging and bioinformatics lab at the harvard medical school, brigham & women\u2019s hospital, and dana-farber cancer institute. http:\/\/modelhub.ai\/"},{"key":"9894_CR41","unstructured":"ModelZoo (2019) A set of pretrained models models hosted on github. https:\/\/github.com\/BVLC\/caffe\/wiki\/Model-Zoo"},{"key":"9894_CR42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.dsp.2017.10.011","volume":"73","author":"G Montavon","year":"2018","unstructured":"Montavon G, Samek W, M\u00fcller K-R (2018) Methods for interpreting and understanding deep neural networks. Digital Signal Process 73:1\u201315","journal-title":"Digital Signal Process"},{"key":"9894_CR43","unstructured":"Nvidia (2019) Digits: A graphical web interface for nvcaffe and tensorflow. https:\/\/docs.nvidia.com\/deeplearning\/digits\/digits-user-guide\/index.html"},{"key":"9894_CR44","unstructured":"Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop"},{"key":"9894_CR45","unstructured":"PyTorch (2019) A set of pretrained pytorch models. https:\/\/pytorch.org\/docs\/stable\/torchvision\/models.html"},{"key":"9894_CR46","unstructured":"Roeder L (2019) Netron: Visualizing deep learning models. https:\/\/github.com\/lutzroeder\/netron"},{"issue":"3","key":"9894_CR47","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet Large Scale Visual Recognition Challenge. Int J Comput Vis (IJCV) 115 (3):211\u2013252. https:\/\/doi.org\/10.1007\/s11263-015-0816-y","journal-title":"Int J Comput Vis (IJCV)"},{"key":"9894_CR48","unstructured":"SageMaker (2018) Sagemaker. https:\/\/aws.amazon.com\/sagemaker\/\/"},{"key":"9894_CR49","unstructured":"Schelter S, B\u00f6se J-H, Kirschnick J, Klein T, Seufert S (2017) Automatically tracking metadata and provenance of machine learning experiments. In: Machine Learning Systems Workshop at NIPS"},{"issue":"4","key":"9894_CR50","first-page":"5","volume":"41","author":"S Schelter","year":"2018","unstructured":"Schelter S, Biessmann F, Januschowski T, Salinas D, Seufert S, Szarvas G, Vartak M, Madden S, Miao H, Deshpande A et al (2018a) On challenges in machine learning model management. IEEE Data Eng Bull 41(4):5\u201315","journal-title":"IEEE Data Eng Bull"},{"key":"9894_CR51","unstructured":"Schelter S, B\u00f6se J-H, Kirschnick J, Klein T, Seufert S (2018b) Declarative metadata management: A missing piece in end-to-end machine learning"},{"key":"9894_CR52","unstructured":"Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Chaudhary V, Young M, Crespo J-F, Dennison D (2015) Hidden technical debt in machine learning systems. In: Advances in neural information processing systems, pp 2503\u20132511"},{"key":"9894_CR53","unstructured":"Seedbank G (2019) A set of models shared via google colab. https:\/\/research.google.com\/seedbank\/"},{"key":"9894_CR54","doi-asserted-by":"crossref","unstructured":"Seide F, Agarwal A (2016) Cntk: Microsoft\u2019s open-source deep-learning toolkit. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 2135\u20132135","DOI":"10.1145\/2939672.2945397"},{"issue":"7587","key":"9894_CR55","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","volume":"529","author":"D Silver","year":"2016","unstructured":"Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van DenDriessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484","journal-title":"Nature"},{"key":"9894_CR56","unstructured":"Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556"},{"key":"9894_CR57","unstructured":"SQL (2019) A c-language library to run sql engine. https:\/\/www.sqlite.org\/index.html"},{"key":"9894_CR58","doi-asserted-by":"crossref","unstructured":"Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: AAAI, vol 4, pp 12","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"9894_CR59","unstructured":"Tantithamthavorn C, Hassan AE, Matsumoto K (2018) The impact of class rebalancing techniques on the performance and interpretation of defect prediction models. IEEE Trans Softw Eng"},{"key":"9894_CR60","unstructured":"Uber (2019) Imeet michelangelo: Uber\u2019s machine learning platform. https:\/\/eng.uber.com\/michelangelo\/"},{"key":"9894_CR61","unstructured":"VanRijn JN, Bischl B, Torgo L, Gao B, Umaashankar V, Fischer S, Winter P, Wiswedel B, Berthold MR, Vanschoren J (2013) Openml: A collaborative science platform. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, pp 645\u2013649"},{"key":"9894_CR62","doi-asserted-by":"crossref","unstructured":"Vartak M, Subramanyam H, Lee W-E, Viswanathan S, Husnoo S, Madden S, Zaharia M (2016) M odel db: a system for machine learning model management. In: Proceedings of the Workshop on Human-In-the-Loop Data Analytics. ACM, pp 14","DOI":"10.1145\/2939502.2939516"},{"key":"9894_CR63","unstructured":"Vartak M (2018a) Infrastructure for model management and model diagnosis. Ph.D. Thesis, Massachusetts Institute of Technology"},{"issue":"4","key":"9894_CR64","first-page":"16","volume":"41","author":"M Vartak","year":"2018","unstructured":"Vartak M, Madden S (2018b) Modeldb: Opportunities and challenges in managing machine learning models. IEEE Data Eng Bull 41(4):16\u201325","journal-title":"IEEE Data Eng Bull"},{"key":"9894_CR65","doi-asserted-by":"crossref","unstructured":"Velazquez M, Anantharaman R, Velazquez S, Lee Y (2019) Rnn-based alzheimer\u2019s disease prediction from prodromal stage using diffusion tensor imaging. In: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, pp 1665\u20131672","DOI":"10.1109\/BIBM47256.2019.8983391"},{"key":"9894_CR66","unstructured":"Yu X, Sohn K, Chandraker M (2018) Video security system using a siamese reconstruction convolutional neural network for pose-invariant face recognition. US Patent App. 15\/803,318"},{"key":"9894_CR67","unstructured":"Zaharia M, Chen A, Davidson A, Ghodsi A, Hong SA, Konwinski A, Murching S, Nykodym T, Ogilvie P, Parkhe M et al (2018) Accelerating the machine learning lifecycle with mlflow. Data Engineering:39"},{"key":"9894_CR68","unstructured":"Zhang A, Lipton ZC, Li M, Smola AssJ (2019) Dive into deep learning. http:\/\/www.d2l.ai"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09894-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10664-020-09894-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09894-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T11:05:15Z","timestamp":1697886315000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10664-020-09894-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,19]]},"references-count":68,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,3]]}},"alternative-id":["9894"],"URL":"https:\/\/doi.org\/10.1007\/s10664-020-09894-9","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,19]]},"assertion":[{"value":"27 November 2020","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 February 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"17"}}