{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T23:42:41Z","timestamp":1793058161391,"version":"4.5.0"},"reference-count":55,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,2,7]],"date-time":"2021-02-07T00:00:00Z","timestamp":1612656000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,2,7]],"date-time":"2021-02-07T00:00:00Z","timestamp":1612656000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cluster Comput"],"published-print":{"date-parts":[[2021,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Deep Learning (DL) has achieved remarkable progress over the last decade on various tasks such as image recognition, speech recognition, and natural language processing. In general, three main crucial aspects fueled this progress: the increasing availability of large amount of digitized data, the increasing availability of affordable parallel and powerful computing resources (e.g., GPU) and the growing number of open source deep learning frameworks that facilitate and ease the development process of deep learning architectures. In practice, the increasing popularity of deep learning frameworks calls for benchmarking studies that can effectively evaluate and understand the performance characteristics of these systems. In this paper, we conduct an extensive experimental evaluation and analysis of six popular deep learning frameworks, namely,<jats:italic>TensorFlow<\/jats:italic>,<jats:italic>MXNet<\/jats:italic>,<jats:italic>PyTorch<\/jats:italic>,<jats:italic>Theano<\/jats:italic>,<jats:italic>Chainer<\/jats:italic>, and<jats:italic>Keras<\/jats:italic>, using three types of DL architectures Convolutional Neural Networks (CNN), Faster Region-based Convolutional Neural Networks (Faster R-CNN), and Long Short Term Memory (LSTM). Our experimental evaluation considers different aspects for its comparison including accuracy, training time, convergence and resource consumption patterns. Our experiments have been conducted on both CPU and GPU environments using different datasets. We report and analyze the performance characteristics of the studied frameworks. In addition, we report a set of insights and important lessons that we have learned from conducting our experiments.<\/jats:p>","DOI":"10.1007\/s10586-021-03240-4","type":"journal-article","created":{"date-parts":[[2021,2,9]],"date-time":"2021-02-09T03:04:50Z","timestamp":1612839890000},"page":"2017-2038","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":67,"title":["DLBench: a comprehensive experimental evaluation of deep learning frameworks"],"prefix":"10.1007","volume":"24","author":[{"given":"Radwa","family":"Elshawi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abdul","family":"Wahab","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmed","family":"Barnawi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2503-523X","authenticated-orcid":false,"given":"Sherif","family":"Sakr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,2,7]]},"reference":[{"key":"3240_CR1","first-page":"265","volume":"16","author":"M Abadi","year":"2016","unstructured":"Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. OSDI 16, 265\u2013283 (2016)","journal-title":"OSDI"},{"issue":"3","key":"3240_CR2","doi-asserted-by":"publisher","first-page":"1781","DOI":"10.1007\/s10586-020-03127-w","volume":"23","author":"A Abugabah","year":"2020","unstructured":"Abugabah, A., AlZubi, A.A., Al-Obeidat, F.N., Alarifi, A., Alwadain, A.: Data mining techniques for analyzing healthcare conditions of urban space-person lung using meta-heuristic optimized neural networks. Clust. Comput. 23(3), 1781\u20131794 (2020)","journal-title":"Clust. Comput."},{"key":"3240_CR3","doi-asserted-by":"crossref","unstructured":"Awan, A.A., Subramoni, H., Panda, D.K.: An in-depth performance characterization of CPU-and GPU-based DNN training on modern architectures. In: Proceedings of the Machine Learning on HPC Environments, p. 8. ACM, (2017)","DOI":"10.1145\/3146347.3146356"},{"key":"3240_CR4","unstructured":"Bahrampour, S., Ramakrishnan, N., Schott, L., Shah, M.: Comparative study of caffe, neon, theano, and torch for deep learning (2016)"},{"key":"3240_CR5","unstructured":"Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., Warde-Farley, D., Bengio, Y.: Theano: new features and speed improvements. arXiv preprint arXiv:1211.5590 (2012)"},{"issue":"1","key":"3240_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/2200000006","volume":"2","author":"Y Bengio","year":"2009","unstructured":"Bengio, Y., et al.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1\u2013127 (2009)","journal-title":"Found. Trends Mach. Learn."},{"key":"3240_CR7","doi-asserted-by":"crossref","unstructured":"Bergstra, J., et al.: Theano: A CPU and GPU math compiler in python. In: Proc. 9th Python in Science Conf, vol. 1 (2010)","DOI":"10.25080\/Majora-92bf1922-003"},{"key":"3240_CR8","unstructured":"Chen, T., et al.: Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 (2015)"},{"key":"3240_CR9","unstructured":"Chollet, F., et al.: Keras: The python deep learning library. Astrophysics Source Code Library (2018)"},{"key":"3240_CR10","unstructured":"Coleman, C., et al.: Dawnbench: an end-to-end deep learning benchmark and competition. Training (2017)"},{"key":"3240_CR11","first-page":"2493","volume":"12","author":"R Collobert","year":"2011","unstructured":"Collobert, R., et al.: Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12, 2493\u20132537 (2011)","journal-title":"J. Mach. Learn. Res."},{"key":"3240_CR12","unstructured":"Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: a matlab-like environment for machine learning. In: BigLearn, NIPS workshop, number EPFL-CONF-192376 (2011)"},{"key":"3240_CR13","first-page":"46","volume":"1","author":"L Dagum","year":"1998","unstructured":"Dagum, L., Menon, R.: Openmp: an industry-standard API for shared-memory programming. Comput. Sci. Eng. 1, 46\u201355 (1998)","journal-title":"Comput. Sci. Eng."},{"issue":"1","key":"3240_CR14","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1007\/s11263-014-0733-5","volume":"111","author":"M Everingham","year":"2015","unstructured":"Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vis. 111(1), 98\u2013136 (2015)","journal-title":"Int. J. Comput. Vis."},{"issue":"4","key":"3240_CR15","doi-asserted-by":"publisher","first-page":"2689","DOI":"10.1007\/s10586-019-03037-6","volume":"23","author":"X Geng","year":"2020","unstructured":"Geng, X., Zhang, H., Zhao, Z., Ma, H.: Interference-aware parallelization for deep learning workload in GPU cluster. Clust. Comput. 23(4), 2689\u20132702 (2020)","journal-title":"Clust. Comput."},{"issue":"2","key":"3240_CR16","doi-asserted-by":"publisher","first-page":"87","DOI":"10.2478\/v10117-011-0021-1","volume":"30","author":"J Hauke","year":"2011","unstructured":"Hauke, J., Kossowski, T.: Comparison of values of pearson\u2019s and spearman\u2019s correlation coefficients on the same sets of data. Quaestiones Geographicae 30(2), 87\u201393 (2011)","journal-title":"Quaestiones Geographicae"},{"key":"3240_CR17","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770\u2013778 (2016)","DOI":"10.1109\/CVPR.2016.90"},{"issue":"7","key":"3240_CR18","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1109\/MC.2008.209","volume":"41","author":"MD Hill","year":"2008","unstructured":"Hill, M.D., Marty, M.R.: Amdahl\u2019s law in the multicore era. Computer 41(7), 33\u201338 (2008)","journal-title":"Computer"},{"issue":"6","key":"3240_CR19","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MSP.2012.2205597","volume":"29","author":"G Hinton","year":"2012","unstructured":"Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82\u201397 (2012)","journal-title":"IEEE Signal Process. Mag."},{"key":"3240_CR20","unstructured":"Huval, B., Wang, T., Tandon, S., Kiske, J., Song, W., Pazhayampallil, J., Andriluka, M., Rajpurkar, P., Migimatsu, T., Cheng-Yue, R., et al.: An empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716 (2015)"},{"key":"3240_CR21","unstructured":"Intel caffe. (2017)"},{"key":"3240_CR22","doi-asserted-by":"crossref","unstructured":"Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675\u2013678. ACM (2014)","DOI":"10.1145\/2647868.2654889"},{"issue":"3","key":"3240_CR23","doi-asserted-by":"publisher","first-page":"1987","DOI":"10.1007\/s10586-019-02959-5","volume":"23","author":"Z Jiang","year":"2020","unstructured":"Jiang, Z., Gao, S.: An intelligent recommendation approach for online advertising based on hybrid deep neural network and parallel computing. Clust. Comput. 23(3), 1987\u20132000 (2020)","journal-title":"Clust. Comput."},{"issue":"3","key":"3240_CR24","doi-asserted-by":"publisher","first-page":"2193","DOI":"10.1007\/s10586-019-02974-6","volume":"23","author":"Y Kim","year":"2020","unstructured":"Kim, Y., Lee, J., Kim, J.-S., Jei, H., Roh, H.: Comprehensive techniques of multi-GPU memory optimization for deep learning acceleration. Clust. Comput. 23(3), 2193\u20132204 (2020)","journal-title":"Clust. Comput."},{"key":"3240_CR25","unstructured":"Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)"},{"key":"3240_CR26","unstructured":"Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097\u20131105 (2012)"},{"key":"3240_CR27","unstructured":"Liu, J., Dutta, J., Li, N., Kurup, U., Shah, M.: Usability study of distributed deep learning frameworks for convolutional neural networks (2018)"},{"key":"3240_CR28","unstructured":"Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologiesvol. 1, pp. 142\u2013150. Association for Computational Linguistics (2011)"},{"key":"3240_CR29","unstructured":"Mahmoud, N., Essam, Y., Shawi, R.E., Sakr, S.: DLBench: an experimental evaluation of deep learning frameworks. In: 2019 IEEE International Congress on Big Data, BigData Congress 2019, Milan, Italy, July 8\u201313, 2019, pp. 149\u2013156 (2019)"},{"key":"3240_CR30","doi-asserted-by":"crossref","unstructured":"Marcus, M., Santorini, B., Marcinkiewicz, M.A.: Building a large annotated corpus of english: the penn treebank (1993)","DOI":"10.21236\/ADA273556"},{"key":"3240_CR31","unstructured":"Mkl-dnn for scalable deep learning. (2017)"},{"key":"3240_CR32","unstructured":"N. Corporation. AI computing leadership from nvidia. In: https:\/\/www.nvidia.com\/en-us\/ (2018)"},{"key":"3240_CR33","unstructured":"Netzer, Y., et al.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011)"},{"key":"3240_CR34","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G.: Tensors and dynamic neural networks in python with strong GPU acceleration (2017)"},{"key":"3240_CR35","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)"},{"key":"3240_CR36","doi-asserted-by":"crossref","unstructured":"Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)","DOI":"10.18653\/v1\/D16-1264"},{"key":"3240_CR37","unstructured":"Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91\u201399 (2015)"},{"key":"3240_CR38","doi-asserted-by":"crossref","unstructured":"Sak, H., Senior, A., Beaufays, F.: Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)","DOI":"10.21437\/Interspeech.2014-80"},{"key":"3240_CR39","doi-asserted-by":"crossref","unstructured":"Seide, F., Agarwal, A.: Cntk: Microsoft\u2019s open-source deep-learning toolkit. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2135. ACM (2016)","DOI":"10.1145\/2939672.2945397"},{"key":"3240_CR40","doi-asserted-by":"crossref","unstructured":"Shams, S., Platania, R., Lee, K., Park, S.-J.: Evaluation of deep learning frameworks over different HPC architectures. In: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pp. 1389\u20131396. IEEE (2017)","DOI":"10.1109\/ICDCS.2017.259"},{"key":"3240_CR41","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1146\/annurev-bioeng-071516-044442","volume":"19","author":"D Shen","year":"2017","unstructured":"Shen, D., Wu, G., Suk, H.-I.: Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221\u2013248 (2017)","journal-title":"Annu. Rev. Biomed. Eng."},{"key":"3240_CR42","doi-asserted-by":"crossref","unstructured":"Shi, S., Wang, Q., Xu, P., Chu, X.: Benchmarking state-of-the-art deep learning software tools. In: IEEE CCBD (2016)","DOI":"10.1109\/CCBD.2016.029"},{"key":"3240_CR43","unstructured":"Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)"},{"key":"3240_CR44","unstructured":"Team, H.: High performance deep learning project. Int. J. Comput. Vis. (2017)"},{"key":"3240_CR45","unstructured":"Tokui, S., Oono, K., Hido, S., Clayton, J.: Chainer: a next-generation open source framework for deep learning. In: Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS) (2015)"},{"key":"3240_CR46","unstructured":"Tokui, S., Oono, K., Hido, S., Clayton, J.: Chainer: a next-generation open source framework for deep learning. In: NIPS Workshops (2015)"},{"key":"3240_CR47","unstructured":"T. report. Worldwide semiannual cognitive\/artificial intelligence systems spending guide. In: International Data Corporation (2017)"},{"issue":"2","key":"3240_CR48","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1007\/s11263-013-0620-5","volume":"104","author":"JR Uijlings","year":"2013","unstructured":"Uijlings, J.R., Van De Sande, K.E., Gevers, T., Smeulders, A.W.: Selective search for object recognition. Int. J. Comput. Vis. 104(2), 154\u2013171 (2013)","journal-title":"Int. J. Comput. Vis."},{"key":"3240_CR49","doi-asserted-by":"publisher","first-page":"102663","DOI":"10.1016\/j.jvcir.2019.102663","volume":"65","author":"Q Wang","year":"2019","unstructured":"Wang, Q., Guo, G.: Benchmarking deep learning techniques for face recognition. J. Vis. Commun. Image Represent. 65, 102663 (2019)","journal-title":"J. Vis. Commun. Image Represent."},{"key":"3240_CR50","doi-asserted-by":"crossref","unstructured":"Woolson, R.: Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials, pp. 1\u20133 (2007)","DOI":"10.1002\/9780471462422.eoct979"},{"key":"3240_CR51","unstructured":"Wu, Y., Liu, L., Pu, C., Cao, W., Sahin, S., Wei, W., Zhang, Q.: A comparative measurement study of deep learning as a service framework. IEEE Trans. Serv. Comput. (2019)"},{"key":"3240_CR52","volume-title":"Openblas: An Optimized Blas Library","author":"Z Xianyi","year":"2016","unstructured":"Xianyi, Z., Qian, W., Saar, W.: Openblas: An Optimized Blas Library. Agosto, Accedido (2016)"},{"key":"3240_CR53","doi-asserted-by":"crossref","unstructured":"Yang, C.-T., Liu, J.-C., Chan, Y.-W., Kristiani, E., Kuo, C.-F.: Performance benchmarking of deep learning framework on intel xeon phi. J. Supercomput. 1\u201325 (2020)","DOI":"10.1007\/s11227-020-03362-3"},{"key":"3240_CR54","doi-asserted-by":"crossref","unstructured":"Zhu, H., Akrout, M., Zheng, B., Pelegris, A., Phanishayee, A., Schroeder, B., Pekhimenko, G.: Tbd: Benchmarking and analyzing deep neural network training. arXiv preprint arXiv:1803.06905 (2018)","DOI":"10.1109\/IISWC.2018.8573476"},{"key":"3240_CR55","doi-asserted-by":"crossref","unstructured":"Zou, S.-X., Chen, C.-Y., Wu, J.-L., Chou, C.-N., Tsao, C.-C., Tung, K.-C., Lin, T.-W., Sung, C.-L., Chang, E.Y.: Distributed training large-scale deep architectures. In: International Conference on Advanced Data Mining and Applications, pp. 18\u201332. Springer (2017)","DOI":"10.1007\/978-3-319-69179-4_2"}],"container-title":["Cluster Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-021-03240-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10586-021-03240-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-021-03240-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T19:33:06Z","timestamp":1671132786000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10586-021-03240-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,7]]},"references-count":55,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,9]]}},"alternative-id":["3240"],"URL":"https:\/\/doi.org\/10.1007\/s10586-021-03240-4","relation":{},"ISSN":["1386-7857","1573-7543"],"issn-type":[{"value":"1386-7857","type":"print"},{"value":"1573-7543","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,7]]},"assertion":[{"value":"17 January 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 January 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 January 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 February 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}