{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T10:34:14Z","timestamp":1772793254701,"version":"3.50.1"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T00:00:00Z","timestamp":1647561600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T00:00:00Z","timestamp":1647561600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Key Research and Development Program of China, Jiangsu Province","award":["BE2020729"],"award-info":[{"award-number":["BE2020729"]}]},{"DOI":"10.13039\/501100013112","name":"Transformation Program of Scientific and Technological Achievements of Jiangsu Provence","doi-asserted-by":"publisher","award":["BA2021002"],"award-info":[{"award-number":["BA2021002"]}],"id":[{"id":"10.13039\/501100013112","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Key Technology Project of China Huaneng Group","award":["No. HNKJ20-H64"],"award-info":[{"award-number":["No. HNKJ20-H64"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["CCF Trans. HPC"],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In edge computing, though distributed training of Deep Neural Networks (DNNs) is expected to exchange massive gradients between parameter servers and working nodes, the high communication cost constrains the training speed. To break this limitation, gradient compression algorithms expect the ultimate compression ratio at the expense of the accuracy of the trained model. Therefore, new gradient compression techniques are necessary to ensure both communication efficiency and model accuracy. This paper introduces a novel technique\u2014an Adaptive Sparse Ternary Gradient Compression (ASTC) scheme, which relies on the number of gradients in model layers to compress gradients. ASTC establishes the model compression selection criterion by gradients\u2019 amount, compresses the network layer that meets the model\u2019s standard, evaluates the gradients\u2019 importance based on entropy to adaptively perform sparse compression, and finally conducts ternary quantization compression and a lossless code scheme on sparse gradients. Using public datasets (MNIST, CIFAR-10, Tiny ImageNet) and deep learning models (CNN, LeNet5, ResNet18) for experimental evaluation, we exhibit excellent results that the training efficiency of ASTC is about 1.6 times, 1.37 times, and 1.1 times higher than that of Top-1, AdaComp, and SBC, respectively. Furthermore, ASTC can be improved by an average of about<jats:inline-formula><jats:alternatives><jats:tex-math>$$1.9\\%$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mrow><mml:mn>1.9<\/mml:mn><mml:mo>%<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>in training accuracy compared with the above approaches.<\/jats:p>","DOI":"10.1007\/s42514-022-00091-2","type":"journal-article","created":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T16:12:53Z","timestamp":1647619973000},"page":"120-134","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Adaptive sparse ternary gradient compression for distributed DNN training in edge computing"],"prefix":"10.1007","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9884-8100","authenticated-orcid":false,"given":"Yingchi","family":"Mao","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jun","family":"Wu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xuesong","family":"Xu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Longbao","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,3,18]]},"reference":[{"key":"91_CR1","unstructured":"Abimbola, B.: Cloud computing concept and roots. arXiv:2102.00981 (2021)"},{"key":"91_CR2","doi-asserted-by":"crossref","unstructured":"Aji, A.F., Heafield, K.: Sparse communication for distributed gradient descent[C]. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 440\u2013445 (2017)","DOI":"10.18653\/v1\/D17-1045"},{"key":"91_CR3","unstructured":"Alistarh, D., Grubic, D., Li, J., et al.: QSGD: communication-efficient SGD via gradient quantization and encoding[C]. In: Advances in Neural Information Processing Systems, pp. 1709\u20131720 (2017)"},{"key":"91_CR4","unstructured":"Alistarh, D., Hoefler, T., Johansson, M., et al.: The convergence of sparsified gradient methods[C]. In: Proceedings of the 32Nd International Conference on Neural Information Processing Systems, pp. 5977\u20135987 (2018)"},{"key":"91_CR5","unstructured":"Bernstein, J., Wang, Y.X., Azizzadenesheli, K., et al.: signSGD: Compressed optimisation for non-convex problems[C]. In: International Conference on Machine Learning, pp. 560\u2013569 (2018)"},{"key":"91_CR6","doi-asserted-by":"crossref","unstructured":"Cao, N.Y., Chatterjee, B., Gong, M.X., et al.: A 65nm image processing SoC supporting multiple DNN models and real-time computation-communication trade-off via actor-critical neuro-controller[C]. In: VLSI Circuits 2020, pp. 1\u20132 (2020)","DOI":"10.1109\/VLSICircuits18222.2020.9162878"},{"key":"91_CR7","doi-asserted-by":"crossref","unstructured":"Chen, C.Y., Choi, J., Brand, D., et al.: Adacomp: adaptive residual gradient compression for data-parallel distributed training[C]. In: Proceedings of the AAAI Conference on Artificial Intelligence (2018)","DOI":"10.1609\/aaai.v32i1.11728"},{"key":"91_CR8","doi-asserted-by":"publisher","first-page":"1655","DOI":"10.1109\/JPROC.2019.2921977","volume":"107","author":"J Chen","year":"2019","unstructured":"Chen, J., Ran, X.: Deep learning with edge computing: a review. Proc. IEEE 107, 1655\u20131674 (2019)","journal-title":"Proc. IEEE"},{"key":"91_CR9","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1145\/2408776.2408794","volume":"56","author":"J Dean","year":"2013","unstructured":"Dean, J., Barroso, L.A.: The tail at scale. Commsunic. ACM 56, 74\u201380 (2013)","journal-title":"Commsunic. ACM"},{"key":"91_CR10","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., et al.: Imagenet: a large-scale hierarchical image database[C]. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248\u2013255 (2009)","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"91_CR11","doi-asserted-by":"crossref","unstructured":"Derbeko, P., Dolev, S., Gudes, E., et al.: MLDStore - DNNs as similitude models for sharing big data (brief announcement)[C]. In: 3rd CSCML 2019, pp. 93\u201396 (2019)","DOI":"10.1007\/978-3-030-20951-3_7"},{"key":"91_CR12","doi-asserted-by":"crossref","unstructured":"Dryden, N., Moon, T., Jacobs, S.A., et al.: Communication quantization for data-parallel training of deep neural networks[C]. In: 2016 2nd Workshop on Machine Learning in HPC Environments (MLHPC), pp. 1\u20138 (2016)","DOI":"10.1109\/MLHPC.2016.004"},{"key":"91_CR13","doi-asserted-by":"crossref","unstructured":"Han, P., Wang, S., Leung, K.K.: Adaptive gradient sparsification for efficient federated learning: an online learning approach. arXiv:2001.04756 (2020)","DOI":"10.1109\/ICDCS47774.2020.00026"},{"key":"91_CR14","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition[C]. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770\u2013778 (2016)","DOI":"10.1109\/CVPR.2016.90"},{"key":"91_CR15","first-page":"1587","volume":"13","author":"T Hoang","year":"2018","unstructured":"Hoang, T., Dinh, C.L., Dusit, N., et al.: A survey of mobile cloud computing: architecture, applications, and approaches. Wirel. Commun. Mob. Comput. 13, 1587\u20131611 (2018)","journal-title":"Wirel. Commun. Mob. Comput."},{"key":"91_CR16","first-page":"749","volume":"12","author":"CVN Index","year":"2017","unstructured":"Index, C.V.N.: Cisco visual networking index: forecast and methodology, 2016\u20132021. Complet. Vis. Netw. Index Forecast 12, 749\u2013759 (2017)","journal-title":"Complet. Vis. Netw. Index Forecast"},{"key":"91_CR17","unstructured":"Kone\u010dn\u00fd, J., McMahan, H.B., Yu, F.X., et al.: Federated learning: strategies for improving communication efficiency[C]. In: NIPS Workshop on Private Multi-Party Machine Learning (2016)"},{"key":"91_CR18","unstructured":"Krizhevsky, A., Nair, V., Hinton, G.: The cifar-10 dataset, online: http:\/\/www.cs.toronto.edu\/kriz\/cifar.html (2014)"},{"key":"91_CR19","unstructured":"Krizhevsky, A., Nair, V., Hinton, G.: The cifar-10 dataset. http:\/\/www.cs.toronto.edu\/kriz\/cifar.html (2014)"},{"key":"91_CR20","doi-asserted-by":"crossref","unstructured":"Kuang, D., Chen, M., Xiao, D., et al.: Entropy-based gradient compression for distributed deep learning[C]. In: 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC\/SmartCity\/DSS), pp. 231\u2013238 (2019)","DOI":"10.1109\/HPCC\/SmartCity\/DSS.2019.00046"},{"key":"91_CR21","unstructured":"LeCun, Y.: The mnist database of handwritten digits. http:\/\/yann.lecun.com\/exdb\/mnist\/ (1998)"},{"key":"91_CR41","doi-asserted-by":"crossref","unstructured":"LeCun, Y., Bottou, L., Bengio, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278\u20132324 (1998)","DOI":"10.1109\/5.726791"},{"key":"91_CR22","doi-asserted-by":"crossref","unstructured":"Li, H.j., Yuan, G., Niu, W., et al.: Real-time mobile acceleration of DNNs: from computer vision to medical applications[C]. In: 26th Asia and South Pacific Design Automation Conference, pp. 581\u2013586 (2021)","DOI":"10.1145\/3394885.3431627"},{"key":"91_CR23","unstructured":"Lin, Y., Han, S., Mao, H., et al.: Deep gradient compression: reducing the communication bandwidth for distributed training[C]. International Conference on Learning Representations (2018)"},{"key":"91_CR24","unstructured":"Minar, M.R., Nather, J.: Recent advances in deep learning: an overview. arXiv:1807.08169 (2018)"},{"key":"91_CR25","doi-asserted-by":"crossref","unstructured":"Munir, M., Siddiqui, S.A., Kusters, F., et al.: TSXplain: demystification of DNN decisions for time-series using natural language and statistical features[C]. In: 26th Asia and South Pacific Design Automation Conference, pp. 426\u2013439 (2019)","DOI":"10.1007\/978-3-030-30493-5_43"},{"key":"91_CR26","unstructured":"Patel, M., Naughton, B., Chan, C., et al.: Mobile-edge computing introductory technical white paper. Mob.-edge Comput. Indust. Initiat. 1089\u20137801 (2014)"},{"issue":"1","key":"91_CR27","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/S0893-6080(98)00116-6","volume":"12","author":"N Qian","year":"1999","unstructured":"Qian, N.: On the momentum term in gradient descent learning algorithms[C]. Neural Netw. 12(1), 145\u2013151 (1999)","journal-title":"Neural Netw."},{"key":"91_CR28","doi-asserted-by":"crossref","unstructured":"Raza, M.R., Varol, A., Varol, N.: Cloud and fog computing: a survey to the concept and challenges[C]. In: 8th International Symposium on Digital Forensics and Security, pp. 1\u20136 (2020)","DOI":"10.1109\/ISDFS49300.2020.9116360"},{"key":"91_CR29","doi-asserted-by":"crossref","unstructured":"Sattler, F., Wiedemann, S., M\u00fcller, K.R., et al.: Sparse binary compression: towards distributed deep learning with minimal communication[C]. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1\u20138 (2019)","DOI":"10.1109\/IJCNN.2019.8852172"},{"issue":"9","key":"91_CR30","doi-asserted-by":"publisher","first-page":"3400","DOI":"10.1109\/TNNLS.2019.2944481","volume":"31","author":"F Sattler","year":"2019","unstructured":"Sattler, F., Wiedemann, S., M\u00fcller, K.R., et al.: Robust and communication-efficient federated learning from non-iid dat[C]. IEEE Trans. Neural Netw. Learn. Syst. 31(9), 3400\u20133413 (2019)","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"91_CR31","doi-asserted-by":"crossref","unstructured":"Seide, F., Fu, H., Droppo, J., et al.: 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs[C]. In: Fifteenth Annual Conference of the International Speech Communication Association, Singapore (2014)","DOI":"10.21437\/Interspeech.2014-274"},{"key":"91_CR32","unstructured":"Stich, S.U., Cordonnier, J.B., Jaggi, M.: Sparsified SGD with memory[C]. In: Proceedings of the 32Nd International Conference on Neural Information Processing Systems, pp. 4452-4463 (2018)"},{"key":"91_CR33","doi-asserted-by":"crossref","unstructured":"Strom, N.: A tonotopic artificial neural network architecture for phoneme probability estimation[C]. In: 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pp. 156\u2013163 (1997)","DOI":"10.1109\/ASRU.1997.659000"},{"key":"91_CR34","doi-asserted-by":"crossref","unstructured":"Strom, N.: Scalable distributed DNN training using commodity GPU cloud computing[C]. In: Sixteenth Annual Conference of the International Speech Communication Association (2015)","DOI":"10.21437\/Interspeech.2015-354"},{"key":"91_CR35","doi-asserted-by":"crossref","unstructured":"Sun, M.S., Zhao, P., Wang, Y.Z., et al.: HSIM-DNN: hardware simulator for computation-, storage- and power-efficient deep neural networks[C]. In: Proceedings of the 2019 on Great Lakes Symposium on VLSI, pp. 81\u201386 (2019)","DOI":"10.1145\/3299874.3317996"},{"key":"91_CR36","unstructured":"Tang, Z., Shi, S., Chu, X., et al.: Communication-efficient distributed deep learning: a comprehensive survey. arXiv:2003.06307 (2020)"},{"key":"91_CR37","unstructured":"Wang, H., Chen, J.R., Wan, X.C., et al.: Domain-specific communication optimization for distributed DNN training. arXiv:2008.08445 (2020)"},{"key":"91_CR38","doi-asserted-by":"crossref","unstructured":"Wang, S.P., Liu, P., Wu, J.J.: Communication usage optimization of gradient sparsification with aggregation in deep learning[C]. In: Proceedings of the 2018 VII International Conference on Network, Communication and Computing, pp. 22\u201326 (2018)","DOI":"10.1145\/3301326.3301347"},{"key":"91_CR39","unstructured":"Wangni, J., Wang, J., Liu, J., et al.: Gradient sparsification for communication-efficient distributed optimization[C]. In: Proceedings of the 32Nd International Conference on Neural Information Processing Systems, pp. 1306\u20131316 (2018)"},{"key":"91_CR40","first-page":"1508","volume":"30","author":"W Wen","year":"2017","unstructured":"Wen, W., Xu, C., Yan, F., et al.: Terngrad: ternary gradients to reduce communication in distributed deep learning[C]. Adv. Neural Inf. Process. Syst. 30, 1508\u20131518 (2017)","journal-title":"Adv. Neural Inf. Process. Syst."}],"container-title":["CCF Transactions on High Performance Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42514-022-00091-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42514-022-00091-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42514-022-00091-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T15:23:00Z","timestamp":1726845780000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42514-022-00091-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,18]]},"references-count":41,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["91"],"URL":"https:\/\/doi.org\/10.1007\/s42514-022-00091-2","relation":{},"ISSN":["2524-4922","2524-4930"],"issn-type":[{"value":"2524-4922","type":"print"},{"value":"2524-4930","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,18]]},"assertion":[{"value":"1 November 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 February 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 March 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}