{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T22:41:03Z","timestamp":1776379263164,"version":"3.51.2"},"reference-count":21,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T00:00:00Z","timestamp":1615334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Running Deep Neural Networks (DNNs) in distributed Internet of Things (IoT) nodes is a promising scheme to enhance the performance of IoT systems. However, due to the limited computing and communication resources of the IoT nodes, the communication efficiency of the distributed DNN training strategy is a problem demanding a prompt solution. In this paper, an adaptive compression strategy based on gradient partition is proposed to solve the problem of high communication overhead between nodes during the distributed training procedure. Firstly, a neural network is trained to predict the gradient distribution of its parameters. According to the distribution characteristics of the gradient, the gradient is divided into the key region and the sparse region. At the same time, combined with the information entropy of gradient distribution, a reasonable threshold is selected to filter the gradient value in the partition, and only the gradient value greater than the threshold is transmitted and updated, to reduce the traffic and improve the distributed training efficiency. The strategy uses gradient sparsity to achieve the maximum compression ratio of 37.1 times, which improves the training efficiency to a certain extent.<\/jats:p>","DOI":"10.3390\/s21061943","type":"journal-article","created":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T20:51:42Z","timestamp":1615409502000},"page":"1943","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["A Partition Based Gradient Compression Algorithm for Distributed Training in AIoT"],"prefix":"10.3390","volume":"21","author":[{"given":"Bingjun","family":"Guo","sequence":"first","affiliation":[{"name":"Department of Computer Science and Technology, North China University of Science and Technology, Tangshan 063210, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yazhi","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Technology, North China University of Science and Technology, Tangshan 063210, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunyang","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Technology, North China University of Science and Technology, Tangshan 063210, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Tang, X., Wang, X., Cattley, R., Xianghong, W., and Xiaoli, T. (2018). Energy harvesting technologies for achieving self-powered wireless sensor networks in machine condition monitoring: A review. Sensors, 18.","DOI":"10.3390\/s18124113"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"11700","DOI":"10.1002\/anie.201201656","article-title":"Nanotechnology-enabled energy harvesting for self-powered micro-\/nanosystems","volume":"51","author":"Wang","year":"2012","journal-title":"Angew. Chem. Int. Ed."},{"key":"ref_3","unstructured":"Zhang, H., Zheng, Z., Xu, S., Wei, D., Qirong, H., Xiaodan, L., Zhiting, H., Jinliang, W., Pengtaol, X., and Xing, E.P. (2017, January 12\u201314). Poseidon: An efficient communication architecture for distributed deep learning on GPU clusters. Proceedings of the 2017 USENIX Annual Technical Conference (USENIXATC 17), Santa Clara, CA, USA."},{"key":"ref_4","unstructured":"Dutta, A., Bergou, E.H., Abdelmoniem, A.M., Yu, H.C., Narayan, S.A., Marco, C., and Panos, K. (2020, January 7\u201312). On the discrepancy between the theoretical analysis and practical implementations of compressed communication for distributed deep learning. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA."},{"key":"ref_5","unstructured":"Yujun, L., Yu, W., Song, H., William, J.D., and Huizi, M. (2017). Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv."},{"key":"ref_6","unstructured":"Wang, L., Wu, W., Zhao, Y., Zhang, J., Liu, H., Bosilca, G., and Fonseca, R. (2018). SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks. arXiv."},{"key":"ref_7","unstructured":"Chmiel, B., Ben-Uri, L., Shkolnik, M., Elad, H., Ron, B., and Daniel, S. (2020). Neural gradients are lognormally distributed: Understanding sparse and quantized training. arXiv."},{"key":"ref_8","unstructured":"Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. (2017). Terngrad: Ternary gradients to reduce communication in distributed deep learning. arXiv."},{"key":"ref_9","unstructured":"Khirirat, S., Feyzmahdavian, H.R., and Johansson, M. (2018). Distributed learning with compressed gradients. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Seide, F., Fu, H., Droppo, J., Gang, L., and Dong, Y. (2014, January 14\u201318). 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. Proceedings of the Fifteenth Annual Conference of the International Speech Communication Association, Singapore.","DOI":"10.21437\/Interspeech.2014-274"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Guo, J., Liu, W., Wang, W., Jizhong, H., Ruixuan, L., Yijun, L., and Songlin, H. (2020, January 4\u20138). Accelerating Distributed Deep Learning By Adaptive Gradient Quantization. Proceedings of the ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9054164"},{"key":"ref_12","unstructured":"Mishchenko, K., Gorbunov, E., and Tak\u00e1\u010d, M. (2019). Distributed learning with compressed gradient differences. arXiv."},{"key":"ref_13","unstructured":"Alistarh, D., Grubic, D., Li, J., Ryota, T., and Milan, V. (2017, January 4\u20139). QSGD: Communication-efficient SGD via gradient quantization and encoding. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Aji, A.F., and Heafield, K. (2017). Sparse communication for distributed gradient descent. arXiv.","DOI":"10.18653\/v1\/D17-1045"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Shi, S., Wang, Q., Chu, X., Bo, L., Yang, Q., Ruihao, L., and Xinxiao, Z. (2020, January 6\u20139). Communication-efficient distributed deep learning with merged gradient sparsification on gpus. Proceedings of the IEEE INFOCOM 2020\u2014IEEE Conference on Computer Communications, Toronto, ON, Canada.","DOI":"10.1109\/INFOCOM41043.2020.9155269"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Chen, C.Y., Choi, J., Daniel, B., Ankur, A., Wei, Z., and Kailash, G. (2017). Adacomp: Adaptive residual gradient compression for data-parallel distributed training. arXiv.","DOI":"10.1609\/aaai.v32i1.11728"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Strom, N. (2015, January 6\u201310). Scalable distributed DNN training using commodity GPU cloud computing. Proceedings of the Sixteenth Annual Conference of the International Speech Communication Association, Dresden, Germany.","DOI":"10.21437\/Interspeech.2015-354"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Dryden, N., Moon, T., Jacobs, S.A., and Van Essen, B. (2016, January 14). Communication quantization for data-parallel training of deep neural networks. Proceedings of the 2016 2nd Workshop on Machine Learning in HPC Environments (MLHPC), Salt Lake City, UT, USA.","DOI":"10.1109\/MLHPC.2016.004"},{"key":"ref_19","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of information theory. Wiley Series in Telecommunications and Signal Processing Hoboken, John Wiley & Sons."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Abadi, M., Isard, M., and Murray, D.G. (2017, January 18). A computational model for TensorFlow: An introduction. Proceedings of the 1st ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, Barcelona, Spain.","DOI":"10.1145\/3088525.3088527"},{"key":"ref_21","unstructured":"Tsuzuku, Y., Imachi, H., and Akiba, T. (2018). Variance-based gradient compression for efficient distributed deep learning. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/1943\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:33:23Z","timestamp":1760160803000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/1943"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,10]]},"references-count":21,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["s21061943"],"URL":"https:\/\/doi.org\/10.3390\/s21061943","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,10]]}}}