{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T23:40:31Z","timestamp":1792798831550,"version":"4.5.0"},"reference-count":28,"publisher":"Association for Computing Machinery (ACM)","issue":"5s","license":[{"start":{"date-parts":[[2019,10,8]],"date-time":"2019-10-08T00:00:00Z","timestamp":1570492800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2019,10,31]]},"abstract":"<jats:p>\n            Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a\n            <jats:italic>distributed inference<\/jats:italic>\n            paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communication-aware. Therefore, we propose\n            <jats:italic>Network of Neural Networks<\/jats:italic>\n            (NoNN), a new distributed IoT learning paradigm that compresses a large pretrained \u2018teacher\u2019 deep network into several disjoint and highly-compressed \u2018student\u2019 modules, without loss of accuracy. Moreover, we propose a network science-based knowledge partitioning algorithm for the teacher model, and then train individual students on the resulting disjoint partitions. Extensive experimentation on five image classification datasets, for user-defined memory\/performance budgets, show that NoNN achieves higher accuracy than several baselines and similar accuracy as the teacher model, while using minimal communication among students. Finally, as a case study, we deploy the proposed model for CIFAR-10 dataset on edge devices and demonstrate significant improvements in memory footprint (up to 24\u00d7), performance (up to 12\u00d7), and energy per node (up to 14\u00d7) compared to the large teacher model. We further show that for distributed inference on multiple edge devices, our proposed NoNN model results in up to 33\u00d7 reduction in total latency w.r.t. a state-of-the-art model compression baseline.\n          <\/jats:p>","DOI":"10.1145\/3358205","type":"journal-article","created":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T13:13:05Z","timestamp":1570713185000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":46,"title":["Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT"],"prefix":"10.1145","volume":"18","author":[{"given":"Kartikeya","family":"Bhardwaj","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ching-Yi","family":"Lin","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anderson","family":"Sartor","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Radu","family":"Marculescu","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,10,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems. 2654--266  Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems. 2654--266"},{"key":"e_1_2_1_2_1","volume-title":"ONNX: Open Neural Network Exchange Format. https:\/\/onnx.ai\/","year":"2017","unstructured":"Facebook. 2017 . ONNX: Open Neural Network Exchange Format. https:\/\/onnx.ai\/ Facebook. 2017. ONNX: Open Neural Network Exchange Format. https:\/\/onnx.ai\/"},{"key":"e_1_2_1_3_1","unstructured":"Hongyang Gao Zhengyang Wang and Shuiwang Ji. 2018. ChannelNets: Compact and efficient convolutional neural networks via channel-wise convolutions. In Advances in Neural Information Processing Systems. 5203--5211.  Hongyang Gao Zhengyang Wang and Shuiwang Ji. 2018. ChannelNets: Compact and efficient convolutional neural networks via channel-wise convolutions. In Advances in Neural Information Processing Systems. 5203--5211."},{"key":"e_1_2_1_4_1","volume-title":"Dally","author":"Han Song","year":"2015","unstructured":"Song Han , Huizi Mao , and William J . Dally . 2015 . Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv:1510.00149 (2015). Song Han, Huizi Mao, and William J. Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv:1510.00149 (2015)."},{"key":"e_1_2_1_5_1","unstructured":"Song Han Jeff Pool John Tran and William Dally. 2015. Learning both weights and connections for efficient neural network. In NIPS. 1135--1143.  Song Han Jeff Pool John Tran and William Dally. 2015. Learning both weights and connections for efficient neural network. In NIPS. 1135--1143."},{"key":"e_1_2_1_6_1","volume-title":"Distilling the knowledge in a neural network. arXiv:1503.02531","author":"Hinton Geoffrey","year":"2015","unstructured":"Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . 2015. Distilling the knowledge in a neural network. arXiv:1503.02531 ( 2015 ). Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv:1503.02531 (2015)."},{"key":"e_1_2_1_7_1","unstructured":"Jeremy Howard. 2018. Imagenet in 18 minutes. https:\/\/www.fast.ai\/2018\/08\/10\/fastai-diu-imagenet\/. (2018). Accessed: 2018-10-01.  Jeremy Howard. 2018. Imagenet in 18 minutes. https:\/\/www.fast.ai\/2018\/08\/10\/fastai-diu-imagenet\/. (2018). Accessed: 2018-10-01."},{"key":"e_1_2_1_8_1","first-page":"6869","article-title":"Quantized neural networks: Training neural networks with low precision weights and activations","volume":"18","author":"Itay Hubara","year":"2017","unstructured":"Itay Hubara et al. 2017 . Quantized neural networks: Training neural networks with low precision weights and activations . JMLR 18 , 1 (2017), 6869 -- 6898 . Itay Hubara et al. 2017. Quantized neural networks: Training neural networks with low precision weights and activations. JMLR 18, 1 (2017), 6869--6898.","journal-title":"JMLR"},{"key":"e_1_2_1_9_1","volume-title":"SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and &lt","author":"Iandola Forrest N.","year":"2016","unstructured":"Forrest N. Iandola , Song Han , Matthew W. Moskewicz , Khalid Ashraf , William J. Dally , and Kurt Keutzer . 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and &lt ; 0.5 MB model size. arXiv:1602.07360 ( 2016 ). Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and &lt; 0.5 MB model size. arXiv:1602.07360 (2016)."},{"key":"e_1_2_1_10_1","volume-title":"International Conference on Machine Learning. 1866--1874","author":"Kim Juyong","year":"2017","unstructured":"Juyong Kim , Yookoon Park , Gunhee Kim , and Sung Ju Hwang . 2017 . SplitNet: Learning to semantically split deep networks for parameter reduction and model parallelization . In International Conference on Machine Learning. 1866--1874 . Juyong Kim, Yookoon Park, Gunhee Kim, and Sung Ju Hwang. 2017. SplitNet: Learning to semantically split deep networks for parameter reduction and model parallelization. In International Conference on Machine Learning. 1866--1874."},{"key":"e_1_2_1_11_1","volume-title":"Rush","author":"Kim Yoon","year":"2016","unstructured":"Yoon Kim and Alexander M . Rush . 2016 . Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947 (2016). Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947 (2016)."},{"key":"e_1_2_1_12_1","volume-title":"Deep convolutional neural network inference with floating-point weights and fixed-point activations. arXiv:1703.03073","author":"Lai Liangzhen","year":"2017","unstructured":"Liangzhen Lai , Naveen Suda , and Vikas Chandra . 2017. Deep convolutional neural network inference with floating-point weights and fixed-point activations. arXiv:1703.03073 ( 2017 ). Liangzhen Lai, Naveen Suda, and Vikas Chandra. 2017. Deep convolutional neural network inference with floating-point weights and fixed-point activations. arXiv:1703.03073 (2017)."},{"key":"e_1_2_1_13_1","volume-title":"Pruning filters for efficient convnets. arXiv:1608.08710","author":"Li Hao","year":"2016","unstructured":"Hao Li , Asim Kadav , Igor Durdanovic , Hanan Samet , and Hans Peter Graf . 2016. Pruning filters for efficient convnets. arXiv:1608.08710 ( 2016 ). Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv:1608.08710 (2016)."},{"key":"e_1_2_1_14_1","volume-title":"2017 DATE Conference. IEEE, 1396--1401","author":"Jiachen","unstructured":"Jiachen Mao et al. 2017. Modnn: Local distributed mobile computing system for deep neural network . In 2017 DATE Conference. IEEE, 1396--1401 . Jiachen Mao et al. 2017. Modnn: Local distributed mobile computing system for deep neural network. In 2017 DATE Conference. IEEE, 1396--1401."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCAD.2017.8203852"},{"key":"e_1_2_1_16_1","volume-title":"Watts","author":"Newman Mark","year":"2011","unstructured":"Mark Newman , Albert-Laszlo Barabasi , and Duncan J . Watts . 2011 . The Structure and Dynamics of Networks. Vol. 19 . Princeton University Press . Mark Newman, Albert-Laszlo Barabasi, and Duncan J. Watts. 2011. The Structure and Dynamics of Networks. Vol. 19. Princeton University Press."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0601602103"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206537"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Mark Sandler et al. 2018. Inverted residuals and linear bottlenecks: Mobile networks for classification detection and segmentation. arXiv:1801.04381 (2018).  Mark Sandler et al. 2018. Inverted residuals and linear bottlenecks: Mobile networks for classification detection and segmentation. arXiv:1801.04381 (2018).","DOI":"10.1109\/CVPR.2018.00474"},{"key":"e_1_2_1_20_1","unstructured":"STMicro. 2018. Datasheet for Arm-Based Microcontroller with up to 512KB total storage (including FLASH memory). Product Page: https:\/\/bit.ly\/2I5ZSMR. Datasheet. https:\/\/bit.ly\/2Kz8ehD  STMicro. 2018. Datasheet for Arm-Based Microcontroller with up to 512KB total storage (including FLASH memory). Product Page: https:\/\/bit.ly\/2I5ZSMR. Datasheet. https:\/\/bit.ly\/2Kz8ehD"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2016.7472809"},{"key":"e_1_2_1_22_1","unstructured":"Peter Welinder Steve Branson Takeshi Mita Catherine Wah Florian Schroff Serge Belongie and Pietro Perona. 2010. Caltech-UCSD birds 200. (2010).  Peter Welinder Steve Branson Takeshi Mita Catherine Wah Florian Schroff Serge Belongie and Pietro Perona. 2010. Caltech-UCSD birds 200. (2010)."},{"key":"e_1_2_1_23_1","unstructured":"Tien-Ju Yang etal 2016. Designing energy-efficient convolutional neural networks using energy-aware pruning. arXiv:1611.05128 (2016).  Tien-Ju Yang et al. 2016. Designing energy-efficient convolutional neural networks using energy-aware pruning. arXiv:1611.05128 (2016)."},{"key":"e_1_2_1_24_1","volume-title":"Wide residual networks. BMVC","author":"Zagoruyko Sergey","year":"2016","unstructured":"Sergey Zagoruyko and Nikos Komodakis . 2016. Wide residual networks. BMVC ( 2016 ). Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. BMVC (2016)."},{"key":"e_1_2_1_25_1","volume-title":"Improving the performance of convolutional neural networks via attention transfer. ICLR","author":"Zagoruyko Sergey","year":"2017","unstructured":"Sergey Zagoruyko and Nikos Komodakis . 2017. Improving the performance of convolutional neural networks via attention transfer. ICLR ( 2017 ). Sergey Zagoruyko and Nikos Komodakis. 2017. Improving the performance of convolutional neural networks via attention transfer. ICLR (2017)."},{"key":"e_1_2_1_26_1","unstructured":"Xiangyu Zhang et al. 2017. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. CoRR abs\/1707.01083 (2017).  Xiangyu Zhang et al. 2017. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. CoRR abs\/1707.01083 (2017)."},{"key":"e_1_2_1_27_1","volume-title":"Hello edge: Keyword spotting on microcontrollers. arXiv:1711.07128","author":"Zhang Yundong","year":"2017","unstructured":"Yundong Zhang , Naveen Suda , Liangzhen Lai , and Vikas Chandra . 2017. Hello edge: Keyword spotting on microcontrollers. arXiv:1711.07128 ( 2017 ). Yundong Zhang, Naveen Suda, Liangzhen Lai, and Vikas Chandra. 2017. Hello edge: Keyword spotting on microcontrollers. arXiv:1711.07128 (2017)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2018.2858384"}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3358205","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3358205","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:23:07Z","timestamp":1750202587000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3358205"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,8]]},"references-count":28,"journal-issue":{"issue":"5s","published-print":{"date-parts":[[2019,10,31]]}},"alternative-id":["10.1145\/3358205"],"URL":"https:\/\/doi.org\/10.1145\/3358205","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"value":"1539-9087","type":"print"},{"value":"1558-3465","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,8]]},"assertion":[{"value":"2019-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}