{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:27:24Z","timestamp":1750220844572,"version":"3.41.0"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2019,9,30]],"date-time":"2019-09-30T00:00:00Z","timestamp":1569801600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2019,9,30]]},"abstract":"<jats:p>Many Internet-of-Things (IoT) applications demand fast and accurate understanding of a few key events in their surrounding environment. Deep Convolutional Neural Networks (CNNs) have emerged as an effective approach to understand speech, images, and similar high-dimensional data types. Algorithmic performance of modern CNNs, however, fundamentally relies on learning class-agnostic hierarchical features that only exist in comprehensive training datasets with many classes. As a result, fast inference using CNNs trained on such datasets is prohibitive for most resource-constrained IoT platforms. To bridge this gap, we present a principled and practical methodology for distilling a complex modern CNN that is trained to effectively recognize many different classes of input data into an application-dependent essential core that not only recognizes the few classes of interest to the application accurately but also runs efficiently on platforms with limited resources. Experimental results confirm that our approach strikes a favorable balance between classification accuracy (application constraint), inference efficiency (platform constraint), and productive development of new applications (business constraint).<\/jats:p>","DOI":"10.1145\/3360512","type":"journal-article","created":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T13:13:05Z","timestamp":1570713185000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Distill-Net"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0120-8738","authenticated-orcid":false,"given":"Mohammad","family":"Motamedi","sequence":"first","affiliation":[{"name":"University of California, Davis, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Felix A.","family":"Portillo","sequence":"additional","affiliation":[{"name":"University of California, Davis, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7171-1171","authenticated-orcid":false,"given":"Daniel","family":"Fong","sequence":"additional","affiliation":[{"name":"University of California, Davis, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Soheil","family":"Ghiasi","sequence":"additional","affiliation":[{"name":"University of California, Davis, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,10,9]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dandelion Man\u00e9 Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Retrieved from https:\/\/www.tensorflow.org\/.  Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dandelion Man\u00e9 Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Retrieved from https:\/\/www.tensorflow.org\/."},{"volume-title":"Proceedings of the 1st International Workshop on Deep Learning for Mobile Systems and Applications. ACM, 7--12","author":"Alzantot Moustafa","key":"e_1_2_1_2_1"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISVLSI.2016.111"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2010.05.010"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11277-011-0288-5"},{"key":"e_1_2_1_6_1","unstructured":"Falak Shah Bhagyesh Vikani. 2017. CNN Visualization. Retrieved from https:\/\/github.com\/InFoCusp\/tf_cnnvis\/.  Falak Shah Bhagyesh Vikani. 2017. CNN Visualization. Retrieved from https:\/\/github.com\/InFoCusp\/tf_cnnvis\/."},{"key":"e_1_2_1_7_1","unstructured":"Guobin Chen Wongun Choi Xiang Yu Tony Han and Manmohan Chandraker. 2017. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Processing Systems. 742--751.  Guobin Chen Wongun Choi Xiang Yu Tony Han and Manmohan Chandraker. 2017. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Processing Systems. 742--751."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSSC.2016.2616357"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2018.00215"},{"volume-title":"Hardware-oriented approximation of convolutional neural networks. arXiv preprint arXiv:1604.03168","year":"2016","author":"Gysel Philipp","key":"e_1_2_1_10_1"},{"volume-title":"Proceedings of the ACM\/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA\u201916)","author":"Han Song","key":"e_1_2_1_11_1"},{"volume-title":"Dally","year":"2015","author":"Han Song","key":"e_1_2_1_12_1"},{"volume-title":"Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531","year":"2015","author":"Hinton Geoffrey","key":"e_1_2_1_13_1"},{"volume-title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861","year":"2017","author":"Howard Andrew G","key":"e_1_2_1_14_1"},{"volume-title":"Weinberger","year":"2017","author":"Huang Gao","key":"e_1_2_1_15_1"},{"volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2752--2761","author":"Huang Gao","key":"e_1_2_1_16_1"},{"volume":"1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Huang Gao","key":"e_1_2_1_17_1"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3242044"},{"volume-title":"SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and &lt;0.5 MB model size. arXiv preprint arXiv:1602.07360","year":"2016","author":"Iandola Forrest N.","key":"e_1_2_1_19_1"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654889"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3079856.3080246"},{"key":"e_1_2_1_22_1","unstructured":"Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2014. The CIFAR-10 dataset. Retrieved from https:\/\/www.cs.toronto.edu\/kriz\/cifar.html.  Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2014. The CIFAR-10 dataset. Retrieved from https:\/\/www.cs.toronto.edu\/kriz\/cifar.html."},{"volume-title":"Hinton","year":"2012","author":"Krizhevsky Alex","key":"e_1_2_1_23_1"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2964284.2973801"},{"key":"e_1_2_1_25_1","unstructured":"Yann LeCun. 1998. The MNIST database of handwritten digits. Retrieved from http:\/\/yann.lecun.com\/exdb\/mnist\/.  Yann LeCun. 1998. The MNIST database of handwritten digits. Retrieved from http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"e_1_2_1_26_1","unstructured":"Ji Lin Yongming Rao Jiwen Lu and Jie Zhou. 2017. Runtime neural pruning. In Advances in Neural Information Processing Systems. 2181--2191.  Ji Lin Yongming Rao Jiwen Lu and Jie Zhou. 2017. Runtime neural pruning. In Advances in Neural Information Processing Systems. 2181--2191."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126555"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/LES.2018.2815954"},{"volume-title":"Chung","year":"2015","author":"Ovtcharov Kalin","key":"e_1_2_1_29_1"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2847263.2847265"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"volume-title":"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538","year":"2017","author":"Shazeer Noam","key":"e_1_2_1_32_1"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"volume-title":"Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud. arXiv preprint arXiv:1710.07368","year":"2017","author":"Wu Bichen","key":"e_1_2_1_34_1"},{"volume-title":"Zeiler and Rob Fergus","year":"2014","author":"Matthew","key":"e_1_2_1_35_1"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2684746.2689060"}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3360512","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3360512","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:22:58Z","timestamp":1750202578000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3360512"}},"subtitle":["Application-Specific Distillation of Deep Convolutional Neural Networks for Resource-Constrained IoT Platforms"],"short-title":[],"issued":{"date-parts":[[2019,9,30]]},"references-count":36,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2019,9,30]]}},"alternative-id":["10.1145\/3360512"],"URL":"https:\/\/doi.org\/10.1145\/3360512","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"type":"print","value":"1539-9087"},{"type":"electronic","value":"1558-3465"}],"subject":[],"published":{"date-parts":[[2019,9,30]]},"assertion":[{"value":"2018-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}