{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:08:46Z","timestamp":1760242126902,"version":"build-2065373602"},"reference-count":29,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2018,12,28]],"date-time":"2018-12-28T00:00:00Z","timestamp":1545955200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Convolutional Neural Networks (CNNs) are brain-inspired computational models designed to recognize patterns. Recent advances demonstrate that CNNs are able to achieve, and often exceed, human capabilities in many application domains. Made of several millions of parameters, even the simplest CNN shows large model size. This characteristic is a serious concern for the deployment on resource-constrained embedded-systems, where compression stages are needed to meet the stringent hardware constraints. In this paper, we introduce a novel accuracy-driven compressive training algorithm. It consists of a two-stage flow: first, layers are sorted by means of heuristic rules according to their significance; second, a modified stochastic gradient descent optimization is applied on less significant layers such that their representation is collapsed into a constrained subspace. Experimental results demonstrate that our approach achieves remarkable compression rates with low accuracy loss (&lt;1%).<\/jats:p>","DOI":"10.3390\/fi11010007","type":"journal-article","created":{"date-parts":[[2018,12,28]],"date-time":"2018-12-28T11:52:42Z","timestamp":1545997962000},"page":"7","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Layer-Wise Compressive Training for Convolutional Neural Networks"],"prefix":"10.3390","volume":"11","author":[{"given":"Matteo","family":"Grimaldi","sequence":"first","affiliation":[{"name":"Department of Control and Computer Engineering, Politecnico di Torino, Turin 10129, Italy"}]},{"given":"Valerio","family":"Tenace","sequence":"additional","affiliation":[{"name":"Department of Control and Computer Engineering, Politecnico di Torino, Turin 10129, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5881-3811","authenticated-orcid":false,"given":"Andrea","family":"Calimera","sequence":"additional","affiliation":[{"name":"Department of Control and Computer Engineering, Politecnico di Torino, Turin 10129, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2018,12,28]]},"reference":[{"key":"ref_1","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, Nevada."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Poplin, R., Varadarajan, A.V., Blumer, K., Liu, Y., McConnell, M.V., Corrado, G.S., Peng, L., and Webster, D.R. (arXiv, 2017). Predicting cardiovascular risk factors from retinal fundus photographs using deep learning, arXiv.","DOI":"10.1038\/s41551-018-0195-0"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Milletari, F., Navab, N., and Ahmadi, S.A. (2016, January 25\u201328). V-net: Fully convolutional neural networks for volumetric medical image segmentation. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.","DOI":"10.1109\/3DV.2016.79"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"Litjens","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_5","unstructured":"Schl\u00fcter, J., and Grill, T. (2015, January 26\u201330). Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks. Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR 2015), Malaga, Spain."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wang, J., Chen, Y., Hao, S., Peng, X., and Hu, L. (2018). Deep learning for sensor-based activity recognition: A survey. Pattern Recognit. Lett., in press.","DOI":"10.1016\/j.patrec.2018.02.010"},{"key":"ref_7","unstructured":"Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., and Zhang, J. (arXiv, 2016). End to end learning for self-driving cars, arXiv."},{"key":"ref_8","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (arXiv, 2015). Distilling the knowledge in a neural network, arXiv."},{"key":"ref_9","unstructured":"Ba, J., and Caruana, R. (arXiv, 2014). Do deep nets really need to be deep?, arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Rigamonti, R., Sironi, A., Lepetit, V., and Fua, P. (2013, January 23\u201328). Learning separable filters. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.355"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Jaderberg, M., Vedaldi, A., and Zisserman, A. (arXiv, 2014). Speeding up convolutional neural networks with low rank expansions, arXiv.","DOI":"10.5244\/C.28.88"},{"key":"ref_12","unstructured":"LeCun, Y., Denker, J.S., and Solla, S.A. (1990). Optimal brain damage. Advances in Neural Information Processing Systems, Morgan Kaufmann Publishers Inc."},{"key":"ref_13","unstructured":"Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H.P. (arXiv, 2016). Pruning filters for efficient convnets, arXiv."},{"key":"ref_14","unstructured":"Han, S., Mao, H., and Dally, W.J. (arXiv, 2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv."},{"key":"ref_15","unstructured":"Gong, Y., Liu, L., Yang, M., and Bourdev, L. (arXiv, 2014). Compressing deep convolutional networks using vector quantization, arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wu, J., Leng, C., Wang, Y., Hu, Q., and Cheng, J. (2016, January 27\u201330). Quantized convolutional neural networks for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.521"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016, January 11\u201314). Xnor-net: Imagenet classification using binary convolutional neural networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46493-0_32"},{"key":"ref_18","unstructured":"Courbariaux, M., Bengio, Y., and David, J.P. (arXiv, 2015). Binaryconnect: Training deep neural networks with binary weights during propagations, arXiv."},{"key":"ref_19","unstructured":"Zhu, C., Han, S., Mao, H., and Dally, W.J. (arXiv, 2016). Trained ternary quantization, arXiv."},{"key":"ref_20","unstructured":"Li, F.B., and Zhang, B.L. (arXiv, 2016). Ternary Weight Networks, arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hashemi, S., Anthony, N., Tann, H., Bahar, R.I., and Reda, S. (2017, January 27\u201331). Understanding the impact of precision quantization on the accuracy and energy of neural networks. Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE), Lausanne, Switzerland.","DOI":"10.23919\/DATE.2017.7927224"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Grimaldi, M., Pugliese, F., Tenace, V., and Calimera, A. (2018, January 4). A compression-driven training framework for embedded deep neural networks. Proceedings of the Workshop on INTelligent Embedded Systems Architectures and Applications, Turin, Italy.","DOI":"10.1145\/3285017.3285021"},{"key":"ref_23","unstructured":"Han, S., Pool, J., Tran, J., and Dally, W. (arXiv, 2015). Learning both weights and connections for efficient neural network, arXiv."},{"key":"ref_24","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 4\u20139). Automatic differentiation in PyTorch. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_25","unstructured":"Simonyan, K., and Zisserman, A. (arXiv, 2014). Very deep convolutional networks for large-scale image recognition, arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_27","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features From Tiny Images, University of Toronto. Technical Report."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_29","unstructured":"Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. (arXiv, 2016). Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients, arXiv."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/11\/1\/7\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:36:27Z","timestamp":1760196987000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/11\/1\/7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,12,28]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,1]]}},"alternative-id":["fi11010007"],"URL":"https:\/\/doi.org\/10.3390\/fi11010007","relation":{},"ISSN":["1999-5903"],"issn-type":[{"type":"electronic","value":"1999-5903"}],"subject":[],"published":{"date-parts":[[2018,12,28]]}}}