{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T22:58:59Z","timestamp":1772837939309,"version":"3.50.1"},"reference-count":26,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,3,31]],"date-time":"2022-03-31T00:00:00Z","timestamp":1648684800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Recurrent neural networks (RNNs) are neural networks (NN) designed for time-series applications. There is a growing interest in running RNNs to support these applications on edge devices. However, RNNs have large memory and computational demands that make them challenging to implement on edge devices. Quantization is used to shrink the size and the computational needs of such models by decreasing weights and activation precision. Further, the delta networks method increases the sparsity in activation vectors by relying on the temporal relationship between successive input sequences to eliminate repeated computations and memory accesses. In this paper, we study the effect of quantization on LSTM-, GRU-, LiGRU-, and SRU-based RNN models for speech recognition on the TIMIT dataset. We show how to apply post-training quantization on these models with a minimal increase in the error by skipping quantization of selected paths. In addition, we show that the quantization of activation vectors in RNNs to integer precision leads to considerable sparsity if the delta networks method is applied. Then, we propose a method for increasing the sparsity in the activation vectors while minimizing the error and maximizing the percentage of eliminated computations. The proposed quantization method managed to compress the four models more than 85%, with an error increase of 0.6, 0, 2.1, and 0.2 percentage points, respectively. By applying the delta networks method to the quantized models, more than 50% of the operations can be eliminated, in most cases with only a minor increase in the error. Comparing the four models to each other under the quantization and delta networks method, we found that compressed LSTM-based models are the most-optimum solutions at low-error-rates constraints. The compressed SRU-based models are the smallest in size, suitable when higher error rates are acceptable, and the compressed LiGRU-based models have the highest number of eliminated operations.<\/jats:p>","DOI":"10.3390\/info13040176","type":"journal-article","created":{"date-parts":[[2022,3,31]],"date-time":"2022-03-31T21:34:29Z","timestamp":1648762469000},"page":"176","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Shrink and Eliminate: A Study of Post-Training Quantization and Repeated Operations Elimination in RNN Models"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4674-3809","authenticated-orcid":false,"given":"Nesma M.","family":"Rezk","sequence":"first","affiliation":[{"name":"School of Information Technology, Halmstad Univeristy, 30118 Halmstad, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0562-2082","authenticated-orcid":false,"given":"Tomas","family":"Nordstr\u00f6m","sequence":"additional","affiliation":[{"name":"Department of Applied Physics and Electronics, Ume\u00e5 University, 90187 Ume\u00e5, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4932-4036","authenticated-orcid":false,"given":"Zain","family":"Ul-Abdin","sequence":"additional","affiliation":[{"name":"School of Information Technology, Halmstad Univeristy, 30118 Halmstad, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"57967","DOI":"10.1109\/ACCESS.2020.2982416","article-title":"Recurrent Neural Networks: An Embedded Computing Perspective","volume":"8","author":"Rezk","year":"2020","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gao, C., Neil, D., Ceolini, E., Liu, S.C., and Delbruck, T. (2018). DeltaRNN: A Power-Efficient Recurrent Neural Network Accelerator, Association for Computing Machinery.","DOI":"10.1145\/3174243.3174261"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Jo, J., Kung, J., Lee, S., and Lee, Y. (2019, January 29\u201331). Similarity-based LSTM architecture for energy-efficient edge-level speech recognition. Proceedings of the 2019 IEEE\/ACM International Symposium on Low Power Electronics and Design (ISLPED), Lausanne, Switzerland.","DOI":"10.1109\/ISLPED.2019.8824862"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Rezk, N.M., Nordstr\u00f6m, T., Stathis, D., Ul-Abdin, Z., Aksoy, E.E., and Hemani, A. (2021). MOHAQ: Multi-objective Hardware-Aware Quantization of Recurrent Neural Networks. arXiv.","DOI":"10.1016\/j.sysarc.2022.102778"},{"key":"ref_5","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). Post training 4-bit quantization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_6","unstructured":"Nagel, M., Amjad, R.A., Van Baalen, M., Louizos, C., and Blankevoort, T. (2020, January 13\u201318). Up or down? Adaptive rounding for post-training quantization. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3245","DOI":"10.1007\/s10994-021-06053-z","article-title":"Loss aware post-training quantization","volume":"110","author":"Nahshan","year":"2021","journal-title":"Mach. Learn."},{"key":"ref_8","unstructured":"Zhao, R., Hu, Y., Dotzel, J., De Sa, C., and Zhang, Z. (2019, January 10\u201315). Improving neural network quantization without retraining using outlier channel splitting. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M.W., and Keutzer, K. (2020, January 13\u201319). Zeroq: A novel zero shot quantization framework. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01318"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Aji, A.F., and Heafield, K. (2019). Neural Machine Translation with 4-Bit Precision and Beyond. arXiv.","DOI":"10.18653\/v1\/2020.ngt-1.4"},{"key":"ref_11","unstructured":"Chen, T., Goodfellow, I., and Shlens, J. (2016, January 2\u20134). Net2Net: Accelerating Learning via Knowledge Transfer. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico."},{"key":"ref_12","unstructured":"Neil, D., Lee, J.H., Delbruck, T., and Liu, S.C. (2017, January 6\u201311). Delta networks for optimized recurrent network computation. Proceedings of the International Conference on Machine Learning, Sydney, NSW, Australia."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Gao, C., Rios-Navarro, A., Chen, X., Delbruck, T., and Liu, S.C. (September, January 31). EdgeDRNN: Enabling low-latency recurrent neural network edge inference. Proceedings of the 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), Genova, Italy.","DOI":"10.1109\/AICAS48895.2020.9074001"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Shan, B., and Fang, Y. (2021). DRAC: A delta recurrent neural network-based arithmetic coding algorithm for edge computing. Complex Intell. Syst., 1\u20137.","DOI":"10.1007\/s40747-021-00455-1"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Cho, K., van Merrienboer, B., Bahdanau, D., and Bengio, Y. (2014, January 25). On the properties of neural machine translation: Encoder-decoder approaches. Proceedings of the SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar.","DOI":"10.3115\/v1\/W14-4012"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1109\/TETCI.2017.2762739","article-title":"Light Gated Recurrent Units for Speech Recognition","volume":"2","author":"Ravanelli","year":"2018","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lei, T., Zhang, Y., Wang, S., Dai, H., and Artzi, Y. (November, January 31). Simple Recurrent Units for Highly Parallelizable Recurrence. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.","DOI":"10.18653\/v1\/D18-1477"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ravanelli, M., Parcollet, T., and Bengio, Y. (2019, January 12\u201317). The pytorch-kaldi speech recognition toolkit. Proceedings of the ICASSP 2019\u20132019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8683713"},{"key":"ref_20","unstructured":"Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., and Schwarz, P. (2011, January 11\u201315). The Kaldi Speech Recognition Toolkit. Proceedings of the IEEE 2011 Workshop on Automatic Speech Recognition and Understanding, Waikoloa, HI, USA."},{"key":"ref_21","unstructured":"Garofolo, J., Lamel, L.F., Fisher, W.M., Fiscus, J.G., Pallett, D.S., and DARPA TIMIT Acoustic-Phonetic Continous Speech Corpus CD-ROM (2022, February 20). NIST Speech Disc 1-1.1, Available online: https:\/\/nvlpubs.nist.gov\/nistpubs\/Legacy\/IR\/nistir4930.pdf."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wang, S., Li, Z., Ding, C., Yuan, B., Qiu, Q., Wang, Y., and Liang, Y. (2018, January 25\u201327). C-LSTM: Enabling efficient LSTM using structured compression techniques on FPGAs. Proceedings of the 2018 ACM\/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, CA, USA.","DOI":"10.1145\/3174243.3174253"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Sharma, H., Park, J., Suda, N., Lai, L., Chau, B., Chandra, V., and Esmaeilzadeh, H. (2018, January 1\u20136). Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. Proceedings of the 2018 ACM\/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), Los Angeles, CA, USA.","DOI":"10.1109\/ISCA.2018.00069"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1109\/JSSC.2018.2865489","article-title":"UNPU: An Energy-Efficient Deep Neural Network Accelerator With Fully Variable Weight Bit Precision","volume":"54","author":"Lee","year":"2019","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_25","unstructured":"Sung, W., Shin, S., and Hwang, K. (2015). Resiliency of Deep Neural Networks under Quantization. arXiv."},{"key":"ref_26","unstructured":"Sung, W., and Park, J. (2018). Single Stream Parallelization of Recurrent Neural Networks for Low Power and Fast Inference. arXiv."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/4\/176\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:47:28Z","timestamp":1760136448000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/4\/176"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,31]]},"references-count":26,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["info13040176"],"URL":"https:\/\/doi.org\/10.3390\/info13040176","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,31]]}}}