{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T16:42:48Z","timestamp":1782837768993,"version":"3.54.5"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"9","license":[{"start":{"date-parts":[[2020,7,9]],"date-time":"2020-07-09T00:00:00Z","timestamp":1594252800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,7,9]],"date-time":"2020-07-09T00:00:00Z","timestamp":1594252800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Sign Process Syst"],"published-print":{"date-parts":[[2020,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Long-Short Term Memory (LSTM) can retain memory and learn from data sequences. It gives state-of-the-art accuracy in many applications such as speech recognition, natural language processing and video classifications. Field-Programmable Gate Arrays (FPGAs) have been used to speed up the inference of LSTMs, but FPGA-based LSTM accelerators are limited by the size of on-chip memory and the bandwidth of external memory on FPGA boards. We propose a novel hardware architecture to overcome data dependency and a new blocking-batching strategy to reuse the LSTM weights fetched from external memory to optimize the performance of systems with size-limited on-chip memory for large machine learning models. Evaluation results show that our architecture can achieve 20.8 GOPS\/W, which is among the highest for the FPGA-based LSTM designs storing weights in off-chip memory. Our design achieves 1.65 times higher performance-per-watt efficiency and 2.48 times higher performance-per-DSP efficiency when compared with the current state-of-the-art designs of LSTM using weights stored in off-chip memory. Compared with CPU and GPU implementations, our FPGA implementation is 23.7 and 1.3 times faster while consuming 208 and 19.2 times lower energy respectively, which shows that our approach enables large LSTM systems to be processed efficiently on FPGAs with high performance and low power consumption.<\/jats:p>","DOI":"10.1007\/s11265-020-01549-8","type":"journal-article","created":{"date-parts":[[2020,7,9]],"date-time":"2020-07-09T12:05:05Z","timestamp":1594296305000},"page":"965-979","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Mapping Large LSTMs to FPGAs with Weight Reuse"],"prefix":"10.1007","volume":"92","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9263-6529","authenticated-orcid":false,"given":"Zhiqiang","family":"Que","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yongxin","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongxiang","family":"Fan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiuxi","family":"Meng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyu","family":"Niu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wayne","family":"Luk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,7,9]]},"reference":[{"key":"1549_CR1","doi-asserted-by":"crossref","unstructured":"Amin, H., & et al. (1997). Piecewise linear approximation applied to nonlinear function of a neural network. IEEE Proceedings-Circuits, Devices and Systems, 144.","DOI":"10.1049\/ip-cds:19971587"},{"key":"1549_CR2","doi-asserted-by":"crossref","unstructured":"Ardakani, A., Ji, Z., & Gross, W.J. (2018). Learning to skip ineffectual recurrent computations in LSTMs. arXiv:https:\/\/arxiv.org\/abs\/1811.10396.","DOI":"10.23919\/DATE.2019.8714765"},{"key":"1549_CR3","doi-asserted-by":"crossref","unstructured":"Cao, S., & et al. (2019). Efficient and effective sparse LSTM on FPGA with Bank-Balanced sparsity. In Proceedings of the 2019 ACM\/SIGDA International Symposium on Field-Programmable Gate Arrays. ACM.","DOI":"10.1145\/3289602.3293898"},{"key":"1549_CR4","unstructured":"Chang, A.X.M., Martini, B., & Culurciello, E. (2015). Recurrent neural networks hardware implementation on FPGA. arXiv:https:\/\/arxiv.org\/abs\/1511.05552."},{"key":"1549_CR5","unstructured":"Cooijmans, T., Ballas, N., Laurent, C., G\u00fcl\u00e7ehre, \u00c7., & Courville, A. (2016). Recurrent batch normalization. arXiv:https:\/\/arxiv.org\/abs\/1603.09025."},{"key":"1549_CR6","doi-asserted-by":"crossref","unstructured":"Donahue, J., & et al. (2015). Long-term Recurrent Convolutional Networks for Visual Recognition and Description. In Proceedings of the IEEE conference on computer vision and pattern recognition.","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"1549_CR7","doi-asserted-by":"crossref","unstructured":"Eyben, F., & et al. (2009). From speech to letters-using a novel neural network architecture for grapheme based ASR. In Automatic speech recognition & understanding, 2009. ASRU 2009. IEEE workshop.","DOI":"10.1109\/ASRU.2009.5373257"},{"key":"1549_CR8","doi-asserted-by":"crossref","unstructured":"Fan, H., & et al. (2018). A Real-Time Object Detection Accelerator with Compressed SSDLite on FPGA, International conference on field-programmable technology (FPT). IEEE.","DOI":"10.1109\/FPT.2018.00014"},{"key":"1549_CR9","doi-asserted-by":"crossref","unstructured":"Fan, H., & et al. (2018). Reconfigurable Acceleration of 3d-CNNs for Human Action Recognition with Block Floating-Point Representation. In 28th international conference on field programmable logic and applications (FPL). IEEE.","DOI":"10.1109\/FPL.2018.00056"},{"key":"1549_CR10","doi-asserted-by":"crossref","unstructured":"Fan, H., & et al. (2019). F-E3D: FPGA-based Acceleration of an Efficient 3D Convolutional Neural Network for Human Action Recognition. In IEEE 30Th international conference on application-specific systems, architectures and processors (ASAP). IEEE.","DOI":"10.1109\/ASAP.2019.00-44"},{"key":"1549_CR11","doi-asserted-by":"crossref","unstructured":"Ferreira, J.C., & et al. (2016). An FPGA implementation of a long short-term memory neural network. In Reconfigurable computing and FPGAs (reconfig). IEEE.","DOI":"10.1109\/ReConFig.2016.7857151"},{"key":"1549_CR12","doi-asserted-by":"crossref","unstructured":"Fowers, J., & et al. (2018). A configurable Cloud-Scale DNN processor for Real-Time AI. In Proceedings of the 45th Annual International Symposium on Computer Architecture.","DOI":"10.1109\/ISCA.2018.00012"},{"key":"1549_CR13","doi-asserted-by":"crossref","unstructured":"Gao, C., & et al. (2018). DeltaRNN: A power-efficient recurrent neural network accelerator. In Proceedings of the 2018 ACM\/SIGDA International Symposium on Field-Programmable Gate Arrays. ACM.","DOI":"10.1145\/3174243.3174261"},{"key":"1549_CR14","doi-asserted-by":"crossref","unstructured":"Guan, Y., & et al. (2017). FP-DNN: An Automated Framework for Mapping Deep Neural Networks onto FPGAs with RTL-HLS Hybrid Templates. In Field-programmable custom computing machines (FCCM). IEEE.","DOI":"10.1109\/FCCM.2017.25"},{"key":"1549_CR15","doi-asserted-by":"crossref","unstructured":"Guan, Y., & et al. (2017). FPGA-based Accelerator for Long Short-Term Memory Recurrent Neural Networks. In 22Nd asia and south pacific design automation conference (ASP-DAC). IEEE.","DOI":"10.1109\/ASPDAC.2017.7858394"},{"key":"1549_CR16","doi-asserted-by":"crossref","unstructured":"Han, S., & et al. (2017). ESE: Efficient speech recognition engine with sparse LSTM on FPGA. In Proceedings of the 2017 ACM\/SIGDA International Symposium on Field-Programmable Gate Arrays.","DOI":"10.1145\/3020078.3021745"},{"issue":"8","key":"1549_CR17","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term memory. Neural computation, 9 (8), 1735\u20131780.","journal-title":"Neural computation"},{"key":"1549_CR18","doi-asserted-by":"crossref","unstructured":"Jacob, B., & et al. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704\u20132713.","DOI":"10.1109\/CVPR.2018.00286"},{"key":"1549_CR19","unstructured":"Khurram Soomro, A.R.Z., & Shah, M. (2012). UCF101: A dataset of 101 human action classes from videos in the wild CRCV-TR-12-01."},{"key":"1549_CR20","doi-asserted-by":"crossref","unstructured":"Li, Z., & et al. (2019). E-RNN: Design optimization for efficient recurrent neural networks in FPGAs. In International symposium on high performance computer architecture (HPCA). IEEE.","DOI":"10.1109\/HPCA.2019.00028"},{"key":"1549_CR21","doi-asserted-by":"crossref","unstructured":"Nurvitadhi, E., & et al. (2016). Accelerating binarized neural networks: Comparison of FPGA, CPU, GPU, and ASIC. In International conference on field-programmable technology (FPT). IEEE.","DOI":"10.1109\/FPT.2016.7929192"},{"key":"1549_CR22","doi-asserted-by":"crossref","unstructured":"Nurvitadhi, E., & et al. (2016). Accelerating recurrent neural networks in analytics servers: Comparison of FPGA, CPU, GPU, and ASIC. In 26th international conference on field programmable logic and applications (FPL). IEEE.","DOI":"10.1109\/FPL.2016.7577314"},{"key":"1549_CR23","doi-asserted-by":"crossref","unstructured":"Nurvitadhi, E., & et al. (2017). Can FPGAs beat GPUs in accelerating next-generation deep neural networks?. In Proceedings of the 2017 ACM\/SIGDA International Symposium on Field-Programmable Gate Arrays ACM.","DOI":"10.1145\/3020078.3021740"},{"key":"1549_CR24","doi-asserted-by":"crossref","unstructured":"Nurvitadhi, E., & et al. (2019). Why Compete When You Can Work together: FPGA-ASIC Integration for Persistent RNNs. In 27th annual international symposium on field-programmable custom computing machines (FCCM). IEEE.","DOI":"10.1109\/FCCM.2019.00035"},{"key":"1549_CR25","doi-asserted-by":"crossref","unstructured":"Que, Z., & et al. (2019). Efficient Weight Reuse for Large LSTMs. In 30th international conference on application-specific systems, architectures and processors (ASAP). IEEE.","DOI":"10.1109\/ASAP.2019.00-42"},{"key":"1549_CR26","doi-asserted-by":"crossref","unstructured":"Que, Z., & et al. (2019). Real-time Anomaly Detection for Flight Testing using AutoEncoder and LSTM. In International conference on field-programmable technology (FPT). IEEE.","DOI":"10.1109\/ICFPT47387.2019.00072"},{"key":"1549_CR27","doi-asserted-by":"crossref","unstructured":"Que, Z., & et al. (2020). Optimizing reconfigurable recurrent neural networks. In Field-programmable custom computing machines (FCCM). IEEE.","DOI":"10.1109\/FCCM48280.2020.00011"},{"key":"1549_CR28","doi-asserted-by":"crossref","unstructured":"Rizakis, M., & et al. (2018). Approximate FPGA-based LSTMs under computation time constraints. arXiv:https:\/\/arxiv.org\/abs\/1801.02190.","DOI":"10.1007\/978-3-319-78890-6_1"},{"key":"1549_CR29","doi-asserted-by":"crossref","unstructured":"Rybalkin, V., Pappalardo, A., Ghaffar, M.M., Gambardella, G., Wehn, N., & Blott, M. (2018). FINN-L: Library extensions and design trade-off analysis for variable precision LSTM networks on FPGAs. In 28th international conference on field programmable logic and applications (FPL). IEEE.","DOI":"10.1109\/FPL.2018.00024"},{"key":"1549_CR30","doi-asserted-by":"crossref","unstructured":"Rybalkin, V., & et al. (2017). Hardware architecture of bidirectional long short-term memory neural network for optical character recognition. In Proceedings of the Conference on Design, Automation & Test in Europe.","DOI":"10.23919\/DATE.2017.7927210"},{"key":"1549_CR31","doi-asserted-by":"crossref","unstructured":"Sun, Z., & et al. (2018). FPGA Acceleration of LSTM based on data for test flight. In International conference on smart cloud (smartcloud). IEEE.","DOI":"10.1109\/SmartCloud.2018.00009"},{"key":"1549_CR32","doi-asserted-by":"crossref","unstructured":"Wang, S., & et al. (2018). C-LSTM: Enabling efficient LSTM using structured compression techniques on FPGAs. In Proceedings of the ACM\/SIGDA International Symposium on Field-Programmable Gate Arrays. ACM.","DOI":"10.1145\/3174243.3174253"},{"key":"1549_CR33","doi-asserted-by":"crossref","unstructured":"Xiong, P., & et al. (2018). Application of transfer learning in continuous time series for anomaly detection in commercial aircraft flight data. In International conference on smart cloud (smartcloud). IEEE.","DOI":"10.1109\/SmartCloud.2018.00011"},{"key":"1549_CR34","doi-asserted-by":"crossref","unstructured":"Yue-Hei Ng, J., & et al. (2015). Beyond short snippets: Deep networks for video classification. In Proceedings of the conference on computer vision and pattern recognition. IEEE.","DOI":"10.1109\/CVPR.2015.7299101"},{"key":"1549_CR35","unstructured":"Zhang, M., & et al. (2018). DeepCPU: Serving RNN-based deep learning models 10x faster. In: 2018 {USENIX} Annual technical conference ({USENIX}{ATC} 18)."},{"key":"1549_CR36","doi-asserted-by":"crossref","unstructured":"Zhang, X., & et al. (2017). High-performance video content recognition with long-term recurrent convolutional network for FPGA. In Field programmable logic and applications (FPL). IEEE.","DOI":"10.23919\/FPL.2017.8056833"},{"key":"1549_CR37","doi-asserted-by":"crossref","unstructured":"Zhao, R., & et al. (2017). Optimizing CNN-based object detection algorithms on embedded FPGA platforms. In International symposium on applied reconfigurable computing. Springer.","DOI":"10.1007\/978-3-319-56258-2_22"}],"container-title":["Journal of Signal Processing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11265-020-01549-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11265-020-01549-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11265-020-01549-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,8]],"date-time":"2021-07-08T23:40:25Z","timestamp":1625787625000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11265-020-01549-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,9]]},"references-count":37,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2020,9]]}},"alternative-id":["1549"],"URL":"https:\/\/doi.org\/10.1007\/s11265-020-01549-8","relation":{},"ISSN":["1939-8018","1939-8115"],"issn-type":[{"value":"1939-8018","type":"print"},{"value":"1939-8115","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,9]]},"assertion":[{"value":"30 November 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 March 2020","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 May 2020","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 July 2020","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}