{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,27]],"date-time":"2025-11-27T06:42:19Z","timestamp":1764225739991,"version":"3.41.0"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,11,9]],"date-time":"2021-11-09T00:00:00Z","timestamp":1636416000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Reconfigurable Technol. Syst."],"published-print":{"date-parts":[[2022,3,31]]},"abstract":"<jats:p>Multidimensional Long Short-Term Memory (MD-LSTM) neural network is an extension of one-dimensional LSTM for data with more than one dimension. MD-LSTM achieves state-of-the-art results in various applications, including handwritten text recognition, medical imaging, and many more. However, its implementation suffers from the inherently sequential execution that tremendously slows down both training and inference compared to other neural networks.<\/jats:p>\n          <jats:p>\n            The main goal of the current research is to provide acceleration for inference of MD-LSTM. We advocate that\n            <jats:bold>Field-Programmable Gate Array (FPGA)<\/jats:bold>\n            is an alternative platform for deep learning that can offer a solution when the massive parallelism of GPUs does not provide the necessary performance required by the application.\n          <\/jats:p>\n          <jats:p>\n            In this article, we present the first hardware architecture for MD-LSTM. We conduct a systematic exploration to analyze a tradeoff between precision and accuracy. We use a challenging dataset for semantic segmentation, namely historical document image binarization from the DIBCO 2017 contest and a well-known MNIST dataset for handwritten digit recognition. Based on our new architecture, we implement FPGA-based accelerators that outperform Nvidia Geforce RTX 2080 Ti with respect to throughput by up to 9.9\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:inline-graphic xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" content-type=\"gif\" xlink:href=\"3469661-inline1.gif\"\/>\n            <\/jats:inline-formula>\n            and Nvidia Jetson AGX Xavier with respect to energy efficiency by up to 48\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:inline-graphic xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" content-type=\"gif\" xlink:href=\"3469661-inline2.gif\"\/>\n            <\/jats:inline-formula>\n            . Our accelerators achieve higher throughput, energy efficiency, and resource efficiency than FPGA-based implementations of\n            <jats:bold>convolutional neural networks (CNNs)<\/jats:bold>\n            for semantic segmentation tasks. For the handwritten digit recognition task, our FPGA implementations provide higher accuracy and can be considered as a solution when accuracy is a priority. Furthermore, they outperform earlier FPGA implementations of one-dimensional LSTMs with respect to throughput, energy efficiency, and resource efficiency.\n          <\/jats:p>","DOI":"10.1145\/3469661","type":"journal-article","created":{"date-parts":[[2021,11,9]],"date-time":"2021-11-09T21:11:34Z","timestamp":1636492294000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["When Massive GPU Parallelism Ain\u2019t Enough: A Novel Hardware Architecture of 2D-LSTM Neural Network"],"prefix":"10.1145","volume":"15","author":[{"given":"Vladimir","family":"Rybalkin","sequence":"first","affiliation":[{"name":"Microelectronic Systems Design Research Group, University of Kaiserslautern, Kaiserslautern, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonas","family":"Ney","sequence":"additional","affiliation":[{"name":"Microelectronic Systems Design Research Group, University of Kaiserslautern, Kaiserslautern, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Menbere Kina","family":"Tekleyohannes","sequence":"additional","affiliation":[{"name":"Microelectronic Systems Design Research Group, University of Kaiserslautern, Kaiserslautern, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Norbert","family":"Wehn","sequence":"additional","affiliation":[{"name":"Microelectronic Systems Design Research Group, University of Kaiserslautern, Kaiserslautern, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,11,9]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2016.7472621"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.5555\/1776814.1776875"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICFHR.2016.0052"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10032-019-00325-0"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298977"},{"key":"e_1_3_2_8_2","first-page":"2998","volume-title":"Advances in Neural Information Processing Systems","author":"Stollenga Marijn F.","year":"2015","unstructured":"Marijn F. Stollenga, Wonmin Byeon, Marcus Liwicki, and Juergen Schmidhuber. 2015. Parallel multi-dimensional lstm, with application to fast biomedical volumetric image segmentation. In Advances in Neural Information Processing Systems. 2998\u20133006."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-26350-3"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2017.20"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2019.00064"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2017.228"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.660"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.3390\/rs11151774"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10032-019-00332-1"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3268344"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/FPL.2019.00037"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-17227-5_26"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/FPT.2018.00081"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2809544.2809561"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.18287\/2412-6179-2019-43-5-825-832"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASAP.2019.00-21"},{"key":"e_1_3_2_25_2","first-page":"3213","article-title":"The cityscapes dataset for semantic urban scene understanding","author":"Cordts Marius","year":"2016","unstructured":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201916). 3213\u20133223.","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201916)"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2008.04.005"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btu080"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijleo.2019.164025"},{"key":"e_1_3_2_29_2","article-title":"Deep high-resolution representation learning for visual recognition","author":"Wang Jingdong","year":"2020","unstructured":"Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et\u00a0al. 2020. Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. (2020).","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.106"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01219-9_25"},{"key":"e_1_3_2_33_2","unstructured":"Nal Kalchbrenner Ivo Danihelka and Alex Graves. 2015. Grid long short-term memory. arXiv:1507.01526. Retrieved from https:\/\/arxiv.org\/abs\/1507.01526."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.3007050"},{"key":"e_1_3_2_35_2","unstructured":"Aaron van den Oord Nal Kalchbrenner and Koray Kavukcuoglu. 2016. Pixel recurrent neural networks. In International Conference on Machine Learning . PMLR 1747\u20131756."},{"key":"e_1_3_2_36_2","first-page":"489","article-title":"ICFHR 2018 competition on handwritten document image binarization (H-DIBCO 2018)","author":"Pratikakis Ioannis","year":"2018","unstructured":"Ioannis Pratikakis, Konstantinos Zagoris, Panagiotis Kaddas, and Basilios Gatos. 2018. ICFHR 2018 competition on handwritten document image binarization (H-DIBCO 2018). In Proceedings of the 16th International Conference on Frontiers in Handwriting Recognition (ICFHR\u201918), 489\u2013493.","journal-title":"Proceedings of the 16th International Conference on Frontiers in Handwriting Recognition (ICFHR\u201918)"},{"key":"e_1_3_2_37_2","unstructured":"READ-COOP SCE: Revolutionizing Access to Handwritten Documents. Retrieved June 16 2021 from https:\/\/readcoop.eu. ([n. d.])."},{"key":"e_1_3_2_38_2","first-page":"509","volume-title":"Proceedings of the Asian Conference on Computer Vision","author":"Karpinski Romain","year":"2018","unstructured":"Romain Karpinski and Abdel Bela\u00efd. 2018. Combination of two fully convolutional neural networks for robust binarization. In Proceedings of the Asian Conference on Computer Vision. Springer, 509\u2013524."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00745"},{"key":"e_1_3_2_40_2","first-page":"1097","volume-title":"Advances in Neural Information Processing Systems","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097\u20131105."},{"key":"e_1_3_2_41_2","unstructured":"PG338\u2014Zynq DPU v3.1 IP Product Guide (v3.1). Retrieved June 16 2021 from https:\/\/www.xilinx.com\/support\/documentation\/ip_documentation\/dpu\/v3_1\/pg338-dpu.pdf."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3373087.3375301"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2016.7471828"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2017.7966166"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.09.046"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46493-0_32"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3020078.3021744"},{"key":"e_1_3_2_48_2","article-title":"Recurrent neural networks hardware implementation on FPGA","author":"Chang Andre Xian Ming","year":"2015","unstructured":"Andre Xian Ming Chang, Berin Martini, and Eugenio Culurciello. 2015. Recurrent neural networks hardware implementation on FPGA. arXiv:1511.05552. Retrieved from https:\/\/arxiv.org\/abs\/1511.05552.","journal-title":"arXiv:1511.05552"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ReConFig.2016.7857151"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/SiPS.2016.48"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASPDAC.2017.7858394"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCT50939.2020.9295665"},{"key":"e_1_3_2_53_2","first-page":"469","volume-title":"Proceedings of the IEEE 38th International Conference on Computer Design (ICCD\u201920)","author":"Zhang Xinyi","year":"2020","unstructured":"Xinyi Zhang, Weiwen Jiang, and Jingtong Hu. 2020. Achieving full parallelism in LSTM via a unified accelerator design. In Proceedings of the IEEE 38th International Conference on Computer Design (ICCD\u201920). IEEE, 469\u2013477."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/FCCM.2017.25"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11265-020-01549-8"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.5555\/3130379.3130707"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/FPL.2018.00024"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.23919\/FPL.2017.8056833"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3020078.3021745"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3174243.3174253"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISCAS45731.2020.9180869"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3289602.3293898"},{"key":"e_1_3_2_63_2","article-title":"BRDS: An FPGA-based LSTM accelerator with row-balanced dual-ratio sparsification","author":"Ghasemzadeh Seyed Abolfazl","year":"2021","unstructured":"Seyed Abolfazl Ghasemzadeh, Erfan Bank Tavakoli, Mehdi Kamal, Ali Afzali-Kusha, and Massoud Pedram. 2021. BRDS: An FPGA-based LSTM accelerator with row-balanced dual-ratio sparsification. arXiv:2101.02667. Retrieved from https:\/\/arxiv.org\/abs\/2101.02667.","journal-title":"arXiv:2101.02667"},{"key":"e_1_3_2_64_2","unstructured":"PyTorch\u2014An Open Source Machine Leraning Framework. Retrieved June 16 2021 from http:\/\/pytorch.org\/."},{"key":"e_1_3_2_65_2","unstructured":"Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv and Yoshua Bengio. 2017. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research 18 1 (2017) 6869\u20136898."},{"key":"e_1_3_2_66_2","article-title":"DoReFa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients","author":"Zhou Shuchang","year":"2016","unstructured":"Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. 2016. DoReFa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv:1606.06160. Retrieved from https:\/\/arxiv.org\/abs\/1606.06160.","journal-title":"arXiv:1606.06160"},{"key":"e_1_3_2_67_2","first-page":"249","volume-title":"Proceedings of the 13th International Conference on Artificial Intelligence and Statistics","author":"Glorot Xavier","year":"2010","unstructured":"Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. 249\u2013256."},{"key":"e_1_3_2_68_2","unstructured":"DMA\/Bridge Subsystem for PCI Express v4.1. Retrieved June 16 2021 from https:\/\/www.xilinx.com\/support\/documentation\/ip_documentation\/xdma\/v4_1\/pg195-pcie-dma.pdf."},{"key":"e_1_3_2_69_2","unstructured":"UltraScale Architecture-Based FPGAs Memory IP v1.4. Retrieved June 16 2021 from https:\/\/www.xilinx.com\/support\/documentation\/ip_documentation\/ultrascale_memory_ip\/v1_4\/pg150-ultrascale-memory-ip.pdf."},{"key":"e_1_3_2_70_2","unstructured":"Zynq UltraScale+ MPSoC Power Advantage Tool. Retrieved June 16 2021 from https:\/\/xilinx-wiki.atlassian.net\/wiki\/spaces\/A\/pages\/18841813\/Zynq+UltraScale+MPSoC+Power+Management."},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3206098.3206111"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2017.58"},{"key":"e_1_3_2_73_2","first-page":"139","volume-title":"Robust Binarization of Stereo and Monocular Document Images Using Percentile Filter","author":"Afzal M. Z.","year":"2014","unstructured":"M. Z. Afzal, M. Kramer, Syed Saqib Bukhari, M. R. Yousefi, Faisal Shafait, and T. M. Breuel. 2014. Robust Binarization of Stereo and Monocular Document Images Using Percentile Filter. Vol. 1. Springer, 139\u2013149. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-319-05167-3_11"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209280.3209538"},{"key":"e_1_3_2_75_2","unstructured":"Narrenschif. Retrieved June 16 2021 from http:\/\/kallimachos.de\/kallimachos\/index.php\/Narragonien."},{"key":"e_1_3_2_76_2","unstructured":"Digital Multimeter Vilcraft. Retrieved June 16 2021 from http:\/\/www.produktinfo.conrad.com\/datenblaetter\/100000-124999\/124608-an-01-ml-VOLTCRAFT_VC_870_DMM__K__de_en_fr_nl.pdf."},{"key":"e_1_3_2_77_2","article-title":"The implementation of a deep recurrent neural network language model on a Xilinx FPGA","author":"Hao Yufeng","year":"2017","unstructured":"Yufeng Hao and Steven Quigley. 2017. The implementation of a deep recurrent neural network language model on a Xilinx FPGA. arXiv:1710.10296. Retrieved from https:\/\/arxiv.org\/abs\/1710.10296.","journal-title":"arXiv:1710.10296"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240302.3240322"}],"container-title":["ACM Transactions on Reconfigurable Technology and Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3469661","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3469661","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:29Z","timestamp":1750191449000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3469661"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,9]]},"references-count":77,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,3,31]]}},"alternative-id":["10.1145\/3469661"],"URL":"https:\/\/doi.org\/10.1145\/3469661","relation":{},"ISSN":["1936-7406","1936-7414"],"issn-type":[{"type":"print","value":"1936-7406"},{"type":"electronic","value":"1936-7414"}],"subject":[],"published":{"date-parts":[[2021,11,9]]},"assertion":[{"value":"2020-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}