{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T02:08:52Z","timestamp":1769738932714,"version":"3.49.0"},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,7,18]],"date-time":"2023-07-18T00:00:00Z","timestamp":1689638400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2022YFB4500405"],"award-info":[{"award-number":["2022YFB4500405"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62174162"],"award-info":[{"award-number":["62174162"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Singapore Government\u2019s Research, Innovation and Enterprise 2020 Plan","award":["A1687b0033"],"award-info":[{"award-number":["A1687b0033"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Des. Autom. Electron. Syst."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>Deformable convolution networks (DCNs) proposed to address image recognition with geometric or photometric variations typically involve deformable convolution that convolves on arbitrary locations of input features. The locations change with different inputs and induce considerable dynamic and irregular memory accesses that cannot be handled by classic neural network accelerators (NNAs). Moreover, bilinear interpolation (BLI) operation, which is required to obtain deformed features in DCNs, also cannot be deployed on existing NNAs directly. Although a general purposed processor (GPP) seated along with classic NNAs can process the deformable convolution, the processing on GPP can be extremely slow due to the limited parallel computing capability and massive additional data movement. To address the problem, we develop a DCN accelerator on existing NNAs to support both the standard convolution and deformable convolution. Specifically, for the dynamic and irregular accesses in DCNs, we have both the input and output features divided into tiles and build a tile dependency table (TDT) to track the irregular tile dependency at runtime. With the TDT, we further develop an on-chip tile scheduler to handle the dynamic and irregular accesses efficiently. In addition, we propose a novel mapping strategy to enable parallel BLI processing on NNAs and apply layer fusion techniques for more energy-efficient DCN processing. According to our experiments, the proposed accelerator achieves orders of magnitude higher performance and energy efficiency compared to the typical computing architectures including ARM, ARM+TPU, and GPU with 6.6% chip area penalty to a classic NNA.<\/jats:p>","DOI":"10.1145\/3597431","type":"journal-article","created":{"date-parts":[[2023,5,15]],"date-time":"2023-05-15T12:06:04Z","timestamp":1684152364000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Accelerating Deformable Convolution Networks with Dynamic and Irregular Memory Accesses"],"prefix":"10.1145","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3226-0750","authenticated-orcid":false,"given":"Cheng","family":"Chu","sequence":"first","affiliation":[{"name":"Indiana University Bloomington, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5542-7306","authenticated-orcid":false,"given":"Cheng","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4204-0898","authenticated-orcid":false,"given":"Dawen","family":"Xu","sequence":"additional","affiliation":[{"name":"Hefei University of Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5172-4736","authenticated-orcid":false,"given":"Ying","family":"Wang","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3415-3676","authenticated-orcid":false,"given":"Tao","family":"Luo","sequence":"additional","affiliation":[{"name":"Institute of High Performance Computing, A*STAR, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8082-4218","authenticated-orcid":false,"given":"Huawei","family":"Li","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0874-814X","authenticated-orcid":false,"given":"Xiaowei","family":"Li","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,7,18]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.89"},{"key":"e_1_3_2_3_2","first-page":"326","volume-title":"Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS\u201919)","author":"Cao Zeyu","year":"2019","unstructured":"Zeyu Cao, Xiaorun Li, and Liaoying Zhao. 2019. Object detection in VHR image using transfer learning with deformable convolution. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS\u201919). IEEE, 326\u2013329."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00968"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/CCHI.2019.8901912"},{"key":"e_1_3_2_6_2","article-title":"Restricted deformable convolution-based road scene semantic segmentation using surround view cameras","author":"Deng Liuyuan","year":"2019","unstructured":"Liuyuan Deng, Ming Yang, Hao Li, Tianyi Li, Bing Hu, and Chunxiang Wang. 2019. Restricted deformable convolution-based road scene semantic segmentation using surround view cameras. IEEE Trans. Intell. Transport. Syst. (2019).","journal-title":"IEEE Trans. Intell. Transport. Syst."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00902"},{"key":"e_1_3_2_9_2","first-page":"521","volume-title":"Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS\u201920)","author":"Diao Yingyu","year":"2020","unstructured":"Yingyu Diao, Jingzhou Chen, and Yuntao Qian. 2020. Multi-label remote sensing image classification with deformable convolutions and graph neural networks. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS\u201920). IEEE, 521\u2013524."},{"key":"e_1_3_2_10_2","unstructured":"Natasha Goenawan Mandy Lu. 2018. DeformSketchNet: Deformable Convolutional Networks for Sketch Classification. Retrieved from http:\/\/cs230.stanford.edu\/files_winter_2018\/projects\/6940505.pdf."},{"key":"e_1_3_2_11_2","first-page":"117660R","volume-title":"Proceedings of the International Workshop on Advanced Imaging Technology (IWAIT\u201921) 2021","author":"Lai Siew Cheng","year":"2021","unstructured":"Siew Cheng Lai, Hung Khoon Tan, and Phooi Yee Lau. 2021. 3D deformable convolution for action classification in videos. In Proceedings of the International Workshop on Advanced Imaging Technology (IWAIT\u201921) 2021. International Society for Optics and Photonics, 117660R."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01231-1_33"},{"key":"e_1_3_2_13_2","first-page":"136","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV\u201918)","author":"Weng Junwu","year":"2018","unstructured":"Junwu Weng, Mengyuan Liu, Xudong Jiang, and Junsong Yuan. 2018. Deformable pose traversal convolution for 3D action and gesture recognition. In Proceedings of the European Conference on Computer Vision (ECCV\u201918). 136\u2013152."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01258-8_21"},{"key":"e_1_3_2_15_2","unstructured":"Khoi-Nguyen C. Mac Dhiraj Joshi Raymond A. Yeh Jinjun Xiong Rogerio R. Feris and Minh N. Do. 2018. Locally-consistent deformable convolution networks for fine-grained action detection. Retrieved from https:\/\/arXiv:1811.08815."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN48605.2020.9206944"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2019.00278"},{"key":"e_1_3_2_18_2","first-page":"48","volume-title":"Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS\u201919)","author":"Huang Qijing","year":"2019","unstructured":"Qijing Huang, Dequan Wang, Yizhao Gao, Yaohui Cai, Zhen Dong, Bichen Wu, Kurt Keutzer, and John Wawrzynek. 2019. Algorithm-hardware co-design for deformable convolution. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS\u201919). IEEE, 48\u201351."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP40778.2020.9191028"},{"key":"e_1_3_2_20_2","first-page":"367","volume-title":"Proceedings of the ACM\/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA\u201916)","author":"Chen Yu-Hsin","year":"2016","unstructured":"Yu-Hsin Chen, Joel Emer, and Vivienne Sze. 2016. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. In Proceedings of the ACM\/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA\u201916). IEEE, 367\u2013379."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001138"},{"key":"e_1_3_2_22_2","first-page":"132","volume-title":"Proceedings of the IEEE 13th International Symposium on Biomedical Imaging (ISBI\u201916)","author":"Bromberger Michael","year":"2016","unstructured":"Michael Bromberger, Pascal Bastian, Jan-Philip Bergeest, Christian Conrad, Vincent Heuveline, Karl Rohr, and Wolfgang Karl. 2016. FPGA-accelerated Richardson-Lucy deconvolution for 3D image data. In Proceedings of the IEEE 13th International Symposium on Biomedical Imaging (ISBI\u201916). IEEE, 132\u2013135."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/2897937.2898003"},{"key":"e_1_3_2_24_2","article-title":"Caffeine: Towards uniformed representation and acceleration for deep convolutional neural networks","author":"Zhang Chen","year":"2018","unstructured":"Chen Zhang, Guangyu Sun, Zhenman Fang, Peipei Zhou, Peichen Pan, and Jason Cong. 2018. Caffeine: Towards uniformed representation and acceleration for deep convolutional neural networks. IEEE Trans. Comput.-Aided Design Integr. Circ. Syst. (2018).","journal-title":"IEEE Trans. Comput.-Aided Design Integr. Circ. Syst."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2021.3124763"},{"key":"e_1_3_2_26_2","first-page":"478","volume-title":"Proceedings of the IEEE 38th International Conference on Computer Design (ICCD\u201920)","author":"Xu Dawen","year":"2020","unstructured":"Dawen Xu, Cheng Chu, Qianlong Wang, Cheng Liu, Ying Wang, Lei Zhang, Huaguo Liang, and Kwang-Ting Cheng. 2020. A hybrid computing architecture for fault-tolerant deep learning accelerators. In Proceedings of the IEEE 38th International Conference on Computer Design (ICCD\u201920). IEEE, 478\u2013485."},{"key":"e_1_3_2_27_2","first-page":"22","volume-title":"Proceedings of the International Conference on Computer-Aided Design","author":"Xu Dawen","year":"2018","unstructured":"Dawen Xu, Kaijie Tu, Ying Wang, Cheng Liu, Bingsheng He, and Huawei Li. 2018. FCN-engine: Accelerating deconvolutional layers in classic CNN processors. In Proceedings of the International Conference on Computer-Aided Design. ACM, 22."},{"key":"e_1_3_2_28_2","first-page":"650","volume-title":"Proceedings of the 45th Annual International Symposium on Computer Architecture","author":"Yazdanbakhsh Amir","year":"2018","unstructured":"Amir Yazdanbakhsh, Kambiz Samadi, Nam Sung Kim, and Hadi Esmaeilzadeh. 2018. Ganax: A unified MIMD-SIMD acceleration for generative adversarial networks. In Proceedings of the 45th Annual International Symposium on Computer Architecture. IEEE Press, 650\u2013661."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2018.2857258"},{"key":"e_1_3_2_30_2","unstructured":"Xinyu Zhang Srinjoy Das Ojash Neopane and Ken Kreutz-Delgado. 2017. A design methodology for efficient implementation of deconvolutional neural networks on an FPGA. Retrieved from https:\/\/arXiv:1705.02583."},{"key":"e_1_3_2_31_2","first-page":"933","volume-title":"Proceedings of the 51st Annual IEEE\/ACM International Symposium on Microarchitecture (MICRO\u201918)","author":"Hegde Kartik","year":"2018","unstructured":"Kartik Hegde, Rohit Agrawal, Yulun Yao, and Christopher W. Fletcher. 2018. Morph: Flexible acceleration for 3D CNN-based video understanding. In Proceedings of the 51st Annual IEEE\/ACM International Symposium on Microarchitecture (MICRO\u201918). IEEE, 933\u2013946."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2017.2699229"},{"key":"e_1_3_2_33_2","first-page":"287","volume-title":"Proceedings of the 28th International Conference on Field Programmable Logic and Applications (FPL\u201918)","author":"Fan Hongxiang","year":"2018","unstructured":"Hongxiang Fan, Ho-Cheung Ng, Shuanglong Liu, Zhiqiang Que, Xinyu Niu, and Wayne Luk. 2018. Reconfigurable acceleration of 3D-CNNs for human action recognition with block floating-point representation. In Proceedings of the 28th International Conference on Field Programmable Logic and Applications (FPL\u201918). IEEE, 287\u20132877."},{"key":"e_1_3_2_34_2","volume-title":"Proceedings of the Great Lakes Symposium on VLSI (GLSVLSI\u201921)","author":"Chu C.","year":"2021","unstructured":"C. Chu, Fan Chen, D. Xu, and Y. Wang. 2021. RECOIN: A low-power processing-in-ReRAM architecture for deformable convolution. In Proceedings of the Great Lakes Symposium on VLSI (GLSVLSI\u201921)."},{"key":"e_1_3_2_35_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Esser Steven K.","year":"2019","unstructured":"Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. 2019. Learned Step Size Quantization. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_36_2","unstructured":"Python Cache Hierarchy Simulator. (n.d.). Retrieved from https:\/\/github.com\/RRZE-HPC\/pycachesim."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.5555\/3195638.3195664"},{"key":"e_1_3_2_38_2","first-page":"140","volume-title":"Proceedings of the IEEE Workshop on Signal Processing Systems (SiPS\u201921)","author":"Yu Yue","year":"2021","unstructured":"Yue Yu, Jiapeng Luo, Wendong Mao, and Zhongfeng Wang. 2021. A memory-efficient hardware architecture for deformable convolutional networks. In Proceedings of the IEEE Workshop on Signal Processing Systems (SiPS\u201921). IEEE, 140\u2013145."},{"key":"e_1_3_2_39_2","first-page":"1","volume-title":"Proceedings of the IEEE Workshop on Signal Processing Systems (SiPS\u201922)","author":"Meng Yuan","year":"2022","unstructured":"Yuan Meng, Hongjiang Men, and Viktor Prasanna. 2022. Accelerator design and exploration for deformable convolution networks. In Proceedings of the IEEE Workshop on Signal Processing Systems (SiPS\u201922). IEEE, 1\u20136."},{"key":"e_1_3_2_40_2","unstructured":"Ananda Samajdar Yuhao Zhu Paul Whatmough Matthew Mattina and Tushar Krishna. 2018. SCALE-Sim: Systolic CNN accelerator simulator. Retrieved from https:\/\/arXiv:1811.02883."},{"key":"e_1_3_2_41_2","unstructured":"Retrieved from https:\/\/www.micron.com\/support\/tools-and-utilities\/power-calc."},{"key":"e_1_3_2_42_2","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from https:\/\/arXiv:1409.1556."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2016.2644615"}],"container-title":["ACM Transactions on Design Automation of Electronic Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597431","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597431","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:44Z","timestamp":1750182524000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597431"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,18]]},"references-count":42,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3597431"],"URL":"https:\/\/doi.org\/10.1145\/3597431","relation":{},"ISSN":["1084-4309","1557-7309"],"issn-type":[{"value":"1084-4309","type":"print"},{"value":"1557-7309","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,18]]},"assertion":[{"value":"2022-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-05-04","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}