{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T08:00:46Z","timestamp":1772870446294,"version":"3.50.1"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,11,10]],"date-time":"2020-11-10T00:00:00Z","timestamp":1604966400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100004750","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1816833, 1719160, 1725447, 1730309"],"award-info":[{"award-number":["1816833, 1719160, 1725447, 1730309"]}],"id":[{"id":"10.13039\/501100004750","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2020,12,31]]},"abstract":"<jats:p>The tremendous impact of deep learning algorithms over a wide range of application domains has encouraged a surge of neural network (NN) accelerator research. Facilitating the NN accelerator design calls for guidance from an evolving benchmark suite that incorporates emerging NN models. Nevertheless, existing NN benchmarks are not suitable for guiding NN accelerator designs. These benchmarks are either selected for general-purpose processors without considering unique characteristics of NN accelerators or lack quantitative analysis to guarantee their completeness during the benchmark construction, update, and customization.<\/jats:p>\n          <jats:p>In light of the shortcomings of prior benchmarks, we propose a novel benchmarking methodology for NN accelerators with a quantitative analysis of application performance features and a comprehensive awareness of software-hardware co-design. Specifically, we decouple the benchmarking process into three stages: First, we characterize the NN workloads with quantitative metrics and select the representative applications for the benchmark suite to ensure diversity and completeness. Second, we refine the selected applications according to the customized model compression techniques provided by specific software-hardware co-design. Finally, we evaluate a variety of accelerator designs on the generated benchmark suite. To demonstrate the effectiveness of our benchmarking methodology, we conduct a case study of composing an NN benchmark from the TensorFlow Model Zoo and compress these selected models with various model compression techniques. Finally, we evaluate compressed models on various architectures, including GPU, Neurocube, DianNao, and Cambricon-X.<\/jats:p>","DOI":"10.1145\/3417709","type":"journal-article","created":{"date-parts":[[2020,11,10]],"date-time":"2020-11-10T23:16:11Z","timestamp":1605050171000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["NNBench-X"],"prefix":"10.1145","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7285-6682","authenticated-orcid":false,"given":"Xinfeng","family":"Xie","sequence":"first","affiliation":[{"name":"University of California, Santa Barbara, California, United States"}]},{"given":"Xing","family":"Hu","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara, California, United States"}]},{"given":"Peng","family":"Gu","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara, California, United States"}]},{"given":"Shuangchen","family":"Li","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara, California, United States"}]},{"given":"Yu","family":"Ji","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara, California, United States"}]},{"given":"Yuan","family":"Xie","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara, California, United States"}]}],"member":"320","published-online":{"date-parts":[[2020,11,10]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"MLPerf. 2018. MLPerf. Retrieved from https:\/\/mlperf.org\/.  MLPerf. 2018. MLPerf. Retrieved from https:\/\/mlperf.org\/."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI\u201918)","volume":"16","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , Manjunath Kudlur , Josh Levenberg , Rajat Monga , Sherry Moore , Derek G. Murray , Benoit Steiner , Paul Tucker , Vijay Vasudevan , Pete Warden , Martin Wicke , Yuan Yu , and Xiaoqiang Zheng . 2016 . TensorFlow: A system for large-scale machine learning . In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI\u201918) , Vol. 16 . 265--283. Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI\u201918), Vol. 16. 265--283."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/IISWC.2016.7581275"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001138"},{"key":"e_1_2_1_5_1","unstructured":"Amazon. 2020. Amazon EC2 F1 Instances. Retrieved from https:\/\/aws.amazon.com\/ec2\/instance-types\/f1\/.  Amazon. 2020. Amazon EC2 F1 Instances. Retrieved from https:\/\/aws.amazon.com\/ec2\/instance-types\/f1\/."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2019.2930057"},{"key":"e_1_2_1_7_1","unstructured":"Baidu. 2018. DeepBench. Retrieved from https:\/\/github.com\/baidu-research\/DeepBench.  Baidu. 2018. DeepBench. Retrieved from https:\/\/github.com\/baidu-research\/DeepBench."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems. 515--528","author":"Banerjee Subho S.","unstructured":"Subho S. Banerjee , Zbigniew T. Kalbarczyk , and Ravishankar K. Iyer . 2019. AcMC 2: Accelerating Markov chain Monte Carlo algorithms for probabilistic models . In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems. 515--528 . Subho S. Banerjee, Zbigniew T. Kalbarczyk, and Ravishankar K. Iyer. 2019. AcMC 2: Accelerating Markov chain Monte Carlo algorithms for probabilistic models. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems. 515--528."},{"key":"e_1_2_1_9_1","volume-title":"Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424","author":"Blundell Charles","year":"2015","unstructured":"Charles Blundell , Julien Cornebise , Koray Kavukcuoglu , and Daan Wierstra . 2015. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424 ( 2015 ). Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424 (2015)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TED.2015.2439635"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/IISWC.2012.6402898"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2541940.2541967"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MICRO.2014.58"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSSC.2016.2616357"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001140"},{"key":"e_1_2_1_16_1","unstructured":"PyTorch Core team. 2017. PyTorch. Retrieved from http:\/\/pytorch.org\/.  PyTorch Core team. 2017. PyTorch. Retrieved from http:\/\/pytorch.org\/."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/781131.781159"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 64--72","author":"Finn Chelsea","year":"2016","unstructured":"Chelsea Finn , Ian Goodfellow , and Sergey Levine . 2016 . Unsupervised learning for physical interaction through video prediction . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 64--72 . Chelsea Finn, Ian Goodfellow, and Sergey Levine. 2016. Unsupervised learning for physical interaction through video prediction. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 64--72."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA.2018.00012"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.81"},{"key":"e_1_2_1_21_1","unstructured":"Google. 2018. TensorFlow Models. Retrieved from https:\/\/github.com\/tensorflow\/models.  Google. 2018. TensorFlow Models. Retrieved from https:\/\/github.com\/tensorflow\/models."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA45697.2020.00071"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2017.39"},{"key":"e_1_2_1_24_1","volume-title":"Dally","author":"Han Song","year":"2016","unstructured":"Song Han , Xingyu Liu , Huizi Mao , Jing Pu , Ardavan Pedram , Mark A. Horowitz , and William J . Dally . 2016 . EIE : Efficient inference engine on compressed deep neural network. In Proceedings of the 43rd International Symposium on Computer Architecture. IEEE Press , 243--254. Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. 2016. EIE: Efficient inference engine on compressed deep neural network. In Proceedings of the 43rd International Symposium on Computer Architecture. IEEE Press, 243--254."},{"key":"e_1_2_1_25_1","volume-title":"Dally","author":"Han Song","year":"2015","unstructured":"Song Han , Huizi Mao , and William J . Dally . 2015 . Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149 (2015). Song Han, Huizi Mao, and William J. Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149 (2015)."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 1135--1143","author":"Han Song","year":"2015","unstructured":"Song Han , Jeff Pool , John Tran , and William Dally . 2015 . Learning both weights and connections for efficient neural network . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 1135--1143 . Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 1135--1143."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_28_1","volume-title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861","author":"Howard Andrew G.","year":"2017","unstructured":"Andrew G. Howard , Menglong Zhu , Bo Chen , Dmitry Kalenichenko , Weijun Wang , Tobias Weyand , Marco Andreetto , and Hartwig Adam . 2017 . Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017). Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)."},{"key":"e_1_2_1_29_1","volume-title":"SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and&lt","author":"Iandola Forrest N.","year":"2016","unstructured":"Forrest N. Iandola , Song Han , Matthew W. Moskewicz , Khalid Ashraf , William J. Dally , and Kurt Keutzer . 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and&lt ; 0.5 MB model size. arXiv preprint arXiv:1602.07360 ( 2016 ). Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and&lt; 0.5 MB model size. arXiv preprint arXiv:1602.07360 (2016)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304048"},{"key":"e_1_2_1_31_1","volume-title":"Joel Shor, and George Toderici.","author":"Johnston Nick","year":"2017","unstructured":"Nick Johnston , Damien Vincent , David Minnen , Michele Covell , Saurabh Singh , Troy Chinen , Sung Jin Hwang , Joel Shor, and George Toderici. 2017 . Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. arXiv preprint arXiv:1703.10114 (2017). Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, and George Toderici. 2017. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. arXiv preprint arXiv:1703.10114 (2017)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3079856.3080246"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001178"},{"key":"e_1_2_1_34_1","volume-title":"Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530","author":"Kim Yong-Deok","year":"2015","unstructured":"Yong-Deok Kim , Eunhyeok Park , Sungjoo Yoo , Taelim Choi , Lu Yang , and Dongjun Shin . 2015. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530 ( 2015 ). Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. 2015. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530 (2015)."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3294--3302","author":"Kiros Ryan","year":"2015","unstructured":"Ryan Kiros , Yukun Zhu , Ruslan R. Salakhutdinov , Richard Zemel , Raquel Urtasun , Antonio Torralba , and Sanja Fidler . 2015 . Skip-thought vectors . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3294--3302 . Ryan Kiros, Yukun Zhu, Ruslan R. Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3294--3302."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304028"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3352460.3358252"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2772--2782","author":"Nachum Ofir","year":"2017","unstructured":"Ofir Nachum , Mohammad Norouzi , Kelvin Xu , and Dale Schuurmans . 2017 . Bridging the gap between value and policy based reinforcement learning . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2772--2782 . Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. 2017. Bridging the gap between value and policy based reinforcement learning. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2772--2782."},{"key":"e_1_2_1_39_1","volume-title":"Bayesian Learning for Neural Networks","author":"Neal Radford M.","unstructured":"Radford M. Neal . 2012. Bayesian Learning for Neural Networks . Vol. 118 . Springer Science 8 Business Media. Radford M. Neal. 2012. Bayesian Learning for Neural Networks. Vol. 118. Springer Science 8 Business Media."},{"key":"e_1_2_1_40_1","unstructured":"Nvidia. 2017. cuDNN. Retrieved from https:\/\/developer.nvidia.com\/cudnn.  Nvidia. 2017. cuDNN. Retrieved from https:\/\/developer.nvidia.com\/cudnn."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2847263.2847265"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46493-0_32"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001165"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the ACM\/IEEE 47th International Symposium on Computer Architecture (ISCA\u201920)","author":"Reddi V. J.","unstructured":"V. J. Reddi , C. Cheng , D. Kanter , P. Mattson , G. Schmuelling , C. Wu , B. Anderson , M. Breughe , M. Charlebois , W. Chou , R. Chukka , C. Coleman , S. Davis , P. Deng , G. Diamos , J. Duke , D. Fick , J. S. Gardner , I. Hubara , S. Idgunji , T. B. Jablin , J. Jiao , T. S. John , P. Kanwar , D. Lee , J. Liao , A. Lokhmotov , F. Massa , P. Meng , P. Micikevicius , C. Osborne , G. Pekhimenko , A. T. R. Rajan , D. Sequeira , A. Sirasao , F. Sun , H. Tang , M. Thomson , F. Wei , E. Wu , L. Xu , K. Yamada , B. Yu , G. Yuan , A. Zhong , P. Zhang , and Y. Zhou . 2020. MLPerf inference benchmark . In Proceedings of the ACM\/IEEE 47th International Symposium on Computer Architecture (ISCA\u201920) . 446--459. V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, R. Chukka, C. Coleman, S. Davis, P. Deng, G. Diamos, J. Duke, D. Fick, J. S. Gardner, I. Hubara, S. Idgunji, T. B. Jablin, J. Jiao, T. S. John, P. Kanwar, D. Lee, J. Liao, A. Lokhmotov, F. Massa, P. Meng, P. Micikevicius, C. Osborne, G. Pekhimenko, A. T. R. Rajan, D. Sequeira, A. Sirasao, F. Sun, H. Tang, M. Thomson, F. Wei, E. Wu, L. Xu, K. Yamada, B. Yu, G. Yuan, A. Zhong, P. Zhang, and Y. Zhou. 2020. MLPerf inference benchmark. In Proceedings of the ACM\/IEEE 47th International Symposium on Computer Architecture (ISCA\u201920). 446--459."},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 91--99","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren , Kaiming He , Ross Girshick , and Jian Sun . 2015 . Faster R-CNN: Towards real-time object detection with region proposal networks . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 91--99 . Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 91--99."},{"key":"e_1_2_1_46_1","volume-title":"A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685","author":"Rush Alexander M.","year":"2015","unstructured":"Alexander M. Rush , Sumit Chopra , and Jason Weston . 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685 ( 2015 ). Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685 (2015)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.5555\/3195638.3195659"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCBD.2016.029"},{"key":"e_1_2_1_49_1","volume-title":"Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)."},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3104--3112","author":"Sutskever Ilya","unstructured":"Ilya Sutskever , Oriol Vinyals , and Quoc V. Le . 2014. Sequence to sequence learning with neural networks . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3104--3112 . Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3104--3112."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_2_1_52_1","volume-title":"BENCHIP: Benchmarking intelligence processors. arXiv preprint arXiv:1710.08315","author":"Tao Jin-Hua","year":"2017","unstructured":"Jin-Hua Tao , Zi-Dong Du , Qi Guo , Hui-Ying Lan , Lei Zhang , Sheng-Yuan Zhou , Cong Liu , Hai-Feng Liu , Shan Tang , Allen Rush , Willian Chen , Shao-Li Liu , Yun-Ji Chen , and Tian-Shi Chen . 2017 . BENCHIP: Benchmarking intelligence processors. arXiv preprint arXiv:1710.08315 (2017). Jin-Hua Tao, Zi-Dong Du, Qi Guo, Hui-Ying Lan, Lei Zhang, Sheng-Yuan Zhou, Cong Liu, Hai-Feng Liu, Shan Tang, Allen Rush, Willian Chen, Shao-Li Liu, Yun-Ji Chen, and Tian-Shi Chen. 2017. BENCHIP: Benchmarking intelligence processors. arXiv preprint arXiv:1710.08315 (2017)."},{"key":"e_1_2_1_53_1","volume-title":"David Minnen, Joel Shor, and Michele Covell.","author":"Toderici George","year":"2016","unstructured":"George Toderici , Damien Vincent , Nick Johnston , Sung Jin Hwang , David Minnen, Joel Shor, and Michele Covell. 2016 . Full resolution image compression with recurrent neural networks. arXiv preprint (2016). George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell. 2016. Full resolution image compression with recurrent neural networks. arXiv preprint (2016)."},{"key":"e_1_2_1_54_1","volume-title":"QGAN: Quantized generative adversarial networks. arXiv preprint arXiv:1901.08263","author":"Wang Peiqi","year":"2019","unstructured":"Peiqi Wang , Dongsheng Wang , Yu Ji , Xinfeng Xie , Haoxuan Song , XuXin Liu , Yongqiang Lyu , and Yuan Xie . 2019 . QGAN: Quantized generative adversarial networks. arXiv preprint arXiv:1901.08263 (2019). Peiqi Wang, Dongsheng Wang, Yu Ji, Xinfeng Xie, Haoxuan Song, XuXin Liu, Yongqiang Lyu, and Yuan Xie. 2019. QGAN: Quantized generative adversarial networks. arXiv preprint arXiv:1901.08263 (2019)."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 604--614","author":"Wang Peiqi","year":"2018","unstructured":"Peiqi Wang , Xinfeng Xie , Lei Deng , Guoqi Li , Dongsheng Wang , and Yuan Xie . 2018 . HitNet: Hybrid ternary recurrent neural network . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 604--614 . Peiqi Wang, Xinfeng Xie, Lei Deng, Guoqi Li, Dongsheng Wang, and Yuan Xie. 2018. HitNet: Hybrid ternary recurrent neural network. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 604--614."},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2074--2082","author":"Wen Wei","year":"2016","unstructured":"Wei Wen , Chunpeng Wu , Yandan Wang , Yiran Chen , and Hai Li . 2016 . Learning structured sparsity in deep neural networks . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2074--2082 . Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016. Learning structured sparsity in deep neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2074--2082."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498765.1498785"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3122788"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/LCA.2019.2898196"},{"key":"e_1_2_1_60_1","unstructured":"Xilinx. 2020. Xilinx xDNN Processing Engine. Retrieved from https:\/\/github.com\/Xilinx\/ml-suite.  Xilinx. 2020. Xilinx xDNN Processing Engine. Retrieved from https:\/\/github.com\/Xilinx\/ml-suite."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3079856.3080215"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2684746.2689060"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2017.2785257"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/MICRO.2016.7783723"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240765.3240801"}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3417709","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3417709","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:14Z","timestamp":1750197674000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3417709"}},"subtitle":["A Benchmarking Methodology for Neural Network Accelerator Designs"],"short-title":[],"issued":{"date-parts":[[2020,11,10]]},"references-count":65,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,12,31]]}},"alternative-id":["10.1145\/3417709"],"URL":"https:\/\/doi.org\/10.1145\/3417709","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"value":"1544-3566","type":"print"},{"value":"1544-3973","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,10]]},"assertion":[{"value":"2019-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-11-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}