{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T15:11:27Z","timestamp":1778771487368,"version":"3.51.4"},"reference-count":42,"publisher":"The Royal Society","issue":"2315","license":[{"start":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T00:00:00Z","timestamp":1772064000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T00:00:00Z","timestamp":1772064000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100022984","name":"Amazon","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100022984","id-type":"DOI","asserted-by":"publisher"}]},{"name":"SRC AIHW"},{"name":"SRC JUMP 2.0 COCOSYS"},{"name":"DARPA YFA"},{"DOI":"10.13039\/100017567","name":"Apple Inc.","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100017567","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004340","name":"Siemens","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004340","id-type":"DOI","asserted-by":"publisher"}]},{"name":"PORTAL Affiliates Program"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,2,26]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>As neural network models continue to grow in scale and complexity, specialized hardware accelerators have emerged to meet the increased demand for compute and memory. These accelerators employ a wide range of architectural innovations, making it challenging to perform fair comparisons and isolate the impact of specific design decisions. Traditional evaluation metrics, such as tera operations per second (TOPS) and TOPS per watt (TOPS\/W), are heavily influenced by external factors such as technology node, clock frequency, the scale of the design and workload variations, limiting their effectiveness for meaningful analysis. In this work, we propose a methodology for benchmarking neural network accelerators using Voyager, a high-level synthesis (HLS)-based accelerator generator. Voyager enables the creation of baseline accelerators matched in compute scale and technology node capable of running identical workloads. This enables fair, apples-to-apples comparisons across diverse accelerator architectures. We showcase this methodology with a range of case studies, including those on technology scaling and involving comparisons with state-of-the-art digital accelerators, in-memory computing-based accelerators, and sparsity-aware designs. Our results demonstrate that Voyager-generated designs serve as well-optimized baselines that enable systematic evaluation of accelerator architectures.<\/jats:p>\n                  <jats:p>This article is part of the discussion meeting issue \u2018Bits, neurons and qubits for sustainable AI\u2019.<\/jats:p>","DOI":"10.1098\/rsta.2024.0514","type":"journal-article","created":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T20:47:48Z","timestamp":1772311668000},"source":"Crossref","is-referenced-by-count":1,"title":["A methodology for accurate benchmarking of neural network accelerators using a high-level synthesis-based hardware generator"],"prefix":"10.1098","volume":"384","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4179-1692","authenticated-orcid":false,"given":"Kartik","family":"Prabhu","sequence":"first","affiliation":[{"name":"Stanford University 1 , , ,","place":["Stanford, CA, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeffrey","family":"Yu","sequence":"additional","affiliation":[{"name":"Stanford University 1 , , ,","place":["Stanford, CA, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xinyuan Allen","family":"Pan","sequence":"additional","affiliation":[{"name":"Stanford University 1 , , ,","place":["Stanford, CA, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8834-8663","authenticated-orcid":false,"given":"Priyanka","family":"Raina","sequence":"additional","affiliation":[{"name":"Stanford University 1 , , ,","place":["Stanford, CA, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"175","published-online":{"date-parts":[[2026,2,26]]},"reference":[{"key":"2026051410314761200_R1","doi-asserted-by":"crossref","unstructured":"He\n              K\n            , ZhangX, RenS, SunJ. 2016Deep residual learning for image recognition. In2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 770\u2013778. IEEE. (doi:10.1109\/CVPR.2016.90)","DOI":"10.1109\/CVPR.2016.90"},{"key":"2026051410314761200_R2","unstructured":"Bahdanau\n              D\n            , ChoK, BengioY. 2016Neural machine translation by jointly learning to align and translate. arxiv. Seehttps:\/\/arxiv.org\/abs\/1409.0473."},{"key":"2026051410314761200_R3","unstructured":"OpenAI\n          . 2024Sora: a video generation model. Seehttps:\/\/openai.com\/sora\/ (accessed31-May-2025)."},{"key":"2026051410314761200_R4","unstructured":"Bojarski\n              M\n            \n            et al\n          . 2016End to end learning for self-driving cars. arxiv. Seehttps:\/\/arxiv.org\/abs\/1604.07316."},{"key":"2026051410314761200_R5","unstructured":"Ronneberger\n              O\n            , FischerP, BroxT. 2015U-net: convolutional networks for biomedical image segmentation. arxiv. Seehttps:\/\/arxiv.org\/abs\/1505.04597."},{"key":"2026051410314761200_R6","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1162\/neco.1989.1.4.541","article-title":"Backpropagation applied to handwritten zip code recognition","volume":"1","author":"LeCun","year":"1989","journal-title":"Neural Comput."},{"key":"2026051410314761200_R7","first-page":"318","volume-title":"Parallel distributed processing: explorations in the microstructure of cognition: foundations","author":"Rumelhart","year":"1987"},{"key":"2026051410314761200_R8","unstructured":"Vaswani\n              A\n            , ShazeerN, ParmarN, UszkoreitJ, JonesL, GomezAN, KaiserL, PolosukhinI. 2023Attention is all you need. arxiv. Seehttps:\/\/arxiv.org\/abs\/1706.03762."},{"key":"2026051410314761200_R9","unstructured":"Gong\n              Y\n            , LiuL, YangM, BourdevL. 2014Compressing deep convolutional networks using vector quantization. arxiv. Seehttps:\/\/arxiv.org\/abs\/1412.6115."},{"key":"2026051410314761200_R10","unstructured":"Han\n              S\n            , MaoH, DallyWJ. 2016Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. arxiv. Seehttps:\/\/arxiv.org\/abs\/1510.00149."},{"key":"2026051410314761200_R11","unstructured":"Zhuang\n              B\n            , ShenC, TanM, LiuL, ReidI. 2017Towards effective low-bitwidth convolutional neural networks. arxiv. Seehttps:\/\/arxiv.org\/abs\/1711.00205."},{"key":"2026051410314761200_R12","unstructured":"Molchanov\n              P\n            , TyreeS, KarrasT, AilaT, KautzJ. 2017Pruning convolutional neural networks for resource efficient inference. arxiv. Seehttps:\/\/arxiv.org\/abs\/1611.06440."},{"key":"2026051410314761200_R13","unstructured":"Kung\n              HT\n            , LeisersonCE. 1979Systolic arrays (for VLSI). InSparse Matrix Proc. 1978, vol. 1, pp. 256\u2013282, Philadelphia, PA, USA: Society for Industrial and Applied Mathematics."},{"key":"2026051410314761200_R14","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1038\/nphys2566","article-title":"The parallel approach","volume":"9","author":"Di Ventra","year":"2013","journal-title":"Nat. Phys."},{"key":"2026051410314761200_R15","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1016\/j.mssp.2004.09.127","article-title":"Non-volatile memory technologies: emerging concepts and new materials","volume":"7","author":"Bez","year":"2004","journal-title":"Mater. Sci. Semicond. Process."},{"key":"2026051410314761200_R16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/JSSC.2025.3553765","article-title":"How to keep pushing ML accelerator performance? Know your rooflines!","volume":"60","author":"Verhelst","year":"2025","journal-title":"IEEE J. Solid-State Circuits"},{"key":"2026051410314761200_R17","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1145\/1498765.1498785","article-title":"Roofline: an insightful visual performance model for multicore architectures","volume":"52","author":"Williams","year":"2009","journal-title":"Commun. ACM"},{"key":"2026051410314761200_R18","doi-asserted-by":"crossref","unstructured":"Dong\n              P\n            \n            et al\n          . 2025A 28 nm 0.22 uJ\/token memory-compute-intensity-aware CNN-transformer accelerator with hybrid-attention-based layer-fusion and cascaded pruning for semantic-segmentation. In2025 IEEE Int. Solid-State Circuits Conf. (ISSCC), vol. 68, pp. 01\u201303, IEEE. (doi:10.1109\/ISSCC49661.2025.10904499)","DOI":"10.1109\/ISSCC49661.2025.10904499"},{"key":"2026051410314761200_R19","doi-asserted-by":"crossref","unstructured":"Moon\n              S\n            , LiM, ChenGK, KnagPC, KrishnamurthyRK, SeokM. 2025T-REX: a 68-to-567us\/token 0.41-to-3.95uJ\/token transformer accelerator with reduced external memory access and enhanced hardware utilization in 16 nm FinFET. In2025 IEEE Int. Solid-State Circuits Conf. (ISSCC), vol. 68, pp. 406\u2013408, IEEE. (doi:10.1109\/ISSCC49661.2025.10904793)","DOI":"10.1109\/ISSCC49661.2025.10904793"},{"key":"2026051410314761200_R20","doi-asserted-by":"crossref","unstructured":"Ji\n              Z\n            , WangH, WangM, KhwaWS, ChangMF, HanS, ChandrakasanAP. 2023A fully-integrated energy-scalable transformer accelerator supporting adaptive model configuration and word elimination for language understanding on edge devices. In2023 IEEE\/ACM Int. Symposium on Low Power Electronics and Design (ISLPED), pp. 1\u20136. IEEE. (doi:10.1109\/ISLPED58423.2023.10244459)","DOI":"10.1109\/ISLPED58423.2023.10244459"},{"key":"2026051410314761200_R21","doi-asserted-by":"crossref","unstructured":"Ignatov\n              A\n            , TimofteR, KulikA, YangS, WangK, BaumF, WuM, XuL, Van GoolL. 2019AI benchmark: all about deep learning on smartphones in 2019. In2019 IEEE\/CVF Int. Conf. on Computer Vision Workshop (ICCVW), pp. 3617\u20133635. IEEE. (doi:10.1109\/ICCVW.2019.00447)","DOI":"10.1109\/ICCVW.2019.00447"},{"key":"2026051410314761200_R22","unstructured":"Coleman\n              C\n            \n            et al\n          . 2017DAWNBench: an end-to-end deep learning benchmark and competition. InNeurIPS ML systems workshop. https:\/\/neurips.cc\/virtual\/2017\/9937."},{"key":"2026051410314761200_R23","unstructured":"Reddi\n              VJ\n            \n            et al\n          . 2020MLPerf inference benchmark. InProc. of the ACM\/IEEE 47th Annual Int. Symposium on Computer Architecture, ISCA \u201920, pp. 446\u2013459. IEEE Press. (doi:10.1109\/ISCA45697.2020.00045)"},{"key":"2026051410314761200_R24","unstructured":"Prabhu\n              K\n            , YuJ, PanXA, XieZ, AleshireA, ChenZ, RatnaniAA, RainaP. 2025Voyager: an end-to-end framework for design-space exploration and generation of DNN accelerators. arxiv. (doi:10.48550\/arXiv.2509.15205)"},{"key":"2026051410314761200_R25","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1109\/TC.2021.3059962","article-title":"ZigZag: enlarging joint architecture-mapping design space exploration for DNN accelerators","volume":"70","author":"Mei","year":"2021","journal-title":"IEEE Trans. Comput."},{"key":"2026051410314761200_R26","doi-asserted-by":"crossref","unstructured":"Parashar\n              A\n            \n            et al\n          . 2019Timeloop: a systematic approach to DNN accelerator evaluation. In2019 IEEE Int. Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 304\u2013315. IEEE. (doi:10.1109\/ISPASS.2019.00042)","DOI":"10.1109\/ISPASS.2019.00042"},{"key":"2026051410314761200_R27","doi-asserted-by":"crossref","unstructured":"Yang\n              X\n            \n            et al\n          . 2020Interstellar: using Halide\u2019s scheduling language to analyze DNN accelerators. InProc. of the Twenty-Fifth Int. Conf. on Architectural Support for Programming Languages and Operating Systems, ASPLOS \u201920, pp. 369\u2013383. New York, NY, USA: Association for Computing Machinery. (doi:10.1145\/3373376.3378514)","DOI":"10.1145\/3373376.3378514"},{"key":"2026051410314761200_R28","unstructured":"Sijstermans\n              F\n            \n          . 2018The NVIDIA deep learning accelerator. In2018 IEEE Hot Chips 30 Symposium (HCS), Cupertino, CA, USA, August 19-21, 2018. IEEE."},{"key":"2026051410314761200_R29","doi-asserted-by":"crossref","unstructured":"Venkatesan\n              R\n            \n            et al\n          . 2019MAGNet: a modular accelerator generator for neural networks. In2019 IEEE\/ACM Int. Conf. on Computer-Aided Design (ICCAD), pp. 1\u20138. IEEE. (doi:10.1109\/ICCAD45719.2019.8942127)","DOI":"10.1109\/ICCAD45719.2019.8942127"},{"key":"2026051410314761200_R30","doi-asserted-by":"crossref","unstructured":"Genc\n              H\n            \n            et al\n          . 2021Gemmini: enabling systematic deep-learning architecture evaluation via full-stack integration. InProc. of the 58th Annual Design Automation Conf. (DAC). IEEE. (doi:10.1109\/DAC18074.2021.9586216)","DOI":"10.1109\/DAC18074.2021.9586216"},{"key":"2026051410314761200_R31","unstructured":"Rouhani\n              BD\n            \n            et al\n          . 2023Microscaling data formats for deep learning. arxiv. Seehttps:\/\/arxiv.org\/abs\/2310.10537."},{"key":"2026051410314761200_R32","doi-asserted-by":"crossref","first-page":"478","DOI":"10.1109\/JPROC.2020.2981715","article-title":"A density metric for semiconductor technology","volume":"108","author":"Wong","year":"2020","journal-title":"Proc. IEEE"},{"key":"2026051410314761200_R33","doi-asserted-by":"crossref","unstructured":"Venkatesan\n              R\n            \n            et al\n          . 2019A 0.11 pJ\/op, 0.32-128 TOPS, scalable multi-chip-module-based deep neural network accelerator designed with a high-productivity VLSI methodology. In2019 IEEE Hot Chips 31 Symposium (HCS), Cupertino, CA, USA, August 18-20, 2019, pp. 1\u201324. IEEE. (doi:10.1109\/HOTCHIPS.2019.8875657)","DOI":"10.1109\/HOTCHIPS.2019.8875657"},{"key":"2026051410314761200_R34","doi-asserted-by":"crossref","unstructured":"Zimmer\n              B\n            \n            et al\n          . 2019A 0.11 pJ\/op, 0.32-128 TOPS, scalable multi-chip-module-based deep neural network accelerator with ground-reference signaling in 16 nm. In2019 Symposium on VLSI Circuits, pp. C300\u2013C301. IEEE. (doi:10.23919\/VLSIC.2019.8778056)","DOI":"10.23919\/VLSIC.2019.8778056"},{"key":"2026051410314761200_R35","doi-asserted-by":"crossref","first-page":"920","DOI":"10.1109\/JSSC.2019.2960488","article-title":"A 0.32\u2013128 TOPS, scalable multi-chip-module-based deep neural network inference accelerator with ground-referenced signaling in 16 nm","volume":"55","author":"Zimmer","year":"2020","journal-title":"IEEE J. Solid-State Circuits"},{"key":"2026051410314761200_R36","doi-asserted-by":"crossref","unstructured":"Keller\n              B\n            , VenkatesanR, DaiS, TellSG, ZimmerB, DallyWJ, Thomas GrayC, KhailanyB. 2022A 17\u201395.6 TOPS\/W deep learning inference accelerator with per-vector scaled 4-bit quantization for transformers in 5 nm. In2022 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits), pp. 16\u201317. IEEE. (doi:10.1109\/VLSITechnologyandCir46769.2022.9830277)","DOI":"10.1109\/VLSITechnologyandCir46769.2022.9830277"},{"key":"2026051410314761200_R37","doi-asserted-by":"crossref","first-page":"1129","DOI":"10.1109\/JSSC.2023.3234893","article-title":"A 95.6-TOPS\/W deep learning inference accelerator with per-vector scaled 4-bit quantization in 5 nm","volume":"58","author":"Keller","year":"2023","journal-title":"IEEE J. Solid-State Circuits"},{"key":"2026051410314761200_R38","doi-asserted-by":"crossref","unstructured":"Tang\n              C\n            , LinX, HuangZ, SunW, JiaH, LiuY. 2024A 28 nm 4.35 TOPS\/mm2 transformer accelerator with basis-vector based ultra storage compression, decomposed computation and unified LUT-assisted cores. In2024 IEEE Symposium on VLSI Technology and Circuits, pp. 1\u20132. IEEE. (doi:10.1109\/VLSITechnologyandCir46783.2024.10631311)","DOI":"10.1109\/VLSITechnologyandCir46783.2024.10631311"},{"key":"2026051410314761200_R39","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1109\/JSSC.2022.3214064","article-title":"DIANA: an end-to-end hybrid digital and analog neural network SoC for the edge","volume":"58","author":"Houshmand","year":"2023","journal-title":"IEEE J. Solid-State Circuits"},{"key":"2026051410314761200_R40","doi-asserted-by":"crossref","unstructured":"Zhang\n              JF\n            , LeeCE, LiuC, ShaoYS, KecklerSW, ZhangZ. 2019SNAP: a 1.67 \u2014 21.55 TOPS\/W sparse neural acceleration processor for unstructured sparse deep neural network inference in 16 nm CMOS. In2019 Symposium on VLSI Circuits, pp. C306\u2013C307. IEEE. (doi:10.23919\/VLSIC.2019.8778193)","DOI":"10.23919\/VLSIC.2019.8778193"},{"key":"2026051410314761200_R41","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1109\/JSSC.2020.3043870","article-title":"SNAP: an efficient sparse neural acceleration processor for unstructured sparse deep neural network inference","volume":"56","author":"Zhang","year":"2021","journal-title":"IEEE J. Solid-State Circuits"},{"key":"2026051410314761200_R42","doi-asserted-by":"crossref","unstructured":"Yu\n              H\n            , WenT, ChengG, SunJ, HanQ, ShiJ. 2020Low-bit quantization needs good distribution. In2020 IEEE\/CVF Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2909\u20132918. IEEE. (doi:10.1109\/CVPRW50498.2020.00348)","DOI":"10.1109\/CVPRW50498.2020.00348"}],"container-title":["Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/royalsocietypublishing.org\/rsta\/article-pdf\/doi\/10.1098\/rsta.2024.0514\/6117277\/rsta.2024.0514.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/royalsocietypublishing.org\/rsta\/article-pdf\/doi\/10.1098\/rsta.2024.0514\/6117277\/rsta.2024.0514.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T14:43:29Z","timestamp":1778769809000},"score":1,"resource":{"primary":{"URL":"https:\/\/royalsocietypublishing.org\/rsta\/article\/384\/2315\/20240514\/480524\/A-methodology-for-accurate-benchmarking-of-neural"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,26]]},"references-count":42,"journal-issue":{"issue":"2315","published-online":{"date-parts":[[2026,2,26]]},"published-print":{"date-parts":[[2026,2,26]]}},"URL":"https:\/\/doi.org\/10.1098\/rsta.2024.0514","relation":{},"ISSN":["1364-503X","1471-2962"],"issn-type":[{"value":"1364-503X","type":"print"},{"value":"1471-2962","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,26]]},"article-number":"20240514"}}