{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T20:37:44Z","timestamp":1790714264404,"version":"4.1.0"},"publisher-location":"New York, NY, USA","reference-count":51,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,6,3]],"date-time":"2021-06-03T00:00:00Z","timestamp":1622678400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Swiss National Science Foundation","award":["185778"],"award-info":[{"award-number":["185778"]}]},{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["678880,955606"],"award-info":[{"award-number":["678880,955606"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,6,3]]},"DOI":"10.1145\/3447818.3460360","type":"proceedings-article","created":{"date-parts":[[2021,6,4]],"date-time":"2021-06-04T15:09:36Z","timestamp":1622819376000},"page":"63-74","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":30,"title":["NPBench"],"prefix":"10.1145","author":[{"given":"Alexandros Nikolaos","family":"Ziogas","sequence":"first","affiliation":[{"name":"ETH Zurich"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tal","family":"Ben-Nun","sequence":"additional","affiliation":[{"name":"ETH Zurich"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Timo","family":"Schneider","sequence":"additional","affiliation":[{"name":"ETH Zurich"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Torsten","family":"Hoefler","sequence":"additional","affiliation":[{"name":"ETH Zurich"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,6,4]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"The SciPy community. [n.d.]. Ndarray Indexing. https:\/\/numpy.org\/devdocs\/reference\/arrays.indexing.html Retrieved 2021-02-04 from  The SciPy community. [n.d.]. Ndarray Indexing. https:\/\/numpy.org\/devdocs\/reference\/arrays.indexing.html Retrieved 2021-02-04 from"},{"key":"e_1_3_2_1_2_1","unstructured":"Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dan Man\u00e9 Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. http:\/\/tensorflow.org\/ Software available from tensorflow.org.  Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dan Man\u00e9 Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. http:\/\/tensorflow.org\/ Software available from tensorflow.org."},{"key":"e_1_3_2_1_3_1","unstructured":"Anaconda Inc. [n.d.]a. A 5 minute guide to Numba. https:\/\/numba.readthedocs.io\/en\/stable\/user\/5minguide.html Retrieved 2021-02-04 from  Anaconda Inc. [n.d.]a. A 5 minute guide to Numba. https:\/\/numba.readthedocs.io\/en\/stable\/user\/5minguide.html Retrieved 2021-02-04 from"},{"key":"e_1_3_2_1_4_1","unstructured":"Anaconda Inc. [n.d.]b. airspeed velocity of an unladen numba. http:\/\/numba.pydata.org\/numba-benchmark\/ Retrieved 2021-01-28 from  Anaconda Inc. [n.d.]b. airspeed velocity of an unladen numba. http:\/\/numba.pydata.org\/numba-benchmark\/ Retrieved 2021-01-28 from"},{"key":"e_1_3_2_1_5_1","unstructured":"Anaconda Inc. [n.d.]c. Example: Histogram. https:\/\/numba.pydata.org\/numba-examples\/examples\/density_estimation\/histogram\/results.html Retrieved 2021-02-04 from  Anaconda Inc. [n.d.]c. Example: Histogram. https:\/\/numba.pydata.org\/numba-examples\/examples\/density_estimation\/histogram\/results.html Retrieved 2021-02-04 from"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1562764.1562783"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-09766-4_133"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1175\/MWR-D-10-05013.1"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.21105\/jose.00021"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2010.118"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3295500.3356173"},{"key":"e_1_3_2_1_13_1","volume-title":"cuDNN: Efficient Primitives for Deep Learning. CoRR abs\/1410.0759","author":"Chetlur Sharan","year":"2014","unstructured":"Sharan Chetlur , Cliff Woolley , Philippe Vandermersch , Jonathan Cohen , John Tran , Bryan Catanzaro , and Evan Shelhamer . 2014. cuDNN: Efficient Primitives for Deep Learning. CoRR abs\/1410.0759 ( 2014 ). [arxiv]1410.0759 http:\/\/arxiv.org\/abs\/1410.0759 Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cuDNN: Efficient Primitives for Deep Learning. CoRR abs\/1410.0759 (2014). [arxiv]1410.0759 http:\/\/arxiv.org\/abs\/1410.0759"},{"key":"e_1_3_2_1_14_1","unstructured":"COSMO. 1998. Consortium for Small-scale Modeling. http:\/\/www.cosmo-model.org Retrieved 2021-02-04 from  COSMO. 1998. Consortium for Small-scale Modeling. http:\/\/www.cosmo-model.org Retrieved 2021-02-04 from"},{"key":"e_1_3_2_1_15_1","unstructured":"Cython. [n.d.]. Cython Demos\/benchmarks. https:\/\/github.com\/cython\/cython\/tree\/master\/Demos\/benchmarks Retrieved 2021-01-28 from  Cython. [n.d.]. Cython Demos\/benchmarks. https:\/\/github.com\/cython\/cython\/tree\/master\/Demos\/benchmarks Retrieved 2021-01-28 from"},{"key":"e_1_3_2_1_16_1","volume-title":"Dask: Library for dynamic task scheduling. https:\/\/dask.org","author":"Team Dask Development","year":"2016","unstructured":"Dask Development Team . 2016 . Dask: Library for dynamic task scheduling. https:\/\/dask.org Dask Development Team. 2016. Dask: Library for dynamic task scheduling. https:\/\/dask.org"},{"key":"e_1_3_2_1_17_1","volume-title":"Breakthroughs in statistics","author":"Efron Bradley","unstructured":"Bradley Efron . 1992. Bootstrap methods: another look at the jackknife . In Breakthroughs in statistics . Springer , 569--593. Bradley Efron. 1992. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics. Springer, 569--593."},{"key":"e_1_3_2_1_18_1","unstructured":"Python Software Foundation. [n.d.]. PEP 8 -- Style Guide for Python Code. https:\/\/www.python.org\/dev\/peps\/pep-0008  Python Software Foundation. [n.d.]. PEP 8 -- Style Guide for Python Code. https:\/\/www.python.org\/dev\/peps\/pep-0008"},{"key":"e_1_3_2_1_19_1","unstructured":"GitHub. 2020. The 2020 State of the Octoverse. https:\/\/octoverse.github.com\/  GitHub. 2020. The 2020 State of the Octoverse. https:\/\/octoverse.github.com\/"},{"key":"e_1_3_2_1_20_1","unstructured":"Serge Guelton. [n.d.]. Numpy Benchmarks. https:\/\/github.com\/serge-sans-paille\/numpy-benchmarks Retrieved 2021-01-28 from  Serge Guelton. [n.d.]. Numpy Benchmarks. https:\/\/github.com\/serge-sans-paille\/numpy-benchmarks Retrieved 2021-01-28 from"},{"key":"e_1_3_2_1_21_1","unstructured":"Serge Guelton. [n.d.]. Pythran. https:\/\/github.com\/serge-sans-paille\/pythran.  Serge Guelton. [n.d.]. Pythran. https:\/\/github.com\/serge-sans-paille\/pythran."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1088\/1749-4680\/8\/1\/014001"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2649-2"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807591.2807644"},{"key":"e_1_3_2_1_27_1","unstructured":"Intel Corporation. [n.d.]. oneAPI Deep Neural Network Library (oneDNN). https:\/\/github.com\/oneapi-src\/oneDNN Retrieved 2021-02-01 from  Intel Corporation. [n.d.]. oneAPI Deep Neural Network Library (oneDNN). https:\/\/github.com\/oneapi-src\/oneDNN Retrieved 2021-02-01 from"},{"key":"e_1_3_2_1_28_1","volume-title":"Proceedings of the 7th European Conference on Python in Science (EuroSciPy","author":"Kieffer J\u00e9r\u00f4me","year":"2014","unstructured":"J\u00e9r\u00f4me Kieffer and Giannis Ashiotis . 2014 . PyFAI: a Python library for high performance azimuthal integration on GPU . In Proceedings of the 7th European Conference on Python in Science (EuroSciPy 2014). [arxiv]1412.6367 [astro-ph.IM] J\u00e9r\u00f4me Kieffer and Giannis Ashiotis. 2014. PyFAI: a Python library for high performance azimuthal integration on GPU. In Proceedings of the 7th European Conference on Python in Science (EuroSciPy 2014). [arxiv]1412.6367 [astro-ph.IM]"},{"key":"e_1_3_2_1_29_1","volume-title":"Positioning and Power in Academic Publishing: Players","author":"Kluyver Thomas","unstructured":"Thomas Kluyver , Benjamin Ragan-Kelley , Fernando P\u00e9rez , Brian Granger , Matthias Bussonnier , Jonathan Frederic , Kyle Kelley , Jessica Hamrick , Jason Grout , Sylvain Corlay , Paul Ivanov , Dami\u00e1n Avila , Safia Abdalla , Carol Willing , and Jupyter development team. 2016. Jupyter Notebooks - a publishing format for reproducible computational workflows . In Positioning and Power in Academic Publishing: Players , Agents and Agendas, Fernando Loizides and Birgit Scmidt (Eds.). IOS Press , Netherlands , 87--90. https:\/\/eprints.soton.ac.uk\/403913\/ Thomas Kluyver, Benjamin Ragan-Kelley, Fernando P\u00e9rez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Dami\u00e1n Avila, Safia Abdalla, Carol Willing, and Jupyter development team. 2016. Jupyter Notebooks - a publishing format for reproducible computational workflows. In Positioning and Power in Academic Publishing: Players, Agents and Agendas, Fernando Loizides and Birgit Scmidt (Eds.). IOS Press, Netherlands, 87--90. https:\/\/eprints.soton.ac.uk\/403913\/"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2833157.2833162"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1989.1.4.541"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.103"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/jcc.21787"},{"key":"e_1_3_2_1_34_1","unstructured":"Philip Mocz. 2020. nbody-python: Create Your Own N-body Simulation (With Python). https:\/\/github.com\/pmocz\/nbody-python.  Philip Mocz. 2020. nbody-python: Create Your Own N-body Simulation (With Python). https:\/\/github.com\/pmocz\/nbody-python."},{"key":"e_1_3_2_1_35_1","unstructured":"Ryosuke Okuta Yuya Unno Daisuke Nishino Shohei Hido and Crissman Loomis. 2017. CuPy: A NumPy-Compatible Library for NVIDIA GPU Calculations. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS). http:\/\/learningsys.org\/nips17\/assets\/papers\/paper_16.pdf  Ryosuke Okuta Yuya Unno Daisuke Nishino Shohei Hido and Crissman Loomis. 2017. CuPy: A NumPy-Compatible Library for NVIDIA GPU Calculations. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS) . http:\/\/learningsys.org\/nips17\/assets\/papers\/paper_16.pdf"},{"key":"e_1_3_2_1_36_1","unstructured":"Øystein Sture. [n.d.]. Implementation of crc16 (CRC-16-CCITT) in python. https:\/\/gist.github.com\/oysstu\/68072c44c02879a2abf94ef350d1c7c6 Retrieved 2021-02-04 from  Øystein Sture. [n.d.]. Implementation of crc16 (CRC-16-CCITT) in python. https:\/\/gist.github.com\/oysstu\/68072c44c02879a2abf94ef350d1c7c6 Retrieved 2021-02-04 from"},{"key":"e_1_3_2_1_37_1","volume-title":"PyTorch: An Imperative Style","author":"Paszke Adam","unstructured":"Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , Alban Desmaison , Andreas Kopf , Edward Yang , Zachary DeVito , Martin Raison , Alykhan Tejani , Sasank Chilamkurthy , Benoit Steiner , Lu Fang , Junjie Bai , and Soumith Chintala . 2019. PyTorch: An Imperative Style , High-Performance Deep Learning Library . In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlch\u00e9-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlch\u00e9-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf"},{"key":"e_1_3_2_1_38_1","volume-title":"Polybench: The polyhedral benchmark suite. URL: http:\/\/www. cs. ucla. edu\/pouchet\/software\/polybench 437","author":"Pouchet Louis-No\u00ebl","year":"2012","unstructured":"Louis-No\u00ebl Pouchet 2012 . Polybench: The polyhedral benchmark suite. URL: http:\/\/www. cs. ucla. edu\/pouchet\/software\/polybench 437 (2012). Louis-No\u00ebl Pouchet et al. 2012. Polybench: The polyhedral benchmark suite. URL: http:\/\/www. cs. ucla. edu\/pouchet\/software\/polybench 437 (2012)."},{"key":"e_1_3_2_1_39_1","unstructured":"PyPy. [n.d.]. Benchmarks. https:\/\/foss.heptapod.net\/pypy\/benchmarks.  PyPy. [n.d.]. Benchmarks. https:\/\/foss.heptapod.net\/pypy\/benchmarks."},{"key":"e_1_3_2_1_40_1","unstructured":"Python Software Foundation. [n.d.]. CPython. https:\/\/github.com\/python\/cpython.  Python Software Foundation. [n.d.]. CPython. https:\/\/github.com\/python\/cpython."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.25080\/Majora-629e541a-00e"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1176617.1176753"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.225783"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_1_45_1","unstructured":"Scalable Parallel Computing Lab. [n.d.]. DaCe - Data-Centric Parallel Programming. https:\/\/github.com\/spcl\/dace.  Scalable Parallel Computing Lab. [n.d.]. DaCe - Data-Centric Parallel Programming. https:\/\/github.com\/spcl\/dace."},{"key":"e_1_3_2_1_46_1","unstructured":"Serge Guelton Pierrick Brunet et al. [n.d.]. Pythran. https:\/\/pythran.readthedocs.io\/ Retrieved 2021-02-04 from  Serge Guelton Pierrick Brunet et al. [n.d.]. Pythran. https:\/\/pythran.readthedocs.io\/ Retrieved 2021-02-04 from"},{"key":"e_1_3_2_1_47_1","unstructured":"Stefan Behnel Robert Bradshaw Dag Sverre Seljebotn Greg Ewing William Stein Gabriel Gellner et al. [n.d.]. Cython for NumPy users. https:\/\/cython.readthedocs.io\/en\/latest\/src\/userguide\/numpy_tutorial.html Retrieved 2021-02-04 from  Stefan Behnel Robert Bradshaw Dag Sverre Seljebotn Greg Ewing William Stein Gabriel Gellner et al. [n.d.]. Cython for NumPy users. https:\/\/cython.readthedocs.io\/en\/latest\/src\/userguide\/numpy_tutorial.html Retrieved 2021-02-04 from"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1063\/1.4990384"},{"key":"e_1_3_2_1_49_1","unstructured":"Victor Stinner. 2017. The Python Performance Benchmark Suite. https:\/\/pyperformance.readthedocs.io\/ Retrieved 2021-01-28 from  Victor Stinner. 2017. The Python Performance Benchmark Suite. https:\/\/pyperformance.readthedocs.io\/ Retrieved 2021-01-28 from"},{"key":"e_1_3_2_1_50_1","unstructured":"Swiss National Supercomputing Centre (CSCS). [n.d.]. GT4Py. https:\/\/github.com\/GridTools\/gt4py Retrieved 2021-02-01 from  Swiss National Supercomputing Centre (CSCS). [n.d.]. GT4Py. https:\/\/github.com\/GridTools\/gt4py Retrieved 2021-02-01 from"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-019-0686-2"},{"key":"e_1_3_2_1_52_1","unstructured":"David Wheeler. [n.d.]. SLOCCount. https:\/\/dwheeler.com\/sloccount\/ Retrieved 2021-02-04 from  David Wheeler. [n.d.]. SLOCCount. https:\/\/dwheeler.com\/sloccount\/ Retrieved 2021-02-04 from"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3295500.3357156"}],"event":{"name":"ICS '21: 2021 International Conference on Supercomputing","location":"Virtual Event USA","acronym":"ICS '21","sponsor":["SIGARCH ACM Special Interest Group on Computer Architecture"]},"container-title":["Proceedings of the ACM International Conference on Supercomputing"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3447818.3460360","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3447818.3460360","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:06Z","timestamp":1750193286000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3447818.3460360"}},"subtitle":["a benchmarking suite for high-performance NumPy"],"short-title":[],"issued":{"date-parts":[[2021,6,3]]},"references-count":51,"alternative-id":["10.1145\/3447818.3460360","10.1145\/3447818"],"URL":"https:\/\/doi.org\/10.1145\/3447818.3460360","relation":{},"subject":[],"published":{"date-parts":[[2021,6,3]]},"assertion":[{"value":"2021-06-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}