{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T05:06:09Z","timestamp":1781067969876,"version":"3.54.1"},"reference-count":45,"publisher":"Verein zur Forderung des Open Access Publizierens in den Quantenwissenschaften","license":[{"start":{"date-parts":[[2020,11,15]],"date-time":"2020-11-15T00:00:00Z","timestamp":1605398400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["quantum-journal.org"],"crossmark-restriction":false},"short-container-title":["Quantum"],"abstract":"We propose a very large family of benchmarks for probing the performance of quantum computers. We call them {volumetric benchmarks} (VBs) because they generalize IBM's benchmark for measuring quantum volume \\cite{Cross18}. The quantum volume benchmark defines a family of {square} circuits whose depth d<\/mml:mi><\/mml:math> and width w<\/mml:mi><\/mml:math> are the same. A volumetric benchmark defines a family of {rectangular} quantum circuits, for which d<\/mml:mi><\/mml:math> and w<\/mml:mi><\/mml:math> are uncoupled to allow the study of time\/space performance trade-offs. Each VB defines a mapping from circuit shapes -- (<\/mml:mo>w<\/mml:mi>,<\/mml:mo>d<\/mml:mi>)<\/mml:mo><\/mml:math> pairs -- to test suites C<\/mml:mi><\/mml:mrow><\/mml:mrow>(<\/mml:mo>w<\/mml:mi>,<\/mml:mo>d<\/mml:mi>)<\/mml:mo><\/mml:math>. A test suite is an ensemble of test circuits that share a common structure. The test suite C<\/mml:mi><\/mml:mrow><\/mml:mrow><\/mml:math> for a given circuit shape may be a single circuit C<\/mml:mi><\/mml:math>, a specific list of circuits {<\/mml:mo>C<\/mml:mi>1<\/mml:mn><\/mml:msub>&#x2026;<\/mml:mo>C<\/mml:mi>N<\/mml:mi><\/mml:msub>}<\/mml:mo><\/mml:math> that must all be run, or a large set of possible circuits equipped with a distribution P<\/mml:mi>r<\/mml:mi>(<\/mml:mo>C<\/mml:mi>)<\/mml:mo><\/mml:math>. The circuits in a given VB share a structure, which is limited only by designers' creativity. We list some known benchmarks, and other circuit families, that fit into the VB framework: several families of random circuits, periodic circuits, and algorithm-inspired circuits. The last ingredient defining a benchmark is a success criterion that defines when a processor is judged to have ``passed'' a given test circuit. We discuss several options. Benchmark data can be analyzed in many ways to extract many properties, but we propose a simple, universal graphical summary of results that illustrates the Pareto frontier of the d<\/mml:mi><\/mml:math> vs w<\/mml:mi><\/mml:math> trade-off for the processor being benchmarked.<\/jats:p>","DOI":"10.22331\/q-2020-11-15-362","type":"journal-article","created":{"date-parts":[[2020,11,15]],"date-time":"2020-11-15T15:55:07Z","timestamp":1605455707000},"page":"362","update-policy":"https:\/\/doi.org\/10.22331\/q-crossmark-policy-page","source":"Crossref","is-referenced-by-count":76,"title":["A volumetric framework for quantum computer benchmarks"],"prefix":"10.22331","volume":"4","author":[{"given":"Robin","family":"Blume-Kohout","sequence":"first","affiliation":[{"name":"Quantum Performance Laboratory, Sandia National Laboratories\\vspace-.1cm \\ Albuquerque, NM 87185 and Livermore, California 94550"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4679-4542","authenticated-orcid":false,"given":"Kevin C.","family":"Young","sequence":"additional","affiliation":[{"name":"Quantum Performance Laboratory, Sandia National Laboratories\\vspace-.1cm \\ Albuquerque, NM 87185 and Livermore, California 94550"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"9598","published-online":{"date-parts":[[2020,11,15]]},"reference":[{"key":"0","doi-asserted-by":"publisher","unstructured":"Andrew W Cross, Lev S Bishop, Sarah Sheldon, Paul D Nation, and Jay M Gambetta. Validating quantum computers using randomized model circuits. Phys. Rev. A, 100 (3): 032328, September 2019. ISSN 1050-2947. 10.1103\/PhysRevA.100.032328.","DOI":"10.1103\/PhysRevA.100.032328"},{"key":"1","doi-asserted-by":"publisher","unstructured":"Rami Barends, Julian Kelly, et al. Superconducting quantum circuits at the surface code threshold for fault tolerance. Nature, 508 (7497): 500, 2014. 10.1038\/nature13171.","DOI":"10.1038\/nature13171"},{"key":"2","doi-asserted-by":"publisher","unstructured":"Simon J Devitt. Performing quantum computing experiments in the cloud. Phys. Rev. A, 94 (3): 032329, September 2016. ISSN 1050-2947. 10.1103\/PhysRevA.94.032329.","DOI":"10.1103\/PhysRevA.94.032329"},{"key":"3","doi-asserted-by":"publisher","unstructured":"S Debnath, N M Linke, C Figgatt, K A Landsman, K Wright, and C Monroe. Demonstration of a small programmable quantum computer with atomic qubits. Nature, 536 (7614): 63\u201366, August 2016. ISSN 0028-0836, 1476-4687. 10.1038\/nature18648.","DOI":"10.1038\/nature18648"},{"key":"4","doi-asserted-by":"publisher","unstructured":"Norbert M Linke, Dmitri Maslov, Martin Roetteler, Shantanu Debnath, Caroline Figgatt, Kevin A Landsman, Kenneth Wright, and Christopher Monroe. Experimental comparison of two quantum computing architectures. Proceedings of the National Academy of Sciences, 114 (13): 3305\u20133310, 2017. 10.1073\/pnas.1618020114.","DOI":"10.1073\/pnas.1618020114"},{"key":"5","unstructured":"James R Wootton. Benchmarking of quantum processors with random circuits. arXiv:1806.02736, 2018."},{"key":"6","doi-asserted-by":"publisher","unstructured":"Julian Kelly, Rami Barends, et al. State preservation by repetitive error detection in a superconducting quantum circuit. Nature, 519 (7541): 66, 2015. 10.1038\/nature14270.","DOI":"10.1038\/nature14270"},{"key":"7","unstructured":"IonQ press release. [Online], 2018. https:\/\/ionq.co\/news\/december-11-2018 [2019-03-18]."},{"key":"8","unstructured":"IBM Q ``Tokyo'' Specifications. [Online], 2019a. https:\/\/www.research.ibm.com\/ibm-q\/technology\/devices\/#ibmq-20-tokyo [2019-03-18]."},{"key":"9","doi-asserted-by":"publisher","unstructured":"Frank Arute, Kunal Arya, et al. Quantum supremacy using a programmable superconducting processor. Nature, 574 (7779): 505\u2013510, October 2019. ISSN 0028-0836, 1476-4687. 10.1038\/s41586-019-1666-5.","DOI":"10.1038\/s41586-019-1666-5"},{"key":"10","unstructured":"IBM news room. [Online], 2019b. https:\/\/newsroom.ibm.com\/2019-09-18-IBM-Opens-Quantum-Computation-Center-in-New-York-Brings-Worlds-Largest-Fleet-of-Quantum-Computing-Systems-Online-Unveils-New-53-Qubit-Quantum-System-for-Broad-Use [2019-09-18]."},{"key":"11","doi-asserted-by":"publisher","unstructured":"Fernando G S L Brand\u00e3o, Aram W Harrow, and Micha\u0142 Horodecki. Local random quantum circuits are approximate Polynomial-Designs. Commun. Math. Phys., 346 (2): 397\u2013434, September 2016. ISSN 1432-0916. 10.1007\/s00220-016-2706-8.","DOI":"10.1007\/s00220-016-2706-8"},{"key":"12","doi-asserted-by":"publisher","unstructured":"Joseph Emerson, Robert Alicki, and Karol \u017byczkowski. Scalable noise estimation with random unitary operators. Journal of Optics B: Quantum and Semiclassical Optics, 7 (10): S347, 2005. 10.1088\/1464-4266\/7\/10\/021.","DOI":"10.1088\/1464-4266\/7\/10\/021"},{"key":"13","doi-asserted-by":"publisher","unstructured":"Joseph Emerson, Marcus Silva, Osama Moussa, Colm Ryan, Martin Laforest, Jonathan Baugh, David G Cory, and Raymond Laflamme. Symmetrized characterization of noisy quantum processes. Science, 317 (5846): 1893\u20131896, 2007. 10.1126\/science.1145699.","DOI":"10.1126\/science.1145699"},{"key":"14","doi-asserted-by":"publisher","unstructured":"Emanuel Knill, Dietrich Leibfried, et al. Randomized benchmarking of quantum gates. Physical Review A, 77 (1): 012307, 2008. 10.1103\/PhysRevA.77.012307.","DOI":"10.1103\/PhysRevA.77.012307"},{"key":"15","doi-asserted-by":"publisher","unstructured":"Easwar Magesan, Jay M Gambetta, and Joseph Emerson. Scalable and robust randomized benchmarking of quantum processes. Physical Review Letters, 106 (18): 180504, 2011. 10.1103\/PhysRevLett.106.180504.","DOI":"10.1103\/PhysRevLett.106.180504"},{"key":"16","doi-asserted-by":"publisher","unstructured":"Easwar Magesan, Jay M Gambetta, and Joseph Emerson. Characterizing quantum gates via randomized benchmarking. Physical Review A, 85 (4): 042311, 2012. 10.1103\/PhysRevA.85.042311.","DOI":"10.1103\/PhysRevA.85.042311"},{"key":"17","doi-asserted-by":"publisher","unstructured":"Robin Blume-Kohout, John King Gamble, Erik Nielsen, Kenneth Rudinger, Jonathan Mizrahi, Kevin Fortier, and Peter Maunz. Demonstration of qubit operations below a rigorous fault tolerance threshold with gate set tomography. Nature Communications, 8: 14485, 2017. 10.1038\/ncomms16226.","DOI":"10.1038\/ncomms16226"},{"key":"18","unstructured":"Daniel Greenbaum. Introduction to quantum gate set tomography. arXiv:1509.02921, 2015."},{"key":"19","doi-asserted-by":"publisher","unstructured":"Juan P Dehollain, Juha T Muhonen, Robin Blume-Kohout, Kenneth M Rudinger, John King Gamble, Erik Nielsen, Arne Laucht, Stephanie Simmons, Rachpon Kalra, Andrew S Dzurak, and Andrea Morello. Optimization of a solid-state electron spin qubit using gate set tomography. New J. Phys., 18 (10): 103018, October 2016. ISSN 1367-2630. 10.1088\/1367-2630\/18\/10\/103018.","DOI":"10.1088\/1367-2630\/18\/10\/103018"},{"key":"20","doi-asserted-by":"publisher","unstructured":"Jack J Dongarra, Piotr Luszczek, and Antoine Petitet. The LINPACK benchmark: past, present and future. Concurrency and Computation: Practice and Experience, 15 (9): 803\u2013820, 2003. 10.1002\/cpe.728.","DOI":"10.1002\/cpe.728"},{"key":"21","unstructured":"Joseph Emerson. Benchmarking into the future. IARPA LogiQ Principal Investigators Meeting, 6 September, 2017."},{"key":"22","unstructured":"Joseph Emerson. (private communication, 2019)."},{"key":"23","doi-asserted-by":"publisher","unstructured":"Alexander Erhard, Joel J Wallman, Lukas Postler, Michael Meth, Roman Stricker, Esteban A Martinez, Philipp Schindler, Thomas Monz, Joseph Emerson, and Rainer Blatt. Characterizing large-scale quantum computers via cycle benchmarking. Nat. Commun., 10 (1): 5347, November 2019. ISSN 2041-1723. 10.1038\/s41467-019-13068-7.","DOI":"10.1038\/s41467-019-13068-7"},{"key":"24","unstructured":"Lev Bishop. Is the `Quantum Volume' a fair metric for future, elaborate, high value quantum computations? Quantum Computing Stack Exchange, 2018. https:\/\/quantumcomputing.stackexchange.com\/a\/4001 [2019-03-21]."},{"key":"25","doi-asserted-by":"publisher","unstructured":"Peter W Shor. Algorithms for quantum computation: Discrete logarithms and factoring. In Proceedings 35th Annual Symposium on Foundations of Computer Science, pages 124\u2013134. IEEE, 1994. 10.1109\/SFCS.1994.365700.","DOI":"10.1109\/SFCS.1994.365700"},{"key":"26","doi-asserted-by":"publisher","unstructured":"Lov K Grover. Quantum mechanics helps in searching for a needle in a haystack. Physical Review Letters, 79 (2): 325, 1997. 10.1103\/PhysRevLett.79.325.","DOI":"10.1103\/PhysRevLett.79.325"},{"key":"27","doi-asserted-by":"publisher","unstructured":"Richard Cleve and John Watrous. Fast parallel circuits for the quantum Fourier transform. In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 526\u2013536. IEEE, 2000. 10.1109\/SFCS.2000.892140.","DOI":"10.1109\/SFCS.2000.892140"},{"key":"28","doi-asserted-by":"publisher","unstructured":"Yasuhiro Sekino and Leonard Susskind. Fast scramblers. Journal of High Energy Physics, 2008 (10): 065, 2008. 10.1088\/1126-6708\/2008\/10\/065.","DOI":"10.1088\/1126-6708\/2008\/10\/065"},{"key":"29","unstructured":"Scott Aaronson and Lijie Chen. Complexity-Theoretic foundations of quantum supremacy experiments. December 2016."},{"key":"30","doi-asserted-by":"publisher","unstructured":"Steven J van Enk and Robin Blume-Kohout. When quantum tomography goes wrong: drift of quantum sources and other errors. New Journal of Physics, 15 (2): 025024, 2013. 10.1088\/1367-2630\/15\/2\/025024.","DOI":"10.1088\/1367-2630\/15\/2\/025024"},{"key":"31","doi-asserted-by":"publisher","unstructured":"Kenneth Rudinger, Timothy Proctor, Dylan Langharst, Mohan Sarovar, Kevin Young, and Robin Blume-Kohout. Probing Context-Dependent errors in quantum processors. Phys. Rev. X, 9 (2): 021045, June 2019. 10.1103\/PhysRevX.9.021045.","DOI":"10.1103\/PhysRevX.9.021045"},{"key":"32","doi-asserted-by":"publisher","unstructured":"Kristine Boone, Arnaud Carignan-Dugas, Joel J Wallman, and Joseph Emerson. Randomized benchmarking under different gate sets. Phys. Rev. A, 99 (3): 032329, March 2019. ISSN 1050-2947. 10.1103\/PhysRevA.99.032329.","DOI":"10.1103\/PhysRevA.99.032329"},{"key":"33","doi-asserted-by":"publisher","unstructured":"Timothy J Proctor, Arnaud Carignan-Dugas, Kenneth Rudinger, Erik Nielsen, Robin Blume-Kohout, and Kevin Young. Direct randomized benchmarking for multiqubit devices. Phys. Rev. Lett., 123 (3): 030503, July 2019. ISSN 0031-9007, 1079-7114. 10.1103\/PhysRevLett.123.030503.","DOI":"10.1103\/PhysRevLett.123.030503"},{"key":"34","doi-asserted-by":"publisher","unstructured":"Jay M Gambetta, Antonio D C\u00f3rcoles, et al. Characterization of addressability by simultaneous randomized benchmarking. Physical Review Letters, 109 (24): 240504, 2012. 10.1103\/PhysRevLett.109.240504.","DOI":"10.1103\/PhysRevLett.109.240504"},{"key":"35","doi-asserted-by":"publisher","unstructured":"Sergio Boixo, Sergei V Isakov, Vadim N Smelyanskiy, Ryan Babbush, Nan Ding, Zhang Jiang, Michael J Bremner, John M Martinis, and Hartmut Neven. Characterizing quantum supremacy in near-term devices. Nature Physics, 14 (6): 595, 2018. 10.1038\/s41567-018-0124-x.","DOI":"10.1038\/s41567-018-0124-x"},{"key":"36","doi-asserted-by":"publisher","unstructured":"Charles Neill, Pedran Roushan, et al. A blueprint for demonstrating quantum supremacy with superconducting qubits. Science, 360 (6385): 195\u2013199, 2018. 10.1126\/science.aao4309.","DOI":"10.1126\/science.aao4309"},{"key":"37","doi-asserted-by":"publisher","unstructured":"Shelby Kimmel, Guang Hao Low, and Theodore J Yoder. Robust calibration of a universal single-qubit gate set via robust phase estimation. Physical Review A, 92 (6): 062315, 2015. 10.1103\/PhysRevA.92.062315.","DOI":"10.1103\/PhysRevA.92.062315"},{"key":"38","doi-asserted-by":"publisher","unstructured":"Richard Cleve, Artur Ekert, Chiara Macchiavello, and Michele Mosca. Quantum algorithms revisited. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 454 (1969): 339\u2013354, 1998. 10.1098\/rspa.1998.0164.","DOI":"10.1098\/rspa.1998.0164"},{"key":"39","doi-asserted-by":"publisher","unstructured":"Alberto Peruzzo, Jarrod McClean, et al. A variational eigenvalue solver on a photonic quantum processor. Nature Communications, 5: 4213, 2014. 10.1038\/ncomms5213.","DOI":"10.1038\/ncomms5213"},{"key":"40","doi-asserted-by":"publisher","unstructured":"David C McKay, Christopher J Wood, Sarah Sheldon, Jerry M Chow, and Jay M Gambetta. Efficient $Z$ gates for quantum computing. Phys. Rev. A, 96 (2): 022330, August 2017. ISSN 1050-2947. 10.1103\/PhysRevA.96.022330.","DOI":"10.1103\/PhysRevA.96.022330"},{"key":"41","doi-asserted-by":"publisher","unstructured":"M A Rol, L Ciorciaro, F K Malinowski, B M Tarasinski, R E Sagastizabal, C C Bultink, Y Salathe, N Haandbaek, J Sedivy, and L DiCarlo. Time-domain characterization and correction of on-chip distortion of control pulses in a quantum processor. Appl. Phys. Lett., 116 (5): 054001, February 2020. ISSN 0003-6951. 10.1063\/1.5133894.","DOI":"10.1063\/1.5133894"},{"key":"42","doi-asserted-by":"publisher","unstructured":"Adam Bouland, Bill Fefferman, Chinmay Nirkhe, and Umesh Vazirani. On the complexity and verification of quantum random circuit sampling. Nat. Phys., 15 (2): 159\u2013163, February 2019. ISSN 1745-2473, 1745-2481. 10.1038\/s41567-018-0318-2.","DOI":"10.1038\/s41567-018-0318-2"},{"key":"43","unstructured":"Robin Blume-Kohout et al. Idle Tomography. In preparation, 2020."},{"key":"44","doi-asserted-by":"publisher","unstructured":"Seth Lloyd. Universal quantum simulators. Science, 273: 1073\u20131078, 1996. 10.1126\/science.273.5278.1073.","DOI":"10.1126\/science.273.5278.1073"}],"container-title":["Quantum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/quantum-journal.org\/papers\/q-2020-11-15-362\/pdf\/","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"}],"deposited":{"date-parts":[[2022,2,9]],"date-time":"2022-02-09T12:30:58Z","timestamp":1644409858000},"score":1,"resource":{"primary":{"URL":"https:\/\/quantum-journal.org\/papers\/q-2020-11-15-362\/"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,15]]},"references-count":45,"URL":"https:\/\/doi.org\/10.22331\/q-2020-11-15-362","archive":["CLOCKSS"],"relation":{},"ISSN":["2521-327X"],"issn-type":[{"value":"2521-327X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,15]]},"article-number":"362"}}