{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T19:04:59Z","timestamp":1772910299047,"version":"3.50.1"},"reference-count":57,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T00:00:00Z","timestamp":1665705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Major Science Program of Henan Province","award":["221100210400"],"award-info":[{"award-number":["221100210400"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The quantum computer has been claimed to show more quantum advantage than the classical computer in solving some specific problems. Many companies and research institutes try to develop quantum computers with different physical implementations. Currently, most people only focus on the number of qubits in a quantum computer and consider it as a standard to evaluate the performance of the quantum computer intuitively. However, it is quite misleading in most times, especially for investors or governments. This is because the quantum computer works in a quite different way than classical computers. Thus, quantum benchmarking is of great importance. Currently, many quantum benchmarks are proposed from different aspects. In this paper, we review the existing performance benchmarking protocols, models, and metrics. We classify the benchmarking techniques into three categories: physical benchmarking, aggregative benchmarking, and application-level benchmarking. We also discuss the future trend for quantum computer\u2019s benchmarking and propose setting up the QTOP100.<\/jats:p>","DOI":"10.3390\/e24101467","type":"journal-article","created":{"date-parts":[[2022,10,17]],"date-time":"2022-10-17T00:04:55Z","timestamp":1665965095000},"page":"1467","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["SoK: Benchmarking the Performance of a Quantum Computer"],"prefix":"10.3390","volume":"24","author":[{"given":"Junchao","family":"Wang","sequence":"first","affiliation":[{"name":"State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou 450002, China"},{"name":"CAS Key Laboratory of Quantum Information, University of Science and Technology of China, Hefei 230026, China"}]},{"given":"Guoping","family":"Guo","sequence":"additional","affiliation":[{"name":"CAS Key Laboratory of Quantum Information, University of Science and Technology of China, Hefei 230026, China"}]},{"given":"Zheng","family":"Shan","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou 450002, China"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1460","DOI":"10.1126\/science.abe8770","article-title":"Quantum computational advantage using photons","volume":"370","author":"Zhong","year":"2020","journal-title":"Science"},{"key":"ref_2","first-page":"89","article-title":"A short note on Shor\u2019s factoring algorithm","volume":"27","author":"Buhrman","year":"1996","journal-title":"Assoc. Comput. Mach."},{"key":"ref_3","unstructured":"Grover, L.K. (1996, January 22\u201324). A fast quantum mechanical algorithm for database search. Proceedings of the 28th Annual ACM Symposium on the Theory of Computing(STOC), Philadelphia, PA, USA."},{"key":"ref_4","unstructured":"Farhi, E., Goldstone, J., and Gutmann, S. (2021, April 03). A Quantum Approximate Optimization Algorithm. Quantum Phys., Available online: https:\/\/export.arxiv.org\/abs\/1411.4028."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e84","DOI":"10.1002\/que2.84","article-title":"Quantum Computation for Pricing the Collateralized Debt Obligations","volume":"3","author":"Tang","year":"2021","journal-title":"Quantum Eng."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"6:1","DOI":"10.1147\/JRD.2018.2888987","article-title":"Potential of quantum computing for drug discovery","volume":"62","author":"Cao","year":"2018","journal-title":"IBM J. Res. Dev."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1007\/s42484-021-00042-0","article-title":"Semi-supervised time series classification method for quantum computing","volume":"3","author":"Yarkoni","year":"2021","journal-title":"Quantum Mach. Intell."},{"key":"ref_8","unstructured":"(2021, July 09). How Is the Quantum Computer Performance Assessed. Available online: https:\/\/quantum.ieee.org\/education\/quantum-supremacy-and-quantum-computer-performance."},{"key":"ref_9","unstructured":"(2021, November 17). IBM Quantum Experience. Available online: https:\/\/quantum-computing.ibm.com."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1038\/d41586-019-03781-0","article-title":"Instead of \u2018supremacy\u2019 use \u2018quantum advantage\u2019","volume":"576","author":"Mueck","year":"2019","journal-title":"Nature"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"021318","DOI":"10.1063\/1.5089550","article-title":"A Quantum Engineer\u2019s Guide to Superconduct-ing Qubits","volume":"6","author":"Krantz","year":"2019","journal-title":"Appl. Phys. Rev."},{"key":"ref_12","first-page":"1293","article-title":"Surface code quantum computing with error rates over 1%","volume":"83","author":"Wang","year":"2011","journal-title":"Phys. Rev. A"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"79","DOI":"10.22331\/q-2018-08-06-79","article-title":"Quantum Computing in the NISQ era and beyond","volume":"2","author":"Preskill","year":"2018","journal-title":"Quantum"},{"key":"ref_14","unstructured":"Nielsen, M.A., and Chuang, I.L. (2010). Introduction and overview. Quantum Computation and Quantum Information, Cambridge University Press."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"5347","DOI":"10.1038\/s41467-019-13068-7","article-title":"Characterizing large-scale quantum computers via cycle benchmarking","volume":"10","author":"Erhard","year":"2019","journal-title":"Nat. Commun."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"012307","DOI":"10.1103\/PhysRevA.77.012307","article-title":"Randomized benchmarking of quantum gates","volume":"77","author":"Knill","year":"2008","journal-title":"Phys. Rev. A"},{"key":"ref_17","unstructured":"Bishop, L.S., Bravyi, S., Cross, A., Gambetta, J.M., and Smolin, J. (2021, November 17). Quantum Volume. Available online: https:\/\/storageconsortium.de\/content\/sites\/default\/files\/quantum-volumehp08co1vbo0cc8fr.pdf."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"032328","DOI":"10.1103\/PhysRevA.100.032328","article-title":"Validating quantum computers using randomized model circuits","volume":"100","author":"Cross","year":"2019","journal-title":"Phys. Rev. A"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"362","DOI":"10.22331\/q-2020-11-15-362","article-title":"A volumetric framework for quantum computer benchmarks","volume":"4","author":"Young","year":"2020","journal-title":"Quantum"},{"key":"ref_20","unstructured":"(2021, March 28). IonQ Quantum Computer 4 Million Quantum Volume. Available online: https:\/\/www.nextbigfuture.com\/2021\/03\/ionq-quantum-computer-4-million-quantum-volume-and-16x-error-correction.html."},{"key":"ref_21","unstructured":"(2020, March 01). Scott Aaronson: Turn Down the Quantum Volume. Available online: https:\/\/www.scottaaronson.com\/blog\/?p=4649."},{"key":"ref_22","unstructured":"(2020, December 01). Scaling IonQ\u2019s Quantum Computers: The Roadmap. Available online: https:\/\/ionq.com\/posts\/december-09-2020-scaling-quantum-computer-roadmap."},{"key":"ref_23","unstructured":"(2021, March 01). Algorithmic Qubit Calculator. Available online: https:\/\/ionq.com\/algorithmic-qubit-calculator\/."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1038\/s41567-021-01409-7","article-title":"Measuring the Capabilities of Quantum Computers","volume":"18","author":"Proctor","year":"2020","journal-title":"Nat. Phys."},{"key":"ref_25","unstructured":"Wack, A., Paik, H., Javadi-Abhari, A., Jurcevic, P., Faro, I., Gambetta, J.M., and Johnson, B.R. (2021). Quality, Speed, and Scale: Three key attributes to measure the performance of near-term quantum computers. arXiv."},{"key":"ref_26","unstructured":"Mesman, K., Al-Ars, Z., and Mller, M. (2021). QPack: Quantum Approximate Optimization Algorithms as universal benchmark for quantum computers. arXiv."},{"key":"ref_27","unstructured":"(2021, June 01). qScore. Available online: https:\/\/github.com\/myQLM\/qscore."},{"key":"ref_28","unstructured":"(2021, June 01). New Cambridge Quantum Algorithm Sets a Benchmark in Performance and Effectively Outperforms Existing Methods. Available online: https:\/\/quantumzeitgeist.com\/new-cambridge-quantum-algorithm-sets-a-benchmark-in-performance-and-effectively-outperforms-existing-methods\/."},{"key":"ref_29","unstructured":"Dallaire-Demers, P.L., St\u0119ch\u0142y, M., Gonthier, J.F., Bashige, N.T., Romero, J., and Cao, Y. (2021). An application benchmark for fermionic quantum simulations. Am. Phys. Soc."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1038\/s41534-019-0157-8","article-title":"A generative modeling approach for benchmarking and training shallow quantum circuits","volume":"5","author":"Benedetti","year":"2019","journal-title":"NPJ Quantum Inf."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"415","DOI":"10.22331\/q-2021-03-22-415","article-title":"Application-Motivated, Holistic Benchmarking of a Full Quantum Computing Stack","volume":"5","author":"Mills","year":"2020","journal-title":"Quantum"},{"key":"ref_32","unstructured":"Lubinski, T., Johri, S., Varosy, P., Coleman, J., Zhao, L., Necaise, J., Baldwin, C.H., Mayer, K., and Proctor, T. (2021). Application-Oriented Performance Benchmarks for Quantum Computing. arXiv."},{"key":"ref_33","first-page":"062412","article-title":"Random circuit block-encoded matrix and a proposal of quantum LINPACK benchmark","volume":"33","author":"Dong","year":"2020","journal-title":"Phys. Rev. A"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1038\/s41534-019-0209-0","article-title":"Quantum chemistry as a benchmark for near-term quantum computers","volume":"5","author":"McCaskey","year":"2019","journal-title":"npj Quantum Inf."},{"key":"ref_35","unstructured":"Li, A., and Krishnamoorthy, S. (2020). QASMBench: A Low-level QASM Benchmark Suite for NISQ Evaluation and Simulation. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1038\/s41534-021-00478-z","article-title":"Analyzing the Performance of Variational Quantum Factoring on a Superconducting Quantum Processor","volume":"7","author":"Karamlou","year":"2020","journal-title":"NPJ Quantum Inf."},{"key":"ref_37","unstructured":"(2021, April 01). Zapata: Benchmarking Quantum Devices. Available online: https:\/\/www.zapatacomputing.com\/benchmarking-quantum-devices."},{"key":"ref_38","unstructured":"(2021, April 01). Application-Oriented Quantum Benchmarks. Available online: https:\/\/github.com\/SRI-International\/QC-App-Oriented-Benchmarks."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Dongarra, J.J., Luszczek, P., and Petitet, A. (2003). The LINPACK Benchmark: Past, Present and Future, John Wiley & Sons, Ltd.","DOI":"10.1002\/cpe.728"},{"key":"ref_40","unstructured":"(2021, April 01). XACC. Available online: https:\/\/github.com\/eclipse\/xacc."},{"key":"ref_41","unstructured":"(2021, June 01). QASMBench. Available online: http:\/\/github.com\/pnnl\/QASMBench."},{"key":"ref_42","unstructured":"(2021, June 01). Qconvert. Available online: http:\/\/quantum-circuit.com\/qconvert."},{"key":"ref_43","unstructured":"Preskill, J. (2012). Quantum computing and the entanglement frontier. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1126\/science.1252319","article-title":"Defining and detecting quantum speedup","volume":"345","author":"Wang","year":"2014","journal-title":"Science"},{"key":"ref_45","first-page":"303","article-title":"Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum computer","volume":"41","author":"Shor","year":"1997","journal-title":"Siam J. Comput."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"433","DOI":"10.22331\/q-2021-04-15-433","article-title":"How to factor 2048 bit RSA integers in 8 hours using 20 million noisy qubits","volume":"5","author":"Gidney","year":"2019","journal-title":"Quantum"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Beauregard, S. (2002). Circuit for Shor\u2019s algorithm using 2n+3 qubits. arXiv, preprint.","DOI":"10.26421\/QIC3.2-8"},{"key":"ref_48","unstructured":"Pavlidis, A., and Gizopoulos, D. (2012). Fast quantum modular exponentiation architecture for Shor\u2019s factorization algorithm. arXiv, preprint."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1007\/s11128-021-03398-1","article-title":"Resource analysis of quantum computing with noisy qubits for Shor\u2019s factoring algorithms","volume":"21","author":"Ha","year":"2022","journal-title":"Quantum Inf. Process."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1038\/s41567-018-0318-2","article-title":"On the complexity and verification of quantum random circuit sampling","volume":"6121","author":"Bouland","year":"2019","journal-title":"Nat. Phys."},{"key":"ref_51","first-page":"180501","article-title":"Strong quantum computational advantage using a superconducting quantum processor","volume":"127","author":"Wu","year":"2021","journal-title":"Quantum Phys."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1038\/nphoton.2013.175","article-title":"Boson sampling on a chip","volume":"7","author":"Ralph","year":"2013","journal-title":"Nat. Photonics"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Pan, F., Chen, K., and Zhang, P. (2021). Solving the sampling problem of the Sycamore quantum supremacy circuits. arXiv, preprint.","DOI":"10.1103\/PhysRevLett.129.090502"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Liu, Y., Liu, X., Li, F., Fu, H., Yang, Y., Song, J., Zhao, P., Wang, Z., Peng, D., and Chen, H. (2021, January 14\u201319). Closing the \u201cQuantum Supremacy\u201d Gap: Achieving Real-Time Simulation of a Random Quantum Circuit Using a New Sunway Supercomputer. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, MO, USA.","DOI":"10.1145\/3458817.3487399"},{"key":"ref_55","unstructured":"Ronagh, P. (2019). Quantum Algorithms for Solving Dynamic Programming Problems. arXiv, preprint."},{"key":"ref_56","unstructured":"(2021, April 01). TOP500. Available online: https:\/\/top500.org\/lists\/green500\/."},{"key":"ref_57","unstructured":"(2021, April 01). Green500. Available online: https:\/\/graph500.org\/."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/10\/1467\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:54:16Z","timestamp":1760144056000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/10\/1467"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,14]]},"references-count":57,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["e24101467"],"URL":"https:\/\/doi.org\/10.3390\/e24101467","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,14]]}}}