{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T14:58:21Z","timestamp":1784300301136,"version":"3.55.0"},"reference-count":91,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2025,4,16]],"date-time":"2025-04-16T00:00:00Z","timestamp":1744761600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,4,16]],"date-time":"2025-04-16T00:00:00Z","timestamp":1744761600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100020959","name":"JST-Mirai Program","doi-asserted-by":"publisher","award":["JMPJMI20B8"],"award-info":[{"award-number":["JMPJMI20B8"]}],"id":[{"id":"10.13039\/501100020959","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"Norges Forskningsr\u00e5d","doi-asserted-by":"publisher","award":["Qu-Test project #299827"],"award-info":[{"award-number":["Qu-Test project #299827"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100020586","name":"storbyuniversitetet","doi-asserted-by":"publisher","award":["Quantum Hub initiative"],"award-info":[{"award-number":["Quantum Hub initiative"]}],"id":[{"id":"10.13039\/100020586","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Simula Research Laboratory","award":["Internal Strategic Project on Quantum Software Engineering"],"award-info":[{"award-number":["Internal Strategic Project on Quantum Software Engineering"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>As a new research area, quantum software testing lacks systematic testing benchmarks to assess testing techniques\u2019 effectiveness. Recently, some open-source benchmarks and mutation analysis tools have emerged. However, there is insufficient evidence on how various quantum circuit characteristics (e.g., circuit depth, number of quantum gates), algorithms (e.g., Quantum Approximate Optimization Algorithm), and mutation characteristics (e.g., mutation operators) affect the detection of mutants in quantum circuits. Studying such relations is important to systematically design faulty benchmarks with varied attributes (e.g., the difficulty in detecting a seeded fault) to facilitate assessing the cost-effectiveness of quantum software testing techniques efficiently. To this end, we present a large-scale empirical evaluation with more than 700K faulty benchmarks (quantum circuits) generated by mutating 382 real-world quantum circuits. Based on the results, we provide valuable insights for researchers to define systematic quantum mutation analysis techniques. We also provide a tool to recommend mutants to users based on chosen characteristics (e.g., a quantum algorithm type) and the required difficulty of detecting mutants. Finally, we also provide faulty benchmarks that can already be used to assess the cost-effectiveness of quantum software testing techniques.<\/jats:p>","DOI":"10.1007\/s10664-025-10643-z","type":"journal-article","created":{"date-parts":[[2025,4,16]],"date-time":"2025-04-16T09:08:52Z","timestamp":1744794532000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Quantum circuit mutants: Empirical analysis and recommendations"],"prefix":"10.1007","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-3315-1664","authenticated-orcid":false,"given":"E\u00f1aut","family":"Mendiluze Usandizaga","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shaukat","family":"Ali","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tao","family":"Yue","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paolo","family":"Arcaini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,4,16]]},"reference":[{"key":"10643_CR1","doi-asserted-by":"publisher","unstructured":"Abreu R, Fernandes JP, Llana L, Tavares G (2022) Metamorphic Testing of Oracle Quantum Programs. In: 2022 IEEE\/ACM 3rd International Workshop on Quantum Software Engineering (Q-SE), pp. 16\u201323. doi: https:\/\/doi.org\/10.1145\/3528230.3529189","DOI":"10.1145\/3528230.3529189"},{"key":"10643_CR2","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1016\/j.infsof.2016.04.003","volume":"81","author":"A Alberto","year":"2017","unstructured":"Alberto A, Cavalcanti A, Gaudel M, Sim\u00f1o A (2017) Formal mutation testing for Circus. Information and Software Technology 81:131\u2013153. https:\/\/doi.org\/10.1016\/j.infsof.2016.04.003","journal-title":"Information and Software Technology"},{"key":"10643_CR3","doi-asserted-by":"publisher","unstructured":"Ali S, Arcaini P, Wang X, Yue T (2021) Assessing the effectiveness of input and output coverage criteria for testing quantum programs. In: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). pp. 13\u201323. dhttps:\/\/doi.org\/10.1016\/j.infsof.2016.04.003.","DOI":"10.1016\/j.infsof.2016.04.003."},{"issue":"4","key":"10643_CR4","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1145\/3512340","volume":"65","author":"S Ali","year":"2022","unstructured":"Ali S, Yue T, Abreu R (2022) When software engineering meets quantum computing. Commun. ACM 65(4):84\u201388. https:\/\/doi.org\/10.1145\/3512340","journal-title":"Commun. ACM"},{"key":"10643_CR5","doi-asserted-by":"publisher","DOI":"10.1017\/9781316771273","volume-title":"Introduction to software testing","author":"P Ammann","year":"2016","unstructured":"Ammann P, Offutt J (2016) Introduction to software testing. Cambridge University Press, Cambridge"},{"key":"10643_CR6","doi-asserted-by":"publisher","unstructured":"Anbalagan, P., & Xie, T. (2008). Automated generation of pointcut mutants for testing pointcuts in AspectJ programs. In: 2008 19th international symposium on software reliability engineering (issre) (pp. 239\u2013248). dhttps:\/\/doi.org\/10.1109\/ISSRE.2008.58.","DOI":"10.1109\/ISSRE.2008.58."},{"key":"10643_CR7","doi-asserted-by":"publisher","unstructured":"Arcaini P, Gargantini A, Riccobene E (2017) MutRex: A mutation-based generator of fault detecting strings for regular expressions. In: 2017 ieee international conference on software testing, verification and validation workshops (icstw) (pp. 87\u201396). https:\/\/doi.org\/10.1109\/ICSTW.2017.23.","DOI":"10.1109\/ICSTW.2017.23."},{"key":"10643_CR8","doi-asserted-by":"publisher","unstructured":"Arcaini, P., Gargantini, A., Riccobene, E. (2019). Fault-based test generation for regular expressions by mutation. In: Software Testing, Verification and Reliability, 29(1-2), e1664, https:\/\/doi.org\/10.1002\/stvr.1664","DOI":"10.1002\/stvr.1664"},{"issue":"1","key":"10643_CR9","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1049\/qtc2.12004","volume":"2","author":"U Azad","year":"2021","unstructured":"Azad U, Papneja A, Saini R, Behera BK, Panigrahi PK (2021) Circuit centric quantum architecture design. IET Quantum Communication 2(1):14\u201325","journal-title":"IET Quantum Communication"},{"key":"10643_CR10","doi-asserted-by":"publisher","unstructured":"Bottaci L (2010) Type sensitive application of mutation operators for dynamically typed programs. 2010 third international conference on software testing, verification, and validation workshops pp. 126\u2013131. dhttps:\/\/doi.org\/10.1109\/ICSTW.2010.56.","DOI":"10.1109\/ICSTW.2010.56."},{"key":"10643_CR11","doi-asserted-by":"publisher","unstructured":"Boubeta-Puig J, Medina-Bulo I, Garc\u00eda-Dom\u00ednguez A (2011) Analogies and differences between mutation operators for WS-BPEL 2.0 and other languages. In: 2011 ieee fourth international conference on software testing, verification and validation workshops pp. 398\u2013407. https:\/\/doi.org\/10.1109\/ICSTW.2011.52.","DOI":"10.1109\/ICSTW.2011.52."},{"key":"10643_CR12","doi-asserted-by":"publisher","unstructured":"Campos J, Souto A (2021) QBugs: A collection of reproducible bugs in quantum algorithms and a supporting infrastructure to enable controlled quantum software testing and debugging experiments. In: 2021 ieee\/acm 2nd international workshop on quantum software engineering (q-se) pp. 28\u201332. Los Alamitos, CA, USA: IEEE Computer Society. https:\/\/doi.org\/10.1109\/Q-SE52541.2021.00013.","DOI":"10.1109\/Q-SE52541.2021.00013."},{"key":"10643_CR13","doi-asserted-by":"publisher","unstructured":"Childs AM, Schoute E, Unsal CM (2019) Circuit Transformations for Quantum Architectures. W.\u00a0van Dam and L.\u00a0Man\u010dinska (Eds.), In: 14th conference on the theory of quantum computation, communication and cryptography (tqc 2019) Vol.\u00a0135, pp. 3:1\u20133:24. Dagstuhl, Germany: Schloss Dagstuhl \u2013 Leibniz-Zentrum f\u00fcr Informatik. https:\/\/doi.org\/10.4230\/LIPIcs.TQC.2019.3","DOI":"10.4230\/LIPIcs.TQC.2019.3"},{"key":"10643_CR14","doi-asserted-by":"publisher","unstructured":"Cicero A, Maleki MA, Azhar MW, Kockum AF, Trancoso P (2024) Simulation of quantum computers: Review and acceleration opportunities. In: CoRR, abs\/2410.12660, https:\/\/doi.org\/10.48550\/ARXIV.2410.12660","DOI":"10.48550\/ARXIV.2410.12660"},{"key":"10643_CR15","doi-asserted-by":"publisher","unstructured":"Cleophas TJ, Zwinderman AH (2018) Bayesian pearson correlation analysis. In: Modern bayesian statistics in clinical research. pp. 111\u2013118. Cham: Springer International Publishing. dhttps:\/\/doi.org\/10.1007\/978-3-319-92747-3_11","DOI":"10.1007\/978-3-319-92747-3_11"},{"key":"10643_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TEM.2022.3175633","volume":"71","author":"M Coccia","year":"2022","unstructured":"Coccia M, Roshani S, Mosleh M (2022) Evolution of quantum computing: Theoretical and innovation management implications for emerging quantum industry. IEEE Transactions on Engineering Management 71:1\u201311. https:\/\/doi.org\/10.1109\/TEM.2022.3175633","journal-title":"IEEE Transactions on Engineering Management"},{"key":"10643_CR17","doi-asserted-by":"publisher","DOI":"10.4324\/9780203771587","volume-title":"Statistical power analysis for the behavioral sciences","author":"J Cohen","year":"2013","unstructured":"Cohen J (2013) Statistical power analysis for the behavioral sciences. Routledge, London"},{"key":"10643_CR18","doi-asserted-by":"publisher","unstructured":"Cross AW, Bishop LS, Smolin JA, Gambetta JM (2017) Open quantum assembly language. In: arXiv:1707.03429, https:\/\/doi.org\/10.48550\/arXiv.1707.03429","DOI":"10.48550\/arXiv.1707.03429"},{"key":"10643_CR19","doi-asserted-by":"publisher","unstructured":"Delamaro ME, Deng L, Durelli VHS, Li N, Offutt J (2014) Experimental evaluation of SDL and one-op mutation for C. In: 2014 Ieee Seventh International Conference on Software Testing, Verification And Validation pp. 203\u2013212. dohttps:\/\/doi.org\/10.1109\/ICSTW.2011.52","DOI":"10.1109\/ICSTW.2011.52"},{"key":"10643_CR20","doi-asserted-by":"publisher","unstructured":"Delamaro ME, Offutt J, Ammann P (2014) Designing deletion mutation operators. In: 2014 Ieee Seventh International Conference on Software Testing, Verification and Validation. pp. 11\u201320. https:\/\/doi.org\/10.1109\/ICST.2014.12","DOI":"10.1109\/ICST.2014.12"},{"issue":"4\u20135","key":"10643_CR21","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1630","volume":"27","author":"P Delgado-P\u00e9rez","year":"2017","unstructured":"Delgado-P\u00e9rez P, Segura S, Medina-Bulo I (2017) Assessment of C++ object-oriented mutation operators: A selective mutation approach. Software Testing, Verification and Reliability 27(4\u20135):e1630","journal-title":"Software Testing, Verification and Reliability"},{"key":"10643_CR22","doi-asserted-by":"publisher","unstructured":"Deng L, Mirzaei N, Ammann P, Offutt J (2015) Towards mutation analysis of Android apps. In: 2015 Ieee Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) pp. 1\u201310. https:\/\/doi.org\/10.1109\/ICSTW.2015.7107450","DOI":"10.1109\/ICSTW.2015.7107450"},{"key":"10643_CR23","doi-asserted-by":"publisher","unstructured":"Dom\u00ednguez-Jim\u00e9nez J, Estero-Botaro A, Garc\u00eda-Dom\u00ednguez A, Medina-Bulo I (2011) Evolutionary mutation testing. Information and Software Technology 53(10):1108\u20131123. https:\/\/doi.org\/10.1016\/j.infsof.2011.03.008 (Special Section on Mutation Testing)","DOI":"10.1016\/j.infsof.2011.03.008"},{"key":"10643_CR24","doi-asserted-by":"publisher","unstructured":"Durelli VHS, De\u00a0Souza NM, Delamaro ME (2017) Are deletion mutants easier to identify manually? In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 149\u2013158). dhttps:\/\/doi.org\/10.1109\/ICSTW.2017.32","DOI":"10.1109\/ICSTW.2017.32"},{"key":"10643_CR25","doi-asserted-by":"publisher","unstructured":"Estero-Botaro A, Palomo-Lozano F, Medina-Bulo I (2010) Quantitative evaluation of mutation operators for WS-BPEL compositions. In: 2010 Third International Conference on Software Testing, Verification, and Validation Workshops. pp. 142\u2013150. doi: https:\/\/doi.org\/10.1109\/ICSTW.2010.36.","DOI":"10.1109\/ICSTW.2010.36."},{"key":"10643_CR26","doi-asserted-by":"publisher","unstructured":"Ferrari FC, Maldonado JC, Rashid A (2008) Mutation testing for aspect-oriented programs. In: 2008 1st International Conference on Software Testing, Verification, and Validation. pp. 52\u201361. doi: https:\/\/doi.org\/10.1109\/ICST.2008.37","DOI":"10.1109\/ICST.2008.37"},{"key":"10643_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TQE.2022.3195061","volume":"3","author":"D Fortunato","year":"2022","unstructured":"Fortunato D, Campos J, Abreu R (2022) Mutation testing of quantum programs: A case study with Qiskit. IEEE Transactions on Quantum Engineering 3:1\u201317. https:\/\/doi.org\/10.1109\/TQE.2022.3195061","journal-title":"IEEE Transactions on Quantum Engineering"},{"key":"10643_CR28","doi-asserted-by":"publisher","unstructured":"Fortunato D, Campos J, Abreu R (2022b) Mutation testing of quantum programs written in QISKit. in: Proceedings of the ACM\/IEEE 44th International Conference on Software Engineering: Companion Proceedings. pp. 358\u2013359. New York, NY, USA: Association for Computing Machinery. dohttps:\/\/doi.org\/10.1145\/3510454.3528649.","DOI":"10.1145\/3510454.3528649."},{"key":"10643_CR29","doi-asserted-by":"publisher","unstructured":"Fortunato D, Campos J, Abreu R (2022c) QMutPy: A mutation testing tool for quantum algorithms and applications in qiskit. In: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 797\u2013800. New York, NY, USA: Association for Computing Machinery. doi: https:\/\/doi.org\/10.1145\/3533767.3543296","DOI":"10.1145\/3533767.3543296"},{"issue":"4","key":"10643_CR30","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2419","volume":"35","author":"A Garc\u00eda de la Barrera","year":"2023","unstructured":"Garc\u00eda de la Barrera A, Garc\u00eda-Rodr\u00edguez de Guzm\u00e1n I, Polo M, Piattini M (2023) Quantum software testing: State of the art. Journal of Software: Evolution and Process 35(4):e2419. https:\/\/doi.org\/10.1002\/smr.2419","journal-title":"Journal of Software: Evolution and Process"},{"key":"10643_CR31","doi-asserted-by":"publisher","unstructured":"Gligoric M, Badame S, Johnson R (2011) SMutant: A tool for type-sensitive mutation testing in a dynamic language. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. pp. 424\u2013427. New York, NY, USA: Association for Computing Machinery. dohttps:\/\/doi.org\/10.1145\/2025113.2025181","DOI":"10.1145\/2025113.2025181"},{"key":"10643_CR32","doi-asserted-by":"publisher","unstructured":"Gopinath R, Alipour A, Ahmed I, Jensen C, Groce A (2015) How hard does mutation analysis have to be, anyway? In: 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). pp. 216\u2013227. doi: https:\/\/doi.org\/10.1109\/ISSRE.2015.7381815.","DOI":"10.1109\/ISSRE.2015.7381815."},{"key":"10643_CR33","doi-asserted-by":"publisher","unstructured":"Gopinath, R., & Walkingshaw, E. (2017). How good are your types? using mutation analysis to evaluate the effectiveness of type annotations. In: 2017 IEEE International Conference On Software Testing, Verification And Validation Workshops (ICSTW). pp. 122\u2013127. doi: https:\/\/doi.org\/10.1109\/ICSTW.2017.28.","DOI":"10.1109\/ICSTW.2017.28."},{"key":"10643_CR34","doi-asserted-by":"publisher","unstructured":"Hietala K, Rand R, Hung S-H, Li L, Hicks M (2021) Proving Quantum Programs Correct. L.\u00a0Cohen and C.\u00a0Kaliszyk (Eds.), In: 12th International Conference On Interactive Theorem Proving (ITP 2021) (Vol.\u00a0193, pp. 21:1\u201321:19). Dagstuhl, Germany: Schloss Dagstuhl \u2013 Leibniz-Zentrum f\u00fcr Informatik. doi: https:\/\/doi.org\/10.4230\/LIPIcs.ITP.2021.21.","DOI":"10.4230\/LIPIcs.ITP.2021.21."},{"issue":"10","key":"10643_CR35","doi-asserted-by":"publisher","first-page":"576","DOI":"10.1145\/363235.363259","volume":"12","author":"CAR Hoare","year":"1969","unstructured":"Hoare CAR (1969) An axiomatic basis for computer programming. Commun. ACM 12(10):576\u2013580. https:\/\/doi.org\/10.1145\/363235.363259","journal-title":"Commun. ACM"},{"key":"10643_CR36","doi-asserted-by":"publisher","unstructured":"Honarvar S, Mousavi MR, Nagarajan R (2020) Property-based testing of quantum programs in Q#. In: Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops. pp. 430\u2013435. New York, NY, USA: Association for Computing Machinery. dohttps:\/\/doi.org\/10.1145\/3387940.3391459.","DOI":"10.1145\/3387940.3391459."},{"key":"10643_CR37","doi-asserted-by":"publisher","unstructured":"Hu J, Li N, Offutt J (2011) An analysis of OO mutation operators. In: 2011 ieee fourth international conference on software testing, verification and validation workshops. pp. 334\u2013341. doi: https:\/\/doi.org\/10.1109\/ICSTW.2011.47","DOI":"10.1109\/ICSTW.2011.47"},{"key":"10643_CR38","unstructured":"IBM (2023). Ibm quantum composer."},{"key":"10643_CR39","doi-asserted-by":"publisher","unstructured":"Jain K, Palmskog K, Celik A, Arias EJG, Gligoric M (2020) mCoq: mutation analysis for Coq verification projects. In: Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: Companion Proceedings (pp. 89\u201392). New York, NY, USA: Association for Computing Machinery.doi: https:\/\/doi.org\/10.1145\/3377812.3382156","DOI":"10.1145\/3377812.3382156"},{"issue":"10","key":"10643_CR40","doi-asserted-by":"publisher","first-page":"1379","DOI":"10.1016\/j.infsof.2009.04.016","volume":"51","author":"Y Jia","year":"2009","unstructured":"Jia Y, Harman M (2009) Higher order mutation testing. Information and Software Technology 51(10):1379\u20131393. https:\/\/doi.org\/10.1016\/j.infsof.2009.04.016","journal-title":"Information and Software Technology"},{"issue":"5","key":"10643_CR41","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1109\/TSE.2010.62","volume":"37","author":"Y Jia","year":"2011","unstructured":"Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering 37(5):649\u2013678. https:\/\/doi.org\/10.1109\/TSE.2010.62","journal-title":"IEEE Transactions on Software Engineering"},{"key":"10643_CR42","doi-asserted-by":"publisher","unstructured":"Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing? In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. pp. 654\u2013665. New York, NY, USA: Association for Computing Machinery. dohttps:\/\/doi.org\/10.1145\/2635868.2635929","DOI":"10.1145\/2635868.2635929"},{"issue":"1","key":"10643_CR43","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/npjqi.2015.18","volume":"1","author":"A Kalev","year":"2015","unstructured":"Kalev A, Kosut RL, Deutsch IH (2015) Quantum tomography protocols with positivity are compressed sensing protocols. Npj Quantum Information 1(1):1\u20136","journal-title":"Npj Quantum Information"},{"key":"10643_CR44","doi-asserted-by":"crossref","unstructured":"Kintis M, Papadakis M, Malevris N (2010) Evaluating mutation testing alternatives: A collateral experiment. In: 2010 Asia Pacific Software Engineering Conference. pp. 300\u2013309","DOI":"10.1109\/APSEC.2010.42"},{"issue":"3","key":"10643_CR45","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevResearch.4.033173","volume":"4","author":"M Kohda","year":"2022","unstructured":"Kohda M, Imai R, Kanno K, Mitarai K, Mizukami W, Nakagawa YO (2022) Quantum expectation-value estimation by computational basis sampling. Physical review research 4(3):033173","journal-title":"Physical review research"},{"issue":"1","key":"10643_CR46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3624483","volume":"5","author":"M Lewis","year":"2023","unstructured":"Lewis M, Soudjani S, Zuliani P (2023) Formal verification of quantum programs: Theory, tools, and challenges. ACM Transactions on Quantum Computing 5(1):1\u201335","journal-title":"ACM Transactions on Quantum Computing"},{"issue":"2","key":"10643_CR47","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3550488","volume":"4","author":"A Li","year":"2023","unstructured":"Li A, Stein S, Krishnamoorthy S, Ang J (2023) QASMBench: A low-level quantum benchmark suite for nisq evaluation and simulation. ACM Transactions on Quantum Computing 4(2):1\u201326. https:\/\/doi.org\/10.1145\/3550488","journal-title":"ACM Transactions on Quantum Computing"},{"key":"10643_CR48","doi-asserted-by":"publisher","unstructured":"Ma Y-S, Offutt J, Kwon Y-R (2006) MuJava: A mutation system for Java. In: Proceedings of the 28th international conference on software engineering. pp. 827\u2013830. New York, NY, USA: Association for Computing Machinery. dohttps:\/\/doi.org\/10.1145\/1134285.1134425.","DOI":"10.1145\/1134285.1134425."},{"issue":"1","key":"10643_CR49","doi-asserted-by":"publisher","first-page":"3115","DOI":"10.1038\/ncomms4115","volume":"5","author":"M Malik","year":"2014","unstructured":"Malik M, Mirhosseini M, Lavery MP, Leach J, Padgett MJ, Boyd RW (2014) Direct measurement of a 27-dimensional orbital-angular-momentum state vector. Nature communications 5(1):3115","journal-title":"Nature communications"},{"key":"10643_CR50","doi-asserted-by":"publisher","unstructured":"Mendiluze\u00a0Usandizaga E, Ali S, Arcaini P, Yue T (2022) Muskit: A mutation analysis tool for quantum software testing. In: Proceedings of the 36th ieee\/acm international conference on automated software engineering. pp. 1266\u20131270. IEEE Press. doi: https:\/\/doi.org\/10.1109\/ASE51524.2021.9678563","DOI":"10.1109\/ASE51524.2021.9678563"},{"key":"10643_CR51","doi-asserted-by":"crossref","unstructured":"Mendiluze\u00a0Usandizaga E, Ali S, Yue T, Arcaini P (2025) Supplementary material for the paper \u201cQuantum Circuit Mutants: Empirical Analysis and Recommendations\u201d. URL: https:\/\/github.com\/EnautMendi\/Quantum-Circuit-Mutants-Empirical-Evaluation","DOI":"10.1007\/s10664-025-10643-z"},{"issue":"5","key":"10643_CR52","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1109\/TSE.2014.2371458","volume":"41","author":"S Mirshokraie","year":"2015","unstructured":"Mirshokraie S, Mesbah A, Pattabiraman K (2015) Guided mutation testing for javascript web applications. IEEE Transactions on Software Engineering 41(5):429\u2013444. https:\/\/doi.org\/10.1109\/TSE.2014.2371458","journal-title":"IEEE Transactions on Software Engineering"},{"key":"10643_CR53","doi-asserted-by":"publisher","unstructured":"Mogos G (2016) Quantum random number generator vs. random number generator. In: 2016 International Conference on Communications (COMM). pp. 423\u2013426. dohttps:\/\/doi.org\/10.1109\/ICComm.2016.7528306","DOI":"10.1109\/ICComm.2016.7528306"},{"key":"10643_CR54","doi-asserted-by":"publisher","unstructured":"Muqeet A, Ali S, Arcaini P (2024) Quantum Program Testing Through Commuting Pauli Strings on IBM\u2019S Quantum Computers. In: Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. pp. 2130\u20132141. New York, NY, USA: Association for Computing Machinery. doi: https:\/\/doi.org\/10.1145\/3691620.3695275.","DOI":"10.1145\/3691620.3695275."},{"key":"10643_CR55","doi-asserted-by":"publisher","DOI":"10.1145\/3712002","author":"JM Murillo","year":"2025","unstructured":"Murillo JM, Garcia-Alonso J, Moguel E, Barzen J, Leymann F, Ali S, Wimmer M (2025) Quantum software engineering: Roadmap and challenges ahead. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3712002","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"issue":"1","key":"10643_CR56","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1145\/125489.125473","volume":"1","author":"AJ Offutt","year":"1992","unstructured":"Offutt AJ (1992) Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1(1):5\u201320. https:\/\/doi.org\/10.1145\/125489.125473","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"10643_CR57","doi-asserted-by":"publisher","unstructured":"Oliveira RA, Al\u00e9groth E, Gao Z, Memon A (2015) Definition and evaluation of mutation operators for GUI-level mutation analysis. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). pp. 1\u201310. dohttps:\/\/doi.org\/10.1109\/ICSTW.2015.7107457","DOI":"10.1109\/ICSTW.2015.7107457"},{"key":"10643_CR58","doi-asserted-by":"publisher","unstructured":"Paltenghi M, Pradel M (2024) Analyzing quantum programs with LintQ: A static analysis framework for qiskit. Proc ACM Softw Eng 1(FSE). https:\/\/doi.org\/10.1145\/3660802","DOI":"10.1145\/3660802"},{"key":"10643_CR59","doi-asserted-by":"publisher","unstructured":"Papadakis M, Henard C, Harman M, Jia Y, Le\u00a0Traon Y (2016) Threats to the validity of mutation-based test assessment. In: Proceedings of the 25th International Symposium on Software Testing and Analysis. pp. 354\u2013365. New York, NY, USA: Association for Computing Machinery. dohttps:\/\/doi.org\/10.1145\/2931037.2931040","DOI":"10.1145\/2931037.2931040"},{"key":"10643_CR60","doi-asserted-by":"publisher","unstructured":"Papadakis M, Jia Y, Harman M, Le\u00a0Traon Y (2015) Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In: 2015 IEEE\/ACM 37th Ieee International Conference on Software Engineering. Vol.\u00a01, pp. 936\u2013946. doi: https:\/\/doi.org\/10.1109\/ICSE.2015.103","DOI":"10.1109\/ICSE.2015.103"},{"key":"10643_CR61","doi-asserted-by":"publisher","unstructured":"Papadakis M, Kintis M, Zhang J, Jia Y, Le\u00a0Traon Y, Harman M (2019) Chapter six - mutation testing advances: An analysis and survey. In: A.M.\u00a0Memon (Ed.), (Vol.\u00a0112, pp. 275\u2013378). Elsevier. doi: https:\/\/doi.org\/10.1016\/bs.adcom.2018.03.015.","DOI":"10.1016\/bs.adcom.2018.03.015."},{"key":"10643_CR62","doi-asserted-by":"publisher","unstructured":"Pontolillo G, Mousavi MR (2023) A multi-lingual benchmark for property-based testing of quantum programs. In: Proceedings of the 3rd international workshop on quantum software engineering. pp. 1\u20137. New York, NY, USA: Association for Computing Machinery. doi: https:\/\/doi.org\/10.1145\/3528230.3528395","DOI":"10.1145\/3528230.3528395"},{"key":"10643_CR63","doi-asserted-by":"publisher","unstructured":"Praphamontripong U, Offutt J (2010) Applying mutation testing to web applications. In: 2010 Third International Conference on Software Testing, Verification, and Validation Workshops. pp. 132\u2013141. doi: https:\/\/doi.org\/10.1109\/ICSTW.2010.38","DOI":"10.1109\/ICSTW.2010.38"},{"key":"10643_CR64","doi-asserted-by":"publisher","unstructured":"Praphamontripong U, Offutt J, Deng L, Gu J (2016) An experimental evaluation of web mutation operators. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). pp. 102\u2013111. doi: https:\/\/doi.org\/10.1109\/ICSTW.2016.17.","DOI":"10.1109\/ICSTW.2016.17."},{"key":"10643_CR65","doi-asserted-by":"crossref","unstructured":"Quetschlich N, Burgholzer L, Wille R (2023) MQT Bench: Benchmarking software and design automation tools for quantum computing. In: Quantum, (MQT Bench is available at https:\/\/www.cda.cit.tum.de\/mqtbench\/","DOI":"10.22331\/q-2023-07-20-1062"},{"key":"10643_CR66","doi-asserted-by":"publisher","DOI":"10.1145\/3715106","author":"NCL Ramalho","year":"2025","unstructured":"Ramalho NCL, Amario de Souza H, Lordello Chaim M (2025) Testing and debugging quantum programs: The road to 2030. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3715106","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10643_CR67","doi-asserted-by":"crossref","unstructured":"Rand R, Paykin J, Zdancewic S (2018) QWIRE practice: Formal verification of quantum circuits in Coq. In: arXiv:1803.00699","DOI":"10.4204\/EPTCS.266.8"},{"key":"10643_CR68","doi-asserted-by":"publisher","unstructured":"Rao AC, Raouf A, Dhadyalla G, Pasupuleti V (2017) Mutation testing based evaluation of formal verification tools. In: 2017 International Conference on Dependable Systems and Their Applications (DSA). pp. 1\u20137. dohttps:\/\/doi.org\/10.1109\/DSA.2017.10","DOI":"10.1109\/DSA.2017.10"},{"key":"10643_CR69","doi-asserted-by":"publisher","unstructured":"Siami\u00a0Namin A, Andrews JH, Murdoch DJ (2008) Sufficient mutation operators for measuring test effectiveness. In: Proceedings of the 30th International Conference on Software Engineering. pp. 351\u2013360. New York, NY, USA: Association for Computing Machinery. doi: https:\/\/doi.org\/10.1145\/1368088.1368136","DOI":"10.1145\/1368088.1368136"},{"key":"10643_CR70","doi-asserted-by":"crossref","unstructured":"Smith BH, Williams L (2007) An empirical evaluation of the MuJava mutation operators. In: Testing: Academic and industrial conference practice and research techniques-MUTATION (TAICPART-MUTATION 2007). IEEE, pp. 193\u2013202","DOI":"10.1109\/TAIC.PART.2007.12"},{"key":"10643_CR71","doi-asserted-by":"crossref","unstructured":"Tannu SS, Qureshi MK (2019) Mitigating measurement errors in quantum computers by exploiting state-dependent bias. In: Proceedings of the 52nd annual IEEE\/ACM International Symposium on Microarchitecture. pp. 279\u2013290","DOI":"10.1145\/3352460.3358265"},{"key":"10643_CR72","doi-asserted-by":"publisher","unstructured":"Usaola MP, Rojas G, Rodr\u00edguez I, Hern\u00e1ndez S (2017) An architecture for the development of mutation operators. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). pp. 143\u2013148. doi: https:\/\/doi.org\/10.1109\/ICSTW.2017.31.","DOI":"10.1109\/ICSTW.2017.31."},{"key":"10643_CR73","doi-asserted-by":"publisher","unstructured":"Wang X, Arcaini P, Yue T, Ali S (2021a) Application of combinatorial testing to quantum programs. In: 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). pp. 179\u2013188. doi: https:\/\/doi.org\/10.1109\/QRS54544.2021.00029.","DOI":"10.1109\/QRS54544.2021.00029."},{"key":"10643_CR74","doi-asserted-by":"publisher","unstructured":"Wang X, Arcaini P, Yue T, Ali S (2021b) Generating failing test suites for quantum programs with search. In: Search-based Software Engineering. pp. 9\u201325. Cham: Springer International Publishing. doi: https:\/\/doi.org\/10.1007\/978-3-030-88106-1_2.","DOI":"10.1007\/978-3-030-88106-1_2."},{"key":"10643_CR75","doi-asserted-by":"publisher","unstructured":"Wang X, Arcaini P, Yue T, Ali S (2022) Quito: A coverage-guided test generator for quantum programs. In: Proceedings of the 36th IEEE\/ACM International Conference on Automated Software Engineering. pp. 1237\u20131241. IEEE Press. doi: https:\/\/doi.org\/10.1109\/ASE51524.2021.9678798.","DOI":"10.1109\/ASE51524.2021.9678798."},{"key":"10643_CR76","doi-asserted-by":"publisher","unstructured":"Wang X, Yu T, Arcaini P, Yue T, Ali S (2022) Mutation-based test generation for quantum programs with multi-objective search. In: Proceedings of the Genetic and Evolutionary Computation Conference. pp. 1345\u20131353. New York, NY, USA: Association for Computing Machinery. doi: https:\/\/doi.org\/10.1145\/3512290.3528869","DOI":"10.1145\/3512290.3528869"},{"key":"10643_CR77","volume-title":"Qiskit pocket guide","author":"JL Weaver","year":"2022","unstructured":"Weaver JL, Harkins FJ (2022) Qiskit pocket guide. O\u2019Reilly Media Inc, Sebastopol"},{"key":"10643_CR78","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511813948","volume-title":"Quantum measurement and control","author":"HM Wiseman","year":"2009","unstructured":"Wiseman HM, Milburn GJ (2009) Quantum measurement and control. Cambridge University Press, Cambridge"},{"issue":"3","key":"10643_CR79","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1016\/0950-5849(93)90053-6","volume":"35","author":"M Woodward","year":"1993","unstructured":"Woodward M (1993) Mutation testing-its origin and evolution. Information and Software Technology 35(3):163\u2013169. https:\/\/doi.org\/10.1016\/0950-5849(93)90053-6","journal-title":"Information and Software Technology"},{"issue":"5886","key":"10643_CR80","doi-asserted-by":"publisher","first-page":"802","DOI":"10.1038\/299802a0","volume":"299","author":"WK Wootters","year":"1982","unstructured":"Wootters WK, Zurek WH (1982) A single quantum cannot be cloned. Nature 299(5886):802\u2013803. https:\/\/doi.org\/10.1038\/299802a0","journal-title":"Nature"},{"key":"10643_CR81","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1016\/j.infsof.2016.03.002","volume":"81","author":"F Wu","year":"2017","unstructured":"Wu F, Nanavati J, Harman M, Jia Y, Krinke J (2017) Memory mutation testing. Information and Software Technology 81:97\u2013111. https:\/\/doi.org\/10.1016\/j.infsof.2016.03.002","journal-title":"Information and Software Technology"},{"key":"10643_CR82","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511813887","volume-title":"Quantum computing for computer scientists","author":"NS Yanofsky","year":"2008","unstructured":"Yanofsky NS, Mannucci MA (2008) Quantum computing for computer scientists. Cambridge University Press, Cambridge"},{"key":"10643_CR83","doi-asserted-by":"publisher","unstructured":"Yao, X., Harman, M., Jia, Y. (2014). A study of equivalent and stubborn mutation operators using human analysis of equivalence. In: Proceedings of the 36th International Conference on Software Engineering. pp. 919\u2013930. New York, NY, USA: Association for Computing Machinery. dohttps:\/\/doi.org\/10.1145\/2568225.2568265","DOI":"10.1145\/2568225.2568265"},{"key":"10643_CR84","doi-asserted-by":"publisher","unstructured":"Ye J, Xia S, Zhang F, Arcaini P, Ma L, Zhao J, Ishikawa F (2023) QuraTest: Integrating quantum specific features in quantum program testing. In:2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). pp. 1149\u20131161. doi: https:\/\/doi.org\/10.1109\/ASE56229.2023.00196","DOI":"10.1109\/ASE56229.2023.00196"},{"key":"10643_CR85","doi-asserted-by":"publisher","unstructured":"Zhang H, Yue T, Ali S, Liu C (2016) Towards mutation analysis for use cases. In: Proceedings of the ACM\/IEEE 19th International Conference on Model Driven Engineering Languages and Systems. pp. 363\u2013373. New York, NY, USA: Association for Computing Machinery. doi: https:\/\/doi.org\/10.1145\/2976767.2976784","DOI":"10.1145\/2976767.2976784"},{"key":"10643_CR86","doi-asserted-by":"publisher","unstructured":"Zhang L, Hou S-S, Hu J-J, Xie T, Mei H (2010) Is operator-based mutant selection superior to random mutant selection? In: Proceedings of the 32nd ACM\/IEEE International Conference On Software Engineering - Volume 1.ICSE \u201910. Cape Town, South Africa: Association for Computing Machinery. pp. 435\u2013444. doi: https:\/\/doi.org\/10.1145\/1806799.1806863","DOI":"10.1145\/1806799.1806863"},{"key":"10643_CR87","unstructured":"Zhao J (2020) Quantum software engineering: Landscapes and horizons. In: CoRR, abs\/2007.07047, arXiv:2007.07047"},{"key":"10643_CR88","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111805","volume":"205","author":"P Zhao","year":"2023","unstructured":"Zhao P, Miao Z, Lan S, Zhao J (2023) Bugs4Q: A benchmark of existing bugs to enable controlled testing and debugging studies for quantum programs. Journal of Systems and Software 205:111805. https:\/\/doi.org\/10.1016\/j.jss.2023.111805","journal-title":"Journal of Systems and Software"},{"key":"10643_CR89","doi-asserted-by":"publisher","unstructured":"Zhao, P., Wu, X., Li, Z., Zhao, J. (2023). QChecker: Detecting bugs in quantum programs via static analysis. 2023 IEEE\/ACM 4th International Workshop on Quantum Software Engineering (Q-SE). pp. 50\u201357. dohttps:\/\/doi.org\/10.1109\/Q-SE59154.2023.00014","DOI":"10.1109\/Q-SE59154.2023.00014"},{"key":"10643_CR90","doi-asserted-by":"crossref","unstructured":"Zhao P, Zhao J, Ma L (2021) Identifying bug patterns in quantum programs. In: 2021 IEEE\/ACM 2nd International Workshop on Quantum Software Engineering (Q-SE) (pp. 16\u201321). Los Alamitos, CA, USA: IEEE Computer Society. 10.1109\/Q-SE52541.2021.00011","DOI":"10.1109\/Q-SE52541.2021.00011"},{"key":"10643_CR91","doi-asserted-by":"publisher","unstructured":"Zhao P, Zhao J, Miao Z, Lan S (2021) Bugs4Q: A benchmark of real bugs for quantum programs. In: 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). pp. 1373\u20131376. doi: https:\/\/doi.org\/10.1109\/ASE51524.2021.9678908","DOI":"10.1109\/ASE51524.2021.9678908"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-025-10643-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-025-10643-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-025-10643-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,5]],"date-time":"2025-06-05T09:59:19Z","timestamp":1749117559000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-025-10643-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,16]]},"references-count":91,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10643"],"URL":"https:\/\/doi.org\/10.1007\/s10664-025-10643-z","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,16]]},"assertion":[{"value":"24 March 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 April 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of Interests\/Competing Interests"}}],"article-number":"100"}}