{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T03:11:56Z","timestamp":1775790716278,"version":"3.50.1"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,6,27]],"date-time":"2024-06-27T00:00:00Z","timestamp":1719446400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61832015"],"award-info":[{"award-number":["61832015"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"JSPS KAKENHI","award":["JP23H03372"],"award-info":[{"award-number":["JP23H03372"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>Quantum computing has emerged as a promising field with the potential to revolutionize various domains by harnessing the principles of quantum mechanics. As quantum hardware and algorithms continue to advance, developing high-quality quantum software has become crucial. However, testing quantum programs poses unique challenges due to the distinctive characteristics of quantum systems and the complexity of multi-subroutine programs. This article addresses the specific testing requirements of multi-subroutine quantum programs. We begin by investigating critical properties by surveying existing quantum libraries and providing insights into the challenges of testing these programs. Building upon this understanding, we focus on testing criteria and techniques based on the whole testing process perspective, spanning from unit testing to integration testing. We delve into various aspects, including IO analysis, quantum relation checking, structural testing, behavior testing, integration of subroutine pairs, and test case generation. We also introduce novel testing principles and criteria to guide the testing process. We conduct comprehensive testing on typical quantum subroutines, including diverse mutants and randomized inputs, to evaluate our proposed approach. The analysis of failures provides valuable insights into the effectiveness of our testing methodology. Additionally, we present case studies on representative multi-subroutine quantum programs, demonstrating the practical application and effectiveness of our proposed testing principles and criteria.<\/jats:p>","DOI":"10.1145\/3656339","type":"journal-article","created":{"date-parts":[[2024,4,5]],"date-time":"2024-04-05T12:00:37Z","timestamp":1712318437000},"page":"1-61","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration Testing"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-9255-9335","authenticated-orcid":false,"given":"Peixun","family":"Long","sequence":"first","affiliation":[{"name":"Institute of Software, Chinese Academy of Sciences, Beijing, China and University of the Chinese Academy of Sciences, Beijing China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8083-4352","authenticated-orcid":false,"given":"Jianjun","family":"Zhao","sequence":"additional","affiliation":[{"name":"Kyushu University, Fukuoka, Japan"}]}],"member":"320","published-online":{"date-parts":[[2024,6,27]]},"reference":[{"key":"e_1_3_3_2_2","volume-title":"Scaffold: Quantum Programming Language","author":"Abhari Ali J.","year":"2012","unstructured":"Ali J. Abhari, Arvin Faruque, Mohammad J. Dousti, Lukas Svec, Oana Catu, Amlan Chakrabati, Chen-Fu Chiang, Seth Vanderwilt, John Black, and Fred Chong. 2012. Scaffold: Quantum Programming Language. Technical Report. Department of Computer Science, Princeton University."},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3528230.3529189"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","unstructured":"Gadi Aleksandrowicz Thomas Alexander Panagiotis Barkoutsos Luciano Bello Yael Ben-Haim David Bucher Francisco Jose Cabrera-Hern\u00e1ndez Jorge Carballo-Franquis Adrian Chen Chun-Fu Chen Jerry M. Chow Antonio D. C\u00f3rcoles-Gonzales Abigail J. Cross Andrew Cross Juan Cruz-Benito Chris Culver Salvador De La Puente Gonz\u00e1lez Enrique De La Torre Delton Ding Eugene Dumitrescu Ivan Duran Pieter Eendebak Mark Everitt Ismael Faro Sertage Albert Frisch Andreas Fuhrer Jay Gambetta Borja Godoy Gago Juan Gomez-Mosquera Donny Greenberg Ikko Hamamura Vojtech Havlicek Joe Hellmers \u0141ukasz Herok Hiroshi Horii Shaohan Hu Takashi Imamichi Toshinari Itoko Ali Javadi-Abhari Naoki Kanazawa Anton Karazeev Kevin Krsulich Peng Liu Yang Luh Yunho Maeng Manoel Marques Francisco Jose Mart\u00edn-Fern\u00e1ndez Douglas T. McClure David McKay Srujan Meesala Antonio Mezzacapo Nikolaj Moll Diego Moreda Rodr\u00edguez Giacomo Nannicini Paul Nation Pauline Ollitrault Lee James O\u2019Riordan Hanhee Paik Jes\u00fas P\u00e9rez Anna Phan Marco Pistoia Viktor Prutyanov Max Reuter Julia Rice Abd\u00f3n Rodr\u00edguez Davila Raymond Harry Putra Rudy Mingi Ryu Ninad Sathaye Chris Schnabel Eddie Schoute Kanav Setia Yunong Shi Adenilton Silva Yukio Siraichi Seyon Sivarajah John A. Smolin Mathias Soeken Hitomi Takahashi Ivano Tavernelli Charles Taylor Pete Taylour Kenso Trabing Matthew Treinish Wes Turner Desiree Vogt-Lee Christophe Vuillot Jonathan A. Wildstrom Jessica Wilson Erick Winston Christopher Wood Stephen Wood Stefan W\u00f6rner Ismail Yunus Akhalwaya and Christa Zoufal. 2019. Qiskit: An Open-Source Framework for Quantum Computing. Retrieved April 12 2024 from 10.5281\/zenodo.2562111","DOI":"10.5281\/zenodo.2562111"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00014"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3412451.3428499"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/CMPASS.1994.318466"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1017\/9781316771273"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.5555\/1540612"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1137\/S0097539796302452"},{"key":"e_1_3_3_11_2","article-title":"Quantum state discrimination","author":"Barnett Stephen M.","year":"2008","unstructured":"Stephen M. Barnett and Sarah Croke. 2008. Quantum state discrimination. arXiv preprint arXiv:0810.1970v1 (2008).","journal-title":"arXiv preprint arXiv:0810.1970v1"},{"key":"e_1_3_3_12_2","volume-title":"Test-Driven Development: By Example","author":"Beck Kent","year":"2003","unstructured":"Kent Beck. 2003. Test-Driven Development: By Example. Addison-Wesley Professional."},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1088\/2058-9565\/ab4eb5"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3385412.3386007"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.5555\/870802"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.87.167902"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICC.1994.368821"},{"key":"e_1_3_3_18_2","doi-asserted-by":"crossref","unstructured":"Tony Cai Donggyu Kim Yazhen Wang Ming Yuan and Harrison H. Zhou. 2016. Optimal large-scale quantum state tomography with Pauli measurements. arXiv:1603.07559 (2016).","DOI":"10.1214\/15-AOS1382"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.22331\/q-2020-03-26-248"},{"key":"e_1_3_3_20_2","article-title":"Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01","author":"Chen T. Y.","year":"1998","unstructured":"T. Y. Chen, S. C. Cheung, and S. M. Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01. Hong Kong University of Science and Technology.","journal-title":"Hong Kong University of Science and Technology."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1080\/09500349708231894"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.605761"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.1994.341392"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.127.100501"},{"issue":"1","key":"e_1_3_3_25_2","first-page":"Article 12, 25","article-title":"Ket Quantum Programming","volume":"18","author":"Rosa Evandro Chagas Ribeiro Da","year":"2021","unstructured":"Evandro Chagas Ribeiro Da Rosa and Rafael De Santiago. 2021. Ket Quantum Programming. ACM Journal on Emerging Technologies in Computing Systems 18, 1 (2021), Article 12, 25 pages.","journal-title":"ACM Journal on Emerging Technologies in Computing Systems"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.5555\/1243380"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1098\/rspa.1985.0070"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.88.217901"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/Q-SE52541.2021.00008"},{"key":"e_1_3_3_30_2","volume-title":"An Exploratory Study on the Usage of Quantum Programming Languages","author":"Ferreira Felipe Cavalcanti","year":"2022","unstructured":"Felipe Cavalcanti Ferreira. 2022. An Exploratory Study on the Usage of Quantum Programming Languages. Ph. D. Dissertation. Repositorio da Universidade de Lisboa."},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.106.230501"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TQE.2022.3195061"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3543296"},{"key":"e_1_3_3_34_2","first-page":"e2419","article-title":"Quantum software testing: State of the art","author":"Barrera Antonio Garc\u00eda de la","year":"2021","unstructured":"Antonio Garc\u00eda de la Barrera, Ignacio Garc\u00eda-Rodr\u00edguez de Guzm\u00e1n, Macario Polo, and Mario Piattini. 2021. Quantum software testing: State of the art. Journal of Software: Evolution and Process 35, 4 (2021), e2419.","journal-title":"Journal of Software: Evolution and Process"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11831-019-09372-6"},{"key":"e_1_3_3_36_2","unstructured":"Google AI Quantum Team. 2018. Cirq. Retrieved April 12 2024 from https:\/\/github.com\/quantumlib\/Cirq"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-38986-3_10"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.105.150401"},{"key":"e_1_3_3_39_2","article-title":"Creating superpositions that correspond to efficiently integrable probability distributions","author":"Grover Lov","year":"2002","unstructured":"Lov Grover and Terry Rudolph. 2002. Creating superpositions that correspond to efficiently integrable probability distributions. arXiv preprint arXiv:quant-ph\/0208112 (2002).","journal-title":"arXiv preprint arXiv:quant-ph\/0208112"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/237814.237866"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TQE.2023.3275868"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.103.150502"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391459"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3582016.3582030"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3307650.3322213"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/Q-SE52541.2021.00013"},{"key":"e_1_3_3_47_2","unstructured":"E. Knill. 1995. Approximation by quantum circuits. arxiv:quant-ph\/9508006 [quant-ph] (1995)."},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.5555\/270146"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428218"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378488"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA51647.2021.00025"},{"key":"e_1_3_3_52_2","article-title":"Equivalence, identity, and unitarity checking in black-box testing of quantum programs","author":"Long Peixun","year":"2023","unstructured":"Peixun Long and Jianjun Zhao. 2023. Equivalence, identity, and unitarity checking in black-box testing of quantum programs. arXiv preprint arXiv:2307.01481 (2023).","journal-title":"arXiv preprint arXiv:2307.01481"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678563"},{"key":"e_1_3_3_54_2","doi-asserted-by":"crossref","unstructured":"Ying Mingsheng. 2016. Foundations of Quantum Programming. Morgan Kaufmann San Francisco CA USA.","DOI":"10.1016\/B978-0-12-802306-8.00004-5"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00023"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377816.3381731"},{"key":"e_1_3_3_57_2","article-title":"On testing and debugging quantum software","author":"Miranskyy Andriy","year":"2021","unstructured":"Andriy Miranskyy, Lei Zhang, and Javad Doliskani. 2021. On testing and debugging quantum software. arXiv preprint arXiv:2103.09172 (2021).","journal-title":"arXiv preprint arXiv:2103.09172"},{"key":"e_1_3_3_58_2","article-title":"Noise-aware quantum software testing","author":"Muqeet Asmar","year":"2023","unstructured":"Asmar Muqeet, Tao Yue, Shaukat Ali, and Paolo Arcaini. 2023. Noise-aware quantum software testing. arXiv preprint arXiv:2306.16992 (2023).","journal-title":"arXiv preprint arXiv:2306.16992"},{"key":"e_1_3_3_59_2","volume-title":"Quantum Computing: Progress and Prospects","author":"Sciences Engineering National Academies of","year":"2019","unstructured":"Engineering National Academies of Sciences, Engineering, and Medicine, and the Intelligence Community Studies Board. 2019. Quantum Computing: Progress and Prospects, Emily Grumbling and Mark Horowitz (Eds.). National Academies Press."},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511976667"},{"key":"e_1_3_3_61_2","volume-title":"Structured Quantum Programming","author":"\u00d6mer Bernhard","year":"2003","unstructured":"Bernhard \u00d6mer. 2003. Structured Quantum Programming. Ph. D. Dissertation. Institute for Theoretical Physics, Technical University of Vienna."},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/Q-SE52541.2021.00009"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3392183"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.22331\/q-2018-08-06-79"},{"key":"e_1_3_3_65_2","unstructured":"Raphael Seidel Sebastian Bock Nikolay Tcholtchev and Manfred Hauswirth. 2022. Qrisp: A framework for compilable high-level programming of gate-based quantum computers. In Proceedings of the 27th ACM SIGPLAN International Conference on Functional Programming (ICFP\u201922)."},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1137\/S0036144598347011"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183895.3183901"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.40.2847"},{"key":"e_1_3_3_69_2","article-title":"QuanFuzz: Fuzz testing of quantum program","author":"Wang Jiyuan","year":"2018","unstructured":"Jiyuan Wang, Ming Gao, Yu Jiang, Jianguang Lou, Yue Gao, Dongmei Zhang, and Jiaguang Sun. 2018. QuanFuzz: Fuzz testing of quantum program. arXiv preprint arXiv:1810.10310 (2018).","journal-title":"arXiv preprint arXiv:1810.10310"},{"key":"e_1_3_3_70_2","doi-asserted-by":"crossref","unstructured":"Jiyuan Wang Qian Zhang Guoqing Harry Xu and Miryung Kim. 2021. QDiff: Differential testing of quantum software stacks. In Proceedings of the 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201921).","DOI":"10.1109\/ASE51524.2021.9678792"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS54544.2021.00029"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-88106-1_2"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3512290.3528869"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00196"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.physleta.2005.12.097"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.75.062306"},{"key":"e_1_3_3_77_2","article-title":"Quantum software engineering: Landscapes and horizons","author":"Zhao Jianjun","year":"2020","unstructured":"Jianjun Zhao. 2020. Quantum software engineering: Landscapes and horizons. arXiv preprint arXiv:2007.07047 (2020).","journal-title":"arXiv preprint arXiv:2007.07047"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678908"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3656339","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3656339","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:48:59Z","timestamp":1750286939000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3656339"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,27]]},"references-count":77,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3656339"],"URL":"https:\/\/doi.org\/10.1145\/3656339","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,27]]},"assertion":[{"value":"2023-06-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-07","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}