{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T02:36:17Z","timestamp":1784169377378,"version":"3.55.0"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA1","license":[{"start":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T00:00:00Z","timestamp":1651190400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["851895"],"award-info":[{"award-number":["851895"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2022,4,29]]},"abstract":"<jats:p>The interest in quantum computing is growing, and with it, the importance of software platforms to develop quantum programs. Ensuring the correctness of such platforms is important, and it requires a thorough understanding of the bugs they typically suffer from. To address this need, this paper presents the first in-depth study of bugs in quantum computing platforms. We gather and inspect a set of 223 real-world bugs from 18 open-source quantum computing platforms. Our study shows that a significant fraction of these bugs (39.9%) are quantum-specific, calling for dedicated approaches to prevent and find them. The bugs are spread across various components, but quantum-specific bugs occur particularly often in components that represent, compile, and optimize quantum programming abstractions. Many quantum-specific bugs manifest through unexpected outputs, rather than more obvious signs of misbehavior, such as crashes. Finally, we present a hierarchy of recurrent bug patterns, including ten novel, quantum-specific patterns. Our findings not only show the importance and prevalence bugs in quantum computing platforms, but they help developers to avoid common mistakes and tool builders to tackle the challenge of preventing, finding, and fixing these bugs.<\/jats:p>","DOI":"10.1145\/3527330","type":"journal-article","created":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T15:42:03Z","timestamp":1651246923000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":58,"title":["Bugs in Quantum computing platforms: an empirical study"],"prefix":"10.1145","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2266-453X","authenticated-orcid":false,"given":"Matteo","family":"Paltenghi","sequence":"first","affiliation":[{"name":"University of Stuttgart, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1623-498X","authenticated-orcid":false,"given":"Michael","family":"Pradel","sequence":"additional","affiliation":[{"name":"University of Stuttgart, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,4,29]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Overview on Quantum Initiatives Worldwide - Update Mid","year":"2021","unstructured":"2021. Overview on Quantum Initiatives Worldwide - Update Mid 2021 . https:\/\/www.qureca.com\/overview-on-quantum-initiatives-worldwide-update-mid-2021\/. 2021. Overview on Quantum Initiatives Worldwide - Update Mid 2021. https:\/\/www.qureca.com\/overview-on-quantum-initiatives-worldwide-update-mid-2021\/."},{"key":"e_1_2_1_2_1","unstructured":"2021. Qiskit\/Qiskit. https:\/\/github.com\/Qiskit\/qiskit.  2021. Qiskit\/Qiskit. https:\/\/github.com\/Qiskit\/qiskit."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2012.28"},{"key":"e_1_2_1_4_1","volume-title":"6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.","author":"Allamanis Miltiadis","year":"2018","unstructured":"Miltiadis Allamanis , Marc Brockschmidt , and Mahmoud Khademi . 2018 . Learning to Represent Programs with Graphs . In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to Represent Programs with Graphs. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360585"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178372.3179521"},{"key":"e_1_2_1_7_1","first-page":"2640","volume-title":"Proceedings of the 38th International Conference on Machine Learning. PMLR, 780\u2013791","author":"Berabi Berkay","year":"2021","unstructured":"Berkay Berabi , Jingxuan He , Veselin Raychev , and Martin Vechev . 2021 . TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer . In Proceedings of the 38th International Conference on Machine Learning. PMLR, 780\u2013791 . issn: 2640 - 3498 Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. 2021. TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 780\u2013791. issn:2640-3498"},{"key":"e_1_2_1_8_1","volume-title":"Carsten Blank, Alain Delgado, Soran Jahangiri, Keri McKiernan, Johannes Jakob Meyer, Zeyue Niu, Antal Sz\u00e1va, and Nathan Killoran.","author":"Bergholm Ville","year":"2020","unstructured":"Ville Bergholm , Josh Izaac , Maria Schuld , Christian Gogolin , M. Sohaib Alam , Shahnawaz Ahmed , Juan Miguel Arrazola , Carsten Blank, Alain Delgado, Soran Jahangiri, Keri McKiernan, Johannes Jakob Meyer, Zeyue Niu, Antal Sz\u00e1va, and Nathan Killoran. 2020 . PennyLane: Automatic Differentiation of Hybrid Quantum-Classical Computations . arXiv:1811.04968 [physics, physics:quant-ph], Feb., arxiv:1811.04968. Ville Bergholm, Josh Izaac, Maria Schuld, Christian Gogolin, M. Sohaib Alam, Shahnawaz Ahmed, Juan Miguel Arrazola, Carsten Blank, Alain Delgado, Soran Jahangiri, Keri McKiernan, Johannes Jakob Meyer, Zeyue Niu, Antal Sz\u00e1va, and Nathan Killoran. 2020. PennyLane: Automatic Differentiation of Hybrid Quantum-Classical Computations. arXiv:1811.04968 [physics, physics:quant-ph], Feb., arxiv:1811.04968."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3385412.3386007"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Jos\u00e9 Campos and Andr\u00e9 Souto. 2021. QBugs: A Collection of Reproducible Bugs in Quantum Algorithms and a Supporting Infrastructure to Enable Controlled Quantum Software Testing and Debugging Experiments. arXiv:2103.16968 [cs] March arxiv:2103.16968.  Jos\u00e9 Campos and Andr\u00e9 Souto. 2021. QBugs: A Collection of Reproducible Bugs in Quantum Algorithms and a Supporting Infrastructure to Enable Controlled Quantum Software Testing and Debugging Experiments. arXiv:2103.16968 [cs] March arxiv:2103.16968.","DOI":"10.1109\/Q-SE52541.2021.00013"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3363562"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/502034.502042"},{"key":"e_1_2_1_13_1","volume-title":"Gambetta","author":"Cross Andrew W.","year":"2017","unstructured":"Andrew W. Cross , Lev S. Bishop , John A. Smolin , and Jay M . Gambetta . 2017 . Open Quantum Assembly Language . arXiv:1707.03429 [quant-ph], July, arxiv:1707.03429. Andrew W. Cross, Lev S. Bishop, John A. Smolin, and Jay M. Gambetta. 2017. Open Quantum Assembly Language. arXiv:1707.03429 [quant-ph], July, arxiv:1707.03429."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5182845"},{"key":"e_1_2_1_15_1","volume-title":"8th International Conference on Learning Representations, ICLR 2020","author":"Dinella Elizabeth","year":"2020","unstructured":"Elizabeth Dinella , Hanjun Dai , Ziyang Li , Mayur Naik , Le Song , and Ke Wang . 2020 . Hoppity: Learning Graph Transformations to Detect and Fix Bugs in Programs . In 8th International Conference on Learning Representations, ICLR 2020 , Addis Ababa, Ethiopia , April 26-30, 2020. OpenReview.net. https:\/\/openreview.net\/forum?id=SJeqs6EFvB Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. 2020. Hoppity: Learning Graph Transformations to Detect and Fix Bugs in Programs. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https:\/\/openreview.net\/forum?id=SJeqs6EFvB"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416576"},{"key":"e_1_2_1_17_1","unstructured":"Edward Farhi Jeffrey Goldstone and Sam Gutmann. 2014. A Quantum Approximate Optimization Algorithm. arXiv:1411.4028 [quant-ph] Nov. arxiv:1411.4028.  Edward Farhi Jeffrey Goldstone and Sam Gutmann. 2014. A Quantum Approximate Optimization Algorithm. arXiv:1411.4028 [quant-ph] Nov. arxiv:1411.4028."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0208561"},{"key":"e_1_2_1_19_1","unstructured":"Doug Finke. 2021. Relative Popularity of Different Quantum Programming Platforms - Quantum Computing Report. https:\/\/web.archive.org\/web\/20210619213740\/https:\/\/quantumcomputingreport.com\/relative-popularity-of-different-quantum-programming-platforms\/.  Doug Finke. 2021. Relative Popularity of Different Quantum Programming Platforms - Quantum Computing Report. https:\/\/web.archive.org\/web\/20210619213740\/https:\/\/quantumcomputingreport.com\/relative-popularity-of-different-quantum-programming-platforms\/."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428287"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s42354-021-0330-z"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462177"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462177"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/237814.237866"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2961111.2962602"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428201"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.103.150502"},{"key":"e_1_2_1_28_1","volume-title":"Global Relational Models of Source Code. In 8th International Conference on Learning Representations, ICLR 2020","author":"Hellendoorn Vincent J.","year":"2020","unstructured":"Vincent J. Hellendoorn , Charles Sutton , Rishabh Singh , Petros Maniatis , and David Bieber . 2020 . Global Relational Models of Source Code. In 8th International Conference on Learning Representations, ICLR 2020 , Addis Ababa, Ethiopia , April 26-30, 2020. OpenReview.net. https:\/\/openreview.net\/forum?id=B1lnbRNtwr Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. Global Relational Models of Source Code. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https:\/\/openreview.net\/forum?id=B1lnbRNtwr"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3434318"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.4230\/OASIcs.PLATEAU.2018.4"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3307650.3322213"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2345156.2254075"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 37th International Conference on Machine Learning, ICML 2020","volume":"5121","author":"Kanade Aditya","year":"2020","unstructured":"Aditya Kanade , Petros Maniatis , Gogul Balakrishnan , and Kensen Shi . 2020 . Learning and Evaluating Contextual Embedding of Source Code . In Proceedings of the 37th International Conference on Machine Learning, ICML 2020 , 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research , Vol. 119). PMLR, 5110\u2013 5121 . https:\/\/doi.org\/10.48550\/ARXIV.2001.00059 Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and Evaluating Contextual Embedding of Source Code. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 5110\u20135121. https:\/\/doi.org\/10.48550\/ARXIV.2001.00059"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387491"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1088\/2058-9565\/abe567"},{"key":"e_1_2_1_37_1","volume-title":"Max H. Gordon, Yousef Hindy, Aaron Robertson, Purva Thakre, Nathan Shammah, and William J. Zeng.","author":"LaRose Ryan","year":"2021","unstructured":"Ryan LaRose , Andrea Mari , Sarah Kaiser , Peter J. Karalekas , Andre A. Alves , Piotr Czarnik , Mohamed El Mandouh , Max H. Gordon, Yousef Hindy, Aaron Robertson, Purva Thakre, Nathan Shammah, and William J. Zeng. 2021 . Mitiq : A Software Package for Error Mitigation on Noisy Quantum Computers . arXiv:2009.04417 [quant-ph], Aug., arxiv:2009.04417. Ryan LaRose, Andrea Mari, Sarah Kaiser, Peter J. Karalekas, Andre A. Alves, Piotr Czarnik, Mohamed El Mandouh, Max H. Gordon, Yousef Hindy, Aaron Robertson, Purva Thakre, Nathan Shammah, and William J. Zeng. 2021. Mitiq: A Software Package for Error Mitigation on Noisy Quantum Computers. arXiv:2009.04417 [quant-ph], Aug., arxiv:2009.04417."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594334"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858965.2814319"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318162"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1538788.1538814"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428218"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380345"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2006.28"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1346281.1346323"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2013.18"},{"key":"e_1_2_1_47_1","first-page":"100","article-title":"Differential Testing for Software","volume":"10","author":"McKeeman William M.","year":"1998","unstructured":"William M. McKeeman . 1998 . Differential Testing for Software . Digit. Tech. J. , 10 , 1 (1998), 100 \u2013 107 . William M. McKeeman. 1998. Differential Testing for Software. Digit. Tech. J., 10, 1 (1998), 100\u2013107.","journal-title":"Digit. Tech. J."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428198"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360569"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378477"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454040"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468539"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884848"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133928"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884829"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468591"},{"key":"e_1_2_1_58_1","unstructured":"Yunong Shi Runzhou Tao Xupeng Li Ali Javadi-Abhari Andrew W. Cross Frederic T. Chong and Ronghui Gu. 2020. CertiQ: A Mostly-automated Verification of a Realistic Quantum Compiler. arXiv:1908.08963 [quant-ph] Nov. arxiv:1908.08963.  Yunong Shi Runzhou Tao Xupeng Li Ali Javadi-Abhari Andrew W. Cross Frederic T. Chong and Ronghui Gu. 2020. CertiQ: A Mostly-automated Verification of a Realistic Quantum Compiler. arXiv:1908.08963 [quant-ph] Nov. arxiv:1908.08963."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0036144598347011"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931074"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.22331\/q-2021-10-06-559"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183895.3183901"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454029"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2676726.2676995"},{"key":"e_1_2_1_65_1","volume-title":"International Conference on Learning Representations.","author":"Vasic Marko","year":"2018","unstructured":"Marko Vasic , Aditya Kanade , Petros Maniatis , David Bieber , and Rishabh Singh . 2018 . Neural Program Repair by Jointly Learning to Localize and Repair . In International Conference on Learning Representations. Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh. 2018. Neural Program Repair by Jointly Learning to Localize and Repair. In International Conference on Learning Representations."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678792"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00138"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1038\/299802a0"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993316.1993532"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454061"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3140587.3062379"},{"key":"e_1_2_1_72_1","unstructured":"Jianjun Zhao. 2021. Quantum Software Engineering: Landscapes and Horizons. arXiv:2007.07047 [quant-ph] Dec. arxiv:2007.07047.  Jianjun Zhao. 2021. Quantum Software Engineering: Landscapes and Horizons. arXiv:2007.07047 [quant-ph] Dec. arxiv:2007.07047."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/Q-SE52541.2021.00011"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678908"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380922"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527330","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3527330","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:53Z","timestamp":1750191533000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527330"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,29]]},"references-count":75,"journal-issue":{"issue":"OOPSLA1","published-print":{"date-parts":[[2022,4,29]]}},"alternative-id":["10.1145\/3527330"],"URL":"https:\/\/doi.org\/10.1145\/3527330","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,29]]},"assertion":[{"value":"2022-04-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}