{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T05:16:26Z","timestamp":1779340586000,"version":"3.51.4"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T00:00:00Z","timestamp":1724803200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T00:00:00Z","timestamp":1724803200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001775","name":"University of Technology Sydney","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001775","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Quantum Mach. Intell."],"published-print":{"date-parts":[[2024,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Interpretability of artificial intelligence (AI) methods, particularly deep neural networks, is of great interest. This heightened focus stems from the widespread use of AI-backed systems. These systems, often relying on intricate neural architectures, can exhibit behavior that is challenging to explain and comprehend. The interpretability of such models is a crucial component of building trusted systems. Many methods exist to approach this problem, but they do not apply straightforwardly to the quantum setting. Here, we explore the interpretability of quantum neural networks using local model-agnostic interpretability measures commonly utilized for classical neural networks. Following this analysis, we generalize a classical technique called LIME, introducing Q-LIME, which produces explanations of quantum neural networks. A feature of our explanations is the delineation of the region in which data samples have been given a random label, likely subjects of inherently random quantum measurements. We view this as a step toward understanding how to build responsible and accountable quantum AI models.<\/jats:p>","DOI":"10.1007\/s42484-024-00191-y","type":"journal-article","created":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T13:02:31Z","timestamp":1724850151000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["On the interpretability of quantum neural networks"],"prefix":"10.1007","volume":"6","author":[{"given":"Lirand\u00eb","family":"Pira","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chris","family":"Ferrie","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,8,28]]},"reference":[{"issue":"7","key":"191_CR1","doi-asserted-by":"publisher","first-page":"e0130140","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach S, Binder A, Montavon G, Klauschen F, M\u00fcller KR, Samek W (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7):e0130140","journal-title":"PLoS ONE"},{"key":"191_CR2","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1038\/nature23474","volume":"549","author":"J Biamonte","year":"2016","unstructured":"Biamonte J, Wittek P, Pancotti N, Rebentrost P, Wiebe N, Lloyd S (2016) Quantum Machine Learning. Nature 549:195\u201320. https:\/\/doi.org\/10.1038\/nature23474","journal-title":"Nature"},{"key":"191_CR3","doi-asserted-by":"crossref","unstructured":"Burge I, Barbeau M, Garcia-Alfaro J (2023) A quantum algorithm for shapley value estimation. arXiv preprint arXiv:2301.04727","DOI":"10.1109\/QCE57702.2023.00024"},{"issue":"9","key":"191_CR4","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1038\/s42254-021-00348-9","volume":"3","author":"M Cerezo","year":"2021","unstructured":"Cerezo M, Arrasmith A, Babbush R, Benjamin SC, Endo S, Fujii K, McClean JR, Mitarai K, Yuan X, Cincio L, Coles PJ (2021) Variational quantum algorithms. Nature Rev Phys 3(9):625\u201364. https:\/\/doi.org\/10.1038\/s42254-021-00348-9","journal-title":"Variational quantum algorithms. Nature Rev Phys"},{"key":"191_CR5","doi-asserted-by":"publisher","unstructured":"Cerezo M, Verdon G, Huang HY, Cincio L, Coles P (2022) Challenges and opportunities in quantum machine learning. Nature Comput Sci. https:\/\/doi.org\/10.1038\/s43588-022-00311-3","DOI":"10.1038\/s43588-022-00311-3"},{"key":"191_CR6","unstructured":"Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608"},{"issue":"1","key":"191_CR7","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1145\/3359786","volume":"63","author":"M Du","year":"2019","unstructured":"Du M, Liu N, Hu X (2019) Techniques for interpretable machine learning. Commun ACM 63(1):68\u201377","journal-title":"Commun ACM"},{"key":"191_CR8","doi-asserted-by":"publisher","unstructured":"Farhi E, Neven H (2018) Classification with quantum neural networks on near term processors. arXiv preprint https:\/\/doi.org\/10.48550\/arXiv.1802.06002","DOI":"10.48550\/arXiv.1802.06002"},{"key":"191_CR9","doi-asserted-by":"crossref","unstructured":"Fisher RA (1936) The use of multiple measurements in taxonomic problems. https:\/\/archive.ics.uci.edu\/ml\/datasets\/iris","DOI":"10.1111\/j.1469-1809.1936.tb02137.x"},{"key":"191_CR10","unstructured":"Ghorbani A, Abid A, Zou J (2018) Interpretation of neural networks is fragile"},{"key":"191_CR11","doi-asserted-by":"crossref","unstructured":"Goldstein A, Kapelner A, Bleich J, Pitkin E (2014) Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation","DOI":"10.1080\/10618600.2014.907095"},{"key":"191_CR12","unstructured":"Goodfellow I, Bengio Y, Courville A (2016) Deep Learning MIT Press,. http:\/\/www.deeplearningbook.org"},{"issue":"5","key":"191_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3236009","volume":"51","author":"R Guidotti","year":"2018","unstructured":"Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D (2018) A survey of methods for explaining black box models. ACM Comput Surv (CSUR) 51(5):1\u201342","journal-title":"ACM Comput Surv (CSUR)"},{"key":"191_CR14","doi-asserted-by":"publisher","unstructured":"Heese R, Gerlach T, M\u00fccke S, M\u00fcller S, Jakobs M, Piatkowski N (2023) Explaining quantum circuits with shapley values: Towards explainable quantum machine learning. arXiv preprint https:\/\/doi.org\/10.48550\/arXiv.2301.09138","DOI":"10.48550\/arXiv.2301.09138"},{"key":"191_CR15","unstructured":"IBM (2021) Qiskit: An Open-Source Framework for Quantum Computing. https:\/\/qiskit.org\/. Accessed on 1 May 2023"},{"key":"191_CR16","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) In: Advances in Neural Information Processing Systems, vol\u00a025, (eds). by Pereira F, Burges C, Bottou L, Weinberger K. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2012\/file\/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf"},{"issue":"3","key":"191_CR17","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton ZC (2018) The mythos of model interpretability. Queue 16(3):31\u201357","journal-title":"Queue"},{"key":"191_CR18","unstructured":"Lundberg SM, Lee SI (2017) In: Advances in Neural Information Processing Systems, vol.\u00a030 Curran Associates, Inc., https:\/\/proceedings.neurips.cc\/paper\/2017\/file\/8a20a8621978632d76c43dfd28b67767-Paper.pdf"},{"key":"191_CR19","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2021.11.001","author":"L McCoy","year":"2022","unstructured":"McCoy L, Brenna C, Chen S, Vold K, Das S (2022) Believing in black boxes: Must machine learning in healthcare be explainable to be evidence-based? Journal of Clinical Epidemiolog. https:\/\/doi.org\/10.1016\/j.jclinepi.2021.11.001","journal-title":"Journal of Clinical Epidemiolog"},{"key":"191_CR20","doi-asserted-by":"publisher","unstructured":"Mercaldo F, Ciaramella G, Iadarola G, Storto M, Martinelli F, Santone A (2022) Towards explainable quantum machine learning for mobile malware detection and classification. Appl Sci 12.https:\/\/doi.org\/10.3390\/app122312025","DOI":"10.3390\/app122312025"},{"key":"191_CR21","volume-title":"Machine Learning","author":"TM Mitchell","year":"1997","unstructured":"Mitchell TM (1997) Machine Learning, 1st edn. McGraw-Hill Inc, USA","edition":"1"},{"key":"191_CR22","unstructured":"Molnar C (2022) Interpretable Machine Learning, 2nd edn. https:\/\/christophm.github.io\/interpretable-ml-book"},{"key":"191_CR23","doi-asserted-by":"publisher","unstructured":"Molnar C, Casalicchio G, Bischl B (2020) In: ECML PKDD 2020 Workshops Springer International Publishing, pp 417\u2013431. https:\/\/doi.org\/10.1007\/978-3-030-65965-3_28","DOI":"10.1007\/978-3-030-65965-3_28"},{"key":"191_CR24","doi-asserted-by":"crossref","unstructured":"Olah C, Mordvintsev A, Tyka M (2017) Feature visualization: How neural networks build up their understanding of images. Distill 2(11)","DOI":"10.23915\/distill.00007"},{"key":"191_CR25","unstructured":"Pira L, Ferrie C (2023) Interpret QNN: Explicability and Inexplicability in the Interpretation of Quantum Neural Networks. https:\/\/github.com\/lirandepira\/interpret-qnn. Accessed on 24 July 2023"},{"key":"191_CR26","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining association for computing machinery, New York, NY, USA, KDD \u201916, pp 1135\u2013114","DOI":"10.1145\/2939672.2939778"},{"key":"191_CR27","doi-asserted-by":"crossref","unstructured":"Roscher R, Bohn B, Duarte MF, Garcke J (2020) Explainable machine learning for scientific insights and discoveries. Ieee Access 8:42,200\u201342,216","DOI":"10.1109\/ACCESS.2020.2976199"},{"key":"191_CR28","doi-asserted-by":"crossref","unstructured":"Rudin C, Chen C, Chen Z, Huang H, Semenova L, Zhong C (2021) Interpretable machine learning: Fundamental principles and 10 grand challenges","DOI":"10.1214\/21-SS133"},{"key":"191_CR29","volume-title":"Artificial Intelligence: A Modern Approach","author":"S Russell","year":"2010","unstructured":"Russell S, Norvig P (2010) Artificial Intelligence: A Modern Approach, 3rd edn. Prentice Hall","edition":"3"},{"key":"191_CR30","doi-asserted-by":"publisher","unstructured":"Sarkar A (2022) Is explainable AI a race against model complexity? arXiv preprint https:\/\/doi.org\/10.48550\/arXiv.2205.10119","DOI":"10.48550\/arXiv.2205.10119"},{"key":"191_CR31","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-96424-9","author":"M Schuld","year":"2018","unstructured":"Schuld M, Petruccione F (2018) Supervised Learning with Quantum Computers. Quantum Science and Technology Springer International Publishin. https:\/\/doi.org\/10.1007\/978-3-319-96424-9","journal-title":"Quantum Science and Technology Springer International Publishin"},{"key":"191_CR32","doi-asserted-by":"crossref","unstructured":"Schuld M, Petruccione F (2021) Machine Learning with Quantum Computers Springer International Publishing","DOI":"10.1007\/978-3-030-83098-4"},{"key":"191_CR33","doi-asserted-by":"publisher","unstructured":"Simonyan K, Vedaldi A, Zisserman A (2013) Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint https:\/\/doi.org\/10.48550\/arXiv.1312.6034","DOI":"10.48550\/arXiv.1312.6034"},{"issue":"4","key":"191_CR34","first-page":"482","volume":"19","author":"JC Spall","year":"1998","unstructured":"Spall JC (1998) An overview of the simultaneous perturbation method for efficient optimization. J Hopkins APL Tech Dig 19(4):482\u2013492","journal-title":"J Hopkins APL Tech Dig"},{"key":"191_CR35","doi-asserted-by":"publisher","unstructured":"Steinm\u00fcller P, Schulz T, Graf F, Herr D (2022) explainable ai for quantum machine learning. arXiv preprint https:\/\/doi.org\/10.48550\/arXiv.2211.01441","DOI":"10.48550\/arXiv.2211.01441"},{"key":"191_CR36","unstructured":"Weitz G, Pira L, Ferrie C, Combes J (2023) Sub-universal variational circuits for combinatorial optimization problem"},{"key":"191_CR37","unstructured":"Yosinski J, Clune J, Nguyen A, Fuchs T, Lipson H (2015) In: Proceedings of the 32nd international conference on machine learning (ICML), pp 1582\u20131591"},{"key":"191_CR38","doi-asserted-by":"crossref","unstructured":"Youssry A, Paz-Silva GA, Ferrie C (2020) Characterization and control of open quantum systems beyond quantum noise spectroscopy. npj Quantum Information 6:95","DOI":"10.1038\/s41534-020-00332-8"},{"key":"191_CR39","doi-asserted-by":"crossref","unstructured":"Youssry A, Paz-Silva GA, Ferrie C (2023) Noise detection with spectator qubits and quantum feature engineering. New J Phys 25:073004","DOI":"10.1088\/1367-2630\/ace2e4"},{"key":"191_CR40","doi-asserted-by":"publisher","unstructured":"Youssry A, Yang Y, Chapman RJ, Haylock B, Lenzini F, Lobino M, Peruzzo A (2023) Experimental graybox quantum system identification and control. arXiv preprint https:\/\/doi.org\/10.48550\/arXiv.2206.12201","DOI":"10.48550\/arXiv.2206.12201"},{"key":"191_CR41","doi-asserted-by":"crossref","unstructured":"Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. European conference on computer vision pp 818\u2013833","DOI":"10.1007\/978-3-319-10590-1_53"},{"issue":"5","key":"191_CR42","doi-asserted-by":"publisher","first-page":"726","DOI":"10.1109\/tetci.2021.3100641","volume":"5","author":"Y Zhang","year":"2021","unstructured":"Zhang Y, Tino P, Leonardis A, Tang K (2021) A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence 5(5):726\u20137. https:\/\/doi.org\/10.1109\/tetci.2021.3100641","journal-title":"IEEE Transactions on Emerging Topics in Computational Intelligence"}],"container-title":["Quantum Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42484-024-00191-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42484-024-00191-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42484-024-00191-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,23]],"date-time":"2024-12-23T16:07:30Z","timestamp":1734970050000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42484-024-00191-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,28]]},"references-count":42,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,12]]}},"alternative-id":["191"],"URL":"https:\/\/doi.org\/10.1007\/s42484-024-00191-y","relation":{},"ISSN":["2524-4906","2524-4914"],"issn-type":[{"value":"2524-4906","type":"print"},{"value":"2524-4914","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,28]]},"assertion":[{"value":"8 January 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 August 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 August 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no Conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"52"}}