{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,22]],"date-time":"2025-02-22T00:00:22Z","timestamp":1740182422841,"version":"3.37.3"},"reference-count":23,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T00:00:00Z","timestamp":1722902400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T00:00:00Z","timestamp":1722902400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"name":"FCT Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","award":["LA\/P\/0063\/2020"],"award-info":[{"award-number":["LA\/P\/0063\/2020"]}]},{"name":"IBEX","award":["PTDC\/CC1-COM\/4280\/2021"],"award-info":[{"award-number":["PTDC\/CC1-COM\/4280\/2021"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>This research explores the trainability of Parameterized Quantum Circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and the mapping of these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad6830","type":"journal-article","created":{"date-parts":[[2024,7,26]],"date-time":"2024-07-26T22:57:32Z","timestamp":1722034652000},"page":"035037","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Trainability issues in quantum policy gradients"],"prefix":"10.1088","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6659-9277","authenticated-orcid":true,"given":"Andr\u00e9","family":"Sequeira","sequence":"first","affiliation":[]},{"given":"Luis","family":"Paulo Santos","sequence":"additional","affiliation":[]},{"given":"Luis","family":"Soares Barbosa","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2024,8,6]]},"reference":[{"key":"mlstad6830bib1","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1038\/s43588-021-00084-1","article-title":"The power of quantum neural networks","volume":"1","author":"Abbas","year":"2021","journal-title":"Nat. Comput. Sci."},{"key":"mlstad6830bib2","doi-asserted-by":"publisher","first-page":"558","DOI":"10.22331\/q-2021-10-05-558","article-title":"Effect of barren plateaus on gradient-free optimization","volume":"5","author":"Arrasmith","year":"2021","journal-title":"Quantum"},{"article-title":"PennyLane: automatic differentiation of hybrid quantum-classical computations","year":"2022","author":"Bergholm","key":"mlstad6830bib3"},{"key":"mlstad6830bib4","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.103.L030401","article-title":"Universal variational quantum computation","volume":"103","author":"Biamonte","year":"2021","journal-title":"Phys. Rev. A"},{"key":"mlstad6830bib5","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1038\/s42254-021-00348-9","article-title":"Variational quantum algorithms","volume":"3","author":"Cerezo","year":"2021","journal-title":"Nat. Rev. Phys."},{"key":"mlstad6830bib6","doi-asserted-by":"publisher","first-page":"1791","DOI":"10.1038\/s41467-021-21728-w","article-title":"Cost function dependent barren plateaus in shallow parametrized quantum circuits","volume":"12","author":"Cerezo","year":"2021","journal-title":"Nat. Commun."},{"article-title":"Does provable absence of barren plateaus imply classical simulability? or, why we need to rethink variational quantum computing","year":"2024","author":"Cerezo","key":"mlstad6830bib7"},{"key":"mlstad6830bib8","doi-asserted-by":"publisher","first-page":"141007","DOI":"10.1109\/ACCESS.2020.3010470","article-title":"Variational quantum circuits for deep reinforcement learning","volume":"8","author":"Chen","year":"2020","journal-title":"IEEE Access"},{"key":"mlstad6830bib9","doi-asserted-by":"crossref","DOI":"10.22331\/q-2023-11-29-1191","article-title":"Quantum deep hedging","author":"Cherrat","year":"2023"},{"article-title":"Variational quantum policies for reinforcement learning","year":"2021","author":"Jerbi","key":"mlstad6830bib10"},{"article-title":"Quantum policy gradient algorithms","year":"2022","author":"Jerbi","key":"mlstad6830bib11"},{"key":"mlstad6830bib12","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.128.050402","article-title":"On the practical usefulness of the hardware efficient ansatz","volume":"128","author":"Leone","year":"2022","journal-title":"Phys. Rev. Lett."},{"key":"mlstad6830bib13","doi-asserted-by":"publisher","first-page":"4812","DOI":"10.1038\/s41467-018-07090-4","article-title":"Barren plateaus in quantum neural network training landscapes","volume":"9","author":"McClean","year":"2018","journal-title":"Nat. Commun."},{"article-title":"Quantum policy gradient algorithm with optimized action decoding","year":"2023","author":"Meyer","key":"mlstad6830bib14"},{"article-title":"A unified theory of barren plateaus for deep parametrized quantum circuits","year":"2023","author":"Ragone","key":"mlstad6830bib15"},{"article-title":"Trainability barriers and opportunities in quantum generative modeling","year":"2023","author":"Rudolph","key":"mlstad6830bib16"},{"key":"mlstad6830bib17","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.99.032331","article-title":"Evaluating analytic gradients on quantum hardware","volume":"99","author":"Schuld","year":"2019","journal-title":"Phys. Rev. A"},{"article-title":"Proximal policy optimization algorithms","year":"2017","author":"Schulman","key":"mlstad6830bib18"},{"key":"mlstad6830bib19","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1007\/s42484-023-00101-8","article-title":"Policy gradients using variational quantum circuits","volume":"5","author":"Sequeira","year":"2023","journal-title":"Quantum Mach. Intell."},{"key":"mlstad6830bib20","doi-asserted-by":"publisher","first-page":"720","DOI":"10.22331\/q-2022-05-24-720","article-title":"Quantum agents in the gym: a variational quantum algorithm for deep q-learning","volume":"6","author":"Skolik","year":"2022","journal-title":"Quantum"},{"article-title":"Subtleties in the trainability of quantum machine learning models","year":"2021","author":"Thanasilp","key":"mlstad6830bib21"},{"key":"mlstad6830bib22","doi-asserted-by":"publisher","DOI":"10.1088\/1751-8121\/abfac7","article-title":"On barren plateaus and cost function locality in variational quantum algorithms","volume":"54","author":"Uvarov","year":"2021","journal-title":"J. Phys. A: Math. Theor."},{"key":"mlstad6830bib23","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Mach. Learn."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T09:42:11Z","timestamp":1722937331000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad6830"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,6]]},"references-count":23,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,8,6]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad6830","relation":{},"ISSN":["2632-2153"],"issn-type":[{"type":"electronic","value":"2632-2153"}],"subject":[],"published":{"date-parts":[[2024,8,6]]},"assertion":[{"value":"Trainability issues in quantum policy gradients","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2024-04-17","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-07-26","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-08-06","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}