{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T21:42:56Z","timestamp":1792791776148,"version":"4.5.0"},"reference-count":51,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"82","content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2023,9,13]]},"abstract":"<jats:p>A central question in robotics is how to design a control system for an agile mobile robot. This paper studies this question systematically, focusing on a challenging setting: autonomous drone racing. We show that a neural network controller trained with reinforcement learning (RL) outperformed optimal control (OC) methods in this setting. We then investigated which fundamental factors have contributed to the success of RL or have limited OC. Our study indicates that the fundamental advantage of RL over OC is not that it optimizes its objective better but that it optimizes a better objective. OC decomposes the problem into planning and control with an explicit intermediate representation, such as a trajectory, that serves as an interface. This decomposition limits the range of behaviors that can be expressed by the controller, leading to inferior control performance when facing unmodeled effects. In contrast, RL can directly optimize a task-level objective and can leverage domain randomization to cope with model uncertainty, allowing the discovery of more robust control responses. Our findings allowed us to push an agile drone to its maximum performance, achieving a peak acceleration greater than 12 times the gravitational acceleration and a peak velocity of 108 kilometers per hour. Our policy achieved superhuman control within minutes of training on a standard workstation. This work presents a milestone in agile robotics and sheds light on the role of RL and OC in robot control.<\/jats:p>","DOI":"10.1126\/scirobotics.adg1462","type":"journal-article","created":{"date-parts":[[2023,9,13]],"date-time":"2023-09-13T13:58:37Z","timestamp":1694613517000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":244,"title":["Reaching the limit in autonomous racing: Optimal control versus reinforcement learning"],"prefix":"10.1126","volume":"8","author":[{"given":"Yunlong","family":"Song","sequence":"first","affiliation":[{"name":"University of Zurich, Zurich, Switzerland."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angel","family":"Romero","sequence":"additional","affiliation":[{"name":"University of Zurich, Zurich, Switzerland."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthias","family":"M\u00fcller","sequence":"additional","affiliation":[{"name":"Intel Labs, Jackson, WY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vladlen","family":"Koltun","sequence":"additional","affiliation":[{"name":"Intel Labs, Jackson, WY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Davide","family":"Scaramuzza","sequence":"additional","affiliation":[{"name":"University of Zurich, Zurich, Switzerland."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2016.2578706"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","unstructured":"M. Maurer J.\u00a0C.\u00a0Gerdes B.\u00a0Lenz H.\u00a0Winner Autonomous Driving: Technical Legal and Social Aspects (Springer Nature 2016).","DOI":"10.1007\/978-3-662-48847-8"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1002\/rob.21887"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tre.2020.102090"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aaw1975"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06419-4"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2939-8"},{"key":"e_1_3_2_9_2","doi-asserted-by":"crossref","unstructured":"T. Tzanetos M.\u00a0Aung J.\u00a0Balaram H.\u00a0F.\u00a0Grip J.\u00a0T.\u00a0Karras T.\u00a0K.\u00a0Canham G.\u00a0Kubiak J.\u00a0Anderson G.\u00a0Merewether M.\u00a0Starch M.\u00a0Pauken S.\u00a0Cappucci M.\u00a0Chase M.\u00a0Golombek O.\u00a0Toupet M.\u00a0C.\u00a0Smart S.\u00a0Dawson E.\u00a0B.\u00a0Ramirez J.\u00a0Lam R.\u00a0Stern N.\u00a0Chahat J.\u00a0Ravich R.\u00a0Hogg B.\u00a0Pipenberg M.\u00a0Keennon K.\u00a0H.\u00a0Williford Ingenuity Mars helicopter: From technology demonstration to extraterrestrial scout in Proceedings of the 2022 IEEE Aerospace Conference (AERO) (IEEE 2022) pp.\u00a01\u201319.","DOI":"10.1109\/AERO53065.2022.9843428"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"H.\u00a0F.\u00a0Grip D.\u00a0Conway D.\u00a0Conway J.\u00a0Lam N.\u00a0Williams M.\u00a0P.\u00a0Golombek R.\u00a0Brockers M.\u00a0Mischna M.\u00a0R.\u00a0Cacan Flying a helicopter on mars: How ingenuity\u2019s flights were planned executed and analyzed in Proceedings of the 2022 IEEE Aerospace Conference (AERO) (IEEE 2022) pp.\u00a01\u201317.","DOI":"10.1109\/AERO53065.2022.9843813"},{"key":"e_1_3_2_11_2","unstructured":"E. Arthur Jr. J.-C. Ho Applied Optimal Control: Optimization Estimation and Control (Hemisphere 1975)."},{"key":"e_1_3_2_12_2","unstructured":"D. Bertsekas Dynamic Programming and Optimal Control: Volume I (Athena Scientific 2012) vol. 1."},{"key":"e_1_3_2_13_2","unstructured":"R.\u00a0S.\u00a0Sutton A.\u00a0G.\u00a0Barto Reinforcement Learning: An Introduction (MIT Press 2018)."},{"key":"e_1_3_2_14_2","unstructured":"D. Bertsekas Reinforcement Learning and Optimal Control (Athena Scientific 2019)."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1724-z"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-04301-9"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-015-9479-3"},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","unstructured":"G. Bledt M.\u00a0J.\u00a0Powell M.\u00a0J.\u00a0Powell B.\u00a0Katz J.\u00a0D.\u00a0Carlo P.\u00a0M.\u00a0Wensing S.\u00a0Kim MIT Cheetah 3: Design and control of a robust dynamic quadruped robot in Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE 2018) pp.\u00a02245\u20132252.","DOI":"10.1109\/IROS.2018.8593885"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2018.2800124"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","unstructured":"D. Falanga P.\u00a0Foehn P.\u00a0Lu D.\u00a0Scaramuzza PAMPC: Perception-aware model predictive control for quadrotors in Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE 2018) pp.\u00a01\u20138.","DOI":"10.1109\/IROS.2018.8593739"},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","unstructured":"M. Neunert C.\u00a0De Crousaz F.\u00a0Furrer M.\u00a0Kamel F.\u00a0Farshidian R.\u00a0Siegwart J.\u00a0Buchli Fast nonlinear model predictive control for unified trajectory optimization and tracking in Proceedings of the 2016 IEEE International Conference On Robotics and Automation (ICRA) (IEEE 2016) pp.\u00a01398\u20131404.","DOI":"10.1109\/ICRA.2016.7487274"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2022.3173711"},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","unstructured":"P.-B. Wieber R.\u00a0Tedrake S.\u00a0Kuindersma Modeling and control of legged robots in Springer Handbook of Robotics (Springer 2016) pp.\u00a01203\u20131234.","DOI":"10.1007\/978-3-319-32552-1_48"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1177\/02783649221102473"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abc5986"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abk2822"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11370-018-00271-6"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.55417\/fr.2022042"},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","unstructured":"P. Foehn D.\u00a0Brescianini E.\u00a0Kaufmann T.\u00a0Cieslewski M.\u00a0Gehrig M.\u00a0Muglikar D.\u00a0Scaramuzza Alphapilot: Autonomous drone racing Robotics: Science and Systems (RSS) (2020).","DOI":"10.15607\/RSS.2020.XVI.081"},{"key":"e_1_3_2_32_2","unstructured":"C. De Wagter F.\u00a0Paredes-Vall\u00e9s N.\u00a0Sheth G.\u00a0de Croon The artificial intelligence behind the winning entry to the 2019 AI Robotic Racing Competition. arXiv:2109.14985 [cs.RO] (30 September 2021)."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abh1221"},{"key":"e_1_3_2_34_2","unstructured":"Y. Song S.\u00a0Naji E.\u00a0Kaufmann A.\u00a0Loquercio D.\u00a0Scaramuzza Flightmare: A flexible quadrotor simulator in Conference on Robot Learning (PMLR 2020) pp.1147\u20131157."},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"L. Bauersfeld E.\u00a0Kaufmann P.\u00a0Foehn S.\u00a0Sun D.\u00a0Scaramuzza NeuroBEM: Hybrid Aerodynamic Quadrotor Model (RSS: Robotics Science and Systems 2021).","DOI":"10.15607\/RSS.2021.XVII.042"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abl6259"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2020.12.073"},{"key":"e_1_3_2_38_2","unstructured":"J. Schulman F.\u00a0Wolski P.\u00a0Dhariwal A.\u00a0Radford O.\u00a0Klimov Proximal policy optimization algorithms arXiv:1707.06347 [cs.LG] (28 August 2017)."},{"key":"e_1_3_2_39_2","first-page":"1","article-title":"Stable-baselines3: Reliable reinforcement learning implementations","volume":"22","author":"Raffin A.","year":"2021","unstructured":"A.\u00a0Raffin, A.\u00a0Hill, A.\u00a0Gleave, A.\u00a0Kanervisto, M.\u00a0Ernestus, N.\u00a0Dormann, Stable-baselines3: Reliable reinforcement learning implementations. J. Mach. Learn. Res. 22, 1\u20138 (2021).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","unstructured":"E. Kaufmann L.\u00a0Bauersfeld D.\u00a0Scaramuzza A benchmark comparison of learned control policies for agile quadrotor flight in Proceedings of the 2022 International Conference on Robotics and Automation (ICRA) (IEEE 2022) pp.\u00a010504\u201310510.","DOI":"10.1109\/ICRA46639.2022.9811564"},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","unstructured":"D. Mellinger V.\u00a0Kumar Minimum snap trajectory generation and control for quadrotors in Proceedings of the 2011 IEEE International Conference on Robotics and Automation (IEEE 2011) pp.\u00a02520\u20132525.","DOI":"10.1109\/ICRA.2011.5980409"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364911434236"},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","unstructured":"C. Richter A.\u00a0Bry N.\u00a0Roy Polynomial trajectory planning for aggressive quadrotor flight in dense indoor environments in Proceedings of the International Symposium of Robotics Research (ISRR) (Sage Publications 2013) pp.\u00a0649\u2013666.","DOI":"10.1007\/978-3-319-28872-7_37"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-012-9282-3"},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","unstructured":"W.\u00a0V.\u00a0Loock G.\u00a0Pipeleers J.\u00a0Swevers Time-optimal quadrotor flight in Proceedings of the 2013 European Control Conference (ECC) (IEEE 2013) pp.\u00a01788\u20131792 (2013).","DOI":"10.23919\/ECC.2013.6669253"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCST.2017.2709268"},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","unstructured":"C. de Wagter F.\u00a0Paredes-Vall\u00e9s N.\u00a0Sheth G.\u00a0D.\u00a0Croon The sensing state-estimation and control behind the winning entry to the 2019 artificial intelligence robotic racing competition. Field Robot . 2 1263\u20131290 (2022).","DOI":"10.55417\/fr.2022042"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2019.2942989"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2017.2720851"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3301273"},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Y. Song M.\u00a0Steinweg E.\u00a0Kaufmann D.\u00a0Scaramuzza Autonomous drone racing with deep reinforcement learning in Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE 2021) pp.\u00a01205\u20131212.","DOI":"10.1109\/IROS51168.2021.9636053"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2019.2930489"}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.adg1462","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,16]],"date-time":"2024-01-16T07:00:02Z","timestamp":1705388402000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.adg1462"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,13]]},"references-count":51,"journal-issue":{"issue":"82","published-print":{"date-parts":[[2023,9,13]]}},"alternative-id":["10.1126\/scirobotics.adg1462"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.adg1462","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,13]]},"assertion":[{"value":"2022-12-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-05-23","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eadg1462"}}