{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T16:51:41Z","timestamp":1778259101378,"version":"3.51.4"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,11,10]],"date-time":"2021-11-10T00:00:00Z","timestamp":1636502400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,10]],"date-time":"2021-11-10T00:00:00Z","timestamp":1636502400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100014374","name":"Universitat Polit\u00e8cnica de Catalunya","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100014374","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Intell Robot Syst"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>A deep reinforcement learning approach for solving the quadrotor path following and obstacle avoidance problem is proposed in this paper. The problem is solved with two agents: one for the path following task and another one for the obstacle avoidance task. A novel structure is proposed, where the action computed by the obstacle avoidance agent becomes the state of the path following agent. Compared to traditional deep reinforcement learning approaches, the proposed method allows to interpret the training process outcomes, is faster and can be safely trained on the real quadrotor. Both agents implement the Deep Deterministic Policy Gradient algorithm. The path following agent was developed in a previous work. The obstacle avoidance agent uses the information provided by a low-cost LIDAR to detect obstacles around the vehicle. Since LIDAR has a narrow field-of-view, an approach for providing the agent with a memory of the previously seen obstacles is developed. A detailed description of the process of defining the state vector, the reward function and the action of this agent is given. The agents are programmed in python\/tensorflow and are trained and tested in the RotorS\/gazebo platform. Simulations results prove the validity of the proposed approach.<\/jats:p>","DOI":"10.1007\/s10846-021-01491-2","type":"journal-article","created":{"date-parts":[[2021,11,10]],"date-time":"2021-11-10T16:51:17Z","timestamp":1636563077000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Quadrotor Path Following and Reactive Obstacle Avoidance with Deep Reinforcement Learning"],"prefix":"10.1007","volume":"103","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8822-2681","authenticated-orcid":false,"given":"Bartomeu","family":"Rub\u00ed","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bernardo","family":"Morcego","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ramon","family":"P\u00e9rez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,11,10]]},"reference":[{"key":"1491_CR1","doi-asserted-by":"crossref","unstructured":"Caicedo, J.C., Lazebnik, S.: Active object localization with deep reinforcement learning, in. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2488\u20132496 (2015)","DOI":"10.1109\/ICCV.2015.286"},{"key":"1491_CR2","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","volume":"529","author":"D Silver","year":"2016","unstructured":"Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T.P., Leach, M., Kavukcuoglu, K., Graepel, T., Hassabis, D.: Mastering the game of go with deep neural networks and tree search. Nature 529, 484\u2013489 (2016)","journal-title":"Nature"},{"key":"1491_CR3","doi-asserted-by":"crossref","unstructured":"Yu, R., Shi, Z., Huang, C., Li, T., Ma, Q.: Deep reinforcement learning based optimal trajectory tracking control of autonomous underwater vehicle. In: 2017 36th Chinese Control Conference (CCC), pp. 4958\u20134965 (2017)","DOI":"10.23919\/ChiCC.2017.8028138"},{"key":"1491_CR4","doi-asserted-by":"crossref","unstructured":"Tuyen, L.P., Chung, T.: Controlling bicycle using deep deterministic policy gradient algorithm. In: 2017 14TH International Conference on Ubiquitous Robots and Ambien Intelligence (Urai), pp. 413\u2013417 (2017)","DOI":"10.1109\/URAI.2017.7992765"},{"issue":"3","key":"1491_CR5","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1109\/JAS.2016.7508798","volume":"3","author":"L Li","year":"2016","unstructured":"Li, L., Lv, Y., Wang, F.: Traffic signal timing via deep reinforcement learning. IEEE\/CAA J. Automatica Sinica 3(3), 247\u2013254 (2016)","journal-title":"IEEE\/CAA J. Automatica Sinica"},{"key":"1491_CR6","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s10514-020-09951-8","volume":"45","author":"B Rub\u00ed","year":"2020","unstructured":"Rub\u00ed, B., Morcego, B., P\u00e9rez, R.: Deep reinforcement learning for quadrotor path following with adaptive velocity. Auton. Robot. 45, 119\u2013134 (2020)","journal-title":"Auton. Robot."},{"issue":"1","key":"1491_CR7","first-page":"65","volume":"57","author":"C Goerzen","year":"2009","unstructured":"Goerzen, C., Kong, Z., Mettler, B.: A survey of motion planning algorithms from the perspective of autonomous uav guidance. J. Intell. Robot. Syst. 57(1), 65 (2009)","journal-title":"J. Intell. Robot. Syst."},{"key":"1491_CR8","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/j.arcontrol.2019.10.001","volume":"48","author":"S Huang","year":"2019","unstructured":"Huang, S., Teo, R.S.H., Tan, K.K.: Collision avoidance of multi unmanned aerial vehicles: A review. Ann. Rev. Control 48, 147\u2013164 (2019)","journal-title":"Ann. Rev. Control"},{"key":"1491_CR9","doi-asserted-by":"publisher","first-page":"105139","DOI":"10.1109\/ACCESS.2020.3000064","volume":"8","author":"JN Yasin","year":"2020","unstructured":"Yasin, J.N., Mohamed, S.A.S., Haghbayan, M., Heikkonen, J., Tenhunen, H., Plosila, J.: Unmanned aerial vehicles (uavs): Collision avoidance systems and approaches. IEEE Access 8, 105139\u2013105155 (2020)","journal-title":"IEEE Access"},{"key":"1491_CR10","doi-asserted-by":"crossref","unstructured":"Minguez, J., Lamiraux, F., Laumond, J. -P.: Motion Planning and Obstacle Avoidance. In: Springer Handbook Of Robotics, pp 827\u2013852. Springer, Berlin (2008)","DOI":"10.1007\/978-3-540-30301-5_36"},{"key":"1491_CR11","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1016\/j.robot.2017.10.009","volume":"100","author":"Z Ma","year":"2018","unstructured":"Ma, Z., Wang, C., Niu, Y., Wang, X., Shen, L.: A saliency-based reinforcement learning approach for a uav to avoid flying obstacles. Robot. Auton. Syst. 100, 108\u2013118 (2018)","journal-title":"Robot. Auton. Syst."},{"key":"1491_CR12","doi-asserted-by":"crossref","unstructured":"Eslamiat, H., Li, Y., Wang, N., Sanyal, A.K., Qiu, Q.: Autonomous waypoint planning, optimal trajectory generation and nonlinear tracking control for multi-rotor uavs. In: 2019 18TH European Control Conference (ECC), pp. 2695\u20132700 (2019)","DOI":"10.23919\/ECC.2019.8795855"},{"key":"1491_CR13","doi-asserted-by":"crossref","unstructured":"Yang, S., Meng, Z., Chen, X., Xie, R.: Real-time obstacle avoidance with deep reinforcement learning three-dimensional autonomous obstacle avoidance for uav. In: Proceeding of the 2019 International Conference on Robotics, Intelligent Control and Artificial Intelligence (2019)","DOI":"10.1145\/3366194.3366251"},{"key":"1491_CR14","doi-asserted-by":"crossref","unstructured":"Wu, K., Esfahani, M.A., Yuan, S., Wang, H.: Depth-based obstacle avoidance through deep reinforcement learning. In: Proceedings of the 5th International Conference on Mechatronics and Robotics Engineering, pp. 102\u2013106 (2019)","DOI":"10.1145\/3314493.3314495"},{"key":"1491_CR15","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1007\/s10846-019-01073-3","volume":"98","author":"C Yan","year":"2020","unstructured":"Yan, C., Xiang, X., Wang, C.: Towards real-time path planning through deep reinforcement learning for a uav in dynamic environments. J. Intell. Robot. Syst 98, 297\u2013309 (2020)","journal-title":"J. Intell. Robot. Syst"},{"key":"1491_CR16","doi-asserted-by":"crossref","unstructured":"Han, X., Wang, J., Xue, J., Zhang, Q.: Intelligent decision-making for 3-dimensional dynamic obstacle avoidance of uav based on deep reinforcement learning. In: 2019 11th International Conference on Wireless Communications and Signal Processing (Wcsp), pp. 1\u20136 (2019)","DOI":"10.1109\/WCSP.2019.8928110"},{"key":"1491_CR17","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1016\/j.neucom.2017.06.066","volume":"272","author":"Y Cheng","year":"2018","unstructured":"Cheng, Y., Zhang, W.: Concise deep reinforcement learning obstacle avoidance for underactuated unmanned marine vessels. Neurocomputing 272, 63\u201373 (2018)","journal-title":"Neurocomputing"},{"key":"1491_CR18","doi-asserted-by":"crossref","unstructured":"Lu, X., Cao, Y., Zhao, Z., Yan, Y.: Deep reinforcement learning based collision avoidance algorithm for differential drive robot. In: Intelligent Robotics and Applications ICIRA 2018 (2018)","DOI":"10.1007\/978-3-319-97586-3_17"},{"key":"1491_CR19","doi-asserted-by":"publisher","first-page":"29064","DOI":"10.1109\/ACCESS.2020.2971780","volume":"8","author":"B Li","year":"2020","unstructured":"Li, B., Wu, Y.: Path planning for uav ground target tracking via deep reinforcement learning. IEEE Access 8, 29064\u201329074 (2020)","journal-title":"IEEE Access"},{"issue":"3","key":"1491_CR20","doi-asserted-by":"publisher","first-page":"2124","DOI":"10.1109\/TVT.2018.2890773","volume":"68","author":"C Wang","year":"2019","unstructured":"Wang, C., Wang, J., Shen, Y., Zhang, X.: Autonomous navigation of uavs in large-scale complex environments: A deep reinforcement learning approach. IEEE Trans. Veh. Technol. 68(3), 2124\u20132136 (2019)","journal-title":"IEEE Trans. Veh. Technol."},{"key":"1491_CR21","doi-asserted-by":"crossref","unstructured":"Sampedro, C., Bavle, H., Rodriguez-Ramos, A., de la Puente, P., Campoy, P.: Laser-based reactive navigation for multirotor aerial robots using deep reinforcement learning. In: 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1024\u20131031 (2018)","DOI":"10.1109\/IROS.2018.8593706"},{"key":"1491_CR22","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M.A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518, 529\u2013533 (2015)","journal-title":"Nature"},{"key":"1491_CR23","unstructured":"Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 (2015)"},{"key":"1491_CR24","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. In: 2016 International Conference on Learning Representations (ICLR) (2016)"},{"key":"1491_CR25","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.: Deterministic policy gradient algorithms. In: Proceedings of the 31st International Conference on Machine Learning, pp. 387\u2013395 (2014)"},{"key":"1491_CR26","doi-asserted-by":"crossref","unstructured":"Rub\u00ed, B., Ruiz, A., P\u00e9rez, R., Morcego, B.: Path-flyer: A benchmark of quadrotor path following algorithms. In: 2019 15th IEEE International Conference on Controol and Automation (2019)","DOI":"10.1109\/ICCA.2019.8899563"},{"issue":"8","key":"1491_CR27","doi-asserted-by":"publisher","first-page":"1362","DOI":"10.1109\/TAC.2007.902731","volume":"52","author":"AP Aguiar","year":"2007","unstructured":"Aguiar, A.P., Hespanha, J.P.: Trajectory-tracking and path-following of underactuated autonomous vehicles with parametric modeling uncertainty. IEEE Trans. Autom. Control 52(8), 1362\u20131379 (2007)","journal-title":"IEEE Trans. Autom. Control"},{"key":"1491_CR28","doi-asserted-by":"crossref","unstructured":"Cabecinhas, D., Cunha, R., Silvestre, C.: Rotorcraft path following control for extended flight envelope coverage. In: Proceedings of the 48th IEEE Conference on Decision and Control, Held Jointly with the 28th Chinese Control Conference (CDC\/CCC 2009), pp. 3460\u20133465 (2009)","DOI":"10.1109\/CDC.2009.5400665"},{"issue":"3","key":"1491_CR29","doi-asserted-by":"publisher","first-page":"598","DOI":"10.1016\/j.automatica.2007.06.030","volume":"44","author":"AP Aguiar","year":"2008","unstructured":"Aguiar, A.P., Hespanha, J.P., Kokotovic, P.V.: Performance limitations in reference tracking and path following for nonlinear systems. Automatica 44(3), 598\u2013610 (2008)","journal-title":"Automatica"},{"key":"1491_CR30","doi-asserted-by":"crossref","unstructured":"Kaminer, I., Yakimenko, O., Pascoal, A., Ghabcheloo, R.: Path generation, path following and coordinated control for time-critical missions of multiple UAVs. In: Proceedings of the American Control Conference, vol. 1\u201312, p 4906 (2006)","DOI":"10.1109\/ACC.2006.1657498"},{"key":"1491_CR31","first-page":"595","volume-title":"RotorS\u2014A Modular Gazebo MAV Simulator Framework","author":"F Furrer","year":"2016","unstructured":"Furrer, F., Burri, M., Achtelik, M., Siegwart, R.: RotorS\u2014A Modular Gazebo MAV Simulator Framework, pp 595\u2013625. Springer International Publishing, Cham (2016)"},{"issue":"4","key":"1491_CR32","doi-asserted-by":"publisher","first-page":"3852","DOI":"10.1109\/LRA.2018.2856922","volume":"3","author":"S Stev\u0161i\u0107","year":"2018","unstructured":"Stev\u0161i\u0107, S., N\u00e4geli, T., Alonso-Mora, J., Hilliges, O.: Sample efficient learning of path following and obstacle avoidance behavior for quadrotors. IEEE Robot. Autom. Lett. 3(4), 3852\u20133859 (2018)","journal-title":"IEEE Robot. Autom. Lett."},{"key":"1491_CR33","first-page":"6","volume":"12","author":"J Park","year":"2020","unstructured":"Park, J., Cho, N.: Collision avoidance of hexacopter uav based on lidar data in dynamic environment. Remote. Sens. 12, 6 (2020)","journal-title":"Remote. Sens."},{"key":"1491_CR34","doi-asserted-by":"crossref","unstructured":"Al-Kaff, A., Garc\u00eda, F., Mart\u00edn, D., de la Escalera, A., Armingol, J.M.: Obstacle detection and avoidance system based on monocular camera and size expansion algorithm for uavs. Sensors (Basel, Switzerland) 17 (2017)","DOI":"10.3390\/s17051061"},{"key":"1491_CR35","doi-asserted-by":"crossref","unstructured":"Amrouche, M., Marinho, T., Stipanovi\u0107, D.: Vision based collision avoidance for multi-agent systems using avoidance functions. In: 2020 European Control Conference (ECC), pp. 1683\u20131688 (2020)","DOI":"10.23919\/ECC51009.2020.9143739"}],"container-title":["Journal of Intelligent &amp; Robotic Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10846-021-01491-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10846-021-01491-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10846-021-01491-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T06:04:37Z","timestamp":1640844277000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10846-021-01491-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,10]]},"references-count":35,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1491"],"URL":"https:\/\/doi.org\/10.1007\/s10846-021-01491-2","relation":{},"ISSN":["0921-0296","1573-0409"],"issn-type":[{"value":"0921-0296","type":"print"},{"value":"1573-0409","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,10]]},"assertion":[{"value":"2 September 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 August 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}],"article-number":"62"}}