{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T10:30:28Z","timestamp":1779273028789,"version":"3.51.4"},"reference-count":32,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2020,4,18]],"date-time":"2020-04-18T00:00:00Z","timestamp":1587168000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Counter-drone technology by using artificial intelligence (AI) is an emerging technology and it is rapidly developing. Considering the recent advances in AI, counter-drone systems with AI can be very accurate and efficient to fight against drones. The time required to engage with the target can be less than other methods based on human intervention, such as bringing down a malicious drone by a machine-gun. Also, AI can identify and classify the target with a high precision in order to prevent a false interdiction with the targeted object. We believe that counter-drone technology with AI will bring important advantages to the threats coming from some drones and will help the skies to become safer and more secure. In this study, a deep reinforcement learning (DRL) architecture is proposed to counter a drone with another drone, the learning drone, which will autonomously avoid all kind of obstacles inside a suburban neighborhood environment. The environment in a simulator that has stationary obstacles such as trees, cables, parked cars, and houses. In addition, another non-malicious third drone, acting as moving obstacle inside the environment was also included. In this way, the learning drone is trained to detect stationary and moving obstacles, and to counter and catch the target drone without crashing with any other obstacle inside the neighborhood. The learning drone has a front camera and it can capture continuously depth images. Every depth image is part of the state used in DRL architecture. There are also scalar state parameters such as velocities, distances to the target, distances to some defined geofences and track, and elevation angles. The state image and scalars are processed by a neural network that joints the two state parts into a unique flow. Moreover, transfer learning is tested by using the weights of the first full-trained model. With transfer learning, one of the best jump-starts achieved higher mean rewards (close to 35 more) at the beginning of training. Transfer learning also shows that the number of crashes during training can be reduced, with a total number of crashed episodes reduced by 65%, when all ground obstacles are included.<\/jats:p>","DOI":"10.3390\/s20082320","type":"journal-article","created":{"date-parts":[[2020,4,21]],"date-time":"2020-04-21T04:49:38Z","timestamp":1587444578000},"page":"2320","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Counter a Drone in a Complex Neighborhood Area by Deep Reinforcement Learning"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9258-1919","authenticated-orcid":false,"given":"Ender","family":"\u00c7etin","sequence":"first","affiliation":[{"name":"Aerospace Science and Technology, UPC BarcelonaTECH, 08860 Castelldefels, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0100-724X","authenticated-orcid":false,"given":"Cristina","family":"Barrado","sequence":"additional","affiliation":[{"name":"Computer Architecture Department, UPC BarcelonaTECH, 08860 Castelldefels, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7587-8702","authenticated-orcid":false,"given":"Enric","family":"Pastor","sequence":"additional","affiliation":[{"name":"Computer Architecture Department, UPC BarcelonaTECH, 08860 Castelldefels, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,4,18]]},"reference":[{"key":"ref_1","unstructured":"(2019, May 26). European ATM Master Plan: Roadmap for the Safe Integration of Drones into All Classes of Airspace. Available online: https:\/\/www.sesarju.eu\/node\/2993."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Fabra, F., Zamora, W., Sang\u00fcesa, J., Calafate, C.T., Cano, J.C., and Manzoni, P. (2019). A Distributed Approach for Collision Avoidance between Multirotor UAVs Following Planned Missions. Sensors, 19.","DOI":"10.3390\/s19102404"},{"key":"ref_3","unstructured":"(2019, August 23). Flights Diverted after Gatwick Airport. Available online: https:\/\/www.bbc.com\/news\/uk-england-sussex-48086013."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"460","DOI":"10.14429\/dsj.68.12442","article-title":"Countering UAVs\u2014The Mover of Research in Military Technology","volume":"68","author":"Kratky","year":"2018","journal-title":"Def. Sci. J."},{"key":"ref_5","unstructured":"Michel, A.H. (2020, April 17). Counter-Drone Systems; Center for the Study of the Drone at Bard College. Available online: https:\/\/dronecenter.bard.edu\/counter-drone-systems."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Akhloufi, M.A., Arola, S., and Bonnet, A. (2019). Drones Chasing Drones: Reinforcement Learning and Deep Search Area Proposal. Drones, 3.","DOI":"10.3390\/drones3030058"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Anwar, A., and Raychowdhury, A. (2019). Autonomous Navigation via Deep Reinforcement Learning for Resource Constraint Edge Nodes using Transfer Learning. arXiv.","DOI":"10.1109\/ACCESS.2020.2971172"},{"key":"ref_9","unstructured":"(2019, January 29). Unreal Engine 4. Available online: https:\/\/www.unrealengine.com\/en-US\/what-is-unreal-engine-4."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Kouris, A., and Bouganis, C.S. (2018, January 1\u20135). Learning to Fly by MySelf: A Self-Supervised CNN-based Approach for Autonomous Navigation. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8594204"},{"key":"ref_11","unstructured":"Lu, X., Xiao, L., Dai, C., and Dai, H. (2018). UAV-aided cellular communications with deep reinforcement learning against jamming. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1007\/s10846-018-0891-8","article-title":"A deep reinforcement learning strategy for UAV autonomous landing on a moving platform","volume":"93","author":"Sampedro","year":"2019","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press Cambridge.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2042","DOI":"10.1109\/TNNLS.2017.2773458","article-title":"Optimal and autonomous control using reinforcement learning: A survey","volume":"29","author":"Kiumarsi","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_16","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.A. (2013). Playing Atari with Deep Reinforcement Learning. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1037\/0033-295X.102.3.419","article-title":"Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory","volume":"102","author":"McClelland","year":"1995","journal-title":"Psychol. Rev."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Riedmiller, M. (2005, January 3\u20137). Neural fitted Q iteration-first experiences with a data efficient neural reinforcement learning method. Proceedings of the 16th European Conference on Machine Learning, Porto, Portugal.","DOI":"10.1007\/11564096_32"},{"key":"ref_20","unstructured":"Lin, L.J. (1993). Reinforcement Learning for Robots Using Neural Networks. [Ph.D. Thesis, Carnegie-Mellon University]."},{"key":"ref_21","unstructured":"Hasselt, H.V. (2010, January 6\u20139). Double Q-learning. Proceedings of the Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, Vancouver, BC, Canada."},{"key":"ref_22","unstructured":"Kersandt, K. (2017). Deep Reinforcement Learning as Control Method for Autonomous UAVs. [Master\u2019s Thesis, Universitat Polit\u00e8cnica de Catalunya]."},{"key":"ref_23","first-page":"1633","article-title":"Transfer learning for reinforcement learning domains: A survey","volume":"10","author":"Taylor","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Shah, S., Dey, D., Lovett, C., and Kapoor, A. (2017). AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. Field and Service Robotics, Springer.","DOI":"10.1007\/978-3-319-67361-5_40"},{"key":"ref_25","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2020, April 17). OpenAI Gym 2016. Available online: https:\/\/arxiv.org\/abs\/1606.01540."},{"key":"ref_26","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2015). TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. arXiv."},{"key":"ref_27","unstructured":"(2016). Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv."},{"key":"ref_28","unstructured":"Plappert, M. (2020, April 17). keras-rl. Available online: https:\/\/github.com\/keras-rl\/keras-rl."},{"key":"ref_29","unstructured":"Von Bothmer, F. (2018). Missing Man: Contextualising Legal Reviews for Autonomous Weapon Systems. [Ph.D. Thesis, Universit\u00e4t St. Gallen]."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Gurriet, T., and Ciarletta, L. (2016, January 7\u201310). Towards a generic and modular geofencing strategy for civilian UAVs. Proceedings of the 2016 International Conference on Unmanned Aircraft Systems (ICUAS), Arlington, VA, USA.","DOI":"10.1109\/ICUAS.2016.7502603"},{"key":"ref_31","unstructured":"(2019, May 01). AirSim Documentation. Available online: https:\/\/microsoft.github.io\/AirSim."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Samek, W., Wiegand, T., and M\u00fcller, K.R. (2018). Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models. arXiv.","DOI":"10.1007\/978-3-030-28954-6_1"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/8\/2320\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:31:30Z","timestamp":1760362290000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/8\/2320"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,18]]},"references-count":32,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2020,4]]}},"alternative-id":["s20082320"],"URL":"https:\/\/doi.org\/10.3390\/s20082320","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,18]]}}}