{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T15:17:01Z","timestamp":1781191021533,"version":"3.54.1"},"reference-count":37,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,5,12]],"date-time":"2023-05-12T00:00:00Z","timestamp":1683849600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Unmanned aerial vehicles (UAVs) can be used to relay sensing information and computational workloads from ground users (GUs) to a remote base station (RBS) for further processing. In this paper, we employ multiple UAVs to assist with the collection of sensing information in a terrestrial wireless sensor network. All of the information collected by the UAVs can be forwarded to the RBS. We aim to improve the energy efficiency for sensing-data collection and transmission by optimizing UAV trajectory, scheduling, and access-control strategies. Considering a time-slotted frame structure, UAV flight, sensing, and information-forwarding sub-slots are confined to each time slot. This motivates the trade-off study between UAV access-control and trajectory planning. More sensing data in one time slot will take up more UAV buffer space and require a longer transmission time for information forwarding. We solve this problem by a multi-agent deep reinforcement learning approach that takes into consideration a dynamic network environment with uncertain information about the GU spatial distribution and traffic demands. We further devise a hierarchical learning framework with reduced action and state spaces to improve the learning efficiency by exploiting the distributed structure of the UAV-assisted wireless sensor network. Simulation results show that UAV trajectory planning with access control can significantly improve UAV energy efficiency. The hierarchical learning method is more stable in learning and can also achieve higher sensing performance.<\/jats:p>","DOI":"10.3390\/s23104691","type":"journal-article","created":{"date-parts":[[2023,5,12]],"date-time":"2023-05-12T09:56:18Z","timestamp":1683885378000},"page":"4691","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Deep Reinforcement Learning for Joint Trajectory Planning, Transmission Scheduling, and Access Control in UAV-Assisted Wireless Sensor Networks"],"prefix":"10.3390","volume":"23","author":[{"given":"Xiaoling","family":"Luo","sequence":"first","affiliation":[{"name":"School of Information Engineering, Wuhan University of Technology, Wuhan 430070, China"},{"name":"China Three Gorges Corporation, Wuhan 430010, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Che","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computer Sciences, Minnan Normal University, Zhangzhou 363000, China"},{"name":"School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen 518107, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunnian","family":"Zeng","sequence":"additional","affiliation":[{"name":"School of Information Engineering, Wuhan University of Technology, Wuhan 430070, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chengtao","family":"Li","sequence":"additional","affiliation":[{"name":"China Three Gorges Corporation, Wuhan 430010, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jing","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4874-8766","authenticated-orcid":false,"given":"Shimin","family":"Gong","sequence":"additional","affiliation":[{"name":"School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen 518107, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Lagkas, T., Argyriou, V., Bibi, S., and Sarigiannidis, P. (2018). UAV IoT framework views and challenges: Towards protecting drones as \u201cThings\u201d. Sensors, 18.","DOI":"10.3390\/s18114015"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1109\/COMST.2015.2495297","article-title":"Survey of important issues in UAV communication networks","volume":"18","author":"Gupta","year":"2015","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"3129","DOI":"10.1109\/JSAC.2021.3088676","article-title":"UAV-aided backscatter communications: Performance analysis and trajectory optimization","volume":"39","author":"Han","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"926","DOI":"10.1109\/TWC.2020.3029225","article-title":"Energy-efficient UAV backscatter communication with joint trajectory design and resource optimization","volume":"20","author":"Yang","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/MWC.2018.1800160","article-title":"UAV-assisted emergency networks in disasters","volume":"26","author":"Zhao","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"15717","DOI":"10.3390\/s150715717","article-title":"UAV deployment exercise for mapping purposes: Evaluation of emergency response applications","volume":"15","author":"Boccardo","year":"2015","journal-title":"Sensors"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"8958","DOI":"10.1109\/JIOT.2019.2925567","article-title":"Localization and clustering based on swarm intelligence in UAV networks for emergency communications","volume":"6","author":"Arafat","year":"2019","journal-title":"IEEE Internet Things J."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2624","DOI":"10.1109\/COMST.2016.2560343","article-title":"Survey on unmanned aerial vehicle networks for civil applications: A communications viewpoint","volume":"18","author":"Hayat","year":"2016","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/MCOM.2017.1700452","article-title":"An amateur drone surveillance system based on the cognitive Internet of Things","volume":"56","author":"Ding","year":"2018","journal-title":"IEEE Commun. Mag."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3193","DOI":"10.1109\/JSAC.2021.3088669","article-title":"Multi-UAV trajectory planning for energy-efficient content coverage: A decentralized learning-based approach","volume":"39","author":"Zhao","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1621","DOI":"10.1109\/TWC.2021.3105821","article-title":"UAV relay-assisted emergency communications in IoT networks: Resource allocation and trajectory optimization","volume":"21","author":"Tran","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3192","DOI":"10.1109\/TWC.2019.2911939","article-title":"3D trajectory optimization in Rician fading for UAV-enabled data harvesting","volume":"18","author":"You","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"7867","DOI":"10.1109\/TWC.2022.3162704","article-title":"IRS empowered UAV wireless communication with resource allocation, reflecting design and trajectory optimization","volume":"21","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"5704","DOI":"10.1109\/JIOT.2022.3161571","article-title":"Intelligent offloading and resource allocation in heterogeneous aerial access IoT networks","volume":"10","author":"Lakew","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kang, H., Chang, X., Mi\u0161i\u0107, J., Mi\u0161i\u0107, V.B., Fan, J., and Liu, Y. (2023). Cooperative UAV Resource Allocation and Task Offloading in Hierarchical Aerial Computing Systems: A MAPPO Based Approach. IEEE Internet Things J.","DOI":"10.1109\/JIOT.2023.3240173"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"3899","DOI":"10.1109\/JIOT.2021.3102185","article-title":"Trajectory Design for UAV-Based Internet of Things Data Collection: A Deep Reinforcement Learning Approach","volume":"9","author":"Wang","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, M., Long, Y., Gong, S., and Xu, J. (2021, January 20\u201322). Adaptive Network Formation and Trajectory Optimization for Multi-UAV-Assisted Wireless Data Offloading. Proceedings of the 2021 IEEE 23rd International Conference on High Performance Computing & Communications, Haikou, China.","DOI":"10.1109\/HPCC-DSS-SmartCity-DependSys53884.2021.00153"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"9107","DOI":"10.1109\/TVT.2022.3175592","article-title":"Distributed federated deep reinforcement learning based trajectory optimization for air-ground cooperative emergency networks","volume":"71","author":"Wu","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1109\/JIOT.2022.3201017","article-title":"Joint Multi-Domain Resource Allocation and Trajectory Optimization in UAV-Assisted Maritime IoT Networks","volume":"10","author":"Qian","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"986","DOI":"10.1109\/TVT.2022.3202525","article-title":"Hierarchical Multi-Agent Deep Reinforcement Learning for Energy-Efficient Hybrid Computation Offloading","volume":"72","author":"Zhou","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Gong, S., Cui, L., Gu, B., Lyu, B., Hoang, D.T., and Niyato, D. (2023). Hierarchical Deep Reinforcement Learning for Age-of-Information Minimization in IRS-aided and Wireless-powered Wireless Networks. IEEE Trans. Wirel. Commun.","DOI":"10.1109\/TWC.2023.3259721"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Gong, S., Wang, M., Gu, B., Zhang, W., Hoang, D.T., and Niyato, D. (2023). Bayesian Optimization Enhanced Deep Reinforcement Learning for Trajectory Planning and Network Formation in Multi-UAV Networks. IEEE Trans. Veh. Technol.","DOI":"10.1109\/TVT.2023.3262778"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"15372","DOI":"10.1109\/JIOT.2021.3064376","article-title":"An efficient strategy for accurate detection and localization of UAV swarms","volume":"8","author":"Zheng","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3160","DOI":"10.1109\/JSAC.2021.3088718","article-title":"Deep reinforcement learning based three-dimensional area coverage with UAV swarm","volume":"39","author":"Mou","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_25","first-page":"15372","article-title":"Multi-UAV Navigation for Partially Observable Communication Coverage by Graph Reinforcement Learning","volume":"8","author":"Ye","year":"2022","journal-title":"IEEE Trans. Mobil. Comput."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5362","DOI":"10.1109\/TVT.2021.3062418","article-title":"Orchestrated scheduling and multi-agent deep reinforcement learning for cloud-assisted multi-UAV charging systems","volume":"70","author":"Jung","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_27","first-page":"2003","article-title":"Average peak age-of-information minimization in UAV-assisted IoT networks","volume":"68","author":"Dhillon","year":"2018","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Du, Y., Wang, K., Yang, K., and Zhang, G. (2018, January 9\u201313). Energy-efficient resource allocation in UAV based MEC system for IoT devices. Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/GLOCOM.2018.8647789"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"7610","DOI":"10.1109\/TWC.2021.3086503","article-title":"Deep reinforcement learning-based resource allocation in cooperative UAV-assisted wireless networks","volume":"20","author":"Luong","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Singh, S.K., Agrawal, K., Singh, K., Chen, Y.M., and Li, C.P. (2022). Ergodic Capacity and Placement Optimization for RSMA-Enabled UAV-Assisted Communication. IEEE Syst. J.","DOI":"10.1109\/JSYST.2022.3220249"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Singh, S.K., Agrawal, K., Singh, K., Chen, Y.M., and Li, C.P. (2022). Performance Analysis and Optimization of RSMA Enabled UAV-Aided IBL and FBL Communication with Imperfect SIC and CSI. IEEE Trans. Wirel. Commun.","DOI":"10.1109\/TWC.2022.3220785"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"22716","DOI":"10.1109\/ACCESS.2018.2826650","article-title":"Non-orthogonal multiple access for unmanned aerial vehicle assisted communication","volume":"6","author":"Sohail","year":"2018","journal-title":"IEEE Access"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1196","DOI":"10.1109\/TVT.2022.3206145","article-title":"Stackelberg game-based deployment design and radio resource allocation in coordinated UAVs-assisted vehicular communication networks","volume":"72","author":"Hosseini","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_34","first-page":"2296","article-title":"Backscatter communication assisted by reconfigurable intelligent surfaces","volume":"28","author":"Liang","year":"2022","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"835","DOI":"10.1109\/TCCN.2019.2925080","article-title":"Exploiting backscatter-aided relay communications with hybrid access model in device-to-device networks","volume":"5","author":"Gong","year":"2019","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_36","unstructured":"Lowe, R., Wu, Y.I., Tamar, A., Harb, J., Pieter, A., and Mordatch, I. (2017, January 4\u20139). Multi-agent actor-critic for mixed cooperative-competitive environments. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_37","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016). Continuous control with deep reinforcement learning. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/10\/4691\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:33:40Z","timestamp":1760124820000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/10\/4691"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,12]]},"references-count":37,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["s23104691"],"URL":"https:\/\/doi.org\/10.3390\/s23104691","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,12]]}}}