{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T04:29:51Z","timestamp":1772166591825,"version":"3.50.1"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,4,7]],"date-time":"2021-04-07T00:00:00Z","timestamp":1617753600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,4,7]],"date-time":"2021-04-07T00:00:00Z","timestamp":1617753600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["742648"],"award-info":[{"award-number":["742648"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001866","name":"Fonds National de la Recherche Luxembourg","doi-asserted-by":"publisher","award":["11632107"],"award-info":[{"award-number":["11632107"]}],"id":[{"id":"10.13039\/501100001866","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001866","name":"Fonds National de la Recherche Luxembourg","doi-asserted-by":"publisher","award":["C17\/IS\/11691338"],"award-info":[{"award-number":["C17\/IS\/11691338"]}],"id":[{"id":"10.13039\/501100001866","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001866","name":"Fonds National de la Recherche Luxembourg","doi-asserted-by":"publisher","award":["C9\/IS\/13713801"],"award-info":[{"award-number":["C9\/IS\/13713801"]}],"id":[{"id":"10.13039\/501100001866","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001866","name":"Fonds National de la Recherche Luxembourg","doi-asserted-by":"publisher","award":["12173206"],"award-info":[{"award-number":["12173206"]}],"id":[{"id":"10.13039\/501100001866","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Wireless Com Network"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In unmanned aerial vehicle (UAV)-assisted networks, UAV acts as an aerial base station which acquires the requested data via backhaul link and then serves ground users (GUs) through an access network. In this paper, we investigate an energy minimization problem with a limited power supply for both backhaul and access links. The difficulties for solving such a non-convex and combinatorial problem lie at the high computational complexity\/time. In solution development, we consider the approaches from both actor-critic deep reinforcement learning (AC-DRL) and optimization perspectives. First, two offline non-learning algorithms, i.e., an optimal and a heuristic algorithms, based on piecewise linear approximation and relaxation are developed as benchmarks. Second, toward real-time decision-making, we improve the conventional AC-DRL and propose two learning schemes: AC-based user group scheduling and backhaul power allocation (ACGP), and joint AC-based user group scheduling and optimization-based backhaul power allocation (ACGOP). Numerical results show that the computation time of both ACGP and ACGOP is reduced tenfold to hundredfold compared to the offline approaches, and ACGOP is better than ACGP in energy savings. The results also verify the superiority of proposed learning solutions in terms of guaranteeing the feasibility and minimizing the system energy compared to the conventional AC-DRL.<\/jats:p>","DOI":"10.1186\/s13638-021-01960-0","type":"journal-article","created":{"date-parts":[[2021,4,7]],"date-time":"2021-04-07T09:06:05Z","timestamp":1617786365000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Actor-critic learning-based energy optimization for UAV access and backhaul networks"],"prefix":"10.1186","volume":"2021","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4118-039X","authenticated-orcid":false,"given":"Yaxiong","family":"Yuan","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Lei","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thang X.","family":"Vu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Symeon","family":"Chatzinotas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sumei","family":"Sun","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bj\u00f6rn","family":"Ottersten","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,4,7]]},"reference":[{"key":"1960_CR1","doi-asserted-by":"crossref","unstructured":"Y. Yuan, L. Lei, T. X. Vu, S. Chatzinotas, B. Ottersten, Actor-critic deep reinforcement learning for energy minimization in uav-aided networks, In 2020 European Conference on Networks and Communications (EuCNC), (2020)","DOI":"10.1109\/EuCNC48522.2020.9200931"},{"issue":"3","key":"1960_CR2","doi-asserted-by":"publisher","first-page":"2334","DOI":"10.1109\/COMST.2019.2902862","volume":"21","author":"M Mozaffari","year":"2019","unstructured":"M. Mozaffari, W. Saad, M. Bennis, Y. Nam, M. Debbah, A tutorial on uavs for wireless networks: Applications, challenges, and open problems. IEEE Communications Surveys Tutorials 21(3), 2334\u20132360 (2019)","journal-title":"IEEE Communications Surveys Tutorials"},{"key":"1960_CR3","doi-asserted-by":"crossref","unstructured":"M.M.U. Chowdhury, S.J. Maeng, E. Bulut, I. G\u00fcven\u00e7, 3D trajectory optimization in uav-assisted cellular networks considering antenna radiation pattern and backhaul constraint. IEEE Transactions on Aerospace and Electronic Systems (2020)","DOI":"10.1109\/TAES.2020.2981233"},{"key":"1960_CR4","doi-asserted-by":"crossref","first-page":"21215","DOI":"10.1109\/ACCESS.2020.2969357","volume":"8","author":"S Ahmed","year":"2020","unstructured":"S. Ahmed, M.Z. Chowdhury, Y.M. Jang, Energy-efficient uav-to-user scheduling to maximize throughput in wireless networks. IEEE Access 8, 21215\u201321225 (2020)","journal-title":"IEEE Access"},{"issue":"4","key":"1960_CR5","doi-asserted-by":"publisher","first-page":"2329","DOI":"10.1109\/TWC.2019.2902559","volume":"18","author":"Y Zeng","year":"2019","unstructured":"Y. Zeng, J. Xu, R. Zhang, Energy minimization for wireless communication with rotary-wing uav. IEEE Transactions on Wireless Communications 18(4), 2329\u20132345 (2019)","journal-title":"IEEE Transactions on Wireless Communications"},{"issue":"1","key":"1960_CR6","doi-asserted-by":"publisher","first-page":"363","DOI":"10.1109\/JSYST.2019.2911895","volume":"14","author":"H Yang","year":"2020","unstructured":"H. Yang, X. Xie, Energy-efficient joint scheduling and resource management for uav-enabled multicell networks. IEEE Systems Journal 14(1), 363\u2013374 (2020)","journal-title":"IEEE Systems Journal"},{"issue":"6","key":"1960_CR7","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1109\/MCOM.2018.1700431","volume":"56","author":"H Wang","year":"2018","unstructured":"H. Wang, G. Ding, F. Gao, J. Chen, J. Wang, L. Wang, Power control in uav-supported ultra dense networks: Communications, caching, and energy transfer. IEEE Communications Magazine 56(6), 28\u201334 (2018)","journal-title":"IEEE Communications Magazine"},{"issue":"4","key":"1960_CR8","doi-asserted-by":"publisher","first-page":"849","DOI":"10.1109\/LCOMM.2020.2965120","volume":"24","author":"S Ahmed","year":"2020","unstructured":"S. Ahmed, M.Z. Chowdhury, Y.M. Jang, Energy-efficient uav relaying communications to serve ground nodes. IEEE Communications Letters 24(4), 849\u2013852 (2020)","journal-title":"IEEE Communications Letters"},{"key":"1960_CR9","doi-asserted-by":"publisher","first-page":"60940","DOI":"10.1109\/ACCESS.2020.2983516","volume":"8","author":"C Qiu","year":"2020","unstructured":"C. Qiu, Z. Wei, Z. Feng, P. Zhang, Backhaul-aware trajectory optimization of fixed-wing uav-mounted base station for continuous available wireless service. IEEE Access 8, 60940\u201360950 (2020)","journal-title":"IEEE Access"},{"key":"1960_CR10","doi-asserted-by":"crossref","unstructured":"M. Youssef, J. Farah, C. Abdel Nour, C. Douillard, Full-duplex and backhaul-constrained uav-enabled networks using noma, IEEE Transactions on Vehicular Technology, (2020)","DOI":"10.1109\/TVT.2020.3001432"},{"key":"1960_CR11","unstructured":"Z. Xu, L. Li, H. Xu, A. Gao, X. Li, W. Chen, Z. Han, Precoding design for drone small cells cluster network with massive mimo: A game theoretical approach, In 2018 14th International Wireless Communications Mobile Computing Conference (IWCMC), pp. 1477\u20131482, (2018)"},{"issue":"5","key":"1960_CR12","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1109\/CC.2018.8387985","volume":"15","author":"Q Song","year":"2018","unstructured":"Q. Song, F. Zheng, Energy efficient multi-antenna uav-enabled mobile relay. China Communications 15(5), 41\u201350 (2018)","journal-title":"China Communications"},{"key":"1960_CR13","doi-asserted-by":"crossref","unstructured":"L. Lei, L. You, G. Dai, T.X. Vu, D. Yuan, S. Chatzinotas, A Deep Learning Approach for Optimizing Content Delivering in Cache Enabled HetNet, In IEEE International Symposium on Wireless Communication Systems (ISWCS), pp. 449\u2013453, Aug. (2017)","DOI":"10.1109\/ISWCS.2017.8108157"},{"key":"1960_CR14","doi-asserted-by":"crossref","unstructured":"F. Ghavimi, R. Jantti, Energy-efficient uav communications with interference management: Deep learning framework, In 2020 IEEE Wireless Communications and Networking Conference Workshops (WCNCW), (2020)","DOI":"10.1109\/WCNCW48565.2020.9124759"},{"key":"1960_CR15","doi-asserted-by":"crossref","unstructured":"W. Liu, P. Si, E. Sun, M. Li, C. Fang, Y. Zhang, Green mobility management in uav-assisted iot based on dueling dqn, In ICC 2019 - 2019 IEEE International Conference on Communications (ICC), (2019)","DOI":"10.1109\/ICC.2019.8762097"},{"key":"1960_CR16","volume-title":"Reinforcement learning An introduction","author":"RS Sutton","year":"2018","unstructured":"R.S. Sutton, A. Barto, Reinforcement Learning An Introduction (MIT Press, London, 2018)"},{"issue":"9","key":"1960_CR17","doi-asserted-by":"publisher","first-page":"2059","DOI":"10.1109\/JSAC.2018.2864373","volume":"36","author":"CH Liu","year":"2018","unstructured":"C.H. Liu, Z. Chen, J. Tang, J. Xu, C. Piao, Energy-efficient uav control for effective and fair communication coverage: A deep reinforcement learning approach. IEEE Journal on Selected Areas in Communications 36(9), 2059\u20132070 (2018)","journal-title":"IEEE Journal on Selected Areas in Communications"},{"key":"1960_CR18","doi-asserted-by":"publisher","first-page":"53172","DOI":"10.1109\/ACCESS.2020.2981403","volume":"8","author":"H Qi","year":"2020","unstructured":"H. Qi, Z. Hu, H. Huang, X. Wen, Z. Lu, Energy efficient 3-d uav control for persistent communication service and fairness: A deep reinforcement learning approach. IEEE Access 8, 53172\u201353184 (2020)","journal-title":"IEEE Access"},{"key":"1960_CR19","doi-asserted-by":"crossref","unstructured":"M.H. Casta\u00f1eda Garcia, M. Iwanow, R.A. Stirling-Gallacher, Los mimo design based on multiple optimum antenna separations, In 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall), (2018)","DOI":"10.1109\/VTCFall.2018.8690668"},{"key":"1960_CR20","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511807213","volume-title":"Fundamentals of wireless communication","author":"D Tse","year":"2005","unstructured":"D. Tse, P. Viswanath, Fundamentals of Wireless Communication (Cambridge University Press, Cambridge, 2005)"},{"key":"1960_CR21","doi-asserted-by":"crossref","unstructured":"Y. Yuan, T.X. Vu, L. Lei, S. Chatzinotas, B. Ottersten, Joint user grouping and power allocation for MISO systems: learning to schedule, In 2019 27th European Signal Processing Conference (EUSIPCO), (2019)","DOI":"10.23919\/EUSIPCO.2019.8902514"},{"key":"1960_CR22","doi-asserted-by":"publisher","first-page":"107769","DOI":"10.1109\/ACCESS.2019.2933173","volume":"7","author":"C Yan","year":"2019","unstructured":"C. Yan, L. Fu, J. Zhang, J. Wang, A comprehensive survey on uav communication channel modeling. IEEE Access 7, 107769\u2013107792 (2019)","journal-title":"IEEE Access"},{"key":"1960_CR23","doi-asserted-by":"crossref","unstructured":"H.D. Tran, T.X. Vu, S. Chatzinotas, S. Shahbazpanahi, B. Ottersten, Coarse trajectory design for energy minimization in uav-enabled wireless communications with latency constraints. IEEE Transactions on Vehicular Technology (2020)","DOI":"10.1109\/TVT.2020.3001403"},{"key":"1960_CR24","doi-asserted-by":"publisher","DOI":"10.2514\/4.478390","volume-title":"Flight performance of fixed and rotary wing aircraft","author":"A Filippone","year":"2006","unstructured":"A. Filippone, Flight Performance of Fixed and Rotary Wing Aircraft (Elsevier, London, 2006)"},{"key":"1960_CR25","volume-title":"Network flows: theory, algorithms, and applications, Upper Saddle River, NJ","author":"RK Ahuja","year":"1993","unstructured":"R.K. Ahuja, T.L. Magnanti, J.B. Orlin, Network Flows: Theory, Algorithms, and Applications, Upper Saddle River, NJ (Prentice Hall, USA, 1993)"},{"issue":"1","key":"1960_CR26","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1007\/BF01580665","volume":"10","author":"GP McCormick","year":"1976","unstructured":"G.P. McCormick, Computability of global solutions to factorable nonconvex programs: Part i: Convex underestimating problems. Mathematical programming 10(1), 147\u2013175 (1976)","journal-title":"Mathematical programming"},{"key":"1960_CR27","unstructured":"C.H. Papadimitriou, K. Steiglitz, Combinatorial Optimization: Algorithms and Complexity, Mineola, NY (Dover, USA, 1998)"},{"key":"1960_CR28","volume-title":"Linear Programming","author":"K Murty","year":"1983","unstructured":"K. Murty, Linear Programming (Wiley, New York, NY, USA, 1983)"},{"issue":"1","key":"1960_CR29","doi-asserted-by":"publisher","first-page":"680","DOI":"10.1109\/TWC.2017.2769644","volume":"17","author":"Y Wei","year":"2018","unstructured":"Y. Wei, F.R. Yu, M. Song, Z. Han, User scheduling and resource allocation in hetnets with hybrid energy supply: An actor-critic reinforcement learning approach. IEEE Transactions on Wireless Communications 17(1), 680\u2013692 (2018)","journal-title":"IEEE Transactions on Wireless Communications"},{"key":"1960_CR30","unstructured":"J. Schulman, P. Moritz, S. Levine, M. Jordan, P. Abbeel, High-dimensional continuous control using generalized advantage estimation, arXiv preprint arXiv:1506.02438, (2015)"},{"key":"1960_CR31","doi-asserted-by":"crossref","unstructured":"Y. Lu, H. Lu, L. Cao, F. Wu, D. Zhu, Learning deterministic policy with target for power control in wireless networks, In 2018 IEEE Global Communications Conference (GLOBECOM), (2018)","DOI":"10.1109\/GLOCOM.2018.8648056"},{"key":"1960_CR32","volume-title":"Introduction to algorithms","author":"TH Cormen","year":"2009","unstructured":"T.H. Cormen, C.E. Leiserson, R.L. Rivest, C. Stein, Introduction to Algorithms (MIT Press, London, 2009)."},{"key":"1960_CR33","doi-asserted-by":"crossref","unstructured":"T. Yang, Y. Hu, M.C. Gursoy, A. Schmeink, R. Mathar, Deep reinforcement learning based resource allocation in low latency edge computing networks, In International Symposium on Wireless Communication Systems (ISWCS), (2018)","DOI":"10.1109\/ISWCS.2018.8491089"},{"issue":"3","key":"1960_CR34","doi-asserted-by":"publisher","first-page":"528","DOI":"10.1109\/JSAC.2005.862421","volume":"24","author":"T Yoo","year":"2006","unstructured":"T. Yoo, A. Goldsmith, On the optimality of multiantenna broadcast scheduling using zero-forcing beamforming. IEEE J Sel Areas Commun 24(3), 528\u2013541 (2006)","journal-title":"IEEE Journal on Selected Areas in Communications"},{"issue":"7","key":"1960_CR35","doi-asserted-by":"publisher","first-page":"5906","DOI":"10.1109\/JIOT.2019.2952677","volume":"7","author":"H Mei","year":"2019","unstructured":"H. Mei, K. Yang, Q. Liu, K. Wang, Joint trajectory-resource optimization in UAV-enabled edge-cloud system with virtualized mobile clone. IEEE Internet Things J 7(7), 5906\u20135921 (2019)","journal-title":"IEEE Internet of Things Journal"}],"container-title":["EURASIP Journal on Wireless Communications and Networking"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13638-021-01960-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s13638-021-01960-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13638-021-01960-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T12:38:06Z","timestamp":1671799086000},"score":1,"resource":{"primary":{"URL":"https:\/\/jwcn-eurasipjournals.springeropen.com\/articles\/10.1186\/s13638-021-01960-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,7]]},"references-count":35,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1960"],"URL":"https:\/\/doi.org\/10.1186\/s13638-021-01960-0","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-95073\/v1","asserted-by":"object"}]},"ISSN":["1687-1499"],"issn-type":[{"value":"1687-1499","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,7]]},"assertion":[{"value":"16 October 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 March 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 April 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"78"}}