{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T14:30:53Z","timestamp":1784212253339,"version":"3.55.0"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,1,26]],"date-time":"2024-01-26T00:00:00Z","timestamp":1706227200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,26]],"date-time":"2024-01-26T00:00:00Z","timestamp":1706227200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62261013"],"award-info":[{"award-number":["62261013"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["EURASIP J. Adv. Signal Process."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Cell-free massive multiple-input multiple-output (CF-mMIMO) has attracted considerable attention due to its potential for delivering high data rates and energy efficiency (EE). In this paper, we investigate the resource allocation of downlink in CF-mMIMO systems. A hierarchical depth deterministic strategy gradient (H-DDPG) framework is proposed to jointly optimize the access point (AP) clustering and power allocation. The framework uses two-layer control networks operating on different timescales to enhance EE of downlinks in CF-mMIMO systems by cooperatively optimizing AP clustering and power allocation. In this framework, the high-level processing of system-level problems, namely AP clustering, enhances the wireless network configuration by utilizing DDPG on the large timescale while meeting the minimum spectral efficiency (SE) constraints for each user. The low layer solves the link-level sub-problem, that is, power allocation, and reduces interference between APs and improves transmission performance by utilizing DDPG on a small timescale while meeting the maximum transmit power constraint of each AP. Two corresponding DDPG agents are trained separately, allowing them to learn from the environment and gradually improve their policies to maximize the system EE. Numerical results validate the effectiveness of the proposed algorithm in term of its convergence speed, SE, and EE.<\/jats:p>","DOI":"10.1186\/s13634-024-01111-9","type":"journal-article","created":{"date-parts":[[2024,1,26]],"date-time":"2024-01-26T02:07:22Z","timestamp":1706234842000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Energy-efficient access point clustering and power allocation in cell-free massive MIMO networks: a hierarchical deep reinforcement learning approach"],"prefix":"10.1186","volume":"2024","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1084-3699","authenticated-orcid":false,"given":"Fangqing","family":"Tan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Quanxuan","family":"Deng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,1,26]]},"reference":[{"issue":"11","key":"1111_CR1","doi-asserted-by":"publisher","first-page":"3590","DOI":"10.1109\/TWC.2010.092810.091092","volume":"9","author":"TL Marzetta","year":"2010","unstructured":"T.L. Marzetta, Noncooperative cellular wireless with unlimited numbers of base station antennas. IEEE Trans. Wireless Commun.Commun. 9(11), 3590\u20133600 (2010)","journal-title":"IEEE Trans. Wireless Commun.Commun."},{"issue":"1","key":"1111_CR2","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1109\/MWC.2018.8304383","volume":"25","author":"L Daza","year":"2018","unstructured":"L. Daza, S. Misra, Fundamentals of massive MIMO. IEEE Wirel. Commun.Wirel. Commun. 25(1), 9\u20139 (2018)","journal-title":"IEEE Wirel. Commun.Wirel. Commun."},{"issue":"1","key":"1111_CR3","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1109\/TWC.2017.2768423","volume":"17","author":"E Bj\u00f6rnson","year":"2018","unstructured":"E. Bj\u00f6rnson, J. Hoydis, L. Sanguinetti, Massive MIMO has unlimited capacity. IEEE Trans. Wireless Commun.Commun. 17(1), 574\u2013590 (2018)","journal-title":"IEEE Trans. Wireless Commun.Commun."},{"issue":"3\u20134","key":"1111_CR4","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1561\/2000000093","volume":"11","author":"E Bj\u00f6rnson","year":"2017","unstructured":"E. Bj\u00f6rnson, J. Hoydis, L. Sanguinetti, Massive MIMO networks Spectral, energy, and hardware efficiency. Found. Trends Signal Process. 11(3\u20134), 154\u2013655 (2017)","journal-title":"Found. Trends Signal Process."},{"issue":"1","key":"1111_CR5","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1109\/MCOM.001.2000208","volume":"59","author":"M Matthaiou","year":"2021","unstructured":"M. Matthaiou, O. Yurduseven, H.Q. Ngo, D. Morales-Jimenez, S.L. Cotton, V.F. Fusco, The road to 6G: ten physical layer challenges for communications engineers. IEEE Commun. Mag.Commun. Mag. 59(1), 64\u201369 (2021)","journal-title":"IEEE Commun. Mag.Commun. Mag."},{"issue":"3","key":"1111_CR6","doi-asserted-by":"publisher","first-page":"1834","DOI":"10.1109\/TWC.2017.2655515","volume":"16","author":"HQ Ngo","year":"2017","unstructured":"H.Q. Ngo, A. Ashikhmin, H. Yang, E.G. Larsson, T.L. Marzetta, Cell-free massive MIMO versus small cells. IEEE Trans. Wireless Commun.Commun. 16(3), 1834\u20131850 (2017)","journal-title":"IEEE Trans. Wireless Commun.Commun."},{"issue":"4","key":"1111_CR7","doi-asserted-by":"publisher","first-page":"321","DOI":"10.23919\/JCIN.2021.9663100","volume":"6","author":"H He","year":"2021","unstructured":"H. He, X. Yu, J. Zhang, S. Song, K.B. Letaief, Cell-free massive MIMO for 6G wireless communication networks. J. Commun. Inf. Netw. 6(4), 321\u2013335 (2021)","journal-title":"J. Commun. Inf. Netw."},{"issue":"3\u20134","key":"1111_CR8","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1561\/2000000109","volume":"14","author":"\u00d6T Demir","year":"2021","unstructured":"\u00d6.T. Demir, E. Bj\u00f6rnson, L. Sanguinetti, Foundations of user-centric cell-free massive MIMO. Found. Trends Signal Process. 14(3\u20134), 162\u2013472 (2021)","journal-title":"Found. Trends Signal Process."},{"key":"1111_CR9","first-page":"348","volume":"2","author":"TX Doan","year":"2020","unstructured":"T.X. Doan, T.Q. Trinh, An overview of emerging technologies for 5G: full-duplex relaying cognitive radio networks, device-to-device communications and cell-free massive MIMO. J. Thu Dau Mot Univ. 2, 348\u2013364 (2020)","journal-title":"J. Thu Dau Mot Univ."},{"key":"1111_CR10","unstructured":"M. Fallgren, G. Fodor and A. Forsgren, \u201cOptimization approach to joint cell, channel and power allocation in wireless communication networks,\u201d 2012 In: Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pp. 829\u2013833, 2012."},{"key":"1111_CR11","doi-asserted-by":"crossref","unstructured":"V. M. T. Palhares, R. C. de Lamare, A. R. Flores and L. T. N. Landau, \u201cIterative MMSE precoding and power allocation in cell-free massive MIMO systems,\u201d\u00a02021 IEEE Statistical Signal Processing Workshop (SSP), pp.181\u2013185, 2021.","DOI":"10.1109\/SSP49050.2021.9513855"},{"issue":"13","key":"1111_CR12","doi-asserted-by":"publisher","first-page":"3334","DOI":"10.1109\/TSP.2013.2253772","volume":"61","author":"M Hong","year":"2013","unstructured":"M. Hong, A. Garcia, J. Barrera, S.G. Wilson, Joint access point selection and power allocation for uplink wireless networks. IEEE Trans. Signal Process. 61(13), 3334\u20133347 (2013)","journal-title":"IEEE Trans. Signal Process."},{"issue":"12","key":"1111_CR13","first-page":"2022","volume":"11","author":"X Lin","year":"1944","unstructured":"X. Lin, F. Xu, J. Fu, Y. Wang, Resource allocation for TDD cell-free massive MIMO systems. Electronics 11(12), 2022 (1944)","journal-title":"Electronics"},{"key":"1111_CR14","first-page":"1003","volume":"70","author":"W Jess","year":"2021","unstructured":"W. Jess, K. Arulkumaran, M. Crosby, The societal implications of deep reinforcement learning. J. Artif. Intell. Res. 70, 1003\u20131030 (2021)","journal-title":"J. Artif. Intell. Res."},{"key":"1111_CR15","doi-asserted-by":"publisher","first-page":"13768","DOI":"10.1109\/TVT.2020.3027013","volume":"69","author":"J Jang","year":"2020","unstructured":"J. Jang, H.J. Yang, Deep reinforcement learning-based resource allocation and power control in small cells with limited information exchange. IEEE Trans. Veh. Technol.Veh. Technol. 69, 13768\u201313783 (2020)","journal-title":"IEEE Trans. Veh. Technol.Veh. Technol."},{"key":"1111_CR16","doi-asserted-by":"publisher","first-page":"2239","DOI":"10.1109\/JSAC.2019.2933973","volume":"37","author":"YS Nasir","year":"2018","unstructured":"Y.S. Nasir, D. Guo, Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks. IEEE J. Sel. Areas Commun.Commun. 37, 2239\u20132250 (2018)","journal-title":"IEEE J. Sel. Areas Commun.Commun."},{"issue":"1","key":"1111_CR17","doi-asserted-by":"publisher","first-page":"957","DOI":"10.1109\/TVT.2022.3211908","volume":"72","author":"IM Braga","year":"2023","unstructured":"I.M. Braga, R.P. Antonioli, G. Fodor, Y.C.B. Silva, W.C. Freitas, Decentralized joint pilot and data power control based on deep reinforcement learning for the uplink of cell-free systems. IEEE Trans. Veh. Technol.Veh. Technol. 72(1), 957\u2013972 (2023)","journal-title":"IEEE Trans. Veh. Technol.Veh. Technol."},{"issue":"6","key":"1111_CR18","doi-asserted-by":"publisher","first-page":"6772","DOI":"10.1109\/TVT.2022.3162585","volume":"71","author":"L Luo","year":"2022","unstructured":"L. Luo, J. Zhang, S. Chen, X. Zhang, B. Ai, D.W.K. Ng, Downlink power control for cell-free massive MIMO with deep reinforcement learning. IEEE Trans. Veh. Technol.Veh. Technol. 71(6), 6772\u20136777 (2022)","journal-title":"IEEE Trans. Veh. Technol.Veh. Technol."},{"key":"1111_CR19","doi-asserted-by":"crossref","unstructured":"M. Rahmani, M. Bashar, M. J. Dehghani, P. Xiao, R. Tafazolli and M. Debbah, \u201cDeep reinforcement learning-based power allocation in uplink cell-free massive MIMO,\u201d 2022 IEEE Wireless Communications and Networking Conference (WCNC), pp. 459\u2013464, 2022.","DOI":"10.1109\/WCNC51071.2022.9771964"},{"issue":"6","key":"1111_CR20","doi-asserted-by":"publisher","first-page":"205","DOI":"10.23919\/JCC.2022.06.016","volume":"19","author":"Z Li","year":"2022","unstructured":"Z. Li, C. Hu, W. Wang, Y. Li, G. Wei, Joint access point selection and resource allocation in MEC-assisted network: a reinforcement learning based approach. China Commun. 19(6), 205\u2013218 (2022)","journal-title":"China Commun."},{"key":"1111_CR21","doi-asserted-by":"crossref","unstructured":"J. Tan, L. Zhang and Y. -C. Liang, \u201cDeep reinforcement learning for channel selection and power control in D2D networks,\u201d\u00a02019 IEEE Global Communications Conference (GLOBECOM), pp.1\u20136, 2019.","DOI":"10.1109\/GLOBECOM38437.2019.9014074"},{"issue":"3","key":"1111_CR22","doi-asserted-by":"publisher","first-page":"416","DOI":"10.1109\/LWC.2019.2958814","volume":"9","author":"H Zhang","year":"2020","unstructured":"H. Zhang, S. Chong, X. Zhang, N. Lin, A deep reinforcement learning based D2D relay selection and power level allocation in mmwave vehicular networks. IEEE Wireless Commun. Lett. 9(3), 416\u2013419 (2020)","journal-title":"IEEE Wireless Commun. Lett."},{"key":"1111_CR23","doi-asserted-by":"crossref","unstructured":"Z. Liu, J. Zhang, Z. Liu, H. Xiao and B. Ai, \u201cDouble-layer power control for mobile cell-free XL-MIMO with multi-agent reinforcement learning,\u201d\u00a0IEEE Transactions on Wireless Communications, 2023, early access.","DOI":"10.1109\/TWC.2023.3321334"},{"key":"1111_CR24","doi-asserted-by":"crossref","unstructured":"D. Zha, K.H. Lai, Q. Tan, S. Ding, N. Zou, X. Hu, \u201cTowards automated imbalanced learning with deep hierarchical reinforcement learning,\u201d In: Proceedings of the 31st ACM International Conference on Information, 2022.","DOI":"10.1145\/3511808.3557474"},{"key":"1111_CR25","doi-asserted-by":"publisher","first-page":"25473","DOI":"10.1109\/ACCESS.2021.3056670","volume":"9","author":"S Liu","year":"2021","unstructured":"S. Liu, J. Wu, J. He, Dynamic multichannel sensing in cognitive radio: hierarchical reinforcement learning. IEEE Access 9, 25473\u201325481 (2021)","journal-title":"IEEE Access"},{"key":"1111_CR26","unstructured":"W. Yuan, H. Mu\u00f1oz-Avila, \u201cHierarchical reinforcement learning for deep goal reasoning: an expressiveness analysis,\u201d ArXiv, 2020."},{"issue":"4","key":"1111_CR27","doi-asserted-by":"publisher","first-page":"949","DOI":"10.1109\/JSAC.2020.3020110","volume":"39","author":"F Tan","year":"2021","unstructured":"F. Tan, P. Wu, Y.-C. Wu, M. Xia, Energy-efficient non-orthogonal multicast and unicast transmission of cell-free massive MIMO systems with SWIPT. IEEE J. Sel. Areas Commun.Commun. 39(4), 949\u2013968 (2021)","journal-title":"IEEE J. Sel. Areas Commun.Commun."},{"key":"1111_CR28","doi-asserted-by":"crossref","unstructured":"S. Chen, J. Zhang, E. Bj\u00f6rnson, \u00d6. Demir, B. Ai, \u201cEnergy-efficient cell-free massive MIMO through sparse large-scale fading processing,\u201d IEEE Transactions on Wireless Communications, pp. 1\u201337, May. 2023, Early Access.","DOI":"10.1109\/TWC.2023.3270299"},{"issue":"1","key":"1111_CR29","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1109\/TGCN.2017.2770215","volume":"2","author":"HQ Ngo","year":"2018","unstructured":"H.Q. Ngo, L.-N. Tran, T.Q. Duong, M. Matthaiou, E.G. Larsson, On the total energy efficiency of cell-free massive MIMO. IEEE Trans. Green Commun. Netw. 2(1), 25\u201339 (2018)","journal-title":"IEEE Trans. Green Commun. Netw."},{"key":"1111_CR30","doi-asserted-by":"publisher","first-page":"102953","DOI":"10.1109\/ACCESS.2021.3097243","volume":"9","author":"Y Zhao","year":"2021","unstructured":"Y. Zhao, I.G. Niemegeers, S.M.H. De Groot, Dynamic power allocation for cell-free massive MIMO: deep reinforcement learning methods. IEEE Access 9, 102953\u2013102965 (2021)","journal-title":"IEEE Access"},{"key":"1111_CR31","doi-asserted-by":"crossref","unstructured":"R. Y. Chang, S. -F. Han and F. -T. Chien, \u201cReinforcement learning-based joint cooperation clustering and content caching in cell-free massive MIMO networks,\u201d 2021 IEEE 94th vehicular technology conference (VTC2021-Fall), Norman, OK, USA, 2021, pp. 1-7","DOI":"10.1109\/VTC2021-Fall52928.2021.9625449"},{"issue":"1","key":"1111_CR32","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1109\/TGCN.2022.3196013","volume":"7","author":"N Ghiasi","year":"2023","unstructured":"N. Ghiasi, S. Mashhadi, S. Farahmand, S.M. Razavizadeh, I. Lee, Energy efficient AP selection for cell-free massive MIMO systems: Deep reinforcement learning approach. IEEE Trans. Green Commun. Netw. 7(1), 29\u201341 (2023)","journal-title":"IEEE Trans. Green Commun. Netw."},{"key":"1111_CR33","doi-asserted-by":"crossref","unstructured":"K. Yu, C. Zhao, G. Wu, and G. Y. Li, \u201cDistributed two-tier DRL framework for cell-free network: Association, beamforming and power allocation,\u201d arXiv e-prints, 2023.","DOI":"10.1109\/ICC45041.2023.10279009"},{"issue":"10","key":"1111_CR34","doi-asserted-by":"publisher","first-page":"2344","DOI":"10.1109\/LCOMM.2020.3000067","volume":"24","author":"A Zhou","year":"2020","unstructured":"A. Zhou, J. Wu, E.G. Larsson, P. Fan, Max-min optimal beamforming for cell-free massive MIMO. IEEE Commun. Lett.Commun. Lett. 24(10), 2344\u20132348 (2020)","journal-title":"IEEE Commun. Lett.Commun. Lett."},{"issue":"12","key":"1111_CR35","doi-asserted-by":"publisher","first-page":"8459","DOI":"10.1109\/TCOMM.2021.3110214","volume":"69","author":"J Zhang","year":"2021","unstructured":"J. Zhang, J. Zhang, E. Bj\u00f6rnson, B. Ai, Local partial zero-forcing combining for cell-free massive MIMO systems. IEEE Trans. Commun.Commun. 69(12), 8459\u20138473 (2021)","journal-title":"IEEE Trans. Commun.Commun."},{"issue":"11","key":"1111_CR36","doi-asserted-by":"publisher","first-page":"5299","DOI":"10.1109\/TWC.2019.2935434","volume":"18","author":"\u00d6 \u00d6zdogan","year":"2019","unstructured":"\u00d6. \u00d6zdogan, E. Bj\u00f6rnson, J. Zhang, Performance of cell-free massive MIMO with rician fading and phase shifts. IEEE Trans. Wireless Commun.Commun. 18(11), 5299\u20135315 (2019)","journal-title":"IEEE Trans. Wireless Commun.Commun."},{"key":"1111_CR37","doi-asserted-by":"crossref","unstructured":"Y. Zhang, J. Zhang, Y. Jin, S. Buzzi and B. Ai, \u201cDeep learning-based power control for uplink cell-free massive MIMO systems,\u201d 2021 IEEE global communications conference (GLOBECOM), Madrid, Spain, 2021, pp. 1-6","DOI":"10.1109\/GLOBECOM46510.2021.9685827"},{"issue":"7","key":"1111_CR38","doi-asserted-by":"publisher","first-page":"4247","DOI":"10.1109\/TCOMM.2020.2987311","volume":"68","author":"E Bj\u00f6rnson","year":"2020","unstructured":"E. Bj\u00f6rnson, L. Sanguinetti, Scalable cell-free massive MIMO systems. IEEE Trans. Commun.Commun. 68(7), 4247\u20134261 (2020)","journal-title":"IEEE Trans. Commun.Commun."}],"container-title":["EURASIP Journal on Advances in Signal Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-024-01111-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13634-024-01111-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-024-01111-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,26]],"date-time":"2024-01-26T02:11:45Z","timestamp":1706235105000},"score":1,"resource":{"primary":{"URL":"https:\/\/asp-eurasipjournals.springeropen.com\/articles\/10.1186\/s13634-024-01111-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,26]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["1111"],"URL":"https:\/\/doi.org\/10.1186\/s13634-024-01111-9","relation":{},"ISSN":["1687-6180"],"issn-type":[{"value":"1687-6180","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,26]]},"assertion":[{"value":"1 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 January 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 January 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no Competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"18"}}