{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T02:31:30Z","timestamp":1773196290979,"version":"3.50.1"},"reference-count":72,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2021,11,27]],"date-time":"2021-11-27T00:00:00Z","timestamp":1637971200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Technology development Program funded by the Ministry of SMEs and Startups (MSS, Korea).","award":["no.G21S290100202"],"award-info":[{"award-number":["no.G21S290100202"]}]},{"name":"Gachon University Research","award":["202008460005"],"award-info":[{"award-number":["202008460005"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Wireless networking using GHz or THz spectra has encouraged mobile service providers to deploy small cells to improve link quality and cell capacity using mmWave backhaul links. As green networking for less CO2 emission is mandatory to confront global climate change, we need energy efficient network management for such denser small-cell heterogeneous networks (HetNets) that already suffer from observable power consumption. We establish a dual-objective optimization model that minimizes energy consumption by switching off unused small cells while maximizing user throughput, which is a mixed integer linear problem (MILP). Recently, the deep reinforcement learning (DRL) algorithm has been applied to many NP-hard problems of the wireless networking field, such as radio resource allocation, association and power saving, which can induce a near-optimal solution with fast inference time as an online solution. In this paper, we investigate the feasibility of the DRL algorithm for a dual-objective problem, energy efficient routing and throughput maximization, which has not been explored before. We propose a proximal policy (PPO)-based multi-objective algorithm using the actor-critic model that is realized as an optimistic linear support framework in which the PPO algorithm searches for feasible solutions iteratively. Experimental results show that our algorithm can achieve throughput and energy savings comparable to the CPLEX.<\/jats:p>","DOI":"10.3390\/s21237925","type":"journal-article","created":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T01:45:02Z","timestamp":1638323102000},"page":"7925","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Multi-Objective Optimization of Energy Saving and Throughput in Heterogeneous Networks Using Deep Reinforcement Learning"],"prefix":"10.3390","volume":"21","author":[{"given":"Kyungho","family":"Ryu","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Gachon University, Seongnam 13120, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0955-3421","authenticated-orcid":false,"given":"Wooseong","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Gachon University, Seongnam 13120, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1109\/ACCESS.2013.2260813","article-title":"Millimeter wave mobile communications for 5G cellular: It will work!","volume":"1","author":"Rappaport","year":"2013","journal-title":"Access IEEE"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Sun, S., MacCartney, G.R., Samimi, M.K., Nie, S., and Rappaport, T.S. (2014, January 10\u201314). Millimeter wave multi-beam antenna combining for 5G cellular link improvement in New York City. Proceedings of the 2014 IEEE International Conference on Communications (ICC), Sydney, NSW, Australia.","DOI":"10.1109\/ICC.2014.6884191"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Maccartney, G.R., and Rappaport, T.S. (2014, January 10\u201314). 73 GHz millimeter wave propagation measurements for outdoor urban mobile and backhaul communications in New York City. Proceedings of the 2014 IEEE International Conference on Communications (ICC), Sydney, NSW, Australia.","DOI":"10.1109\/ICC.2014.6884090"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1109\/MCOM.2014.6894457","article-title":"Millimeter-wave access and backhauling: The solution to the exponential data traffic increase in 5G mobile communications systems?","volume":"52","author":"Dehos","year":"2014","journal-title":"Commun. Mag. IEEE"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1109\/MCOM.2015.7010531","article-title":"Multi-gigabit millimeter wave wireless communications for 5G: From fixed access to cellular networks","volume":"53","author":"Wang","year":"2015","journal-title":"Commun. Mag. IEEE"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1109\/MCOM.2015.7010534","article-title":"Point-to-multipoint in-band mmwave backhaul for 5G networks","volume":"53","author":"Taori","year":"2015","journal-title":"IEEE Commun. Mag."},{"key":"ref_7","unstructured":"Zhu, Y., Niu, Y., Li, J., Wu, D.O., Li, Y., and Jin, D. (2021, October 10). QoS-Aware Scheduling for Small Cell Millimeter Wave Mesh Backhaul. Available online: https:\/\/ieeexplore.ieee.org\/document\/7511065."},{"key":"ref_8","unstructured":"Nakamura, M., Tran, G.K., and Sakaguchi, K. (2021, October 10). Interference Management for Millimeter-Wave Mesh Backhaul Networks. Available online: https:\/\/ieeexplore.ieee.org\/document\/8651725."},{"key":"ref_9","unstructured":"Zola, E., Kassler, A.J., and Kim, W. (2021, October 10). Joint User Association and Energy Aware Routing for Green Small Cell Mmwave Backhaul Networks. Available online: https:\/\/ieeexplore.ieee.org\/document\/7925706."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1743","DOI":"10.1109\/ACCESS.2016.2556011","article-title":"5G backhaul challenges and emerging research directions: A survey","volume":"4","author":"Jaber","year":"2016","journal-title":"IEEE Access"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1109\/MCOM.2010.5621969","article-title":"Challenges and enabling technologies for energy aware mobile radio networks","volume":"48","author":"Correia","year":"2010","journal-title":"IEEE Commun. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/MCOM.2011.5978418","article-title":"Sleep mode techniques for small cell deployments","volume":"49","author":"Ashraf","year":"2011","journal-title":"IEEE Commun. Mag."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"840","DOI":"10.1109\/JSAC.2013.130503","article-title":"Energy efficient heterogeneous cellular networks","volume":"31","author":"Soh","year":"2013","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1016\/j.comnet.2014.10.028","article-title":"Energy-efficient BS switching-off and cell topology management for macro\/femto environments","volume":"78","author":"Nuaymi","year":"2015","journal-title":"Comput. Netw."},{"key":"ref_15","unstructured":"Liu, C., Pan, Z., Liu, N., and You, X. (2021, October 10). A Novel Energy Saving Strategy for LTE HetNet. Available online: https:\/\/ieeexplore.ieee.org\/document\/6096845."},{"key":"ref_16","unstructured":"Bhaumik, S., Narlikar, G., Chattopadhyay, S., and Kanugovi, S. (2021, October 10). Breathe to Stay Cool: ADJUSTING Cell Sizes to Reduce Energy Consumption. Available online: https:\/\/dl.acm.org\/doi\/10.1145\/1851290.1851300."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3709","DOI":"10.1109\/JSAC.2016.2611846","article-title":"Green full-duplex self-backhaul and energy harvesting small cell networks with massive MIMO","volume":"34","author":"Chen","year":"2016","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1810","DOI":"10.1109\/TVT.2016.2565539","article-title":"Energy-and spectrum-efficient user association in millimeter-wave backhaul small-cell networks","volume":"66","author":"Mesodiakaki","year":"2016","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2457","DOI":"10.1109\/ACCESS.2017.2783544","article-title":"Energy-efficient resource allocation for mmWave massive MIMO HetNets with wireless backhaul","volume":"6","author":"Hao","year":"2017","journal-title":"IEEE Access"},{"key":"ref_20","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_21","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2021, October 10). Deterministic Policy Gradient Algorithms. Available online: http:\/\/proceedings.mlr.press\/v32\/silver14.pdf."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_23","unstructured":"Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2021, October 10). Trust Region Policy Optimization. Available online: https:\/\/arxiv.org\/abs\/1502.05477."},{"key":"ref_24","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv."},{"key":"ref_25","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2021, October 10). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Available online: https:\/\/arxiv.org\/abs\/1801.01290."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1109\/TCCN.2018.2809722","article-title":"Deep reinforcement learning for dynamic multichannel access in wireless networks","volume":"4","author":"Wang","year":"2018","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_27","unstructured":"Zhong, C., Lu, Z., Gursoy, M.C., and Velipasalar, S. (2021, October 10). Actor-Critic Deep Reinforcement Learning for Dynamic Multichannel Access. Available online: https:\/\/ieeexplore.ieee.org\/document\/8646405."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1109\/TCCN.2019.2952909","article-title":"A deep actor-critic reinforcement learning framework for dynamic multichannel access","volume":"5","author":"Zhong","year":"2019","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_29","unstructured":"Naparstek, O., and Cohen, K. (2021, October 10). Deep Multi-User Reinforcement Learning for Dynamic Spectrum Access in Multichannel Wireless Networks. Available online: https:\/\/arxiv.org\/abs\/1704.02613."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1109\/TWC.2018.2879433","article-title":"Deep multi-user reinforcement learning for distributed dynamic spectrum access","volume":"18","author":"Naparstek","year":"2018","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"464","DOI":"10.1109\/TCCN.2020.2982895","article-title":"Deep reinforcement learning for dynamic spectrum sensing and aggregation in multi-channel wireless networks","volume":"6","author":"Li","year":"2020","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"25473","DOI":"10.1109\/ACCESS.2021.3056670","article-title":"Dynamic Multichannel Sensing in Cognitive Radio: Hierarchical Reinforcement Learning","volume":"9","author":"Liu","year":"2021","journal-title":"IEEE Access"},{"key":"ref_33","unstructured":"He, Y., Yu, F.R., Zhao, N., Yin, H., and Boukerche, A. (2021, October 10). Deep Reinforcement Learning (DRL)-Based Resource Management in Softwaredefined and Virtualized Vehicular Ad Hoc Networks. Available online: https:\/\/www.semanticscholar.org\/paper\/Deep-Reinforcement-Learning-(DRL)-based-Resource-in-He-Yu\/e1d5360a49ee5269298a54c49d171661ffc245d6."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"9800","DOI":"10.1109\/JIOT.2020.3020067","article-title":"Drone-cell trajectory planning and resource allocation for highly mobile networks: A hierarchical DRL approach","volume":"8","author":"Shi","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"7279","DOI":"10.1109\/JIOT.2020.2982699","article-title":"DRL-based energy-efficient resource allocation frameworks for uplink NOMA systems","volume":"7","author":"Wang","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"5083","DOI":"10.1109\/TWC.2021.3065523","article-title":"Resource allocation in uplink NOMA-IoT networks: A reinforcement-learning approach","volume":"20","author":"Ahsan","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Rahimi, A.M., Ziaeddini, A., and Gonglee, S. (2021). A novel approach to efficient resource allocation in load-balanced cellular networks using hierarchical DRL. J. Ambient. Intell. Humaniz. Comput., 1\u201315.","DOI":"10.1007\/s12652-021-03174-0"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"3500","DOI":"10.1109\/ACCESS.2018.2886216","article-title":"Deep reinforcement learning paradigm for performance optimization of channel observation\u2013based MAC protocols in dense WLANs","volume":"7","author":"Ali","year":"2018","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1277","DOI":"10.1109\/JSAC.2019.2904329","article-title":"Deep-reinforcement learning multiple access for heterogeneous wireless networks","volume":"37","author":"Yu","year":"2019","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"108088","DOI":"10.1109\/ACCESS.2020.3000893","article-title":"Learn to Schedule (LEASCH): A Deep reinforcement learning approach for radio resource scheduling in the 5G MAC layer","volume":"8","author":"Correia","year":"2020","journal-title":"IEEE Access"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2211","DOI":"10.1109\/JSAC.2019.2933887","article-title":"Robust coordinated reinforcement learning for MAC design in sensor networks","volume":"37","author":"Nisioti","year":"2019","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2059","DOI":"10.1109\/JSAC.2018.2864373","article-title":"Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach","volume":"36","author":"Liu","year":"2018","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Liu, J., Zhao, B., Xin, Q., Su, J., and Ou, W. (2020). DRL-ER: An Intelligent Energy-aware Routing Protocol with Guaranteed Delay Bounds in Satellite Mega-constellations. IEEE Trans. Netw. Sci. Eng.","DOI":"10.1109\/TNSE.2020.3039499"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"El Amine, A., Dini, P., and Nuaymi, L. (2021, October 10). Reinforcement Learning for Delay-Constrained Energy-Aware Small Cells with Multi-Sleeping Control. Available online: https:\/\/ieeexplore.ieee.org\/document\/9145431.","DOI":"10.1109\/ICCWorkshops49005.2020.9145431"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"133474","DOI":"10.1109\/ACCESS.2019.2939827","article-title":"Channel access and power control for energy-efficient delay-aware heterogeneous cellular networks for smart grid communications using deep reinforcement learning","volume":"7","author":"Asuhaimi","year":"2019","journal-title":"IEEE Access"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Hsieh, C.K., Chan, K.L., and Chien, F.T. (2021). Energy-efficient power allocation and user association in heterogeneous networks with deep reinforcement learning. Appl. Sci., 11.","DOI":"10.3390\/app11094135"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1613\/jair.3987","article-title":"A survey of multi-objective sequential decision-making","volume":"48","author":"Roijers","year":"2013","journal-title":"J. Artif. Intell. Res."},{"key":"ref_48","unstructured":"Mossalam, H., Assael, Y.M., Roijers, D.M., and Whiteson, S. (2016). Multi-objective deep reinforcement learning. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"989","DOI":"10.1109\/JSTSP.2019.2925975","article-title":"Deep CNN-based channel estimation for mmWave massive MIMO systems","volume":"13","author":"Dong","year":"2019","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"7601","DOI":"10.1109\/JIOT.2020.2986442","article-title":"A novel OFDM autoencoder featuring CNN-based channel estimation for internet of vessels","volume":"7","author":"Lin","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"5859","DOI":"10.1109\/TCOMM.2021.3085895","article-title":"Dual CNN based Channel Estimation for MIMO-OFDM Systems","volume":"69","author":"Jiang","year":"2021","journal-title":"IEEE Trans. Commun."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"66496","DOI":"10.1109\/ACCESS.2019.2918136","article-title":"Fusion methods for CNN-based automatic modulation classification","volume":"7","author":"Zheng","year":"2019","journal-title":"IEEE Access"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1109\/LCOMM.2020.2968030","article-title":"MCNet: An efficient CNN architecture for robust automatic modulation classification","volume":"24","author":"Hua","year":"2020","journal-title":"IEEE Commun. Lett."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"1038","DOI":"10.1109\/LCOMM.2020.2970922","article-title":"CNN-based automatic modulation classification for beyond 5G communications","volume":"24","author":"Hermawan","year":"2020","journal-title":"IEEE Commun. Lett."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Vinayakumar, R., Soman, K., and Poornachandran, P. (2017, January 13\u201316). Applying convolutional neural network for network intrusion detection. Proceedings of the 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Udupi, India.","DOI":"10.1109\/ICACCI.2017.8126009"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Kim, J., Kim, J., Kim, H., Shim, M., and Choi, E. (2020). CNN-based network intrusion detection against denial-of-service attacks. Electronics, 9.","DOI":"10.3390\/electronics9060916"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"17265","DOI":"10.1007\/s00500-020-05017-0","article-title":"A deep learning approach for effective intrusion detection in wireless networks using CNN","volume":"24","author":"Riyaz","year":"2020","journal-title":"Soft Comput."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Azizjon, M., Jumabek, A., and Kim, W. (2021, October 10). 1D CNN Based Network Intrusion Detection with Normalization on Imbalanced Data. Available online: https:\/\/ieeexplore.ieee.org\/document\/9064976.","DOI":"10.1109\/ICAIIC48513.2020.9064976"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"5141","DOI":"10.1109\/TWC.2019.2933417","article-title":"Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks","volume":"18","author":"Zhao","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"4535","DOI":"10.1109\/TWC.2020.2984758","article-title":"Intelligent user association for symbiotic radio networks using deep reinforcement learning","volume":"19","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"102069","DOI":"10.1016\/j.adhoc.2019.102069","article-title":"A deep reinforcement learning for user association and power control in heterogeneous networks","volume":"102","author":"Ding","year":"2020","journal-title":"Ad Hoc Netw."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/MVT.2020.3015184","article-title":"Green Deep Reinforcement Learning for Radio Resource Management: Architecture, Algorithm Compression, and Challenges","volume":"16","author":"Du","year":"2020","journal-title":"IEEE Veh. Technol. Mag."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"12175","DOI":"10.1109\/TVT.2020.3013990","article-title":"Edge intelligence for energy-efficient computation offloading and resource allocation in 5G beyond","volume":"69","author":"Dai","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","article-title":"Deep learning in neural networks: An overview","volume":"61","author":"Schmidhuber","year":"2015","journal-title":"Neural Netw."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/S0004-3702(99)00052-1","article-title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","volume":"112","author":"Sutton","year":"1999","journal-title":"Artif. Intell."},{"key":"ref_66","unstructured":"Sutton, R.S., McAllester, D.A., Singh, S.P., and Mansour, Y. (2021, October 10). Policy Gradient Methods for Reinforcement Learning with Function Approximation. Available online: https:\/\/proceedings.neurips.cc\/paper\/1999\/file\/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1109\/MWC.2011.6056691","article-title":"How much energy is needed to run a wireless network?","volume":"18","author":"Auer","year":"2011","journal-title":"IEEE Wirel. Commun."},{"key":"ref_68","unstructured":"(2021, October 10). 36.814. Evolved Universal Terrestrial Radio Access (E-UTRA); Further Advancements for E-UTRA Physical Layer Aspects. Available online: https:\/\/www.scirp.org\/(S(czeh2tfqyw2orz553k1w0r45))\/reference\/ReferencesPapers.aspx?ReferenceID=998750."},{"key":"ref_69","unstructured":"Mesodiakaki, A., Adelantado, F., Antonopoulos, A., Kartsakli, E., Alonso, L., and Verikoukis, C. (2021, October 10). Energy Impact of Outdoor Small Cell Backhaul in Green Heterogeneous Networks. Available online: https:\/\/ieeexplore.ieee.org\/document\/7033196?arnumber=7033196."},{"key":"ref_70","unstructured":"(2021, October 10). gTSC0020, gAPZ0039, gRSC0016, gRSC0015, gTSC0023, gAPZ0042. Technical Report, Gotmic. Available online: www.gotmic.se."},{"key":"ref_71","unstructured":"Fujimoto, S., Hoof, H., and Meger, D. (2021, October 10). Addressing Function Approximation Error in Actor-Critic Methods. Available online: https:\/\/arxiv.org\/abs\/1802.09477."},{"key":"ref_72","unstructured":"Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2015). High-dimensional continuous control using generalized advantage estimation. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/7925\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:36:47Z","timestamp":1760168207000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/7925"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,27]]},"references-count":72,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21237925"],"URL":"https:\/\/doi.org\/10.3390\/s21237925","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,27]]}}}