{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T15:55:44Z","timestamp":1781193344969,"version":"3.54.1"},"reference-count":43,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,2,23]],"date-time":"2022-02-23T00:00:00Z","timestamp":1645574400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Shanghai 2020 Action Plan of Technological Innovation","award":["20dz1201400"],"award-info":[{"award-number":["20dz1201400"]}]},{"name":"Shanghai Sailing Program","award":["20YF1410900"],"award-info":[{"award-number":["20YF1410900"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>In mobile edge computing systems, the edge server placement problem is mainly tackled as a multi-objective optimization problem and solved with mixed integer programming, heuristic or meta-heuristic algorithms, etc. These methods, however, have profound defect implications such as poor scalability, local optimal solutions, and parameter tuning difficulties. To overcome these defects, we propose a novel edge server placement algorithm based on deep q-network and reinforcement learning, dubbed DQN-ESPA, which can achieve optimal placements without relying on previous placement experience. In DQN-ESPA, the edge server placement problem is modeled as a Markov decision process, which is formalized with the state space, action space and reward function, and it is subsequently solved using a reinforcement learning algorithm. Experimental results using real datasets from Shanghai Telecom show that DQN-ESPA outperforms state-of-the-art algorithms such as simulated annealing placement algorithm (SAPA), Top-K placement algorithm (TKPA), K-Means placement algorithm (KMPA), and random placement algorithm (RPA). In particular, with a comprehensive consideration of access delay and workload balance, DQN-ESPA achieves up to 13.40% and 15.54% better placement performance for 100 and 300 edge servers respectively.<\/jats:p>","DOI":"10.3390\/e24030317","type":"journal-article","created":{"date-parts":[[2022,2,23]],"date-time":"2022-02-23T09:34:38Z","timestamp":1645608878000},"page":"317","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["An Edge Server Placement Method Based on Reinforcement Learning"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7062-4404","authenticated-orcid":false,"given":"Fei","family":"Luo","sequence":"first","affiliation":[{"name":"School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuai","family":"Zheng","sequence":"additional","affiliation":[{"name":"School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weichao","family":"Ding","sequence":"additional","affiliation":[{"name":"School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0517-1231","authenticated-orcid":false,"given":"Joel","family":"Fuentes","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technologies, Universidad del B\u00edo-B\u00edo, Chill\u00e1n 3780000, Chile"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yong","family":"Li","sequence":"additional","affiliation":[{"name":"School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1397","DOI":"10.1109\/TWC.2016.2633522","article-title":"Energy-efficient resource allocation for mobile-edge computation offloading","volume":"16","author":"You","year":"2016","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3590","DOI":"10.1109\/JSAC.2016.2611964","article-title":"Dynamic computation offloading for mobile-edge computing with energy harvesting devices","volume":"34","author":"Mao","year":"2016","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.jnca.2017.01.012","article-title":"MIST: Fog-based data analytics scheme with cost-efficient resource provisioning for IoT crowdsensing applications","volume":"82","author":"Arkian","year":"2017","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1109\/TETC.2015.2508382","article-title":"Cost efficient resource management in fog computing supported medical cyber-physical system","volume":"5","author":"Gu","year":"2015","journal-title":"IEEE Trans. Emerg. Top. Comput."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Taleb, T., and Ksentini, A. (2013, January 9\u201313). An analytical model for follow me cloud. Proceedings of the 2013 IEEE Global Communications Conference (GLOBECOM), Atlanta, GA, USA.","DOI":"10.1109\/GLOCOM.2013.6831252"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1109\/TCC.2016.2525987","article-title":"Follow-me cloud: When cloud services follow mobile users","volume":"7","author":"Taleb","year":"2016","journal-title":"IEEE Trans. Cloud Comput."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1109\/TMC.2019.2957804","article-title":"Delay-aware microservice coordination in mobile edge computing: A reinforcement learning approach","volume":"20","author":"Wang","year":"2019","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.jpdc.2017.09.014","article-title":"QoS prediction for service recommendations in mobile edge computing","volume":"127","author":"Wang","year":"2019","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ma, L., Wu, J., and Chen, L. (2017, January 14\u201317). DOTA: Delay bounded optimal cloudlet deployment and user association in WMANs. Proceedings of the 2017 17th IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Madrid, Spain.","DOI":"10.1109\/CCGRID.2017.34"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Xu, Z., Liang, W., Xu, W., Jia, M., and Guo, S. (2015, January 26\u201329). Capacitated cloudlet placements in wireless metropolitan area networks. Proceedings of the 2015 IEEE 40th Conference on Local Computer Networks (LCN), Clearwater Beach, FL, USA.","DOI":"10.1109\/LCN.2015.7366372"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Meng, J., Shi, W., Tan, H., and Li, X. (2017, January 10\u201311). Cloudlet placement and minimum-delay routing in cloudlet computing. Proceedings of the 2017 3rd International Conference on Big Data Computing and Communications (BIGCOM), Chengdu, China.","DOI":"10.1109\/BIGCOM.2017.58"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1016\/j.jpdc.2018.06.008","article-title":"Edge server placement in mobile edge computing","volume":"127","author":"Wang","year":"2019","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1002\/spe.2685","article-title":"User allocation-aware edge cloud placement in mobile edge computing","volume":"50","author":"Guo","year":"2020","journal-title":"Software Pract. Exp."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhang, R., Yan, F., Xia, W., Xing, S., Wu, Y., and Shen, L. (2017, January 4\u20138). An optimal roadside unit placement method for vanet localization. Proceedings of the GLOBECOM 2017-2017 IEEE Global Communications Conference, Singapore.","DOI":"10.1109\/GLOCOM.2017.8253971"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"847","DOI":"10.1016\/j.future.2019.07.019","article-title":"Optimization of lightweight task offloading strategy for mobile edge computing based on deep reinforcement learning","volume":"102","author":"Lu","year":"2020","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_16","unstructured":"Curran, W., Brys, T., Taylor, M., and Smart, W. (2015). Using PCA to efficiently represent state spaces. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Dolui, K., and Datta, S.K. (2017, January 6\u20139). Comparison of edge computing implementations: Fog computing, cloudlet and mobile edge computing. Proceedings of the 2017 Global Internet of Things Summit (GIoTS), Geneva, Switzerland.","DOI":"10.1109\/GIOTS.2017.8016213"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1016\/j.jpdc.2021.03.007","article-title":"Edge computing server placement with capacitated location allocation","volume":"153","author":"Lhderanta","year":"2021","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3778","DOI":"10.1007\/s12083-021-01208-0","article-title":"Optimize the placement of edge server between workload balancing and system delay in smart city","volume":"14","author":"Zhao","year":"2021","journal-title":"Peer-to-Peer Netw. Appl."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Liang, Y., Liu, H., and Rajan, D. (2012, January 6\u20139). Optimal placement and configuration of roadside units in vehicular networks. Proceedings of the 2012 IEEE 75th Vehicular Technology Conference (VTC Spring), Yokohama, Japan.","DOI":"10.1109\/VETECS.2012.6240345"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wang, Z., Zheng, J., Wu, Y., and Mitton, N. (2017, January 21\u201325). A centrality-based RSU deployment approach for vehicular ad hoc networks. Proceedings of the 2017 IEEE International Conference on Communications (ICC), Paris, France.","DOI":"10.1109\/ICC.2017.7996986"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Aslam, B., Amjad, F., and Zou, C.C. (2012, January 1\u20134). Optimal roadside units placement in urban areas for vehicular networks. Proceedings of the 2012 IEEE Symposium on Computers and Communications (ISCC), Cappadocia, Turkey.","DOI":"10.1109\/ISCC.2012.6249333"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"432","DOI":"10.1016\/j.comcom.2009.11.021","article-title":"Planning roadside infrastructure for information dissemination in intelligent transportation systems","volume":"33","author":"Trullols","year":"2010","journal-title":"Comput. Commun."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"851","DOI":"10.1049\/iet-its.2014.0051","article-title":"Optimal road side units placement model based on binary integer programming for efficient traffic information advertisement and discovery in vehicular environment","volume":"9","author":"Balouchzahi","year":"2015","journal-title":"IET Intell. Transp. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Premsankar, G., Ghaddar, B., Di Francesco, M., and Verago, R. (2018, January 23\u201327). Efficient placement of edge computing devices for vehicular applications in smart cities. Proceedings of the NOMS 2018\u20142018 IEEE\/IFIP Network Operations and Management Symposium, Taipei, Taiwan.","DOI":"10.1109\/NOMS.2018.8406256"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Fan, Q., and Ansari, N. (2017, January 21\u201325). Cost aware cloudlet placement for big data processing at the edge. Proceedings of the 2017 IEEE International Conference on Communications (ICC), Paris, France.","DOI":"10.1109\/ICC.2017.7996722"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"725","DOI":"10.1109\/TCC.2015.2449834","article-title":"Optimal cloudlet placement and user to cloudlet allocation in wireless metropolitan area networks","volume":"5","author":"Jia","year":"2015","journal-title":"IEEE Trans. Cloud Comput."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lewis, G., Echeverr\u00eda, S., Simanta, S., Bradshaw, B., and Root, J. (2014, January 6\u20138). Tactical cloudlets: Moving cloud computing to the edge. Proceedings of the 2014 IEEE Military Communications Conference, Baltimore, MD, USA.","DOI":"10.1109\/MILCOM.2014.238"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1373","DOI":"10.1093\/comjnl\/bxu122","article-title":"Deduplication-based energy efficient storage system in cloud environment","volume":"58","author":"Li","year":"2015","journal-title":"Comput. J."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1109\/TETC.2016.2517930","article-title":"Pricing and repurchasing for big data processing in multi-clouds","volume":"4","author":"Li","year":"2016","journal-title":"IEEE Trans. Emerg. Top. Comput."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zeng, F., Ren, Y., Deng, X., and Li, W. (2019). Cost-effective edge server placement in wireless metropolitan area networks. Sensors, 19.","DOI":"10.3390\/s19010032"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Mao, H., Alizadeh, M., Menache, I., and Kandula, S. (2016, January 9\u201310). Resource management with deep reinforcement learning. Proceedings of the 15th ACM Workshop on Hot Topics in Networks, Atlanta, GA, USA.","DOI":"10.1145\/3005745.3005750"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1007\/s11761-017-0219-8","article-title":"Optimized IoT service placement in the fog","volume":"11","author":"Skarlat","year":"2017","journal-title":"Serv. Oriented Comput. Appl."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1440","DOI":"10.1109\/TC.2015.2435781","article-title":"Cost aware service placement and load dispatching in mobile cloud systems","volume":"65","author":"Yang","year":"2015","journal-title":"IEEE Trans. Comput."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Messaoudi, F., Ksentini, A., and Bertin, P. (2017, January 4\u20138). On using edge computing for computation offloading in mobile network. Proceedings of the GLOBECOM 2017-2017 IEEE Global Communications Conference, Singapore.","DOI":"10.1109\/GLOCOM.2017.8254635"},{"key":"ref_36","first-page":"279","article-title":"Technical note q-learning","volume":"8","author":"Watkins","year":"1993","journal-title":"Reinf. Learn."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1137\/17M1122815","article-title":"Stable optimal control and semicontractive dynamic programming","volume":"56","author":"Bertsekas","year":"2018","journal-title":"SIAM J. Control. Optim."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Scheffler, K., and Young, S. (2002, January 24\u201327). Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. Proceedings of the Second International Conference on Human Language Technology Research, Citeseer, San Francisco, CA, USA.","DOI":"10.3115\/1289189.1289246"},{"key":"ref_39","unstructured":"Yang, G.S., Chen, E.K., and An, C.W. (2004, January 26\u201329). Mobile robot navigation using neural Q-learning. Proceedings of the 2004 International Conference on Machine Learning and Cybernetics (IEEE Cat. No. 04EX826), Shanghai, China."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/j.engappai.2004.08.018","article-title":"Application of reinforcement learning for agent-based production scheduling","volume":"18","author":"Wang","year":"2005","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2170","DOI":"10.1109\/TSP.2007.893228","article-title":"Q-Learning Algorithms for Constrained Markov Decision Processes With Randomized Monotone Policies: Application to MIMO Transmission Control","volume":"55","author":"Djonin","year":"2007","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1126\/science.153.3731.34","article-title":"Dynamic programming","volume":"153","author":"Bellman","year":"1966","journal-title":"Science"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/3\/317\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:25:39Z","timestamp":1760135139000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/3\/317"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,23]]},"references-count":43,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["e24030317"],"URL":"https:\/\/doi.org\/10.3390\/e24030317","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,23]]}}}