{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T18:06:34Z","timestamp":1782324394456,"version":"3.54.5"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,10,2]],"date-time":"2021-10-02T00:00:00Z","timestamp":1633132800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,10,2]],"date-time":"2021-10-02T00:00:00Z","timestamp":1633132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"natural science foundation of china","doi-asserted-by":"crossref","award":["62071377"],"award-info":[{"award-number":["62071377"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"key project of natural science foundation of shaanxi province","award":["2021JM-465"],"award-info":[{"award-number":["2021JM-465"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["EURASIP J. Adv. Signal Process."],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper investigates a computing offloading policy and the allocation of computational resource for multiple user equipments (UEs) in device-to-device (D2D)-aided fog radio access networks (F-RANs). Concerning the dynamically changing wireless environment where the channel state information (CSI) is difficult to predict and know exactly, we formulate the problem of task offloading and resource optimization as a mixed-integer nonlinear programming problem to maximize the total utility of all UEs. Concerning the non-convex property of the formulated problem, we decouple the original problem into two phases to solve. Firstly, a centralized deep reinforcement learning (DRL) algorithm called dueling deep Q-network (DDQN) is utilized to obtain the most suitable offloading mode for each UE. Particularly, to reduce the complexity of the proposed offloading scheme-based DDQN algorithm, a pre-processing procedure is adopted. Then, a distributed deep Q-network (DQN) algorithm based on the training result of the DDQN algorithm is further proposed to allocate the appropriate computational resource for each UE. Combining these two phases, the optimal offloading policy and resource allocation for each UE are finally achieved. Simulation results demonstrate the performance gains of the proposed scheme compared with other existing baseline schemes.<\/jats:p>","DOI":"10.1186\/s13634-021-00802-x","type":"journal-article","created":{"date-parts":[[2021,10,4]],"date-time":"2021-10-04T14:45:42Z","timestamp":1633358742000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["A reinforcement learning-based computing offloading and resource allocation scheme in F-RAN"],"prefix":"10.1186","volume":"2021","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8968-5178","authenticated-orcid":false,"given":"Fan","family":"Jiang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rongxin","family":"Ma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Youjun","family":"Gao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zesheng","family":"Gu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,10,2]]},"reference":[{"key":"802_CR1","unstructured":"M. Latva-Aho, K. Lepp\u00e4nen, Key drivers and research challenges for 6G ubiquitous wireless intelligence (white paper). Oulu, Finland: 6G Flagship, (2019)"},{"key":"802_CR2","doi-asserted-by":"publisher","first-page":"104876","DOI":"10.1109\/ACCESS.2019.2929075","volume":"7","author":"Y Lan","year":"2019","unstructured":"Y. Lan, X. Wang, D. Wang, Z. Liu et al., Task caching, offloading, and resource allocation in D2D-aided fog computing networks. IEEE Access 7, 104876\u2013104891 (2019)","journal-title":"IEEE Access"},{"key":"802_CR3","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.3015522","author":"M Yang","year":"2020","unstructured":"M. Yang, H. Zhu, H. Wang, Y. Koucheryavy, K. Samouylov, H. Qian, An online learning approach to computation offloading in dynamic fog networks. IEEE Internet Things J (2020). https:\/\/doi.org\/10.1109\/JIOT.2020.3015522","journal-title":"IEEE Internet Things J"},{"key":"802_CR4","doi-asserted-by":"publisher","first-page":"108310","DOI":"10.1109\/ACCESS.2020.3000832","volume":"8","author":"Y Ma","year":"2020","unstructured":"Y. Ma, H. Wang, J. Xiong, J. Diao, D. Ma, Joint allocation on communication and computing resources for fog radio access networks. IEEE Access 8, 108310\u2013108323 (2020). https:\/\/doi.org\/10.1109\/ACCESS.2020.3000832","journal-title":"IEEE Access"},{"issue":"10","key":"802_CR5","doi-asserted-by":"publisher","first-page":"6790","DOI":"10.1109\/TWC.2018.2864559","volume":"17","author":"M Chen","year":"2018","unstructured":"M. Chen, B. Liang, M. Dong, Multi-user multi-task offloading and resource allocation in mobile cloud systems. IEEE Trans Wirel Commun 17(10), 6790\u20136805 (2018). https:\/\/doi.org\/10.1109\/TWC.2018.2864559","journal-title":"IEEE Trans Wirel Commun"},{"issue":"3","key":"802_CR6","first-page":"32","volume":"16","author":"Q Li","year":"2019","unstructured":"Q. Li, J. Zhao, Y. Gong et al., Energy-efficient computation offloading and resource allocation in fog computing for internet of everything. China Commun 16(3), 32\u201341 (2019)","journal-title":"China Commun"},{"key":"802_CR7","doi-asserted-by":"publisher","first-page":"104876","DOI":"10.1109\/ACCESS.2019.2929075","volume":"7","author":"Y Lan","year":"2019","unstructured":"Y. Lan, X. Wang, D. Wang et al., Task caching, offloading, and resource allocation in D2D-aided fog computing networks. IEEE Access 7, 104876\u2013104891 (2019)","journal-title":"IEEE Access"},{"key":"802_CR8","doi-asserted-by":"crossref","unstructured":"X. Chen, J. Zhang, When D2D meets cloud: hybrid mobile task offloading in fog computing, in 2017 IEEE International Conference on Communications (ICC). IEEE , 1\u20136 (2017)","DOI":"10.1109\/ICC.2017.7996590"},{"key":"802_CR9","doi-asserted-by":"crossref","unstructured":"A. Bozorgchenani, D. Tarchi, G.E. Corazza, Mobile edge computing partial offloading techniques for mobile urban scenarios, in IEEE Global Communications Conference (GLOBECOM), IEEE, (2018), pp. 1\u20136","DOI":"10.1109\/GLOCOM.2018.8647240"},{"key":"802_CR10","unstructured":"N. Docomo, White paper 5g evolution and 6g. Accessed on 1(2020)"},{"key":"802_CR11","doi-asserted-by":"crossref","unstructured":"F. Jiang, W. Liu, J. Wang, X. Liu, Q-learning based task offloading and resource allocation scheme for internet of vehicles, in 2020 IEEE\/CIC International Conference on Communications in China (ICCC), Chongqing, China, (2020), pp. 460\u2013465, https:\/\/doi.org\/10.1109\/ICCC49849.2020.9238925","DOI":"10.1109\/ICCC49849.2020.9238925"},{"key":"802_CR12","doi-asserted-by":"publisher","first-page":"179349","DOI":"10.1109\/ACCESS.2019.2959348","volume":"7","author":"H Ke","year":"2019","unstructured":"H. Ke, J. Wang, H. Wang, Y. Ge, Joint optimization of data offloading and resource allocation with renewable energy aware for IoT devices: a deep reinforcement learning approach. IEEE Access 7, 179349\u2013179363 (2019). https:\/\/doi.org\/10.1109\/ACCESS.2019.2959348","journal-title":"IEEE Access"},{"key":"802_CR13","doi-asserted-by":"crossref","unstructured":"S. Nath, Y. Li, J. Wu, P. Fan, Multi-user multi-channel computation offloading and resource allocation for mobile edge computing, in ICC 2020\u20142020 IEEE International Conference on Communications (ICC), Dublin, Ireland, (2020), pp. 1-6, https:\/\/doi.org\/10.1109\/ICC40277.2020.9149124","DOI":"10.1109\/ICC40277.2020.9149124"},{"issue":"3","key":"802_CR14","doi-asserted-by":"publisher","first-page":"4005","DOI":"10.1109\/JIOT.2018.2876279","volume":"6","author":"X Chen","year":"2019","unstructured":"X. Chen, H. Zhang, C. Wu, S. Mao, Y. Ji, M. Bennis, Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning. IEEE Internet Things J 6(3), 4005\u20134018 (2019). https:\/\/doi.org\/10.1109\/JIOT.2018.2876279","journal-title":"IEEE Internet Things J"},{"key":"802_CR15","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.3009540","author":"J Baek","year":"2020","unstructured":"J. Baek, G. Kaddoum, Heterogeneous task offloading and resource allocations via deep recurrent reinforcement learning in partial observable multi-fog networks. IEEE Internet Things J (2020). https:\/\/doi.org\/10.1109\/JIOT.2020.3009540","journal-title":"IEEE Internet Things J"},{"key":"802_CR16","unstructured":"Z. Wang, T. Schaul, M. Hessel, et al. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581 (2015)"},{"key":"802_CR17","doi-asserted-by":"crossref","unstructured":"Y. Ouyang, Task offloading algorithm of vehicle edge computing environment based on Dueling-DQN, in Journal of Physics: Conference Series, Vol. 1873. No. 1. IOP Publishing, (2021)","DOI":"10.1088\/1742-6596\/1873\/1\/012046"},{"key":"802_CR18","doi-asserted-by":"publisher","first-page":"118192","DOI":"10.1109\/ACCESS.2020.3004861","volume":"8","author":"S Song","year":"2020","unstructured":"S. Song, Z. Fang, Z. Zhang, C. Chen, H. Sun, Semi-online computational offloading by dueling deep-Q network for user behavior prediction. IEEE Access 8, 118192\u2013118204 (2020). https:\/\/doi.org\/10.1109\/ACCESS.2020.3004861","journal-title":"IEEE Access"},{"key":"802_CR19","doi-asserted-by":"crossref","unstructured":"F. Jiang, R. Ma, C. Sun, Z. Gu, Dueling deep Q-network learning based computing offloading scheme for F-RAN, in IEEE 31st Annual International Symposium on Personal. Indoor and Mobile Radio Communications, London, UK 2020, 1\u20136 (2020). https:\/\/doi.org\/10.1109\/PIMRC48278.2020.9217355","DOI":"10.1109\/PIMRC48278.2020.9217355"},{"key":"802_CR20","doi-asserted-by":"publisher","first-page":"97505","DOI":"10.1109\/ACCESS.2019.2927836","volume":"7","author":"F Jiang","year":"2019","unstructured":"F. Jiang, Z. Yuan, C. Sun, J. Wang, Deep Q-learning-based content caching with update strategy for fog radio access networks. IEEE Access 7, 97505\u201397514 (2019). https:\/\/doi.org\/10.1109\/ACCESS.2019.2927836","journal-title":"IEEE Access"},{"key":"802_CR21","doi-asserted-by":"publisher","first-page":"19324","DOI":"10.1109\/ACCESS.2018.2819690","volume":"6","author":"J Zhang","year":"2018","unstructured":"J. Zhang, W. Xia, F. Yan, L. Shen, Joint computation offloading and resource allocation optimization in heterogeneous networks with mobile edge computing. IEEE Access 6, 19324\u201319337 (2018)","journal-title":"IEEE Access"},{"issue":"1","key":"802_CR22","doi-asserted-by":"publisher","first-page":"89","DOI":"10.32604\/cmc.2019.04836","volume":"59","author":"Y Wei","year":"2019","unstructured":"Y. Wei, Z. Wang, D. Guo, F.R. Yu, Deep q-learning based computation offloading strategy for mobile edge computing. Comput. Mater. Continua 59(1), 89\u2013104 (2019). https:\/\/doi.org\/10.32604\/cmc.2019.04836","journal-title":"Comput. Mater. Continua"},{"issue":"2","key":"802_CR23","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1109\/JSAC.2020.3020659","volume":"39","author":"L Zhang","year":"2021","unstructured":"L. Zhang, B. Cao, Y. Li, M. Peng, G. Feng, A multi-stage stochastic programming-based offloading policy for fog enabled IoT-eHealth. IEEE J. Sel. Areas Commun. 39(2), 411\u2013425 (2021). https:\/\/doi.org\/10.1109\/JSAC.2020.3020659","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"802_CR24","doi-asserted-by":"crossref","unstructured":"A.A. Majeed, P. Kilpatrick, I. Spence, B. Varghese, Modelling fog offloading performance, in 2020 IEEE 4th International Conference on Fog and Edge Computing (ICFEC), Melbourne, VIC, Australia, 2020, pp. 29\u201338, https:\/\/doi.org\/10.1109\/ICFEC50348.2020.00011","DOI":"10.1109\/ICFEC50348.2020.00011"},{"key":"802_CR25","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2020.3036871","author":"M Tang","year":"2020","unstructured":"M. Tang, V.W.S. Wong, Deep reinforcement learning for task offloading in mobile edge computing systems. IEEE Trans. Mob. Comput. (2020). https:\/\/doi.org\/10.1109\/TMC.2020.3036871","journal-title":"IEEE Trans. Mob. Comput."},{"key":"802_CR26","doi-asserted-by":"publisher","first-page":"85204","DOI":"10.1109\/ACCESS.2020.2991773","volume":"8","author":"Y Li","year":"2020","unstructured":"Y. Li, F. Qi, Z. Wang, X. Yu, S. Shao, Distributed edge computing offloading algorithm based on deep reinforcement learning. IEEE Access 8, 85204\u201385215 (2020). https:\/\/doi.org\/10.1109\/ACCESS.2020.2991773","journal-title":"IEEE Access"},{"key":"802_CR27","doi-asserted-by":"crossref","unstructured":"D. Van Le, C. Tham, A deep reinforcement learning based offloading scheme in ad-hoc mobile clouds, in IEEE INFOCOM 2018\u2014IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), Honolulu, HI, (2018), pp. 760\u2013765, https:\/\/doi.org\/10.1109\/INFCOMW.2018.8406881","DOI":"10.1109\/INFCOMW.2018.8406881"},{"key":"802_CR28","doi-asserted-by":"publisher","first-page":"66588","DOI":"10.1109\/ACCESS.2020.2985679","volume":"8","author":"C Huang","year":"2020","unstructured":"C. Huang, P. Chen, Joint demand forecasting and DQN-based control for energy-aware mobile traffic offloading. IEEE Access 8, 66588\u201366597 (2020). https:\/\/doi.org\/10.1109\/ACCESS.2020.2985679","journal-title":"IEEE Access"},{"issue":"3","key":"802_CR29","doi-asserted-by":"publisher","first-page":"4436","DOI":"10.1109\/JIOT.2018.2882783","volume":"6","author":"Z Wei","year":"2019","unstructured":"Z. Wei, B. Zhao, J. Su, X. Lu, Dynamic edge computation offloading for internet of things with energy harvesting: a learning method. IEEE Internet Things J 6(3), 4436\u20134447 (2019). https:\/\/doi.org\/10.1109\/JIOT.2018.2882783","journal-title":"IEEE Internet Things J"},{"key":"802_CR30","doi-asserted-by":"crossref","unstructured":"Y. Huang, G. Wei, Y. Wang, V-D D3QN: the variant of double deep Q-learning network with dueling architecture, in 2018 37th Chinese Control Conference (CCC), Wuhan, (2018), pp. 9130\u20139135, https:\/\/doi.org\/10.23919\/ChiCC.2018.8483478","DOI":"10.23919\/ChiCC.2018.8483478"},{"key":"802_CR31","doi-asserted-by":"publisher","first-page":"186474","DOI":"10.1109\/ACCESS.2020.3029868","volume":"8","author":"B-A Han","year":"2020","unstructured":"B.-A. Han, J.-J. Yang, Research on adaptive job shop scheduling problems based on dueling double DQN. IEEE Access 8, 186474\u2013186495 (2020). https:\/\/doi.org\/10.1109\/ACCESS.2020.3029868","journal-title":"IEEE Access"},{"key":"802_CR32","doi-asserted-by":"crossref","unstructured":"H. Sasaki, T. Horiuchi, S. Kato, A study on vision-based mobile robot learning by deep Q-network, in 2017 56th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE), Kanazawa, (2017), pp. 799\u2013804, https:\/\/doi.org\/10.23919\/SICE.2017.8105597","DOI":"10.23919\/SICE.2017.8105597"},{"key":"802_CR33","doi-asserted-by":"publisher","first-page":"40797","DOI":"10.1109\/ACCESS.2019.2907618","volume":"7","author":"H Ge","year":"2019","unstructured":"H. Ge, Y. Song, C. Wu, J. Ren, G. Tan, Cooperative deep Q-learning with Q-value transfer for multi-intersection signal control. IEEE Access 7, 40797\u201340809 (2019). https:\/\/doi.org\/10.1109\/ACCESS.2019.2907618","journal-title":"IEEE Access"},{"issue":"3","key":"802_CR34","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1109\/TCC.2014.2350471","volume":"4","author":"K Elgazzar","year":"2016","unstructured":"K. Elgazzar, P. Martin, H.S. Hassanein, Cloud-assisted computation offloading to support mobile services. IEEE Trans. Cloud Comput. 4(3), 279\u2013292 (2016). https:\/\/doi.org\/10.1109\/TCC.2014.2350471","journal-title":"IEEE Trans. Cloud Comput."},{"key":"802_CR35","doi-asserted-by":"crossref","unstructured":"P. Ajay Rao, B. Navaneesh Kumar, S. Cadabam, T. Praveena, Distributed deep reinforcement learning using tensorflow, in 2017 International Conference on Current Trends in Computer, Electrical, Electronics and Communication (CTCEEC), Mysore, (2017), pp. 171\u2013174, https:\/\/doi.org\/10.1109\/CTCEEC.2017.8455196","DOI":"10.1109\/CTCEEC.2017.8455196"}],"container-title":["EURASIP Journal on Advances in Signal Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-021-00802-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13634-021-00802-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-021-00802-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,4]],"date-time":"2021-10-04T15:18:04Z","timestamp":1633360684000},"score":1,"resource":{"primary":{"URL":"https:\/\/asp-eurasipjournals.springeropen.com\/articles\/10.1186\/s13634-021-00802-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,2]]},"references-count":35,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["802"],"URL":"https:\/\/doi.org\/10.1186\/s13634-021-00802-x","relation":{},"ISSN":["1687-6180"],"issn-type":[{"value":"1687-6180","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,2]]},"assertion":[{"value":"7 May 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 October 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"The manuscript does not contain any individual person\u2019s data in any form (including individual details, images, or videos) and therefore the consent to publish is not applicable to this article.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"91"}}