{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T15:16:23Z","timestamp":1772118983627,"version":"3.50.1"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T00:00:00Z","timestamp":1670457600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T00:00:00Z","timestamp":1670457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"The work of this paper is supported by the National Science Foundation of China","award":["No. 62171387"],"award-info":[{"award-number":["No. 62171387"]}]},{"name":"The work of this paper is supported by the National Science Foundation of China","award":["No. 62171387"],"award-info":[{"award-number":["No. 62171387"]}]},{"name":"The work of this paper is supported by the National Science Foundation of China","award":["No. 62171387"],"award-info":[{"award-number":["No. 62171387"]}]},{"name":"The work of this paper is supported by the National Science Foundation of China","award":["No. 62171387"],"award-info":[{"award-number":["No. 62171387"]}]},{"name":"The work of this paper is supported by the National Science Foundation of China","award":["No. 62171387"],"award-info":[{"award-number":["No. 62171387"]}]},{"name":"The work of this paper is supported by the National Science Foundation of China","award":["No. 62171387"],"award-info":[{"award-number":["No. 62171387"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Cloud Comp"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Multi-cloud computing is becoming a promising paradigm to provide abundant computation resources for Internet-of-Things (IoT) devices. For a multi-device multi-cloud network, the real-time computing requirements, frequently varied wireless channel gains and changeable network scale, make the system more dynamic. It is critical to satisfy the dynamic nature of network with different constraints of IoT devices in multi-cloud environment. In this paper, we establish a continuous-discrete hybrid decision offloading model, each device should learn to make coordinated actions, including cloud server selection, offloading ratio and local computation capacity. Therefore, both continuous-discrete hybrid decision and coordination among IoT devices are challenging. To this end, we first develop a probabilistic method to relax the discrete action (e.g. cloud server selection) to a continuous set. Then, by leveraging a centralized training and distributed execution strategy, we design a cooperative multi-agent deep reinforcement learning (CMADRL) based framework to minimize the total system cost in terms of the energy consumption of IoT device and the renting charge of cloud servers. Each IoT device acts as an agent, which not only learns efficient decentralized policies, but also relieves IoT devices\u2019 computing pressure. Experimental results demonstrate that the proposed CMADRL could efficiently learn dynamic offloading polices at each IoT device, and significantly outperform the four state-of-the-art DRL based agents and two heuristic algorithms with lower system cost.<\/jats:p>","DOI":"10.1186\/s13677-022-00372-9","type":"journal-article","created":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T12:02:59Z","timestamp":1670500979000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Task offloading in hybrid-decision-based multi-cloud computing network: a cooperative multi-agent deep reinforcement learning"],"prefix":"10.1186","volume":"11","author":[{"given":"Juan","family":"Chen","sequence":"first","affiliation":[]},{"given":"Peng","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Xianhua","family":"Niu","sequence":"additional","affiliation":[]},{"given":"Zongling","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Ling","family":"Xiong","sequence":"additional","affiliation":[]},{"given":"Canghong","family":"Shi","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,12,8]]},"reference":[{"issue":"3","key":"372_CR1","doi-asserted-by":"publisher","first-page":"2009","DOI":"10.1109\/COMST.2020.2989392","volume":"22","author":"K Gai","year":"2020","unstructured":"Gai K, Guo J, Zhu L, Yu S (2020) Blockchain meets cloud computing: a survey. IEEE Commun Surv Tutorials 22(3):2009\u20132030","journal-title":"IEEE Commun Surv Tutorials"},{"key":"372_CR2","doi-asserted-by":"publisher","DOI":"10.1016\/j.buildenv.2022.109513","author":"K Li","year":"2022","unstructured":"Li K, Zhao J, Hu J et al (2022) Dynamic energy efficient task offloading and resource allocation for noma-enabled iot in smart buildings and environment. Build Environ. https:\/\/doi.org\/10.1016\/j.buildenv.2022.109513","journal-title":"Build Environ"},{"key":"372_CR3","doi-asserted-by":"publisher","unstructured":"Chen C, Zeng Y, Li H, Liu Y, Wan S (2022) A multi-hop task offloading decision model in mec-enabled internet of vehicles. IEEE Internet Things J: 1. https:\/\/doi.org\/10.1109\/JIOT.2022.3143529","DOI":"10.1109\/JIOT.2022.3143529"},{"key":"372_CR4","doi-asserted-by":"publisher","unstructured":"Chen Y, Zhao F, Lu Y, Chen X () Dynamic task offloading for mobile edge computing with hybrid energy supply. Tsinghua Sci Technol https:\/\/doi.org\/10.26599\/TST.2021.9010050","DOI":"10.26599\/TST.2021.9010050"},{"key":"372_CR5","doi-asserted-by":"crossref","unstructured":"Chen Y, Xing H, Ma Z, et\u00a0al (2022) Cost-efficient edge caching for noma-enabled iot services. China Communications","DOI":"10.1155\/2022\/8072493"},{"issue":"4","key":"372_CR6","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1145\/1721654.1721672","volume":"53","author":"M Armbrust","year":"2010","unstructured":"Armbrust M, Fox A, Griffith R, Joseph AD, Katz R, Konwinski A, Lee G, Patterson D, Rabkin A, Stoica I et al (2010) A view of cloud computing. Commun ACM 53(4):50\u201358","journal-title":"Commun ACM"},{"issue":"18","key":"372_CR7","doi-asserted-by":"publisher","first-page":"1587","DOI":"10.1002\/wcm.1203","volume":"13","author":"HT Dinh","year":"2013","unstructured":"Dinh HT, Lee C, Niyato D, Wang P (2013) A survey of mobile cloud computing: architecture, applications, and approaches. Wirel Commun Mob Comput 13(18):1587\u20131611","journal-title":"Wirel Commun Mob Comput"},{"key":"372_CR8","doi-asserted-by":"publisher","unstructured":"Huang J, Tong Z, Feng Z (2022) Geographical poi recommendation for internet of things: A federated learning approach using matrix factorization. Int J Commun Syst e5161 https:\/\/doi.org\/10.1002\/dac.5161","DOI":"10.1002\/dac.5161"},{"key":"372_CR9","doi-asserted-by":"publisher","unstructured":"Apostolopoulos PA, Fragkos G, Tsiropoulou EE, Papavassiliou S (2021) Data offloading in uav-assisted multi-access edge computing systems under resource uncertainty. IEEE Trans Mob Comput: 1. https:\/\/doi.org\/10.1109\/TMC.2021.3069911","DOI":"10.1109\/TMC.2021.3069911"},{"key":"372_CR10","doi-asserted-by":"publisher","unstructured":"Tang X (2021) Reliability-aware cost-efficient scientific workflows scheduling strategy on multi-cloud systems. IEEE Trans Cloud Comput: 1. https:\/\/doi.org\/10.1109\/TCC.2021.3057422","DOI":"10.1109\/TCC.2021.3057422"},{"key":"372_CR11","doi-asserted-by":"publisher","unstructured":"Addya SK, Satpathy A, Ghosh BC, Chakraborty S, Ghosh SK, Das SK (2021)\u00a0CoMCLOUD: Virtual machine coalition for multi-tier applications over multi-cloud environments. IEEE Trans Cloud Comput: 1. https:\/\/doi.org\/10.1109\/TCC.2021.3122445","DOI":"10.1109\/TCC.2021.3122445"},{"issue":"3","key":"372_CR12","doi-asserted-by":"publisher","first-page":"683","DOI":"10.1109\/TPDS.2021.3100298","volume":"33","author":"X Chen","year":"2022","unstructured":"Chen X, Zhang J, Lin B, Chen Z, Wolter K, Min G (2022) Energy-efficient offloading for dnn-based smart iot systems in cloud-edge environments. IEEE Trans Parallel Distrib Syst 33(3):683\u2013697. https:\/\/doi.org\/10.1109\/TPDS.2021.3100298","journal-title":"IEEE Trans Parallel Distrib Syst"},{"key":"372_CR13","doi-asserted-by":"publisher","unstructured":"Chen Y, Zhao J, Wu Y (2022) QoE-aware decentralized task offloading and resource allocation for end-edge-cloud systems: A game-theoretical approach. IEEE Trans Mob Comput. https:\/\/doi.org\/10.1109\/TMC.2022.3223119","DOI":"10.1109\/TMC.2022.3223119"},{"key":"372_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.buildenv.2022.109218","author":"J Xu","year":"2022","unstructured":"Xu J, Li D, Gu W et al (2022) Uav-assisted task offloading for iot in smart buildings and environment via deep reinforcement learning. Build Environ. https:\/\/doi.org\/10.1016\/j.buildenv.2022.109218","journal-title":"Build Environ"},{"issue":"3","key":"372_CR15","doi-asserted-by":"publisher","first-page":"940","DOI":"10.1109\/TMC.2020.3017079","volume":"21","author":"S Tuli","year":"2020","unstructured":"Tuli S, Ilager S, Ramamohanarao K, Buyya R (2020) Dynamic scheduling for stochastic edge-cloud computing environments using a3c learning and residual recurrent neural networks. IEEE Trans Mob Comput 21(3):940\u2013954. https:\/\/doi.org\/10.1109\/TMC.2020.3017079","journal-title":"IEEE Trans Mob Comput"},{"issue":"4","key":"372_CR16","doi-asserted-by":"publisher","first-page":"2565","DOI":"10.1109\/TWC.2020.3043038","volume":"20","author":"Y Zhang","year":"2020","unstructured":"Zhang Y, Di B, Zheng Z, Lin J, Song L (2020) Distributed multi-cloud multi-access edge computing by multi-agent reinforcement learning. IEEE Trans Wirel Commun 20(4):2565\u20132578","journal-title":"IEEE Trans Wirel Commun"},{"issue":"8","key":"372_CR17","doi-asserted-by":"publisher","first-page":"1911","DOI":"10.1109\/TPDS.2021.3132422","volume":"33","author":"Z Chen","year":"2022","unstructured":"Chen Z, Hu J, Min G, Luo C, El-Ghazawi T (2022) Adaptive and efficient resource allocation in cloud datacenters using actor-critic deep reinforcement learning. IEEE Trans Parallel Distrib Syst 33(8):1911\u20131923. https:\/\/doi.org\/10.1109\/TPDS.2021.3132422","journal-title":"IEEE Trans Parallel Distrib Syst"},{"key":"372_CR18","doi-asserted-by":"crossref","unstructured":"Zhou P, Wu G, Alzahrani B, Barnawi A, Alhindi A, Chen M (2021) Reinforcement learning for task placement in collaborative cloud-edge computing. In: 2021 IEEE Global Communications Conference (GLOBECOM). IEEE,\u00a0Madrid, pp 1\u20136","DOI":"10.1109\/GLOBECOM46510.2021.9685049"},{"issue":"3","key":"372_CR19","doi-asserted-by":"publisher","first-page":"3448","DOI":"10.1109\/TNSM.2021.3087258","volume":"18","author":"G Qu","year":"2021","unstructured":"Qu G, Wu H, Li R, Jiao P (2021) Dmro: A deep meta reinforcement learning-based task offloading framework for edge-cloud computing. IEEE Trans Netw Serv Manag 18(3):3448\u20133459","journal-title":"IEEE Trans Netw Serv Manag"},{"issue":"16","key":"372_CR20","doi-asserted-by":"publisher","first-page":"12610","DOI":"10.1109\/JIOT.2020.3014970","volume":"8","author":"L Chen","year":"2020","unstructured":"Chen L, Xu Y, Lu Z, Wu J, Gai K, Hung PC, Qiu M (2020) Iot microservice deployment in edge-cloud hybrid environment using reinforcement learning. IEEE Internet Things J 8(16):12610\u201312622","journal-title":"IEEE Internet Things J"},{"issue":"18","key":"372_CR21","doi-asserted-by":"publisher","first-page":"16742","DOI":"10.1109\/JIOT.2022.3164441","volume":"9","author":"Y Chen","year":"2022","unstructured":"Chen Y, Sun Y, Wang C, Taleb T (2022) Dynamic task allocation and service migration in edge-cloud iot system based on deep reinforcement learning. IEEE Internet Things J 9(18):16742\u201316757. https:\/\/doi.org\/10.1109\/JIOT.2022.3164441","journal-title":"IEEE Internet Things J"},{"issue":"10","key":"372_CR22","doi-asserted-by":"publisher","first-page":"9303","DOI":"10.1109\/JIOT.2020.3000527","volume":"7","author":"J Zhang","year":"2020","unstructured":"Zhang J, Du J, Shen Y, Wang J (2020) Dynamic computation offloading with energy harvesting devices: A hybrid-decision-based deep reinforcement learning approach. IEEE Internet Things J 7(10):9303\u20139317","journal-title":"IEEE Internet Things J"},{"key":"372_CR23","doi-asserted-by":"publisher","unstructured":"Oroojlooyjadid A, Hajinezhad D (2019) A review of cooperative multi-agent deep reinforcement learning. https:\/\/doi.org\/10.48550\/arXiv.1908.03963","DOI":"10.48550\/arXiv.1908.03963"},{"issue":"10","key":"372_CR24","doi-asserted-by":"publisher","first-page":"4738","DOI":"10.1109\/TVT.2014.2372852","volume":"64","author":"O Mu\u00f1oz","year":"2015","unstructured":"Mu\u00f1oz O, Pascual-Iserte A, Vidal J (2015) Optimization of radio and computational resources for energy efficiency in latency-constrained application offloading. IEEE Trans Veh Technol 64(10):4738\u20134755. https:\/\/doi.org\/10.1109\/TVT.2014.2372852","journal-title":"IEEE Trans Veh Technol"},{"issue":"5","key":"372_CR25","doi-asserted-by":"publisher","first-page":"4584","DOI":"10.1109\/TVT.2021.3133586","volume":"71","author":"Y Chen","year":"2022","unstructured":"Chen Y, Zhao F, Chen X, Wu Y (2022) Efficient multi-vehicle task offloading for mobile edge computing in 6g networks. IEEE Trans Veh Technol 71(5):4584\u20134595. https:\/\/doi.org\/10.1109\/TVT.2021.3133586","journal-title":"IEEE Trans Veh Technol"},{"issue":"9","key":"372_CR26","doi-asserted-by":"publisher","first-page":"2968","DOI":"10.1109\/LCOMM.2021.3094842","volume":"25","author":"J Chen","year":"2021","unstructured":"Chen J, Wu Z (2021) Dynamic computation offloading with energy harvesting devices: A graph-based deep reinforcement learning approach. IEEE Commun Lett 25(9):2968\u20132972. https:\/\/doi.org\/10.1109\/LCOMM.2021.3094842","journal-title":"IEEE Commun Lett"},{"issue":"24","key":"372_CR27","doi-asserted-by":"publisher","first-page":"17508","DOI":"10.1109\/JIOT.2021.3081694","volume":"8","author":"J Chen","year":"2021","unstructured":"Chen J, Xing H, Xiao Z, Xu L, Tao T (2021) A drl agent for jointly optimizing computation offloading and resource allocation in mec. IEEE Internet Things J 8(24):17508\u201317524. https:\/\/doi.org\/10.1109\/JIOT.2021.3081694","journal-title":"IEEE Internet Things J"},{"key":"372_CR28","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1016\/j.jpdc.2022.03.010","volume":"165","author":"C Chen","year":"2022","unstructured":"Chen C, Jiang J, Zhou Y, Lv N, Liang X, Wan S (2022) An edge intelligence empowered flooding process prediction using internet of things in smart city. J Parallel Distrib Comput 165:66\u201378","journal-title":"J Parallel Distrib Comput"},{"key":"372_CR29","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1016\/j.future.2022.09.007","volume":"139","author":"J Huang","year":"2023","unstructured":"Huang J, Gao H, Wan S et al (2023) Aoi-aware energy control and computation offloading for industrial iot. Futur Gener Comput Syst 139:29\u201337","journal-title":"Futur Gener Comput Syst"},{"key":"372_CR30","doi-asserted-by":"crossref","unstructured":"Chen C, Li H, Li H, Fu R, Liu Y, Wan S (2022) Efficiency and fairness oriented dynamic task offloading in internet of vehicles. IEEE Trans Green Commun Netw","DOI":"10.1109\/TGCN.2022.3167643"},{"key":"372_CR31","doi-asserted-by":"publisher","unstructured":"Lowe R, Wu Y, Tamar A, Harb J (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. https:\/\/doi.org\/10.48550\/arXiv.1706.02275","DOI":"10.48550\/arXiv.1706.02275"},{"key":"372_CR32","doi-asserted-by":"publisher","unstructured":"Fujimoto S, Hoof HV, Meger D (2018) Addressing function approximation error in actor-critic methods. https:\/\/doi.org\/10.48550\/arXiv.1802.09477","DOI":"10.48550\/arXiv.1802.09477"},{"key":"372_CR33","doi-asserted-by":"crossref","unstructured":"Chen Y, Gu W, Xu J, et\u00a0al (2022a) Dynamic task offloading for digital twin-empowered mobile edge computing via deep reinforcement learning. China Commun","DOI":"10.1002\/dac.5154"},{"issue":"1","key":"372_CR34","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1109\/TCCN.2021.3093436","volume":"8","author":"Z Chen","year":"2022","unstructured":"Chen Z, Zhang L, Pei Y, Jiang C, Yin L (2022) Noma-based multi-user mobile edge computation offloading via cooperative multi-agent deep reinforcement learning. IEEE Trans Cogn Commun Netw 8(1):350\u2013364. https:\/\/doi.org\/10.1109\/TCCN.2021.3093436","journal-title":"IEEE Trans Cogn Commun Netw"}],"container-title":["Journal of Cloud Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13677-022-00372-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13677-022-00372-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13677-022-00372-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T12:11:51Z","timestamp":1670501511000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofcloudcomputing.springeropen.com\/articles\/10.1186\/s13677-022-00372-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,8]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["372"],"URL":"https:\/\/doi.org\/10.1186\/s13677-022-00372-9","relation":{},"ISSN":["2192-113X"],"issn-type":[{"value":"2192-113X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,8]]},"assertion":[{"value":"28 July 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 November 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 December 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"90"}}