{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,5]],"date-time":"2025-04-05T05:53:09Z","timestamp":1743832389173},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,9,13]],"date-time":"2023-09-13T00:00:00Z","timestamp":1694563200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,13]],"date-time":"2023-09-13T00:00:00Z","timestamp":1694563200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Cloud Comp"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In resource constrained edge environment, multiple service providers can compete to rent the limited resources to cache their service instances on edge servers close to end users, thereby significantly reducing the service delay and improving quality of service (QoS). However, service providers renting the resources of different edge servers to deploy their service instances can incur different resource usage costs and service delay. To make full use of the limited resources of the edge servers to further reduce resource usage costs, multiple service providers on an edge server can form a coalition and share the limited resource of an edge server. In this paper, we investigate the service caching problem of multiple service providers in resource constrained edge environment, and propose an independent learners-based services caching scheme (ILSCS) which adopts a stateless Q-learning to learn an optimal service caching scheme. To verify the effectiveness of ILSCS scheme, we implement COALITION, RANDOM, MDU, and MCS four baseline algorithms, and compare the total collaboration cost and service latency of ILSCS scheme with these of these four baseline algorithms under different experimental parameter settings. The extensive experimental results show that the ILSCS scheme can achieve lower total collaboration cost and service latency.<\/jats:p>","DOI":"10.1186\/s13677-023-00506-7","type":"journal-article","created":{"date-parts":[[2023,9,13]],"date-time":"2023-09-13T08:02:29Z","timestamp":1694592149000},"update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Stateless Q-learning algorithm for service caching in resource constrained edge environment"],"prefix":"10.1186","volume":"12","author":[{"given":"Binbin","family":"Huang","sequence":"first","affiliation":[]},{"given":"Ziqi","family":"Ran","sequence":"additional","affiliation":[]},{"given":"Dongjin","family":"Yu","sequence":"additional","affiliation":[]},{"given":"Yuanyuan","family":"Xiang","sequence":"additional","affiliation":[]},{"given":"Xiaoying","family":"Shi","sequence":"additional","affiliation":[]},{"given":"Zhongjin","family":"Li","sequence":"additional","affiliation":[]},{"given":"Zhengqian","family":"Xu","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,9,13]]},"reference":[{"key":"506_CR1","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1016\/j.jpdc.2019.10.009","volume":"136","author":"T Wang","year":"2020","unstructured":"Wang T, Mei Y, Jia W, Zheng X, Wang G, Xie M (2020) Edge-based differential privacy computing for sensor\u2013cloud systems. J Parallel Distrib Comput 136:75\u201385. https:\/\/doi.org\/10.1016\/j.jpdc.2019.10.009","journal-title":"J Parallel Distrib Comput"},{"key":"506_CR2","doi-asserted-by":"crossref","unstructured":"J. Xu, L. Chen, and P. Zhou (2018) \u201cJoint service caching and task offloading for mobile edge computing in dense networks,\u201d arXiv","DOI":"10.1109\/INFOCOM.2018.8485977"},{"key":"506_CR3","doi-asserted-by":"publisher","unstructured":"S. Li, L. Da Xu, and S. Zhao (2018) \u201c5G Internet of Things: A survey A R T I C L E I N F O,\u201d J. Ind. Inf. Integr. 10:1\u20139. Available: https:\/\/doi.org\/10.1016\/j.jii.2018.01.005","DOI":"10.1016\/j.jii.2018.01.005"},{"key":"506_CR4","doi-asserted-by":"publisher","unstructured":"Zhang, Junna, Jiawei Chen, et al.\u00a0 (2023) \u201cDependent Task Offloading Mechanism for Cloud\u2013Edge-Device Collaboration.\u201d J Netw Comput Appl.\u00a0 216:103656, https:\/\/doi.org\/10.1016\/j.jnca.2023.103656","DOI":"10.1016\/j.jnca.2023.103656"},{"key":"506_CR5","doi-asserted-by":"publisher","unstructured":"Zhang, Junna, Xiaoyan Zhao, et al.\u00a0 (2022) \u201cA Composite Service Provisioning Mechanism in Edge Computing.\u201d Mobile Inf Syst. 2022:1\u201316, https:\/\/doi.org\/10.1155\/2022\/9031201","DOI":"10.1155\/2022\/9031201"},{"key":"506_CR6","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1016\/j.future.2020.07.016","volume":"113","author":"X Xia","year":"2020","unstructured":"Xia X, Chen F, He Q, Cui G, Lai P, Abdelrazek M, Grundy J, Jin H (2020) Graph-based data caching optimization for edge computing. Future Gener Syst 113:228\u2013239. https:\/\/doi.org\/10.1016\/j.future.2020.07.016","journal-title":"Future Gener Syst"},{"issue":"2","key":"506_CR7","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1109\/TPDS.2020.3016344","volume":"32","author":"X Xia","year":"2020","unstructured":"Xia X, Chen F, He Q, Grundy J, Abdelrazek M, Jin H (2020) Online collaborative data caching in edge computing. IEEE Trans Parallel Distrib Syst 32(2):281\u2013294. https:\/\/doi.org\/10.1109\/TPDS.2020.3016344","journal-title":"IEEE Trans Parallel Distrib Syst"},{"key":"506_CR8","doi-asserted-by":"publisher","unstructured":"B. Huang, X. Liu, Y. Xiang, D. Yu, S. Deng and S. Wang,\u00a0 (2022) \u201cReinforcement learning for cost-effective IoT service caching at the edge. \u201dJ Parallel Distributed Comput. 168\u00a0\u00a0https:\/\/doi.org\/10.1016\/j.jpdc.2022.06.008","DOI":"10.1016\/j.jpdc.2022.06.008"},{"issue":"5","key":"506_CR9","doi-asserted-by":"publisher","first-page":"2635","DOI":"10.1109\/TSC.2021.3062017","volume":"15","author":"X Xia","year":"2022","unstructured":"Xia X, Chen F, Grundy J, Abdelrazek M, Jin H, He Q (2022) Constrained app data caching over edge server graphs in edge computing environment. IEEE Trans Serv Comput 15(5):2635\u20132647. https:\/\/doi.org\/10.1109\/TSC.2021.3062017","journal-title":"IEEE Trans Serv Comput"},{"issue":"1","key":"506_CR10","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1002\/itl2.208","volume":"4","author":"B Du","year":"2021","unstructured":"Du B (2021) Mobile edge computation induced caching strategy for huge online education with college teachers and students. Internet Technol Lett 4(1):2\u20137. https:\/\/doi.org\/10.1002\/itl2.208","journal-title":"Internet Technol Lett"},{"key":"506_CR11","doi-asserted-by":"publisher","first-page":"141331","DOI":"10.1109\/ACCESS.2020.3012442","volume":"8","author":"M Rim","year":"2020","unstructured":"Rim M, Kang CG (2020) Content Prefetching of Mobile Caching Devices in Cooperative D2D Communication Systems. IEEE Access 8:141331\u2013141341. https:\/\/doi.org\/10.1109\/ACCESS.2020.3012442","journal-title":"IEEE Access"},{"key":"506_CR12","doi-asserted-by":"publisher","first-page":"120788","DOI":"10.1109\/ACCESS.2019.2936866","volume":"7","author":"K Qi","year":"2019","unstructured":"Qi K, Han S, Yang C (2019) Learning a Hybrid Proactive and Reactive Caching Policy in Wireless Edge under Dynamic Popularity. IEEE Access 7:120788\u2013120801. https:\/\/doi.org\/10.1109\/ACCESS.2019.2936866","journal-title":"IEEE Access"},{"key":"506_CR13","doi-asserted-by":"publisher","unstructured":"Wang W, Lan RN, Gu JX, et al (2017) Edge caching at base stations with device-to-device offloading. IEEE Access 5:6399\u20136410. https:\/\/doi.org\/10.1109\/ACCESS.2017.2679198","DOI":"10.1109\/ACCESS.2017.2679198"},{"key":"506_CR14","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1109\/ojcoms.2020.2978585","volume":"1","author":"G Ahani","year":"2020","unstructured":"Ahani G, Yuan D (2020) \u201cOptimal scheduling of content caching subject to deadline.\u201d arXiv. 1:293\u2013307. https:\/\/doi.org\/10.1109\/ojcoms.2020.2978585","journal-title":"arXiv."},{"key":"506_CR15","doi-asserted-by":"publisher","unstructured":"C. K. Kim, T. Kim, A. Cho, and S. K. Lee (2020) \u201cDelay-aware distributed caching scheme in edge network,\u201d Conex. 2020 - Proc. 16th Int. Conf. Emerg. Netw. Exp. Technol. 544\u2013545, https:\/\/doi.org\/10.1145\/3386367.3431664","DOI":"10.1145\/3386367.3431664"},{"key":"506_CR16","doi-asserted-by":"publisher","first-page":"142747","DOI":"10.1109\/ACCESS.2020.3009683","volume":"8","author":"H Gu","year":"2020","unstructured":"Gu H, Wang H (2020) A Distributed Caching Scheme Using Non-Cooperative Game for Mobile Edge Networks. IEEE Access 8:142747\u2013142757. https:\/\/doi.org\/10.1109\/ACCESS.2020.3009683","journal-title":"IEEE Access"},{"key":"506_CR17","doi-asserted-by":"publisher","unstructured":"M. Kim, H. Cho, Y. Cui, and J. Lee\u00a0 (2020) \u201cService Caching and Computation Resource Allocation for Large-Scale Edge Computing-Enabled Networks,\u201d 2020 IEEE Glob. Commun. Conf. GLOBECOM 2020 - Proc.,\u00a0\u00a0https:\/\/doi.org\/10.1109\/GLOBECOM42002.2020.9322297","DOI":"10.1109\/GLOBECOM42002.2020.9322297"},{"issue":"1","key":"506_CR18","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1002\/itl2.160","volume":"4","author":"Y Ren","year":"2021","unstructured":"Ren Y (2021) Game theory based cooperative caching strategy in information-centric networking. Internet Technol Lett 4(1):2\u20135. https:\/\/doi.org\/10.1002\/itl2.160","journal-title":"Internet Technol Lett"},{"key":"506_CR19","doi-asserted-by":"publisher","unstructured":"Song, Jiongjiong, et al.\u00a0 (2017) \u201cLearning Based Content Caching and Sharing for Wireless Networks.\u201d IEEE Transactions on Communications. 1\u20131, https:\/\/doi.org\/10.1109\/tcomm.2017.2713384","DOI":"10.1109\/tcomm.2017.2713384"},{"key":"506_CR20","doi-asserted-by":"publisher","first-page":"106","DOI":"10.3390\/network2010008","volume":"2","author":"S Lu","year":"2022","unstructured":"Lu S et al (2022) A Dynamic Service Placement Based on Deep Reinforcement Learning in Mobile Edge Computing. Network. 2:106\u2013122. https:\/\/doi.org\/10.3390\/network2010008","journal-title":"Network."},{"key":"506_CR21","doi-asserted-by":"publisher","first-page":"119306","DOI":"10.1016\/j.ins.2023.119306","volume":"644","author":"Y Chen","year":"2023","unstructured":"Chen Y et al (2023) A Distributed Game Theoretical Approach for Credibility-Guaranteed Multimedia Data Offloading in MEC. Inf Sci. 644:119306.\u00a0\u00a0https:\/\/doi.org\/10.1016\/j.ins.2023.119306","journal-title":"Inf Sci."},{"key":"506_CR22","doi-asserted-by":"publisher","DOI":"10.1109\/TNSM.2023.3258191","author":"J Liang","year":"2023","unstructured":"Liang J, Ma B, Feng Z, Huang J (2023) Reliability-aware Task Processing and Offloading for Data-intensive Applications in Edge Computing. IEEE Trans Netw Serv Manage. https:\/\/doi.org\/10.1109\/TNSM.2023.3258191","journal-title":"IEEE Trans Netw Serv Manage"},{"key":"506_CR23","doi-asserted-by":"publisher","unstructured":"Chen Y, Hu J, Zhao J, Min G. QoS-Aware Computation Offloading in LEO Satellite Edge Computing for IoT: A Game-Theoretical Approach[J]. Chinese Journal of Electronics. https:\/\/doi.org\/10.23919\/cje.2022.00.412","DOI":"10.23919\/cje.2022.00.412"},{"key":"506_CR24","doi-asserted-by":"publisher","unstructured":"Chen, Ying, Jie Zhao, Jintao Hu, et al.\u00a0 (2023) \u201cDistributed Task Offloading and Resource Purchasing in Noma-Enabled Mobile Edge Computing: Hierarchical Game Theoretical Approaches.\u201d ACM Transactions on Embedded Computing Systems.\u00a0https:\/\/doi.org\/10.1145\/3597023","DOI":"10.1145\/3597023"},{"key":"506_CR25","doi-asserted-by":"publisher","first-page":"101277","DOI":"10.1016\/j.swevo.2023.101277","volume":"78","author":"J Zhang","year":"2023","unstructured":"Zhang J, Chen D, Yang Q et al (2023) Proximity Ranking-Based Multimodal Differential Evolution. Swarm and Evolutionary Computation. 78:101277. https:\/\/doi.org\/10.1016\/j.swevo.2023.101277","journal-title":"Swarm and Evolutionary Computation."},{"key":"506_CR26","doi-asserted-by":"publisher","unstructured":"Z Xu, L Zhou, S Chi-Kin Chau, W Liang, Q Xia, P Zhou\u00a0 (2020) \u201cCollaborate or Separate? Distributed Service Caching in Mobile Edge Clouds.\u201d\u00a0 IEEE INFOCOM.\u00a0 2020-July, 3:2066\u20132075.\u00a0https:\/\/doi.org\/10.1109\/INFOCOM41043.2020.9155365","DOI":"10.1109\/INFOCOM41043.2020.9155365"},{"key":"506_CR27","doi-asserted-by":"publisher","first-page":"11261","DOI":"10.1109\/jiot.2021.3125957","volume":"9","author":"X Zhang","year":"2022","unstructured":"Zhang X et al (2022) Joint Edge Server Placement and Service Placement in Mobile-Edge Computing. IEEE Internet of Things Journal. 9:11261\u201311274. https:\/\/doi.org\/10.1109\/jiot.2021.3125957","journal-title":"IEEE Internet of Things Journal."},{"key":"506_CR28","doi-asserted-by":"publisher","unstructured":"Y. Chen, W. Gu, J. Xu, Y. Zhang and G. Min, \"Dynamic task offloading for digital twin-empowered mobile edge computing via deep reinforcement learning,\" in China Communications, https:\/\/doi.org\/10.23919\/JCC.ea.2022-0372.202302.","DOI":"10.23919\/JCC.ea.2022-0372.202302"},{"key":"506_CR29","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2023.3251561","author":"F Liu","year":"2023","unstructured":"Liu F, Huang J, Wang X (2023) Joint Task Offloading and Resource Allocation for Device-Edge-Cloud Collaboration with Subtask Dependencies. IEEE Transactions on Cloud Computing. https:\/\/doi.org\/10.1109\/TCC.2023.3251561","journal-title":"IEEE Transactions on Cloud Computing"},{"issue":"2","key":"506_CR30","doi-asserted-by":"publisher","first-page":"2500","DOI":"10.1109\/JSYST.2023.3249217","volume":"17","author":"J Huang","year":"2023","unstructured":"Huang J, Wan J, Lv B, Ye Q, Chen Y (2023) Joint Computation Offloading and Resource Allocation for Edge-Cloud Collaboration in Internet of Vehicles via Deep Reinforcement Learning. IEEE Syst J 17(2):2500\u20132511. https:\/\/doi.org\/10.1109\/JSYST.2023.3249217","journal-title":"IEEE Syst J"},{"key":"506_CR31","doi-asserted-by":"publisher","unstructured":"Y. Chen, J. Zhao, Y. Wu, J. Huang and X. S. Shen, (2022) \"QoE-Aware Decentralized Task Offloading and Resource Allocation for End-Edge-Cloud Systems: A Game-Theoretical Approach,\" in IEEE Transactions on Mobile Computing.\u00a0\u00a0https:\/\/doi.org\/10.1109\/TMC.2022.3223119","DOI":"10.1109\/TMC.2022.3223119"},{"issue":"3","key":"506_CR32","doi-asserted-by":"publisher","first-page":"1610","DOI":"10.1109\/TWC.2019.2894403","volume":"18","author":"W Jiang","year":"2019","unstructured":"Jiang W, Feng G, Qin S, Yum TSP, Cao G (2019) Multi-Agent Reinforcement Learning for Efficient Content Caching in Mobile D2D Networks. IEEE Trans Wirel Commun 18(3):1610\u20131622. https:\/\/doi.org\/10.1109\/TWC.2019.2894403","journal-title":"IEEE Trans Wirel Commun"},{"key":"506_CR33","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1007\/3-540-44826-8_2","volume":"2636","author":"S Kapetanakis","year":"2003","unstructured":"Kapetanakis S, Kudenko D, Strens MJA (2003) Reinforcement learning approaches to coordination in cooperative multi-agent systems. Lect. Notes Artif. Intell. (Subseries Lect. Notes Comput. Sci. 2636:18\u201332. https:\/\/doi.org\/10.1007\/3-540-44826-8_2","journal-title":"Lect. Notes Artif. Intell. (Subseries Lect. Notes Comput. Sci."},{"issue":"5","key":"506_CR34","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1134\/S1064230710050102","volume":"49","author":"AA Korbut","year":"2010","unstructured":"Korbut AA, Sigal IK (2010) Exact and greedy solutions of the knapsack problem: The ratio of values of objective functions. J Comput Syst Sci Int 49(5):757\u2013764. https:\/\/doi.org\/10.1134\/S1064230710050102","journal-title":"J Comput Syst Sci Int"}],"container-title":["Journal of Cloud Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13677-023-00506-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13677-023-00506-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13677-023-00506-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,19]],"date-time":"2023-11-19T06:09:16Z","timestamp":1700374156000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofcloudcomputing.springeropen.com\/articles\/10.1186\/s13677-023-00506-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,13]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["506"],"URL":"https:\/\/doi.org\/10.1186\/s13677-023-00506-7","relation":{},"ISSN":["2192-113X"],"issn-type":[{"value":"2192-113X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,13]]},"assertion":[{"value":"29 July 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 September 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"(applicable for both human and\/ or animal studies. Ethical committees, Internal Review Boards and guidelines followed must be named. When applicable, additional headings with statements on consent to participate and consent to publish are also required). This declaration is not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to particpate"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"132"}}