{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:56:27Z","timestamp":1760234187475,"version":"build-2065373602"},"reference-count":26,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T00:00:00Z","timestamp":1618790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["No. 2020R1I1A3061195"],"award-info":[{"award-number":["No. 2020R1I1A3061195"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>As the demand for video streaming has been rapidly increasing recently, new technologies for improving the efficiency of video streaming have attracted much attention. In this paper, we thus investigate how to improve the efficiency of video streaming by using clients\u2019 cache storage considering exclusive OR (XOR) coding-based video streaming where multiple different video contents can be simultaneously transmitted in one transmission as long as prerequisite conditions are satisfied, and the efficiency of video streaming can be thus significantly enhanced. We also propose a new cache update scheme using reinforcement learning. The proposed scheme uses a K-actor-critic (K-AC) network that can mitigate the disadvantage of actor-critic networks by yielding K candidate outputs and by selecting the final output with the highest value out of the K candidates. The K-AC exists in each client, and each client can train it by using only locally available information without any feedback or signaling so that the proposed cache update scheme is a completely decentralized scheme. The performance of the proposed cache update scheme was analyzed in terms of the average number of transmissions for XOR coding-based video streaming and was compared to that of conventional cache update schemes. Our numerical results show that the proposed cache update scheme can reduce the number of transmissions up to 24% when the number of videos is 100, the number of clients is 50, and the cache size is 5.<\/jats:p>","DOI":"10.3390\/s21082867","type":"journal-article","created":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T21:59:49Z","timestamp":1618869589000},"page":"2867","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A New Cache Update Scheme Using Reinforcement Learning for Coded Video Streaming Systems"],"prefix":"10.3390","volume":"21","author":[{"given":"Yu-Sin","family":"Kim","sequence":"first","affiliation":[{"name":"Algorithm Team, Carvi, Seoul 08513, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeong-Min","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering, Gyeongsang National University, Gyeongnam 53064, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jong-Yeol","family":"Ryu","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering, Gyeongsang National University, Gyeongnam 53064, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4832-5827","authenticated-orcid":false,"given":"Tae-Won","family":"Ban","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering, Gyeongsang National University, Gyeongnam 53064, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,19]]},"reference":[{"key":"ref_1","unstructured":"Cisco (2019). Cisco visual networking index: Global mobile data traffic forecast update, 2017\u20132022. Update, 2017, 2022."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gill, P., Arlitt, M., Li, Z., and Mahanti, A. (2007, January 24\u201326). YouTube Traffic Characterization: A View From the Edge. Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement 20067, San Diego, CA, USA.","DOI":"10.1145\/1298306.1298310"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/MCOM.2018.1700736","article-title":"NR Wide Bandwidth Operations","volume":"56","author":"Jeon","year":"2018","journal-title":"IEEE Commun. Mag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1109\/TGCN.2017.2770215","article-title":"On the Total Energy Efficiency of Cell-Free Massive MIMO","volume":"2","author":"Ngo","year":"2018","journal-title":"IEEE Trans. Green Commun. Netw."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1109\/MCOM.2017.1600788","article-title":"Energy Efficiency Challenges of 5G Small Cell Networks","volume":"55","author":"Ge","year":"2017","journal-title":"IEEE Commun. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2995","DOI":"10.1109\/TWC.2016.2514418","article-title":"Exploiting Caching and Multicast for 5G Wireless Networks","volume":"15","author":"Poularakis","year":"2016","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kanrar, S., and Mandal, N.K. (2017, January 15\u201316). Traffic analysis and control at proxy server. Proceedings of the 2017 International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India.","DOI":"10.1109\/ICCONS.2017.8250702"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"6996","DOI":"10.1109\/TIT.2018.2835507","article-title":"Noisy broadcast networks with receiver caching","volume":"64","author":"Bidokhti","year":"2018","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"842","DOI":"10.1109\/TVT.2017.2739639","article-title":"Interference cancelation at receivers in cache-enabled wireless networks","volume":"67","author":"Yang","year":"2018","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Arbabjolfaei, F., Bandemer, B., Kim, Y., \u015ea\u015fo\u011flu, E., and Wang, L. (2013, January 7\u201312). On the capacity region for index coding. Proceedings of the 2013 IEEE International Symposium on Information Theory, Turkey, Sunday.","DOI":"10.1109\/ISIT.2013.6620369"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2825","DOI":"10.1109\/TIT.2006.874540","article-title":"Coding on demand by an informed source (iscod) for efficient broadcast of different supplemental data to caching clients","volume":"52","author":"Birk","year":"2006","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"6043","DOI":"10.1109\/TWC.2019.2942032","article-title":"User-Cache Aided Transmission With Index Coding in K -User Downlink Channels","volume":"18","author":"Son","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ban, T.-W., Lee, W., and Ryu, J. (2020). An efficient coded streaming using clients\u2019 cache. Sensors, 20.","DOI":"10.3390\/s20216220"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ascigil, O., Phan, T.K., Tasiopoulos, A.G., Sourlas, V., Psaras, I., and Pavlou, G. (2017, January 11\u201314). On uncoordinated service placement in edge-clouds. Proceedings of the 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), Hong Kong, China.","DOI":"10.1109\/CloudCom.2017.46"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Aimtongkham, P., So-In, C., and Sanguanpong, S. (2016, January 13\u201315). A novel web caching scheme using hybrid least frequently used and support vector machine. Proceedings of the 2016 13th International Joint Conference on Computer Science and Software Engineering (JCSSE), Khon Kaen, Thailand.","DOI":"10.1109\/JCSSE.2016.7748932"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Nogueira, J., Gonzalez, D., Guardalben, L., and Sargento, S. (2016, January 27\u201330). Over-The-Top Catch-up TV content-aware caching. Proceedings of the 2016 IEEE Symposium on Computers and Communication (ISCC), Messina, Italy.","DOI":"10.1109\/ISCC.2016.7543869"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1145\/170036.170081","article-title":"The LRU\u2013K Page Replacement Algorithm For Database Disk Buffering","volume":"22","author":"Weikum","year":"1993","journal-title":"ACM SIGMOD Rec."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Abrahamsson, H., and Bj\u00f6rkman, M. (2013, January 28\u201331). Caching for IPTV distribution with time-shift. Proceedings of the 2013 International Conference on Computing, Networking and Communications (ICNC), San Diego, CA, USA.","DOI":"10.1109\/ICCNC.2013.6504212"},{"key":"ref_19","unstructured":"Abdelkrim, E., Salahuddin, M.A., Elbiaze, H., and Glitho, R. (2016, January 4\u20138). A Hybrid Regression Model for Video Popularity-Based Cache Replacement in Content Delivery Networks. Proceedings of the 2016 IEEE Global Communications Conference (GLOBECOM), Washington, DC, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Maniotis, P., and Thomos, N. (2021). Viewport-Aware Deep Reinforcement Learning Approach for 360\u00b0 Video Caching. IEEE Trans. Multimed., to be published.","DOI":"10.1109\/TMM.2021.3052339"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zhong, C., Gursoy, M.C., and Velipasalar, S. (2018, January 21\u201323). A deep reinforcement learning-based framework for content caching. Proceedings of the 2018 52nd Annual Conference on Information Sciences and Systems (CISS), Princeton, NJ, USA.","DOI":"10.1109\/CISS.2018.8362276"},{"key":"ref_22","unstructured":"Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Breslau, L., Cao, F., Phillips, G., and Shenker, S. (1999, January 21\u201325). Web caching and Zipf-like distributions: Evidence and implications. Proceedings of the IEEE INFOCOM \u201999. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320), New York, NY, USA.","DOI":"10.1109\/INFCOM.1999.749260"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wang, Y., Li, Y., Song, Y., and Rong, X. (2020). The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition. Appl. Sci., 10.","DOI":"10.3390\/app10051897"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mei\u00dfner, P., Watschke, H., Winter, J., and Vietor, T. (2020). Artificial Neural Networks-Based Material Parameter Identification for Numerical Simulations of Additively Manufactured Parts by Material Extrusion. Polymers, 12.","DOI":"10.3390\/polym12122949"},{"key":"ref_26","unstructured":"Kingma, D.P., and Ba, J. (2014, January 12). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, New Orleans, LA, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/8\/2867\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:49:52Z","timestamp":1760161792000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/8\/2867"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,19]]},"references-count":26,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["s21082867"],"URL":"https:\/\/doi.org\/10.3390\/s21082867","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,4,19]]}}}