{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T15:59:47Z","timestamp":1767974387640,"version":"3.49.0"},"reference-count":53,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,5,15]],"date-time":"2023-05-15T00:00:00Z","timestamp":1684108800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DP200102299"],"award-info":[{"award-number":["DP200102299"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Making internet-of-things (IoT)-based applications context-aware demands large amounts of raw data to be collected, interpreted, stored, and reused or repurposed if needed from many domains and applications. Context is transient but interpreted data can be distinguished from IoT data in many aspects. Managing context in cache is a novel area of research that has been given very little attention. Performance metric-driven adaptive context caching (ACOCA) can have a profound impact on the performance and cost efficiency of context-management platforms (CMPs) when responding to context queries in realtime. Our paper proposes an ACOCA mechanism to maximize both the cost and performance efficiency of a CMP in near realtime. Our novel mechanism encompasses the entire context-management life cycle. This, in turn, distinctively addresses the problems of efficiently selecting context for caching and managing the additional costs of context management in the cache. We demonstrate that our mechanism results in long-term efficiencies for the CMP that have not been observed in any previous study. The mechanism employs a novel, scalable, and selective context-caching agent implemented using the twin delayed deep deterministic policy gradient method. It further incorporates an adaptive context-refresh switching policy, a time-aware eviction policy, and a latent caching decision management policy. We point out in our findings that the additional complexity of adaptation introduced to the CMP through ACOCA is significantly justified, considering the cost and performance gains achieved. Our algorithm is evaluated using a real-world inspired heterogeneous context-query load and a data set based on parking-related traffic in Melbourne, Australia. This paper presents and benchmarks the proposed scheme against traditional and context-aware caching policies. We demonstrate that ACOCA outperforms the benchmarks in both cost and performance efficiency, i.e., up to 68.6%, 84.7%, and 67% more cost efficient compared to traditional data caching policies to cache context, redirector mode, and context-aware adaptive data caching under real-world-like circumstances.<\/jats:p>","DOI":"10.3390\/s23104767","type":"journal-article","created":{"date-parts":[[2023,5,16]],"date-time":"2023-05-16T02:27:04Z","timestamp":1684204024000},"page":"4767","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Adaptive Context Caching for IoT-Based Applications: A Reinforcement Learning Approach"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9287-3420","authenticated-orcid":false,"given":"Shakthi","family":"Weerasinghe","sequence":"first","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3145, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1990-5734","authenticated-orcid":false,"given":"Arkady","family":"Zaslavsky","sequence":"additional","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3145, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5339-9305","authenticated-orcid":false,"given":"Seng Wai","family":"Loke","sequence":"additional","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3145, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alireza","family":"Hassani","sequence":"additional","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3145, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7562-6257","authenticated-orcid":false,"given":"Alexey","family":"Medvedev","sequence":"additional","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3145, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amin","family":"Abken","sequence":"additional","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3145, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1109\/SURV.2013.042313.00197","article-title":"Context Aware Computing for The Internet of Things: A Survey","volume":"16","author":"Perera","year":"2014","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1007\/3-540-48157-5_29","article-title":"Towards a Better Understanding of Context and Context-Awareness","volume":"Volume 1707","author":"Gellersen","year":"1999","journal-title":"Handheld and Ubiquitous Computing"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"3432","DOI":"10.1109\/TNSM.2021.3056891","article-title":"Caching Popular Transient IoT Contents in an SDN-Based Edge Infrastructure","volume":"18","author":"Ruggeri","year":"2021","journal-title":"IEEE Trans. Netw. Serv. Manag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3011","DOI":"10.1109\/TCOMM.2021.3059305","article-title":"Contextual Learning for Content Caching With Unknown Time-Varying Popularity Profiles via Incremental Clustering","volume":"69","author":"Liu","year":"2021","journal-title":"IEEE Trans. Commun."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e4261","DOI":"10.1002\/ett.4261","article-title":"Value-aware Cache Replacement in Edge Networks for Internet of Things","volume":"32","author":"Peng","year":"2021","journal-title":"Trans. Emerg. Telecommun. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Jagarlamudi, K.S., Zaslavsky, A., Loke, S.W., Hassani, A., and Medvedev, A. (2021, January 6\u20138). Quality and Cost Aware Service Selection in IoT-Context Management Platforms. Proceedings of the 2021 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics), Melbourne, Australia.","DOI":"10.1109\/iThings-GreenCom-CPSCom-SmartData-Cybermatics53846.2021.00028"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hassani, A., Medvedev, A., Haghighi, P.D., Ling, S., Indrawan-Santiago, M., Zaslavsky, A., and Jayaraman, P.P. (2018, January 19\u201323). Context-as-a-Service Platform: Exchange and Share Context in an IoT Ecosystem. Proceedings of the 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Athens, Greece.","DOI":"10.1109\/PERCOMW.2018.8480240"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Lehmann, O., Bauer, M., Becker, C., and Nicklas, D. (2004, January 14\u201317). From Home to World\u2014Supporting Context-Aware Applications through World Models. Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications, Orlando, FL, USA.","DOI":"10.1109\/PERCOM.2004.1276867"},{"key":"ref_9","unstructured":"(2022, March 14). FIWARE-Orion. Available online: https:\/\/github.com\/telefonicaid\/fiware-orion."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Weerasinghe, S., Zaslavsky, A., Loke, S.W., Medvedev, A., Abken, A., and Hassani, A. (2023). Context Caching for IoT-based Applications: Opportunities and Challenges. IEEE Internet Things J.","DOI":"10.1109\/IOTM.001.2200247"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Weerasinghe, S., Zaslavsky, A., Loke, S.W., Medvedev, A., and Abken, A. (2022, January 25\u201329). Estimating the Lifetime of Transient Context for Adaptive Caching in IoT Applications. Proceedings of the ACM Symposium on Applied Computing, Brno, Czech Republic.","DOI":"10.1145\/3477314.3507075"},{"key":"ref_12","unstructured":"Medvedev, A. (2020). Performance and Cost Driven Data Storage and Processing for IoT Context Management Platforms. [Doctoral Thesis, Monash University]."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sheng, S., Chen, P., Chen, Z., Wu, L., and Jiang, H. (2020, January 18). Edge Caching for IoT Transient Data Using Deep Reinforcement Learning. Proceedings of the IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society, Singapore.","DOI":"10.1109\/IECON43393.2020.9255111"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Lung, C.-H., Lambadaris, I., and St-Hilaire, M. (2018, January 20\u201324). IoT Data Lifetime-Based Cooperative Caching Scheme for ICN-IoT Networks. Proceedings of the 2018 IEEE International Conference on Communications (ICC), Kansas City, MO, USA.","DOI":"10.1109\/ICC.2018.8422100"},{"key":"ref_15","unstructured":"Weerasinghe, S., Zaslavsky, A., Loke, S.W., Hassani, A., Abken, A., and Medvedev, A. (2022). From Traditional Adaptive Data Caching to Adaptive Context Caching: A Survey. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Boytsov, A., and Zaslavsky, A. (2011, January 12\u201314). From Sensory Data to Situation Awareness: Enhanced Context Spaces Theory Approach. Proceedings of the 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing, Sydney, Australia.","DOI":"10.1109\/DASC.2011.55"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Sun, Y., Uysal-Biyikoglu, E., Yates, R., Koksal, C.E., and Shroff, N.B. (2016, January 10\u201315). Update or Wait: How to Keep Your Data Fresh. Proceedings of the IEEE INFOCOM 2016\u2014The 35th Annual IEEE International Conference on Computer Communications, San Francisco, CA, USA.","DOI":"10.1109\/INFOCOM.2016.7524524"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Schwefel, H.-P., Hansen, M.B., and Olsen, R.L. (2007, January 3\u20137). Adaptive Caching Strategies for Context Management Systems. Proceedings of the 2007 IEEE 18th International Symposium on Personal, Indoor and Mobile Radio Communications, Athens, Greece.","DOI":"10.1109\/PIMRC.2007.4394813"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zameel, A., Najmuldeen, M., and Gormus, S. (2019, January 28\u201330). Context-Aware Caching in Wireless IoT Networks. Proceedings of the 2019 11th International Conference on Electrical and Electronics Engineering (ELECO), Bursa, Turkey.","DOI":"10.23919\/ELECO47770.2019.8990647"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Li, Q., Shi, W., Xiao, Y., Ge, X., and Pandharipande, A. (2018, January 9\u201313). Content Size-Aware Edge Caching: A Size-Weighted Popularity-Based Approach. Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/GLOCOM.2018.8647794"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Weerasinghe, S., Zaslavsky, A., Loke, S.W., Abken, A., Hassani, A., and Medvedev, A. (2023, January 27\u201331). Adaptive Context Caching for Efficient Distributed Context Management Systems. Proceedings of the ACM Symposium on Applied Computing, Tallinn, Estonia.","DOI":"10.1145\/3555776.3577602"},{"key":"ref_22","unstructured":"Cidon, A., Eisenman, A., Alizadeh, M., and Katti, S. (2016, January 16\u201318). Cliffhanger: Scaling Performance Cliffs in Web Memory Caches. Proceedings of the NSDI\u201916: Proceedings of the 13th Usenix Conference on Networked Systems Design and Implementation, Santa Clara, CA, USA."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Arcaini, P., Riccobene, E., and Scandurra, P. (2015, January 18\u201319). Modeling and Analyzing MAPE-K Feedback Loops for Self-Adaptation. Proceedings of the 2015 IEEE\/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, Florence, Italy.","DOI":"10.1109\/SEAMS.2015.10"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1109\/COMST.2022.3205377","article-title":"A Survey on Evaluating the Quality of Autonomic Internet of Things Applications","volume":"25","author":"Fizza","year":"2022","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1109\/TPDS.2018.2868642","article-title":"On Cost-Driven Collaborative Data Caching: A New Model Approach","volume":"30","author":"Wang","year":"2019","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2074","DOI":"10.1109\/JIOT.2018.2882583","article-title":"Caching Transient Data for Internet of Things: A Deep Reinforcement Learning Approach","volume":"6","author":"Zhu","year":"2019","journal-title":"IEEE Internet Things J."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Khargharia, H.S., Jayaraman, P.P., Banerjee, A., Zaslavsky, A., Hassani, A., Abken, A., and Kumar, A. (2022, January 10\u201316). Probabilistic Analysis of Context Caching in Internet of Things Applications. Proceedings of the 2022 IEEE International Conference on Services Computing (SCC), Barcelona, Spain.","DOI":"10.1109\/SCC55611.2022.00025"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/2192-113X-1-7","article-title":"Context Caches in the Clouds","volume":"1","author":"Kiani","year":"2012","journal-title":"J. Cloud Comput. Adv. Syst. Appl."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Wang, Y., and Friderikos, V. (2020). A Survey of Deep Learning for Data Caching in Edge Network. Informatics, 7.","DOI":"10.3390\/informatics7040043"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"103005","DOI":"10.1016\/j.jnca.2021.103005","article-title":"Applying Machine Learning Techniques for Caching in Next-Generation Edge Networks: A Comprehensive Survey","volume":"181","author":"Shuja","year":"2021","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Guo, Y., Lama, P., Rao, J., and Zhou, X. (2013, January 20\u201324). V-Cache: Towards Flexible Resource Provisioning for Multi-Tier Applications in IaaS Clouds. Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing, Cambridge, MA, USA.","DOI":"10.1109\/IPDPS.2013.12"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2896380","article-title":"A Unified Approach to the Performance Analysis of Caching Systems","volume":"1","author":"Garetto","year":"2016","journal-title":"ACM Trans. Model. Perform. Eval. Comput. Syst."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1024","DOI":"10.1109\/TCCN.2019.2936193","article-title":"Deep Reinforcement Learning for Adaptive Caching in Hierarchical Content Delivery Networks","volume":"5","author":"Sadeghi","year":"2019","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Al-Turjman, F., Imran, M., and Vasilakos, A. (2017). Value-Based Caching in Information-Centric Wireless Body Area Networks. Sensors, 17.","DOI":"10.3390\/s17010181"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1331","DOI":"10.1109\/JSAC.2018.2844985","article-title":"A Reinforcement-Learning Approach to Proactive Caching in Wireless Networks","volume":"36","author":"Somuyiwa","year":"2018","journal-title":"IEEE J. Select. Areas Commun."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Nasehzadeh, A., and Wang, P. (2020, January 9). A Deep Reinforcement Learning-Based Caching Strategy for Internet of Things. Proceedings of the 2020 IEEE\/CIC International Conference on Communications in China (ICCC), Chongqing, China.","DOI":"10.1109\/ICCC49849.2020.9238811"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Al-Turjman, F.M., Al-Fagih, A.E., and Hassanein, H.S. (2013, January 21\u201324). A Value-Based Cache Replacement Approach for Information-Centric Networks. Proceedings of the 38th Annual IEEE Conference on Local Computer Networks\u2014Workshops, Sydney, Australia.","DOI":"10.1109\/LCNW.2013.6758526"},{"key":"ref_38","unstructured":"Weerasinghe, S., Zaslavsky, A., Loke, S.W., Abken, A., and Hassani, A. (2022). Reinforcement Learning Based Approaches to Adaptive Context Caching in Distributed Context Management Systems. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1007\/978-3-319-46301-8_10","article-title":"Storing and Indexing IoT Context for Smart City Applications","volume":"Volume 9870","author":"Galinina","year":"2016","journal-title":"Internet of Things, Smart Spaces, and Next Generation Networks and Systems"},{"key":"ref_40","unstructured":"(2022, December 27). FIWARE-Orion Components. Available online: https:\/\/www.fiware.org\/catalogue\/."},{"key":"ref_41","unstructured":"Jung, J., and Berger, A.W. (April, January 30). Hari Balakrishnan Modeling TTL-Based Internet Caches. Proceedings of the IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428), San Francisco, CA, USA."},{"key":"ref_42","unstructured":"Larson, R.C., and Odoni, A.R. (2007). Urban Operations Research, Dynamic Ideas."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1145\/3558053.3558057","article-title":"Estimating the Dynamic Lifetime of Transient Context in near Real-Time for Cost-Efficient Adaptive Caching","volume":"22","author":"Weerasinghe","year":"2022","journal-title":"SIGAPP Appl. Comput. Rev."},{"key":"ref_44","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2019). Continuous Control with Deep Reinforcement Learning. arXiv."},{"key":"ref_45","unstructured":"(2022, October 19). Appendix to Adaptive Context Caching for IoT-Based Applications. Available online: https:\/\/bit.ly\/3eEMJxc."},{"key":"ref_46","unstructured":"Fujimoto, S. (2018, January 10\u201315). Addressing Function Approximation Error in Actor-Critic Methods. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"5886","DOI":"10.1109\/TCOMM.2021.3086535","article-title":"Caching Transient Content for IoT Sensing: Multi-Agent Soft Actor-Critic","volume":"69","author":"Wu","year":"2021","journal-title":"IEEE Trans. Commun."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Hassani, A., Medvedev, A., Delir Haghighi, P., Ling, S., Zaslavsky, A., and Prakash Jayaraman, P. (2019). Context Definition and Query Language: Conceptual Specification, Implementation, and Evaluation. Sensors, 19.","DOI":"10.3390\/s19061478"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"2408","DOI":"10.1109\/TKDE.2018.2831214","article-title":"Similarity Metrics for SQL Query Clustering","volume":"30","author":"Kul","year":"2018","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Kul, G., Luong, D., Xie, T., Coonan, P., Chandola, V., Kennedy, O., and Upadhyaya, S. (2016, January 11\u201315). Ettu: Analyzing Query Intents in Corporate Databases. Proceedings of the 25th International Conference Companion on World Wide Web\u2014WWW \u201916 Companion, Montreal, QC, Canada.","DOI":"10.1145\/2872518.2888608"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Yang, J., McAuley, J., Leskovec, J., LePendu, P., and Shah, N. (2014, January 7\u201311). Finding Progression Stages in Time-Evolving Event Sequences. Proceedings of the 23rd international conference on World wide web\u2014WWW \u201914, Seoul, Korea.","DOI":"10.1145\/2566486.2568044"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Sheikh, R., and Kharbutli, M. (2010, January 3\u20136). Improving Cache Performance by Combining Cost-Sensitivity and Locality Principles in Cache Replacement Algorithms. Proceedings of the 2010 IEEE International Conference on Computer Design, Amsterdam, The Netherlands.","DOI":"10.1109\/ICCD.2010.5647594"},{"key":"ref_53","unstructured":"Weerasinghe, S., Zaslavsky, A., Hassani, A., Loke, S.W., Medvedev, A., and Abken, A. (2022). Context Query Simulation for Smart Carparking Scenarios in the Melbourne CDB. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/10\/4767\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:35:22Z","timestamp":1760124922000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/10\/4767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,15]]},"references-count":53,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["s23104767"],"URL":"https:\/\/doi.org\/10.3390\/s23104767","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,15]]}}}