{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:48:09Z","timestamp":1760240889583,"version":"build-2065373602"},"reference-count":41,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2019,10,11]],"date-time":"2019-10-11T00:00:00Z","timestamp":1570752000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["NRF-2016R1E1A1A01943474, NRF-2018R1D1A1A02086102"],"award-info":[{"award-number":["NRF-2016R1E1A1A01943474, NRF-2018R1D1A1A02086102"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Hyperconnectivity via modern Internet of Things (IoT) technologies has recently driven us to envision \u201cdigital twin\u201d, in which physical attributes are all embedded, and their latest updates are synchronized on digital spaces in a timely fashion. From the point of view of cyberphysical system (CPS) architectures, the goals of digital twin include providing common programming abstraction on the same level of databases, thereby facilitating seamless integration of real-world physical objects and digital assets at several different system layers. However, the inherent limitations of sampling and observing physical attributes often pose issues related to data uncertainty in practice. In this paper, we propose a learning-based data management scheme where the implementation is layered between sensors attached to physical attributes and domain-specific applications, thereby mitigating the data uncertainty between them. To do so, we present a sensor data management framework, namely D2WIN, which adopts reinforcement learning (RL) techniques to manage the data quality for CPS applications and autonomous systems. To deal with the scale issue incurred by many physical attributes and sensor streams when adopting RL, we propose an action embedding strategy that exploits their distance-based similarity in the physical space coordination. We introduce two embedding methods, i.e., a user-defined function and a generative model, for different conditions. Through experiments, we demonstrate that the D2WIN framework with the action embedding outperforms several known heuristics in terms of achievable data quality under certain resource restrictions. We also test the framework with an autonomous driving simulator, clearly showing its benefit. For example, with only 30% of updates selectively applied by the learned policy, the driving agent maintains its performance about 96.2%, as compared to the ideal condition with full updates.<\/jats:p>","DOI":"10.3390\/s19204410","type":"journal-article","created":{"date-parts":[[2019,10,11]],"date-time":"2019-10-11T10:53:03Z","timestamp":1570791183000},"page":"4410","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Resource-Efficient Sensor Data Management for Autonomous Systems Using Deep Reinforcement Learning"],"prefix":"10.3390","volume":"19","author":[{"given":"Seunghwan","family":"Jeong","sequence":"first","affiliation":[{"name":"Department of Software, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gwangpyo","family":"Yoo","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Minjong","family":"Yoo","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ikjun","family":"Yeom","sequence":"additional","affiliation":[{"name":"Department of Software, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Honguk","family":"Woo","sequence":"additional","affiliation":[{"name":"Department of Software, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"23020","DOI":"10.3390\/s150923020","article-title":"Optimization and Control of Cyberphysical Vehicle Systems","volume":"15","author":"Bradley","year":"2015","journal-title":"Sensors"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1016\/j.procir.2016.11.152","article-title":"The Digital Twin: Realizing the Cyber-Physical Production System for Industry 4.0","volume":"61","author":"Uhlemann","year":"2017","journal-title":"Procedia CIRP"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Madni, A.M., Madni, C.C., and Lucero, S.D. (2019). Leveraging Digital Twin Technology in Model-Based Systems Engineering. Systems, 7.","DOI":"10.3390\/systems7010007"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.promfg.2017.04.043","article-title":"The Digital Twin: Demonstrating the Potential of Real Time Data Acquisition in Production Systems","volume":"9","author":"Uhlemann","year":"2017","journal-title":"Procedia Manuf."},{"key":"ref_5","unstructured":"Deshpande, A., Guestrin, C., Madden, S.R., Hellerstein, J.M., and Hong, W. (September, January 31). Model-driven Data Acquisition in Sensor Networks. Proceedings of the Thirtieth International Conference on Very Large Data Bases, Toronto, ON, Canada."},{"key":"ref_6","unstructured":"(2019, July 30). Donkey Simulator. Available online: https:\/\/docs.donkeycar.com\/guide\/simulator\/."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Deshpande, A., and Madden, S. (2006;, January 27\u201329). MauveDB: supporting model-based user views in database systems. Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, Chicago, IL, USA.","DOI":"10.1145\/1142473.1142483"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Morison, A.M., Murphy, T., and Woods, D.D. (2016, January 17\u201322). Seeing Through Multiple Sensors into Distant Scenes: The Essential Power of Viewpoint Control. Proceedings of the International Conference Human-Computer Interaction Platforms and Techniques, Toronto, ON, Canada.","DOI":"10.1007\/978-3-319-39516-6_37"},{"key":"ref_9","unstructured":"(2019, August 17). Autopilot. Available online: https:\/\/www.tesla.com\/autopilot?redirect=no."},{"key":"ref_10","unstructured":"(2019, August 18). Airsim Image APIs. Available online: https:\/\/microsoft.github.io\/AirSim\/docs\/image_apis\/."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1595","DOI":"10.1007\/s10586-015-0484-2","article-title":"Random task scheduling scheme based on reinforcement learning in cloud computing","volume":"18","author":"Peng","year":"2015","journal-title":"Cluster Comput."},{"key":"ref_12","unstructured":"Bao, Y., Peng, Y., and Wu, C. (May, January 29). Deep Learning-based Job Placement in Distributed Machine Learning Clusters. Proceedings of the IEEE Conference on Computer Communications, Paris, France."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Mao, H., Alizadeh, M., Menache, I., and Kandula, S. (2016, January 9\u201310). Resource management with deep reinforcement learning. Proceedings of the 15th ACM Workshop on Hot Topics in Networks, Atlanta, GA, USA.","DOI":"10.1145\/3005745.3005750"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mao, H., Schwarzkopf, M., Venkatakrishnan, S.B., Meng, Z., and Alizadeh, M. (2019, January 19\u201324). Learning Scheduling Algorithms for Data Processing Clusters. Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM), Beijing, China.","DOI":"10.1145\/3341302.3342080"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chinchali, S., Hu, P., Chu, T., Sharma, M., Bansal, M., Misra, R., Pavone, M., and Katti, S. (2018, January 2\u20137). Cellular Network Traffic Scheduling with Deep Reinforcement Learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11339"},{"key":"ref_16","unstructured":"Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J.M., and Stoica, I. (2018). Learning to Optimize Join Queries with Deep Reinforcement Learning. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1109\/TKDE.2004.61","article-title":"Managing deadline miss ratio and sensor data freshness in real-time databases","volume":"16","author":"Kang","year":"2004","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhou, Y., and Kang, K.D. (2010, January 6\u20139). Deadline assignment and tardiness control for real-time data services. Proceedings of the 2010 22nd Euromicro Conference on Real-Time Systems, Brussels, Belgium.","DOI":"10.1109\/ECRTS.2010.20"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_20","unstructured":"Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Bartle, R.G., and Bartle, R.G. (1995). The Elements of Integration and Lebesgue Measure, A Wiley-Interscience.","DOI":"10.1002\/9781118164471"},{"key":"ref_22","unstructured":"Ho, J., and Ermon, S. (2016). Generative adversarial imitation learning. Advances in Neural Information Processing Systems, Neural Information Processing Systems Foundation."},{"key":"ref_23","unstructured":"(2019, July 30). TensorFlow. Available online: https:\/\/www.tensorflow.org."},{"key":"ref_24","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv."},{"key":"ref_25","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv."},{"key":"ref_26","unstructured":"Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. (2016, January 19\u201324). Benchmarking deep reinforcement learning for continuous control. Proceedings of the 33rd International International Conference on Machine Learning (ICML), New York, NY, USA."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/BF01264051","article-title":"Real-time databases","volume":"1","author":"Ramamritham","year":"1993","journal-title":"Distrib. Parallel Databases"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3245","DOI":"10.1109\/TKDE.2015.2441725","article-title":"Deadline Assignment and Feedback Control for Differentiated Real-Time Data Services","volume":"27","author":"Zhou","year":"2015","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_29","unstructured":"Mnih, V., Heess, N., Graves, A., and Kavukcuoglu, K. (2014, January 8\u201313). Recurrent models of visual attention. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Elgabli, A., Khan, H., Krouka, M., and Bennis, M. (2018). Reinforcement Learning Based Scheduling Algorithm for Optimizing Age of Information in Ultra Reliable Low Latency Networks. arXiv.","DOI":"10.1109\/ISCC47284.2019.8969641"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.jnca.2019.04.010","article-title":"DA-DRLS: Drift adaptive deep reinforcement learning based scheduling for IoT resource management","volume":"138","author":"Chowdhury","year":"2019","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_32","unstructured":"Pazis, J., and Parr, R. (July, January 28). Generalized value functions for large action sets. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"70","DOI":"10.2352\/ISSN.2470-1173.2017.19.AVM-023","article-title":"Deep reinforcement learning framework for autonomous driving","volume":"2017","author":"Sallab","year":"2017","journal-title":"Electron. Imaging"},{"key":"ref_34","unstructured":"(2019, July 30). Using Keras and Deep Deterministic Policy Gradient to Play TORCS. Available online: https:\/\/yanpanlau.github.io\/2016\/10\/11\/Torcs-Keras.html."},{"key":"ref_35","unstructured":"Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 19\u201324). Asynchronous methods for deep reinforcement learning. Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1109\/MCI.2019.2901089","article-title":"Reinforcement Learning and Deep Learning Based Lateral Control for Autonomous Driving [Application Notes]","volume":"14","author":"Li","year":"2019","journal-title":"IEEE Comput. Intell. Mag."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Kaushik, M., Prasad, V., Krishna, K.M., and Ravindran, B. (2018, January 26\u201330). Overtaking maneuvers in simulated highway driving using deep reinforcement learning. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.","DOI":"10.1109\/IVS.2018.8500718"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Wu, K., Abolfazli Esfahani, M., Yuan, S., and Wang, H. (2018). Learn to steer through deep reinforcement learning. Sensors, 18.","DOI":"10.3390\/s18113650"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhou, X., Gao, Y., and Guan, L. (2019). Towards goal-directed navigation through combining learning based global and local planners. Sensors, 19.","DOI":"10.3390\/s19010176"},{"key":"ref_40","unstructured":"(2019, July 30). TORCS\u2014The Open Racing Car Simulator. Available online: https:\/\/sourceforge.net\/projects\/torcs\/."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Jaritz, M., de Charette, R., Toromanoff, M., Perot, E., and Nashashibi, F. (2018, January 21\u201326). End-to-end race driving with deep reinforcement learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8460934"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/20\/4410\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:25:24Z","timestamp":1760189124000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/20\/4410"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,11]]},"references-count":41,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2019,10]]}},"alternative-id":["s19204410"],"URL":"https:\/\/doi.org\/10.3390\/s19204410","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2019,10,11]]}}}