{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T05:55:50Z","timestamp":1771048550950,"version":"3.50.1"},"reference-count":34,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,7,6]],"date-time":"2020-07-06T00:00:00Z","timestamp":1593993600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"US National Science Foundation","doi-asserted-by":"crossref","award":["CNS1514425 and CNS151304"],"award-info":[{"award-number":["CNS1514425 and CNS151304"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Cyber-Phys. Syst."],"published-print":{"date-parts":[[2020,10,31]]},"abstract":"<jats:p>Markov Decision Processes (MDPs) provide important capabilities for facilitating the dynamic adaptation and self-optimization of cyber physical systems at runtime. In recent years, this has primarily taken the form of Reinforcement Learning (RL) techniques that eliminate some MDP components for the purpose of reducing computational requirements. In this work, we show that recent advancements in Compact MDP Models (CMMs) provide sufficient cause to question this trend when designing wireless sensor network nodes. In this work, a novel CMM-based approach to designing self-aware wireless sensor nodes is presented and compared to Q-Learning, a popular RL technique. We show that a certain class of CPS nodes is not well served by RL methods and contrast RL versus CMM methods in this context. Through both simulation and a prototype implementation, we demonstrate that CMM methods can provide significantly better runtime adaptation performance relative to Q-Learning, with comparable resource requirements.<\/jats:p>","DOI":"10.1145\/3372153","type":"journal-article","created":{"date-parts":[[2020,7,6]],"date-time":"2020-07-06T21:14:20Z","timestamp":1594070060000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Runtime Adaptation in Wireless Sensor Nodes Using Structured Learning"],"prefix":"10.1145","volume":"4","author":[{"given":"Adrian","family":"Sapio","sequence":"first","affiliation":[{"name":"University of Maryland, College Park"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuvra S.","family":"Bhattacharyya","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park and Tampere University of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marilyn","family":"Wolf","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,7,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCS.1984.1104824"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/92.845896"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/43.766730"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence. 1104--1111","author":"Boutilier C.","unstructured":"C. Boutilier , R. Dearden , and M. Goldszmidt . 1995. Exploiting structure in policy construction . In Proceedings of the International Joint Conference on Artificial Intelligence. 1104--1111 . C. Boutilier, R. Dearden, and M. Goldszmidt. 1995. Exploiting structure in policy construction. In Proceedings of the International Joint Conference on Artificial Intelligence. 1104--1111."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2009.158"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the IEEE 12th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob\u201916)","author":"Dawaliby S.","unstructured":"S. Dawaliby , A. Bradai , and Y. Pousset . 2016. In depth performance evaluation of LTE-M for M2M communications . In Proceedings of the IEEE 12th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob\u201916) . 1--8. S. Dawaliby, A. Bradai, and Y. Pousset. 2016. In depth performance evaluation of LTE-M for M2M communications. In Proceedings of the IEEE 12th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob\u201916). 1--8."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI\u201995)","volume":"2","author":"Dean T.","unstructured":"T. Dean and S. Lin . 1995. Decomposition techniques for planning in stochastic domains . In Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI\u201995) , Vol. 2 . 1121--1127. T. Dean and S. Lin. 1995. Decomposition techniques for planning in stochastic domains. In Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI\u201995), Vol. 2. 1121--1127."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS\u201918)","author":"Debizet Y.","unstructured":"Y. Debizet , G. Lallement , F. Abouzeid , P. Roche , and J. Autran . 2018. Q-learning-based adaptive power management for iot system-on-chips with embedded power states . In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS\u201918) . 1--5. Y. Debizet, G. Lallement, F. Abouzeid, P. Roche, and J. Autran. 2018. Q-learning-based adaptive power management for iot system-on-chips with embedded power states. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS\u201918). 1--5."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872936"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201918)","author":"Esterle L.","unstructured":"L. Esterle and B. Rinner . 2018. An architecture for self -aware IoT applications . In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201918) . 6588--6592. L. Esterle and B. Rinner. 2018. An architecture for self -aware IoT applications. In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201918). 6588--6592."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3017994"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence (UAI\u201999)","author":"Hoey J.","unstructured":"J. Hoey , R. St-Aubin , A. Hu , and C. Boutilier . 1999. SPUDD: Stochastic planning using decision diagrams . In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence (UAI\u201999) . Morgan Kaufmann, San Francisco, CA, 279--288. J. Hoey, R. St-Aubin, A. Hu, and C. Boutilier. 1999. SPUDD: Stochastic planning using decision diagrams. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence (UAI\u201999). Morgan Kaufmann, San Francisco, CA, 279--288."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the International Conference on Hardware\/Software Codesign and System Synthesis. 1--10","author":"Hsieh C.-M.","unstructured":"C.-M. Hsieh , F. Samie , M. S. Srouji , M. Wang , Z. Wang , and J. Henkel . 2014. Hardware\/software co-design for a wireless sensor network platform . In Proceedings of the International Conference on Hardware\/Software Codesign and System Synthesis. 1--10 . C.-M. Hsieh, F. Samie, M. S. Srouji, M. Wang, Z. Wang, and J. Henkel. 2014. Hardware\/software co-design for a wireless sensor network platform. In Proceedings of the International Conference on Hardware\/Software Codesign and System Synthesis. 1--10."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TETC.2014.2316518"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/1248547.1248628"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10617-010-9054-2"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1274858.1274870"},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"P. R. Lewis M. Platzner B. Rinner J. Torresen and X. Yao. 2016. Self-aware Computing Systems: An Engineering Approach. Springer.  P. R. Lewis M. Platzner B. Rinner J. Torresen and X. Yao. 2016. Self-aware Computing Systems: An Engineering Approach. Springer.","DOI":"10.1007\/978-3-319-39675-0"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVLSI.2014.2342213"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Design, Automation Test in Europe Conference Exhibition (DATE\u201910)","author":"Liu W.","unstructured":"W. Liu , Y. Tan , and Q. Qiu . 2010. Enhanced q-learning algorithm for dynamic power management with performance constraint . In Proceedings of the Design, Automation Test in Europe Conference Exhibition (DATE\u201910) . 602--605. W. Liu, Y. Tan, and Q. Qiu. 2010. Enhanced q-learning algorithm for dynamic power management with performance constraint. In Proceedings of the Design, Automation Test in Europe Conference Exhibition (DATE\u201910). 602--605."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC\u201912). 1903","author":"Modayil J.","year":"1910","unstructured":"J. Modayil , A. White , P. M. Pilarski , and R. S. Sutton . 2012. Acquiring a broad range of empirical knowledge in real time by temporal-difference learning . In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC\u201912). 1903 -- 1910 . J. Modayil, A. White, P. M. Pilarski, and R. S. Sutton. 2012. Acquiring a broad range of empirical knowledge in real time by temporal-difference learning. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC\u201912). 1903--1910."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the Design, Automation and Test in Europe Conference and Exhibition. 1--6.","author":"Moser C.","unstructured":"C. Moser , L. Thiele , D. Brunelli , and L. Benini . 2007. Adaptive power management in energy harvesting systems . In Proceedings of the Design, Automation and Test in Europe Conference and Exhibition. 1--6. C. Moser, L. Thiele, D. Brunelli, and L. Benini. 2007. Adaptive power management in energy harvesting systems. In Proceedings of the Design, Automation and Test in Europe Conference and Exhibition. 1--6."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 14th Mexican International Conference on Artificial Intelligence (MICAI\u201915)","author":"Ruiz S.","year":"2015","unstructured":"S. Ruiz and B. Hernandez . 2015. A parallel solver for markov decision process in crowd simulations . In Proceedings of the 14th Mexican International Conference on Artificial Intelligence (MICAI\u201915) . 107--116. DOI:https:\/\/doi.org\/10.1109\/MICAI. 2015 .23 10.1109\/MICAI.2015.23 S. Ruiz and B. Hernandez. 2015. A parallel solver for markov decision process in crowd simulations. In Proceedings of the 14th Mexican International Conference on Artificial Intelligence (MICAI\u201915). 107--116. DOI:https:\/\/doi.org\/10.1109\/MICAI.2015.23"},{"key":"e_1_2_1_24_1","volume-title":"Artificial Intelligence: A Modern Approach","author":"Russell S.","year":"2009","unstructured":"S. Russell and P. Norvig . 2009 . Artificial Intelligence: A Modern Approach ( 3 rd ed.). Pearson . S. Russell and P. Norvig. 2009. Artificial Intelligence: A Modern Approach (3rd ed.). Pearson.","edition":"3"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the Conference on Design and Architectures for Signal and Image Processing (DASIP\u201918)","author":"Sapio A.","unstructured":"A. Sapio , S. Bhattacharyya , and M. Wolf . 2018. Efficient solving of markov decision processes on GPUs using parallelized sparse matrices . In Proceedings of the Conference on Design and Architectures for Signal and Image Processing (DASIP\u201918) . A. Sapio, S. Bhattacharyya, and M. Wolf. 2018. Efficient solving of markov decision processes on GPUs using parallelized sparse matrices. In Proceedings of the Conference on Design and Architectures for Signal and Image Processing (DASIP\u201918)."},{"key":"#cr-split#-e_1_2_1_26_1.1","doi-asserted-by":"crossref","unstructured":"A. Sapio L. Li J. Wu M. Wolf and S. S. Bhattacharyya. 2017. Reconfigurable digital channelizer design using factored markov decision processes. J. Sign. Process. Syst. (Dec. 2017). DOI:https:\/\/doi.org\/10.1007\/s11265-017-1319-6 10.1007\/s11265-017-1319-6","DOI":"10.1007\/s11265-017-1319-6"},{"key":"#cr-split#-e_1_2_1_26_1.2","doi-asserted-by":"crossref","unstructured":"A. Sapio L. Li J. Wu M. Wolf and S. S. Bhattacharyya. 2017. Reconfigurable digital channelizer design using factored markov decision processes. J. Sign. Process. Syst. (Dec. 2017). DOI:https:\/\/doi.org\/10.1007\/s11265-017-1319-6","DOI":"10.1007\/s11265-017-1319-6"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the IEEE Global Conference on Signal and Information Processing. 595--599","author":"Sapio A.","unstructured":"A. Sapio , M. Wolf , and S. S. Bhattacharyya . 2016. Compact modeling and management of reconfiguration in digital channelizer implementation . In Proceedings of the IEEE Global Conference on Signal and Information Processing. 595--599 . A. Sapio, M. Wolf, and S. S. Bhattacharyya. 2016. Compact modeling and management of reconfiguration in digital channelizer implementation. In Proceedings of the IEEE Global Conference on Signal and Information Processing. 595--599."},{"key":"e_1_2_1_28_1","unstructured":"O. Sigaud and O. Buffet (Eds.). 2010. Markov Decision Processes in Artificial Intelligence. Wiley.  O. Sigaud and O. Buffet (Eds.). 2010. Markov Decision Processes in Artificial Intelligence. Wiley."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/551283"},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"A. D. Tijsma M. M. Drugan and M. A. Wiering. 2016. Comparing exploration strategies for Q-Learning in random stochastic mazes. In Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI\u201916). 1--8.  A. D. Tijsma M. M. Drugan and M. A. Wiering. 2016. Comparing exploration strategies for Q-Learning in random stochastic mazes. In Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI\u201916). 1--8.","DOI":"10.1109\/SSCI.2016.7849366"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the IEEE International Conference on Communications. 221--225","author":"Wei Y.","unstructured":"Y. Wei , X. Wang , F. Guo , G. Hogan , and M. Collier . 2015. Energy saving local control policy for Green Reconfigurable Routers . In Proceedings of the IEEE International Conference on Communications. 221--225 . Y. Wei, X. Wang, F. Guo, G. Hogan, and M. Collier. 2015. Energy saving local control policy for Green Reconfigurable Routers. In Proceedings of the IEEE International Conference on Communications. 221--225."},{"key":"e_1_2_1_32_1","volume-title":"Efficient GPU algorithms for parallel decomposition of graphs into strongly connected and maximal end components. Formal Methods Syst. Des. 48, 3 (01","author":"Wijs Anton","year":"2016","unstructured":"Anton Wijs , Joost-Pieter Katoen , and Dragan Bo\u0161na\u010dki . 2016. Efficient GPU algorithms for parallel decomposition of graphs into strongly connected and maximal end components. Formal Methods Syst. Des. 48, 3 (01 Jun. 2016 ), 274--300. DOI:https:\/\/doi.org\/10.1007\/s10703-016-0246-7 10.1007\/s10703-016-0246-7 Anton Wijs, Joost-Pieter Katoen, and Dragan Bo\u0161na\u010dki. 2016. Efficient GPU algorithms for parallel decomposition of graphs into strongly connected and maximal end components. Formal Methods Syst. Des. 48, 3 (01 Jun. 2016), 274--300. DOI:https:\/\/doi.org\/10.1007\/s10703-016-0246-7"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the IEEE International Conference on Computer Design (ICCD\u201912)","author":"Yue S.","unstructured":"S. Yue , D. Zhu , Y. Wang , and M. Pedram . 2012. Reinforcement learning based dynamic power management with a hybrid power supply . In Proceedings of the IEEE International Conference on Computer Design (ICCD\u201912) . 81--86. S. Yue, D. Zhu, Y. Wang, and M. Pedram. 2012. Reinforcement learning based dynamic power management with a hybrid power supply. In Proceedings of the IEEE International Conference on Computer Design (ICCD\u201912). 81--86."}],"container-title":["ACM Transactions on Cyber-Physical Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3372153","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3372153","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:45:07Z","timestamp":1750203907000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3372153"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,6]]},"references-count":34,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,10,31]]}},"alternative-id":["10.1145\/3372153"],"URL":"https:\/\/doi.org\/10.1145\/3372153","relation":{},"ISSN":["2378-962X","2378-9638"],"issn-type":[{"value":"2378-962X","type":"print"},{"value":"2378-9638","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,6]]},"assertion":[{"value":"2018-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-07-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}