{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T17:04:35Z","timestamp":1785603875408,"version":"3.56.0"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,7,31]],"date-time":"2020-07-31T00:00:00Z","timestamp":1596153600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"CONIX Research Center"},{"DOI":"10.13039\/100000185","name":"DARPA","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Sen. Netw."],"published-print":{"date-parts":[[2020,11,30]]},"abstract":"<jats:p>Many modern smart building applications are supported by wireless sensors to sense physical parameters, given the flexibility they offer and the reduced cost of deployment. However, most wireless sensors are powered by batteries today, and large deployments are inhibited by the requirement of periodic battery replacement. Energy harvesting sensors provide an attractive alternative, but they need to provide adequate quality of service to applications given the uncertainty of energy availability. We propose ACES, which uses reinforcement learning to maximize sensing quality of energy harvesting sensors for periodic and event-driven indoor sensing with available energy. Our custom-built sensor platform uses a supercapacitor to store energy and Bluetooth Low Energy to relay sensors data. Using simulations and real deployments, we use the data collected to continually adapt the sensing of each node to changing environmental patterns and transfer learning to reduce the training time in real deployments. In our 60-node deployment lasting 2 weeks, nodes stop operations only 0.1% of the time, and collection of data is comparable with current battery-powered nodes. We show that ACES reduces the node duty-cycle period by an average of 33% compared to three prior reinforcement learning techniques while continuously learning environmental changes over time.<\/jats:p>","DOI":"10.1145\/3404191","type":"journal-article","created":{"date-parts":[[2020,7,31]],"date-time":"2020-07-31T19:02:16Z","timestamp":1596222136000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":46,"title":["ACES"],"prefix":"10.1145","volume":"16","author":[{"given":"Francesco","family":"Fraternali","sequence":"first","affiliation":[{"name":"University of California, San Diego"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bharathan","family":"Balaji","sequence":"additional","affiliation":[{"name":"Amazon, WA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuvraj","family":"Agarwal","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rajesh K.","family":"Gupta","sequence":"additional","affiliation":[{"name":"University of California, San Diego"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,7,31]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Retrieved","year":"2018"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1878431.1878433"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TGCN.2018.2801725"},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"S. F. Barrett and D. J. Pack. 2006. Microcontrollers Fundamentals for Engineers and Scientists. Synthesis Lectures on Digital Circuits and Systems. Morgan 8 Claypool.  S. F. Barrett and D. J. Pack. 2006. Microcontrollers Fundamentals for Engineers and Scientists. Synthesis Lectures on Digital Circuits and Systems. Morgan 8 Claypool.","DOI":"10.1007\/978-3-031-79737-8"},{"key":"e_1_2_1_5_1","unstructured":"Felix Berkenkamp Matteo Turchetta Angela Schoellig and Andreas Krause. 2017. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing Systems. 908--918.  Felix Berkenkamp Matteo Turchetta Angela Schoellig and Andreas Krause. 2017. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing Systems. 908--918."},{"key":"e_1_2_1_6_1","unstructured":"Steven Bohez Abbas Abdolmaleki Michael Neunert Jonas Buchli Nicolas Heess and Raia Hadsell. 2019. Value constrained model-free continuous control. arXiv:1902.04623.  Steven Bohez Abbas Abdolmaleki Michael Neunert Jonas Buchli Nicolas Heess and Raia Hadsell. 2019. Value constrained model-free continuous control. arXiv:1902.04623."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/2893711.2893793"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2674061.2674083"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675683.2675692"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2014.2306798"},{"key":"e_1_2_1_11_1","volume-title":"Retrieved","author":"Sanio Semiconductor","year":"2007"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 2013 IEEE Wireless Communications and Networking Conference (WCNC\u201913)","author":"Dai Haipeng","year":"2013"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517351.2517363"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/WF-IoT.2016.7845391"},{"key":"e_1_2_1_15_1","unstructured":"adestotech.com. 2020. Home Page. Retrieved from https:\/\/www.adestotech.com\/.  adestotech.com. 2020. Home Page. Retrieved from https:\/\/www.adestotech.com\/."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMTT.2017.2687866"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025528.2025537"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3090075"},{"key":"e_1_2_1_19_1","volume-title":"Retrieved","author":"Envirosystems Cypress","year":"2015"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3276774.3282823"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 5th ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Building (BuildSys\u201918)","author":"Fraternali Francesco"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3279755.3279760"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2012.2218595"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2013.01.010"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3131672.3131674"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3131672.3131673"},{"key":"e_1_2_1_27_1","volume-title":"IHS Markit Says. Retrieved","author":"Howell Jenalea","year":"2017"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1165573.1165616"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the International Conference on Industrial, Engineering, and Other Applications of Applied Intelligent Systems.","author":"Hsu Roy Chaoming","year":"2009"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TETC.2014.2316518"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 2009 International Conference on Computational Science and Engineering","volume":"2","author":"Hsu R. C."},{"key":"e_1_2_1_32_1","volume-title":"Retrieved","year":"2018"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2627369.2631644"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1329125.1329241"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1274858.1274870"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740070.2626319"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigDataCongress.2015.124"},{"key":"e_1_2_1_38_1","volume-title":"LIPIcs-Leibniz International Proceedings in Informatics","volume":"71","author":"Lucia Brandon","year":"2017"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133920"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3005745.3005750"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2426656.2426679"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.icte.2017.12.005"},{"key":"e_1_2_1_43_1","volume-title":"Retrieved","author":"DAQ.","year":"2019"},{"key":"e_1_2_1_44_1","volume-title":"et\u00a0al","author":"Mnih Volodymyr","year":"2015"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2009.158"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the IEEE International Conference on RFID (RFID\u201915)","author":"Naderiparizi Saman"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp\u201915)","author":"Naderiparizi Saman"},{"key":"e_1_2_1_48_1","volume-title":"Retrieved","year":"2018"},{"key":"e_1_2_1_49_1","volume-title":"Retrieved","author":"GE.","year":"2017"},{"key":"e_1_2_1_50_1","first-page":"32","article-title":"A comparison of high-level full-system power models","volume":"8","author":"Rivoire Suzanne","year":"2008","journal-title":"HotPower"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the International Conference on Machine Learning. 1889--1897","author":"Schulman John","year":"2015"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126495"},{"key":"e_1_2_1_53_1","volume-title":"Energy harvesting sensor nodes: Survey and implications","author":"Sudevalayam Sujesha","year":"2011"},{"key":"e_1_2_1_54_1","volume-title":"Barto","author":"Sutton Richard S.","year":"1998"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3090090"},{"key":"e_1_2_1_56_1","article-title":"Transfer learning for reinforcement learning domains: A survey","author":"Taylor Matthew E.","year":"2009","journal-title":"Journal of Machine Learning Research 10"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/AHS.2009.50"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSAC.2015.2391531"},{"key":"e_1_2_1_59_1","volume-title":"Retrieved","year":"2018"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the 2007 4th Annual IEEE Communications Society Conference on Sensor, Mesh, and Ad Hoc Communications and Networks. IEEE","author":"Vigorito Christopher M."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/SAHCN.2006.288433"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-94268-1_40"},{"key":"e_1_2_1_65_1","unstructured":"www.enocean.com. 2019. EnOcean Motion Detector. Retrieved from https:\/\/www.enocean.com\/en\/products\/enocean_modules_24ghz_ble\/easyfit-motion-detector-with-illumination-sensor-emdcb\/.  www.enocean.com. 2019. EnOcean Motion Detector. Retrieved from https:\/\/www.enocean.com\/en\/products\/enocean_modules_24ghz_ble\/easyfit-motion-detector-with-illumination-sensor-emdcb\/."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/1824766.1824774"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2011.08.007"}],"container-title":["ACM Transactions on Sensor Networks"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3404191","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3404191","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3404191","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:42Z","timestamp":1750195902000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3404191"}},"subtitle":["Automatic Configuration of Energy Harvesting Sensors with Reinforcement Learning"],"short-title":[],"issued":{"date-parts":[[2020,7,31]]},"references-count":66,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,11,30]]}},"alternative-id":["10.1145\/3404191"],"URL":"https:\/\/doi.org\/10.1145\/3404191","relation":{},"ISSN":["1550-4859","1550-4867"],"issn-type":[{"value":"1550-4859","type":"print"},{"value":"1550-4867","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,31]]},"assertion":[{"value":"2019-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-07-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}