{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T15:08:54Z","timestamp":1783436934775,"version":"3.54.6"},"reference-count":58,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T00:00:00Z","timestamp":1695772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Innovation Research Group Project of NSFC","award":["61921003"],"award-info":[{"award-number":["61921003"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62222202: 62232004"],"award-info":[{"award-number":["62222202: 62232004"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2023,9,27]]},"abstract":"<jats:p>Energy disaggregation is a key enabling technology for residential power usage monitoring, which benefits various applications such as carbon emission monitoring and human activity recognition. However, existing methods are difficult to balance the accuracy and usage burden (device costs, data labeling and prior knowledge). As the high penetration of smart speakers offers a low-cost way for sound-assisted residential power usage monitoring, this work aims to combine a smart speaker and a smart meter in a house to liberate the system from a high usage burden. However, it is still challenging to extract and leverage the consistent\/complementary information (two types of relationships between acoustic and power features) from acoustic and power data without data labeling or prior knowledge. To this end, we design COMFORT, a cross-modality system for self-supervised power usage monitoring, including (i) a cross-modality learning component to automatically learn the consistent and complementary information, and (ii) a cross-modality inference component to utilize the consistent and complementary information. We implement and evaluate COMFORT with a self-collected dataset from six houses in 14 days, demonstrating that COMFORT finds the most appliances (98%), improves the appliance recognition performance in F-measure by at least 41.1%, and reduces the Mean Absolute Error (MAE) of energy disaggregation by at least 30.4% over other alternative solutions.<\/jats:p>","DOI":"10.1145\/3610905","type":"journal-article","created":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T15:45:03Z","timestamp":1695829503000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Combining Smart Speaker and Smart Meter to Infer Your Residential Power Usage by Self-supervised Cross-modal Learning"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5671-9637","authenticated-orcid":false,"given":"Guanzhou","family":"Zhu","sequence":"first","affiliation":[{"name":"State Key Laboratory of Network and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7337-9168","authenticated-orcid":false,"given":"Dong","family":"Zhao","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Network and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8450-8417","authenticated-orcid":false,"given":"Kuo","family":"Tian","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Network and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0262-3323","authenticated-orcid":false,"given":"Zhengyuan","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Network and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2310-860X","authenticated-orcid":false,"given":"Rui","family":"Yuan","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Network and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7199-5047","authenticated-orcid":false,"given":"Huadong","family":"Ma","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Network and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,9,27]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"https:\/\/www.watttime.org\/ Accessed","year":"2023","unstructured":"2022. WattTime. https:\/\/www.watttime.org\/ Accessed February 6, 2023."},{"key":"e_1_2_1_2_1","volume-title":"Bayesian online changepoint detection. arXiv preprint arXiv:0710.3742","author":"Adams Ryan Prescott","year":"2007","unstructured":"Ryan Prescott Adams and David JC MacKay. 2007. Bayesian online changepoint detection. arXiv preprint arXiv:0710.3742 (2007)."},{"key":"e_1_2_1_3_1","first-page":"9758","article-title":"Self-supervised learning by cross-modal audio-video clustering","volume":"33","author":"Alwassel Humam","year":"2020","unstructured":"Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. 2020. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems 33 (2020), 9758--9770.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11343"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.3301850"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.pmcj.2013.02.002"},{"key":"e_1_2_1_7_1","first-page":"1","article-title":"IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data","volume":"5","author":"Bhalla Sejal","year":"2021","unstructured":"Sejal Bhalla, Mayank Goel, and Rushil Khurana. 2021. IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data. Proceedings of the ACM IMWUT 5, 4 (2021), 1--20.","journal-title":"Proceedings of the ACM IMWUT"},{"key":"e_1_2_1_8_1","first-page":"1","article-title":"Leveraging sound and wrist motion to detect activities of daily living with commodity smartwatches","volume":"6","author":"Bhattacharya Sarnab","year":"2022","unstructured":"Sarnab Bhattacharya, Rebecca Adaimi, and Edison Thomaz. 2022. Leveraging sound and wrist motion to detect activities of daily living with commodity smartwatches. Proceedings of the ACM IMWUT 6, 2 (2022), 1--28.","journal-title":"Proceedings of the ACM IMWUT"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00791"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460120.3485365"},{"key":"e_1_2_1_11_1","first-page":"1","article-title":"Midas: Generating mmWave Radar Data from Videos for Training Pervasive and Privacy-preserving Human Sensing Tasks","volume":"7","author":"Deng Kaikai","year":"2023","unstructured":"Kaikai Deng, Dong Zhao, Qiaoyue Han, Zihan Zhang, Shuyue Wang, Anfu Zhou, and Huadong Ma. 2023. Midas: Generating mmWave Radar Data from Videos for Training Pervasive and Privacy-preserving Human Sensing Tasks. Proceedings of the ACM IMWUT 7, 1 (2023), 1--26.","journal-title":"Proceedings of the ACM IMWUT"},{"key":"e_1_2_1_12_1","unstructured":"Drohoo. 2022. Smart Meter ZHDB-002. http:\/\/www.drohoo.com\/znkgcz\/ Accessed April 6 2022."},{"key":"e_1_2_1_13_1","unstructured":"Drohoo. 2022. Smart Plugin ZHCZ-003. http:\/\/www.drohoo.com\/znkgcz\/ Accessed April 6 2022."},{"key":"e_1_2_1_14_1","volume-title":"Tri Kurniawan Wijaya, and Karl Aberer","author":"Eberle Julien","year":"2015","unstructured":"Julien Eberle, Tri Kurniawan Wijaya, and Karl Aberer. 2015. Online unsupervised state recognition in sensor data. In IEEE PerCom. 29--36."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2528282.2528300"},{"key":"e_1_2_1_16_1","volume-title":"Towards data-efficient modeling for wake word spotting","author":"Gao Yixin","unstructured":"Yixin Gao, Yuriy Mishchenko, Anish Shah, Spyros Matsoukas, and Shiv Vitaladevuni. 2020. Towards data-efficient modeling for wake word spotting. In IEEE ICASSP. 7479--7483."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/1946459.1946461"},{"key":"e_1_2_1_18_1","volume-title":"Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al.","author":"Hershey Shawn","year":"2017","unstructured":"Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In IEEE ICASSP. 131--135."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2632048.2636084"},{"key":"e_1_2_1_20_1","volume-title":"Dominique Fourer, and Laurence Mi\u00e8geville.","author":"Houidi Sarra","year":"2017","unstructured":"Sarra Houidi, Fran\u00e7ois Auger, Houda Ben Attia Sethom, Dominique Fourer, and Laurence Mi\u00e8geville. 2017. Relevant feature selection for home appliances recognition. In Electrimacs 2017."},{"key":"e_1_2_1_21_1","volume-title":"International Conference on Learning Representations.","author":"Hsu Chen-Yu","year":"2019","unstructured":"Chen-Yu Hsu, Abbas Zeitoun, Guang-He Lee, Dina Katabi, and Tommi Jaakkola. 2019. Self-supervised learning of appliance usage. In International Conference on Learning Representations."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00947"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1620545.1620582"},{"key":"e_1_2_1_24_1","unstructured":"J Zico Kolter and Tommi Jaakkola. 2012. Approximate inference in additive factorial hmms with application to energy disaggregation. In Artificial intelligence and statistics. PMLR 1472--1482."},{"key":"e_1_2_1_25_1","volume-title":"SIGKDD Workshop","volume":"25","author":"Zico Kolter J","year":"2011","unstructured":"J Zico Kolter and Matthew J Johnson. 2011. REDD: A public data set for energy disaggregation research. In SIGKDD Workshop, San Diego, CA, Vol. 25. Citeseer, 59--62."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2632048.2636069"},{"key":"e_1_2_1_27_1","volume-title":"Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710","author":"Li Hao","year":"2016","unstructured":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)."},{"key":"e_1_2_1_28_1","first-page":"1","article-title":"IndoTrack: Device-free indoor human tracking with commodity Wi-Fi","volume":"1","author":"Li Xiang","year":"2017","unstructured":"Xiang Li, Daqing Zhang, Qin Lv, Jie Xiong, Shengjie Li, Yue Zhang, and Hong Mei. 2017. IndoTrack: Device-free indoor human tracking with commodity Wi-Fi. Proceedings of the ACM IMWUT 1, 3 (2017), 1--22.","journal-title":"Proceedings of the ACM IMWUT"},{"key":"e_1_2_1_29_1","unstructured":"Zhenning Li Kyle Gluesenkamp Bo Shen Jeffrey Munk Helia Zandi Praveen Cheekatamarla and Steve Kowalski. 2022. Seamlessly Fuel Flexible Heat Pump with Optimal Model-based Control Strategies to Reduce Peak Demand Utility Cost and CO2 Emission. Technical Report. Oak Ridge National Lab (ORNL) Oak Ridge TN (United States)."},{"key":"e_1_2_1_30_1","first-page":"1","article-title":"Audio-based activities of daily living (adl) recognition with large-scale acoustic embeddings from online videos","volume":"3","author":"Liang Dawei","year":"2019","unstructured":"Dawei Liang and Edison Thomaz. 2019. Audio-based activities of daily living (adl) recognition with large-scale acoustic embeddings from online videos. Proceedings of the ACM IMWUT 3, 1 (2019), 1--18.","journal-title":"Proceedings of the ACM IMWUT"},{"key":"e_1_2_1_31_1","volume-title":"Annual Conference on Neural Information Processing Systems.","author":"Lin Ji","year":"2022","unstructured":"Ji Lin, Ligeng Zhu, Wei-ming Chen, Wei-chen Wang, Chuang Gan, and Song Han. 2022. On-Device Training Under 256KB Memory. In Annual Conference on Neural Information Processing Systems."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s42835-020-00465-y"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3495243.3560519"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/WoWMoM.2013.6583496"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the ACM UBICOMP. 271--288","author":"Patel Shwetak N","year":"2007","unstructured":"Shwetak N Patel, Thomas Robertson, Julie A Kientz, Matthew S Reynolds, and Gregory D Abowd. 2007. At the flick of a switch: detecting and classifying unique electrical events on the residential power line. In Proceedings of the ACM UBICOMP. 271--288."},{"key":"e_1_2_1_36_1","volume-title":"Md Abdullah Al Hafiz Khan, and Nirmalya Roy","author":"Pathak Nilavra","year":"2015","unstructured":"Nilavra Pathak, Md Abdullah Al Hafiz Khan, and Nirmalya Roy. 2015. Acoustic based appliance state identifications for fine-grained energy analytics. In IEEE PerCom. 63--70."},{"key":"e_1_2_1_37_1","volume-title":"Supero: A sensor system for unsupervised residential power usage monitoring","author":"Phillips Dennis E","year":"2013","unstructured":"Dennis E Phillips, Rui Tan, Mohammad-Mahdi Moazzami, Guoliang Xing, Jinzhu Chen, and David KY Yau. 2013. Supero: A sensor system for unsupervised residential power usage monitoring. In IEEE PerCom. 66--75."},{"key":"e_1_2_1_38_1","volume-title":"Clustering by fast search and find of density peaks. science 344, 6191","author":"Rodriguez Alex","year":"2014","unstructured":"Alex Rodriguez and Alessandro Laio. 2014. Clustering by fast search and find of density peaks. science 344, 6191 (2014), 1492--1496."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.pmcj.2016.01.003"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.3009358"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2602044.2602058"},{"key":"e_1_2_1_42_1","volume-title":"ReSpeaker 6-Mic Circular Array kit for Raspberry Pi. https:\/\/wiki.seeedstudio.com\/ReSpeaker_6-Mic_Circular_Array_kit_for_Raspberry_Pi\/ Accessed","year":"2022","unstructured":"Seeed. 2022. ReSpeaker 6-Mic Circular Array kit for Raspberry Pi. https:\/\/wiki.seeedstudio.com\/ReSpeaker_6-Mic_Circular_Array_kit_for_Raspberry_Pi\/ Accessed November 3, 2022."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v27i1.8485"},{"key":"e_1_2_1_44_1","volume-title":"Carbon Reduction Potential through Optimal Control of Grid Interactive Heat Pump Water Heaters across the United States. Available at SSRN 4371455","author":"Sheikh Imran","year":"2022","unstructured":"Imran Sheikh and Charles Barnhart. 2022. Carbon Reduction Potential through Optimal Control of Grid Interactive Heat Pump Water Heaters across the United States. Available at SSRN 4371455 (2022)."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33011150"},{"key":"e_1_2_1_46_1","first-page":"1","article-title":"Vibrosense: Recognizing home activities by deep learning subtle vibrations on an interior surface of a house from a single point using laser doppler vibrometry","volume":"4","author":"Sun Wei","year":"2020","unstructured":"Wei Sun, Tuochao Chen, Jiayi Zheng, Zhenyu Lei, Lucy Wang, Benjamin Steeper, Peng He, Matthew Dressa, Feng Tian, and Cheng Zhang. 2020. Vibrosense: Recognizing home activities by deep learning subtle vibrations on an interior surface of a house from a single point using laser doppler vibrometry. Proceedings of the ACM IMWUT 4, 3 (2020), 1--28.","journal-title":"Proceedings of the ACM IMWUT"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1878431.1878439"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58621-8_45"},{"key":"e_1_2_1_49_1","first-page":"1","article-title":"Zero-shot learning for imu-based activity recognition using video embeddings","volume":"5","author":"Tong Catherine","year":"2021","unstructured":"Catherine Tong, Jinchen Ge, and Nicholas D Lane. 2021. Zero-shot learning for imu-based activity recognition using video embeddings. Proceedings of the ACM IMWUT 5, 4 (2021), 1--23.","journal-title":"Proceedings of the ACM IMWUT"},{"key":"e_1_2_1_50_1","volume-title":"EnergySniffer: Home energy monitoring system using smart phones","author":"Uddin Mostafa","unstructured":"Mostafa Uddin and Tamer Nadeem. 2012. EnergySniffer: Home energy monitoring system using smart phones. In IEEE IWCMC. 159--164."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/PERCOMW.2018.8480314"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11901"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1963.10500845"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376875"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scs.2018.12.021"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11873"},{"key":"e_1_2_1_57_1","first-page":"1","article-title":"IF-ConvTransformer: A framework for human activity recognition using IMU fusion and ConvTransformer","volume":"6","author":"Zhang Ye","year":"2022","unstructured":"Ye Zhang, Longguang Wang, Huiling Chen, Aosheng Tian, Shilin Zhou, and Yulan Guo. 2022. IF-ConvTransformer: A framework for human activity recognition using IMU fusion and ConvTransformer. Proceedings of the ACM IMWUT 6, 2 (2022), 1--26.","journal-title":"Proceedings of the ACM IMWUT"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2019.12.396"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610905","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3610905","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T16:29:19Z","timestamp":1753720159000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610905"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,27]]},"references-count":58,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9,27]]}},"alternative-id":["10.1145\/3610905"],"URL":"https:\/\/doi.org\/10.1145\/3610905","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,27]]},"assertion":[{"value":"2023-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}