{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T19:13:22Z","timestamp":1775675602478,"version":"3.50.1"},"reference-count":42,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2021,7,19]],"date-time":"2021-07-19T00:00:00Z","timestamp":1626652800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2020R1F1A1049314"],"award-info":[{"award-number":["2020R1F1A1049314"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Chung-Ang University Graduate Research Scholarship","award":["2020"],"award-info":[{"award-number":["2020"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper proposes a privacy-preserving energy management of a shared energy storage system (SESS) for multiple smart buildings using federated reinforcement learning (FRL). To preserve the privacy of energy scheduling of buildings connected to the SESS, we present a distributed deep reinforcement learning (DRL) framework using the FRL method, which consists of a global server (GS) and local building energy management systems (LBEMSs). In the framework, the LBEMS DRL agents share only a randomly selected part of their trained neural network for energy consumption models with the GS without consumer\u2019s energy consumption data. Using the shared models, the GS executes two processes: (i) construction and broadcast of a global model of energy consumption to the LBEMS agents for retraining their local models and (ii) training of the SESS DRL agent\u2019s energy charging and discharging from and to the utility and buildings. Simulation studies are conducted using one SESS and three smart buildings with solar photovoltaic systems. The results demonstrate that the proposed approach can schedule the charging and discharging of the SESS and an optimal energy consumption of heating, ventilation, and air conditioning systems in smart buildings under heterogeneous building environments while preserving the privacy of buildings\u2019 energy consumption.<\/jats:p>","DOI":"10.3390\/s21144898","type":"journal-article","created":{"date-parts":[[2021,7,19]],"date-time":"2021-07-19T04:55:40Z","timestamp":1626670540000},"page":"4898","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":34,"title":["Privacy-Preserving Energy Management of a Shared Energy Storage System for Smart Buildings: A Federated Deep Reinforcement Learning Approach"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0736-2325","authenticated-orcid":false,"given":"Sangyoon","family":"Lee","sequence":"first","affiliation":[{"name":"School of Electrical and Electronics Engineering, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 156-756, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9810-948X","authenticated-orcid":false,"given":"Le","family":"Xie","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9248-9522","authenticated-orcid":false,"given":"Dae-Hyun","family":"Choi","sequence":"additional","affiliation":[{"name":"School of Electrical and Electronics Engineering, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 156-756, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2021,7,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Bhusal, N., Abdelmalak, M., and Benidris, M. (2019, January 20\u201322). Optimum locations of utility-scale shared energy storage systems. Proceedings of the 2019 8th International Conference on Power Systems (ICPS), Jaipur, India.","DOI":"10.1109\/ICPS48983.2019.9067540"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1113","DOI":"10.1016\/j.energy.2018.12.185","article-title":"A multi-objective risk-based robust optimization approach to energy management in smart residential buildings under combined demand and supply uncertainty","volume":"170","author":"Golpira","year":"2019","journal-title":"Energy"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2714","DOI":"10.1109\/TIA.2018.2803728","article-title":"Energy management in electrical smart grid environment using robust optimization algorithm","volume":"54","author":"Melhem","year":"2018","journal-title":"IEEE Trans. Ind. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1109\/TII.2017.2728803","article-title":"A stochastic home energy management system considering satisfaction cost and response fatigue","volume":"14","author":"Siano","year":"2018","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1016\/j.jclepro.2018.05.103","article-title":"Stochastic multi-objective energy management in residential microgrids with combined cooling, heating, and power units considering battery energy storage systems and plug-in hybrid electric vehicles","volume":"195","author":"Sedighizadeh","year":"2018","journal-title":"J. Clean. Prod."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"38748","DOI":"10.1109\/ACCESS.2019.2906311","article-title":"Probabilistic algorithm for predictive control with full-complexity models in non-residential buildings","volume":"7","author":"Cambronero","year":"2019","journal-title":"IEEE Access."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1109\/TII.2020.2971530","article-title":"Predictive home energy management system with photovoltaic array, heat pump, and plug-in electric vehicle","volume":"17","author":"Yousefi","year":"2021","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1109\/JIOT.2017.2765359","article-title":"Distributed real-time HVAC control for cost-efficient commercial buildings under smart grid environment","volume":"5","author":"Yu","year":"2018","journal-title":"IEEE Internet Things J."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"5074","DOI":"10.1109\/TII.2018.2802454","article-title":"A heuristic-based smart HVAC energy management scheme for university buildings","volume":"14","author":"Jindal","year":"2018","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1646","DOI":"10.1109\/TSG.2017.2775209","article-title":"Online Energy Management for a Sustainable Smart Home With an HVAC Load and Random Occupancy","volume":"10","author":"Yu","year":"2017","journal-title":"IEEE Trans. Smart Grid."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1017","DOI":"10.1109\/TIA.2017.2781639","article-title":"Multi-objective optimization model of source-load-storage synergetic dispatch for a building energy management system based on TOU price demand response","volume":"54","author":"Wang","year":"2017","journal-title":"IEEE Trans. Ind. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"4466","DOI":"10.1109\/TSG.2020.2980318","article-title":"Demand-side management with shared energy storage system in smart grid","volume":"11","author":"Jo","year":"2020","journal-title":"IEEE Trans. Smart Grid."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"11688","DOI":"10.1109\/ACCESS.2017.2717923","article-title":"Towards cost minimization with renewable energy sharing in cooperative residential communities","volume":"5","author":"Ye","year":"2017","journal-title":"IEEE Access."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"185821","DOI":"10.1109\/ACCESS.2019.2961389","article-title":"Credit-based distributed real-time energy storage sharing management","volume":"7","author":"Zhu","year":"2019","journal-title":"IEEE Access."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3963","DOI":"10.1109\/TSG.2018.2844877","article-title":"Sharing solar PV and energy storage in apartment buildings: Resource allocation and pricing","volume":"10","author":"Fleischhacker","year":"2019","journal-title":"IEEE Trans. Smart Grid."},{"key":"ref_16","first-page":"82","article-title":"Privacy aware stochastic games for distributed end-user energy storage sharing","volume":"4","author":"Yao","year":"2018","journal-title":"IEEE Trans. Signal Inf. Process. Netw."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3698","DOI":"10.1109\/TSG.2018.2834219","article-title":"On-Line Building energy optimization using deep reinforcement learning","volume":"10","author":"Mocanu","year":"2019","journal-title":"IEEE Trans. Smart Grid."},{"key":"ref_18","unstructured":"Wei, T., Ren, S., and Zhu, Q. (2019). Deep reinforcement learning for joint datacenter and HVAC load control in distributed mixed-use buildings. IEEE Trans. Sustain. Comput."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Chen, B., Cai, Z., and Berges, M. (2019, January 13\u201314). Gnu-RL: A precocial reinforcement learning solution for building HVAC control using a differentiable MPC policy. Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, New York, NY, USA.","DOI":"10.1145\/3360322.3360849"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1109\/TSG.2020.3011739","article-title":"Multi-agent deep reinforcement learning for HVAC control in commercial buildings","volume":"12","author":"Yu","year":"2021","journal-title":"IEEE Trans. Smart Grid."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yu, L., Qin, S., Zhang, M., Shen, C., Jiang, T., and Guan, X. (2021). A review of deep reinforcement learning for smart building energy management. IEEE Internet Things J.","DOI":"10.1109\/JIOT.2021.3078462"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2751","DOI":"10.1109\/JIOT.2019.2957289","article-title":"Deep reinforcement learning for smart home energy management","volume":"7","author":"Yu","year":"2019","journal-title":"IEEE Internet Things J."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3146","DOI":"10.1109\/TSG.2020.2967430","article-title":"Deep reinforcement learning method for demand response management of interruptible load","volume":"11","author":"Wang","year":"2020","journal-title":"IEEE Trans. Smart Grid."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"82194","DOI":"10.1109\/ACCESS.2019.2924030","article-title":"Demand response management for industrial facilities: A deep reinforcement learning approach","volume":"7","author":"Huang","year":"2019","journal-title":"IEEE Access."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"5039","DOI":"10.1109\/TSG.2020.2996274","article-title":"Deep reinforcement learning-based controller for SOC management of multi-electrical energy storage system","volume":"11","author":"Gorostiza","year":"2020","journal-title":"IEEE Trans. Smart Grid."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1522","DOI":"10.1109\/TIFS.2020.3036247","article-title":"Energy management strategy for smart meter privacy and cost saving","volume":"16","author":"You","year":"2021","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_27","first-page":"42","article-title":"PPM: Privacy policy manager for home energy management system","volume":"9","author":"Rahman","year":"2018","journal-title":"J. Wirel. Mob. Netw. Ubiquitous Comput. Dependable Appl."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1109\/JIOT.2017.2771370","article-title":"Smart meter privacy: Exploiting the potential of household energy storage units","volume":"5","author":"Sun","year":"2018","journal-title":"IEEE Internet Things J."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Jia, R., Dong, R., Sastry, S.S., and Sapnos, C.J. (2017, January 18\u201320). Privacy-enhanced architecture for occupancy-based HVAC control. Proceedings of the 2017 ACM\/IEEE 8th international conference on cyber-physical systems (ICCPS), Pittsburgh, PA, USA.","DOI":"10.1145\/3055004.3055007"},{"key":"ref_30","unstructured":"McMahan, H.B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2021, May 03). Communcation-Efficient Learning of Deep Networks from Decentralized Data. Available online: https:\/\/arxiv.org\/pdf\/1602.05629."},{"key":"ref_31","unstructured":"Zhou, H.H., Feng, W., Lin, Y., Xu, Q., and Yang, Q. (2021, May 03). Federated Deep Reinforcement Learning. Available online: https:\/\/arxiv.org\/abs\/1901.08277."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"4555","DOI":"10.1109\/LRA.2019.2931179","article-title":"Lifelong federated reinforcement learning: A learning architecture for navigation in cloud robotic systems","volume":"4","author":"Liu","year":"2019","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Nadiger, C., Kumar, A., and Abdelhak, S. (2019, January 3\u20135). Federated reinforcement learning for fast personalization. Proceedings of the 2019 IEEE Second International Conference on Artificial Intelligenca and Knowledge Engineering(AIKE), Sardinia, Italy.","DOI":"10.1109\/AIKE.2019.00031"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1109\/JCN.2020.000015","article-title":"AFRL: Adaptive federated reinforcement learning for intelligent jamming defense in FANET","volume":"22","author":"Mowla","year":"2020","journal-title":"J. Commun. Netw."},{"key":"ref_35","unstructured":"Lee, S., and Choi, D.-H. (2020). Federated reinforcement learning for energy management of multiple smart homes with distributed energy resources. IEEE Trans. Ind. Inform."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Shokri, R., and Shmatikov, V. (October, January 29). Privacy-preserving deep learning. Proceedings of the 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA.","DOI":"10.1109\/ALLERTON.2015.7447103"},{"key":"ref_37","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21\u201326). Deterministic policy gradient algorithms. Proceedings of the 31st International Conference on Machine Learning (ICML), Beijing, China."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1145\/203330.203343","article-title":"Temporal difference learning and TD-gammon","volume":"38","author":"Tesau","year":"1995","journal-title":"Commun. ACM"},{"key":"ref_39","unstructured":"Kingma, D.P., and Ba, J. (2021, May 03). Adam: A Method for Stochastic Optimization. Available online: https:\/\/arxiv.org\/abs\/1412.6980."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"4939","DOI":"10.1109\/TSG.2018.2871171","article-title":"Transactive real-time electric vehicle charging management for commercial buildings with PV on-site generation","volume":"10","author":"Liu","year":"2019","journal-title":"IEEE Trans. Smart Grid."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/TIA.2018.2866155","article-title":"Aggregation of users in a residential\/commercial building managed by a building energy management system (BEMS)","volume":"55","author":"Martirano","year":"2019","journal-title":"IEEE Trans. Ind. Appl."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Ostadijafari, M., and Dubey, A. (2020). Tube-based model predictive controller for building\u2019s heating ventilation and air conditioning (HVAC) system. IEEE Syst. J.","DOI":"10.1109\/CCTA.2019.8920657"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/14\/4898\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:31:39Z","timestamp":1760164299000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/14\/4898"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,19]]},"references-count":42,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2021,7]]}},"alternative-id":["s21144898"],"URL":"https:\/\/doi.org\/10.3390\/s21144898","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,19]]}}}