{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T05:22:37Z","timestamp":1772601757217,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2021,12,7]],"date-time":"2021-12-07T00:00:00Z","timestamp":1638835200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Regional Development Fund (FEDER)","award":["NORTE 01-0145-FEDER-000062"],"award-info":[{"award-number":["NORTE 01-0145-FEDER-000062"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Energies"],"abstract":"<jats:p>Heating, ventilating, and air conditioning (HVAC) units account for a significant consumption share in buildings, namely office buildings. Therefore, this paper addresses the possibility of having an intelligent and more cost-effective solution for the management of HVAC units in office buildings. The method applied in this paper divides the addressed problem into three steps: (i) the continuous acquisition of data provided by an open-source building energy management systems, (ii) the proposed learning and predictive model able to predict if users will be working in a given location, and (iii) the proposed decision model to manage the HVAC units according to the prediction of users, current environmental context, and current energy prices. The results show that the proposed predictive model was able to achieve a 93.8% accuracy and that the proposed decision tree enabled the maintenance of users\u2019 comfort. The results demonstrate that the proposed solution is able to run in real-time in a real office building, making it a possible solution for smart buildings.<\/jats:p>","DOI":"10.3390\/en14248210","type":"journal-article","created":{"date-parts":[[2021,12,7]],"date-time":"2021-12-07T09:52:29Z","timestamp":1638870749000},"page":"8210","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Energy Management Model for HVAC Control Supported by Reinforcement Learning"],"prefix":"10.3390","volume":"14","author":[{"given":"Pedro","family":"Macieira","sequence":"first","affiliation":[{"name":"GECAD\u2014Research Group on Intelligent Engineering and Computing for Advanced Innovation and Development, Polytechnic of Porto (P.PORTO), P-4200-072 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8597-3383","authenticated-orcid":false,"given":"Luis","family":"Gomes","sequence":"additional","affiliation":[{"name":"GECAD\u2014Research Group on Intelligent Engineering and Computing for Advanced Innovation and Development, Polytechnic of Porto (P.PORTO), P-4200-072 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4560-9544","authenticated-orcid":false,"given":"Zita","family":"Vale","sequence":"additional","affiliation":[{"name":"GECAD\u2014Research Group on Intelligent Engineering and Computing for Advanced Innovation and Development, Polytechnic of Porto (P.PORTO), P-4200-072 Porto, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"102048","DOI":"10.1016\/j.scs.2020.102048","article-title":"Flexibility management model of home appliances to support DSO requests in smart grids","volume":"55","author":"Lezama","year":"2020","journal-title":"Sustain. Cities Soc."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1016\/j.ijepes.2018.08.019","article-title":"Stochastic interval-based optimal offering model for residential energy management systems by household owners","volume":"105","author":"Gazafroudi","year":"2019","journal-title":"Int. J. Electr. Power Energy Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2751","DOI":"10.1109\/JIOT.2019.2957289","article-title":"Deep Reinforcement Learning for Smart Home Energy Management","volume":"7","author":"Yu","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Liere-Netheler, I., Schuldt, F., von Maydell, K., and Agert, C. (2020). Simulation of Incidental Distributed Generation Curtailment to Maximize the Integration of Renewable Energy Generation in Power Systems. Energies, 13.","DOI":"10.3390\/en13164173"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1720","DOI":"10.1109\/TNNLS.2015.2461491","article-title":"Adaptive Portfolio Optimization for Multiple Electricity Markets Participation","volume":"27","author":"Pinto","year":"2016","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"64169","DOI":"10.1109\/ACCESS.2020.2985254","article-title":"Multi-Agent Microgrid Management System for Single-Board Computers: A Case Study on Peer-to-Peer Energy Trading","volume":"8","author":"Gomes","year":"2020","journal-title":"IEEE Access"},{"key":"ref_7","unstructured":"Yu, L., Qin, S., Zhang, M., Shen, C., Jiang, T., and Guan, X. (2020). Deep Reinforcement Learning for Smart Building Energy Management: A Survey. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Gomes, L., Ramos, C., Jozi, A., Serra, B., Paiva, L., and Vale, Z. (2019). IoH: A Platform for the Intelligence of Home with a Context Awareness and Ambient Intelligence Approach. Futur. Internet, 11.","DOI":"10.3390\/fi11030058"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3311950","article-title":"Machine learning for smart building applications: Review and taxonomy","volume":"52","author":"Djenouri","year":"2019","journal-title":"ACM Comput. Surv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1016\/j.enbuild.2007.03.007","article-title":"A review on buildings energy consumption information","volume":"40","author":"Ortiz","year":"2008","journal-title":"Energy Build."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"107810","DOI":"10.1016\/j.buildenv.2021.107810","article-title":"Occupancy-based HVAC control systems in buildings: A state-of-the-art review","volume":"197","author":"Haghighat","year":"2021","journal-title":"Build. Environ."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"102480","DOI":"10.1016\/j.scs.2020.102480","article-title":"Adaptive-predictive control strategy for HVAC systems in smart buildings\u2014A review","volume":"63","author":"Gholamzadehmir","year":"2020","journal-title":"Sustain. Cities Soc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.apenergy.2018.02.085","article-title":"An algorithm for optimal management of aggregated HVAC power demand using smart thermostats","volume":"217","author":"Adhikari","year":"2018","journal-title":"Appl. Energy"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"102275","DOI":"10.1016\/j.scs.2020.102275","article-title":"A survey on hyperparameters optimization algorithms of forecasting models in smart grid","volume":"61","author":"Khalid","year":"2020","journal-title":"Sustain. Cities Soc."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1471","DOI":"10.1016\/j.apenergy.2019.01.070","article-title":"Human-in-the-loop HVAC operations: A quantitative review on occupancy, comfort, and energy-efficiency dimensions","volume":"239","author":"Jung","year":"2019","journal-title":"Appl. Energy"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"118154","DOI":"10.1016\/j.jclepro.2019.118154","article-title":"Agent-based architecture for demand side management using real-time resources\u2019 priorities and a deterministic optimization algorithm","volume":"241","author":"Gomes","year":"2019","journal-title":"J. Clean. Prod."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Petrosanu, D.M., Carutasu, G., Carutasu, N.L., and P\u00eerjan, A. (2019). A review of the recent developments in integrating machine learning models with sensor devices in the smart buildings sector with a view to attaining enhanced sensing, energy efficiency, and optimal building management. Energies, 12.","DOI":"10.3390\/en12244745"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1016\/j.enbuild.2018.09.033","article-title":"Non-intrusive occupancy monitoring for energy conservation in commercial buildings","volume":"179","author":"Ardakanian","year":"2018","journal-title":"Energy Build."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/j.enbuild.2018.11.025","article-title":"Occupancy detection of residential buildings using smart meter data: A large-scale study","volume":"183","author":"Razavi","year":"2019","journal-title":"Energy Build."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1016\/j.procs.2019.08.069","article-title":"Real-Time Occupancy Estimation Using WiFi Network to Optimize HVAC Operation","volume":"155","author":"Simma","year":"2019","journal-title":"Procedia Comput. Sci."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"111377","DOI":"10.1016\/j.enbuild.2021.111377","article-title":"Occupancy-based HVAC control using deep learning algorithms for estimating online preconditioning time in residential buildings","volume":"252","author":"Haghighat","year":"2021","journal-title":"Energy Build."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"110860","DOI":"10.1016\/j.enbuild.2021.110860","article-title":"Reinforcement learning of occupant behavior model for cross-building transfer learning to various HVAC control systems","volume":"238","author":"Deng","year":"2021","journal-title":"Energy Build."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wei, T., Wang, Y., and Zhu, Q. (2017, January 18\u201322). Deep Reinforcement Learning for Building HVAC Control. Proceedings of the 54th Annual Design Automation Conference, Austin, TX, USA.","DOI":"10.1145\/3061639.3062224"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"16111","DOI":"10.1109\/ACCESS.2020.2966545","article-title":"Advanced fuzzy-logic-based context-driven control for HVAC management systems in buildings","volume":"8","author":"Escobar","year":"2020","journal-title":"IEEE Access"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Lou, R., Hallinan, K.P., Huang, K., and Reissman, T. (2020). Smart Wifi Thermostat-Enabled Thermal Comfort Control in Residences. Sustainability, 12.","DOI":"10.3390\/su12051919"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"109791","DOI":"10.1016\/j.enbuild.2020.109791","article-title":"Energy saving impact of occupancy-driven thermostat for residential buildings","volume":"211","author":"Wang","year":"2020","journal-title":"Energy Build."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1950","DOI":"10.1109\/TASE.2020.2983486","article-title":"HVAC Energy Cost Optimization for a Multizone Building via a Decentralized Approach","volume":"17","author":"Yang","year":"2020","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"21684","DOI":"10.1109\/ACCESS.2017.2759800","article-title":"Day Ahead Scheduling to Optimize Industrial HVAC Energy Cost Based on Peak\/OFF-Peak Tariff and Weather Forecasting","volume":"5","author":"Khan","year":"2017","journal-title":"IEEE Access"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.egyr.2019.08.029","article-title":"Air conditioner consumption optimization in an office building considering user comfort","volume":"6","author":"Khorram","year":"2020","journal-title":"Energy Rep."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Corbin, C.D., Makhmalbaf, A., Huang, S., Mendon, V.V., Zhao, M., Somasundaram, S., Liu, G., Ngo, H., and Katipamula, S. (2016). Transactive Control of Commercial Building HVAC Systems, Pacific Northwest National Lab.(PNNL).","DOI":"10.2172\/1406830"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"103237","DOI":"10.1016\/j.autcon.2020.103237","article-title":"Real-time space occupancy sensing and human motion analysis using deep learning for indoor air quality control","volume":"116","author":"Mutis","year":"2020","journal-title":"Autom. Constr."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1109\/TSG.2020.3011739","article-title":"Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings","volume":"12","author":"Yu","year":"2021","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"123402","DOI":"10.1109\/ACCESS.2019.2937639","article-title":"Autonomic management architecture for multi-HVAC systems in smart buildings","volume":"7","author":"Aguilar","year":"2019","journal-title":"IEEE Access"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Garces-Jimenez, A., Gomez-Pulido, J.-M., Gallego-Salvador, N., and Garcia-Tejedor, A.J. (2021). Genetic and Swarm Algorithms for Optimizing the Control of Building HVAC Systems Using Real Data: A Comparative Study. Mathematics, 9.","DOI":"10.3390\/math9182181"},{"key":"ref_35","unstructured":"Collier, M., and Llorens, H.U. (2018). Deep Contextual Multi-armed Bandits. arXiv."},{"key":"ref_36","unstructured":"Riquelme, C., Tucker, G., and Snoek, J. (May, January 30). Deep Bayesian bandits showdown: An empirical comparison of Bayesian deep networks for Thompson sampling. Proceedings of the 6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings, Vancouver, BC, Canada."}],"container-title":["Energies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1996-1073\/14\/24\/8210\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:42:44Z","timestamp":1760168564000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1996-1073\/14\/24\/8210"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,7]]},"references-count":36,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["en14248210"],"URL":"https:\/\/doi.org\/10.3390\/en14248210","relation":{},"ISSN":["1996-1073"],"issn-type":[{"value":"1996-1073","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,7]]}}}