{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T15:01:16Z","timestamp":1767970876018,"version":"3.49.0"},"reference-count":58,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,2,18]],"date-time":"2025-02-18T00:00:00Z","timestamp":1739836800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"PhD fellowship of the Research Foundation Flanders","award":["1S08622N"],"award-info":[{"award-number":["1S08622N"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2025,4,30]]},"abstract":"<jats:p>The conventional approach for controlling the supply temperature in collective space heating networks relies on a predefined heating curve determined by outdoor temperature and heat emitter type. This prioritises thermal comfort but lacks energetic and financial optimisation. This research proposes an adaptive supply temperature control in well-insulated dwellings, responsive to diverse environmental parameters. The approach considers variable electricity prices and accommodates different indoor temperature set points in dwellings. The study evaluates the effectiveness of two Deep Reinforcement Learning (DRL) algorithms, i.e., Proximal Policy Optimisation (PPO) and Deep Q-Network (DQN), across various scenarios. Results reveal that DQN excels in collective space heating systems with underfloor heating in each dwelling, while PPO proves superior for radiator-based systems. Both outperform the traditional heating curve, achieving up to 13.77% (DQN) and 16.15% (PPO) cost reduction while guaranteeing thermal comfort. Additionally, the research highlights the capability of DRL-based methods to dynamically set the supply temperature based on a cloud of set points, showcasing adaptability to diverse environmental factors and addressing the growing significance of indoor heat gains in well-insulated dwellings. This innovative approach holds promise for more efficient and environmentally conscious heating strategies within collective space heating networks.<\/jats:p>","DOI":"10.1145\/3709010","type":"journal-article","created":{"date-parts":[[2024,12,19]],"date-time":"2024-12-19T15:56:07Z","timestamp":1734623767000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Model-Free Deep Reinforcement Learning for Adaptive Supply Temperature Control in Collective Space Heating Systems"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0159-175X","authenticated-orcid":false,"given":"Sara","family":"Ghane","sequence":"first","affiliation":[{"name":"IDLab \u2013 Faculty of Applied Engineering, University of Antwerp \u2013 imec, Antwerp, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8179-3468","authenticated-orcid":false,"given":"Stef","family":"Jacobs","sequence":"additional","affiliation":[{"name":"EMIB, Faculty of Applied Engineering \u2013 Electromechanics, University of Antwerp, Antwerp, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5611-6331","authenticated-orcid":false,"given":"Thomas","family":"Huybrechts","sequence":"additional","affiliation":[{"name":"IDLab \u2013 Faculty of Applied Engineering, University of Antwerp \u2013 imec, Antwerp, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8029-4720","authenticated-orcid":false,"given":"Peter","family":"Hellinckx","sequence":"additional","affiliation":[{"name":"Modelling for Sustainability (M4S) \u2013 Faculty of Applied Engineering, University of Antwerp, Antwerp, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9355-6566","authenticated-orcid":false,"given":"Siegfried","family":"Mercelis","sequence":"additional","affiliation":[{"name":"IDLab \u2013 Faculty of Applied Engineering, University of Antwerp \u2013 imec, Antwerp, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5238-6123","authenticated-orcid":false,"given":"Ivan","family":"Verhaert","sequence":"additional","affiliation":[{"name":"EMIB, Faculty of Applied Engineering \u2013 Electromechanics, University of Antwerp, Antwerp, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7946-4884","authenticated-orcid":false,"given":"Erik","family":"Mannens","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Antwerp, Antwerp, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,2,18]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jobe.2024.110085"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scs.2021.103445"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.apenergy.2021.118346"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.enbuild.2017.08.052"},{"key":"e_1_3_1_6_2","unstructured":"European Commission. 2020. Renovation Wave. Retrieved from https:\/\/energy.ec.europa.eu\/topics\/energy-efficiency\/energy-efficient-buildings\/renovation-wave_en"},{"key":"e_1_3_1_7_2","volume-title":"Proceedings of the REHVA Annual Meeting Conference: Low Carbon Technologies in HVAC","author":"Pauw Margot De","year":"2018","unstructured":"Margot De Pauw, Freek Van Riet, Jef De Schutter, Simon Binnemans, Jeroen Van der Veken, and Ivan Verhaert. 2018. A methodology to compare collective heating systems with individual heating systems in buildings. In Proceedings of the REHVA Annual Meeting Conference: Low Carbon Technologies in HVAC. REHVA."},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.apenergy.2020.116117"},{"key":"e_1_3_1_9_2","unstructured":"Elexys. 2022. Spot BelpexDAM Prices for Electricity in Belgium. Retrieved January 1 2023 from https:\/\/my.elexys.be\/MarketInformation\/SpotBelpex.aspx"},{"key":"e_1_3_1_10_2","unstructured":"European Commission. 2020. A Renovation Wave for Europe - Greening our Buildings Creating Jobs Improving Lives. Retrieved from https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?qid=1603122220757&uri=CELEX:52020DC0662"},{"key":"e_1_3_1_11_2","unstructured":"European Commission. 2021. A European Green Deal: Striving to be the First Climate-Neutral Continent. Retrieved from https:\/\/commission.europa.eu\/strategy-and-policy\/priorities-2019-2024\/european-green-deal_en"},{"key":"e_1_3_1_12_2","unstructured":"European Environment Agency. 2023. Greenhouse Gas Emissions from Energy Use in Buildings in Europe. Retrieved from https:\/\/www.eea.europa.eu\/ims\/greenhouse-gas-emissions-from-energy"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.applthermaleng.2015.01.067"},{"key":"e_1_3_1_14_2","unstructured":"Guanyu Gao Jie Li and Yonggang Wen. 2019. Energy-efficient thermal comfort control in smart buildings via deep reinforcement learning. arXiv:1901.04693. Retrieved from https:\/\/arxiv.org\/abs\/1901.04693"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISC253183.2021.9562966"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jobe.2020.101739"},{"key":"e_1_3_1_17_2","first-page":"270","volume-title":"Energy","author":"Han Gwangwoo","year":"2023","unstructured":"Gwangwoo Han, Hong-Jin Joo, Hee-Won Lim, Young-Sub An, Wang-Je Lee, and Kyoung-Ho Lee. 2023. Data-driven heat pump operation strategy using rainbow deep reinforcement learning for significant reduction of electricity cost. Energy 270 (2023), 126913."},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scs.2019.101748"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ETFA52439.2022.9921461"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.23919\/JCIN.2019.8917869"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.3390\/en16083435"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/IECON49645.2022.9968547"},{"key":"e_1_3_1_23_2","volume-title":"Proceedings of the CausalML Workshop","author":"Jagannatha Abhyuday","year":"2018","unstructured":"Abhyuday Jagannatha, Philip Thomas, and Hong Yu. 2018. Towards high confidence off-policy reinforcement learning for clinical applications. In Proceedings of the CausalML Workshop, ICML."},{"key":"e_1_3_1_24_2","unstructured":"Joint Research Centre. 2023. Residential heating: Heat pumps would knock down energy consumption and emissions. Retrieved from https:\/\/joint-research-centre.ec.europa.eu\/jrc-news-and-updates\/residential-heating-heat-pumps-would-knock-down-energy-consumption-and-emissions-2023-06-21_en"},{"key":"e_1_3_1_25_2","unstructured":"Adrien Le-Coz Tahar Nabil and Francois Courtot. 2020. Towards optimal district heating temperature control in China with deep reinforcement learning. arXiv:2012.09508. Retrieved from https:\/\/arxiv.org\/abs\/2012.09508"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMST.2020.2988367"},{"key":"e_1_3_1_27_2","first-page":"3053","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Liang Eric","year":"2018","unstructured":"Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2018. RLlib: Abstractions for distributed reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 3053\u20133062."},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13705-019-0192-5"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.egyai.2020.100043"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.energy.2014.02.089"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compeleceng.2019.07.019"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2021.105400"},{"key":"e_1_3_1_33_2","unstructured":"Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv:1312.5602 Retrieved from https:\/\/arxiv.org\/abs\/1312.5602"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_3_1_35_2","unstructured":"Adam Nagy Hussain Kazmi Farah Cheaib and Johan Driesen. 2018. Deep reinforcement learning for optimal control of space heating. arXiv:1805.03777. Retrieved from https:\/\/arxiv.org\/abs\/1805.03777"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459991"},{"key":"e_1_3_1_37_2","first-page":"229","volume-title":"Energy","author":"Pinto Giuseppe","year":"2021","unstructured":"Giuseppe Pinto, Marco Savino Piscitelli, Jos\u00e9 Ram\u00f3n V\u00e1zquez-Canteli, Zolt\u00e1n Nagy, and Alfonso Capozzoli. 2021. Coordinated energy management for a cluster of buildings through deep reinforcement learning. Energy 229 (2021), 120725."},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.23919\/ACC45564.2020.9147629"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.3390\/en8088300"},{"key":"e_1_3_1_40_2","unstructured":"John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. Retrieved from https:\/\/arxiv.org\/abs\/1707.06347"},{"key":"e_1_3_1_41_2","unstructured":"Solar Energy Laboratory Univ. of Wisconsin-Madison. 2009. TRNSYS 17 Volume 4 Mathematical Reference: Type 60 (Stratified Fluid Storage Tank with Internal Heat Exchangers) 390\u2013396."},{"key":"e_1_3_1_42_2","unstructured":"Solar Energy Laboratory Univ. of Wisconsin-Madison. 2014. TRNSYS 17 Volume 8 Weather Data."},{"key":"e_1_3_1_43_2","unstructured":"EPEX SPOT. 2021. Basics of the Power Market. Retrieved from https:\/\/www.epexspot.com\/en\/basicspowermarket"},{"key":"e_1_3_1_44_2","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. The MIT Press."},{"key":"e_1_3_1_45_2","unstructured":"Unicorn systems. 2022. Entsoe Transparency Platform. Retrieved from https:\/\/transparency.entsoe.eu\/transmission-domain\/r2\/dayAheadPrices\/"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.2760\/27877"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.34641\/clima.2022.134"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"e_1_3_1_49_2","volume-title":"Hydronic Design of Hybrid Thermal Production Systems in Buildings","author":"Riet Freek Van","year":"2019","unstructured":"Freek Van Riet. 2019. Hydronic Design of Hybrid Thermal Production Systems in Buildings. University of Antwerp."},{"key":"e_1_3_1_50_2","first-page":"1","volume-title":"Proceedings of the 10th International Conference on System Simulation in Buildings","author":"Riet Freek Van","year":"2018","unstructured":"Freek Van Riet, Gunther Steenacker, and Ivan Verhaert. 2018. A new approach to model transport Delay in branched pipes. In Proceedings of the 10th International Conference on System Simulation in Buildings, 1\u201311."},{"key":"e_1_3_1_51_2","unstructured":"Vlaams Energie \u2013 en Klimaatagentschap. 2022. Energiebesluit bijlage V \u2013 Bepalingsmethode EPW 2022. Retrieved from https:\/\/www.vlaanderen.be\/epb-pedia\/epb-regelgeving\/energiebesluit-en-bijlagen\/energiebesluit-bijlage-v"},{"key":"e_1_3_1_52_2","unstructured":"VLAIO. 2021. Instal 2020 project: Integraal ontwerp van installaties voor sanitair en verwarming (Dutch). VIS 135098 (2014-2018). Retrieved from https:\/\/www.instal2020.be"},{"key":"e_1_3_1_53_2","unstructured":"VLAIO. 2021. Productie En distributie van Sanitair warm water: Selectie en dimensionering (Dutch). TETRA 120145 (2012-2014). Retrieved from https:\/\/www.tetra-sww.be\/"},{"key":"e_1_3_1_54_2","doi-asserted-by":"crossref","unstructured":"Zhe Wang and Tianzhen Hong. 2020. Reinforcement learning for building controls: The opportunities and challenges. Applied Energy 269 (2020) 115036.","DOI":"10.1016\/j.apenergy.2020.115036"},{"key":"e_1_3_1_55_2","first-page":"1995","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Wang Ziyu","year":"2016","unstructured":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016. Dueling network architectures for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1995\u20132003."},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSG.2020.2976771"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.enbuild.2019.07.029"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1134\/S0040601522060088"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.buildenv.2019.106269"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3709010","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3709010","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:55Z","timestamp":1750295875000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3709010"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,18]]},"references-count":58,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,4,30]]}},"alternative-id":["10.1145\/3709010"],"URL":"https:\/\/doi.org\/10.1145\/3709010","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,18]]},"assertion":[{"value":"2024-03-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}