{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T16:59:09Z","timestamp":1776445149361,"version":"3.51.2"},"reference-count":31,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T00:00:00Z","timestamp":1726444800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Metrology"],"abstract":"<jats:p>As modern systems become more complex, their control strategy no longer relies only on measurement data from probes; it also requires information from mathematical models for non-measurable places. On the other hand, those mathematical models can lead to unbearable computation times due to their own complexity, making the control process non-viable. To overcome this problem, it is possible to implement any kind of surrogate model that enables the computation of such estimates within an acceptable time frame, which allows for making decisions. Using a Physics-Informed Neural Network as a surrogate model, it is possible to compute the temperature distribution at each time step, replacing the need for running direct numerical simulations. This approach enables the use of a Deep Reinforcement Learning algorithm to train a control strategy. On this work, we considered a one-dimensional heat conduction problem, in which temperature distribution feeds a control system. Such control system has the objective of reacing and maintaining constant temperature value at a specific location of the 1D problem by activating a heat source; the desired location somehow cannot be directly measured so, the PINN approach allows to estimate its temperature with a minimum computational workload. With this approach, the control training becomes much faster without the need of performing numerical simulations or laboratory measurements.<\/jats:p>","DOI":"10.3390\/metrology4030030","type":"journal-article","created":{"date-parts":[[2024,9,18]],"date-time":"2024-09-18T01:52:31Z","timestamp":1726624351000},"page":"489-505","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Heat Conduction Control Using Deep Q-Learning Approach with Physics-Informed Neural Networks"],"prefix":"10.3390","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5234-150X","authenticated-orcid":false,"given":"Nelson D.","family":"Gon\u00e7alves","sequence":"first","affiliation":[{"name":"Institute of Science and Innovation in Mechanical and Industrial Engineering (INEGI), Rua Dr. Roberto Frias, 400, 4200-465 Porto, Portugal"},{"name":"LAETA\u2014Associated Laboratory of Energy, Transports and Aerospace, 4200-265 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0009-2833","authenticated-orcid":false,"given":"Jhonny","family":"de S\u00e1 Rodrigues","sequence":"additional","affiliation":[{"name":"Institute of Science and Innovation in Mechanical and Industrial Engineering (INEGI), Rua Dr. Roberto Frias, 400, 4200-465 Porto, Portugal"},{"name":"LAETA\u2014Associated Laboratory of Energy, Transports and Aerospace, 4200-265 Porto, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2024,9,16]]},"reference":[{"key":"ref_1","unstructured":"Evans, L.C. (2022). Partial Differential Equations, American Mathematical Society."},{"key":"ref_2","unstructured":"Iakovlev, V., Heinonen, M., and L\u00e4hdesm\u00e4ki, H. (2020). Learning continuous-time pdes from sparse data with graph neural networks. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Haitsiukevich, K., and Ilin, A. (2023, January 18\u201323). Improved training of physics-informed neural networks with model ensembles. Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), Gold Coast, Australia.","DOI":"10.1109\/IJCNN54540.2023.10191822"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"109008","DOI":"10.1016\/j.ijheatfluidflow.2022.109008","article-title":"Deep reinforcement learning for active control of flow over a circular cylinder with rotational oscillations","volume":"96","author":"Han","year":"2022","journal-title":"Int. J. Heat Fluid Flow"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1017\/jfm.2016.615","article-title":"Reynolds averaged turbulence modelling using deep neural networks with embedded invariance","volume":"807","author":"Ling","year":"2016","journal-title":"J. Fluid Mech."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Li, F.-F. (2014, January 23\u201328). Large-scale video classification with convolutional neural networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.223"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1109\/MSP.2012.2205597","article-title":"Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups","volume":"29","author":"Hinton","year":"2012","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"987","DOI":"10.1109\/72.712178","article-title":"Artificial neural networks for solving ordinary and partial differential equations","volume":"9","author":"Lagaris","year":"1998","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1016\/j.ifacol.2022.09.117","article-title":"Physics-informed neural networks-based model predictive control for multi-link manipulators","volume":"55","author":"Nicodemus","year":"2022","journal-title":"IFAC-PapersOnLine"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/j.ifacol.2022.09.070","article-title":"Physics informed neural networks for power transformer dynamic thermal modelling","volume":"55","author":"Laneryd","year":"2022","journal-title":"IFAC-PapersOnLine"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.ifacol.2022.09.015","article-title":"Generalized feedforward control using physics\u2014Informed neural networks","volume":"55","author":"Bolderman","year":"2022","journal-title":"IFAC-PapersOnLine"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"A3055","DOI":"10.1137\/20M1318043","article-title":"Understanding and mitigating gradient flow pathologies in physics-informed neural networks","volume":"43","author":"Wang","year":"2021","journal-title":"SIAM J. Sci. Comput."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"113938","DOI":"10.1016\/j.cma.2021.113938","article-title":"On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks","volume":"384","author":"Wang","year":"2021","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"ref_15","unstructured":"Wang, S., Sankaran, S., and Perdikaris, P. (2022). Respecting causality is all you need for training physics-informed neural networks. arXiv."},{"key":"ref_16","first-page":"26548","article-title":"Characterizing possible failure modes in physics-informed neural networks","volume":"34","author":"Krishnapriyan","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_17","unstructured":"Wight, C.L., and Zhao, J. (2020). Solving Allen-Cahn and Cahn-Hilliard equations using the adaptive physics informed neural networks. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"114474","DOI":"10.1016\/j.cma.2021.114474","article-title":"A novel sequential method to train physics informed neural networks for allen cahn and cahn hilliard equations","volume":"390","author":"Mattey","year":"2022","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Prantikos, K., Tsoukalas, L.H., and Heifetz, A. (2022). Physics-Informed Neural Network Solution of Point Kinetics Equations for a Nuclear Reactor Digital Twin. Energies, 15.","DOI":"10.3390\/en15207697"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"127419","DOI":"10.1016\/j.neucom.2024.127419","article-title":"Physics-informed neural nets for control of dynamical systems","volume":"579","author":"Antonelo","year":"2024","journal-title":"Neurocomputing"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"119565","DOI":"10.1016\/j.renene.2023.119565","article-title":"Multi-scale modeling in thermal conductivity of Polyurethane incorporated with Phase Change Materials using Physics-Informed Neural Networks","volume":"220","author":"Liu","year":"2024","journal-title":"Renew. Energy"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Fan, D., Yang, L., Triantafyllou, M.S., and Karniadakis, G.E. (2020). Reinforcement learning for active flow control in experiments. arXiv.","DOI":"10.1073\/pnas.2004939117"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1007\/s00162-023-00641-6","article-title":"Backpropagation of neural network dynamical models applied to flow control","volume":"37","author":"Wolf","year":"2023","journal-title":"Theor. Comput. Fluid Dyn."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.arcontrol.2018.09.005","article-title":"Reinforcement learning for control: Performance, stability, and deep approximators","volume":"46","author":"Kober","year":"2018","journal-title":"Annu. Rev. Control"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"104973","DOI":"10.1016\/j.compfluid.2021.104973","article-title":"A review on deep reinforcement learning for fluid mechanics","volume":"225","author":"Garnier","year":"2021","journal-title":"Comput. Fluids"},{"key":"ref_27","unstructured":"Rosenblatt, F. (1957). A Perceiving and Recognizing Automation, Cornell Aeronautical Laboratory. Technical Report."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1007\/s42241-020-0028-y","article-title":"Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization","volume":"32","author":"Rabault","year":"2020","journal-title":"J. Hydrodyn."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/S0377-0427(00)00422-2","article-title":"Automatic differentiation of algorithms","volume":"124","author":"Brown","year":"2000","journal-title":"J. Comput. Appl. Math."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_31","unstructured":"Liu, L., Kuo, S.M., and Zhou, M. (2009, January 26\u201329). Virtual sensing techniques and their applications. Proceedings of the 2009 International Conference on Networking, Sensing and Control, Okayama, Japan."}],"container-title":["Metrology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2673-8244\/4\/3\/30\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:57:20Z","timestamp":1760111840000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2673-8244\/4\/3\/30"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,16]]},"references-count":31,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["metrology4030030"],"URL":"https:\/\/doi.org\/10.3390\/metrology4030030","relation":{},"ISSN":["2673-8244"],"issn-type":[{"value":"2673-8244","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,16]]}}}