{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,3]],"date-time":"2025-04-03T04:02:34Z","timestamp":1743652954456,"version":"3.40.3"},"publisher-location":"Cham","reference-count":24,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031857027","type":"print"},{"value":"9783031857034","type":"electronic"}],"license":[{"start":{"date-parts":[[2025,1,1]],"date-time":"2025-01-01T00:00:00Z","timestamp":1735689600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,4,1]],"date-time":"2025-04-01T00:00:00Z","timestamp":1743465600000},"content-version":"vor","delay-in-days":90,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Septoplasty and turbinectomy are among the most frequent but also most debated interventions in the field of rhinology. A previously developed tool enhances surgery planning by physical aspects of respiration, i.e., for the first time a reinforcement learning (RL) algorithm is combined with large-scale computational fluid dynamics (CFD) simulations to plan anti-obstructive surgery. In the current study, an improvement of the tool\u2019s predictive capabilities is investigated for the aforementioned types of surgeries by considering two approaches: (i) training of parallel environments is executed on multiple ranks and the agents of each environment share their experience in a pre-defined interval and (ii) for some of the state-reward combinations the CFD solver is replaced by a Gaussian process regression (GPR) model for an improved computational efficiency. It is found that employing a parallel RL algorithm improves the reliability of the surgery planning tool in finding the global optimum. However, parallel training leads to a larger number of state-reward combinations that need to be computed by the CFD solver. This overhead is compensated by replacing some of the computations with the GPR algorithm, i.e., around <jats:inline-formula>\n              <jats:alternatives>\n                <jats:tex-math>$$6\\%$$<\/jats:tex-math>\n                <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mn>6<\/mml:mn>\n                    <mml:mo>%<\/mml:mo>\n                  <\/mml:mrow>\n                <\/mml:math>\n              <\/jats:alternatives>\n            <\/jats:inline-formula> of the computations can be saved without significantly degrading the predictions\u2019 accuracy. Nevertheless, increasing the number of state-reward combinations predicted\u00a0by the GPR algorithm only works to a certain extent, since this also leads to larger errors.<\/jats:p>","DOI":"10.1007\/978-3-031-85703-4_6","type":"book-chapter","created":{"date-parts":[[2025,4,2]],"date-time":"2025-04-02T04:26:42Z","timestamp":1743568002000},"page":"79-96","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Parallel Reinforcement Learning and\u00a0Gaussian Process Regression for\u00a0Improved Physics-Based Nasal Surgery Planning"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3917-8407","authenticated-orcid":false,"given":"Mario","family":"R\u00fcttgers","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7159-8220","authenticated-orcid":false,"given":"Fabian","family":"H\u00fcbenthal","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6555-9575","authenticated-orcid":false,"given":"Makoto","family":"Tsubokura","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3321-6599","authenticated-orcid":false,"given":"Andreas","family":"Lintermann","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,4,1]]},"reference":[{"key":"6_CR1","doi-asserted-by":"publisher","unstructured":"Chen, J., Hachem, E., Viquerat, J.: Graph neural networks for laminar flow prediction around random two-dimensional shapes. Phys. Fluids 33(12), 123607 (2021). https:\/\/doi.org\/10.1063\/5.0064108","DOI":"10.1063\/5.0064108"},{"key":"6_CR2","doi-asserted-by":"publisher","unstructured":"Gardner, J.R., Pleiss, G., Bindel, D., Weinberger, K.Q., Wilson, A.G.: GPyTorch: blackbox matrix-matrix gaussian process inference with GPU acceleration. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7587\u20137597. NIPS 2018, Curran Associates Inc., Red Hook, NY, USA (2018). https:\/\/doi.org\/10.5555\/3327757.3327857","DOI":"10.5555\/3327757.3327857"},{"key":"6_CR3","doi-asserted-by":"publisher","unstructured":"Germano, M., Piomelli, U., Moin, P., Cabot, W.H.: A dynamic subgrid? Scale eddy viscosity model. Phys. Fluids A Fluid Dyn. 3(7), 1760\u20131765 (1991). https:\/\/doi.org\/10.1063\/1.857955","DOI":"10.1063\/1.857955"},{"key":"6_CR4","doi-asserted-by":"publisher","unstructured":"Guastoni, L., Rabault, J., Schlatter, P., Azizpour, H., Vinuesa, R.: Deep reinforcement learning for turbulent drag reduction in channel flows. Eur. Phys. J. E, Soft Matter 46, 27 (2023). https:\/\/doi.org\/10.1140\/epje\/s10189-023-00285-8","DOI":"10.1140\/epje\/s10189-023-00285-8"},{"key":"6_CR5","doi-asserted-by":"publisher","unstructured":"Ko, J., Klein, D.J., Fox, D., Haehnel, D.: Gaussian processes and reinforcement learning for identification and control of an autonomous blimp. In: Proceedings 2007 IEEE International Conference on Robotics and Automation, pp. 742\u2013747. IEEE (2007). https:\/\/doi.org\/10.1109\/ROBOT.2007.363075","DOI":"10.1109\/ROBOT.2007.363075"},{"key":"6_CR6","doi-asserted-by":"publisher","unstructured":"Krause, D., Th\u00f6rnig, P.: JURECA: Modular supercomputer at J\u00fclich Supercomputing Centre. J. Large-scale Res. Facil. 4, A132 (2018). https:\/\/doi.org\/10.17815\/jlsrf-4-121-1","DOI":"10.17815\/jlsrf-4-121-1"},{"key":"6_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocs.2022.101884","volume":"65","author":"M Kurz","year":"2022","unstructured":"Kurz, M., Offenh\u00e4user, P., Viola, D., Shcherbakov, O., Resch, M., Beck, A.: Deep reinforcement learning for computational fluid dynamics on HPC systems. J. Comput. Sci. 65, 101884 (2022). https:\/\/doi.org\/10.1016\/j.jocs.2022.101884","journal-title":"J. Comput. Sci."},{"issue":"11","key":"6_CR8","doi-asserted-by":"publisher","first-page":"1833","DOI":"10.1016\/j.compbiomed.2013.09.003","volume":"43","author":"A Lintermann","year":"2013","unstructured":"Lintermann, A., Meinke, M., Schr\u00f6der, W.: Fluid mechanics based classification of the respiratory efficiency of several nasal cavities. Comput. Biol. Med. 43(11), 1833\u20131852 (2013). https:\/\/doi.org\/10.1016\/j.compbiomed.2013.09.003","journal-title":"Comput. Biol. Med."},{"key":"6_CR9","doi-asserted-by":"publisher","unstructured":"Lintermann, A., Eitel-Amor, G., Meinke, M., Schr\u00f6der, W.: Lattice-boltzmann solutions with local grid refinement for nasal cavity flows. In: New Results in Numerical and Experimental Fluid Mechanics VIII, Notes on Numerical Fluid Mechanics and Multidisciplinary Design, vol.\u00a0121, pp. 583\u2013590. Springer, Heidelberg (2013). https:\/\/doi.org\/10.1007\/978-3-642-35680-3_69","DOI":"10.1007\/978-3-642-35680-3_69"},{"issue":"7\u20138","key":"6_CR10","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1080\/10618562.2020.1742328","volume":"34","author":"A Lintermann","year":"2020","unstructured":"Lintermann, A., Meinke, M., Schr\u00f6der, W.: Zonal flow solver (ZFS): a highly efficient multi-physics simulation framework. Int. J. Comput. Fluid Dyn. 34(7\u20138), 458\u2013485 (2020). https:\/\/doi.org\/10.1080\/10618562.2020.1742328","journal-title":"Int. J. Comput. Fluid Dyn."},{"key":"6_CR11","doi-asserted-by":"publisher","first-page":"474","DOI":"10.1016\/j.future.2024.05.020","volume":"159","author":"X Liu","year":"2024","unstructured":"Liu, X., et al.: Refining computer tomography data with super-resolution networks to increase the accuracy of respiratory flow simulations. Futur. Gener. Comput. Syst. 159, 474\u2013488 (2024). https:\/\/doi.org\/10.1016\/j.future.2024.05.020","journal-title":"Futur. Gener. Comput. Syst."},{"key":"6_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2021.107321","volume":"158","author":"S Peng","year":"2021","unstructured":"Peng, S., Feng, Q.M.: Reinforcement learning with gaussian processes for condition-based maintenance. Comput. Ind. Eng. 158, 107321 (2021). https:\/\/doi.org\/10.1016\/j.cie.2021.107321","journal-title":"Comput. Ind. Eng."},{"key":"6_CR13","doi-asserted-by":"publisher","DOI":"10.1016\/j.oceaneng.2023.115775","volume":"287","author":"K Portal-Porras","year":"2023","unstructured":"Portal-Porras, K., Fernandez-Gamiz, U., Zulueta, E., Garcia-Fernandez, R., Etxebarria Berrizbeitia, S.: Active flow control on airfoils by reinforcement learning. Ocean Eng. 287, 115775 (2023). https:\/\/doi.org\/10.1016\/j.oceaneng.2023.115775","journal-title":"Ocean Eng."},{"key":"6_CR14","volume-title":"Gaussian Processes for Machine Learning","author":"CE Rasmussen","year":"2006","unstructured":"Rasmussen, C.E., Williams, C.K.: Gaussian Processes for Machine Learning, vol. 2. MIT press Cambridge, MA (2006)"},{"key":"6_CR15","unstructured":"Rasmussen, C., Kuss, M., Thrun, S., Saul, L., Sch\u00f6lkopf, B.: Gaussian processes in reinforcement learning. In: Advances in Neural Information Processing Systems, vol. 16, pp. 751\u2013759 (2004)"},{"key":"6_CR16","doi-asserted-by":"publisher","unstructured":"R\u00fcttgers, M., Waldmann, M., Schr\u00f6der, W., Lintermann, A.: Machine learning-based control of perturbed and heated channel flows. In: Jagode, H., Anzt, H., Ltaief, H., Luszczek, P. (eds.) High Performance Computing, pp. 7\u201322. Springer International Publishing, Cham (2021). https:\/\/doi.org\/10.1007\/978-3-030-90539-2_1","DOI":"10.1007\/978-3-030-90539-2_1"},{"key":"6_CR17","doi-asserted-by":"publisher","first-page":"9080","DOI":"10.1007\/s10489-021-02808-2","volume":"52","author":"M R\u00fcttgers","year":"2022","unstructured":"R\u00fcttgers, M., Waldmann, M., Schr\u00f6der, W., Lintermann, A.: A machine-learning-based method for automatizing lattice-boltzmann simulations of respiratory flows. Appl. Intell. 52, 9080\u20139100 (2022). https:\/\/doi.org\/10.1007\/s10489-021-02808-2","journal-title":"Appl. Intell."},{"key":"6_CR18","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2024.108383","volume":"173","author":"M R\u00fcttgers","year":"2024","unstructured":"R\u00fcttgers, M., Waldmann, M., Vogt, K., Ilgner, J., Schr\u00f6der, W., Lintermann, A.: Automated surgery planning for an obstructed nose by combining computational fluid dynamics with reinforcement learning. Comput. Biol. Med. 173, 108383 (2024). https:\/\/doi.org\/10.1016\/j.compbiomed.2024.108383","journal-title":"Comput. Biol. Med."},{"issue":"1","key":"6_CR19","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1109\/JPROC.2015.2494218","volume":"104","author":"B Shahriari","year":"2016","unstructured":"Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., de Freitas, N.: Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104(1), 148\u2013175 (2016). https:\/\/doi.org\/10.1109\/JPROC.2015.2494218","journal-title":"Proc. IEEE"},{"key":"6_CR20","doi-asserted-by":"publisher","unstructured":"Singh, S., Sutton, R., Kaelbling, P.: Reinforcement learning with replacing eligibility traces. Mach. Learn. 22 (1995). https:\/\/doi.org\/10.1023\/A:1018012322525","DOI":"10.1023\/A:1018012322525"},{"key":"6_CR21","doi-asserted-by":"crossref","unstructured":"Smagorinsky, J.: General circulation experiments with the primitive equations: I. the basic experiment. Monthly Weather Rev. 91(3), 99\u2013164 (1963). https:\/\/doi.org\/10.1175\/1520-0493(1963)091$<$0099:GCEWTP$>$2.3.CO;2","DOI":"10.1175\/1520-0493(1963)091<0099:GCEWTP>2.3.CO;2"},{"key":"6_CR22","doi-asserted-by":"publisher","unstructured":"Viquerat, J., Hachem, E.: Parallel bootstrap-based on-policy deep reinforcement learning for continuous fluid flow control applications. Fluids 8(7) (2023). https:\/\/doi.org\/10.3390\/fluids8070208","DOI":"10.3390\/fluids8070208"},{"key":"6_CR23","doi-asserted-by":"publisher","unstructured":"Waldmann, M., R\u00fcttgers, M., Lintermann, A., Schr\u00f6der, W.: Virtual surgeries of nasal cavities using a coupled lattice-boltzmannlevel-set approach. J. Eng. Sci. Med. Diag. Therapy 5(3) (2022). https:\/\/doi.org\/10.1115\/1.4054042","DOI":"10.1115\/1.4054042"},{"key":"6_CR24","unstructured":"Williams, C., Rasmussen, C.: Gaussian processes for regression. In: Touretzky, D., Mozer, M., Hasselmo, M. (eds.) Advances in Neural Information Processing Systems, vol.\u00a08. MIT Press (1995)"}],"container-title":["Lecture Notes in Computer Science","Parallel Processing and Applied Mathematics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-85703-4_6","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,2]],"date-time":"2025-04-02T04:26:46Z","timestamp":1743568006000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-85703-4_6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025]]},"ISBN":["9783031857027","9783031857034"],"references-count":24,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-85703-4_6","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025]]},"assertion":[{"value":"1 April 2025","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"This statement is to declare that the authors of this manuscript, Mario R\u00fcttgers, Fabian H\u00fcbenthal, Makoto Tsubokura, and Andreas Lintermann do not possess any financial dependence that might bias this work. The authors hereby declare that no conflict of interest exists in this work.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclosure of Interests"}},{"value":"PPAM","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Parallel Processing and Applied Mathematics","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Ostrava","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Czech Republic","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"9 September 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"12 September 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"15","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"ppam2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/ppam.edu.pl\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}