{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T15:26:56Z","timestamp":1768404416880,"version":"3.49.0"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,8,2]],"date-time":"2021-08-02T00:00:00Z","timestamp":1627862400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,8,2]],"date-time":"2021-08-02T00:00:00Z","timestamp":1627862400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Stiftelsen f\u00f6r Strategisk Forskning","award":["BD15-0082"],"award-info":[{"award-number":["BD15-0082"]}]},{"DOI":"10.13039\/501100004063","name":"Knut och Alice Wallenbergs Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004063","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["800999"],"award-info":[{"award-number":["800999"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004359","name":"Vetenskapsr\u00e5det","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004359","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004270","name":"Royal Institute of Technology","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004270","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Supercomput"],"published-print":{"date-parts":[[2022,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In situ visualization on high-performance computing systems allows us to analyze simulation results that would otherwise be impossible, given the size of the simulation data sets and offline post-processing execution time. We develop an in situ adaptor for Paraview Catalyst and Nek5000, a massively parallel Fortran and C code for computational fluid dynamics. We perform a strong scalability test up to 2048 cores on KTH\u2019s Beskow Cray XC40 supercomputer and assess in situ visualization\u2019s impact on the Nek5000 performance. In our study case, a high-fidelity simulation of turbulent flow, we observe that in situ operations significantly limit the strong scalability of the code, reducing the relative parallel efficiency to only <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\approx 21\\%$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mo>\u2248<\/mml:mo>\n                    <mml:mn>21<\/mml:mn>\n                    <mml:mo>%<\/mml:mo>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> on 2048 cores (the relative efficiency of Nek5000 without in situ operations is <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\approx 99\\%$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mo>\u2248<\/mml:mo>\n                    <mml:mn>99<\/mml:mn>\n                    <mml:mo>%<\/mml:mo>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>). Through profiling with Arm MAP, we identified a bottleneck in the image composition step (that uses the Radix-kr algorithm) where a majority of the time is spent on MPI communication. We also identified an imbalance of in situ processing time between rank 0 and all other ranks. In our case, better scaling and load-balancing in the parallel image composition would considerably improve the performance of Nek5000 with in situ capabilities. In general, the result of this study highlights the technical challenges posed by the integration of high-performance simulation codes and data-analysis libraries and their practical use in complex cases, even when efficient algorithms already exist for a certain application scenario.<\/jats:p>","DOI":"10.1007\/s11227-021-03990-3","type":"journal-article","created":{"date-parts":[[2021,8,2]],"date-time":"2021-08-02T10:03:04Z","timestamp":1627898584000},"page":"3605-3620","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["In situ visualization of large-scale turbulence simulations in Nek5000 with ParaView Catalyst"],"prefix":"10.1007","volume":"78","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0790-8460","authenticated-orcid":false,"given":"Marco","family":"Atzori","sequence":"first","affiliation":[]},{"given":"Wiebke","family":"K\u00f6pp","sequence":"additional","affiliation":[]},{"given":"Steven W. D.","family":"Chien","sequence":"additional","affiliation":[]},{"given":"Daniele","family":"Massaro","sequence":"additional","affiliation":[]},{"given":"Ferm\u00edn","family":"Mallor","sequence":"additional","affiliation":[]},{"given":"Adam","family":"Peplinski","sequence":"additional","affiliation":[]},{"given":"Mohamad","family":"Rezaei","sequence":"additional","affiliation":[]},{"given":"Niclas","family":"Jansson","sequence":"additional","affiliation":[]},{"given":"Stefano","family":"Markidis","sequence":"additional","affiliation":[]},{"given":"Ricardo","family":"Vinuesa","sequence":"additional","affiliation":[]},{"given":"Erwin","family":"Laure","sequence":"additional","affiliation":[]},{"given":"Philipp","family":"Schlatter","sequence":"additional","affiliation":[]},{"given":"Tino","family":"Weinkauf","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,8,2]]},"reference":[{"key":"3990_CR1","first-page":"717","volume":"836","author":"J Ahrens","year":"2005","unstructured":"Ahrens J, Geveci B, Law C (2005) ParaView: an end-user tool for large data visualization. Vis Handb 836:717\u2013732","journal-title":"Vis Handb"},{"key":"3990_CR2","doi-asserted-by":"crossref","unstructured":"Ayachit U, Bauer A, Duque EPN, Eisenhauer G, Ferrier N, Gu J, Jansen KE, Loring B, Lukic Z, Menon S, Morozov D, O\u2019Leary P, Ranjan R, Rasquin M, Stone CP, Vishwanath V, Weber GH, Whitlock B, Wolf M, Wu KJ, Bethel EW (2016) Performance analysis, design considerations, and applications of extreme-scale in situ infrastructures. In: SC \u201916: l, pp 921\u2013932","DOI":"10.1109\/SC.2016.78"},{"key":"3990_CR3","doi-asserted-by":"crossref","unstructured":"Ayachit U, Bauer A, Geveci B, O\u2019Leary P, Moreland K, Fabian N, Mauldin J (2015) ParaView catalyst: enabling in situ data analysis and visualization. In: Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, pp 25\u201329","DOI":"10.1145\/2828612.2828624"},{"key":"3990_CR4","doi-asserted-by":"crossref","unstructured":"Bernardoni B, Ferrier N, Insley J, Papka M.E, Patel S, Rizzi S (2018) In situ visualization and analysis to design large scale experiments in computational fluid dynamics. In: 2018 IEEE 8th Symposium on Large Data Analysis and Visualization (LDAV). IEEE, pp 94\u201395","DOI":"10.1109\/LDAV.2018.8739172"},{"key":"3990_CR5","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511546792","volume-title":"High-order methods for incompressible fluid flows","author":"MO Deville","year":"2002","unstructured":"Deville MO, Fischer PF, Mund EH (2002) High-order methods for incompressible fluid flows. Cambridge University Press, New York"},{"key":"3990_CR6","doi-asserted-by":"crossref","unstructured":"Dorier M, Sisneros R, Peterka T, Antoniu G, Semeraro D (2013) Damaris\/viz: a nonintrusive, adaptable and user-friendly in situ visualization framework. In: 2013 IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV). IEEE, pp 67\u201375","DOI":"10.1109\/LDAV.2013.6675160"},{"key":"3990_CR7","doi-asserted-by":"publisher","first-page":"475","DOI":"10.1007\/s10494-013-9482-8","volume":"91","author":"G El Khoury","year":"2013","unstructured":"El Khoury G, Schlatter P, Noorani A, Brethouwer G, Johansson AV (2013) Direct numerical simulation of turbulent pipe flow at moderately high Reynolds numbers. Flow Turbul Combust 91:475\u2013495","journal-title":"Flow Turbul Combust"},{"key":"3990_CR8","unstructured":"Fischer P, Kruse J, Mullen J, Tufo H, Lottes J, Kerkemeier S (2008) Nek5000: Open source spectral element CFD solver. https:\/\/nek5000.mcs.anl.gov\/"},{"key":"3990_CR9","unstructured":"Fischer PF (1989) Spectral element solution of the Navier\u2013Stokes equations on high performance distributed-memory parallel processors. Ph.D. thesis, Massachusetts Institute of Technology"},{"key":"3990_CR10","doi-asserted-by":"crossref","unstructured":"Haimes R (1994) pv3\u2014A distributed system for large-scale unsteady cfd visualization. In: 32nd Aerospace Sciences Meeting and Exhibit","DOI":"10.2514\/6.1994-321"},{"key":"3990_CR11","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1016\/j.ijheatfluidflow.2016.02.001","volume":"61","author":"SM Hosseini","year":"2016","unstructured":"Hosseini SM, Vinuesa R, Schlatter P, Hanifi A, Henningson DS (2016) Direct numerical simulation of the flow around a wing section at moderate Reynolds number. Int J Heat Fluid Flow 61:117\u2013128","journal-title":"Int J Heat Fluid Flow"},{"key":"3990_CR12","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1017\/S0022112095000462","volume":"285","author":"J Jeong","year":"1995","unstructured":"Jeong J, Hussain F (1995) On the identification of a vortex. J Fluid Mech 285:69\u201394","journal-title":"J Fluid Mech"},{"key":"3990_CR13","first-page":"101","volume":"10","author":"W Kendall","year":"2010","unstructured":"Kendall W, Peterka T, Huang J, Shen HW, Ross RB (2010) Accelerating and benchmarking radix-k image compositing at large scale. EGPGV 10:101\u2013110","journal-title":"EGPGV"},{"key":"3990_CR14","unstructured":"Ma KL (1995) Runtime volume visualization for parallel cfd. Tech. Rep. ICASE-95-74, Institute for Computer Applications in Science and Engineering (ICASE)"},{"issue":"6","key":"3990_CR15","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1109\/MCG.2009.120","volume":"29","author":"KL Ma","year":"2009","unstructured":"Ma KL (2009) In situ visualization at extreme scale: challenges and opportunities. IEEE Comput Graph Appl 29(6):14\u201319","journal-title":"IEEE Comput Graph Appl"},{"key":"3990_CR16","doi-asserted-by":"publisher","unstructured":"Merzari E, Rahaman R, Min M, Fischer P (2018) Performance analysis of nek5000 for single-assembly calculations. In: ASME 2018 5th Joint US\u2013European Fluids Engineering Division Summer Meeting: Development and Applications in Computational Fluid Dynamics; Industrial and Environmental Applications of Fluid Mechanics; Fluid Measurement and Instrumentation; Cavitation and Phase Change, vol.\u00a02. https:\/\/doi.org\/10.1115\/FEDSM2018-83517.V002T09A031","DOI":"10.1115\/FEDSM2018-83517.V002T09A031"},{"key":"3990_CR17","doi-asserted-by":"publisher","unstructured":"Moreland K, Kendall W, Peterka T, Huang J (2011) An image compositing solution at scale. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC \u201911. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2063384.2063417","DOI":"10.1145\/2063384.2063417"},{"key":"3990_CR18","doi-asserted-by":"crossref","unstructured":"Offermans N (2019) Aspects of adaptive mesh refinement in the spectral element method. Ph.D. thesis, KTH Royal Institute of Technology","DOI":"10.1007\/978-3-030-04915-7_2"},{"key":"3990_CR19","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/j.jpdc.2019.05.010","volume":"132","author":"E Otero","year":"2019","unstructured":"Otero E, Gong J, Min M, Fischer P, Schlatter P, Laure E (2019) OpenACC acceleration for the PN-PN-2 algorithm in Nek5000. J Parallel Distrib Comput 132:69\u201378","journal-title":"J Parallel Distrib Comput"},{"issue":"3","key":"3990_CR20","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1016\/0021-9991(84)90128-1","volume":"54","author":"AT Patera","year":"1984","unstructured":"Patera AT (1984) A spectral element method for fluid dynamics: laminar flow in a channel expansion. J Comput Phys 54(3):468\u2013488","journal-title":"J Comput Phys"},{"key":"3990_CR21","first-page":"599","volume-title":"Spectral and high order methods for partial differential equations ICOSAHOM 2018","author":"A Peplinski","year":"2019","unstructured":"Peplinski A, Offermans N, Marin O, Fischer P, Schlatter P (2019) Non-conforming elements in nek5000: pressure preconditioning and parallel performance. In: Sherwin S, Moxey D, Peir\u00f3 J, Vincent P, Schwab C (eds) Spectral and high order methods for partial differential equations ICOSAHOM 2018. Springer, Cham, pp 599\u2013609"},{"key":"3990_CR22","doi-asserted-by":"publisher","unstructured":"Peterka T, Goodell D, Ross R, Shen HW, Thakur R (2009) A configurable algorithm for parallel image-compositing applications. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, SC \u201909. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/1654059.1654064","DOI":"10.1145\/1654059.1654064"},{"key":"3990_CR23","doi-asserted-by":"crossref","unstructured":"Rasquin M, Marion P, Vishwanath V, Matthews B, Hereld M, Jansen K, Loy R, Bauer A, Zhou M, Sahni O, Fu J, Liu N, Carothers C, Shephard M, Papka M, Kumaran K, Geveci B (2011) Electronic poster: co-visualization of full data and in situ data extracts from unstructured grid cfd at 160k cores. In: Proceedings of the 2011 Companion on High Performance Computing Networking, Storage and Analysis Companion, SC \u201911 Companion. Association for Computing Machinery, New York, NY, USA, pp 103\u2013104","DOI":"10.1145\/2148600.2148653"},{"key":"3990_CR24","volume-title":"The visualization toolkit-an object-oriented approach to 3D graphics","author":"W Schroeder","year":"2006","unstructured":"Schroeder W, Martin K, Lorensen B (2006) The visualization toolkit-an object-oriented approach to 3D graphics, 4th edn. Kitware, Inc, Washington","edition":"4"},{"key":"3990_CR25","doi-asserted-by":"publisher","first-page":"7","DOI":"10.2312\/pgv.20191105","volume-title":"Eurographics Symposium on Parallel Graphics and Visualization","author":"S Shudler","year":"2019","unstructured":"Shudler S, Ferrier N, Insley J, Papka ME, Patel S, Rizzi S (2019) Fast mesh validation in combustion simulations through in-situ visualization. In: Childs H, Frey S (eds) Eurographics Symposium on Parallel Graphics and Visualization. The Eurographics Association, Aire-la-Ville, pp 7\u201316. https:\/\/doi.org\/10.2312\/pgv.20191105"},{"key":"3990_CR26","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1007\/s10494-020-00152-y","volume":"105","author":"A Tanarro","year":"2020","unstructured":"Tanarro A, Mallor F, Offermans N, Peplinski A, Vinuesa R, Schlatter P (2020) Enabling adaptive mesh refinement for spectral-element simulations of turbulence around wing sections. Flow Turbul Combust 105:415\u2013436. https:\/\/doi.org\/10.1007\/s10494-020-00152-y","journal-title":"Flow Turbul Combust"},{"key":"3990_CR27","doi-asserted-by":"publisher","first-page":"A8","DOI":"10.1017\/jfm.2019.838","volume":"883","author":"A Tanarro","year":"2020","unstructured":"Tanarro A, Vinuesa R, Schlatter P (2020) Effect of adverse pressure gradients on turbulent wing boundary layers. J Fluid Mech 883:A8. https:\/\/doi.org\/10.1017\/jfm.2019.838","journal-title":"J Fluid Mech"},{"key":"3990_CR28","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1016\/j.ijheatfluidflow.2018.04.017","volume":"72","author":"R Vinuesa","year":"2018","unstructured":"Vinuesa R, Negi PS, Atzori M, Hanifi A, Henningson DS, Schlatter P (2018) Turbulent boundary layers around wing sections up to $$Re_c=1,000,000$$. Int J Heat Fluid Flow 72:86\u201399","journal-title":"Int J Heat Fluid Flow"},{"key":"3990_CR29","doi-asserted-by":"crossref","unstructured":"Yi H, Rasquin M, Fang J, Bolotnov I.A (2014) In-situ visualization and computational steering for large-scale simulation of turbulent flows in complex geometries. In: 2014 IEEE International Conference on Big Data (Big Data), pp 567\u2013572","DOI":"10.1109\/BigData.2014.7004275"}],"container-title":["The Journal of Supercomputing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11227-021-03990-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11227-021-03990-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11227-021-03990-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,7]],"date-time":"2022-02-07T13:17:02Z","timestamp":1644239822000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11227-021-03990-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,2]]},"references-count":29,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,2]]}},"alternative-id":["3990"],"URL":"https:\/\/doi.org\/10.1007\/s11227-021-03990-3","relation":{},"ISSN":["0920-8542","1573-0484"],"issn-type":[{"value":"0920-8542","type":"print"},{"value":"1573-0484","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,2]]},"assertion":[{"value":"7 July 2021","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 August 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}