×

STREAmS: a high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows. (English) Zbl 1539.76002

Summary: We present STREAmS, an in-house high-fidelity solver for direct numerical simulations (DNS) of canonical compressible wall-bounded flows, namely turbulent plane channel, zero-pressure gradient turbulent boundary layer and supersonic oblique shock-wave/boundary layer interaction. The solver incorporates state-of-the-art numerical algorithms, specifically designed to cope with the challenging problems associated with the solution of high-speed turbulent flows and can be used across a wide range of Mach numbers, extending from the low subsonic up to the hypersonic regime. From the computational viewpoint, STREAmS is oriented to modern HPC platforms thanks to MPI parallelization and the ability to run on multi-GPU architectures. This paper discusses the main implementation strategies, with particular reference to the CUDA paradigm, the management of a single code for traditional and multi-GPU architectures, and the optimization process to take advantage of the latest generation of NVIDIA GPUs. Performance measurements show that single-GPU optimization more than halves the computing time as compared to the baseline version. At the same time, the asynchronous patterns implemented in STREAmS for MPI communications guarantee very good parallel performance especially in the weak scaling spirit, with efficiency exceeding 97% on 1024 GPUs. For overall evaluation of STREAmS with respect to other compressible solvers, comparison with a recent GPU-enabled community solver is presented. It turns out that, although STREAmS is much more limited in terms of flow configurations that can be addressed, the advantage in terms of accuracy, computing time and memory occupation is substantial, which makes it an ideal candidate for large-scale simulations of high-Reynolds number, compressible wall-bounded turbulent flows. The solver is released open source under GPLv3 license.

MSC:

76-04 Software, source code, etc. for problems pertaining to fluid mechanics
65Y10 Numerical algorithms for specific classes of architectures
76F50 Compressibility effects in turbulence
76F65 Direct numerical and large eddy simulation of turbulence

References:

[1] Harlow, F.; Welch, J., Phys. Fluids, 8, 12, 2182-2189 (1965) · Zbl 1180.76043
[2] Patterson, G.; Orszag, S., Phys. Fluids, 14, 11, 2538-2541 (1971) · Zbl 0225.76033
[3] Kim, J.; Moin, P., J. Comput. Phys., 59, 2, 308-323 (1985) · Zbl 0582.76038
[4] Kim, J.; Moin, P.; Moser, R., J. Fluid Mech., 177, 133-166 (1987) · Zbl 0616.76071
[5] Weller, H.; Tabor, G.; Jasak, H.; Fureby, C., Comput. Phys., 12, 6, 620-631 (1998)
[6] Fisher, P.; Kruse, J.; Mullen, J.; Tufo, H.; Lottes, J.; Kerkemeier, S., NEK5000: open source spectral element CFD solver (2008), URL http://nek5000.mcs.anl.gov/index.php/MainPage
[7] Cantwell, C.; Moxey, D.; Comerford, A.; Bolis, A.; Rocco, G.; Mengaldo, G.; Grazia, D. D.; Yakovlev, S.; Lombard, J.-E.; Ekelschot, D.; Jordi, B.; Hu, H.; Mohamied, Y.; Eskilsson, C.; Nelson, B.; Vos, P.; Biotto, C.; Kirby, R.; Sherwin, S., Comput. Phys. Comm., 192, 205-219 (2015) · Zbl 1380.65465
[8] van der Poel, E.; Ostilla-Mónico, R.; Donners, J.; Verzicco, R., Comput. Fluids, 116, 10-16 (2015) · Zbl 1390.76250
[9] Costa, P., Comput. Math. Appl., 76, 8, 1853-1862 (2018) · Zbl 1442.65156
[10] Dongarra, J.; Luszczek, P., (Padua, D., Encyclopedia of Parallel Computing (2011), Springer US: Springer US Boston, MA), 2055-2057, https://doi.org/10.1007/978-0-387-09766-4_157 · Zbl 1231.68001
[11] 136 GPU-Accelerated Supercomputers Feature in TOP500 — NVIDIA Blog (2019), https://blogs.nvidia.com/blog/2019/11/19/record-gpu-accelerated-supercomputers-top500/. (Accessed 16 January 2020)
[12] The GREEN 500 (2020), https://www.top500.org/green500/. (Accessed 16 January 2020)
[13] Zhu, X.; Phillips, E.; Spandan, V.; Donners, J.; Ruetsch, G.; Romero, J.; Ostilla-Mónico, R.; Yang, Y.; Lohse, D.; Verzicco, R.; Fatica, M.; Stevens, R., Comput. Phys. Commun., 229, 199-210 (2018)
[14] Costa, P.; Phillips, E.; Brandt, L.; Fatica, M., Comput. Math. Appl. (2020)
[15] Honein, A.; Moin, P., J. Comput. Phys., 201, 2, 531-545 (2004) · Zbl 1061.76044
[16] Coppola, G.; Capuano, F.; Pirozzoli, S.; de Luca, L., J. Comput. Phys., 382, 86-104 (2019) · Zbl 1451.76081
[17] Modesti, D.; Pirozzoli, S., Comput. Fluids, 152, 14-23 (2017) · Zbl 1390.76492
[18] Economon, T.; Palacios, F.; Copeland, S.; Lukaczyk, T.; Alonso, J., AIAA J., 54, 3, 828-846 (2015)
[19] Jacobs, C.; Jammy, S.; Sandham, N., J. Comput. Sci., 18, 12-23 (2017)
[20] Legion Webpage (2020), https://legion.stanford.edu/ (Accessed 31 March 2020)
[21] Renzo, M. D.; Fu, L.; Urzay, J., Comput. Phys. Comm., Article 107262 pp. (2020)
[22] Witherden, F.; Farrington, A.; Vincent, P., Comput. Phys. Comm., 185, 11, 3028-3040 (2014) · Zbl 1348.65005
[23] Romero, J.; Crabill, J.; Watkins, J.; Witherden, F.; Jameson, A., Comput. Phys. Comm., 250, Article 107169 pp. (2020) · Zbl 07678519
[24] Pirozzoli, S., Annu. Rev. Fluid Mech., 43, 163-194 (2011) · Zbl 1299.76103
[25] Pirozzoli, S.; Bernardini, M., J. Fluid Mech., 688, 120-168 (2011) · Zbl 1241.76286
[26] Pirozzoli, S.; Bernardini, M., Phys. Fluids, 25, 2, Article 021704 pp. (2013)
[27] Pirozzoli, S.; Bernardini, M.; Grasso, F., J. Fluid Mech., 657, 361-393 (2010) · Zbl 1197.76079
[28] Pirozzoli, S.; Bernardini, M., AIAA J., 49, 1307-1312 (2011)
[29] Bernardini, M.; Pirozzoli, S.; Orlandi, P., Int. J. Heat Fluid Flow, 35, 45-51 (2012)
[30] Bernardini, M.; Pirozzoli, S.; Orlandi, P.; Lele, S., AIAA J., 52, 10, 2261-2269 (2014)
[31] Modesti, D.; Pirozzoli, S., Int. J. Heat Fluid Flow, 59, 33-49 (2016)
[32] Modesti, D.; Pirozzoli, S.; Grasso, F., Int. J. Heat Fluid Flow, 76, 130-140 (2019)
[33] Salvadore, F.; Bernardini, M.; Botti, M., J. Comput. Phys., 235, 129-142 (2013)
[34] Pirozzoli, S., J. Comput. Phys., 229, 19, 7180-7190 (2010) · Zbl 1426.76485
[35] Kennedy, C.; Gruber, A., J. Comput. Phys., 227, 3, 1676-1700 (2008) · Zbl 1290.76135
[36] Jiang, G.; Shu, C. W., J. Comput. Phys., 126, 202 (1996) · Zbl 0877.65065
[37] Ducros, F.; Ferrand, V.; Nicoud, F.; Weber, C.; Darracq, D.; Gacherieu, D.; Poinsot, T., J. Comput. Phys., 152, 2, 517-549 (1999) · Zbl 0955.76045
[38] Spalart, P.; Moser, R.; Rogers, M., J. Comput. Phys., 96, 2, 297-324 (1991) · Zbl 0726.76074
[39] Modesti, D.; Pirozzoli, S., J. Sci. Comput., 75, 308-331 (2018) · Zbl 1388.76211
[40] Simens, M.; Jimenez, J.; Hoyas, S.; Mizuno, Y., J. Comput. Phys., 228, 4218-4231 (2009) · Zbl 1273.76009
[41] Jimenez, J.; Hoyas, S.; Simens, M.; Mizuno, Y., J. Fluid Mech., 657, 335-360 (2010) · Zbl 1197.76063
[42] Poinsot, T.; Lele, S., J. Comput. Phys., 101, 1, 104-129 (1992) · Zbl 0766.76084
[43] Klein, M.; Sadiki, A.; Janicka, J., J. Comput. Phys., 186, 2, 652-665 (2003) · Zbl 1047.76522
[44] Touber, E.; Sandham, N. D., Theor. Comput. Fluid Dyn., 23, 79-107 (2009) · Zbl 1234.76033
[45] Kempf, A.; Wysocki, S.; Pettit, M., Comput. & Fluids, 60, 58-60 (2012) · Zbl 1365.76088
[46] Pirozzoli, S.; Bernardini, M., Supersonic turbulent boundary layers - DNS database (2011), http://newton.dma.uniroma1.it/dnsm2
[47] Smits, A.; Dussauge, J., Turbulent Shear Layers in Supersonic Flow (2006), American Institute of Physics: American Institute of Physics New York
[48] Musker, A., AIAA J., 17, 655-657 (1979) · Zbl 0397.76053
[49] Zhang, C.; Duan, L.; Choudhari, M., AIAA J., 56, 11, 4297-4311 (2018)
[50] Dupont, P.; Haddad, C.; Debiéve, J., J. Fluid Mech., 559, 255-277 (2006) · Zbl 1151.76565
[51] Volpiani, P.; Bernardini, M.; Larsson, J., Phys. Rev. Fluids, 5, 1, Article 014602 pp. (2020)
[52] OpenACC (2020), https://www.openacc.org/, (Accessed 16 January 2020)
[53] OpenMP (2020), https://www.openmp.org/. (Accessed 16 January 2020)
[54] CUDA (2020), https://developer.nvidia.com/cuda-zone, (Accessed 16 January 2020)
[55] CUDA FORTRAN (2020), https://developer.nvidia.com/cuda-fortran. (Accessed 16 January 2020)
[56] OpenCL (2020), https://www.khronos.org/opencl/. (Accessed 16 January 2020)
[57] HIP : C++ Heterogeneous-Compute Interface for Portability (2020), https://gpuopen.com/compute-product/hip-convert-cuda-to-portable-c-code/. (Accessed 16 January 2020)
[58] Moser, R. D.; Kim, J.; Mansour, N. N., Phys. Fluids, 11, 4, 943-945 (1999) · Zbl 1147.76463
[59] Kooij, G.; Botchev, M. A.; Frederix, E. M.; Geurts, B. J.; Horn, S.; Lohse, D.; van der Poel, E. P.; S., O.; Stevens, R. J.; Verzicco, R., Comput. Fluids, 166, 1-8 (2018) · Zbl 1390.76181
[60] Huynh, H. T., ((2007), American Institute of Aeronautics and Astronautics), 2007-4079
[61] Bernardini, M.; Pirozzoli, S.; Orlandi, P., J. Fluid Mech., 742, 171-191 (2014)
[62] Lee, M.; Moser, R., J. Fluid Mech., 774, 395-415 (2015)
[63] Buaria, D.; Sreenivasan, K., Dissipation range of the energy spectrum in high Reynolds number turbulence (2020), arXiv preprint arXiv:2004.06274
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.