×

Development of an unstructured mesh gyrokinetic particle-in-cell code for exascale fusion plasma simulations on GPUs. (English) Zbl 07723495

Summary: This paper presents XGCm, a new unstructured mesh gyrokinetic Particle-in-Cell (PIC) code for modeling fusion plasma. The physical models and aspects of the numerical methods employed are the same as those used in the X-point gyrokinetic code, XGC. The core difference is that XGCm builds on an unstructured mesh-centric infrastructure that supports a distributed mesh, making it scalable in both the number of mesh elements and number of particles. A second advantage of an unstructured mesh infrastructure is its performance is not degraded when generally graded or anisotropic meshes are used. The switch from a particle-centric data infrastructure to a distributed mesh-centric infrastructure required the introduction of new methods to execute core PIC operations and substantial modifications to a number of key algorithms from those used in XGC. First, we present the methods and algorithms used in the development of XGCm, which performs all key computing steps on the GPU accelerators. GPU accelerators are providing the main computational power of current generation U.S. Department of Energy supercomputers. Secondly, we perform code validation and test using the circular geometry cyclone base case and realistic DIII-D geometry case, respectively. The turbulence growth rate shows excellent agreement with existing XGC result in the first case, while ion temperature gradient turbulence growth is further demonstrated in the second case. Finally, we present weak scaling results, using up to full system (27,648 GPUs) of the Oak Ridge National Laboratory’s Summit supercomputer.

MSC:

76M28 Particle methods and lattice-gas methods
76X05 Ionized gas flow in electromagnetic fields; plasmic flow

Software:

XGCm; PUMIPic; PETSc; Kokkos
Full Text: DOI

References:

[1] Committee on the Key Goals and Innovation Needed for a. U. S. Fusion Pilot Plant, Bringing fusion to the U.S. grid (2021) (2021), The National Academies of Sciences, Engineering and Medicine, Tech. Rep.
[2] DÁzevedo, E.; Abbott, S.; Koskela, T.; Worley, P.; Ku, S.; Ethier, S.; Yoon, E.; Shephard, M.; Hager, R.; Lang, J.; Choi, J.; Podhorszki, N.; Klasky, S.; Parashar, M.; Chang, C. S., (Straatsma, T.; Antypas, K.; Williams, T., Exascale Scientific Applications: Scalability and Performance Portability (2017), CRC Press, Taylor & Francis Group), 529-551 · Zbl 1395.82002
[3] Khaziev, R.; Curreli, D., Comput. Phys. Commun., 229, 87-98 (2018)
[4] Madduri, K.; Ibrahim, K. Z.; Williams, S.; Im, E.; Ethier, S.; Shalf, J.; Oliker, L., (SC’11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (2011)), 1-12
[5] Wang, B.; Ethier, S.; Tang, W.; Ibrahim, K. Z.; Madduri, K.; Williams, S.; Oliker, L., Int. J. High Perform. Comput. Appl., 33, 1, 169-188 (2019)
[6] Tang, W.; Lin, Z., (Straatsma, T.; Antypas, K.; Williams, T., Exascale Scientific Applications: Scalability and Performance Portability (2017), CRC Press, Taylor & Francis Group), 507-528 · Zbl 1395.82004
[7] Diamond, G.; Smith, C. W.; Zhang, C.; Yoon, E.; Shephard, M. S., J. Parallel Distrib. Comput., 157, 1-12 (2021)
[8] Ku, S.; Chang, C. S.; Adams, M.; Cummings, J.; Hinton, F.; Keyes, D.; Klasky, S.; Lee, W.; Lin, Z.; Parker, S., J. Mod. Phys. Conf. Ser., 46, 87 (2006), IOP Publishing
[9] Ibanez, D. A., Conformal mesh adaptation on heterogeneous supercomputers (2016), Rensselaer Polytechnic Institute: Rensselaer Polytechnic Institute Troy, NY, Ph.D. thesis
[10] Ibanez, D., Omega_h GitHub repository (2016)
[11] Carter Edwards, H.; Trott, C. R.; Sunderland, D., J. Parallel Distrib. Comput., 74, 12, 3202-3216 (2014)
[12] Burckel, A.; Sauter, O.; Angioni, C.; Candy, J.; Fable, E.; Lapillonne, X., J. Phys. Conf. Ser., 260, 1, Article 012006 pp. (2010)
[13] Merlo, G.; Dominski, J.; Bhattacharjee, A.; Chang, C. S.; Jenko, F.; Ku, S.; Lanti, E.; Parker, S., Phys. Plasmas, 25, 6, Article 062308 pp. (2018)
[14] Lee, W. W., Phys. Fluids, 26, 556-562 (1983) · Zbl 0576.76120
[15] Lee, W. W., J. Comput. Phys., 72, 243-269 (1987) · Zbl 0619.76063
[16] Manuilskiy, I.; Lee, W. W., Phys. Plasmas, 7 (2000)
[17] Ku, S.; Chang, C. S.; Diamond, P. H., Nucl. Fusion, 49, 11, Article 115021 pp. (2009)
[18] Chang, C. S.; Ku, S.; Diamond, P. H.; Lin, Z.; Parker, S.; Hahm, T. S.; Samatova, N., Phys. Plasmas, 16, 5, Article 056108 pp. (2009)
[19] Ku, S.; Chang, C. S.; Hager, R.; Churchill, R. M.; Tynan, G. R.; Cziegler, I.; Greenwald, M.; Hughes, J.; Parker, S. E.; Adams, M. F.; D’Azevedo, E.; Worley, P., Phys. Plasmas, 25, 5, Article 056107 pp. (2018)
[20] Idomura, Y.; Tokuda, S.; Kishimoto, Y., J. Plasma Fusion Res. Ser., 6, 17-22 (2004)
[21] Nishimura, Y.; Lin, Z.; Lewandowski, J.; Ethier, S., J. Comput. Phys., 214, 2, 657-671 (2006) · Zbl 1088.76027
[22] Parker, S. E.; Lee, W. W., Phys. Fluids, B Plasma Phys., 5, 1, 77-86 (1993)
[23] Colonna, G., (Plasma Modeling: Methods and Applications (2016), IOP Publishing), 1-23 · doi:10.1088/978-0-7503-1200-4ch1
[24] Brizard, A. J.; Hahm, T. S., Rev. Mod. Phys., 79, 421-468 (2007) · Zbl 1205.76309
[25] Kleiber, R.; Hatzky, R.; Könies, A.; Mishchenko, A.; Sonnendrücker, E., Phys. Plasmas, 23, 3, Article 032501 pp. (2016)
[26] Hager, R.; Ku, S.; Sharma, A. Y.; Chang, C. S.; Churchill, R. M.; Scheinberg, A., Phys. Plasmas, 29, 11, Article 112308 pp. (2022)
[27] Hahm, T. S., Phys. Fluids, 31, 9, 2670-2673 (1988) · Zbl 0649.76067
[28] Adams, M. F.; Ku, S.; Worley, P.; D’Azevedo, E.; Cummings, J. C.; Chang, C. S., J. Phys. Conf. Ser., 180, Article 012036 pp. (2009)
[29] Dominski, J.; Ku, S.; Chang, C. S., Phys. Plasmas, 25, 5, Article 052304 pp. (2018)
[30] Lu, Z. X.; Lauber, P.; Hayward-Schneider, T.; Bottino, A.; Hoelzl, M., Phys. Plasmas, 26, 12, Article 122503 pp. (2019)
[31] Zhang, F.; Hager, R.; Ku, S.; Chang, C. S.; Jardin, S. C.; Ferraro, N. M.; Seol, E. S.; Yoon, E.; Shephard, M. S., Eng. Comput., 32, 2, 285-293 (2016)
[32] Kreutzer, M.; Hager, G.; Wellein, G.; Fehske, H.; Bishop, A., SIAM J. Sci. Comput., 36, 5, C401-C423 (2014) · Zbl 1307.65055
[33] Besta, M.; Marending, F.; Solomonik, E.; Hoefler, T., (Proceedings of the 31st IEEE International Parallel & Distributed Processing Symposium (IPDPS’17) (2017), IEEE), 32-41
[34] Slattery, S.; Reeve, S. T.; Junghans, C.; Lebrun-Grandié, D.; Bird, R.; Chen, G.; Fogerty, S.; Qiu, Y.; Schulz, S.; Scheinberg, A.; Isner, A.; Chong, K.; Moore, S.; Germann, T.; Belak, J.; Mniszewski Cabana, S., J. Open Sour. Softw., 7, 72, 4115 (2022)
[35] Balay, S.; Abhyankar, S.; Adams, M. F.; Benson, S.; Brown, J.; Brune, P.; Buschelman, K.; Constantinescu, E.; Dalcin, L.; Dener, A.; Eijkhout, V.; Gropp, W. D.; Hapla, V.; Isaac, T.; Jolivet, P.; Karpeev, D.; Kaushik, D.; Knepley, M. G.; Kong, F.; Kruger, S.; May, D. A.; McInnes, L. C.; Mills, R. T.; Mitchell, L.; Munson, T.; Roman, J. E.; Rupp, K.; Sanan, P.; Sarich, J.; Smith, B. F.; Zampini, S.; Zhang, H.; Zhang, H.; Zhang, J., PETSc/TAO Users Manual (2022), Argonne National Laboratory, Tech. Rep. ANL-21/39 - Revision 3.17
[36] Balay, S.; Gropp, W. D.; McInnes, L. C.; Smith, B. F., (Arge, E.; Bruaset, A. M.; Langtangen, H. P., Modern Software Tools in Scientific Computing (1997), Birkhäuser Press), 163-202
[37] Balay, S.; Abhyankar, S.; Adams, M. F.; Benson, S.; Brown, J.; Brune, P.; Buschelman, K.; Constantinescu, E. M.; Dalcin, L.; Dener, A.; Eijkhout, V.; Gropp, W. D.; Hapla, V.; Isaac, T.; Jolivet, P.; Karpeev, D.; Kaushik, D.; Knepley, M. G.; Kong, F.; Kruger, S.; May, D. A.; McInnes, L. C.; Mills, R. T.; Mitchell, L.; Munson, T.; Roman, J. E.; Rupp, K.; Sanan, P.; Sarich, J.; Smith, B. F.; Zampini, S.; Zhang, H.; Zhang, H.; Zhang, J.; page, PETSc Web (2022)
[38] Wesson, J., Tokamaks (2011), Oxford University Press: Oxford University Press Oxford, UK · Zbl 1111.82054
[39] Moritaka, T.; Hager, R.; Cole, M.; Lazerson, S.; Chang, C. S.; Ku, S.; Matsuoka, S.; Satake, S.; Ishiguro, S., Plasma, 2, 2, 179-200 (2019)
[40] Mills, R. T.; Adams, M. F.; Balay, S.; Brown, J.; Dener, A.; Knepley, M.; Kruger, S. E.; Morgan, H.; Munson, T.; Rupp, K.; Smith, B. F.; Zampini, S.; Zhang, H.; Zhang, J., Parallel Comput., 108, Article 102831 pp. (2021)
[41] Sozer, E.; Brehm, C.; Kiris, C. C., (52nd Aerospace Sciences Meeting, AIAA 2014-1440 (2014))
[42] Adam, J.; Gourdin Serveniere, A.; Langdon, A., J. Comput. Phys., 47, 2, 229-244 (1982) · Zbl 0493.76118
[43] Celes, W.; Paulino, G. H.; Espinha, R., Int. J. Numer. Methods Eng., 64, 11, 1529-1556 (2005) · Zbl 1122.74504
[44] Beall, M. W.; Shephard, M. S., Int. J. Numer. Methods Eng., 40, 9, 1573-1596 (1997)
[45] Candy, J.; Waltz, R. E., Phys. Rev. Lett., 91, Article 045001 pp. (2003)
[46] Vazhkudai, S. S.; de Supinski, B. R.; Bland, A. S.; Geist, A.; Sexton, J.; Kahle, J.; Zimmer, C. J.; Atchley, S.; Oral, S.; Maxwell, D. E.; Larrea, V. G.V.; Bertsch, A.; Goldstone, R.; Joubert, W.; Chambreau, C.; Appelhans, D.; Blackmore, R.; Casses, B.; Chochia, G.; Davison, G.; Ezell, M. A.; Gooding, T.; Gonsiorowski, E.; Grinberg, L.; Hanson, B.; Hartner, B.; Karlin, I.; Leininger, M. L.; Leverman, D.; Marroquin, C.; Moody, A.; Ohmacht, M.; Pankajakshan, R.; Pizzano, F.; Rogers, J. H.; Rosenburg, B.; Schmidt, D.; Shankar, M.; Wang, F.; Watson, P.; Walkup, B.; Weems, L. D.; Yin, J., (SC18: International Conference for High Performance Computing, Networking, Storage and Analysis (2018)), 661-672
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.