EMPIRE-PIC: a performance portable unstructured particle-in-cell code. (English) Zbl 1494.78004
Summary: In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in two- and three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems – a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel’s Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs.
MSC:
| 78-04 | Software, source code, etc. for problems pertaining to optics and electromagnetic theory |
| 78M10 | Finite element, Galerkin and related methods applied to problems in optics and electromagnetic theory |
| 78M20 | Finite difference methods applied to problems in optics and electromagnetic theory |
| 78M99 | Basic methods for problems in optics and electromagnetic theory |
| 78A25 | Electromagnetic theory (general) |
| 78A30 | Electro- and magnetostatics |
| 78A35 | Motion of charged particles |
| 78A60 | Lasers, masers, optical bistability, nonlinear optics |
| 65M60 | Finite element, Rayleigh-Ritz and Galerkin methods for initial value and initial-boundary value problems involving PDEs |
| 65M06 | Finite difference methods for initial value and initial-boundary value problems involving PDEs |
| 65N30 | Finite element, Rayleigh-Ritz and Galerkin methods for boundary value problems involving PDEs |
| 65M75 | Probabilistic methods, particle methods, etc. for initial value and initial-boundary value problems involving PDEs |
| 65M25 | Numerical aspects of the method of characteristics for initial value and initial-boundary value problems involving PDEs |
| 65Y05 | Parallel numerical computation |
| 65Y10 | Numerical algorithms for specific classes of architectures |
| 76X05 | Ionized gas flow in electromagnetic fields; plasmic flow |
| 76M28 | Particle methods and lattice-gas methods |
| 35Q60 | PDEs in connection with optics and electromagnetic theory |
Software:
SPIS; Amesos2; OpenACC; OpenCL; OSIRIS; Trilinos; Tempus; Belos; Isorropia; EMPIRE-PIC; Cabana; PSC; Kokkos; PIConGPU; CUDA; GitHub; SYCL; MueLu; oneAPIReferences:
| [1] | Tempus Trilinos package. https://github.com/trilinos/Trilinos/tree/master/ packages/tempus. |
| [2] | OpenCL specification version 2.0. Technical report, Jul 2015. https://www.khronos.org/ registry/OpenCL/specs/opencl-2.0.pdf. |
| [3] | CUDA C++ programming guide. Technical report, NVIDIA, Nov 2019. https://docs. nvidia.com/pdf/CUDA_C_Programming_Guide.pdf. |
| [4] | oneAPI programming model. Technical report, Intel Corporation, Nov 2019. https://www. oneapi.com/. |
| [5] | Top500, 2020. [Online; accessed 30-June-2020]. |
| [6] | M F Adams, S Ethier, and N Wichmann. Performance of particle in cell methods on highly concurrent computational architectures. In Journal of Physics: Conference Series, volume 78, page 012001. IOP Publishing, 2007. |
| [7] | T D Arber, K Bennett, C S Brady, A Lawrence-Douglas, M G Ramsay, N J Sircombe, P Gillies, R G Evans, H Schmitz, A R Bell, and C P Ridgers. Contemporary particle-in-cell approach to laser-plasma modelling. Plasma Physics and Controlled Fusion, 57(11):113001, 2015. |
| [8] | Eric Bavier, Mark Hoemmen, Sivasankaran Rajamanickam, and Heidi Thornquist. Amesos2 and Belos: Direct and Iterative Solvers for Large Sparse Linear Systems. Scientific Program-ming, 20(3):241-255, July 2012. |
| [9] | Donald J Becker, Thomas Sterling, Daniel Savarese, John E Dorband, Udaya A Ranawak, and Charles V Packer. BEOWULF: A Parallel Workstation for Scientific Computation. In Proceedings of the International Conference on Parallel Processing (ICPP’95), pages 11-14, 1995. |
| [10] | Luc Berger-Vergiat, Christian A. Glusa, Jonathan J. Hu, Matthias Mayr, Andrey Prokopenko, Christopher M. Siefert, Raymond S. Tuminaro, and Tobias A. Wiesner. MueLu user’s guide. Technical Report SAND2019-0537, Sandia National Laboratories, 2019. |
| [11] | R F Bird, S J Pennycook, S A Wright, and S A Jarvis. Towards a portable and future-proof particle-in-cell plasma physics code. In 1st International Workshop on OpenCL (IWOCL 13), May 2013. |
| [12] | Robert F Bird, Patrick Gillies, Michael R Bareford, John Andy Herdman, and Stephen A. Jarvis. Performance optimisation of inertial confinement fusion codes using mini-applications. The International Journal of High Performance Computing Applications, 32(4):570-581, 2018. |
| [13] | C. K. Birdsall and A. B. Langdon. Plasma Physics via Computer Simulation. Plasma Physics Series. Institute of Physics Publishing, Bristol BS1 6BE, UK, 1991. |
| [14] | Charles K Birdsall and Dieter Fuss. Clouds-in-clouds, clouds-in-cells physics for many-body plasma simulation. Journal of Computational Physics, 3(4):494 -511, 1969. · Zbl 0938.76088 |
| [15] | Joseph D Blahovec, Lester A Bowers, John W Luginsland, Gerald E Sasser, and John J Wa-trous. 3-d icepic simulations of the relativistic klystron oscillator. IEEE transactions on plasma science, 28(3):821-829, 2000. |
| [16] | OpenMP Architecture Review Board. OpenMP application programming interface ver-sion 5.0. Technical report, Nov 2018. https://www.openmp.org/wp-content/uploads/ OpenMP-API-Specification-5.0.pdf. |
| [17] | P. Bochev, H. C. Edwards, R. C. Kirby, K. Peterson, and D. Ridzal. Solving PDEs with In-trepid. Scientific Programming, 20(2):151-180, 2012. |
| [18] | P. Bochev and J. Hyman. Principles of mimetic discretizations of differential operators. Com-patible spatial discretizations, pages 89-119, 2006. · Zbl 1110.65103 |
| [19] | Pavel B Bochev, Jonathan J Hu, Christopher M Siefert, and Raymond S Tuminaro. An al-gebraic multigrid approach based on a compatible gauge reformulation of Maxwell’s equa-tions. SIAM Journal on Scientific Computing, 31(1):557-583, 2008. · Zbl 1184.78007 |
| [20] | E. G. Boman, U. V. Catalyurek, C. Chevalier, and K. D. Devine. The Zoltan and Isorropia Par-allel Toolkits for Combinatorial Scientific Computing: Partitioning, Ordering, and Coloring. Scientific Programming, 20(2):129-150, 2012. |
| [21] | J Boris. Relativistic Plasma Simulation: Optimization of a Hybrid Code. In Proceedings of the Fourth Conference on Numerical Simulation of Plasmas, pages 3-68, Naval Research Laboratory, Washington, D.C, July 1971. |
| [22] | A. Bossavit. A rationale for ’edge-elements’ in 3-d fields computations. IEEE Transactions on Magnetics, 24(1):74-79, 1988. |
| [23] | K. J. Bowers, B. J. Albright, B. Bergen, L. Yin, K. J. Barker, and D. J. Kerbyson. 0.374 pflop/s trillion-particle kinetic modeling of laser plasma interaction on roadrunner. In Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, SC ’08, pages 63:1-63:11, Piscataway, NJ, USA, 2008. IEEE Press. |
| [24] | K. J. Bowers, B. J. Albright, L. Yin, B. Bergen, and T. J. T. Kwan. Ultrahigh performance three-dimensional electromagnetic relativistic kinetic plasma simulation. Physics of Plasmas, 15(5):055703, 2008. |
| [25] | Franco Brezzi and Michel Fortin. Mixed and hybrid finite element methods, volume 15. Springer Science & Business Media, 2012. · Zbl 0788.73002 |
| [26] | Dominic A.S. Brown, Matthew T. Bettencourt, Steven A. Wright, Satheesh Maheswaran, John P. Jones, and Stephen A. Jarvis. Higher-Order Particle Representation for Particle-in-Cell Simulations. Journal of Computational Physics, 435:110255, June 2021. · Zbl 07503730 |
| [27] | Dominic A.S. Brown, Steven A. Wright, and Stephen A. Jarvis. Performance of a Second Or-der Electrostatic Particle-in-Cell Algorithm on Modern Many-Core Architectures. Electronic Notes in Theoretical Computer Science, 340:67-84, October 2018. |
| [28] | Heiko Burau, Renée Widera, Wolfgang Honig, Guido Juckeland, Alexander Debus, Thomas Kluge, Ulrich Schramm, Tomas E Cowan, Roland Sauerbrey, and Michael Bussmann. PICon-GPU: a fully relativistic particle-in-cell code for a GPU cluster. IEEE Transactions on Plasma Science, 38(10):2831-2839, 2010. |
| [29] | Francis F Chen. Introduction to Plasma Physics and Controlled Fusion. Springer, third edition, 2016. |
| [30] | G. Chen, L. Chacón, and D.C. Barnes. An efficient mixed-precision, hybrid CPU-GPU im-plementation of a nonlinearly implicit one-dimensional particle-in-cell algorithm. Journal of Computational Physics, 231(16):5374-5388, 2012. |
| [31] | OpenACC Language Committee et al. OpenACC application programming interface ver-sion 2.6. Technical report, Nov 2017. https://www.openacc.org/sites/default/files/ inline-files/OpenACC.2.6.final.pdf. |
| [32] | John Dawson. Onedimensional plasma model. The Physics of Fluids, 5(4):445-459, 1962. · Zbl 0116.23001 |
| [33] | John M. Dawson. Particle Simulation of Plasmas. Reviews of Modern Physics, 55:403-447, Apr 1983. |
| [34] | Viktor K. Decyk and Tajendra V. Singh. Adaptable particle-in-cell algorithms for graphical processing units. Computer Physics Communications, 182(3):641-648, 2011. |
| [35] | Thomas H Dupree. Kinetic Theory of Plasma and the Electromagnetic Field. Physics of Fluids (1958-1988), 6(12):1714-1729, 1963. · Zbl 0127.22503 |
| [36] | H. Carter Edwards, Christian R. Trott, and Daniel Sunderland. Kokkos: Enabling Manycore Performance Portability Through Polymorphic Memory Access Patterns. Journal of Parallel and Distributed Computing, 74(12):3202-3216, 2014. Domain-Specific Languages and High-Level Frameworks for High-Performance Computing. |
| [37] | R. A. Fonseca, L. O. Silva, F. S. Tsung, V. K. Decyk, W. Lu, C. Ren, W. B. Mori, S. Deng, S. Lee, T. Katsouleas, and J. C. Adam. OSIRIS: A three-dimensional, fully relativistic particle in cell code for modeling plasma based accelerators. In Peter M. A. Sloot, Alfons G. Hoekstra, C. J. Kenneth Tan, and Jack J. Dongarra, editors, Computational Science ICCS 2002, pages 342-351, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg. · Zbl 1053.81100 |
| [38] | Gregory Fridman, Gary Friedman, Alexander Gutsol, Anatoly B. Shekhter, Victor N. Vasilets, and Alexander Fridman. Applied plasma medicine. Plasma Processes and Polymers, 5(6):503-533, 2008. |
| [39] | Alex Friedman. A second-order implicit particle mover with adjustable damping. Journal of Computational Physics, 90(2):292-312, October 1990. · Zbl 0701.76121 |
| [40] | Kai Germaschewski, William Fox, Stephen Abbott, Narges Ahmadi, Kristofor Maynard, Liang Wang, Hartmut Ruhl, and Amitava Bhattacharjee. The plasma simulation code: A modern carticle-in-cell code with patch-based load-balancing. Journal of Computational Physics, 318:305-326, 2016. · Zbl 1349.76917 |
| [41] | Philip M Gresho and Robert L Lee. Don’t suppress the wiggles-they’re telling you some-thing! Computers & Fluids, 9(2):223-253, June 1981. |
| [42] | Khronos OpenCL Working Group et al. SYCL specification version 1.2.1. Technical report, Nov 2019. https://www.khronos.org/registry/SYCL/specs/sycl-1.2.1.pdf. |
| [43] | Michael A. Heroux, Roscoe A. Bartlett, Vicki E. Howle, Robert J. Hoekstra, Jonathan J. Hu, Tamara G. Kolda, Richard B. Lehoucq, Kevin R. Long, Roger P. Pawlowski, Eric T. Phipps, Andrew G. Salinger, Heidi K. Thornquist, Ray S. Tuminaro, James M. Willenbring, Alan Williams, and Kendall S. Stanley. An Overview of the Trilinos Project. ACM Transactions on Mathematical Software, 31(3):397-423, 2005. · Zbl 1136.65354 |
| [44] | R. Hiptmair. Finite elements in computational electromagnetism. Acta Numerica, 11:237-339, 2002. · Zbl 1123.78320 |
| [45] | R. W. Hockney. Computer experiment of anomalous diffusion. The Physics of Fluids, 9(9):1826-1835, 1966. |
| [46] | Roger W Hockney and James W Eastwood. Computer simulation using particles. crc Press, 1988. · Zbl 0662.76002 |
| [47] | Rich Hornung, Holger Jones, Jeff Keasler, Rob Neely, Olga Pearce, Si Hammond, Christian Trott, Paul Lin, Courtenay Vaughan, Jeanine Cook, Rob Hoekstra, Ben Bergen, Josh Payne, and Geoff Womeldorff. ASC tri-lab co-design level 2 milestone report 2015. Technical report, Lawrence Livermore National Laboratory, Sep 2015. |
| [48] | Jianming Jin. The Finite Element Method in Electromagnetics. Wiley-IEEE Press, 3rd edition, 2014. · Zbl 1001.78001 |
| [49] | Y. L. Klimontovich. The Statistical Theory of Non-Equilibrium Processes in a Plasma: International Series of Monographs in Natural Philosophy, volume 9. Elsevier, 2013. |
| [50] | S. Ku, R. Hager, C.S. Chang, J.M. Kwon, and S.E. Parker. A new hybrid-lagrangian numerical scheme for gyrokinetic simulation of tokamak edge plasma. Journal of Computational Physics, 315:467 -475, 2016. · Zbl 1349.82148 |
| [51] | A. B. Langdon and C. K. Birdsall. Theory of plasma simulation using finitesize particles. The Physics of Fluids, 13(8):2115-2122, 1970. |
| [52] | A Bruce Langdon. On enforcing gauss’ law in electromagnetic particle-in-cell codes. Com-puter Physics Communications, 70(3):447-450, 1992. |
| [53] | R. Marchand. Ptetra, a tool to simulate low orbit satellite-plasma interaction. IEEE Transac-tions on Plasma Science, 40(2):217-229, Feb 2012. |
| [54] | Barry Marder. A method for incorporating gauss’ law into electromagnetic pic codes. Journal of Computational Physics, 68(1):48-55, 1987. · Zbl 0603.65079 |
| [55] | Collin S Meierbachtol, Andrew D Greenwood, John P Verboncoeur, and Balasubramaniam Shanker. Conformal electromagnetic particle in cell: A review. IEEE Transactions on Plasma Science, 43(11):3778-3793, 2015. |
| [56] | Sean T Miller, Eric C Cyr, John N Shadid, Richard Michael Jack Kramer, Edward Geoffrey Phillips, Sidafa Conde, and Roger P Pawlowski. IMEX and exact sequence discretization of the multi-fluid plasma model. Journal of Computational Physics, 397:108806, 2019. · Zbl 1453.76081 |
| [57] | Jean-Claude Nédélec. Mixed finite elements in R 3 . Numerische Mathematik, 35(3):315-341, 1980. · Zbl 0419.65069 |
| [58] | D. R. Nicholson. Introduction to Plasma Theory. Krieger Publishing Company, Malabar, Florida, 1992. |
| [59] | S. J. Pennycook, J. D. Sewall, and V. W. Lee. Implications of a metric for performance porta-bility. Future Generation Computer Systems, pages 947-958, 2017. |
| [60] | Martin Campos Pinto, Marie Mounier, and Eric Sonnendrücker. Handling the divergence constraints in maxwell and vlasov-maxwell simulations. Applied Mathematics and Computa-tion, 272:403-419, 2016. · Zbl 1410.82027 |
| [61] | Timothy D Pointon. Second-order, exact charge conservation for electromagnetic particle-in-cell simulation in complex geometry. Computer Physics Communications, 179(8):535-544, 2008. · Zbl 1197.82015 |
| [62] | F. Rapetti and A. Bossavit. Whitney forms of higher degree. SIAM Journal on Numerical Analysis, 47(3):2369-2386, 2009. · Zbl 1195.78063 |
| [63] | Mario A. Riquelme, Eliot Quataert, and Daniel Verscharen. Particle-in-cell simulations of continuously driven mirror and ion cyclotron instabilities in high beta astrophysical and heliospheric plasmas. The Astrophysical Journal, 800(1):27, Feb 2015. |
| [64] | J. Roussel, F. Rogier, G. Dufour, J. Mateo-Velez, J. Forest, A. Hilgers, D. Rodgers, L. Girard, and D. Payan. Spis open-source code: Methods, capabilities, achievements, and prospects. IEEE Transactions on Plasma Science, 36(5):2360-2368, Oct 2008. |
| [65] | Aaron Scheinberg, Guangye Chen, Stephane Ethier, Stuart Slattery, Robert F. Bird, Pat Wor-ley, and Choong-Seock Chang. Kokkos and fortran in the exascale computing project plasma physics code xgc. 2019. |
| [66] | Stuart Slattery, Christoph Junghans, Damien Lebrun-Grandie, Shane Fogerty, Robert F. Bird, Sam Reeve, Guangye Chen, Rene Halver, Aaron Scheinberg, Cameron Smith, and Evan Weinberg. Ecp-copa/cabana: Version 0.3.0, May 2020. |
| [67] | W. Tang, B. Wang, S. Ethier, G. Kwasniewski, T. Hoefler, K. Z. Ibrahim, K. Madduri, S. Williams, L. Oliker, C. Rosales-Fernandez, and T. Williams. Extreme scale plasma tur-bulence simulations on top supercomputers worldwide. In SC ’16: Proceedings of the Inter-national Conference for High Performance Computing, Networking, Storage and Analysis, pages 502-513, Nov 2016. |
| [68] | J.-L. Vay, A. Almgren, J. Bell, L. Ge, D. P. Grote, M. Hogan, O. Kononenko, R. Lehe, A. My-ers, C. Ng, J. Park, R. Ryne, O. Shapoval, M. Thévenet, and W. Zhang. Warp-x: A new ex-ascale computing platform for beam-plasma simulations. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 909:476-479, 2018. |
| [69] | J.-L. Vay, P. Colella, J. W. Kwan, P. McCorquodale, D. B. Serafini, A. Friedman, D. P. Grote, G. Westenskow, J.-C. Adam, A. Héron, and I. Haber. Application of adaptive mesh refine-ment to particle-in-cell simulations of plasmas and beams. Physics of Plasmas, 11(5):2928-2934, 2004. |
| [70] | John Villasenor and Oscar Buneman. Rigorous charge conservation for local electromagnetic field solvers. Computer Physics Communications, 69(2):306-316, 1992. |
| [71] | B. Wang, S. Ethier, W. Tang, T. Williams, K. Z. Ibrahim, K. Madduri, S. Williams, and L. Oliker. Kinetic turbulence simulations at extreme scale on leadership-class systems. In SC ’13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, pages 1-12, Nov 2013. |
| [72] | E. Wang, S. Wu, Q. Zhang, J. Liu, W. Zhang, Z. Lin, Y. Lu, Y. Du, and X. Zhu. The Gyrokinetic Particle Simulation of Fusion Plasmas on Tianhe-2 Supercomputer. In 2016 7th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems (ScalA), pages 25-32, Nov 2016. |
| [73] | Kane S. Yee. Numerical Solution of Initial Boundary Value Problems Involving Maxwell’s Equations in Isotropic Media. IEEE Transactions on Antennas and Propagation, pages 302-307, 1966. · Zbl 1155.78304 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
