Hemelb: A high performance parallel lattice-Boltzmann code for large scale fluid flow in complex geometries. (English) Zbl 1196.76008
Summary: We describe a parallel lattice-Boltzmann code for efficient simulation of fluid flow in complex geometries. The lattice-Boltzmann model and the structure of the code are discussed. The fluid solver is highly optimized and the resulting computational core is very fast. Furthermore, communication is minimized and the novel topology-aware domain decomposition technique is shown to be very effective for large systems, allowing us to tune code execution in geographically distributed cross-site simulations. The benchmarks presented indicate that very high performance can be achieved.
MSC:
| 76-04 | Software, source code, etc. for problems pertaining to fluid mechanics |
| 65Y05 | Parallel numerical computation |
| 76M28 | Particle methods and lattice-gas methods |
Keywords:
lattice-Boltzmann; computer simulations; parallel computing; fluid dynamics; complex systemsReferences:
| [1] | Chen, S.; Doolen, D., Lattice Boltzmann method for fluid flows, Annu. Rev. Fluid Mech., 30, 329 (1998) · Zbl 1398.76180 |
| [2] | Taylor, C. A.; Draney, M. T.; Ku, J. P.; Parker, D.; Steele, B. N.; Wang, K. C.; Zarins, C. K., Predictive medicine: Computational techniques in therapeutic decision-making, Comput. Aided Surgery, 4, 231 (1999) |
| [3] | Botnar, R.; Rappitsch, G.; Sheidegger, M. B.; Liepsch, D.; Perktold, K.; Boesiger, P., Hemodynamics in the carotid artery bifurcation: a comparison between numerical simulations and in vitro MRI measurements, J. Biomech., 33, 137 (2000) |
| [4] | Mantle, M. D.; Sederman, A. J.; Gladden, L. F., Single- and two-phase flow in fixed-bed reactors: MRI flow visualisation and lattice-Boltzmann simulations, Chem. Eng. Sci., 56, 523 (2001) |
| [5] | Nakashima, Y.; Watanabe, Y., Estimate of transport properties of porous media by microfocus X-ray computed tomography and random walk simulation, Water Resour. Res., 38, 1272 (2002) |
| [6] | Goyen, M.; Ladd, M. E.; Debatin, J. F.; Barkhausen, J.; Truemmler, K. H.; Bosk, S.; Ruehm, S. G., Dynamic 3D MR angiography of the pulmonary arteries in under four seconds, J. Magn. Reson. Imag., 13, 372 (2001) |
| [7] | Thubrikar, M. J.; Robicsek, F., Pressure-induced arterial-wall stress and atherosclerosis, Annals of Thoracic Surgery, 59, 1594 (1995) |
| [8] | Giupponi, G.; Harting, J.; Coveney, P. V., Emergence of rheological properties in lattice Boltzmann simulations of gyroid mesophases, Europhys. Lett., 73, 533 (2006) |
| [9] | Imaeda, K.; Goodman, F., Analysis of non-linear pulsatile flow in arteries, J. Biomech., 13, 1007 (1980) |
| [10] | Succi, S., The Lattice Boltzmann Equation for Fluid Dynamics and Beyond (2001), Oxford University Press · Zbl 0990.76001 |
| [11] | Fang, H.; Wang, Z.; Lin, Z.; Liu, M., Lattice Boltzmann method for simulating the viscous flow in large distensible blood vessels, Phys. Rev. E, 65, 051925 (2002) |
| [12] | Hirabayashi, M.; Ohta, M.; Rufenacht, D. A.; Chopard, B., Characterization of flow reduction properties in an aneurysm due to a stent, Phys. Rev. E, 68, 021918 (2003) |
| [13] | Artoli, A. M.; Hoekstra, A. G.; Sloot, P. M.A., Mesoscopic simulations of systolic flow in the human abdominal aorta, J. Biomech., 39, 873 (2006) · Zbl 1099.76051 |
| [14] | Ouared, R.; Chopard, B., Lattice Boltzmann simulations of blood flow: Non-Newtonian rheology and clotting processes, J. Stat. Phys., 121, 209 (2005) · Zbl 1108.76089 |
| [15] | Noble, D. R.; Georgiadis, J. G.; Buchius, R. O., Comparison of accuracy and performance for lattice Boltzmann and finite difference simulations of steady viscous flow, Int. J. Num. Meth. Fluids, 23, 1 (2003) · Zbl 0866.76069 |
| [16] | Sankaranarayanan, K.; Kevrekidis, I. G.; Sundaresan, S.; Lu, J.; Tryggvason, G., A comparative study of lattice Boltzmann and front-tracking finite-difference methods for bubble simulations, Int. J. Multiphase Flow, 29, 109 (2003) · Zbl 1136.76630 |
| [17] | Breuer, M.; Bernsdorf, J.; Zeiser, T.; Durst, F., Accurate computations of the laminar flow past a square cylinder based on two different methods: Lattice-Boltzmann and finite-volume, Int. J. Heat Fluid Flow, 21, 186 (2000) |
| [18] | Bernsdorf, J.; Durst, F.; Schäfer, M., Comparison of cellular automata and finite volume techniques for simulation of incompressible flows in complex geometries, Int. J. Numer. Meth. Fluids, 29, 251 (1999) · Zbl 0940.76067 |
| [19] | S. Geller, M. Krafczyk, J. Tölke, S. Turek, J. Hron, Benchmark computations based on Lattice Boltzmann, Finite Element and Finite Volume Methods for laminar flows, Ergebnisberichte des Instituts für Angewandte Mathematik, Nummer, 2004, p. 274; S. Geller, M. Krafczyk, J. Tölke, S. Turek, J. Hron, Benchmark computations based on Lattice Boltzmann, Finite Element and Finite Volume Methods for laminar flows, Ergebnisberichte des Instituts für Angewandte Mathematik, Nummer, 2004, p. 274 · Zbl 1177.76313 |
| [20] | Kandhai, D.; Vidal, D. J.-E.; Hoekstra, A. G.; Hoefsloot, H.; Iedema, R.; Sloot, P. M.A., Lattice-Boltzmann and finite element simulations of fluid flow in a SMRX static mixer reactor, Int. J. Num. Meth. Fluids, 31, 1019 (1999) · Zbl 0964.76072 |
| [21] | Martínez, D. O.; Matthaeus, W. H.; Chen, S.; Montgomery, D. C., Comparison of spectral method and lattice Boltzmann simulations of two-dimensional hydrodynamics, Phys. Fluids, 6, 1285 (1994) · Zbl 0826.76069 |
| [22] | Chen, S.; Wang, Z.; Shan, X.; Doolen, G. D., Lattice Boltzmann computational fluid dynamics in three dimensions, Phys. Fluids, 68, 379 (1992) · Zbl 0925.76516 |
| [23] | Zhang, X. X.; Bengough, A. G.; Crawford, J. W.; Young, I. M., A lattice BGK model for advection and anisotropic dispersion equation, Adv. Water Resour., 25, 1 (2002) |
| [24] | Zhang, X. X.; Bengough, A. G.; Deeks, L. K.; Crawford, J. W.; Young, I. M., A novel three-dimensional lattice Boltzmann model for solute transport in variably saturated porous media, Water Resour. Res., 38, 1167 (2002) |
| [25] | Zhang, X. X.; Deeks, L. K.; Bengough, A. G.; Crawford, J. W.; Young, L. M., Determination of soil hydraulic conductivity with the lattice Boltzmann method and soil thin-section technique, J. Hydrology, 306, 59 (2005) |
| [26] | Donath, S.; Zeiser, T.; Hager, G.; Habich, J.; Wellein, G., Optimizing performance of the lattice Boltzmann method for complex structures on cache-based architectures, (Huelsemann, F.; Kowarschik, M.; Ruede, U., Frontiers in Simulation: Simulation Techniques—18th Symposium in Erlangen, September 2005 (ASIM) (2005), SCS Publishing House: SCS Publishing House Erlangen), 728 |
| [27] | Argentini, R.; Bakker, A. F.; Lowe, C. P., Efficiently using memory in lattice Boltzmann simulations, Future Gener. Comput. Syst., 20, 973 (2004) |
| [28] | Kowarschik, M.; Weiß, C., An Overview of Cache Optimization Techniques and Cache-Aware Numerical Algorithms, (Lecture Notes in Computer Science, vol. 2625 (2002), Springer), 213 · Zbl 1024.68768 |
| [29] | Pohl, T.; Kowarschik, M.; Wike, J.; Iglberger, K.; Rude, U., Optimization and profiling of the cache performance of parallel lattice Boltzmann codes in 2D and 3D, J. Parallel Process. Lett., 13, 549 (2003) |
| [30] | Schulz, M.; Krafczyk, M.; Tolke, J.; Rank, E., Parallelization strategies and efficiency of CFD computations in complex geometries using Lattice Boltzmann methods on high-performance computers, (Breuer, M.; Durst, F.; Zenger, C., High-Performance Scientific and Engineering Computing, Proceedings of the 3rd International FORTWIHR Conference on HPSEC. High-Performance Scientific and Engineering Computing, Proceedings of the 3rd International FORTWIHR Conference on HPSEC, Erlangen, 2001 (2002), Springer-Verlag: Springer-Verlag Berlin), 115 |
| [31] | Martys, N. S.; Hagedorn, J. G., Multiscale modeling of fluid transport in heterogeneous materials using discrete Boltzmann methods, Mater. Struct., 35, 650 (2002) |
| [32] | K. Iglberger, Cache optimizations for the lattice Boltzmann method in 3D, Bachelor Thesis, Lehrstuhl für Informatik 10 (Systemsimulation), Institut für Informatik, University of Erlangen-Nuremberg, Germany, 2003; K. Iglberger, Cache optimizations for the lattice Boltzmann method in 3D, Bachelor Thesis, Lehrstuhl für Informatik 10 (Systemsimulation), Institut für Informatik, University of Erlangen-Nuremberg, Germany, 2003 |
| [33] | Wellein, G.; Zeiser, T.; Hager, G.; Donath, S., On the single processor performance of simple lattice Boltzmann kernels, Comput. Fluids, 35, 910 (2006) · Zbl 1177.76335 |
| [34] | Velivelli, A. C.; Bryden, K. M., A cache-efficient implementation of the lattice Boltzmann method for the two-dimensional diffusion equation, Concurrency Comput.: Pract. Exper., 16, 1415 (2004) |
| [35] | S. Donath, On optimized implementations of the lattice Boltzmann method on contemporary high performance architectures, Lehrstuhl für Informatik 10, Institut für Informatik, University of Erlangen-Nuremberg, Bachelor thesis, 2004; S. Donath, On optimized implementations of the lattice Boltzmann method on contemporary high performance architectures, Lehrstuhl für Informatik 10, Institut für Informatik, University of Erlangen-Nuremberg, Bachelor thesis, 2004 |
| [36] | M. Kowarshik, Data locality optimizations for iterative numerical algorithms and cellular automata on hierarchical memory architectures, PhD thesis, Lehrstuhl für Informatik 10, Institut für Informatik, Universität Erlangen-Nüremberg, ISBN 3-936150-39-7, 2004; M. Kowarshik, Data locality optimizations for iterative numerical algorithms and cellular automata on hierarchical memory architectures, PhD thesis, Lehrstuhl für Informatik 10, Institut für Informatik, Universität Erlangen-Nüremberg, ISBN 3-936150-39-7, 2004 |
| [37] | S. Hausmann, Optimization and performance analysis of the lattice Boltzmann method on x86-64 based architectures, Bachelor Thesis, Erlangen, 2005; S. Hausmann, Optimization and performance analysis of the lattice Boltzmann method on x86-64 based architectures, Bachelor Thesis, Erlangen, 2005 |
| [38] | Velivelli, A. C.; Bryden, K. M., Optimizing performance on modern HPC systems: Learning from simple kernel benchmarks, (Proceedings of the 2nd Russian-German Advanced Research Workshop on Computational Science and High Performance Computing (2005), HLRS: HLRS Stuttgart) |
| [39] | J. Ni, J. Zhang, C.-L. Lin, S. Wang, Parallelization of a lattice Boltzmann method for lid-driven cavity flow, in: UI research booth, Supercomputing, 2003; J. Ni, J. Zhang, C.-L. Lin, S. Wang, Parallelization of a lattice Boltzmann method for lid-driven cavity flow, in: UI research booth, Supercomputing, 2003 |
| [40] | Desplat, J.-C.; Pagonabarraga, I.; Bladon, P., LUDWIG: A parallel lattice-Boltzmann code for complex fluids, Comput. Phys. Comm., 134, 273 (2001) · Zbl 1032.76055 |
| [41] | T. Pohl, F. Deserno, N. Thurey, U. Rude, P. Lammers, G. Wellein, T. Zeiser, Performance evaluation of parallel large-scale lattice Boltzmann applications on three supercomputing architectures, in: Proceedings of the ACM/IEEE SC2004 Conference, Supercomputing, 2004, p. 21; T. Pohl, F. Deserno, N. Thurey, U. Rude, P. Lammers, G. Wellein, T. Zeiser, Performance evaluation of parallel large-scale lattice Boltzmann applications on three supercomputing architectures, in: Proceedings of the ACM/IEEE SC2004 Conference, Supercomputing, 2004, p. 21 |
| [42] | Giovanni, P.; Massaioli, F.; Succi, S., High-resolution lattice Boltzmann computing on the IBM SP1 scalable parallel computer, Comput. Phys., 8, 705 (1994) |
| [43] | Amati, G.; Succi, S.; Piva, R., Massively parallel lattice-Boltzmann simulation of turbulent channel flow, Int. J. Mod. Phys. C, 8, 869 (1997) |
| [44] | Wellein, G.; Lammers, P.; Hager, G.; Donath, S.; Zeiser, T., Towards optimal performance for lattice Boltzmann applications on terascale computers, (Deane, A.; etal., Parallel Computational Fluid Dynamics—Theory and Applications. Proceedings of the Parallel CFD 2005 Conference. Parallel Computational Fluid Dynamics—Theory and Applications. Proceedings of the Parallel CFD 2005 Conference, College Park, MD, USA, May 24-27, 2005 (2006), Elsevier), 3 |
| [45] | Karypis, G.; Kumar, V., Parallel multilevel \(k\)-way partitioning scheme for irregular graphs, SIAM Rev., 41, 279 (1999) · Zbl 0918.68073 |
| [46] | Karypis, G.; Kumar, V., Multilevel \(k\)-way partitioning scheme for irregular graphs, J. Parallel Distrib. Comput., 48, 96 (1998) |
| [47] | Karypis, G.; Schloegel, K.; Kumar, V., Parallel graph partitioning and sparse matrix ordering library (2003) |
| [48] | Dupuis, A.; Chopard, B., Lattice gas: An efficient and reusable parallel library based on a graph partitioning technique, (HPCN99. HPCN99, Lecture Notes in Comput. Sci., vol. 1529 (1999), Springer), 319 |
| [49] | Pan, C.; Prins, J. F.; Miller, C. T., A high-performance lattice Boltzmann implementation to model flow in porous media, Comput. Phys. Comm., 158, 89 (2004) · Zbl 1196.76069 |
| [50] | Kumar, V.; Grama, A.; Karypis, G., Parallel Computing (2004), The Benjamin/Cummings Publishing Company Inc.: The Benjamin/Cummings Publishing Company Inc. California |
| [51] | Wang, J.; Zhang, X.; Bengough, A. G.; Crawford, J. W., Domain-decomposition method for parallel lattice Boltzmann simulation of incompressible flow in porous media, Phys. Rev. E, 72, 016706 (2005) |
| [52] | Sagan, H., Space-Filling Curves (1994), Springer-Verlag · Zbl 0806.01019 |
| [53] | M.J. Aftosmis, M.J. Berger, G. Adomavicius, A parallel multilevel method for adaptively refined Cartesian grids with embedded boundaries, in: 38th AIAA Aerospace Sciences Meeting and Exhibit, Reno NV, Jan. 2000; M.J. Aftosmis, M.J. Berger, G. Adomavicius, A parallel multilevel method for adaptively refined Cartesian grids with embedded boundaries, in: 38th AIAA Aerospace Sciences Meeting and Exhibit, Reno NV, Jan. 2000 |
| [54] | M.J. Aftosmis, M.J. Berger, S.M. Murman, Applications of space-filling curves to Cartesian methods for CFD, in: 42nd AIAA Aerospace Sciences Meeting and Exhibit, Jan. 2004; M.J. Aftosmis, M.J. Berger, S.M. Murman, Applications of space-filling curves to Cartesian methods for CFD, in: 42nd AIAA Aerospace Sciences Meeting and Exhibit, Jan. 2004 |
| [55] | Qian, Y.; d’Humières, D.; Lallemand, P., Recovery of Navier-Stokes equations using a lattice-gas Boltzmann method, Europhys. Lett., 17, 479 (1992) · Zbl 1116.76419 |
| [56] | Zou, Q.; Hou, S.; Chen, S.; Doolen, G. D., An improved incompressible lattice Boltzmann model for time independent flows, J. Stat. Phys., 81, 35 (1995) · Zbl 1106.82366 |
| [57] | Bhatnagar, P. L.; Gross, E. P.; Krook, M., A model for collision processes in gases. I. Small amplitude processes in charged and neutral one-component systems, Phys. Rev., 94, 511 (1954) · Zbl 0055.23609 |
| [58] | d’Humières, D.; Lallemand, P., Numerical simulations of hydrodynamics with lattice gas automata in two dimensions, Complex Systems, 1, 599 (1987) |
| [59] | Noble, D. R.; Georgiadis, J. G.; Buckius, R. O., Direct Assessment of lattice Boltzmann hydrodynamics and boundary conditions for recirculating flows, J. Stat. Phys., 81, 17 (1995) · Zbl 1106.82355 |
| [60] | Mei, R.; Shyy, W.; Yu, D., Lattice Boltzmann method for 3-D flows with curved boundary, J. Comput. Phys., 161, 680 (2000) · Zbl 0980.76064 |
| [61] | Guo, Z.; Zheng, C.; Sci, B., An extrapolation method for boundary conditions in lattice Boltzmann method, Phys. Fluids, 14, 2007 (2002) · Zbl 1185.76156 |
| [62] | Filippova, O.; Hänel, D., Grid refinement for lattice-BGK models, J. Comput. Phys., 147, 219 (1998) · Zbl 0917.76061 |
| [63] | Maier, R. S.; Bernard, R. S.; Grunau, D. W., Boundary conditions for the lattice Boltzmann method, Phys. Fluids, 8, 1788 (1996) · Zbl 1027.76632 |
| [64] | Junk, M.; Yang, Z., Asymptotic analysis of lattice Boltzmann boundary conditions, J. Stat. Phys., 121, 3 (2005) · Zbl 1107.82049 |
| [65] | Zhao-Li, G.; Chu-Guang, Z.; Bao-Chang, S., Non-equilibrium extrapolation method for velocity and pressure boundary conditions in the lattice Boltzmann method, Chinese Phys., 11, 366 (2002) |
| [66] | Zou, Q.; He, X., On pressure and velocity boundary conditions for the lattice Boltzmann BGK model, Phys. Fluids, 9, 1591 (1997) · Zbl 1185.76873 |
| [67] | White, F. M., Viscous Fluid Flow (2001), McGraw-Hill: McGraw-Hill New York, p. 123 |
| [68] | Filippova, O.; Schwade, B.; Hänel, D., Multiscale lattice Boltzmann schemes for low Mach number flow, Phil. Trans. Roy. Soc. London A, 360, 467 (2002) · Zbl 1001.76082 |
| [69] | N. Thürey, U. Rude, Optimized free surface fluids on adaptive grids with the lattice Boltzmann method, in: SIGGRAPH 2005 Poster; N. Thürey, U. Rude, Optimized free surface fluids on adaptive grids with the lattice Boltzmann method, in: SIGGRAPH 2005 Poster |
| [70] | Tölke, J.; Freudinger, S.; Krafczyk, M., An adaptive scheme using hierarchical grids for lattice Boltzmann multi-phase flow simulations, Comput. Fluids, 35, 820 (2006) · Zbl 1177.76332 |
| [71] | Morton, G. M., A Computer Oriented Geodetic Data Base and a New Technique in File Sequencing (1966), IBM Ltd: IBM Ltd Ottawa, Ontario |
| [72] | Kandhai, D.; Koponen, A.; Hoekstra, A. G.; Kataja, M.; Timonen, J.; Sloot, P. M.A., Lattice-Boltzmann hydrodynamics on parallel systems, Comput. Phys. Comm., 111, 14 (1998) · Zbl 0942.76062 |
| [73] | George, A.; Liu, J. W.-H., Computer Solution of Large Sparse Positive Definite Systems (1981), Prentice-Hall: Prentice-Hall Englewood Cliffs · Zbl 0516.65010 |
| [74] | T. Goehring, Y. Saad, Heuristic algorithms for automatic graph partitioning, Technical Report UMSI- 94-29, University of Minnesota Supercomputing Institute, 1994; T. Goehring, Y. Saad, Heuristic algorithms for automatic graph partitioning, Technical Report UMSI- 94-29, University of Minnesota Supercomputing Institute, 1994 |
| [75] | P. Ciarlet Jr., F. Lamour, On the validity of a front-oriented approach to partitioning large sparse graphs with a connectivity constraint, Technical Report 94-37, Computer Science Department, UCLA, Los Angeles, CA, 1994; P. Ciarlet Jr., F. Lamour, On the validity of a front-oriented approach to partitioning large sparse graphs with a connectivity constraint, Technical Report 94-37, Computer Science Department, UCLA, Los Angeles, CA, 1994 · Zbl 0861.90119 |
| [76] | X. Wu, V. Taylor, S. Garrick, D. Yu, J. Richard, Performance analysis, modeling and prediction of a parallel multiblock lattice Boltzmann application using prophesy system, in: IEEE International Conference on Cluster Computing, Barcelona, Spain, 2006, p. 25; X. Wu, V. Taylor, S. Garrick, D. Yu, J. Richard, Performance analysis, modeling and prediction of a parallel multiblock lattice Boltzmann application using prophesy system, in: IEEE International Conference on Cluster Computing, Barcelona, Spain, 2006, p. 25 |
| [77] | Dong, S.; Karniadakis, G. E.; Karonis, N. T., Cross-site computations on the TeraGrid, Comput. Sci. Engrg., 7, 14 (2005) |
| [78] | Boghosian, B. M.; Coveney, P. V.; Dong, S.; Finn, L. I.; Jha, S.; Karniadakis, G. E.; Karonis, N. T., Nektar, SPICE and Vortonics: Using federated grids for large scale scientific applications, Cluster Comput., 10, 351 (2007) |
| [79] | S. Jha, P.V. Coveney, M. Harvey, SPICE: Simulated Pore Interactive Computing Environment, in: Supercomputing, 2005, Proceedings of the ACM/IEEE SC 2005 Conference; S. Jha, P.V. Coveney, M. Harvey, SPICE: Simulated Pore Interactive Computing Environment, in: Supercomputing, 2005, Proceedings of the ACM/IEEE SC 2005 Conference |
| [80] | Boghosian, B. M.; Finn, L. I.; Coveney, P. V., Moving the data to the computation: multi-site distributed parallel computation (2006) |
| [84] | Chin, J.; Harting, J.; Jha, S.; Coveney, P. V.; Porter, A. R.; Pickles, S. M., Steering in computational science: mesoscale modelling and simulation, Contemp. Phys., 44, 417 (2003) |
| [85] | Tendler, J. M.; Dodson, S.; Fields, S.; Le, H.; Sinharoy, B., IBM, POWER4 System Microarchitecture, IBM Server Group (2001) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
