Physics-informed distribution transformers via molecular dynamics and deep neural networks. (English) Zbl 07578917
Summary: Generating quasirandom points with high uniformity is a fundamental task in many fields. Existing number-theoretic approaches produce evenly distributed points in \([0, 1]^d\) in asymptotic sense but may not yield a good distribution for a given set size. It is also difficult to extend those techniques to other geometries like a disk or a manifold. In this paper, we present a novel physics-informed framework to transform a given set of points into a distribution with better uniformity. We model each point as a particle and assign the system with a potential energy. Upon minimizing the energy, the uniformity of distribution can be improved correspondingly. Two kinds of schemes are introduced: one based on molecular dynamics and another based on deep neural networks. The new physics-informed framework serves as a black-box transformer that is able to improve given distributions and can be easily extended to other geometries such as disks, spheres, complex manifolds, etc. Various experiments with different geometries are provided to demonstrate that the new framework is able to transform poorly distributed input into one with superior uniformity.
MSC:
| 11Kxx | Probabilistic theory: distribution modulo \(1\); metric theory of algorithms |
| 65Dxx | Numerical approximation and computational geometry (primarily algorithms) |
| 65Nxx | Numerical methods for partial differential equations, boundary value problems |
Keywords:
quasirandom points; deep neural networks; molecular dynamics; residual neural networks; potential theorySoftware:
rhalton; AdaGrad; OpenMM; PyTorch; Adam; nsf; sobol.cc; Algorithm 247; Gromacs; Glow; nflows; NAMD; DESMONDReferences:
| [1] | Niederreiter, H., Random Number Generation and Quasi-Monte Carlo Methods (1992), SIAM · Zbl 0761.65002 |
| [2] | Morokoff, W. J.; Caflisch, R. E., Quasi-random sequences and their discrepancies, SIAM J. Sci. Comput., 15, 6, 1251-1279 (1994) · Zbl 0815.65002 |
| [3] | Dobkin, D.; Eppstein, D.; Mitchell, D., Computing the discrepancy with applications to supersampling patterns, ACM Trans. Graph., 15, 4, 354-376 (1996) |
| [4] | Christensen, P.; Kensler, A.; Kilpatrick, C., Progressive Multi-Jittered Sample Sequences, Computer Graphics Forum, vol. 37, 21-33 (2018), Wiley Online Library |
| [5] | Perrier, H.; Coeurjolly, D.; Xie, F.; Pharr, M.; Hanrahan, P.; Ostromoukhov, V., Sequences with Low-Discrepancy Blue-Noise 2-D Projections, Computer Graphics Forum, vol. 37, 339-353 (2018), Wiley Online Library |
| [6] | Cheng, J.; Druzdzel, M., Computational investigation of low-discrepancy sequences in simulation algorithms for Bayesian networks (2013) |
| [7] | Cai, D.; Nagy, J.; Xi, Y., Fast deterministic approximation of symmetric indefinite kernel matrices with high dimensional datasets, SIAM J. Matrix Anal. Appl., 43, 2, 1003-1028 (2022) · Zbl 1508.65040 |
| [8] | Koksma, J. F., A general theorem from the theory of uniform distribution modulo 1, Math. B (Zutphen), 11, 7-11 (1942) · Zbl 0026.38803 |
| [9] | Niederreiter, H., Point sets and sequences with small discrepancy, Monatshefte Math., 104, 4, 273-337 (1987) · Zbl 0626.10045 |
| [10] | Dick, J.; Pillichshammer, F., Digital Nets and Sequences: Discrepancy Theory and Quasi-Monte Carlo Integration (2010), Cambridge University Press · Zbl 1282.65012 |
| [11] | Halton, J. H., Algorithm 247: radical-inverse quasi-random point sequence, Commun. ACM, 7, 12, 701-702 (1964) |
| [12] | Sobol, I. M., Multidimensional Quadrature Formulas and Haar Functions (1969), Izdat. Nauka: Izdat. Nauka Moscow · Zbl 0195.16903 |
| [13] | Sloan, I., Lattice methods for multiple integration, J. Comput. Appl. Math., 12, 131-143 (1985) · Zbl 0597.65014 |
| [14] | Sloan, I., Lattice Methods for Multiple Integration (1994), Oxford University Press · Zbl 0855.65013 |
| [15] | Faure, H., Good permutations for extreme discrepancy, J. Number Theory, 42, 1, 47-56 (1992) · Zbl 0768.11026 |
| [16] | Owen, A., Quasi-Monte Carlo Sampling, Monte Carlo Ray Tracing: Siggraph, vol. 1, 69-88 (2003) |
| [17] | Owen, A., Randomly permuted \((t, m, s)\)-nets and \((t, s)\)-sequences, (Monte Carlo and Quasi-Monte Carlo Methods in Scientific Computing (1995), Springer), 299-317 · Zbl 0831.65024 |
| [18] | Kocis, L.; Whiten, W., Computational investigations of low-discrepancy sequences, ACM Trans. Math. Softw., 23, 2, 266-294 (1997) · Zbl 0887.65031 |
| [19] | Vandewoestyne, B.; Cools, R., Good permutations for deterministic scrambled Halton sequences in terms of L2-discrepancy, J. Comput. Appl. Math., 189, 1-2, 341-361 (2006) · Zbl 1086.65003 |
| [20] | Karplus, M.; Petsko, G. A., Molecular dynamics simulations in biology, Nature, 347, 6294, 631-639 (1990) |
| [21] | Cuthbert, T. R.; Wagner, N. J.; Paulaitis, M. E.; Murgia, G.; D’Aguanno, B., Molecular dynamics simulation of penetrant diffusion in amorphous polypropylene: diffusion mechanisms and simulation size effects, Macromolecules, 32, 15, 5017-5028 (1999) |
| [22] | Genreith-Schriever, A.; De Souza, R., Field-enhanced ion transport in solids: reexamination with molecular dynamics simulations, Phys. Rev. B, 94, 22, Article 224304 pp. (2016) |
| [23] | Neyts, E. C.; Brault, P., Molecular dynamics simulations for plasma-surface interactions, Plasma Process. Polym., 14, 1-2, Article 1600145 pp. (2017) |
| [24] | Lindblom, O.; Ahlgren, T.; Heinola, K., Molecular dynamics simulations of hydrogen isotope exchange in tungsten vacancies, Nucl. Mater. Energy, 29, Article 101099 pp. (2021) |
| [25] | Rampino, S., Configuration-space sampling in potential energy surface fitting: a space-reduced bond-order grid approach, J. Phys. Chem. A, 120, 27, 4683-4692 (2016) |
| [26] | Schlegel, H., Exploring potential energy surfaces for chemical reactions: an overview of some practical methods, J. Comput. Chem., 24, 12, 1514-1527 (2003) |
| [27] | Kuipers, L.; Niederreiter, H., Uniform Distribution of Sequences (2012), Courier Corporation · Zbl 0568.10001 |
| [28] | Doerr, C.; Gnewuch, M.; Wahlström, M., Calculation of discrepancy measures and applications, (A Panorama of Discrepancy Theory (2014), Springer), 621-678 · Zbl 1358.11087 |
| [29] | Warnock, T. T., Computational investigations of low-discrepancy point sets, (Applications of Number Theory to Numerical Analysis (1972), Elsevier), 319-343 · Zbl 0248.65018 |
| [30] | Joe, S.; Kuo, F., Constructing Sobol′ sequences with better two-dimensional projections, SIAM J. Sci. Comput., 30, 5, 2635-2654 (2008) · Zbl 1171.65364 |
| [31] | Sloan, I.; Kuo, F.; Joe, S., Constructing randomly shifted lattice rules in weighted Sobolev spaces, SIAM J. Numer. Anal., 40, 5, 1650-1665 (2002) · Zbl 1037.65005 |
| [32] | Lyness, J., An introduction to lattice rules and their generator matrices, IMA J. Numer. Anal., 9, 3, 405-419 (1989) · Zbl 0691.65008 |
| [33] | Niederreiter, H., The existence of good extensible polynomial lattice rules, Monatshefte Math., 139, 4, 295-307 (2003) · Zbl 1038.11048 |
| [34] | Cools, R.; Kuo, F.; Nuyens, D., Constructing embedded lattice rules for multivariate integration, SIAM J. Sci. Comput., 28, 6, 2162-2188 (2006) · Zbl 1126.65002 |
| [35] | Kroto, H.; Heath, J.; O’Brien, S.; Curl, R.; Smalley, R., C 60: buckminsterfullerene, Nature, 318, 6042, 162-163 (1985) |
| [36] | Saka, M., Optimum geometry design of geodesic domes using harmony search algorithm, Adv. Struct. Eng., 10, 6, 595-606 (2007) |
| [37] | Tkach, A.; Pauly, M.; Tagliasacchi, A., Sphere-meshes for real-time hand modeling and tracking, ACM Trans. Graph., 35, 6, 1-11 (2016) |
| [38] | Bowers, K., Scalable algorithms for molecular dynamics simulations on commodity clusters, (SC’06: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing (2006), IEEE), 43 |
| [39] | Van Der Spoel, D.; Lindahl, E.; Hess, B.; Groenhof, G.; Mark, A.; Berendsen, H., GROMACS: fast, flexible, and free, J. Comput. Chem., 26, 16, 1701-1718 (2005) |
| [40] | Plimpton, S., Fast parallel algorithms for short-range molecular dynamics, J. Comput. Phys., 117, 1, 1-19 (1995) · Zbl 0830.65120 |
| [41] | Lammps molecular dynamics simulator |
| [42] | Phillips, J., Scalable molecular dynamics with NAMD, J. Comput. Chem., 26, 16, 1781-1802 (2005) |
| [43] | Eastman, P., OpenMM 7: rapid development of high performance algorithms for molecular dynamics, PLoS Comput. Biol., 13, 7, Article e1005659 pp. (2017) |
| [44] | Verlet, L., Computer “experiments” on classical fluids. I. Thermodynamical properties of Lennard-Jones molecules, Phys. Rev., 159, 1, 98 (1967) |
| [45] | Frenkel, D.; Smit, B., Understanding Molecular Simulation: from Algorithms to Applications, vol. 1 (2001), Elsevier |
| [46] | Swope, W.; Andersen, H.; Berens, P.; Wilson, K., A computer simulation method for the calculation of equilibrium constants for the formation of physical clusters of molecules: application to small water clusters, J. Chem. Phys., 76, 1, 637-649 (1982) |
| [47] | Raissi, M.; Perdikaris, P.; Karniadakis, G. E., Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378, 686-707 (2019) · Zbl 1415.68175 |
| [48] | Han, J.; Jentzen, A.; Weinan, E., Solving high-dimensional partial differential equations using deep learning, Proc. Natl. Acad. Sci. USA, 115, 34, 8505-8510 (2018) · Zbl 1416.35137 |
| [49] | Brunton, S. L.; Noack, B. R.; Koumoutsakos, P., Machine learning for fluid mechanics, Annu. Rev. Fluid Mech., 52, 477-508 (2020) · Zbl 1439.76138 |
| [50] | Cybenko, G., Approximation by superpositions of a sigmoidal function, Math. Control Signals Syst., 2, 4, 303-314 (1989) · Zbl 0679.94019 |
| [51] | Goodfellow, I.; Bengio, Y.; Courville, A.; Bengio, Y., Deep Learning, vol. 1 (2016), MIT Press: MIT Press Cambridge · Zbl 1373.68009 |
| [52] | Bottou, L., Online Algorithms and Stochastic Approximations, Online Learning and Neural Networks (1998) · Zbl 0968.68127 |
| [53] | Spall, J., Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control, vol. 65 (2005), John Wiley & Sons |
| [54] | Duchi, J.; Hazan, E.; Singer, Y., Adaptive subgradient methods for online learning and stochastic optimization, J. Mach. Learn. Res., 12, 7 (2011) · Zbl 1280.68164 |
| [55] | Kingma, D.; Ba Adam, J., A method for stochastic optimization, (Bengio, Y.; LeCun, Y., 3rd International Conference on Learning Representations. 3rd International Conference on Learning Representations, ICLR 2015 (2015)) |
| [56] | Korpelevich, G., The extragradient method for finding saddle points and other problems, Matecon, 12, 747-756 (1976) · Zbl 0342.90044 |
| [57] | Nesterov, Y., Dual extrapolation and its applications to solving variational inequalities and related problems, Math. Program., 109, 2, 319-344 (2007) · Zbl 1167.90014 |
| [58] | He, K.; Zhang, X.; Ren, S.; Sun, J., Deep residual learning for image recognition, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)), 770-778 |
| [59] | Melzer, A., Mode spectra of thermally excited two-dimensional dust Coulomb clusters, Phys. Rev. E, 67, 1, Article 016411 pp. (2003) |
| [60] | Radzvilavičius, A.; Anisimovas, E., Topological defect motifs in two-dimensional Coulomb clusters, J. Phys. Condens. Matter, 23, 38, Article 385301 pp. (2011) |
| [61] | Shapir, I.; Hamo, A.; Pecker, S.; Moca, C. P.; Legeza, Ö.; Zarand, G.; Ilani, S., Imaging the electronic Wigner crystal in one dimension, Science, 364, 6443, 870-875 (2019) |
| [62] | Capdeville, Y.; Chaljub, E.; Montagner, J., Coupling the spectral element method with a modal solution for elastic wave propagation in global Earth models, Geophys. J. Int., 152, 1, 34-67 (2003) |
| [63] | Simpson, J., Current and future applications of 3-d global Earth-ionosphere models based on the full-vector Maxwell’s equations FDTD method, Surv. Geophys., 30, 2, 105-130 (2009) |
| [64] | Manoharan, V., Colloidal matter: packing, geometry, and entropy, Science, 349, 6251 (2015) · Zbl 1355.82031 |
| [65] | Gotsman, C.; Gu, X.; Sheffer, A., Fundamentals of spherical parameterization for 3d meshes, (ACM SIGGRAPH 2003 Papers (2003)), 358-363 |
| [66] | Liu, G.-R., Meshfree Methods: Moving Beyond the Finite Element Method (2009), CRC Press |
| [67] | Cai, D.; Huang, H.; Chow, E.; Xi, Y., Data-driven construction of hierarchical matrices with nested bases (2022), arXiv preprint |
| [68] | Yudin, V., Minimum potential energy of a point system of charges, Diskretn. Mat., 4, 2, 115-121 (1992) · Zbl 0769.31004 |
| [69] | Thomson, J., XXIV. On the structure of the atom, Philos. Mag., 7, 39, 237-265 (1904) · JFM 35.0790.01 |
| [70] | Delsarte, P.; Goethals, J.-M.; Seidel, J., Spherical codes and designs, (Geometry and Combinatorics (1991), Elsevier), 68-93 |
| [71] | Fekete, M., Über die Verteilung der Wurzeln bei gewissen algebraischen Gleichungen mit ganzzahligen Koeffizienten, Math. Z., 17, 1, 228-249 (1923) · JFM 49.0047.01 |
| [72] | Smale, S., Mathematical problems for the next century, Math. Intell., 20, 2, 7-15 (1998) · Zbl 0947.01011 |
| [73] | Andreev, N., An extremal property of the icosahedron, East J. Approx., 2, 4, 459-462 (1996) · Zbl 0877.51021 |
| [74] | PyTorch |
| [75] | Paszke, A., PyTorch: an imperative style, high-performance deep learning library, (Wallach, H.; etal., Advances in Neural Information Processing Systems 32 (2019), Curran Associates, Inc.), 8024-8035 |
| [76] | Maas, A.; Hannun, A.; Ng, A., Rectifier nonlinearities improve neural network acoustic models, (Proc. ICML, vol. 30 (2013), Citeseer), 3 |
| [77] | Carrier, J.; Greengard, L.; Rokhlin, V., A fast adaptive multipole algorithm for particle simulations, SIAM J. Sci. Stat. Comput., 9, 4, 669-686 (1988) · Zbl 0656.65004 |
| [78] | Cai, D.; Xia, J., A stable matrix version of the fast multipole method: stabilization strategies and examples, Electron. Trans. Numer. Anal., 54, 581-609 (2021) · Zbl 1475.65026 |
| [79] | Börm, S.; Grasedyck, L.; Hackbusch, W., Introduction to hierarchical matrices with applications, Eng. Anal. Bound. Elem., 27, 5, 405-422 (2003) · Zbl 1035.65042 |
| [80] | Hackbusch, W., Hierarchical Matrices: Algorithms and Analysis, Springer Series in Computational Mathematics (2015), Springer: Springer Berlin Heidelberg · Zbl 1336.65041 |
| [81] | Cai, D.; Chow, E.; Erlandson, L.; Saad, Y.; Xi, Y., SMASH: structured matrix approximation by separation and hierarchy, Numer. Linear Algebra Appl., 25, 6, Article e2204 pp. (2018) · Zbl 1513.65128 |
| [82] | Erlandson, L.; Cai, D.; Xi, Y.; Chow, E., Accelerating parallel hierarchical matrix-vector products via data-driven sampling, (2020 IEEE International Parallel and Distributed Processing Symposium. 2020 IEEE International Parallel and Distributed Processing Symposium, IPDPS (2020)), 749-758 |
| [83] | Papamakarios, G.; Nalisnick, E.; Rezende, D. J.; Mohamed, S.; Lakshminarayanan, B., Normalizing flows for probabilistic modeling and inference (2019), arXiv preprint |
| [84] | Rezende, D.; Mohamed, S., Variational inference with normalizing flows, (International Conference on Machine Learning (2015)), 1530-1538 |
| [85] | Kingma, D. P.; Dhariwal Glow, P., Generative flow with invertible 1x1 convolutions, (Advances in Neural Information Processing Systems, vol. 31 (2018)), 10215-10224 |
| [86] | Durkan, C.; Bekasov, A.; Murray, I.; Papamakarios, G., Neural spline flows, (Advances in Neural Information Processing Systems (2019)), 7509-7520 |
| [87] | De Cao, N.; Aziz, W.; Titov, I., Block neural autoregressive flow, (Proceedings of the 35th Uncertainty in Artificial Intelligence Conference. Proceedings of the 35th Uncertainty in Artificial Intelligence Conference, Proceedings of Machine Learning Research, vol. 115 (2020)), 1263-1273 |
| [88] | Cai, D.; Ji, Y.; He, H.; Ye, Q.; Xi, Y., AUTM flow: atomic unrestricted time machine for monotonic normalizing flows, (The 38th Conference on Uncertainty in Artificial Intelligence (2022)) |
| [89] | Verfürth, R., A posteriori error estimation and adaptive mesh-refinement techniques, J. Comput. Appl. Math., 50, 1, 67-83 (1994) · Zbl 0811.65089 |
| [90] | Cai, D.; Cai, Z., A hybrid a posteriori error estimator for conforming finite element approximations, Comput. Methods Appl. Mech. Eng., 339, 320-340 (2018) · Zbl 1440.65187 |
| [91] | Cai, D.; Cai, Z.; Zhang, S., Robust equilibrated a posteriori error estimator for higher order finite element approximations to diffusion problems, Numer. Math., 144, 1, 1-21 (2020) · Zbl 1447.65130 |
| [92] | Cai, D.; Cai, Z.; Zhang, S., Robust equilibrated error estimator for diffusion problems: mixed finite elements in two dimensions, J. Sci. Comput., 83, 1, 1-22 (2020) · Zbl 1437.65177 |
| [93] | Cai, D.; Cai, Z., Hybrid a posteriori error estimators for conforming finite element approximations to stationary convection-diffusion-reaction equations, arXiv preprint |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
