Robust verification of stochastic simulation codes. (English) Zbl 07517159
Summary: We introduce a robust verification tool for computational codes, which we call Stochastic Robust Extrapolation based Error Quantification (StREEQ). Unlike the prevalent Grid Convergence Index (GCI) [W. L. Oberkampf and C. J. Roy, Verification and validation in scientific computing. Cambridge: Cambridge University Press (2010; Zbl 1211.68499)] method, our approach is suitable for both stochastic and deterministic computational codes and is generalizable to any number of discretization variables. Building on ideas introduced in the Robust Verification [W. Rider et al., J. Comput. Phys. 307, 146–163 (2016; Zbl 1352.65137)] approach, we estimate the converged solution and orders of convergence with uncertainty using multiple fits of a discretization error model. In contrast to Robust Verification, we perform these fits to many bootstrap samples yielding a larger set of predictions with smoother statistics. Here, bootstrap resampling is performed on the lack-of-fit errors for deterministic code responses, and directly on the noisy data set for stochastic responses. This approach lends a degree of robustness to the overall results, capable of yielding precise verification results for sufficiently resolved data sets, and appropriately expanding the uncertainty when the data set does not support a precise result. For stochastic responses, a credibility assessment is also performed to give the analyst an indication of the trustworthiness of the results. This approach is suitable for both code and solution verification, and is particularly useful for solution verification of high-consequence simulations.
MSC:
| 65Mxx | Numerical methods for partial differential equations, initial value and time-dependent initial-boundary value problems |
| 62Gxx | Nonparametric inference |
| 76Pxx | Rarefied gas flows, Boltzmann equation in fluid mechanics |
Keywords:
verification; convergence; error estimation; solution verification; plasma simulation; Monte Carlo methodsReferences:
| [1] | Oberkampf, W. L.; Roy, C. J., Verification and Validation in Scientific Computing (2010), Cambridge University Press · Zbl 1211.68499 |
| [2] | Rider, W. J.; Witkowski, W. R.; Kamm, J. R.; Wildey, T. M., Robust verification analysis, J. Comput. Phys., 307, 146-163 (2016) · Zbl 1352.65137 |
| [3] | Roache, P. J., Fundamentals of Verification and Validation (2009), Hermosa Publishers: Hermosa Publishers Socorro, NM |
| [4] | Roy, C. J., Review of code and solution verification procedures for computational simulation, J. Comput. Phys., 205, 1, 131-156 (2005) · Zbl 1072.65118 |
| [5] | The American Society of Mechanical Engineers, Standard for verification and validation in computational fluid dynamics and heat transfer, ASME V&V 20-2009 (2009) |
| [6] | Birdsall, C. K.; Langdon, A. B., Plasma Physics via Computer Simulation (2005), Taylor and Francis |
| [7] | Hockney, R. W.; Eastwood, J. W., Computer Simulation Using Particles (1988), Taylor and Francis · Zbl 0662.76002 |
| [8] | Seltzer, S. M., An overview of the ETRAN Monte Carlo methods, (Jenkins, T. M.; Nelson, W. R.; Rindi, A., Monte Carlo Transport of Electrons and Photons (1988), Plenum Press), 153-181 |
| [9] | Bird, G. A., Molecular Gas Dynamics and the Direct Simulation of Gas Flows (1994), Clarendon |
| [10] | Riva, F.; Ricci, P.; Halpern, F. D.; Jolliet, S.; Loizu, J.; Mosetto, A., Verification methodology for plasma simulations and application to a scrape-off layer turbulence code, Phys. Plasmas, 21, 062301 (2014) · Zbl 1344.76035 |
| [11] | Riva, F.; Beadle, C. F.; Ricci, P., A methodology for the rigorous verification of particle-in-cell simulations, Phys. Plasmas, 24, 055703 (2017) |
| [12] | Rader, D. J.; Gallis, M. A.; Torczynski, J. R.; Wagner, W., Direct simulation Monte Carlo convergence behavior of the hard-sphere-gas thermal conductivity for Fourier heat flow, Phys. Fluids, 18, Article 077102 pp. (2006) |
| [13] | Ricardson, L. F., The approximate arithmetical solution by finite differences of physical problems involving differential equations, with an application to the stresses in a masonry dam, Philos. Trans. R. Soc. Lond., Ser. A, Contain. Pap. Math. Phys. Character, 210, 307-357 (1911) · JFM 42.0873.02 |
| [14] | Radtke, G. A.; Cartwright, K. L.; Musson, L. C., Stochastic Richardson extrapolation based numerical error estimation with application to kinetic plasma simulations (April 2017), Sandia National Laboratories: Sandia National Laboratories Albuquerque, New Mexico 87185 and Livermore, California 94550, Technical report SAND2017-4165 |
| [15] | Brown, M. B.; Forsythe, A. B., Robust tests for equality of variances, J. Am. Stat. Assoc., 69, 346, 364-367 (1974) · Zbl 0291.62063 |
| [16] | Davison, A. C.; Hinkley, D. V., Bootstrap Methods and Their Application (1997), Cambridge University Press · Zbl 0886.62001 |
| [17] | Hand, M. L., Aspects of linear regression estimation under the criterion of minimizing the maximum absolute residual (1978), Iowa State University, Ph.D. thesis |
| [18] | Santosa, F.; Symes, W. W., Linear inversion of band-limited reflection seismograms, SIAM J. Sci. Stat. Comput., 7, 1307-1330 (1986) · Zbl 0602.73113 |
| [19] | Raier, K., Numerical Analysis, 90-96 (1998) |
| [20] | Engquist, B.; Sjögreen, B., The convergence rate of finite difference schemes in the presence of shocks, SIAM J. Numer. Anal., 35, 2464-2485 (1998) · Zbl 0922.76254 |
| [21] | Nelder, J. A.; Mead, R., A simplex method for function minimization, Comput. J., 7, 308-313 (1965) · Zbl 0229.65053 |
| [22] | Andrae, R.; Schulze-Hartung, T.; Melchoir, P., Dos and don’ts of reduced chi-squared (2010) |
| [23] | Neill, J. W., Testing for lack of fit in nonlinear regression, Ann. Stat., 16, 2, 733-740 (1988) · Zbl 0646.62057 |
| [24] | Doran, H. E.; Kmenta, J., A lack-of-fit test for econometric applications to cross-section data, Rev. Econ. Stat., 68, 2, 346-350 (1986) |
| [25] | Bardsley, W. G.; McGinlay, P. B., The use of non-linear regression analysis and the F test for model discrimination with dose-response curves and ligand binding data, J. Theor. Biol., 126, 2, 183-201 (1987) |
| [26] | Huber, P. J.; Rochetti, E. M., Robust Statistics (2009), Wiley · Zbl 1276.62022 |
| [27] | Leys, C.; Ley, C.; Klein, O.; Bernard, P.; Licata, L., Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median, J. Exp. Soc. Psychol., 49, 764-766 (2013) |
| [28] | Casella, G.; Berger, R. L., Statistical Inference (2002), Duxbury: Duxbury Belmont, CA |
| [29] | Bettencourt, M. T.; Boerner, J. J.; Crozier, P. S.; Fierro, A. S.; Grillet, A. M.; Hooper, R. W.; Hopkins, M. M.; Hughes, T. P.; Meyer, H. E.; Moore, C. H.; Moore, S. G.; Musson, L. C.; Pacheco, J. L., Aleph manual (Sept 2017), Sandia National Laboratories: Sandia National Laboratories Albuquerque, New Mexico 87185, Technical report SAND2017-10343 |
| [30] | Bettencourt, M. T.; Brown, D. A.S.; Cartwright, K. L.; Cyr, E. C.; Glusa, C. A.; Lin, P. T.; Moore, S. G.; McGregor, D. A.O.; Pawlowski, R. P.; Phillips, E. G.; Roberts, N. V.; Wright, S. A.; Maheswaran, S.; Jones, J. P.; Jarvis, S. A., EMPIRE-PIC: a performance portable unstructured particle-in-cell code, Commun. Comput. Phys., 30, 4, 1232-1268 (2021) · Zbl 1494.78004 |
| [31] | Krook, M.; Wu, T. T., Exact solutions of the Boltzmann equation, Phys. Fluids, 20, 10, 1589-1595 (1977) · Zbl 0369.76075 |
| [32] | Bobylev, A. V., Proc. Acad. Sci. USSR, 225 (1975) |
| [33] | Wagner, W., A convergence proof for Bird’s direct simulation Monte Carlo method for the Boltzmann equation, J. Stat. Phys., 66, 3/4, 1011-1044 (1992) · Zbl 0899.76312 |
| [34] | Franke, B. C.; Kensek, R. P.; Laub, T.; Crawford, M. J., Its version 6: the integrated TIGER series of coupled electron/photon Monte Carlo transport codes, revision 5 (2013), Sandia National Laboratories: Sandia National Laboratories Albuquerque, New Mexico 87185, Technical report SAND2008-3331 |
| [35] | Kim, S. H.; Pia, M. G.; Basaglia, T.; Han, M. C.; Hoff, G.; Kim, C. H.; Saracco, P., Validation test of geant4 simulation of electron backscattering, IEEE Trans. Nucl. Sci., 62, 2, 451-479 (2015) |
| [36] | Berger, M. J., Monte Carlo calculation of the penetration and diffusion of fast charged particles, (Fernbach, S.; Rotenburg, M., Methods in Computational Physics, Volume 1 (1963), Academic Press: Academic Press New York), 135-215 |
| [37] | Larsen, E. W., A theoretical derivation of the condensed history algorithm, Ann. Nucl. Energy, 19, 10-12, 701-714 (1992) |
| [38] | Bohren, C. F.; Huffmann, D. R., Absorption and Scattering of Light by Small Particles (2010), Wiley-Interscience: Wiley-Interscience New York |
| [39] | Stratton, J. A., Electromagnetic Theory (2007), Wiley-Interscience |
| [40] | Langston, W. L.; Kotulski, J.; Coats, R.; Jorgenson, R.; Blake, S. A.; Campione, S.; Pung, A.; Zinser, B., Massively parallel frequency domain electromagnetic simulation codes, (Proceedings of the 2018 International Applied Computational Electromagnetics Society Symposium (ACES) (2018), IEEE: IEEE Denver, CO, USA) |
| [41] | Warnick, K.; Chew, W., Error analysis of scattering amplitudes and RCS, Dig. IEEE Antennas Propag. Soc., Int. Symp., 4, 598-601 (2002) |
| [42] | Warnick, K.; Chew, W., Error analysis of the moment method, IEEE Antennas Propag. Mag., 46, 6, 38-53 (2004) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
