Computer experiment designs for accurate prediction. (English) Zbl 1384.62281
Summary: Computer experiments using deterministic simulators are sometimes used to replace or supplement physical system experiments. This paper compares designs for an initial computer simulator experiment based on empirical prediction accuracy; it recommends designs for producing accurate predictions. The basis for the majority of the designs compared is the integrated mean squared prediction error (IMSPE) that is computed assuming a Gaussian process model with a Gaussian correlation function. Designs that minimize the IMSPE with respect to a fixed set of correlation parameters as well as designs that minimize a weighted IMSPE over the correlation parameters are studied. These IMSPE-based designs are compared with three widely-used space-filling designs. The designs are used to predict test surfaces representing a range of stationary and non-stationary functions. For the test conditions examined in this paper, the designs constructed under IMSPE-based criteria are shown to outperform space-filling Latin hypercube designs and maximum projection designs when predicting smooth functions of stationary appearance, while space-filling and maximum projection designs are superior for test functions that exhibit strong non-stationarity.
Keywords:
experimental design; Gaussian process kriging interpolator; IMSPE; Latin hypercube design; maximum projection design; space-filling design; empirical prediction accuracyReferences:
| [1] | Audze, P., Eglais, V.: New approach for planning out of experiments. Probl. Dyn. Strengths 35, 104-107 (1977) |
| [2] | Ba, S., Joseph, V.R.: Composite gaussian process models for emulating expensive functions. Ann. Appl. Stat. 6(4), 1838-1860 (2012) · Zbl 1257.62089 · doi:10.1214/12-AOAS570 |
| [3] | Ba, S., Joseph, V.R.: MaxPro: maximum projection designs (2015). https://CRAN.R-project.org/package=MaxPro, r package version 3.1-2 |
| [4] | Bates, R.A., Riccomagno, E., Schwabe, R., Wynn, H.P.: Lattices and dual lattices in experimental design for Fourier models. In: Proceedings of Workshop on “Quasi-Monte Carlo Methods and Their Applications”, pp 1-14 (1995) · Zbl 1042.62588 |
| [5] | Bates, S.J., Sienz, J., Toropov, V.V.: Formulation of the optimal latin hypercube design of experiments using a permutation genetic algorithm. Adv. Eng. Soft. 34, 493-506 (2003) · doi:10.1016/S0965-9978(03)00042-5 |
| [6] | Bechhofer, R.E., Santner, T.J., Goldsman, D.M.: Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. Wiley, Hoboken (1995) |
| [7] | Bursztyn, D., Steinberg, D.M.: Comparison of designs for computer experiments. J. Stat. Plann. Inference 136, 1103-1119 (2006) · Zbl 1078.62085 · doi:10.1016/j.jspi.2004.08.007 |
| [8] | Currin, C., Mitchell, T.J., Morris, M.D., Ylvisaker, D.: Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J. Am. Stat. Ass. 86, 953-963 (1991) · doi:10.1080/01621459.1991.10475138 |
| [9] | van Dam, E., den Hertog, D., Husslage, B., Rennen, G.: Space-filling designs. http://www.spacefillingdesigns.nl/. Accessed March 2013 (2013) · Zbl 1284.90100 |
| [10] | Draguljić, D., Santner, T.J., Dean, A.M.: Non-collapsing spacing-filling designs for bounded polygonal regions. Technometrics 54, 169-178 (2012) · doi:10.1080/00401706.2012.676951 |
| [11] | Efron, B.: Frequentist accuracy of bayesian estimates. J. R. Stat. Soc. B (2014). doi:10.1111/rssb.12080 · Zbl 1414.62089 |
| [12] | Fogelson, A.; Kuharsky, A.; Yu, H., Computational modeling of blood clotting: coagulation and three-dimensional platelet aggregation, 145-154 (2003), Basel · doi:10.1007/978-3-0348-8043-5_13 |
| [13] | Forrester, A., Sobester, A., Keane, A.: Engineering design via surrogate modelling: a practical guide. Wiley, Chichester (2008) · doi:10.1002/9780470770801 |
| [14] | Givens, G., Hoeting, J.: Computational Statistics. Wiley, Hoboken (2012) · Zbl 1267.62003 · doi:10.1002/9781118555552 |
| [15] | Hajagos, J.G.: Modeling uncertainty in population biology: How the model is written does matter. In: Hanson, K.M., Hemez, F.M. (eds) Proceedings of the SAMO 2004 Conference on Sensitivity Analysis. http://library.lanl.gov/ccw/samo2004/, Los Alamos National Laboratory, Los Alamos, pp. 363-368 (2005) |
| [16] | Higdon, D., Kennedy, M., Cavendish, J., Cafeo, J., Ryne, R.: Combining field data and computer simulations for calibration and prediction. SIAM J. Sci. Comput. 26, 448-466 (2004) · Zbl 1072.62018 · doi:10.1137/S1064827503426693 |
| [17] | Johnson, M.E., Moore, L.M., Ylvisaker, D.: Minimax and maximin distance designs. J. Stat. Plann. Inference 26, 131-148 (1990) · doi:10.1016/0378-3758(90)90122-B |
| [18] | Johnson, R.T., Montgomery, D.C., Jones, B.: An empirical study of the prediction performance of space-filling designs. Int. J. Exp. Des. Process Optim. 2(1), 1-18 (2011) · doi:10.1504/IJEDPO.2011.038048 |
| [19] | Joseph, V.R., Gul, E., Ba, S.: Maximum projection designs for computer experiments. Biometrika 102(2), 371-380 (2015) · Zbl 1452.62593 · doi:10.1093/biomet/asv002 |
| [20] | Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Neural Networks, 1995. Proceedings., IEEE International Conference on, vol 4, pp. 1942-1948 (1995) · Zbl 1094.62084 |
| [21] | Kincaid, D., Cheney, E.: Numerical Analysis: Mathematics of Scientific Computing. American Mathematical Society, Providence (2002) · Zbl 0877.65002 |
| [22] | Leatherman, E.R., Dean, A.M., Santner, T.J.: Computer experiment designs via particle swarm optimization. In: Melas, V., Mignani, S., Monari, P., Salmaso, L. (eds.) Topics in Statistical Simulation: Research Papers from the \[7^{th}7\] th International Workshop on Statistical Simulation, vol. 114, pp. 309-317. Springer, Berlin (2014a) |
| [23] | Leatherman, E.R., Guo, H., Gilbert, S.L., Hutchinson, I.D., Maher, S.A., Santner, T.J.: Using a statistically calibrated biphasic finite element model of the human knee joint to identify robust designs for a meniscal substitute. J. Biomech. Eng. 136(7), 071,007 (2014b) · Zbl 1072.62018 |
| [24] | Liefvendahl, M., Stocki, R.: A study on algorithms for optimization of latin hypercubes. J. Stat. Plann. Inference 136, 3231-3247 (2006) · Zbl 1094.62084 · doi:10.1016/j.jspi.2005.01.007 |
| [25] | MacDonald, B., Ranjan, P., Chipman, H., et al.: Gpfit: an r package for fitting a gaussian process model to deterministic simulator outputs. J. Stat. Soft. 64, i12 (2015) · doi:10.18637/jss.v064.i12 |
| [26] | MATLAB Parametric Empirical Kriging (MPErK) (2013) T.J. Santner Group, The Ohio State University |
| [27] | McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21, 239-245 (1979) · Zbl 0415.62011 |
| [28] | Montgomery, G.P., Truss, L.T.: Combining a statistical design of experiments with formability simulations to predict the formability of pockets in sheet metal parts. In: Technical Report 2001-01-1130, Society of Automotive Engineers (2001) · Zbl 1094.62084 |
| [29] | Morokoff, W.J., Caflisch, R.E.: Quasi-monte carlo integration. J. Comput. Phys. 122, 218-230 (1995) · Zbl 0863.65005 · doi:10.1006/jcph.1995.1209 |
| [30] | Morris, M.D., Mitchell, T.J.: Exploratory designs for computational experiments. J. Stat. Plann. Inference 43, 381-402 (1995) · Zbl 0813.62065 · doi:10.1016/0378-3758(94)00035-T |
| [31] | Nekkanty, S.: Characterization of damage and optimization of thin film coatings on ductile substrates. PhD thesis, Department of Mechanical Engineering, The Ohio State University, Columbus, Ohio, USA (2009) |
| [32] | Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods. SIAM, Philadelphia (1992) · Zbl 0761.65002 · doi:10.1137/1.9781611970081 |
| [33] | Ong, K., Santner, T., Bartel, D.: Robust design for acetabular cup stability accounting for patient and surgical variability. J. Biomech. Eng. 130(031), 001 (2008) |
| [34] | Pronzato, L., Müller, W.G.: Design of computer experiments: space filling and beyond. Stat. Comput. 22, 681-701 (2012) · Zbl 1252.62080 · doi:10.1007/s11222-011-9242-3 |
| [35] | R Core Team (2016) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/ |
| [36] | Sacks, J., Schiller, S.B., Welch, W.J.: Design for computer experiments. Technometrics 31, 41-47 (1989a) · doi:10.1080/00401706.1989.10488474 |
| [37] | Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P.: Design and analysis of computer experiments. Stat. Sci. 4, 409-423 (1989b) · Zbl 0955.62619 · doi:10.1214/ss/1177012413 |
| [38] | Silvestrini, R.T., Montgomery, D.C., Jones, B.: Comparing computer experiments for the Gaussian process model using integrated prediction variance. Qual. Eng. 25, 164-174 (2013) · doi:10.1080/08982112.2012.758284 |
| [39] | Trosset, M.W.: The krigifier: a procedure for generating pseudorandom nonlinear objective functions for computational experimentation. In: Technical Report 35, Institute for Computer Applications in Science and Engineering, NASA Langley Research Center (1999) |
| [40] | Upton, M.L., Guilak, F., Laursen, T.A., Setton, L.A.: Finite element modeling predictions of region-specific cell-matrix mechanics in the meniscus. Biomech. Model. Mechanobiol. 5, 140-149 (2006) · doi:10.1007/s10237-006-0031-4 |
| [41] | Villarreal-Marroquín, M.G., Svenson, J.D., Sun, F., Santner, T.J., Dean, A., Castro, J.M.: A comparison of two metamodel-based methodologies for multiple criteria simulation optimization using an injection molding case study. J. Poly. Eng. 33, 193-209 (2013) |
| [42] | Welch, W.J.: Aced: algorithms for the construction of experimental designs. Am. Stat. 39, 146 (1985) · doi:10.2307/2682827 |
| [43] | Xiong, Y., Chen, W., Apley, D., Ding, X.: A non-stationary covariance-based kriging method for metamodelling in engineering design. Int. J. Numer. Methods Eng. 71(6), 733-756 (2007). doi:10.1002/nme.1969 · Zbl 1194.74553 · doi:10.1002/nme.1969 |
| [44] | Yang, X.S.: Engineering Optimization: An Introduction with Metaheuristic Applications, 1st edn. Wiley Publishing, Hoboken (2010) · doi:10.1002/9780470640425 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
