skip to main content
research-article
Public Access

Multivariate Input Uncertainty in Output Analysis for Stochastic Simulation

Published: 23 October 2016 Publication History

Abstract

When we use simulations to estimate the performance of stochastic systems, the simulation is often driven by input models estimated from finite real-world data. A complete statistical characterization of system performance estimates requires quantifying both input model and simulation estimation errors. The components of input models in many complex systems could be dependent. In this paper, we represent the distribution of a random vector by its marginal distributions and a dependence measure: either product-moment or Spearman rank correlations. To quantify the impact from dependent input model and simulation estimation errors on system performance estimates, we propose a metamodel-assisted bootstrap framework that is applicable to cases when the parametric family of multivariate input distributions is known or unknown. In either case, we first characterize the input models by their moments that are estimated using real-world data. Then, we employ the bootstrap to quantify the input estimation error, and an equation-based stochastic kriging metamodel to propagate the input uncertainty to the output mean, which can also reduce the influence of simulation estimation error due to output variability. Asymptotic analysis provides theoretical support for our approach, while an empirical study demonstrates that it has good finite-sample performance.

Supplementary Material

a5-xie-apndx.pdf (xie.zip)
Supplemental movie, appendix, image and software files for, Multivariate Input Uncertainty in Output Analysis for Stochastic Simulation

References

[1]
Bruce E. Ankenman, Barry L. Nelson, and Jeremy Staum. 2010. Stochastic kriging for simulation metamodeling. Operations Research 58 (2010), 371--382.
[2]
Eusebio Arenal-Gutiérrez, Carlos Matrán, and Juan A. Cuesta-Albertos. 1996. Unconditional Glivenko-Gantelli-type theorems and weak laws of large numbers for bootstrap. Statistics 8 Probability Letters 26 (1996), 365--375.
[3]
Francois Bachoc. 2013. Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification. Computational Statistics 8 Data Analysis 66 (2013), 55--69.
[4]
Russell R. Barton. 2007. Presenting a more complete characterization of uncertainty: Can it be done? In Proceedings of the 2007 INFORMS Simulation Society Research Workshop. INFORMS Simulation Society, Fontainebleau.
[5]
Russell R. Barton. 2012. Tutorial: Input uncertainty in output analysis. In Proceedings of the 2012 Winter Simulation Conference, C. Laroque, J. Himmelspach, R. Pasupathy, O. Rose, and A. M. Uhrmacher (Eds.). IEEE Computer Society, 67--78.
[6]
Russell R. Barton, Barry L. Nelson, and Wei Xie. 2014. Quantifying input uncertainty via simulation confidence intervals. Informs Journal on Computing 26 (2014), 74--87.
[7]
Russell R. Barton and Lee W. Schruben. 1993. Uniform and bootstrap resampling of input distributions. In Proceedings of the 1993 Winter Simulation Conference, G. W. Evans, M. Mollaghasemi, E. C. Russell, and W. E. Biles (Eds.). IEEE Computer Society, 503--508.
[8]
Russell R. Barton and Lee W. Schruben. 2001. Resampling methods for input modeling. In Proceedings of the 2001 Winter Simulation Conference, B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer (Eds.). IEEE Computer Society, 372--378.
[9]
Bahar Biller and Canan G. Corlu. 2011. Accounting for parameter uncertainty in large-scale stochastic simulations with correlated inputs. Operations Research 59 (2011), 661--673.
[10]
Bahar Biller and Soumyadip Ghosh. 2006. Multivariate input processes. In Handbooks in Operations Research and Management Science: Simulation, S. Henderson and B. L. Nelson (Eds.). Elsevier, Chapter 5.
[11]
Patrick Billingsley. 1995. Probability and Measure. Wiley-Interscience, New York.
[12]
Marne C. Cario and Barry L. Nelson. 1997. Modeling and Generating Random Vectors with Arbitrary Marginal Distributions and Correlation Matrix. Technical report. Department of Industrial Engineering and Management Sciences, Northwestern University.
[13]
Xi Chen, Bruce E. Ankenman, and Barry L. Nelson. 2012. The effect of common random numbers on stochastic kriging metamodels. ACM Transactions on Modeling and Computer Simulation 22 (2012), 7:1--7:20.
[14]
Russell C. H. Cheng and Wayne Holland. 1997. Sensitivity of computer simulation experiments to errors in input data. Journal of Statistical Computation and Simulation 57 (1997), 219--241.
[15]
Robert T. Clemen and Terence Reilly. 1999. Correlations and copulas for decision and risk analysis. Management Science 45 (1999), 208--224.
[16]
Sourav Das, Tata S. Rao, and Georgi N. Boshnakov. 2012. On the Estimation of Parameters of Variograms of Spatial Stationary Isotropic Random Processes. Research Report No. 2. The University of Manchester.
[17]
Soumyadip Ghosh and Shane G. Henderson. 2002a. Chessboard distributions and random vectors with specified marginals and covariance matrix. Operations Research 50 (2002), 820--834.
[18]
Soumyadip Ghosh and Shane G. Henderson. 2002b. Properties of the NORTA method in higher dimensions. In Proceedings of the 2002 Winter Simulation Conference, E. Yűcesan, C. H. Chen, J. L. Snowdon, and J. M. Charnes (Eds.). IEEE Computer Society, 263--269.
[19]
Peter Hall. 1988. Rate of convergence in bootstrap approximations. The Annals of Probability 16 (1988), 1665--1684.
[20]
Shane G. Henderson, Belinda A. Chiera, and Roger M. Cooke. 2000. Generating dependent quasi-random numbers. In Proceedings of the 2000 Winter Simulation Conference, J. A. Joines, R. R. Barton, K. Kang, and P. A. Fishwick (Eds.). IEEE Computer Society, 527--536.
[21]
Nicholas J. Higham. 2002. Computing the nearest correlation matrix -- A problem from finance. IMA Journal on Numerical Analysis 22 (2002), 329--343.
[22]
Mark E. Johnson. 1987. Multivariate Statistical Simulation. Wiley, New York.
[23]
Donald R. Jones, Matthias Schonlau, and William J. Welch. 1998. Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13 (1998), 455--492.
[24]
Keebom Kang and Bruce Schmeiser. 1990. Graphical methods for evaluation and comparing confidence-interval procedures. Operations Research 38, 3 (1990), 546--553.
[25]
Shing T. Li and Joseph L. Hammond. 1975. Generation of pseudorandom numbers with specified univariate distributions and correlation coefficients. IEEE Transactions on Systems, Man, and Cybernetics 5 (September 1975), 557--561.
[26]
Jason L. Loeppky, Jerome Sacks, and William J. Welch. 2009. Choosing the sample size of a computer experiment: A practical guide. Technometrics 51 (2009), 366--376.
[27]
Tanmoy Mukhopadhyay, Sushanta Chakroborty, Sondipon Adhikari, and Rajib Chowdhury. 2016. A critical assessment of kriging model variants for high-fidelity uncertainty quantification in dynamics of composite shells. Archives on Computational Methods in Engineering (2016). Online version.
[28]
Bruce W. Schmeiser and Ram Lal. 1982. Bivariate gamma random vectors. Operations Research 30, 2 (1982), 355--374.
[29]
Jun Shao and Dongsheng Tu. 1995. The Jackknife and Bootstrap. Springer-Verlag.
[30]
Eunhye Song, Barry L. Nelson, and C. Dennis Pegden. 2014. Advanced tutorial: Input uncertainty quantification. In Proceedings of the 2014 Winter Simulation Conference, A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buckley, and J. A. Miller (Eds.). IEEE Computer Society.
[31]
A. W. Van Der Vaart. 1998. Asymptotic Statistics. Cambridge University Press, Cambridge, UK.
[32]
William R. Wade. 2010. An Introduction to Analysis (4th ed.). Prentice Hall.
[33]
Wei B. Wu and Jan Mielniczuk. 2010. A new look at measuring dependence. In Dependence in Probability and Statistics, P. Doukhan, G. Lang, D. Surgailis, and G. Teyssière (Eds.). Springer.
[34]
Wei Xie, Barry L. Nelson, and Russell R. Barton. 2014a. A Bayesian framework for quantifying uncertainty in stochastic simulation. Operational Research 62, 6 (2014), 1439--1452.
[35]
Wei Xie, Barry L. Nelson, and Russell R. Barton. 2014b. Statistical uncertainty analysis for stochastic simulation with dependent input models. In Proceedings of the 2014 Winter Simulation Conference, A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buckley, and J. A. Miller (Eds.). IEEE Computer Society.
[36]
Wei Xie, Barry L. Nelson, and Russell R. Barton. 2015. Statistical uncertainty analysis for stochastic simulation. (2015). Working Paper, Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY.
[37]
Wei Xie, Barry L. Nelson, and Jeremy Staum. 2010. The influence of correlation functions on stochastic kriging metamodels. In Proceedings of the 2010 Winter Simulation Conference, B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, and E. Yucesan (Eds.). IEEE Computer Society, 1067--1078.
[38]
Jun Yin, Szu H. Ng, and Kien M. Ng. 2009. A study on the effects of parameter estimation on kriging model’s prediction error in stochastic simulation. In Proceedings of the 2009 Winter Simulation Conference, B. Johansson A. Dunkin M. D. Rossetti, R. R. Hill and R. G. Ingalls (Eds.). IEEE Computer Society, 674--685.

Cited By

View all
  • (2024)Data-Driven Ranking and Selection Under Input UncertaintyOperations Research10.1287/opre.2022.237572:2(781-795)Online publication date: 1-Mar-2024
  • (2024)A Shrinkage Approach to Improve Direct Bootstrap Resampling Under Input UncertaintyINFORMS Journal on Computing10.1287/ijoc.2022.004436:4(1023-1039)Online publication date: 1-Jul-2024
  • (2024)Healthy lifestyle and behavioural intentions: the role of self-identity, self-efficacy subjective norms, and attitudesInternational Journal of Spa and Wellness10.1080/24721735.2024.2374588(1-21)Online publication date: 12-Jul-2024
  • Show More Cited By

Index Terms

  1. Multivariate Input Uncertainty in Output Analysis for Stochastic Simulation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Modeling and Computer Simulation
    ACM Transactions on Modeling and Computer Simulation  Volume 27, Issue 1
    January 2017
    150 pages
    ISSN:1049-3301
    EISSN:1558-1195
    DOI:10.1145/2982568
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 23 October 2016
    Accepted: 01 August 2016
    Revised: 01 July 2016
    Received: 01 October 2014
    Published in TOMACS Volume 27, Issue 1

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Bootstrap
    2. Gaussian process
    3. NORTA
    4. confidence interval
    5. multivariate input uncertainty
    6. output analysis

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)60
    • Downloads (Last 6 weeks)7
    Reflects downloads up to 24 Oct 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Data-Driven Ranking and Selection Under Input UncertaintyOperations Research10.1287/opre.2022.237572:2(781-795)Online publication date: 1-Mar-2024
    • (2024)A Shrinkage Approach to Improve Direct Bootstrap Resampling Under Input UncertaintyINFORMS Journal on Computing10.1287/ijoc.2022.004436:4(1023-1039)Online publication date: 1-Jul-2024
    • (2024)Healthy lifestyle and behavioural intentions: the role of self-identity, self-efficacy subjective norms, and attitudesInternational Journal of Spa and Wellness10.1080/24721735.2024.2374588(1-21)Online publication date: 12-Jul-2024
    • (2022)Subsampling to Enhance Efficiency in Input Uncertainty QuantificationOperations Research10.1287/opre.2021.216870:3(1891-1913)Online publication date: 1-May-2022
    • (2022)Robust Simulation with Likelihood-Ratio Constrained Input UncertaintyINFORMS Journal on Computing10.1287/ijoc.2022.116934:4(2350-2367)Online publication date: 1-Jul-2022
    • (2022)Input Uncertainty in Stochastic SimulationThe Palgrave Handbook of Operations Research10.1007/978-3-030-96935-6_17(573-620)Online publication date: 8-Jul-2022
    • (2021)Multifidelity Modeling for Analysis and Optimization of Serial Production LinesIEEE Transactions on Automatic Control10.1109/TAC.2020.302514366:8(3460-3474)Online publication date: Aug-2021
    • (2020)Stochastic Simulation under Input Uncertainty: A ReviewOperations Research Perspectives10.1016/j.orp.2020.100162(100162)Online publication date: Sep-2020
    • (2018)Metamodel-assisted risk analysis for stochastic simulation with input uncertaintyProceedings of the 2018 Winter Simulation Conference10.5555/3320516.3320731(1766-1777)Online publication date: 9-Dec-2018
    • (2018)Uncertainty Analysis for Natural Gas Transport Pipeline Network Layout: A New Methodology Based on Monte Carlo MethodJournal of Advanced Transportation10.1155/2018/92136482018(1-10)Online publication date: 2018
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media