skip to main content
article

To batch or not to batch?

Published: 01 January 2004 Publication History

Abstract

When designing steady-state computer simulation experiments, one may be faced with the choice of batching observations in one long run or replicating a number of smaller runs. Both methods are potentially useful in the course of undertaking simulation output analysis. The tradeoffs between the two alternatives are well known: batching ameliorates the effects of initialization bias, but produces batch means that might be correlated; replication yields independent sample means, but may suffer from initialization bias at the beginning of each of the runs. We present several new results and specific examples to lend insight as to when one method might be preferred over the other. In steady-state, batching and replication perform similarly in terms of estimating the mean and variance parameter, but replication tends to do better than batching with regard to the performance of confidence intervals for the mean. Such a victory for replication may be hollow, for in the presence of an initial transient, batching often performs better than replication when it comes to point and confidence-interval estimation of the steady-state mean. We conclude---like other classic references---that in the context of estimation of the steady-state mean, batching is typically the wiser approach.

References

[1]
Alexopoulos, C. and Goldsman, D. 2003. Replicated batch means variance estimators in the presence of an initial transient. Tech. rep. School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA.
[2]
Alexopoulos, C. and Seila, A. F. 1998. Output data analysis. In Handbook of Simulation, J. Banks, Ed. John Wiley & Sons, New York, NY, 225--272.
[3]
Andradóttir, S. and Argon, N. 2003. Replicated batch means for steady-state simulations. Tech. rep., School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA.
[4]
Billingsley, P. 1968. Convergence of Probability Measures. John Wiley & Sons, New York, NY.
[5]
Carlstein, E. 1986. The use of subseries for estimating the variance of a general statistic from a stationary sequence. Ann. Stat. 14, 1171--1179.
[6]
Chien, C.-H. 1989. Small sample theory for steady state confidence intervals. Tech. rep., Department of Operations Research, Stanford University, Stanford, CA.
[7]
Chien, C.-H., Goldsman, D., and Melamed, B. 1997. Large-sample results for batch means. Manage. Sci. 43, 1288--1295.
[8]
Evans, M., Hastings, N., and Peacock, B. 2000. Statistical Distributions, 3rd ed. John Wiley & Sons, New York, NY.
[9]
Fishman, G. S. 1998. LABATCH.2: Software for statistical analysis of simulation sample path data. In Proceedings of the 1998 Winter Simulation Conference, D. J. Medeiros, E. F. Watson, J. S. Carson, and M. S. Manivannan, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 131--139.
[10]
Fishman, G. S. 2001. Discrete-Event Simulation: Modeling, Programming, and Analysis. Springer-Verlag, New York, NY.
[11]
Fishman, G. S. and Yarberry, L. S. 1997. An implementation of the batch means method. INFORMS J. Comput. 9, 296--310.
[12]
Gafarian, A. V., Ancker, Jr., C. J., and Morisaku, F. 1978. Evaluation of commonly used rules for detecting "steady-state" in computer simulation. Naval Res. Logist. Quart. 25, 511--529.
[13]
Gallagher, M. A., Bauer, Jr., K. W., and Maybeck, P. S. 1996. Initial data truncation for univariate output of discrete-event simulations using the Kalman filter. Manage. Sci. 42, 559--575.
[14]
Glynn, P. W. and Heidelberger, P. 1991. Analysis of initial transient deletion for replicated steady-state simulations. Operat. Res. Lett. 10, 437--443.
[15]
Glynn, P. W. and Iglehart, D. L. 1990. Simulation output analysis using standardized time series. Math. Operat. Res. 15, 1--16.
[16]
Glynn, P. W. and Whitt, W. 1991. Estimating the asymptotic variance with batch means. Operat. Res. Lett. 10, 431--435.
[17]
Goldsman, D. and Meketon, M. S. 1986. A comparison of several variance estimators. Tech. rep., School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA.
[18]
Goldsman, D., Schruben, L. W., and Swain, J. J. 1994. Tests for transient means in simulated time series. Naval Res. Logist. 41, 171--187.
[19]
Hogg, R. V. and Craig, A. T. 1995. Introduction to Mathematical Statistics, 5th ed. Prentice-Hall, Upper Saddle River, NJ.
[20]
Kang, K. and Goldsman, D. 1990. The correlation between mean and variance estimators in computer simulation. IIE Trans. 22, 15--23.
[21]
Kelton, W. D. 1989. Random initialization methods in simulation. IIE Trans. 21, 355--367.
[22]
Kelton, W. D. and Law, A. M. 1983. A new approach for dealing with the startup problem in discrete event simulation. Naval Res. Logist. 30, 641--658.
[23]
Kelton, W. D. and Law, A. M. 1984. An analytical evaluation of alternative strategies in steady-state simulation. Operat. Res. 32, 169--184.
[24]
Krishman, M. 1968. Series representation of the doubly non-central t distribution. J. Amer. Statist. Assoc. 63, 323, 1004--1012.
[25]
Law, A. M. and Kelton, W. D. 2000. Simulation Modeling and Analysis, 3rd ed. McGraw-Hill, New York, NY.
[26]
Meketon, M. S. and Schmeiser, B. W. 1984. Overlapping batch means: Something for nothing? In Proceedings of the 1984 Winter Simulation Conference, S. Sheppard, U. W. Pooch, and C. D. Pegden, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 227--230.
[27]
NIST. 2002. Dataplot. National Institute of Standards and Technology, Gaithersburg, MD. Available from www.nist.gov (http://www.itl.nist.gov/div898/software/dataplot/; accessed March 27, 2003).
[28]
Patel, J. K. and Read, C. B. 1996. Handbook of the Normal Distribution, 2nd ed. Marcel Dekker, New York, NY.
[29]
Sargent, R. G., Kang, K., and Goldsman, D. 1992. An investigation of finite-sample behavior of confidence interval estimators. Operat. Res. 40, 898--913.
[30]
Schmeiser, B. W. and Yeh, Y. 2002. On choosing a single criterion for confidence-interval procedures. In Proceedings of the 2002 Winter Simulation Conference, E. Yücesan, C.-H. Chen, J. L. Snowdon, and J. M. Charnes, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 345--352.
[31]
Schruben, L. W., Singh, H., and Tierney, L. 1983. Optimal tests for initialization bias in simulation output. Operat. Res. 31, 1167--1178.
[32]
Searle, S. R. 1971. Linear Models. John Wiley & Sons, New York, NY.
[33]
Song, W.-M. and Schmeiser, B. W. 1995. Optimal mean-squared-error batch sizes. Manage. Sci. 41, 110--123.
[34]
Steiger, N. M. and Wilson, J. R. 2001. Convergence properties of the batch-means method for simulation output analysis. INFORMS J. Comput. 13, 4, 277--293.
[35]
Steiger, N. M. and Wilson, J. R. 2002. An improved batch means procedure for simulation output analysis. Manage. Sci. 48, 1569--1586.
[36]
Titus, B. D. 1985. Modified confidence intervals for the mean of an autoregressive process. Tech. rep. 34. Department of Operations Research, Stanford University, Stanford, CA.
[37]
Vassilacopoulos, C. 1989. Testing for initialization bias in simulation output. Simulation 52, 151--153.
[38]
von Neumann, J. 1941. Distribution of the ratio of the mean square successive difference and the variance. Ann. Math. Statist. 12, 367--395.
[39]
Welch, P. D. 1983. The statistical analysis of simulation results. In The Computer Performance Modeling Handbook, S. Lavenberg, Ed. Academic Press, New York, NY, 268--328.
[40]
Whitt, W. 1991. The efficiency of one long run versus independent replications in steady-state simulation. Manage. Sci. 37, 645--666.
[41]
Wilson, J. R. and Pritsker, A. A. B. 1978a. A survey of research on the simulation startup problem. Simulation 31, 2, 55--58.
[42]
Wilson, J. R. and Pritsker, A. A. B. 1978b. Evaluation of startup policies in simulation experiments. Simulation 31, 3, 79--89.
[43]
Yeh, Y. and Schmeiser, B. W. 2001. On the MSE robustness of batching estimators. In Proceedings of the 2001 Winter Simulation Conference, B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 344--347.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation
ACM Transactions on Modeling and Computer Simulation  Volume 14, Issue 1
January 2004
114 pages
ISSN:1049-3301
EISSN:1558-1195
DOI:10.1145/974734
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 January 2004
Published in TOMACS Volume 14, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Batch means
  2. confidence intervals
  3. independent replications
  4. steady-state analysis
  5. stochastic simulation
  6. transient analysis
  7. variance estimation

Qualifiers

  • Article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)43
  • Downloads (Last 6 weeks)3
Reflects downloads up to 24 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2023)Batching Adaptive Variance ReductionACM Transactions on Modeling and Computer Simulation10.1145/357338633:1-2(1-24)Online publication date: 28-Feb-2023
  • (2023)Rebooting simulationIISE Transactions10.1080/24725854.2023.226102856:4(385-397)Online publication date: 8-Nov-2023
  • (2022)Automated and distributed statistical analysis of economic agent-based modelsJournal of Economic Dynamics and Control10.1016/j.jedc.2022.104458143(104458)Online publication date: Oct-2022
  • (2021)Convergence of covariance and spectral density estimates for high-dimensional locally stationary processesThe Annals of Statistics10.1214/20-AOS195449:1Online publication date: 1-Feb-2021
  • (2018)Estimation of the asymptotic variance of univariate and multivariate random fields and statistical inferenceElectronic Journal of Statistics10.1214/18-EJS139812:1Online publication date: 1-Jan-2018
  • (2017)Stochastic co-kriging for steady-state simulation metamodelingProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242326(1-12)Online publication date: 3-Dec-2017
  • (2017)A concise history of simulation output analysisProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242191(1-16)Online publication date: 3-Dec-2017
  • (2017)Gaussian approximation for high dimensional time seriesThe Annals of Statistics10.1214/16-AOS151245:5Online publication date: 1-Oct-2017
  • (2017)High‐order Corrected Estimator of Asymptotic Variance with Optimal BandwidthScandinavian Journal of Statistics10.1111/sjos.1227944:4(866-898)Online publication date: 19-Jun-2017
  • (2017)Stochastic co-kriging for steady-state simulation metamodeling2017 Winter Simulation Conference (WSC)10.1109/WSC.2017.8247913(1750-1761)Online publication date: Dec-2017
  • Show More Cited By

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media