Multiple-comparison procedures for steady-state simulations. (English) Zbl 0894.68178
Summary: Suppose that there are \(k\geq 2\) different systems (i.e., stochastic processes), where each system has an unknown steady-state mean performance and unknown asymptotic variance. We allow for the asymptotic variances to be unequal and for the distributions of \(k\) systems to be different. We consider the problem of running independent, single-stage simulations to make multiple comparisons of the steady-state means of the different systems. We derive asymptotically valid (as the run length of the simulations of the systems tends to infinity) simultaneous confidence intervals for each of the following problems: all pairwise comparisons of means, all contrasts, multiple comparisons with a control and multiple comparisons with the best. Our confidence intervals are based on standardized time series methods, and we establish the asymptotic validity of each under the sole assumption that the stochastic processes representing the simulation output of the different systems satisfy a functional central limit theorem. Although simulation is the context of this paper, the results naturally apply to (asymptotically) stationary time series.
MSC:
| 68U20 | Simulation (MSC2010) |
| 62J15 | Paired and multiple comparisons; multiple testing |
| 65C05 | Monte Carlo methods |
| 62M10 | Time series, auto-correlation, regression, etc. in statistics (GARCH) |
| 60F17 | Functional limit theorems; invariance principles |
References:
| [1] | BANERJEE, S. K. 1961. On confidence interval for two-means problem based on separate estimates of variances and tabulated values of t-table. Sanky a Ser. A 23 359 378. Z. · Zbl 0102.14901 |
| [2] | BILLINGSLEY, P. 1968. Convergence of Probability Measures. Wiley, New York. Z. · Zbl 0172.21201 |
| [3] | BRATLEY, P., FOX, B. L. and SCHRAGE, L. E. 1987. A Guide to Simulation, 2nd ed. Springer, New York. Z. |
| [4] | CARLSTEIN, E. 1986. The use of subseries values for estimating the variance of a general statistic from a stationary sequence. Ann. Statist. 14 1171 1179. Z. · Zbl 0602.62029 · doi:10.1214/aos/1176350057 |
| [5] | CHEN, B.-C. and SARGENT, R. G. 1987. Using standardized time series to estimate confidence intervals for the difference between two stationary stochastic processes. Oper. Res. 35 428 436. Z. JSTOR: · Zbl 0634.62080 · doi:10.1287/opre.35.3.428 |
| [6] | DALAL, S. R. 1978. Simultaneous confidence interval procedures for univariate and multivariate Behrens Fisher ty pe problems. Biometrika 65 221 224. Z. JSTOR: · Zbl 0435.62039 |
| [7] | DAMERDJI, H. and NAKAy AMA, M. K. 1996. Two-stage multiple-comparison procedures for steady-state simulations. NJIT Research Report CIS-96-30, Newark, NJ. Z. |
| [8] | EDWARDS, D. G. and HSU, J. C. 1983. Multiple comparisons with the best treatment J. Amer. Statist. Assoc. 78 965 971. Z. JSTOR: · Zbl 0534.62050 · doi:10.2307/2288211 |
| [9] | ETHIER, S. N. and KURTZ, T. G. 1986. Markov Processes: Characterization and Convergence. Wiley, New York. Z. · Zbl 0592.60049 |
| [10] | GAMES, P. A. 1977. An improved table for simultaneous control of g contrasts. J. Amer. Statist. Assoc. 72 531 534. Z. |
| [11] | GLy NN, P. W. 1990. Diffusion approximations. In Handbooks in Operations Research and Z. Management Science D. Hey man and M. Sobel, eds. 2 145 198. Elsevier Science Publishers, New York. Z. · doi:10.1016/S0927-0507(05)80168-9 |
| [12] | GLy NN, P. W. and IGLEHART, D. L. 1990. Simulation output analysis using standardized time series. Math. Oper. Res. 14 1 16. Z. JSTOR: · Zbl 0704.65110 · doi:10.1287/moor.15.1.1 |
| [13] | GLy NN, P. W. and WHITT, W. 1987. Sufficient conditions for functional-limit-theorem versions of L W. Queueing Sy stems: Theory and Applications 1 279 287. Z. · Zbl 0681.60097 · doi:10.1007/BF01149539 |
| [14] | GLy NN, P. W. and WHITT, W. 1990. Estimating the asy mptotic variance with batch means. Oper. Res. Lett. 10 431 435. Z. · Zbl 0744.62113 · doi:10.1016/0167-6377(91)90019-L |
| [15] | GOLDSMAN, D., KANG, K. and SEILA, A. F. 1993. Cramer von Mises variance estimators for śimulations. Technical report, School of Industrial Sy stems Engineering, Georgia Inst. Technology, Atlanta. Z. |
| [16] | GOLDSMAN, D. and SCHRUBEN, L. W. 1984. Asy mptotic properties of some confidence interval estimators for simulation output. Management Sci. 30 1217 1225. Z. · Zbl 0557.62033 · doi:10.1287/mnsc.30.10.1217 |
| [17] | GOLDSMAN, L. and NELSON, B. L. 1990. Batch-size effects on simulation optmization using multiple comparisons with the best. In Proceedings of the 1990 Winter Simulation Z. Conference O. Balci, R. P. Sadowski and R. E. Nance, eds. 288 293. Institute of Electrical and Electronics Engineers, Piscataway, NJ. Z. |
| [18] | HOCHBERG, Y. and TAMHANE, A. C. 1987. Multiple Comparison Methods. Wiley, New York. Z. · Zbl 0731.62125 |
| [19] | HSU, J. C. 1981. Simultaneous confidence intervals for all distances from the “best.” Ann. Statist. 9 1026 1034. Z. · Zbl 0474.62031 · doi:10.1214/aos/1176345582 |
| [20] | HSU, J. C. 1984a. Ranking and selection and multiple comparisons with the best. In Design of Z. Z Experiments: Ranking and Selection Essay s in Honor of Robert E. Bechhofer T. J.. Santner and A. C. Tamhane, eds. 23 33. Dekker, New York. Z. · Zbl 1007.35009 · doi:10.1007/s002080100304 |
| [21] | HSU, J. C. 1984b. Constrained simultaneous confidence intervals for multiple comparisons with the best. Ann. Statist. 12 1136 1144. Z. · Zbl 0557.62023 · doi:10.1214/aos/1176346732 |
| [22] | MILLER, R. G. 1981. Simultaneous Statistical Inference, 2nd ed. Springer, New York. Z. · Zbl 0463.62002 |
| [23] | NAKAy AMA, M. K. 1994. Two-stage stopping procedures based on standardized time series. Management Sci. 40 1189 1206. · Zbl 0811.62092 · doi:10.1287/mnsc.40.9.1189 |
| [24] | NAKAy AMA, M. K. 1996a. Multiple comparisons with the best in steady-state simulations. In Proceedings of the Second International Workshop on Mathematical Methods in Z. Stochastic Simulation and Experimental Design S. M. Ermakov and V. B. Melas, eds. 230 235. Publishing House of Saint Petersburg Univ., Saint Petersburg, Russia. Z. |
| [25] | NAKAy AMA, M. K. 1996b. Multiple comparisons with the best using common random numbers in steady-state simulations. Research Report CIS-96-22, Computer and Information Science Dept., New Jersey Inst. Technology. Z. |
| [26] | NELSON, B. L. 1993. Robust multiple comparisons under common random numbers. ACM Trans. Modeling Computer Simulation 3 225 243. Z. · Zbl 0842.62061 · doi:10.1145/174153.174158 |
| [27] | NELSON, B. L. and HSU, J. C. 1993. Control-variate models of common random numbers for multiple comparison with the best. Management Sci. 39 989 1001. Z. · Zbl 0789.65106 · doi:10.1287/mnsc.39.8.989 |
| [28] | NEWMAN, C. M. and WRIGHT, A. L. 1981. An invariance principle for certain dependent sequences. Ann. Probab. 9 671 675. Z. · Zbl 0465.60009 · doi:10.1214/aop/1176994374 |
| [29] | SARGENT, R. G., KANG, K. and GOLDSMAN, D. 1992. An investigation of finite sample behavior of confidence interval estimation procedures. Oper. Res. 40 898 913. Z. · Zbl 0753.62062 · doi:10.1287/opre.40.5.898 |
| [30] | SCHMEISER, B. 1982. Batch size effects in the analysis of simulation output. Oper. Res. 30 556 568.Z. JSTOR: · Zbl 0484.65087 · doi:10.1287/opre.30.3.556 |
| [31] | SCHRUBEN, L. W. 1982. Detecting initialization bias in simulation output. Oper. Res. 30 569 590. Z. · Zbl 0484.65043 · doi:10.1287/opre.30.3.569 |
| [32] | SCHRUBEN, L. W. 1983. Confidence interval estimation using standardized time series. Oper. Res. 31 1090 1108. · Zbl 0532.62067 · doi:10.1287/opre.31.6.1090 |
| [33] | SIDAK, Z. 1967. Rectangular confidence regions for the means of multivariate normal distribuťions. J. Amer. Statist. Assoc. 62 626 633. Z. · Zbl 0158.17705 · doi:10.2307/2283989 |
| [34] | SLEPIAN, D. 1962. The one-sided barrier problem for Gaussian noise. Bell Sy st. Tech. J. 41 463 501. Z. SPøTVOLL, E. 1972. Joint confidence intervals for all linear combinations in ANOVA with unknown variances. Biometrika 59 684 685. Z. |
| [35] | TAMHANE, A. C. 1977. Multiple comparisons in model I one-way ANOVA with unequal variances. Comm. Statist. Theory Methods 6 15 32. Z. · Zbl 0365.62071 · doi:10.1080/03610927708827466 |
| [36] | TAMHANE, A. C. 1979. A comparison of procedures for multiple comparisons of means with unequal variances. J. Amer. Statist. Assoc. 74 471 480. Z. · Zbl 0428.62047 · doi:10.2307/2286358 |
| [37] | TOKOL, G., GOLDSMAN, D., OCKERMAN, D. J. and SWAIN, J. J. 1996. Standardized time series L -norm variance estimators for simulations. Management Sci. To appear. p Z. · Zbl 1180.62126 |
| [38] | TUKEY, J. W. 1953. The problem of multiple comparisons. Mimeographed monograph. Z. |
| [39] | YANG, W. and NELSON, B. L. 1991. Using common random numbers and control variates in multiple-comparison procedures. Oper. Res. 39 583 591. Z. · Zbl 0737.65117 · doi:10.1287/opre.39.4.583 |
| [40] | YUAN, M. and NELSON, B. L. 1993. Multiple comparisons with the best for steady-state simulation. ACM Trans. Modeling Computer Simulat. 3 66 79. · Zbl 0844.60081 · doi:10.1145/151527.151551 |
| [41] | NEWARK, NEW JERSEY 07102 E-MAIL: marvin@cis.njit.edu |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
