Higher-order coverage errors of batching methods via Edgeworth expansions on \(t\)-statistics. (English) Zbl 07928780
Summary: While batching methods have been widely used in simulation and statistics, their higher-order coverage behaviors and relative advantages in this regard remain open. We develop techniques to obtain higher-order coverage errors for batching methods by building Edgeworth-type expansions on \(t\)-statistics. The coefficients in these expansions are intricate analytically, but we provide algorithms to estimate the coefficients of the \(n^{- 1}\) error terms via Monte Carlo simulation. We provide insights on the effect of the number of batches on the coverage error, where we demonstrate generally nonmonotonic relations. We also compare different batching methods both theoretically and numerically, and argue that none of the methods is uniformly better than others in terms of coverage. However, when the number of batches is large, sectioned jackknife has the best coverage among all.
MSC:
| 62E20 | Asymptotic distribution theory in statistics |
| 62F40 | Bootstrap, jackknife and other resampling methods |
References:
| [1] | ALEXOPOULOS, C., ARGON, N. T., GOLDSMAN, D., STEIGER, N. M., TOKOL, G. and WILSON, J. R. (2007a). Efficient computation of overlapping variance estimators for simulation. INFORMS J. Comput. 19 314-327. Digital Object Identifier: 10.1287/ijoc.1060.0198 Google Scholar: Lookup Link MathSciNet: MR2344067 · Zbl 1241.62027 · doi:10.1287/ijoc.1060.0198 |
| [2] | ALEXOPOULOS, C., ARGON, N. T., GOLDSMAN, D., TOKOL, G. and WILSON, J. R. (2007b). Overlapping variance estimators for simulation. Oper. Res. 55 1090-1103. Digital Object Identifier: 10.1287/opre.1070.0475 Google Scholar: Lookup Link MathSciNet: MR2372279 · Zbl 1167.62390 · doi:10.1287/opre.1070.0475 |
| [3] | ALEXOPOULOS, C., BOONE, J. H., GOLDSMAN, D., LOLOS, A., DINGEÇ, K. D. and WILSON, J. R. (2020). Steady-state quantile estimation using standardized time series. In Proceedings of the 2020 Winter Simulation Conference 289-300. |
| [4] | ALEXOPOULOS, C., GOLDSMAN, D., MOKASHI, A., NIE, R., SUN, Q., TIEN, K.-W. and WILSON, J. R. (2014). Sequest: A sequential procedure for estimating steady-state quantiles. In Proceedings of the 2014 Winter Simulation Conference 662-673. |
| [5] | ALEXOPOULOS, C., GOLDSMAN, D., MOKASHI, A. C. and WILSON, J. R. (2019). Sequential estimation of steady-state quantiles: Some new developments in methods and software. In Proceedings of the 2019 Winter Simulation Conference 3774-3785. |
| [6] | ALEXOPOULOS, C., GOLDSMAN, D., TANG, P. and WILSON, J. R. (2013). A sequential procedure for estimating the steady-state mean using standardized time series. In Proceedings of the 2013 Winter Simulation Conference 613-622. |
| [7] | ALEXOPOULOS, C., GOLDSMAN, D., TANG, P. and WILSON, J. R. (2016). SPSTS: A sequential procedure for estimating the steady-state mean using standardized time series. IIE Trans. 48 864-880. |
| [8] | ALEXOPOULOS, C. and SEILA, A. F. (1996). Implementing the batch means method in simulation experiments. In Proceedings of the 1996 Winter Simulation Conference 214-221. |
| [9] | ANTONINI, C., ALEXOPOULOS, C., GOLDSMAN, D. and WILSON, J. R. (2009). Area variance estimators for simulation using folded standardized time series. IIE Trans. 41 134-144. |
| [10] | Asmussen, S. and Glynn, P. W. (2007). Stochastic Simulation: Algorithms and Analysis. Stochastic Modelling and Applied Probability 57. Springer, New York. MathSciNet: MR2331321 |
| [11] | Bhattacharya, R. N. and Ghosh, J. K. (1978). On the validity of the formal Edgeworth expansion. Ann. Statist. 6 434-451. Digital Object Identifier: 10.1214/aos/1176344134 Google Scholar: Lookup Link MathSciNet: MR471142 · Zbl 0396.62010 · doi:10.1214/aos/1176344134 |
| [12] | BHATTACHARYA, R. N. and PURI, M. L. (1983). On the order of magnitude of cumulants of von Mises functionals and related statistics. Ann. Probab. 11 346-354. MathSciNet: MR0690132 · Zbl 0527.62025 |
| [13] | BICKEL, P. J., GÖTZE, F. and VAN ZWET, W. R. (1997). Resampling fewer than n observations: Gains, losses, and remedies for losses. Statist. Sinica 7 1-31. · Zbl 0927.62043 |
| [14] | CALLAERT, H., JANSSEN, P. and VERAVERBEKE, N. (1980). An Edgeworth expansion for \(U\)-statistics. Ann. Statist. 8 299-312. MathSciNet: MR0560731 · Zbl 0428.62016 |
| [15] | CALVIN, J. M. and NAKAYAMA, M. K. (2006). Permuted standardized time series for steady-state simulations. Math. Oper. Res. 31 351-368. Digital Object Identifier: 10.1287/moor.1050.0183 Google Scholar: Lookup Link MathSciNet: MR2234001 · Zbl 1278.62138 · doi:10.1287/moor.1050.0183 |
| [16] | CALVIN, J. M. and NAKAYAMA, M. K. (2013). Confidence intervals for quantiles with standardized time series. In Proceedings of the 2013 Winter Simulations Conference 601-612. |
| [17] | CHIEN, C. (1994). Batch size selection for the batch means method. In Proceedings of the 1994 Winter Simulation Conference 345-352. |
| [18] | DONG, J. and GLYNN, P. (2019a). A new approach to sequential stopping for stochastic simulation. |
| [19] | DONG, J. and GLYNN, P. W. (2019b). The asymptotic validity of sequential stopping rules for confidence interval construction using standardized time series. In Proceedings of the 2019 Winter Simulation Conference 332-343. |
| [20] | DUCHI, J. C., GLYNN, P. W. and NAMKOONG, H. (2021). Statistics of robust optimization: A generalized empirical likelihood approach. Math. Oper. Res. 46 946-969. Digital Object Identifier: 10.1287/moor.2020.1085 Google Scholar: Lookup Link MathSciNet: MR4312583 · Zbl 1473.62292 · doi:10.1287/moor.2020.1085 |
| [21] | Flegal, J. M. and Jones, G. L. (2010). Batch means and spectral variance estimators in Markov chain Monte Carlo. Ann. Statist. 38 1034-1070. Digital Object Identifier: 10.1214/09-AOS735 Google Scholar: Lookup Link MathSciNet: MR2604704 · Zbl 1184.62161 · doi:10.1214/09-AOS735 |
| [22] | GEYER, C. J. (1992). Practical Markov chain Monte Carlo. Statist. Sci. 7 473-483. |
| [23] | GLYNN, P. W. and IGLEHART, D. L. (1990). Simulation output analysis using standardized time series. Math. Oper. Res. 15 1-16. Digital Object Identifier: 10.1287/moor.15.1.1 Google Scholar: Lookup Link MathSciNet: MR1038232 · Zbl 0704.65110 · doi:10.1287/moor.15.1.1 |
| [24] | GLYNN, P. W. and LAM, H. (2018). Constructing simulation output intervals under input uncertainty via data sectioning. In Proceedings of the 2018 Winter Simulation Conference 1551-1562. |
| [25] | GOLDSMAN, D. and SCHRUBEN, L. (1990). New confidence interval estimators using standardized time series. Manage. Sci. 36 393-397. · Zbl 0716.62037 |
| [26] | GÖTZE, F. (1984). Expansions for von Mises functionals. Z. Wahrsch. Verw. Gebiete 65 599-625. Digital Object Identifier: 10.1007/BF00531841 Google Scholar: Lookup Link MathSciNet: MR0736148 · Zbl 0518.60051 · doi:10.1007/BF00531841 |
| [27] | GÖTZE, F. and HIPP, C. (1983). Asymptotic expansions for sums of weakly dependent random vectors. Z. Wahrsch. Verw. Gebiete 64 211-239. Digital Object Identifier: 10.1007/BF01844607 Google Scholar: Lookup Link MathSciNet: MR0714144 · Zbl 0497.60022 · doi:10.1007/BF01844607 |
| [28] | HAAS, P. J. (2002). Regenerative Simulation. In Stochastic Petri Nets: Modelling, Stability, Simulation 189-273. Springer, New York, NY. · Zbl 1052.90001 |
| [29] | Hall, P. (1992). The Bootstrap and Edgeworth Expansion. Springer Series in Statistics. Springer, New York. Digital Object Identifier: 10.1007/978-1-4612-4384-7 Google Scholar: Lookup Link MathSciNet: MR1145237 · Zbl 0829.62021 · doi:10.1007/978-1-4612-4384-7 |
| [30] | HE, S. and LAM, H. (2021). Higher-order coverage error analysis for batching and sectioning. In Proceedings of 2021 Winter Simulation Conference 1-12. |
| [31] | HE, S. and LAM, H. (2024a). Supplement to “Higher-order coverage errors of batching methods via Edgeworth expansions on \(t\)-statistics.” https://doi.org/10.1214/24-AOS2377SUPPA |
| [32] | HE, S. and LAM, H. (2024b). Codes for “Higher-order coverage errors of batching methods via Edgeworth expansions on \(t\)-statistics.” https://doi.org/10.1214/24-AOS2377SUPPB |
| [33] | JENSEN, J. L. (1989). Asymptotic expansions for strongly mixing Harris recurrent Markov chains. Scand. J. Stat. 16 47-63. MathSciNet: MR1003968 · Zbl 0674.60067 |
| [34] | Jones, G. L., Haran, M., Caffo, B. S. and Neath, R. (2006). Fixed-width output analysis for Markov chain Monte Carlo. J. Amer. Statist. Assoc. 101 1537-1547. Digital Object Identifier: 10.1198/016214506000000492 Google Scholar: Lookup Link MathSciNet: MR2279478 · Zbl 1171.62316 · doi:10.1198/016214506000000492 |
| [35] | Kleiner, A., Talwalkar, A., Sarkar, P. and Jordan, M. I. (2014). A scalable bootstrap for massive data. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 795-816. Digital Object Identifier: 10.1111/rssb.12050 Google Scholar: Lookup Link MathSciNet: MR3248677 · Zbl 07555464 · doi:10.1111/rssb.12050 |
| [36] | LADA, E. K. and WILSON, J. R. (2006). A wavelet-based spectral procedure for steady-state simulation analysis. European J. Oper. Res. 174 1769-1801. · Zbl 1103.62087 |
| [37] | LEE, S. M. S. and YOUNG, G. A. (1995). Asymptotic iterated bootstrap confidence intervals. Ann. Statist. 23 1301-1330. Digital Object Identifier: 10.1214/aos/1176324710 Google Scholar: Lookup Link MathSciNet: MR1353507 · Zbl 0838.62034 · doi:10.1214/aos/1176324710 |
| [38] | LEWIS, P. A. W. and ORAV, E. J. (1989). Simulation Methodology for Statisticians, Operations Analysts, and Engineers. Vol. I. The Wadsworth & Brooks/Cole Statistics/Probability Series. Wadsworth & Brooks/Cole Advanced Books & Software, Pacific Grove, CA. MathSciNet: MR0969125 · Zbl 0666.65008 |
| [39] | MALINOVSKII, V. K. (1987). Limit theorems for Harris Markov chains, I. Theory Probab. Appl. 31 269-317. |
| [40] | MEKETON, M. S. and SCHMEISER, B. (1984). Overlapping batch means: Something for nothing? In Proceedings of the 16th Conference on Winter Simulation. WSC ’84 226-230. IEEE Press, New York. |
| [41] | MUÑOZ, D. F. and GLYNN, P. W. (1997). A batch means methodology for estimation of a nonlinear function of a steady-state mean. Manage. Sci. 43 1121-1135. · Zbl 0908.62088 |
| [42] | NAKAYAMA, M. K. (2007). Fixed-width multiple-comparison procedures using common random numbers for steady-state simulations. European J. Oper. Res. 182 1330-1349. · Zbl 1128.62082 |
| [43] | NAKAYAMA, M. K. (2014). Confidence intervals for quantiles using sectioning when applying variance-reduction techniques. ACM Trans. Model. Comput. Simul. 24 19. Digital Object Identifier: 10.1145/2558328 Google Scholar: Lookup Link MathSciNet: MR3287088 · Zbl 1322.68251 · doi:10.1145/2558328 |
| [44] | Politis, D. N., Romano, J. P. and Wolf, M. (1999). Subsampling. Springer Series in Statistics. Springer, New York. Digital Object Identifier: 10.1007/978-1-4612-1554-7 Google Scholar: Lookup Link MathSciNet: MR1707286 · Zbl 0931.62035 · doi:10.1007/978-1-4612-1554-7 |
| [45] | POPE, A. (1995). Improving confidence intervals obtained by sectioning in Monte Carlo analysis. In MODSIM 1995 International Congress on Modelling and Simulation. November 27th-30th 150-154, Newcastle, NSW, Australia. |
| [46] | SCHMEISER, B. (1982). Batch size effects in the analysis of simulation output. Oper. Res. 30 556-568. Digital Object Identifier: 10.1287/opre.30.3.556 Google Scholar: Lookup Link MathSciNet: MR0659055 · Zbl 0484.65087 · doi:10.1287/opre.30.3.556 |
| [47] | SCHRUBEN, L. (1983). Confidence interval estimation using standardized time series. Oper. Res. 31 1090-1108. · Zbl 0532.62067 |
| [48] | SKOVGAARD, I. M. (1986). On multivariate Edgeworth expansions. Int. Stat. Rev. 54 169-186. Digital Object Identifier: 10.2307/1403142 Google Scholar: Lookup Link MathSciNet: MR0962933 · Zbl 0615.62018 · doi:10.2307/1403142 |
| [49] | SONG, W. T. and SCHMEISER, B. W. (1995). Optimal mean-squared-error batch sizes. Manage. Sci. 41 110-123. · Zbl 0819.62076 |
| [50] | STEIGER, N. M., LADA, E. K., WILSON, J. R., ALEXOPOULOS, C., GOLDSMAN, D. and ZOUAOUI, F. (2002). ASAP2: An improved batch means procedure for simulation output analysis. In Proceedings of the 2002 Winter Simulation Conference 336-344. |
| [51] | STEIGER, N. M., LADA, E. K., WILSON, J. R., JOINES, J. A., ALEXOPOULOS, C. and GOLDSMAN, D. (2005). ASAP3: A batch means procedure for steady-state simulation analysis. ACM Trans. Model. Comput. Simul. 15 39-73. · Zbl 1478.62244 |
| [52] | STEIGER, N. M. and WILSON, J. R. (2002). An improved batch means procedure for simulation output analysis. Manage. Sci. 48 1569-1586. |
| [53] | SU, Z., PASUPATHY, R., YEH, Y. and GLYNN, P. W. (2024). Overlapping batch confidence intervals on statistical functionals constructed from time series: Application to quantiles, optimization, and estimation. ACM Trans. Model. Comput. Simul. 34 10. MathSciNet: MR4751429 |
| [54] | TAFAZZOLI, A., STEIGER, N. M. and WILSON, J. R. (2011). \(N\)-Skart: A nonsequential skewness- and autoregression-adjusted batch-means procedure for simulation analysis. IEEE Trans. Automat. Control 56 254-264. Digital Object Identifier: 10.1109/TAC.2010.2052137 Google Scholar: Lookup Link MathSciNet: MR2761085 · Zbl 1368.62248 · doi:10.1109/TAC.2010.2052137 |
| [55] | TAFAZZOLI, A., WILSON, J. R., LADA, E. K. and STEIGER, N. M. (2008). Skart: A skewness- and autoregression-adjusted batch-means procedure for simulation analysis. In 2008 Winter Simulation Conference 387-395. |
| [56] | TAKAHASHI, H. (1988). A note on Edgeworth expansion for the von Mises functionals. J. Multivariate Anal. 24 56-65. Digital Object Identifier: 10.1016/0047-259X(88)90102-9 Google Scholar: Lookup Link MathSciNet: MR0925130 · Zbl 0663.62029 · doi:10.1016/0047-259X(88)90102-9 |
| [57] | WITHERS, C. S. (1983). Expansions for the distribution and quantiles of a regular functional of the empirical distribution with applications to nonparametric confidence intervals. Ann. Statist. 11 577-587. · Zbl 0531.62015 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
