Performance evaluation of output analysis methods in steady-state simulations. (English) Zbl 1386.65067
Summary: Output analysis methods of steady-state simulations have extensively been subject of study to evaluate the performance when estimating the mean. However, smaller efforts have been placed on performance evaluation of these methods to estimate variance and quantiles. In this paper, we empirically evaluate the performance of output analysis methods based on multiple replications and batches to estimate mean, variance and quantile with the same set of data. The evaluation of the performance of the methods is based on the empirical coverage of the true value using confidence intervals, the average bias, relative error and mean squared error. The methods are applied to estimate the average, variance and quantiles of waiting time in an M/M/1 queue. The results show that the methods based on non-overlapping batches perform consistently well in all the metrics. The performance of the other methods varies depending on the metric and the parameters of the simulation. In addition, we provide another example of a non-geometric ergodic Markov chain to show that asymptotically valid confidence intervals for quantiles can be obtained using batches and replications.
MSC:
| 65C99 | Probabilistic methods, stochastic differential equations |
| 60J22 | Computational methods in Markov chains |
| 62F10 | Point estimation |
| 62F25 | Parametric tolerance and confidence regions |
References:
| [1] | Asif, H.; El-Alfy, E., Performance evaluation of queuing disciplines for multi-class traffic using opnet simulator, (Proceedings of the 7th WSEAS International Conference on Mathematical Methods and Computational Techniques in Electrical Engineering (2005), WSEAS), 1-6 |
| [2] | Onan, K., A comparative study of production control systems through simulation, (Proceedings of the 12th WSEAS International Conference on Applied Mathematics (2007), WSEAS), 377-382 |
| [3] | Chang-Hyun, P.; Kang-Bae, L.; Hyung-Rim, C.; Soon-Goo, H.; Min-Je, C., Developing a simulation model for the effects of introducing a sharing economy among smes: focusing on the shipping and handling tasks at distribution center, (Proceedings of the 8th WSEAS International Conference on Computer Engineering and Applications (2014), WSEAS), 166-172 |
| [4] | Mose, G.; Mario, D.; Murino, T.; Carmela, S., A simulation based approach for improving healthcare systems, (Proceedings of the 6th WSEAS European Computing Conference (2012), WSEAS), 387-393 |
| [7] | Law, A., Simulation Modeling and Analysis (2014), McGraw-Hill: McGraw-Hill New York, NY |
| [8] | Glynn, P. W.; Heidelberger, P., Analysis of initial transient deletion for replicated steady-state simulations, Oper. Res. Lett., 10, 8, 437-443 (1991) · Zbl 0752.65100 |
| [9] | Grassmann, W. K., Factors affecting warm-up periods in discrete event simulation, Simulation, 90, 1, 11-23 (2014) |
| [10] | Schmeiser, B., Batch size effects in the analysis of simulation output, Oper. Res., 30, 3, 556-568 (1982) · Zbl 0484.65087 |
| [11] | Law, A. M., Confidence intervals in discrete event simulation: a comparison of replication and batch means, Naval Res. Logist. Q., 24, 4, 667-678 (1977) · Zbl 0415.62065 |
| [12] | Argon, N. T.; Andradóttir, S., Replicated batch means for steady-state simulations, Naval Res. Logist. (NRL), 53, 6, 508-524 (2006) · Zbl 1105.62035 |
| [13] | Muñoz, D. F.; Glynn, P. W., A batch means methodology for estimation of a nonlinear function of a steady-state mean, Manage. Sci., 43, 8, 1121-1135 (1997) · Zbl 0908.62088 |
| [15] | Seila, A. F., A batching approach to quantile estimation in regenerative simulations, Manage. Sci., 28, 5, 573-581 (1982) · Zbl 0484.65088 |
| [16] | Muñoz, D. F., On the validity of the batch quantile method for Markov chains, Oper. Res. Lett., 38, 3, 223-226 (2010) · Zbl 1187.90312 |
| [17] | Alexopoulos, C.; Goldsman, D.; Wilson, J. R., A new perspective on batched quantile estimation, (Proceedings of the 2012 Winter Simulation Conference (2012), IEEE), 1-11 |
| [18] | Muñoz, D. F.; Ramírez-López, A., A note on bias and mean squared error in steady-state quantile estimation, Oper. Res. Lett., 43, 4, 374-377 (2015) · Zbl 1408.62059 |
| [19] | Jin, X.; Fu, M. C.; Xiong, X., Probabilistic error bounds for simulation quantile estimators, Manage. Sci., 49, 2, 230-246 (2003) · Zbl 1232.91349 |
| [20] | Alexopoulos, C.; Goldsman, D.; Mokashi, A.; Nie, R.; Sun, Q.; Tien, K.-W.; Wilson, J. R., Sequest: a sequential procedure for estimating steady-state quantiles, (Proceedings of the 2014 Winter Simulation Conference (2014), IEEE), 662-673 |
| [21] | Matsumoto, M.; Nishimura, T., Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator, ACM Trans. Model. Comput. Simul. (TOMACS), 8, 1, 3-30 (1998) · Zbl 0917.65005 |
| [22] | Gross, D., Fundamentals of Queueing Theory (2008), John Wiley & Sons · Zbl 1151.60001 |
| [23] | Meyn, S. P.; Tweedie, R. L., Markov Chains and Stochastic Stability (2012), Springer Science & Business Media · Zbl 0925.60001 |
| [24] | Janssens, G. K.; Deceuninck, W.; Van Breedam, A., Opportunities of robust regression for variance reduction in discrete event simulation, J. Comput. Appl. Math., 64, 1, 163-176 (1995) · Zbl 0847.62060 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
