×

Hierarchical-block conditioning approximations for high-dimensional multivariate normal probabilities. (English) Zbl 1430.62094

Summary: This paper presents a new method to estimate large-scale multivariate normal probabilities. The approach combines a hierarchical representation with processing of the covariance matrix that decomposes the \(n\)-dimensional problem into a sequence of smaller \(m\)-dimensional ones. It also includes a \(d\)-dimensional conditioning method that further decomposes the \(m\)-dimensional problems into smaller \(d\)-dimensional problems. The resulting two-level hierarchical-block conditioning method requires Monte Carlo simulations to be performed only in \(d\) dimensions, with \(d \ll n,\) and allows the complexity of the algorithm’s major cost to be \(O(n \log n)\). The run-time cost of the method depends on two parameters \(m\) and \(d\), where \(m\) represents the diagonal block size and controls the sizes of the blocks of the covariance matrix that are replaced by low-rank approximations, and \(d\) allows a trade-off of accuracy for expensive computations in the evaluation of the probabilities of \(m\)-dimensional blocks. We also introduce an inexpensive block reordering strategy to provide improved accuracy in the overall probability computation. The downside of this method, as with other such conditioning approximations, is the absence of an internal estimate of its error to use in tuning the approximation. Numerical simulations on problems from 2D spatial statistics with dimensions up to 16,384 indicate that the algorithm achieves a \(1\%\) error level and improves the run time over a one-level hierarchical Quasi-Monte Carlo method by a factor between 10 and 15.

MSC:

62G30 Order statistics; empirical distribution functions
62H12 Estimation in multivariate analysis
65C05 Monte Carlo methods

Software:

QSIMVN; R; LAPACK; sn

References:

[1] Anderson, E., Bai, Z., Bischof, C., Blackford, S., Demmel, J., Dongarra, J., Du Croz, J., Greenbaum, A., Hammerling, S., McKenney, A., et al.: LAPACK Users’ Guide, 3rd edn. SIAM Publications, Philadelphia (1999) · Zbl 0934.65030 · doi:10.1137/1.9780898719604
[2] Azzalini, A., Capitanio, A.: The Skew-Normal and Related Families. Cambridge University Press, Cambridge (2014) · Zbl 0924.62050
[3] Brown, B.M., Resnick, S.I.: Extreme values of independent stochastic processes. J. Appl. Probab. 14, 732-739 (1977) · Zbl 0384.60055 · doi:10.2307/3213346
[4] Caflisch, R.E.: Monte Carlo and quasi-Monte Carlo methods. Acta Numer. 7, 1-49 (1998) · Zbl 0949.65003 · doi:10.1017/S0962492900002804
[5] Connors, R.D., Hess, S., Daly, A.: Analytic approximations for computing probit choice probabilities. Transp. A Transp. Sci. 10, 119-139 (2014)
[6] Genton, M.G.: Skew-Elliptical Distributions and Their Applications: A Journey Beyond Normality. CRC Press, Boca Raton (2004) · Zbl 1069.62045 · doi:10.1201/9780203492000
[7] Genton, M.G., Keyes, D.E., Turkiyyah, G.M.: Hierarchical decompositions for the computation of high-dimensional multivariate normal probabilities. J. Comput. Graph. Stat. 27, 268-277 (2018) · Zbl 07498945 · doi:10.1080/10618600.2017.1375936
[8] Genz, A.: Numerical computation of multivariate normal probabilities. J. Comput. Graph. Stat. 1, 141-149 (1992)
[9] Genz, A., Bretz, F.: Computation of Multivariate Normal and t Probabilities, vol. 195. Springer, Berlin (2009) · Zbl 1204.62088
[10] Gupta, R.C., Brown, N.: Reliability studies of the skew-normal distribution and its application to a strength – stress model. Commun. Stat. Theory Methods 30, 2427-2445 (2001) · Zbl 1009.62513 · doi:10.1081/STA-100107696
[11] Hackbusch, W.: Hierarchical Matrices: Algorithms and Analysis, vol. 49. Springer, Berlin (2015) · Zbl 1336.65041 · doi:10.1007/978-3-662-47324-5
[12] Kamakura, W.A.: The estimation of multinomial probit models: a new calibration algorithm. Transp. Sci. 23, 253-265 (1989) · Zbl 0712.62105 · doi:10.1287/trsc.23.4.253
[13] Kan, R., Robotti, C.: On moments of folded and truncated multivariate normal distributions. J. Comput. Graph. Stat. 26(4), 930-934 (2017) · doi:10.1080/10618600.2017.1322092
[14] Kotz, S., Nadarajah, S.: Multivariate t-Distributions and Their Applications. Cambridge University Press, Cambridge (2004) · Zbl 1100.62059 · doi:10.1017/CBO9780511550683
[15] Mendell, N.R., Elston, R.: Multifactorial qualitative traits: genetic analysis and prediction of recurrence risks. Biometrics 30, 41-57 (1974) · doi:10.2307/2529616
[16] Morton, G.M.: A Computer Oriented Geodetic Data Base and a New Technique in File Sequencing. International Business Machines Company, New York (1966)
[17] Muthen, B.: Moments of the censored and truncated bivariate normal distribution. Br. J. Math. Stat. Psychol. 43, 131-143 (1990) · Zbl 0718.62031 · doi:10.1111/j.2044-8317.1990.tb00930.x
[18] Niederreiter, H.: New methods for pseudorandom numbers and pseudorandom vector generation. In: Proceedings of the 24th Conference on Winter Simulation, New York, NY, USA, WSC ’92, pp. 264-269. ACM (1992)
[19] Owen, A., Zhou, Y.: Safe and effective importance sampling. J. Am. Stat. Assoc. 95, 135-143 (2000) · Zbl 0998.65003 · doi:10.1080/01621459.2000.10473909
[20] R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2016)
[21] Schlather, M.: Models for stationary max-stable random fields. Extremes 5, 33-44 (2002) · Zbl 1035.60054 · doi:10.1023/A:1020977924878
[22] Smith, R., Tawn, J., Yuen, H.: Statistics of multivariate extremes. Int. Stat. Rev. 58, 47-58 (1990) · Zbl 0715.62095 · doi:10.2307/1403473
[23] Stewart, G.W.: The efficient generation of random orthogonal matrices with an application to condition estimators. SIAM J. Numer. Anal. 17, 403-409 (1980) · Zbl 0443.65027 · doi:10.1137/0717034
[24] Trinh, G., Genz, A.: Bivariate conditioning approximations for multivariate normal probabilities. Stat. Comput. 25, 989-996 (2015) · Zbl 1332.62055 · doi:10.1007/s11222-014-9468-y
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.