Cross-fertilizing strategies for better EM mountain climbing and DA field exploration: a graphical guide book. (English) Zbl 1329.62040
Summary: In recent years, a variety of extensions and refinements have been developed for data augmentation based model fitting routines. These developments aim to extend the application, improve the speed and/or simplify the implementation of data augmentation methods, such as the deterministic EM algorithm for mode finding and stochastic Gibbs sampler and other auxiliary-variable based methods for posterior sampling. In this overview article we graphically illustrate and compare a number of these extensions, all of which aim to maintain the simplicity and computation stability of their predecessors. We particularly emphasize the usefulness of identifying similarities between the deterministic and stochastic counterparts as we seek more efficient computational strategies. We also demonstrate the applicability of data augmentation methods for handling complex models with highly hierarchical structure, using a high-energy high-resolution spectral imaging model for data from satellite telescopes, such as the Chandra X-ray Observatory.
MSC:
| 62-07 | Data analysis (statistics) (MSC2010) |
| 62-02 | Research exposition (monographs, survey articles) pertaining to statistics |
| 62A09 | Graphical methods in statistics |
Keywords:
AECM; blocking; collapsing; conditional augmentation; ECM; ECME; efficient augmentation; data augmentation; Gibbs sampling; marginal augmentation; model reduction; NEM; nestingReferences:
| [1] | Amit, Y. (1991). On rates of convergence of scholastic relaxation for Gaussian and non-Gaussian distributions. J. Multiple Anal. 38 82-89. · Zbl 0735.60036 · doi:10.1016/0047-259X(91)90033-X |
| [2] | Besag, J. and Green, P. J. (1993). Spatial statistics and Bayesian computation. J. Roy. Statist. Soc. Ser. B 55 25-37. JSTOR: · Zbl 0800.62572 |
| [3] | Dempster, A. P., Laird, N. M. and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm (with discussion). J. Roy. Statist. Soc. Ser. B 39 1-37. JSTOR: · Zbl 0364.62022 |
| [4] | Elvis, M., Matsuoka, M., Siemiginowska, A., Fiore, F., Mihara, T. and Brinkmann, W. (1994). An ASCA GIS spectrum of S5 0014+813 at z =3.384. The Astrophysical Journal 436 L55-L58. |
| [5] | Fessler, J. A. and Hero, A. O. (1994). Space-alternating generalized expectation-maximization algorithm. IEEE Trans. Signal Process. 42 2664-2677. |
| [6] | Fessler, J. A. and Hero, A. O. (1995). Penalized maximum-likelihood image reconstruction using space-alternating generalized EM algorithm. IEEE Trans. Image Process. 4 1417-1438. |
| [7] | Foulley, J.-L. and van Dyk, D. A. (2000). The PX-EM algorithm for fast stable fitting of Henderson’s mixed model. Genetics Selective Evolution 32 143-163. |
| [8] | Gelfand, A. E., Sahu, S. K. and Carlin, B. P. (1995). Efficient parameterization for normal linear mixed models. Biometrika 82 479-488. JSTOR: · Zbl 0832.62064 · doi:10.1093/biomet/82.3.479 |
| [9] | Gelman, A., Carlin, J. B., Stern, H. S. and Rubin, D. B. (2003). Bayesian Data Analysis , 2nd ed. Chapman & Hall, London. · Zbl 1279.62004 |
| [10] | Gelman, A., van Dyk, D. A., Huang, Z. and Boscardin, W. J. (2008). Transformation and parameter-expanded Gibbs samplers for multilevel and generalized linear models. J. Comput. Graph. Statist. 17 95-122. |
| [11] | Geman, S. and Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Analysis and Machine Intelligence 6 721-741. · Zbl 0573.62030 · doi:10.1109/TPAMI.1984.4767596 |
| [12] | Ghosh, J. and Dunson, D. (2009). Default priors and efficient posterior computation in Bayesian factor analysis. J. Comput. Graph. Statist. 18 306-320. |
| [13] | Green, P. J. (1990). On use of the EM algorithm for penalized likelihood estimation. J. Roy. Statist. Soc. Ser. B 52 443-452. JSTOR: · Zbl 0706.62022 |
| [14] | Hans, C. M. and van Dyk, D. A. (2003). Accounting for absorption lines in high energy spectra. In Statistical Challenges in Modern Astronomy III (E. Feigelson and G. Babu, eds.) 429-430. Springer, New York. |
| [15] | Higdon, D. M. (1998). Auxiliary variable methods for Markov chain Monte Carlo with applications. J. Amer. Statist. Assoc. 93 585-595. · Zbl 0953.62103 · doi:10.2307/2670110 |
| [16] | Hobert, J. P. (2001). Discussion of “The art of data augmentation,” by D. A. van Dyk and X. L. Meng. J. Comput. Graph. Statist. 10 59-68. JSTOR: · doi:10.1198/10618600152418584 |
| [17] | Hobert, J. P. and Marchev, D. (2008). A theoretical comparison of the data augmentation, marginal augmentation and PX-DA algorithms. Ann. Statist. 36 532-554. · Zbl 1155.60031 · doi:10.1214/009053607000000569 |
| [18] | Imai, K. and van Dyk, D. A. (2005a). A Bayesian analysis of the multinomial probit model using marginal data augmentation. J. Econometrics 124 311-334. · Zbl 1335.62049 · doi:10.1016/j.jeconom.2004.02.002 |
| [19] | Imai, K. and van Dyk, D. A. (2005b). MNP: R package for fitting multinomial the probit model. J. Statist. Software 14 . |
| [20] | Liu, C. and Rubin, D. B. (1994). The ECME algorithm: A simple extension of EM and ECM with faster monotone convergence. Biometrika 81 633-648. JSTOR: · Zbl 0812.62028 · doi:10.1093/biomet/81.4.633 |
| [21] | Liu, C. and Rubin, D. B. (1995). ML estimation of the t distribution using EM and its extensions, ECM and ECME. Statist. Sinica 5 19-39. · Zbl 0824.62047 |
| [22] | Liu, C., Rubin, D. B. and Wu, Y. N. (1998). Parameter expansion for EM acceleration-the PXEM algorithm. Biometrika 75 755-770. JSTOR: · Zbl 0921.62071 · doi:10.1093/biomet/85.4.755 |
| [23] | Liu, J. S. (1994). The fraction of missing information and convergence rate for data augmentation. In Computing Science and Statistics. Computationally Intensive Statistical Methods. Proceedings of the 26th Symposium on the Interface 490-497. Interface Foundation of North America, Fairfax Station, VA. |
| [24] | Liu, J. S. (2001). Monte Carlo Strategies in Scientific Computing . Springer, New York. · Zbl 0991.65001 |
| [25] | Liu, J. S., Wong, W. H. and Kong, A. (1994). Covariance structure of the Gibbs sampler with applications to comparisons of estimators and augmentation schemes. Biometrika 81 27-40. JSTOR: · Zbl 0811.62080 · doi:10.1093/biomet/81.1.27 |
| [26] | Liu, J. S. and Wu, Y. N. (1999). Parameter expansion for data augmentation. J. Amer. Statist. Assoc. 94 1264-1274. JSTOR: · Zbl 1069.62514 · doi:10.2307/2669940 |
| [27] | Marchev, D. and Hobert, J. P. (2004). Geometric ergodicity of van Dyk and Meng’s algorithm for the multivariate student’s t model. J. Amer. Statist. Assoc. 99 228-238. · Zbl 1089.60518 · doi:10.1198/016214504000000223 |
| [28] | Meng, X.-L. (1994). On the rate of convergence of the ECM algorithm. Ann. Statist. 22 326-339. · Zbl 0803.65146 · doi:10.1214/aos/1176325371 |
| [29] | Meng, X.-L. (1997). The EM algorithm and medical studies: A historical link. Stat. Methods Med. Res. 6 3-23. |
| [30] | Meng, X.-L. and Rubin, D. B. (1991). Using EM to obtain asymptotic variance-covariance matrices: The SEM algorithm. J. Amer. Statist. Assoc. 86 899-909. |
| [31] | Meng, X.-L. and Rubin, D. B. (1993). Maximum likelihood estimation via the ECM algorithm: A general framework. Biometrika 80 267-278. JSTOR: · Zbl 0778.62022 · doi:10.1093/biomet/80.2.267 |
| [32] | Meng, X.-L. and Rubin, D. B. (1994). On the global and componentwise rates of convergence of the EM algorithm. Linear Algebra Appl. 199 413-425. · Zbl 0818.65153 · doi:10.1016/0024-3795(94)90363-8 |
| [33] | Meng, X.-L. and van Dyk, D. A. (1997). The EM algorithm-an old folk song sung to a fast new tune (with discussion). J. Roy. Statist. Soc. Ser. B 59 511-567. JSTOR: · Zbl 1090.62518 · doi:10.1111/1467-9868.00082 |
| [34] | Meng, X.-L. and van Dyk, D. A. (1998). Fast EM implementations for mixed-effects models. J. Roy. Statist. Soc. Ser. B 60 559-578. · Zbl 0909.62073 · doi:10.1111/1467-9868.00140 |
| [35] | Meng, X.-L. and van Dyk, D. A. (1999). Seeking efficient data augmentation schemes via conditional and marginal augmentation. Biometrika 86 301-320. JSTOR: · Zbl 1054.62505 · doi:10.1093/biomet/86.2.301 |
| [36] | Navidi, W. (1997). A graphical illustration of the EM algorithm. Amer. Statist. 51 29-31. |
| [37] | Park, T. and van Dyk, D. A. (2009). Partially collapsed Gibbs samplers: Illustrations and applications. J. Comput. Graph. Statist. 18 283-305. |
| [38] | Park, T., van Dyk, D. A. and Siemiginowska, A. (2008). Searching for narrow emission lines in X-ray spectra: Computation and methods. The Astrophysical Journal 688 807-825. |
| [39] | Pilla, R. S. and Lindsay, B. G. (2001). Alternative EM methods for nonparametric finite mixture models. Biometrika 88 535-550. JSTOR: · Zbl 0984.62024 · doi:10.1093/biomet/88.2.535 |
| [40] | Pope, C. A. and Wong, Y. (2005). Nested Monte Carlo EM algorithm for switching state-space models. IEEE Trans. Knowledge Data Engineering 17 1653-1663. |
| [41] | Protassov, R., van Dyk, D. A., Connors, A., Kashyap, V. and Siemiginowska, A. (2002). Statistics: Handle with care-detecting multiple model components with the likelihood ratio test. The Astrophysical Journal 571 545-559. |
| [42] | Roberts, G. O. (1996). Markov chain concepts related to sampling algorithms. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) 45-57. Chapman & Hall, London. · Zbl 0839.62078 |
| [43] | Schafer, J. L. (1997). Analysis of Incomplete Multivariate Data . Chapman & Hall, London. · Zbl 0997.62510 |
| [44] | Tanner, M. A. and Wong, W. H. (1987). The calculation of posterior distributions by data augmentation (with discussion). J. Amer. Statist. Assoc. 82 528-550. JSTOR: · Zbl 0619.62029 · doi:10.2307/2289457 |
| [45] | Tierney, L. (1994). Markov chains for exploring posterior distributions (with discussion). Ann. Statist. 22 1701-1762. · Zbl 0829.62080 · doi:10.1214/aos/1176325750 |
| [46] | Tierney, L. (1996). Introduction to general state-space Markov chain theory. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) 59-74. Chapman & Hall, London. · Zbl 0849.60072 |
| [47] | Vaida, F. (2005). Convergence of the EM and MM algorithms. Statist. Sinica 15 831-840. · Zbl 1087.62035 |
| [48] | van Dyk, D. and Park, T. (2004). Efficient EM-type algorithms for fitting spectral lines in high-energy astrophysics. In Applied Bayesian Modeling and Causal Inference from Incomplete-Data Perspectives: Contributions by Donald Rubin’s Statistical Family (A. Gelman and X.-L. Meng, eds.) 285-296. Wiley, New York. · Zbl 05274825 |
| [49] | van Dyk, D. and Park, T. (2008). Partially collapsed Gibbs samplers: Theory and methods. J. Amer. Statist. Assoc. 103 790-796. · Zbl 1471.62198 · doi:10.1198/016214508000000409 |
| [50] | van Dyk, D. A. (2000a). Fitting mixed-effects models using efficient EM-type algorithms. J. Comput. Graph. Statist. 9 78-98. JSTOR: · doi:10.2307/1390614 |
| [51] | van Dyk, D. A. (2000b). Nesting EM algorithms for computational efficiency. Statist. Sinica 10 203-225. · Zbl 1009.62018 |
| [52] | van Dyk, D. A. (2009). Marginal MCMC Methods. Statist. Sinica . |
| [53] | van Dyk, D. A., Connors, A., Esch, D. N., Freeman, P., Kang, H., Karovska, M., Kashyap, V., Siemiginowska, A. and Zezas, A. (2006). Deconvolution in high-energy astrophysics: Science, instrumentation, and methods. Bayesian Anal. 1 189-236. · Zbl 1331.85008 · doi:10.1214/06-BA107 |
| [54] | van Dyk, D. A., Connors, A., Kashyap, V. and Siemiginowska, A. (2001). Analysis of energy spectra with low photon counts via Bayesian posterior simulation. The Astrophysical Journal 548 224-243. |
| [55] | van Dyk, D. A. and Kang, H. (2004). Highly structured models for spectral analysis in high-energy astrophysics. Statist. Sci. 19 275-293. · Zbl 1100.62637 · doi:10.1214/088342304000000314 |
| [56] | van Dyk, D. A. and Meng, X.-L. (2001). The art of data augmentation (with discussion). J. Comput. Graph. Statist. 10 1-111. JSTOR: · doi:10.1198/10618600152418584 |
| [57] | van Dyk, D. A., Meng, X.-L. and Rubin, D. B. (1995). Maximum likelihood estimation via the ECM algorithm: Computing the asymptotic variance. Statist. Sinica 5 55-75. · Zbl 0824.62021 |
| [58] | van Dyk, D. A. and Tang, R. (2003). The one-step-late PXEM algorithm. Stat. Comput. 13 137-152. · doi:10.1023/A:1023256509116 |
| [59] | Wei, G. and Tanner, M. A. (1990). A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithm. J. Amer. Statist. Assoc. 85 699-704. |
| [60] | Wu, C. F. J. (1983). On the convergence properties of the EM algorithms. Ann. Statist. 11 95-103. · Zbl 0517.62035 · doi:10.1214/aos/1176346060 |
| [61] | Yu, Y. and Meng, X.-L. (2010). To center or not to center: That is not the question-An ancillarity-sufficiency interweaving strategy (ASIS) for boosting MCMC efficiency (with discussion). J. Comput. Graph. Statist. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
