Forecasting with model uncertainty: representations and risk reduction. (English) Zbl 1410.62167
Summary: We consider forecasting with uncertainty about the choice of predictor variables. The researcher wants to select a model, estimate the parameters, and use the parameter estimates for forecasting. We investigate the distributional properties of a number of different schemes for model choice and parameter estimation, including: in-sample model selection using the Akaike information criterion; out-of-sample model selection; and splitting the data into subsamples for model selection and parameter estimation. Using a weak-predictor local asymptotic scheme, we provide a representation result that facilitates comparison of the distributional properties of the procedures and their associated forecast risks. This representation isolates the source of inefficiency in some of these procedures. We develop a simulation procedure that improves the accuracy of the out-of-sample and split-sample methods uniformly over the local parameter space. We also examine how bootstrap aggregation (bagging) affects the local asymptotic risk of the estimators and their associated forecasts. Numerically, we find that for many values of the local parameter, the out-of-sample and split-sample schemes perform poorly if implemented in the conventional way. But they perform well, if implemented in conjunction with our risk-reduction method or bagging.
MSC:
| 62M20 | Inference from stochastic processes and prediction |
| 62B10 | Statistical aspects of information-theoretic topics |
| 62P20 | Applications of statistics to economics |
Keywords:
model choice; out-of-sample; bagging; shrinkage; Rao-Blackwell theorem; Akaike information criterionReferences:
| [1] | Akaike, H. (1974): “A New Look at the Statistical Model Identification,” IEEE Transactions on Automatic Control, 19, 716-723. · Zbl 0314.62039 |
| [2] | Andreas, B., and W.Stuetzle (2006): “Observations on Bagging,” Statistica Sinica, 16, 323-351. · Zbl 1096.62034 |
| [3] | Andrews, D. W. K. (1993): “Tests for Parameter Instability and Structural Change With Unknown Change Point,” Econometrica, 61 (4), 821-856. 10.2307/2951764 · Zbl 0795.62012 |
| [4] | Ashley, R., C. W.Granger, and R.Schmalensee (1980): “Advertising and Aggregate Consumption: An Analysis of Causality,” Econometrica, 48, 1149-1167. 10.2307/1912176 · Zbl 0442.90012 |
| [5] | Bates, J. M., and C. W.Granger (1969): “The Combination of Forecasts,” Operations Research Quarterly, 20, 451-468. |
| [6] | Breiman, L. (1996): “Bagging Predictors,” Machine Learning, 36, 105-139. · Zbl 0858.68080 |
| [7] | Buckland, S. T., K. P.Burnham, and N. H.Augustin (1997): “Model Selection: An Integral Part of Inference,” Biometrics, 53, 603-618. · Zbl 0885.62118 |
| [8] | Bühlmann, P., and B.Yu (2002): “Analyzing Bagging,” Annals of Statistics, 30, 927-961. 10.1214/aos/1031689014 · Zbl 1029.62037 |
| [9] | Christoffersen, P., and F. X.Diebold (1997): “Optimal Prediction Under Asymmetrical Loss,” Econometric Theory, 13, 806-817. 10.1017/S0266466600006277 |
| [10] | Claeskens, G., and N. L.Hjort (2008): Model Selection and Model Averaging. Cambridge: Cambridge University Press. · Zbl 1166.62001 |
| [11] | Clark, T. E. (2004): “Can out‐of‐Sample Forecast Comparisons Help Prevent Overfitting?” Journal of Forecasting, 23, 115-139. |
| [12] | Davidson, J. (1994): Stochastic Limit Theory. Oxford: Oxford University Press. 10.1093/0198774036.001.0001 |
| [13] | Diebold, F. X., and R. S.Mariano (1995): “Comparing Predictive Accuracy,” Journal of Business and Economic Statistics, 13, 253-263. |
| [14] | Efron, B. (2014): “Estimation and Accuracy After Model Selection (With Discussion),” Journal of the American Statistical Association, 109, 991-1007. 10.1080/01621459.2013.823775 · Zbl 1368.62071 |
| [15] | Elliott, G., and U. K.Müller (2014): “Pre and Post Break Parameter Inference,” Journal of Econometrics, 180 (2), 141-157. · Zbl 1293.62183 |
| [16] | Friedman, J. H., and P.Hall (2007): “On Bagging and Nonlinear Estimation,” Journal of Statistical Planning and Inference, 137, 669-683. 10.1016/j.jspi.2006.06.002 · Zbl 1104.62047 |
| [17] | Gaenssler, P., and K.Ziegler (1994): “A Uniform Law of Large Numbers for Set‐Index Processes With Applications to Empirical and Partial‐Sum Processes,” in Probability in Banach Spaces, 9, ed. by J.Hoffmann‐Jorgensen (ed.), J.Kuelbs (ed.), and M. B.Marcus (ed.)Boston: Birkhäuser. |
| [18] | Giacomini, R., and H.White (2006): “Tests of Conditional Predictive Ability,” Econometrica, 74, 1545-1578. · Zbl 1187.91151 |
| [19] | Goyal, A., and I.Welch (2008): “A Comprehensive Look at the Empirical Performance of Equity Premium Prediction,” Review of Financial Studies, 21, 1455-1508. |
| [20] | Hansen, P. R. (2010): “A Winner”s Curse for Econometric Models: On the Joint Distribution of In‐Sample Fit and Out‐of‐Sample Fit and its Implications for Model Selection.” Report. |
| [21] | Hansen, P. R., and A.Timmermann (2015): “Equivalence Between out‐of‐Sample Forecast Comparisons and Wald Statistics,” Econometrica, 83 (6), 2485-2505. · Zbl 1410.62117 |
| [22] | Hirano, K., and J. H.Wright (2017): “Supplement to ‘Forecasting With Model Uncertainty: Representations and Risk Reduction’,” Econometrica Supplemental Material, 85, http://dx.doi.org/10.3982/ECTA13372. · Zbl 1410.62167 · doi:10.3982/ECTA13372 |
| [23] | Inoue, A., and L.Kilian (2004): “In‐Sample or Out‐of‐Sample Tests of Predictability: Which One Should We Use?” Econometric Reviews, 23, 371-402. · Zbl 1062.62213 |
| [24] | Inoue, A., and L.Kilian (2006): “On the Selection of Forecasting Models,” Journal of Econometrics, 130, 273-306. 10.1016/j.jeconom.2005.03.003 · Zbl 1337.62291 |
| [25] | Leeb, H., and B. M.Pötscher (2005): “Model Selection and Inference: Facts and Fiction,” Econometric Theory, 21, 21-59. · Zbl 1085.62004 |
| [26] | Mallows, C. L. (1973): “Some Comments on Cp,” Technometrics, 15, 661-675. · Zbl 0269.62061 |
| [27] | Meese, R. A., and K.Rogoff (1983): “Empirical Exchange Rate Models of the Seventies: Do They Fit Out of Sample?” Journal of International Economics, 14, 3-24. |
| [28] | Park, J. (2002): “An Invariance Principle for Sieve Bootstrap in Time Series,” Econometric Theory, 18, 469-490. · Zbl 1109.62346 |
| [29] | Sjöstrand, K., L. H.Clemmensen, R.Larsen, and B.Ersbøll (2012): “SpaSM: A Matlab Toolbox for Sparse Statistical Modeling,” Working Paper, Technical University of Denmark. |
| [30] | Stock, J. H., and M. W.Watson (2012): “Generalized Shrinkage Methods for Forecasting Using Many Predictors,” Journal of Business and Economic Statistics, 30, 481-493. |
| [31] | Timmermann, A. (2006): “Forecast Combination,” in Handbook of Economic Forecasting, ed. by C. W.Granger (ed.), G.Elliott (ed.), and A.Timmermann (ed.). Amsterdam: North Holland. · Zbl 1273.91014 |
| [32] | Van Der Vaart, A. W. (1998): Asymptotic Statistics. Cambridge: Cambridge University Press. · Zbl 0910.62001 |
| [33] | West, K. D. (2006): “Forecast Evaluation,” in Handbook of Economic Forecasting, ed. by C. W.Granger (ed.), G.Elliott (ed.), and A.Timmermann (ed.). Amsterdam: North Holland. · Zbl 1273.91014 |
| [34] | Wilson, E. (1934): “The Periodogram of American Business Activity,” Quarterly Journal of Economics, 48 (3), 375-417. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
