Covariate-adjusted Fisher randomization tests for the average treatment effect. (English) Zbl 07414292
Summary: Fisher’s randomization test (frt) delivers exact \(p\)-values under the strong null hypothesis of no treatment effect on any units whatsoever and allows for flexible covariate adjustment to improve the power. Of interest is whether the resulting covariate-adjusted procedure could also be valid for testing the weak null hypothesis of zero average treatment effect. To this end, we evaluate two general strategies for conducting covariate adjustment in frts: the pseudo-outcome strategy that uses the residuals from an outcome model with only the covariates as the pseudo, covariate-adjusted outcomes to form the test statistic, and the model-output strategy that directly uses the output from an outcome model with both the treatment and covariates as the covariate-adjusted test statistic. Based on theory and simulation, we recommend using the ordinary least squares (ols) fit of the observed outcome on the treatment, centered covariates, and their interactions for covariate adjustment, and conducting frt with the robust \(t\)-value of the treatment as the test statistic. The resulting frt is finite-sample exact for testing the strong null hypothesis, asymptotically valid for testing the weak null hypothesis, and more powerful than the unadjusted counterpart under alternatives, all irrespective of whether the linear model is correctly specified or not. We start with complete randomization, and then extend the theory to cluster randomization, stratified randomization, and rerandomization, respectively, giving a recommendation for the test procedure and test statistic under each design. Our theory is design-based, also known as randomization-based, in which we condition on the potential outcomes but average over the random treatment assignment.
MSC:
| 62-XX | Statistics |
| 91-XX | Game theory, economics, finance, and other social and behavioral sciences |
Keywords:
finite-population inference; permutation test; randomization distribution; robust standard error; studentization; super-population inferenceSoftware:
DOSReferences:
| [1] | Angrist, J. D.; Pischke, J.-S., Mostly Harmless Econometrics (2009), Princeton University Press · Zbl 1159.62090 |
| [2] | Athey, S.; Eckles, D.; Imbens, G. W., Exact p-values for network interference, J. Amer. Statist. Assoc., 113, 230-240 (2018) · Zbl 1398.62140 |
| [3] | Bahadur, R. R.; Savage, L. J., The nonexistence of certain statistical procedures in nonparametric problems, Ann. Math. Stat., 27, 1115-1122 (1956) · Zbl 0073.14302 |
| [4] | Banerjee, A. V.; Chassang, S.; Montero, S.; Snowberg, E., A theory of experimenters: Robustness, randomization, and balance, Amer. Econ. Rev., 110, 1206-1230 (2020) |
| [5] | Basse, G.; Ding, P.; Feller, A.; Toulis, P., Randomization tests for peer effects in group formation experiments (2019) · Zbl 1434.62094 |
| [6] | Berk, R.; Pitkin, E.; Brown, L.; Buja, A.; George, E.; Zhao, L., Covariance adjustments for the analysis of randomized field experiments, Eval. Rev., 37, 170-196 (2013) |
| [7] | Bind, M. A.C.; Rubin, D. B., When possible, report a Fisher-exact P value and display its underlying null randomization distribution, Proc. Natl. Acad. Sci. USA, 117, 19151-19158 (2020) |
| [8] | Bloniarz, A.; Liu, H.; Zhang, C.; Sekhon, J.; Yu, B., Lasso adjustments of treatment effect estimates in randomized experiments, Proc. Natl. Acad. Sci. USA, 113, 7383-7390 (2016) · Zbl 1357.62098 |
| [9] | ter Braak, C. J.F., Permutation versus bootstrap significance tests in multiple regression and ANOVA, (Jöckel, K.; Rothe, G.; Sendler, W., Bootstrapping and Related Techniques (1992), Springer-Verlag: Springer-Verlag Berlin), 79-85 |
| [10] | Brillinger, D. R.; Jones, L. V.; Tukey, J. W., The management of weather resources, The Role of Statistics in Weather Resources Management IITechnical Report (1978), US Government Printing Office: US Government Printing Office Washington, DC |
| [11] | Bruhn, M.; McKenzie, D., In pursuit of balance: Randomization in practice in development field experiments, Am. Econ. J.: Appl. Econ., 1, 200-232 (2009) |
| [12] | Bugni, F. A.; Canay, I. A.; Shaikh, A. M., Inference under covariate-adaptive randomization, J. Amer. Statist. Assoc., 113, 1784-1796 (2018) · Zbl 1409.62094 |
| [13] | Bugni, F. A.; Canay, I. A.; Shaikh, A. M., Inference under covariate-adaptive randomization with multiple treatments, Quant. Econ., 10, 1747-1785 (2019) · Zbl 1453.62295 |
| [14] | Canay, I. A.; Romano, J. P.; Shaikh, A. M., Randomization tests under an approximate symmetry assumption, Econometrica, 85, 1013-1030 (2017) · Zbl 1410.62103 |
| [15] | Cattaneo, M. D.; Frandsen, B. R.; Titiunik, R., Randomization inference in the regression discontinuity design: An application to party advantages in the US senate, J. Causal Inference, 3, 1-24 (2015) |
| [16] | Chen, X.; Liu, Y.; Ma, S.; Zhang, Z., Efficient estimation of general treatment effects using neural networks with a diverging number of confounders (2020), arXiv preprint arXiv:2009.07055 |
| [17] | Chong, A.; Cohen, I.; Field, E.; Nakasone, E.; Torero, M., Iron deficiency and schooling attainment in Peru, Am. Econ. J.: Appl. Econ., 8, 222-255 (2016) |
| [18] | Chung, E.; Romano, J. P., Exact and asymptotically robust permutation tests, Ann. Statist., 41, 484-507 (2013) · Zbl 1267.62064 |
| [19] | Chung, E.; Romano, J. P., Asymptotically valid and exact permutation tests based on two-sample \(U\)-statistics, J. Statist. Plann. Inference, 168, 97-105 (2016) · Zbl 1333.62124 |
| [20] | Cohen, P. L.; Fogarty, C. B., Gaussian prepivoting for finite population causal inference (2020) |
| [21] | Cox, D. R., Randomization and concomitant variables in the design of experiments, (Kallianpur, G.; Krishnaiah, P. R.; Ghosh, J. K., Statistics and Probability: Essays in Honor of C. R. Rao (1982), North-Holland, Amsterdam), 197-202 · Zbl 0482.62067 |
| [22] | Dasgupta, T.; Pillai, N.; Rubin, D. B., Causal inference from \(2^K\) factorial designs by using potential outcomes, J. R. Stat. Soc. Ser. B Stat. Methodol., 77, 727-753 (2015) · Zbl 1414.62337 |
| [23] | DiCiccio, C. J.; Romano, J. P., Robust permutation tests for correlation and regression coefficients, J. Amer. Statist. Assoc., 112, 1211-1220 (2017) |
| [24] | Ding, P.; Dasgupta, T., A randomization-based perspective of analysis of variance: a test statistic robust to treatment effect heterogeneity, Biometrika, 105, 45-56 (2018) · Zbl 07072392 |
| [25] | Eicker, F., Limit theorems for regressions with unequal and dependent errors, (Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1 (1967), University of California Press: University of California Press Berkeley, CA), 59-82 · Zbl 0217.51201 |
| [26] | Farrell, M. H.; Liang, T.; Misra, S., Deep neural networks for estimation and inference, Econometrica, 89, 181-213 (2021) · Zbl 1479.62082 |
| [27] | Fisher, R. A., The Design of Experiments (1935), Edinburgh, London: Oliver and Boyd |
| [28] | Fogarty, C. B., On mitigating the analytical limitations of finely stratified experiments, J. R. Stat. Soc. Ser. B Stat. Methodol., 80, 1035-1056 (2018) · Zbl 1407.62288 |
| [29] | Fogarty, C. B., Regression assisted inference for the average treatment effect in paired experiments, Biometrika, 105, 994-1000 (2018) · Zbl 1506.62433 |
| [30] | Freedman, D. A., On regression adjustments to experimental data, Adv. Appl. Math., 40, 180-193 (2008) · Zbl 1130.62003 |
| [31] | Freedman, D.; Lane, D., A nonstochastic interpretation of reported significance levels, J. Bus. Econom. Statist., 1, 292-298 (1983) |
| [32] | Gail, M. H.; Tan, W. Y.; Piantadosi, S., Tests for no treatment effect in randomized clinical trials, Biometrika, 75, 57-64 (1988) · Zbl 0635.62108 |
| [33] | Ganong, P.; Jäger, S., A permutation test for the regression kink design, J. Amer. Statist. Assoc., 113, 494-504 (2018) · Zbl 1398.62109 |
| [34] | Guo, K.; Basse, G., The generalized Oaxaca-Blinder estimator, arXiv, 2004.11615 (2020) |
| [35] | Heckman, J. J.; Karapakula, G., Using a satisficing model of experimenter decision-making to guide finite-sample inference for compromised experiments, Econom. J. (2021), (in press) · Zbl 07546394 |
| [36] | Heckman, J.J., Pinto, R., Shaikh, A.M., 2020. Inference with imperfect randomization: The case of the Perry preschool program. In: University of Chicago, Becker Friedman Institute for Economics Working Paper No. 2020-97. Working paper. University of Chicago. |
| [37] | Hennessy, J.; Dasgupta, T.; Miratrix, L.; Pattanayak, C.; Sarkar, P., A conditional randomization test to account for covariate imbalance in randomized experiments, J. Causal Inference, 4, 61-80 (2016) |
| [38] | Hoeffding, W., The large-sample power of tests based on permutations of observations, Ann. Math. Stat., 23, 169-192 (1952) · Zbl 0046.36403 |
| [39] | Huber, P. J., The behavior of maximum likelihood estimates under nonstandard conditions, (Cam, L. M.L.; Neyman, J., Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1 (1967), Berkeley, California: University of California Press), 221-233 · Zbl 0212.21504 |
| [40] | Janssen, A., Studentized permutation tests for non-iid hypotheses and the generalized Behrens-Fisher problem, Statist. Probab. Lett., 36, 9-21 (1997) · Zbl 1064.62526 |
| [41] | Jiang, F.; Tian, L.; Fu, H.; Hasegawa, T.; Wei, L. J., Robust alternatives to ANCOVA for estimating the treatment effect via a randomized comparative study, J. Amer. Statist. Assoc., 114, 1854-1864 (2019) · Zbl 1428.62339 |
| [42] | Kennedy, P. E., Randomization tests in econometrics, J. Bus. Econom. Statist., 13, 85-94 (1995) |
| [43] | Lee, S.; Shaikh, A. M., Multiple testing and heterogeneous treatment effects: re-evaluating the effect of progresa on school enrollment, J. Appl. Econometrics, 29, 612-626 (2014) |
| [44] | Lehmann, E. L., Nonparametrics: Statistical Methods Based on Ranks (1975), San Francisco: Holden-Day, Inc · Zbl 0354.62038 |
| [45] | Lehmann, E. L.; Romano, J. P., Testing Statistical Hypotheses (2005), New York: Springer · Zbl 1076.62018 |
| [46] | Lei, L.; Ding, P., Regression adjustment in completely randomized experiments with a diverging number of covariates, Biometrika (2020), (in press) |
| [47] | Li, X.; Ding, P., General forms of finite population central limit theorems with applications to causal inference, J. Amer. Statist. Assoc., 112, 1759-1769 (2017) |
| [48] | Li, X.; Ding, P., Rerandomization and regression adjustment, J. R. Stat. Soc. Ser. B Stat. Methodol., 82, 241-268 (2020) · Zbl 1440.62297 |
| [49] | Lin, W., Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique, Ann. Appl. Stat., 7, 295-318 (2013) · Zbl 1454.62217 |
| [50] | Liu, H.; Yang, Y., Regression-adjusted average treatment effect estimates in stratified randomized experiments, Biometrika, 107, 935-948 (2020) · Zbl 1457.62046 |
| [51] | Lu, J., Covariate adjustment in randomization-based causal inference for \(2^K\) factorial designs, Statist. Probab. Lett., 119, 11-20 (2016) · Zbl 1398.62211 |
| [52] | MacKinnon, J. G.; Webb, M. D., Randomization inference for difference-in-differences with few treated clusters, J. Econometrics, 218, 435-450 (2020) · Zbl 1464.62226 |
| [53] | Manly, B. F.J., Randomization, Bootstrap and Monte Carlo Methods in Biology (1997), Chapman & Hall · Zbl 0918.62081 |
| [54] | Middleton, J. A., A unified theory of regression adjustment for design-based inference (2018), arXiv preprint arXiv:1803.06011 |
| [55] | Middleton, J. A.; Aronow, P. M., Unbiased estimation of the average treatment effect in cluster-randomized experiments, Stat. Polit. Policy, 6, 39-75 (2015) |
| [56] | Moore, K. L.; van der Laan, M. J., Covariate adjustment in randomized trials with binary outcomes: targeted maximum likelihood estimation, Stat. Med., 28, 39-64 (2009) |
| [57] | Moore, K. L.; Neugebauer, R.; Valappil, T.; van der Laan, M. J., Robust extraction of covariate information to improve estimation efficiency in randomized trials, Stat. Med., 30, 2389-2408 (2011) |
| [58] | Morgan, K. L.; Rubin, D. B., Rerandomization to improve covariate balance in experiments, Ann. Statist., 40, 1263-1282 (2012) · Zbl 1274.62509 |
| [59] | Mukerjee, R.; Dasgupta, T.; Rubin, D. B., Using standard tools from finite population sampling to improve causal inference for complex experiments, J. Amer. Statist. Assoc., 113, 868-881 (2018) · Zbl 1398.62213 |
| [60] | Negi, A.; Wooldridge, J. M., Revisiting regression adjustment in experiments with heterogeneous treatment effects, Econometric Rev., 40, 504-534 (2021) · Zbl 1490.62472 |
| [61] | Neyman, J., On the application of probability theory to agricultural experiments (with discussion), Statist. Sci., 5, 465-472 (1923) · Zbl 0955.01560 |
| [62] | Neyman, J., Statistical problems in agricultural experimentation (with discussion), Suppl. J. R. Stat. Soc., 2, 107-180 (1935) · JFM 63.1103.02 |
| [63] | Ottoboni, K.; Lewis, F.; Salmaso, L., An empirical comparison of parametric and permutation tests for regression analysis of randomized experiments, Stat. Biopharm. Res., 10, 264-273 (2018) |
| [64] | Pauly, M.; Brunner, E.; Konietschke, F., Asymptotic permutation tests in general factorial designs, J. R. Stat. Soc. Ser. B Stat. Methodol., 77, 461-473 (2015) · Zbl 1414.62339 |
| [65] | Proschan, M. A.; Dodd, L. E., Re-randomization tests in clinical trials, Stat. Med., 38, 2292-2302 (2019) |
| [66] | Raz, J., Testing for no effect when estimating a smooth function by nonparametric regression: a randomization approach, J. Amer. Statist. Assoc., 85, 132-138 (1990) |
| [67] | Romano, J. P., On the behavior of randomization tests without a group invariance assumption, J. Amer. Statist. Assoc., 85, 686-692 (1990) · Zbl 0706.62047 |
| [68] | Rosenbaum, P. R., Reduced sensitivity to hidden bias at upper quantiles in observational studies with dilated treatment effects, Biometrics, 55, 560-564 (1999) · Zbl 1059.62700 |
| [69] | Rosenbaum, P. R., Covariance adjustment in randomized experiments and observational studies, Statist. Sci., 17, 286-327 (2002) · Zbl 1013.62117 |
| [70] | Rosenbaum, P. R., Exact confidence intervals for nonconstant effects by inverting the signed rank test, Am. Stat., 57, 132-138 (2003) · Zbl 1182.62110 |
| [71] | Rosenbaum, P. R., Design of Observational Studies (2010), New York: Springer · Zbl 1308.62005 |
| [72] | Stephens, A. J.; Tchetgen Tchetgen, E. J.; De Gruttola, V., Flexible covariate-adjusted exact tests of randomized treatment effects with application to a trial of HIV education, Ann. Appl. Stat., 7, 2106-2137 (2013) · Zbl 1283.62238 |
| [73] | Su, F.; Ding, P., Model-assisted analyses of cluster-randomized experiments (2021), arXiv:2104.04647 |
| [74] | Tukey, J. W., Tightening the clinical trial, Controlled Clin. Trials, 14, 266-285 (1993) |
| [75] | van der vaart, A. W.; Wellner, J., Weak Convergence and Empirical Processes: With Applications To Statistics (1996), Springer Verlag: Springer Verlag New York · Zbl 0862.60002 |
| [76] | Wager, S.; Du, W.; Taylor, J.; Tibshirani, R. J., High-dimensional regression adjustments in randomized experiments, Proc. Natl. Acad. Sci. USA, 113, 12673-12678 (2016) · Zbl 1407.62264 |
| [77] | White, H., A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity, Econometrica, 48, 817-838 (1980) · Zbl 0459.62051 |
| [78] | Wu, J.; Ding, P., Randomization tests for weak null hypotheses in randomized experiments, J. Amer. Statist. Assoc., 105 (2020), (in press) |
| [79] | Wu, E.; Gagnon-Bartsch, J. A., The LOOP estimator: Adjusting for covariates in randomized experiments, Eval. Rev., 42, 458-488 (2018) |
| [80] | Ye, T.; Yi, Y.; Zhao, Q., Inference on average treatment effect under minimization and other covariate-adaptive randomization methods (2020), arXiv preprint arXiv:2007.09576 |
| [81] | Young, A., Channeling Fisher: Randomization tests and the statistical insignificance of seemingly significant experimental results, Q. J. Econ., 134, 557-598 (2019) · Zbl 1507.62410 |
| [82] | Zhang, M.; Tsiatis, A. A.; Davidian, M., Improving efficiency of inferences in randomized clinical trials using auxiliary covariates, Biometrics, 64, 707-715 (2008) · Zbl 1170.62082 |
| [83] | Zheng, L.; Zelen, M., Multi-center clinical trials: Randomization and ancillary statistics, Ann. Appl. Stat., 2, 582-600 (2008) · Zbl 1400.62303 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
