An online sequential test for qualitative treatment effects. (English) Zbl 07626801
Summary: Tech companies (e.g., Google or Facebook) often use randomized online experiments and/or A/B testing primarily based on the average treatment effects to compare their new product with an old one. However, it is also critically important to detect qualitative treatment effects such that the new one may significantly outperform the existing one only under some specific circumstances. The aim of this paper is to develop a powerful testing procedure to efficiently detect such qualitative treatment effects. We propose a scalable online updating algorithm to implement our test procedure. It has three novelties including adaptive randomization, sequential monitoring, and online updating with guaranteed type-I error control. We also thoroughly examine the theoretical properties of our testing procedure including the limiting distribution of test statistics and the justification of an efficient bootstrap method. Extensive empirical studies are conducted to examine the finite sample performance of our test procedure.
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
Keywords:
A/B testing; qualitative treatment effects; sequential monitoring; adaptive randomization; online updatingReferences:
| [1] | Donald W. K. Andrews and Xiaoxia Shi. Inference based on conditional moment inequalities.Econometrica, 81(2):609-666, 2013. ISSN 0012-9682. doi: 10.3982/ECTA9370. URL http://dx.doi.org/10.3982/ECTA9370. · Zbl 1274.62311 |
| [2] | Donald W. K. Andrews and Xiaoxia Shi. Nonparametric inference based on conditional moment inequalities.J. Econometrics, 179(1):31-45, 2014. ISSN 0304-4076. doi: 10.1016/ j.jeconom.2013.10.005. URLhttp://dx.doi.org/10.1016/j.jeconom.2013.10.005. · Zbl 1293.62065 |
| [3] | Timothy B. Armstrong and Hock Peng Chan. Multiscale adaptive inference on conditional moment inequalities.J. Econometrics, 194(1):24-43, 2016. ISSN 0304-4076. doi: 10.1016/ j.jeconom.2016.04.001. URLhttp://dx.doi.org/10.1016/j.jeconom.2016.04.001. · Zbl 1431.62181 |
| [4] | Susan Athey and Guido Imbens. Recursive partitioning for heterogeneous causal effects. Proc. Natl. Acad. Sci. USA, 113(27):7353-7360, 2016. ISSN 1091-6490. doi: 10.1073/ pnas.1510489113. URLhttps://doi.org/10.1073/pnas.1510489113. · Zbl 1357.62190 |
| [5] | Alexandre Belloni and Roberto I Oliveira. A high dimensional central limit theorem for martingales, with applications to context tree models.arXiv preprint arXiv:1809.02741, 2018. |
| [6] | Alexandre Belloni, Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Some new asymptotic theory for least squares series: Pointwise and uniform results.Journal of Econometrics, 186(2):345-366, 2015. · Zbl 1331.62250 |
| [7] | Bernard Bercu and Abderrahmen Touati. Exponential inequalities for self-normalized martingales with applications.Ann. Appl. Probab., 18(5):1848-1869, 2008. ISSN 1050-5164. doi: 10.1214/07-AAP506. URLhttps://doi.org/10.1214/07-AAP506. · Zbl 1152.60309 |
| [8] | Prabir Burman and Keh-Wei Chen. Nonparametric estimation of a regression function. Ann. Statist., 17(4):1567-1596, 1989. ISSN 0090-5364. doi: 10.1214/aos/1176347382. · Zbl 0744.62054 |
| [9] | Minsu Chang, Sokbae Lee, and Yoon-Jae Whang.Nonparametric tests of conditional treatment effects with an application to single-sex schooling on academic achievements. Econom. J., 18(3):307-346, 2015. ISSN 1368-4221. · Zbl 1521.62068 |
| [10] | Xiaohong Chen and Timothy M. Christensen.Optimal uniform convergence rates and asymptotic normality for series estimators under weak dependence and weak conditions. J. Econometrics, 188(2):447-465, 2015. ISSN 0304-4076. doi: 10.1016/j.jeconom.2015. 03.010. · Zbl 1337.62101 |
| [11] | Xiaohong Chen and Timothy M Christensen. Optimal sup-norm rates and uniform inference on nonlinear functionals of nonparametric iv regression.Quantitative Economics, 9(1): 39-84, 2018. · Zbl 1398.62088 |
| [12] | Victor Chernozhukov, Sokbae Lee, and Adam M. Rosen.Intersection bounds:estimation and inference.Econometrica, 81(2):667-737, 2013.ISSN 0012-9682.doi: 10.3982/ECTA8718. URLhttp://dx.doi.org/10.3982/ECTA8718. · Zbl 1274.62233 |
| [13] | Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Empirical and multiplier bootstraps for suprema of empirical processes of increasing complexity, and related Gaussian couplings.Stochastic Process. Appl., 126(12):3632-3651, 2016. ISSN 0304-4149. doi: 10.1016/j.spa.2016.04.009. URLhttps://doi.org/10.1016/j.spa.2016.04.009. · Zbl 1351.60035 |
| [14] | Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Detailed proof of nazarov’s inequality.arXiv preprint arXiv:1711.10696, 2017. · Zbl 1317.60038 |
| [15] | M Gail and R Simon. Testing for qualitative interactions between treatment effects and patient subsets.Biometrics, 41(2):361-372, 1985. · Zbl 0614.62140 |
| [16] | L. Gunter, J. Zhu, and S. A. Murphy. Variable selection for qualitative interactions.Stat. Methodol., 8(1):42-55, 2011. ISSN 1572-3127. doi: 10.1016/j.stamet.2009.05.003. URL http://dx.doi.org/10.1016/j.stamet.2009.05.003. |
| [17] | Lacey Gunter, Ji Zhu, and Susan Murphy. Variable selection for optimal decision making. InConference on Artificial Intelligence in Medicine in Europe, pages 149-154. Springer, 2007. |
| [18] | Yu-Chin Hsu. Consistent tests for conditional treatment effects.The Econometrics Journal, 20(1):1-22, 2017. · Zbl 1521.62071 |
| [19] | Jianhua Z. Huang. Projection estimation in multiple regression with application to functional ANOVA models.Ann. Statist., 26(1):242-272, 1998.ISSN 0090-5364.doi: 10.1214/aos/1030563984. · Zbl 0930.62042 |
| [20] | Christopher Jennison and Bruce W Turnbull.Group sequential methods with applications to clinical trials. Chapman and Hall/CRC, 1999. · Zbl 0934.62078 |
| [21] | Ramesh Johari, Leo Pekelis, and David J Walsh. Always valid inference: Bringing sequential analysis to a/b testing.arXiv preprint arXiv:1512.04922, 2015. |
| [22] | Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. Peeking at a/b tests: Why it matters, and what to do about it. InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1517-1525. ACM, 2017. |
| [23] | Nianqiao Ju, Diane Hu, Adam Henderson, and Liangjie Hong. A sequential test for selecting the better variant: Online a/b testing, adaptive allocation, and continuous monitoring. InProceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 492-500. ACM, 2019. |
| [24] | Eugene Kharitonov, Aleksandr Vorobev, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. Sequential testing for early stopping of online experiments. InProceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 473-482. ACM, 2015. |
| [25] | K. K. Gordon Lan and David L. DeMets. Discrete sequential boundaries for clinical trials. Biometrika, 70(3):659-663, 1983. ISSN 0006-3444. doi: 10.2307/2336502. URLhttps: //doi.org/10.2307/2336502. · Zbl 0543.62059 |
| [26] | Alexander R. Luedtke and Mark J. van der Laan. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy.Ann. Statist., 44(2):713-742, 2016. ISSN 0090-5364. doi: 10.1214/15-AOS1384. URLhttp://dx.doi.org/10.1214/ 15-AOS1384. · Zbl 1338.62089 |
| [27] | Gregory Morrow and Walter Philipp. An almost sure invariance principle for Hilbert space valued martingales.Trans. Amer. Math. Soc., 273(1):231-251, 1982. ISSN 0002-9947. doi: 10.2307/1999203. URLhttps://doi.org/10.2307/1999203. · Zbl 0508.60014 |
| [28] | Min Qian and Susan A. Murphy. Performance guarantees for individualized treatment rules. Ann. Statist., 39(2):1180-1210, 2011. ISSN 0090-5364. doi: 10.1214/10-AOS864. URL http://dx.doi.org/10.1214/10-AOS864. · Zbl 1216.62178 |
| [29] | Jeremy Roth and Noah Simon. A framework for estimating and testing qualitative interactions with applications to predictive biomarkers.Biostatistics, 19(3):263-280, 2018. ISSN 1468-4357. doi: 10.1093/biostatistics/kxx038. URLhttps://doi.org/10.1093/ biostatistics/kxx038. |
| [30] | Chengchun Shi, Wenbin Lu, and Rui Song. Breaking the curse of nonregularity with subagging: inference of the mean outcome under optimal treatment regimes.Journal of Machine Learning Research, 2020a. · Zbl 1536.62087 |
| [31] | Chengchun Shi, Wenbin Lu, and Rui Song. A sparse random projection-based test for overall qualitative treatment effects.Journal of the American Statistical Association, 115 (531):1201-1213, 2020b. · Zbl 1441.62151 |
| [32] | Chengchun Shi, Sheng Zhang, Rui Song, and Wenbin Lu. Statistical inference of the value function for reinforcement learning in infinite horizon settings.Journal of the Royal Statistical Society, Series B · Zbl 07909595 |
| [33] | Charles J. Stone. Optimal global rates of convergence for nonparametric regression.Ann. Statist., 10(4):1040-1053, 1982. ISSN 0090-5364. · Zbl 0511.62048 |
| [34] | Matt Taddy, Matt Gardner, Liyun Chen, and David Draper. A nonparametric bayesian analysis of heterogenous treatment effects in digital experimentation.Journal of Business & Economic Statistics, 34(4):661-672, 2016. |
| [35] | Aad W. van der Vaart and Jon A. Wellner.Weak convergence and empirical processes. Springer Series in Statistics. Springer-Verlag, New York, 1996. ISBN 0-387-94640-3. doi: 10.1007/978-1-4757-2545-2.URLhttp://dx.doi.org/10.1007/978-1-4757-2545-2. With applications to statistics. · Zbl 0862.60002 |
| [36] | Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests.J. Amer. Statist. Assoc., 113(523):1228-1242, 2018. ISSN 01621459. doi: 10.1080/01621459.2017.1319839. URLhttps://doi.org/10.1080/01621459. 2017.1319839. · Zbl 1402.62056 |
| [37] | C.-F. J. Wu. Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist., 14(4):1261-1350, 1986. ISSN 0090-5364. doi: 10.1214/aos/1176350142. · Zbl 0618.62072 |
| [38] | Fanny Yang, Aaditya Ramdas, Kevin G Jamieson, and Martin J Wainwright. A framework for multi-a (rmed)/b (andit) testing with online fdr control.InAdvances in Neural Information Processing Systems, pages 5957-5966, 2017. |
| [39] | Miao Yu, Wenbin Lu, and Rui Song. A new framework for online testing of heterogeneous treatment effect. InAAAI, pages 10310-10317, 2020. |
| [40] | Baqun Zhang, Anastasios A Tsiatis, Eric B Laber, and Marie Davidian. Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions.Biometrika, 100(3):681-694, 2013. · Zbl 1284.62508 |
| [41] | Li-xin Zhang.Strong approximations of martingale vectors and their applications in Markov-chain adaptive designs.Acta Math. Appl. Sin. Engl. Ser., 20(2):337-352, 2004. ISSN 0168-9673. doi: 10.1007/s10255-004-0173-z. URLhttps://doi.org/10.1007/ s10255-004-0173-z. · Zbl 1057.60031 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
