The A/B testing problem with Gaussian priors. (English) Zbl 1520.91105
Summary: A risk-neutral firm can perform a randomized experiment (A/B test) to learn about the effects of implementing an idea of unknown quality. The firm’s goal is to decide the experiment’s sample size and whether or not the idea should be implemented after observing the experiment’s outcome. We show that when the distribution for idea quality is Gaussian and there are linear costs of experimentation, there are exact formulae for the firm’s optimal implementation decisions, the value of obtaining more data, and optimal experiment sizes. Our formulae – which assume that companies use randomized experiments to help them maximize expected profits – provide a simple alternative to i) the standard rules-of-thumb of power calculations for determining the sample size of an experiment, and also to ii) ad hoc thresholds based on statistical significance to interpret the outcome of an experiment.
MSC:
| 91B06 | Decision theory |
| 91-05 | Experimental work for problems pertaining to game theory, economics, and finance |
References:
| [1] | Athey, S.; Imbens, G. W., The Econometrics of Randomized Experiments, Handbook of Economic Field Experiments, vol. 1, 73-140 (2017), Elsevier |
| [2] | Azevedo, E. M.; Alex, D.; Montiel Olea, J.; Rao, J. M.; Weyl, E. G., A/B testing with fat tails, J. Polit. Econ., 128, 12, 4614-4672 (2020) |
| [3] | Azevedo, E. M.; Deng, A.; Montiel Olea, J. L.; Weyl, E. G., Empirical Bayes estimation of treatment effects with many A/B tests: an overview, (AEA Papers and Proceedings, vol. 109 (2019)), 43-47 |
| [4] | Bross, I., Two-choice selection, J. Am. Stat. Assoc., 45, 252, 530-540 (1950) · Zbl 0039.35602 |
| [5] | DellaVigna, S.; Linos, E., RCTs to scale: comprehensive evidence from two nudge units, Econometrica, 90, 1, 81-116 (2022) |
| [6] | Grundy, P.; Healy, M.; Rees, D., Economic choice of the amount of experimentation, J. R. Stat. Soc., Ser. B, Methodol., 18, 1, 32-49 (1956) · Zbl 0071.13705 |
| [7] | Keppo, J.; Moscarini, G.; Smith, L., The demand for information: more heat than light, J. Econ. Theory, 138, 1, 21-50 (2008) · Zbl 1140.91395 |
| [8] | List, J. A.; Sadoff, S.; Wagner, M., So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design, Exp. Econ., 14, 4, 439 (2011) |
| [9] | Manski, C. F., Statistical treatment rules for heterogeneous populations, Econometrica, 72, 4, 1221-1246 (2004) · Zbl 1142.62308 |
| [10] | Manski, C. F., Treatment choice with trial data: statistical decision theory should supplant hypothesis testing, Am. Stat., 73, sup1, 296-304 (2019) · Zbl 07588212 |
| [11] | Manski, C. F.; Tetenov, A., Sufficient trial size to inform clinical practice, Proc. Natl. Acad. Sci., 113, 38, 10518-10523 (2016) |
| [12] | Manski, C. F.; Tetenov, A., Trial size for near-optimal choice between surveillance and aggressive treatment: reconsidering MSLT-II, Am. Stat., 73, sup1, 305-311 (2019) · Zbl 07588213 |
| [13] | Meltzer, D., Addressing uncertainty in medical cost-effectiveness analysis: implications of expected utility maximization for methods to perform sensitivity analysis and the use of cost-effectiveness analysis to set priorities for medical research, J. Health Econ., 20, 1, 109-129 (2001) |
| [14] | Moscarini, G.; Smith, L., The law of large demand for information, Econometrica, 70, 6, 2351-2366 (2002) · Zbl 1141.91360 |
| [15] | Pierce, L.; Rees-Jones, A.; Blank, C., The negative consequences of loss-framed performance incentives (2021), National Bureau of Economic Research, Technical report |
| [16] | Raiffa, H., Schlaifer, R., 1961. Applied statistical decision theory. · Zbl 0181.21801 |
| [17] | Savage, L., The theory of statistical decision, J. Am. Stat. Assoc., 89, 55-67 (1951) · Zbl 0042.14302 |
| [18] | Somerville, P. N., Some problems of optimum sampling, Biometrika, 41, 3/4, 420-429 (1954) · Zbl 0056.37902 |
| [19] | Stoye, J., Minimax regret treatment choice with finite samples, J. Econom., 151, 1, 70-81 (2009) · Zbl 1431.62031 |
| [20] | Stoye, J., Minimax regret treatment choice with covariates or with limited validity of experiments, J. Econom., 166, 1, 138-156 (2012) · Zbl 1441.62876 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
