What’s trending in difference-in-differences? A synthesis of the recent econometrics literature. (English) Zbl 07704532
Summary: This paper synthesizes recent advances in the econometrics of difference-in-differences (DiD) and provides concrete recommendations for practitioners. We begin by articulating a simple set of “canonical” assumptions under which the econometrics of DiD are well-understood. We then argue that recent advances in DiD methods can be broadly classified as relaxing some components of the canonical DiD setup, with a focus on \((i)\) multiple periods and variation in treatment timing, \((ii)\) potential violations of parallel trends, or \((iii)\) alternative frameworks for inference. Our discussion highlights the different ways that the DiD literature has advanced beyond the canonical model, and helps to clarify when each of the papers will be relevant for empirical work. We conclude by discussing some promising areas for future research.
MSC:
| 62-XX | Statistics |
| 91-XX | Game theory, economics, finance, and other social and behavioral sciences |
Keywords:
difference-in-differences; causal inference; staggered treatment timing; sensitivity analysis; clustering; parallel trends; treatment effect heterogeneityReferences:
| [1] | Abadie, Alberto, Semiparametric Difference-in-Differences Estimators, Rev. Econom. Stud., 72, 1, 1-19 (2005) · Zbl 1112.62132 |
| [2] | Abadie, Alberto, Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects, J. Econ. Lit., 59, 2, 391-425 (2021) |
| [3] | Abadie, Alberto; Athey, Susan; Imbens, Guido W.; Wooldridge, Jeffrey M., Sampling-Based versus Design-Based Uncertainty in Regression Analysis, Econometrica, 88, 1, 265-296 (2020) · Zbl 1467.62119 |
| [4] | Abadie, Alberto; Athey, Susan; Imbens, Guido; Wooldridge, Jeffrey, When should you adjust standard errors for clustering?, Q. J. Econ., 138, 1, 1-35 (2023) · Zbl 07656086 |
| [5] | Abadie, Alberto; Diamond, Alexis; Hainmueller, Jens, Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program, J. Amer. Statist. Assoc., 105, 490, 493-505 (2010) |
| [6] | Abadie, Alberto; Imbens, Guido W., Large sample properties of matching estimators for average treatment effects, Econometrica, 74, 1, 235-267 (2006) · Zbl 1112.62042 |
| [7] | Abadie, Alberto; Imbens, Guido W., On the Failure of the Bootstrap for Matching Estimators, Econometrica, 76, 6, 1537-1557 (2008) · Zbl 1153.91752 |
| [8] | Abadie, Alberto; Imbens, Guido W., Bias-Corrected Matching Estimators for Average Treatment Effects, J. Bus. Econ. Stat., 29, 1, 1-11 (2011) · Zbl 1214.62031 |
| [9] | Abadie, Alberto; Imbens, Guido W., A Martingale Representation for Matching Estimators, J. Amer. Statist. Assoc., 107, 498, 833-843 (2012) · Zbl 1261.62008 |
| [10] | Abadie, Alberto; Spiess, Jann, Robust post-Matching Inference, J. Amer. Statist. Assoc., 117, 538, 983-995 (2022) · Zbl 1507.91124 |
| [11] | Abbring, Jaap H.; van den Berg, Gerard J., The nonparametric identification of treatment effects in duration models, Econometrica, 71, 5, 1491-1517 (2003) · Zbl 1154.62411 |
| [12] | Andrews, Isaiah; Roth, Jonathan; Pakes, Ariel, Inference for Linear Conditional Moment Inequalities, Rev. Econom. Stud. (2023), Forthcoming |
| [13] | Angrist, Joshua D.; Pischke, Jorn-Steffen, Mostly Harmless Econometrics: An Empiricist’s Companion (2009), Princeton University Press: Princeton University Press Princeton · Zbl 1159.62090 |
| [14] | Arellano, M., Practitioners’ Corner: Computing Robust Standard Errors for Within-groups Estimators, Oxf. Bullet. Econ. Stat., 49, 4, 431-434 (1987) |
| [15] | Arkhangelsky, Dmitry; Athey, Susan; Hirshberg, David A.; Imbens, Guido W.; Wager, Stefan, Synthetic Difference-in-Differences, Amer. Econ. Rev., 111, 12, 4088-4118 (2021) |
| [16] | Armstrong, Timothy B.; Kolesár, Michal, Optimal Inference in a Class of Regression Models, Econometrica, 86, 2, 655-683 (2018) · Zbl 1414.62150 |
| [17] | Athey, Susan; Imbens, Guido W., Identification and Inference in Nonlinear Difference-in-Differences Models, Econometrica, 74, 2, 431-497 (2006) · Zbl 1145.62316 |
| [18] | Athey, Susan; Imbens, Guido, Design-based Analysis in Difference-In-Differences Settings with Staggered Adoption, J. Econometrics, 226, 1, 62-79 (2022) · Zbl 07471885 |
| [19] | Baker, Andrew; Larcker, David F.; Wang, Charles C. Y., How much should we trust staggered difference-in-differences estimates?, J. Financ. Econ., 144, 2, 370-395 (2022) |
| [20] | Ben-Michael, Eli; Feller, Avi; Rothstein, Jesse, The Augmented Synthetic Control Method, J. Amer. Statist. Assoc., 116, 536, 1789-1803 (2021) · Zbl 1506.62484 |
| [21] | Ben-Michael, Eli; Feller, Avi; Rothstein, Jesse, Synthetic controls with staggered adoption, J. R. Stat. Soc. Ser. B Stat. Methodol., 84, 2, 351-381 (2022) · Zbl 07593415 |
| [22] | Bertrand, Marianne; Duflo, Esther; Mullainathan, Sendhil, How Much Should We Trust Differences-In-Differences Estimates?, Quart. J. Econ., 119, 1, 249-275 (2004) · Zbl 1053.62132 |
| [23] | Bilinski, Alyssa; Hatfield, Laura A., Seeking evidence of absence: Reconsidering tests of model assumptions (2018) |
| [24] | Bojinov, Iavor; Rambachan, Ashesh; Shephard, Neil, Panel experiments and dynamic causal effects: A finite population perspective, Quant. Econ., 12, 4, 1171-1196 (2021) · Zbl 07497763 |
| [25] | Bonhomme, Stéphane; Sauder, Ulrich, Recovering distributions in difference-in-differences models: A comparison of selective and comprehensive schooling, Rev. Econ. Stat., 93, 2, 479-494 (2011) |
| [26] | Borusyak, Kirill; Jaravel, Xavier, Revisiting Event Study DesignsSSRN Scholarly Paper ID 2826228 (2018), Social Science Research Network: Social Science Research Network Rochester, NY |
| [27] | Borusyak, Kirill; Jaravel, Xavier; Spiess, Jann, Revisiting Event Study Designs: Robust and Efficient Estimation (2021) |
| [28] | Butts, Kyle, Difference-in-Differences Estimation with Spatial Spillovers (2021) |
| [29] | Caetano, Carolina; Callaway, Brantly; Payne, Stroud; Rodrigues, Hugo Sant’Anna, Difference in Differences with Time-Varying Covariates (2022) |
| [30] | Callaway, Brantly; Goodman-Bacon, Andrew; Sant’Anna, Pedro H. C., Difference-in-Differences with a Continuous Treatment (2021) · Zbl 07414289 |
| [31] | Callaway, Brantly; Li, Tong, Quantile treatment effects in difference in differences models with panel data, Quant. Econ., 10, 4, 1579-1618 (2019) · Zbl 1453.62451 |
| [32] | Callaway, Brantly; Sant’Anna, Pedro H. C., Difference-in-Differences with multiple time periods, J. Econometrics, 225, 2, 200-230 (2021) · Zbl 07414289 |
| [33] | Cameron, A. Colin; Gelbach, Jonah B.; Miller, Douglas L., Bootstrap-Based Improvements for Inference With Clustered Errors, Rev. Econ. Stat., 90, 3, 414-427 (2008) |
| [34] | Canay, Ivan A.; Romano, Joseph P.; Shaikh, Azeem M., Randomization Tests Under an Approximate Symmetry Assumption, Econometrica, 85, 3, 1013-1030 (2017) · Zbl 1410.62103 |
| [35] | Canay, Ivan A.; Santos, Andres; Shaikh, Azeem M., The wild bootstrap with a “small” number of “large” clusters, Rev. Econ. Stat., 103, 2, 346-363 (2021) |
| [36] | Card, David; Krueger, Alan B., Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania, Amer. Econ. Rev., 84, 4, 772-793 (1994) |
| [37] | Cengiz, Doruk; Dube, Arindrajit; Lindner, Attila; Zipperer, Ben, The Effect of Minimum Wages on Low-Wage Jobs, Q. J. Econ., 134, 3, 1405-1454 (2019) · Zbl 1482.91111 |
| [38] | Chabé-Ferret, Sylvain, Analysis of the bias of Matching and Difference-in-Difference under alternative earnings and selection processes, J. Econometrics, 185, 1, 110-123 (2015) · Zbl 1331.62457 |
| [39] | Chang, Neng-Chieh, Double/debiased machine learning for difference-in-differences, Econom. J., 23, 177-191 (2020) · Zbl 07546363 |
| [40] | Chernozhukov, Victor; Demirer, Mert; Duflo, Esther; Fernández-Val, Iván, Generic Machine Learning Inference on Heterogeneous Treatment Effects in Randomized Experiments, 1-52 (2020) |
| [41] | Chernozhukov, Victor; Wüthrich, Kaspar; Zhu, Yinchu, An exact and robust conformal inference method for counterfactual and synthetic controls, J. Amer. Statist. Assoc., 116, 536, 1849-1864 (2021) · Zbl 1506.62246 |
| [42] | Conley, Timothy G.; Taber, Christopher R., Inference with “Difference in Differences” with a Small Number of Policy Changes, Rev. Econ. Stat., 93, 1, 113-125 (2011) |
| [43] | Daw, Jamie R.; Hatfield, Laura A., Matching and Regression to the Mean in Difference-in-Differences Analysis, Health Services Research (2018) |
| [44] | de Chaisemartin, Clément; D’Haultfoeuille, Xavier, Fuzzy Differences-in-Differences, Rev. Econom. Stud., 85, 2, 999-1028 (2018) · Zbl 1405.91182 |
| [45] | de Chaisemartin, Clément; D’Haultfoeuille, Xavier, Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects, Amer. Econ. Rev., 110, 9, 2964-2996 (2020) |
| [46] | de Chaisemartin, Clément; D’Haultfoeuille, Xavier, Two-way Fixed Effects Regressions with Several TreatmentsSSRN Scholarly Paper ID 3751060 (2021), Social Science Research Network: Social Science Research Network Rochester, NY |
| [47] | de Chaisemartin, Clément; D’Haultfoeuille, Xavier, Difference-in-Differences Estimators of Intertemporal Treatment Effects (2022) |
| [48] | de Chaisemartin, Clément; D’Haultfoeuille, Xavier, Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey, Econom. J. (2022), Forthcoming |
| [49] | Dette, Holger, Schumann, Martin, 2020. Difference-in-Differences Estimation Under Non-Parallel Trends. Working Paper. |
| [50] | Dettmann, Eva, Flexpaneldid: A Stata Toolbox for Causal Analysis with Varying Treatment Time and DurationSSRN Scholarly Paper ID 3692458 (2020), Social Science Research Network: Social Science Research Network Rochester, NY |
| [51] | Ding, Peng; Li, Fan, A Bracketing Relationship between Difference-in-Differences and Lagged-Dependent-Variable Adjustment, Political Anal., 27, 605-615 (2019) |
| [52] | Donald, Stephen G.; Lang, Kevin, Inference with Difference-in-Differences and Other Panel Data, Rev. Econ. Stat., 89, 2, 221-233 (2007) |
| [53] | Doudchenko, Nikolay; Imbens, Guido W., Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A SynthesisWorking Paper 22791 (2016), National Bureau of Economic Research |
| [54] | Ferman, Bruno; Pinto, Cristine, Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity, Rev. Econ. Stat., 101, 3, 452-467 (2019) |
| [55] | Fisher, R. A., (The Design of Experiments. The Design of Experiments, The design of experiments (1935), Oliver & Boyd: Oliver & Boyd Oxford, England), Pages xi, 251 · JFM 61.0566.03 |
| [56] | Freyaldenhoven, Simon; Hansen, Christian; Pérez, Jorge Pérez; Shapiro, Jesse M., Visualization, identification, and estimation in the linear panel event-study design, Adv. Econ. Econom. Twelfth World Congress (2021) |
| [57] | Freyaldenhoven, Simon; Hansen, Christian; Shapiro, Jesse M., Pre-event Trends in the Panel Event-Study Design, Amer. Econ. Rev., 109, 9, 3307-3338 (2019) |
| [58] | Gardner, John, 2021. Two-stage differences in differences. Working Paper. |
| [59] | Goodman-Bacon, Andrew, Difference-in-differences with variation in treatment timing, J. Econometrics, 225, 2, 254-277 (2021) · Zbl 07414291 |
| [60] | Gruber, Jonathan, The Incidence of Mandated Maternity Benefits, Am. Econ. Rev., 84, 3, 622-641 (1994) |
| [61] | Hagemann, Andreas, Inference with a single treated cluster, 1-23 (2020) |
| [62] | Hagemann, Andreas, Permutation inference with a finite number of heterogeneous clusters (2021) |
| [63] | Hasegawa, Raiden B.; Webster, Daniel W.; Small, Dylan S., Evaluating Missouri’s Handgun Purchaser Law: A Bracketing Method for Addressing Concerns About History Interacting with Group, Epidemiology, 30, 3, 371-379 (2019) |
| [64] | Heckman, James; Ichimura, Hidehiko; Smith, Jeffrey; Todd, Petra, Characterizing Selection Bias Using Experimental Data, Econometrica, 66, 5, 1017-1098 (1998) · Zbl 1055.62573 |
| [65] | Heckman, James J.; Ichimura, Hidehiko; Todd, Petra, Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme, Rev. Econom. Stud., 64, 4, 605-654 (1997) · Zbl 0887.90039 |
| [66] | Holland, Paul W., Statistics and Causal Inference, J. Am. Stat. Assoc., 81, 396, 945-960 (1986) · Zbl 0607.62001 |
| [67] | Huber, Martin; Steinmayr, Andreas, A Framework for Separating Individual-Level Treatment Effects From Spillover Effects, J. Bus. Econ. Stat., 39, 2, 422-436 (2021) |
| [68] | Ibragimov, Rustam; Müller, Ulrich K., Inference with few heterogeneous clusters, Rev. Econ. Stat., 98, 1, 83-96 (2016) |
| [69] | Imai, Kosuke; Kim, In Song, On the use of two-way fixed effects regression models for causal inference with panel data, Political Anal., 29, 3, 405-415 (2021) |
| [70] | Jakiela, Pamela, Simple Diagnostics for Two-Way Fixed Effects (2021) |
| [71] | Kahn-Lang, Ariella; Lang, Kevin, The Promise and Pitfalls of Differences-in-Differences: Reflections on 16 and Pregnant and Other Applications, J. Bus. Econom. Statist., 38, 3, 613-620 (2020) |
| [72] | Keele, Luke J.; Small, Dylan S.; Hsu, Jesse Y.; Fogarty, Colin B., Patterns of Effects and Sensitivity Analysis for Differences-in-Differences (2019) |
| [73] | Khan, Shakeeb; Tamer, Elie, Irregular Identification, Support Conditions, and Inverse Weight Estimation, Econometrica, 78, 6, 2021-2042 (2010) · Zbl 1206.91068 |
| [74] | Lee, Sokbae; Okui, Ryo; Whang, Yoon-Jae Jae, Doubly robust uniform confidence band for the conditional average treatment effect function, J. Appl. Econometrics, 32, 7, 1207-1225 (2017) |
| [75] | Liang, Kung-Yee; Zeger, Scott L., Longitudinal data analysis using generalized linear models, Biometrika, 73, 1, 13-22 (1986) · Zbl 0595.62110 |
| [76] | Lin, Winston, Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique, Ann. Appl. Stat., 7, 1, 295-318 (2013) · Zbl 1454.62217 |
| [77] | Liu, Licheng; Wang, Ye; Xu, Yiqing, A practical guide to counterfactual estimators for causal inference with time-series cross-sectional data, Am. J. Polit. Sci. (2022), Forthcoming |
| [78] | MacKinnon, James G.; Nielsen, Morten Orregaard; Webb, Matthew D., Cluster-robust inference: A guide to empirical practice, J. Econometrics, 232, 2, 272-299 (2023) · Zbl 07648714 |
| [79] | MacKinnon, James G.; Webb, Matthew D., The wild bootstrap for few (treated) clusters, Econom. J., 21, 2, 114-135 (2018) · Zbl 07547388 |
| [80] | Malani, Anup; Reif, Julian, Interpreting pre-trends as anticipation: Impact on estimated treatment effects from tort reform, J. Publ. Econ., 124, 1-17 (2015) |
| [81] | Manski, Charles F.; Pepper, John V., How Do Right-to-Carry Laws Affect Crime Rates? Coping with Ambiguity Using Bounded-Variation Assumptions, Rev. Econ. Stat., 100, 2, 232-244 (2018) |
| [82] | Marcus, Michelle; Sant’Anna, Pedro H. C., The role of parallel trends in event study settings : An application to environmental economics, J. Assoc. Environ. Resour. Econ., 8, 2, 235-275 (2021) |
| [83] | McKenzie, David, Beyond baseline and follow-up: The case for more T in experiments, J. Dev. Econ., 99, 2, 210-221 (2012) |
| [84] | Meyer, Bruce D., Natural and Quasi-Experiments in Economics, J. Bus. Econ. Stat., 13, 2, 151-161 (1995) |
| [85] | Neyman, Jerzy, On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9, Statist. Sci., 5, 4, 465-472 (1923) · Zbl 0955.01560 |
| [86] | Olea, José Luis Montiel; Plagborg-Mller, Mikkel, Simultaneous confidence bands: Theory, implementation, and an application to SVARs, J. Appl. Econometrics, 34, 1, 1-17 (2019) |
| [87] | Rambachan, Ashesh; Roth, Jonathan, Design-Based Uncertainty for Quasi-Experiments (2022) |
| [88] | Rambachan, Ashesh; Roth, Jonathan, A more credible approach to parallel trends, Rev. Econ. Stud. (2022), Forthcoming |
| [89] | Robins, J. M., A New Approach To Causal Inference in Mortality Studies With a Sustained Exposure Period - Application To Control of the Healthy Worker Survivor Effect, Math. Model., 7, 1393-1512 (1986) · Zbl 0614.62136 |
| [90] | Roth, Jonathan, Should We Adjust for the Test for Pre-trends in Difference-in-Difference Designs? (2018) |
| [91] | Roth, Jonathan, Pre-test with Caution: Event-study Estimates After Testing for Parallel Trends, Am. Econ. Rev. Insights, 4, 3, 305-322 (2022) |
| [92] | Roth, Jonathan; Sant’Anna, Pedro H. C., Efficient Estimation for Staggered Rollout Designs (2021) |
| [93] | Roth, Jonathan; Sant’Anna, Pedro H. C., When Is Parallel Trends Sensitive to Functional Form?, Econometrica, 91, 2, 737-747 (2023) · Zbl 1541.62380 |
| [94] | Rubin, Donald B., Estimating causal effects of treatments in randomized and nonrandomized studies, J. Edu. Psychol., 66, 5, 670-688 (1974) |
| [95] | Ryan, Andrew M., Well-Balanced or too Matchy-Matchy? The Controversy over Matching in Difference-in-Differences, Health Serv. Res., 53, 6, 4106-4110 (2018) |
| [96] | Sant’Anna, Pedro H. C.; Zhao, Jun, Doubly robust difference-in-differences estimators, J. Econometrics, 219, 1, 101-122 (2020) · Zbl 1464.62254 |
| [97] | Schmidheiny, Kurt; Siegloch, Sebastian, On Event Studies and Distributed-Lags in Two-Way Fixed Effects Models: Identification, Equivalence, and GeneralizationSSRN Scholarly Paper 3571164 (2020), Social Science Research Network: Social Science Research Network Rochester, NY |
| [98] | Shaikh, Azeem; Toulis, Panos, Randomization tests in observational studies with staggered adoption of treatment, J. Am. Stat. Assoc., 116, 536, 1835-1848 (2021) · Zbl 1506.62254 |
| [99] | Strezhnev, Anton, 2018. Semiparametric weighting estimators for multi-period difference-in-differences designs. Working Paper. |
| [100] | Sun, Liyang; Abraham, Sarah, Estimating dynamic treatment effects in event studies with heterogeneous treatment effects, J. Econometrics, 225, 2, 175-199 (2021) · Zbl 07414288 |
| [101] | Viviano, Davide; Bradic, Jelena, Dynamic covariate balancing: estimating treatment effects over time (2021) · Zbl 07693690 |
| [102] | Wager, Stefan; Athey, Susan, Estimation and Inference of Heterogeneous Treatment Effects using Random Forests, J. Amer. Statist. Assoc., 113, 523, 1228-1242 (2018) · Zbl 1402.62056 |
| [103] | Wooldridge, Jeffrey M., Cluster-Sample Methods in Applied Econometrics, Am. Econ. Rev. P&P, 93, 2, 133-138 (2003) |
| [104] | Wooldridge, Jeffrey M., 2021. Two-Way Fixed Effects, the Two-Way Mundlak Regression, and Difference-in-Differences Estimators. Working Paper, pp. 1-89. |
| [105] | Ye, Ting; Keele, Luke; Hasegawa, Raiden; Small, Dylan S., A Negative Correlation Strategy for Bracketing in Difference-in-Differences (2021) |
| [106] | Zeldow, Bret; Hatfield, Laura A., Confounding and regression adjustment in difference-in-differences studies, Health Serv. Res., 56, 5, 932-941 (2021) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
