Replicating Abadie et al. (2010) with the
+augsynth package
+Delaney & Miratrix
+2024-11-16
+ +This vignette replicates the analysis and figures in Abadie et
+al. (2010) using the augsynth package. In particular,
+our recent extensions to the augsynth package streamline
+permutation-based inference and associated visualizations on behalf of
+the user. In the following, we show how to easily generate results
+similar (but not entirely identical) to Figures 2-7 in the original
+paper.
While the general intuition of Abadie et al.’s analysis is replicated
+in the augsynth package, it is worth noting that covariates
+and model parameters are handled somewhat differently than in the
+original analysis. In particular, unlike the augsynth
+package, the Synth package used in Abadie et al.’s analysis
+reweights a subset of the lagged outcomes and passed auxiliary
+covariates when generating synthetic weights. The augsynth
+package, by contrast, directly balances all pre-treatment
+outcomes along with auxillary covariates, which means it does not
+dynamically determine variable importance weights.
In this vignette we also include Synth code to generate
+the reported Abadie et al.’s results with a high degree of fidelity
+(with the resulting Synth-based plots shown adjacent to
+analogous output from augsynth models) for comparison
+purposes. We do note that we show all the post-treatment years,
+while the original paper only shows the first 10 years
+post-treatment.
Replication with augsynth
+When fitting our model, we found that using
+progfunc = "none" (meaning no ridge regression bias
+adjustment step, which would be seemingly more similar to Abadie) then
+the results deviate more strongly from the tobacco paper, but if we
+allow augsynth to do its default bias removal, we more closely replicate
+the paper. Thus, we use ridge regresssion in this vignette, but then use
+the “R-statistic” permutation approahc from Abadie for inference:
# Basic SCM with covariates
+syn <- augsynth(form = cigsalepcap ~ treated | retail + xxincome + K1_r_15_24 + beer,
+ unit = state,
+ time = year,
+ data = tobacco_70
+ #progfunc = "none"
+)
+syn_rstat <- summary( syn, inf_type = 'permutation_rstat' )To get the outcome plot (Figure 2), call
+plot(plot_type = 'outcomes') on an augsynth
+object or summary object:
p <- plot(syn_rstat, plot_type = 'outcomes raw average') +
+ ggtitle("Replication of Abadie et al. (2010), Figure 2")
Note our version has a bit of “Figure 1” in it, with the raw outcome
+trajectory. If desired, setting plot_type = 'outcomes' will
+drop the raw average of donor units and the synthetic
+counterfactual.
To get Table 1, the comparison of the means of various covariates, we +can use
+covariate_balance_table( syn, pre_period = c( 1975, 1980, 1988 ) ) %>%
+ knitr::kable( digits = 2 )| variable | +Tx | +Co | +Raw | +
|---|---|---|---|
| retail | +89.42 | +90.40 | +87.27 | +
| xxincome | +10.03 | +9.97 | +9.79 | +
| K1_r_15_24 | +17.87 | +17.88 | +17.83 | +
| beer | +24.28 | +23.25 | +23.66 | +
| 1975 | +127.10 | +121.48 | +136.93 | +
| 1980 | +120.20 | +123.65 | +138.09 | +
| 1988 | +90.10 | +96.35 | +113.82 | +
For Figure 3, we compare the estimated impact to that calculated in +the original paper:
+p <- plot(syn_rstat, plot_type = 'estimate only') +
+ ggtitle("Replication of Abadie et al. (2010), Figure 3") +
+ scale_y_continuous(limits = c(-30, 30))
We see a bit more choppyness than the Synth package.
+We can finally plot our result, using
+plot_type = "placebo", to obtain a plot showing each
+placebo trajectory:
plot(syn_rstat, plot_type = 'placebo' ) +
+ ggtitle("Replication of Abadie et al. (2010), Figure 4")
We can get the raw trajectories via
+placebo_distribution( syn_rstat )#> # A tibble: 1,404 × 8
+#> state trt year Yhat Yobs ATT RMSPE rstat
+#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
+#> 1 Alabama 0 1970 91.5 89.8 -1.67 1.14 -1.47
+#> 2 Alabama 0 1971 96.5 95.4 -1.06 1.14 -0.926
+#> 3 Alabama 0 1972 100. 101. 1.08 1.14 0.941
+#> 4 Alabama 0 1973 104. 103. -1.15 1.14 -1.00
+#> 5 Alabama 0 1974 108. 108. 0.0232 1.14 0.0203
+#> 6 Alabama 0 1975 112. 112. -0.269 1.14 -0.236
+#> 7 Alabama 0 1976 115. 116. 1.10 1.14 0.964
+#> 8 Alabama 0 1977 117. 117. -0.149 1.14 -0.130
+#> 9 Alabama 0 1978 120. 123 2.62 1.14 2.29
+#> 10 Alabama 0 1979 120. 121. 1.38 1.14 1.21
+#> # ℹ 1,394 more rowsTo see what makes up our synthetic unit, we can use
+donor_table() to obtain a dataframe with the RMSPE and
+synthetic weight for each of the donor units.
donor_table(syn_rstat) %>%
+ arrange( -abs(weight) )#> # A tibble: 38 × 3
+#> state weight RMSPE
+#> <chr> <dbl> <dbl>
+#> 1 Colorado 0.545 2.68
+#> 2 Connecticut 0.363 2.69
+#> 3 Utah 0.106 2.70
+#> 4 Nevada 0.0935 5.11
+#> 5 Mississippi -0.0792 1.66
+#> 6 Arkansas -0.0689 1.73
+#> 7 Virginia 0.0560 1.95
+#> 8 New Mexico 0.0522 1.43
+#> 9 Oklahoma -0.0480 2.27
+#> 10 Tennessee -0.0445 1.46
+#> # ℹ 28 more rowsCalifornia is apparently mainly a mix of Colorado and Connecticut, +with a bit of Utah and Nevada thrown in. The negative weights, and the +lack of sparse weights, are from the ridge regresssion. Without that +second stage, these weights would generally be sparse, as in the +original paper. For example:
+syn_classic <- augsynth(form = cigsalepcap ~ treated | retail + xxincome + K1_r_15_24 + beer,
+ unit = state,
+ time = year,
+ data = tobacco_70,
+ progfunc = "none" )
+donor_table(syn_classic) %>%
+ filter( weight!= 0 ) %>%
+ arrange( -abs(weight) )#> # A tibble: 5 × 3
+#> state weight RMSPE
+#> <chr> <dbl> <dbl>
+#> 1 Colorado 0.564 4.62
+#> 2 Connecticut 0.342 5.30
+#> 3 Utah 0.0508 24.4
+#> 4 New Mexico 0.0274 3.58
+#> 5 Nevada 0.0156 9.17Sensitivity checks by limiting the donor pool
+The augsynth models can be re-fitted after excluding certain donor
+units via the update_augsynth() method. The main method is
+to set drop to a multiplier for the treatment unit RMSPE.
+All units with a RMSPE larger than that value will be dropped:
update_augsynth(syn) %>% # drops units with >20x treated RMSPE by default
+ summary( inf_type = 'permutation' ) %>%
+ plot(plot_type = 'placebo' ) +
+ ggtitle("Replication of Abadie et al. (2010), Figure 5") + ylim(-51, 91) +
+ annotate('text', y = -48, x = 1970,
+ label = "Removes donors with 20x California's pre-treatment RMSPE",
+ hjust = 0, color = 'darkseagreen', size = 3)
+update_augsynth(syn, drop = 5) %>%
+ summary( inf_type = 'permutation' ) %>%
+ plot(plot_type = 'placebo') +
+ ggtitle("Replication of Abadie et al. (2010), Figure 6") + ylim(-51, 91) +
+ annotate('text', y = -48, x = 1970,
+ label = "Removes donors with 5x California's pre-treatment RMSPE",
+ hjust = 0, color = 'darkseagreen', size = 3)
+update_augsynth(syn, drop = 2) %>%
+ summary( inf_type = 'permutation' ) %>%
+ plot(plot_type = 'placebo') +
+ ggtitle("Replication of Abadie et al. (2010), Figure 7") + ylim(-51, 91) +
+ annotate('text', y = -48, x = 1970,
+ label = "Removes donors with 2x California's pre-treatment RMSPE",
+ hjust = 0, color = 'darkseagreen', size = 3) 
Alternatively, the names of units to be excluded can be passed in
+directly as characters using drop.
drop_states <- c("Iowa", "Arizona", "Alabama", "Illinois", "Indiana", "Idaho", "Connecticut",
+ "New Mexico", "Texas", "Utah", "North Dakota", "South Dakota", "Vermont",
+ "Wisconsin", "West Virginia", "Wyoming", "Tennessee", "Pennsylvania")
+
+update_augsynth(syn, drop = drop_states) %>%
+ summary( inf_type = 'permutation' ) %>%
+ plot(plot_type = 'placebo') 