BEAST – Immunized Doubly-Robust Synthetic Control (Proposition 99)

Estimator:

Immunized Doubly-Robust Synthetic Control (BEAST) – mlsynth.BEAST

Source:

Bléhaut, D’Haultfœuille, L’Hour & Tsybakov (2021), “An alternative to synthetic control for models with many covariates under sparsity” (arXiv:2005.12225), with the authors’ R reference (jeremylhour/alternative-synthetic-control-sparsity).

Replication type:

Cross-validation – a live run of the authors’ own R code on the identical design matrix.

Status:

Done – the balancing coefficients match to five decimals; the immunized ATT reproduces the R (mean \(-22.44\), matched to ~0.02 packs; \(-31.51\) by 2000, to ~0.006), with a per-year path gap under ~0.15 packs and the standard errors matched to ~0.05.

Durable check:

benchmarks/cases/beast_prop99.py (beast_prop99), run against the pinned R reference under benchmarks/reference/beast_prop99/; plus mlsynth/tests/test_beast.py.

The target

The authors ship a Proposition 99 application with the paper (EX1_CaliforniaTobaccoConsumption.R): a panel synthetic control on the Abadie, Diamond & Hainmueller (2010) tobacco panel, where California raises its cigarette tax in 1989 and the donors are the other US states. Rather than fitting the outcome on a simplex, BEAST balances a covariate design – four economic predictors (loginc, p_cig, pct15-24, pc_beer) plus lagged cigarette sales (1975/1980/1988) – by an ℓ₁-penalized exponential tilt, then immunizes the effect with a WLS-Lasso outcome model. This is the paper’s basic informative-covariate regime: a handful of covariates that truly drive the selection.

Demonstrate-first port

Before wiring anything into mlsynth, the four R routines were ported to NumPy and validated cell by cell against a live run of the authors’ code on mlsynth’s exact design matrix (so any residual gap is purely R’s OWL-QN calibration versus the port’s proximal-gradient / FISTA solves, not a difference in inputs):

• CalibrationLasso.R \(\to\) calibration_lasso() – the exponential-tilting balancing weights with iterated Belloni–Chernozhukov–Hansen penalty loadings;

• OrthogonalityReg.R / LassoFISTA.R \(\to\) orthogonality_reg() – the immunizing WLS-Lasso outcome model;

• ImmunizedATT.R \(\to\) immunized_att() – the doubly-robust ATT and its closed-form standard error.

The R reference installs with no CRAN access: lbfgs (which the authors’ code depends on) compiles from the GitHub CRAN mirror against the system R. The captured reference (per-year ATT path and standard errors, the mean, and the 2000 endpoint) is vendored under benchmarks/reference/beast_prop99/ with its provenance and data checksum pinned.

Cell-by-cell agreement

On the identical design matrix, the port reproduces the R:

Quantity	mlsynth (NumPy)	R reference
balancing coefficients \(\boldsymbol\beta\)	agree to 5 decimals	–
mean post-1989 ATT	\(-22.44\)	\(-22.4443\)
ATT by 2000	\(-31.51\)	\(-31.5129\)
ATT path (per year)	max abs diff \(\sim 0.15\)	–
standard-error path	max abs diff \(\sim 0.05\)	–
sum of balancing weights	\(1.0\)	\(1.0\)
covariates selected	\(1\)	\(1\)

The balancing weights sum to one (a valid synthetic control), the ℓ₁ calibration selects a single informative covariate, and the top donor is Utah – the classic Proposition 99 synthetic California. The residual per-year gap in the ATT path (largest about 0.15 packs, averaging out to 0.02 on the mean) is the calibration solver difference (OWL-QN vs proximal-gradient), not a modelling difference.

The honest finding: BEAST’s operating envelope

The replication also mapped out where the method stops working. Run on an over-rich covariate specification – the SparseSC augmented California design, with many more predictors approaching the number of donors – the exponential tilt degenerates in both implementations: the R balancing weights sum to \(0\) and the Python to \(1.72\), neither of which is a valid synthetic control, and no penalty level recovers a sensible balance. There is no free lunch in the high-dimensional regime the title advertises unless the sparsity assumption actually holds.

That finding is what shaped the build. BEAST is deployed for the basic informative-covariate regime, and the estimator carries a balance-validity guard (balance_tol) that raises MlsynthEstimationError when \(\lvert \sum_j W_j - 1\rvert\) exceeds tolerance – turning the degenerate over-saturated case into an explicit error rather than a silently meaningless answer. mlsynth/tests/test_beast.py pins that behaviour (test_oversaturated_regime_is_rejected) alongside the R cross-validation of the basic regime.

Running it

python benchmarks/run_benchmarks.py --case beast_prop99

The case cross-checks the immunized ATT path and standard errors against the vendored R reference, and asserts the valid-balancing (\(\sum_j W_j = 1\)) and sparse-selection properties of the informative Proposition 99 regime.