ESC: short-panel prediction intervals (Prop 99, São Paulo, Monte Carlo)#
This page documents the verification of mlsynth’s Ensemble Synthetic Control (ESC) estimator (Ensemble Synthetic Control, Lian & Pu 2026). ESC’s contribution is inference under a short pre-treatment panel, so it is validated on two empirical case studies and on the paper’s Monte Carlo coverage evidence.
Empirical – California (Proposition 99)#
On the canonical cigarette panel (California plus 38 donor states, treatment from 1989, \(T_0 = 19\)), ESC reproduces the classic synthetic-control result while its prediction interval remains informative. The ensemble counterfactual correlates about \(0.998\) with the convex synthetic California, and the headline effect matches the literature:
Quantity |
ESC |
classic SC |
Abadie et al. |
|---|---|---|---|
avg post-treatment gap (packs) |
\(\approx -22.7\) |
\(-19.5\) |
\(\approx -19\) |
2000 gap (packs) |
\(\approx -33.6\) |
\(-26.6\) |
\(\approx -26\) |
pre-treatment RMSE |
\(\approx 1.64\) |
\(1.66\) |
– |
The ensemble path matches the convex synthetic California in direction and magnitude at a comparable pre-treatment fit, and its \(90\%\) pointwise band places observed California below the lower bound in every post-treatment year, so the effect reads as significant under ESC’s predictive inference. See esc_prop99.py.
Empirical – São Paulo homicides (short pre-window)#
The São Paulo homicide study (Freire 2018; treated 1999, \(T_0 = 9\) annual pre-periods, 24 donor states) is the short-panel regime ESC is designed for. Here the value of ESC is the honesty of its interval rather than a point estimate: the ensemble reports an average effect of about \(-7\) per \(100{,}000\) (with the gap growing to roughly \(-20\) by 2009) and a wide \(90\%\) interval on the number of lives saved — on the order of \([14{,}000,\ 42{,}000]\) around a central \(\approx 32{,}000\). That width nearly spans the range that the individual synthetic-control variants produce as point estimates, so ESC surfaces as one interval the model uncertainty that a single specification hides. The observed series clears the lower bound of the band only in the later post-treatment years, consistent with the across-donor placebo evidence that the effect is large but marginal under nine pre-periods. See esc_saopaulo.py.
Monte Carlo – interval coverage (open reproduction)#
The paper’s central claim is coverage: in short, high-dimensional panels the ESC prediction interval attains close to its nominal level. This did not reproduce from the equations as written. Drawing DGP-1 factor-model panels (\(T_0 = 8\), \(J = 15\)) and measuring pointwise coverage of the realized counterfactual \(Y_{1t}(0)\), a faithful port of the equation-(7) ensemble-quantile band covers only about \(0.5\) to \(0.65\) against a \(0.90\)–\(0.95\) nominal level – under the recommended Local + \(\lambda_{1se}\) rule and across block lengths and perturbation strengths. The reason is structural: the ensemble spread reflects weight-estimation variability, not the treated unit’s irreducible disturbance \(u_t\), so the band is an estimation-uncertainty band rather than a full prediction interval, and adding a resampled residual narrows but does not close the gap.
This is a reproduction gap, not necessarily a defect: the authors’ esc
package (Python/R/Stata) is not yet released, so the predictive component that
would lift coverage to nominal cannot be inspected. A calibrated coverage
benchmark is therefore deferred until the reference implementation is available;
the shipped estimator implements the ensemble band exactly as specified.