Ridge lambda cross-validation – against augsynth on two panels

The ridge penalty in Vanilla Synthetic Control (VanillaSC)’s Augmented SCM (augment="ridge") is selected by leave-one-pre-period-out cross-validation and a 1-SE rule. This page records that selection being cross-validated against the authors’ own R package, ebenmichael/augsynth at commit 7a90ea48 (0.2.0), and two defects it uncovered.

A wrong penalty does not announce itself. It is not an error or an implausible number – it is a well-behaved fit at the wrong amount of shrinkage, which then propagates into every effect and every interval built on top of it.

Two panels, one of which cannot see the problem

panel	shape	role
augsynth’s kansas	89 pre-periods, 49 donors	negative control – the selected penalty is identical whether the code is right or wrong
Song et al. PM2.5, “2+26 cities” 2015	25 pre-periods, 37 donors	diagnostic – the final pre-period is a ~9x outlier fold, so the defects move the answer

The Kansas panel agreed on the selected penalty to fifteen significant figures both before and after the fixes below. Any suite testing only that panel would report the cross-validation as correct while it was not, which is exactly what happened. The second panel was added because its final pre-treatment period sits on the seasonal ramp into the heating season and carries about nine times the held-out error of the other twenty-four, so whether it counts as a fold at all moves the whole curve.

What agreed already

The lambda grid and the selection rule were faithful and are now pinned. create_lambda_list builds lambda_max * scaler^k for k in 0..n_lambda, which is \(n_{\lambda}+1\) points rather than \(n_{\lambda}\) – an easy off-by-one to inherit from R’s seq(0:n_lambda) - 1. lambda_max is the squared largest singular value of the centered control matrix, and matches to 1e-13 on the outcome-only specification. choose_lambda takes the largest lambda whose mean error is within one standard error of the minimum, with the comparison inclusive.

Two defects, and why one hid the other

The fold count was one too many. augsynth’s get_lambda_errors loops i in 1:(ncol(X_c) - holdout_length) and sizes its error matrix to match, so with a holdout length of one the final pre-period is never held out. mlsynth iterated one extra fold. On Kansas that is a 1-in-89 perturbation of an average and changed nothing detectable. On the Song cell the extra fold contributes about nine times the mean error, which moved the curve by up to 22 and the selected penalty from augsynth’s 28.5157 to 11.3523 – a factor of 2.5 in shrinkage.

The standard error used the wrong denominator. augsynth reports sd(x) / sqrt(length(x)) and R’s sd is the sample standard deviation, while numpy’s .std defaults to the population one. The reported error was therefore too small by \(\sqrt{(n-1)/n}\). This is not cosmetic: it feeds the 1-SE threshold directly, so it can change which penalty is chosen.

After both fixes, on the outcome-only specification:

quantity	kansas	song
lambda grid	1e-13	1e-13
CV error curve	1e-10 (was 5.1e-6)	1e-6 (was 22.2)
CV standard error	1e-10	2.5e-7
selected penalty	exact	exact (was 11.35 vs 28.52)

What the fix exposed

Fixing the cross-validation made the covariate cells of Ridge ASCM — Augmented Synthetic Control (Ben-Michael, Feller & Rothstein 2021) disagree more, from about 0.002 to 0.005 on the ATT. That is worth stating plainly, because the surface reading is the opposite: it was a cancellation being removed, not a regression.

Two independent errors had been partly offsetting each other. The fold off-by-one was one; the other was in how the benchmark aggregated auxiliary covariates to one value per unit. augsynth omits missing values column-wise (mean(x, na.rm = TRUE) per covariate, with na.action = NULL passed to model.frame); the harness was omitting them row-wise, discarding every quarter in which either of the two annually reported revenue series was missing. On the Kansas panel that is 56 of 89 pre-treatment quarters, thrown away from all six covariates rather than from the two that are sparse, which put lambda_max at 128.4583 against augsynth’s 128.6077. The wrong fold count happened to select a point on that already-wrong grid that landed nearer augsynth’s answer.

With both corrected, the covariate specification is exact: ATT -0.060937 and pre-fit L2 0.053855 against augsynth’s own values, matching the two outcome-only specifications. The aggregation rule now lives in aggregate_covariates in benchmarks/cases/ascm_kansas.py, and TestCovariateAggregation in mlsynth/tests/test_bilevel_ridge.py pins it – including how far the row-wise rule would move the four fully observed covariates, so the two rules cannot be confused again silently.

The residualized specification still differs, by about 0.004. That one is deliberate and documented on Ridge ASCM — Augmented Synthetic Control (Ben-Michael, Feller & Rothstein 2021): augsynth’s residual lambda-CV is ill-posed once K covariates are projected out, so mlsynth tunes that penalty on the outcome scale instead.

Verification

Pinned in mlsynth/tests/test_ascm_ridge_cv.py (16 tests), which asserts the grid, the curve, the standard error and the selected penalty on both panels, and separately pins the fold count and the fact that the final pre-period is never held out so the off-by-one cannot silently return. Gold is generated by benchmarks/reference/ascm_ridge_cv/reference.R and committed, so the tests need no R toolchain.