DTWSC – Cao and Chadefaux (2025) on the Basque panel

DTWSC: Dynamic Synthetic Control is cross-validated against the authors’ own R package, conflictlab/dsc (MIT), pinned at commit b1cd241. The reference is not vendored; the generator that produces the comparison dump lives in benchmarks/reference/dtwsc_basque/ and the dependency chain it needs is scripted in benchmarks/R/install_dtwsc.sh.

The setup is the reference package’s own README example: Synth’s basque panel, 1955–1997, treatment in 1970, the national aggregate dropped so 16 donors remain, the full 14-predictor Abadie specification, k = 4, filter.width = 5, buffer = 0, n.burn = 3, symmetricP1 for the first alignment and asymmetricP2 for the second.

Headline numbers

method	pre-RMSE	ATT
standard synthetic control	0.0886	-0.6027
DSC, R reference	0.0705	-0.5579
DTWSC, mlsynth’s warp through R’s Synth	0.0705	-0.5592
DTWSC, mlsynth end to end	0.0663	-0.5310
standard SC, mlsynth end to end	0.0845	-0.6232

The paper’s claim reproduces: warping tightens the pre-treatment fit by about 20 percent, and moves the ATT modestly toward zero.

Read the units before comparing any of these. dsc(rescale = TRUE) is the reference’s default, and it maps every unit onto a common pre-treatment range before fitting anything, so every number R reports is in rescaled units rather than in GDP per capita. The mlsynth rows above are run with rescale_units=True so the whole table sits on one scale; without it mlsynth reports in the outcome’s own units, where R’s standard-SC ATT is -0.6405 and not -0.6027.

That distinction is easy to lose, and this page previously lost it: an mlsynth ATT of -0.6026 in GDP per capita was compared against R’s -0.6027 in rescaled units and reported as a four-decimal reproduction. It was a coincidence of two different quantities landing on the same digits. On one scale the two arms agree to about 0.004 on pre-RMSE and 0.03 on the ATT, which is the honest figure and is what benchmarks/cases/dtwsc_basque.py now pins.

Do not read that ATT agreement as agreement between the counterfactual paths, either. Comparing the two pointwise rather than in the mean, the pre-treatment halves sit almost on top of each other – worst single-period gap 0.027 warped and 0.036 unwarped, against a series spanning 2.18 rescaled units – while the post-treatment halves reach 0.19 and 0.16. The paths cross rather than running parallel, so most of that error cancels in the average and a 0.19 divergence presents as a 0.03 ATT gap.

That is the expected signature of the Savitzky–Golay difference described below: it is held down where the pre-treatment fit constrains it and free to grow where nothing does. The pointwise maximum is pinned in the benchmark alongside the ATT, because a regression that tilted the counterfactual while leaving its mean intact would pass an ATT check untouched.

What matches, seam by seam

The comparison holds the input fixed at R’s own filtered panel, so only the warping is being tested.

quantity	agreement
cutoff	16/16 donors exact
weight.a (first-phase speeds)	16/16 bit-identical
avg.weight (second-phase speeds)	16/16, worst 2.2e-16 (one ULP)
warped donor series	16/16, worst 7.1e-15
second-phase window search (j_opt, margin, candidate count)	28/28 windows on the worst-fitting donor
outlier-filter decisions	13888/13888 cells

These rows are pinned in benchmarks/cases/dtwsc_basque.py rather than left as a one-off measurement, so a regression in the warping engine fails the benchmark.

Two findings worth keeping

The alignment kernel is exact, across every step pattern the method uses. mlsynth implements them directly rather than taking a DTW dependency, so a mistyped recursion coefficient is the kind of error that would produce a plausible warp and a wrong answer. Six are supported – symmetricP1, symmetricP2, asymmetricP1, asymmetricP2, typeIc and mori2006 – and each is checked against R dtw 1.23-3 on committed gold in benchmarks/reference/dtwsc_dtw/: 214 admissible alignments agree on the accumulated cost, the normalised cost, and the full index1/index2 warping paths exactly, while 138 inadmissible cases raise where R errors. warp and ref_too_short, which consume those alignments, are checked under all six as well.

Six rather than two because of how the method is meant to be used. The authors do not fix the step pattern; they select it per run by grid search over seven, and the choices shipped in their replication archive span six. An implementation carrying only the two their published example happens to use could reproduce 39 percent of their Basque design, 31 percent of the California one and 50 percent of the German one – so the omission was not a matter of completeness but of whether the paper’s own procedure is reachable at all.

One detail was worth getting right rather than approximating. Normalisation is not uniform across the patterns: the symmetric ones divide the accumulated cost by the query plus reference length, the asymmetric ones by the query’s, and mori2006 by the reference’s. Treating that as a two-way choice leaves every path correct and every normalised distance wrong, which changes nothing visible until an open-ended alignment picks its endpoint – and that endpoint decides which windows ref_too_short rules out of the second-phase search.

One bit was load-bearing. An earlier version of this page recorded a residual disagreement in 40 of the 13888 outlier-filter decisions and attributed it to floating-point ties that no implementation could be expected to reproduce. That was wrong, and the way it was wrong is worth keeping.

second_dtw discards outlying speeds with a type-7 \(Q_3 + 3\, \mathrm{IQR}\) fence. The speeds are small-denominator rationals, so a column routinely looks like \((1, 1, 7/6, 1)\) – and on such a column the fence evaluates to exactly \(7/6\), landing on the very value it is judging. Which side of that comparison the value falls on is then decided by the last bit of arithmetic, and dropping the cell rather than keeping it moves the column mean by \(1/24\).

The last bit was ours to get right. R’s stats::filter smooths the speeds by accumulating \(x_k \cdot (1/w)\) across the window; taking the window’s mean instead computes \((\sum x_k)/w\), which rounds 7/6 to 1.1666666666666667 where R gets 1.1666666666666665. Everything downstream inherited that one bit. Matching R’s summation order makes the first-phase speeds bit-identical and the entire warping engine exact, which is what the table above now records. A tolerance on the fence does not work, and was tried: because the disagreements run in both directions, widening the fence fixes three donors and breaks four.

The general lesson is about the method rather than the port. A statistic whose value turns on which side of an exactly-coincident bound a number falls is fragile by construction, and DSC has one in its inner loop. mlsynth reproduces the reference’s choice here, but a cross-language check of a DSC ATT should not be read as validating the last digit.

A second artefact is inherited from the method rather than the port. Three donors’ warped series end one period short of 1997, so the reference’s own counterfactual is NA there and its ATT is really taken over 1971–1996. mlsynth reproduces that rather than extrapolating over it, and reports the number of dropped periods in res.metadata["n_post_periods_undefined"].

The one remaining difference

Every seam above is measured with R’s own filtered panel substituted for mlsynth’s, which is the honest way to test the warping engine on its own. Run end to end on mlsynth’s own preprocessing instead, 5 of the 16 donors pick up a different alignment and the warped series diverge. That difference is the Savitzky–Golay stage, and it is the only one left.

Before differentiating, the reference extends each series at both ends with a two-period forecast::auto.arima forecast, filters the extended series, and trims back. mlsynth edge-pads instead. The filters agree to 1e-16 across the interior of every series and disagree only over the two points at each end – but a second derivative is small, and at those four points the disagreement reaches 0.083 against a series whose own scale is 0.090. Two of them sit at the start of the pre-treatment window the first alignment learns from.

This is recorded rather than fixed, and deliberately. Closing it would take a bit-exact auto.arima, not a good one. Because the outlier fence described above lands exactly on the values it is testing, an approximate forecast would fall on an arbitrary side of it and would not track the reference any more reliably than edge-padding does – while adding a heavyweight dependency and a stepwise order search to a preprocessing step. The gap is bounded, measured, and pinned in the benchmark case as dtwsc_speed_max_abs_diff_vs_r.

Placebo inference

inference="placebo" implements the paper’s two inferential procedures. Run on the Basque panel through the public API – outcome-only synthetic control, fixed warping hyperparameters, 17 pools and 256 placebo runs:

quantity	mlsynth	paper
efficiency test t	-6.79	-7.91
efficiency test p	9.0e-11	< 0.0001
mean log(MSE_DSC / MSE_SC)	-0.226	-0.18 (from the reported log MSEs)
implied MSE reduction	20 percent	16 percent

The direction and rough magnitude reproduce, but read the table as a comparison of two different procedures rather than a reproduction attempt. The authors’ replication archive (Dataverse 10.7910/DVN/DIUPUA) shows their placebo design differs from mlsynth’s built-in one in five ways: they keep Spain in the donor pool where the estimator page’s example drops it; they use 118 perturbed datasets and 1789 runs against mlsynth’s 17 pools and 256; they give every run its own grid-optimal filter.width / k / step pattern where mlsynth holds one configuration fixed; they fit the full 14-predictor specification on every placebo where mlsynth’s example is outcome-only; and they map each gap back to the outcome’s own units before squaring it, so their reported log MSEs are not on the scale mlsynth reports.

An earlier version of this page said the rule that picked those per-run hyperparameters was absent from the replication package. That was wrong, and in the useful direction: the archive ships the resulting choices themselves, per run and per panel, in Figure_5_*_gridOpt.Rds. The selection rule is indeed not shown, but it is not needed – the choices can be read directly.

So the table above is a comparison of two procedures that happen to measure the same thing, not a replication of the reported statistic. Reproducing the published t means adopting their design wholesale – pool, datasets, per-run hyperparameters, predictors, and units – rather than tightening anything in mlsynth’s own.

The band does not reproduce, and that is worth stating plainly. On this specification the warped band comes out marginally wider than the unwarped one – 2.436 against 2.296, or 0.3 percent – where the paper reports it narrower.

The two results are not in conflict. The efficiency test is a within-run paired comparison of MSE; the band is the cross-run spread of gaps. Lowering each run’s error need not shrink the dispersion across runs, because the band is a tail quantity.

Two candidate explanations have been tested and ruled out. The first was the synthetic-control specification: the paper fits Synth over 14 predictors where the default here matches on the outcome alone, so sc_backend="malo" with the paper’s predictors ought to have tightened the band. It does the opposite – the band widens to 2.9 percent and the efficiency test collapses (\(t = +0.56\), MSE reduction \(-2\) percent). The second was fidelity of the warp itself, which is now bit-exact against the reference; fixing it moved the band from 1.3 percent wider to 0.3 percent wider, so it was a real contributor but not the explanation.

What remains untested is the per-run hyperparameter optimisation, and the pool construction: the paper draws its quantiles over roughly 2000 runs where the figures here use 256, and a 2.5 percent tail quantile is not well determined at that count. Neither has been measured, so neither is claimed.

Treat the efficiency test as the reproducible claim and the band as descriptive.

Cost. The placebo procedure runs one full two-phase warp per donor per pool. Basque with placebo_pairs=0 is 256 warped fits and takes about 12 minutes; the paper’s placebo_pairs=100 construction is roughly 2000 fits.