Validation dashboard

Every estimator in mlsynth is checked against the original authors’ code on real data. This page is generated from the pinned reference bundles the test suite asserts against, so the numbers here cannot drift from what CI enforces. Each row links to the reference implementation, the dataset (with checksum), and the mlsynth case that runs the check.

Coverage: 61 cross-validation checks against original implementations across 35 estimators – 28 reproduce the reference to display precision, 21 to within two percent. A further 2 are captured on the next daily run (see Pending capture). Per-estimator paper replications (Path A / Path B) are catalogued in Replications.

Legend: exact (agreement to display precision), tight (worst relative deviation \(\le 2\%\)), close (\(\le 10\%\)), and documented (looser, with a stated reason on the estimator’s replication page – typically an intrinsically extrapolated or weakly-identified quantity).

Summary

Estimator	Checks	Agreement	Worst max |Δ|
BEAST	1	1 tight	0.15
BFSC	1	1 close	1
BVSS	1	1 tight	0.00041
CLUSTERSC	2	2 exact	0
ClusterSC	2	1 exact · 1 tight	0.036
DPSC	1	1 exact	0
FDID	1	1 exact	0.00032
GEOLIFT	2	2 exact	0.026
LINF	2	1 tight · 1 close	0.39
MAREX	1	1 tight	0.016
MASC	1	1 exact	4.6e-05
MCNNM	1	1 tight	0.81
MLSC	1	1 exact	1e-06
MicroSynth	1	1 tight	0.097
NSC	1	1 close	1.9
ORTHSC	2	1 exact · 1 close	0.037
PDA	5	4 exact · 1 close	0.056
PPSCM	1	1 tight	0.0022
PROPSC	1	1 exact	0
PROXIMAL	3	1 exact · 2 tight	41
RESCM	2	2 tight	0.0013
ROLLDID	1	1 exact	0
SBC	1	1 close	1e+06
SCD	1	1 exact	0
SCMO	1	1 tight	0.011
SCUL	1	1 tight	0.14
SDID	1	1 tight	0.0016
SI	1	1 exact	0
SNN	1	1 exact	0
SPILLSYNTH	4	1 exact · 1 tight · 1 close · 1 documented	7.6
SPSC	2	2 exact	0.0007
SSC	1	1 tight	0.001
SpSyDiD	2	1 exact · 1 close	0.094
TASC	1	1 documented	25
VanillaSC	10	4 exact · 4 tight · 2 close	0.11

BEAST

Reference	Dataset	#	max |Δ|	Verdict	Case
jeremylhour/alternative-synthetic-control-sparsity R (CalibrationLasso/OrthogonalityReg/ImmunizedATT)	augmented_cali_long.csv (974ae6ad6ab7…)	13	0.15	tight	beast_prop99

BFSC

Reference	Dataset	#	max |Δ|	Verdict	Case
author appendix Stan (via Rscript + rstan)	—	3	1	close	bfsc_prop99

BVSS

Reference	Dataset	#	max |Δ|	Verdict	Case
authors’ two-coordinate Gibbs (example2_fspda_2.R primitives), live run, captured	china_watches_long.csv (1ce8146af9a9…)	6	0.00041	tight	bvss_watches

CLUSTERSC

Reference	Dataset	#	max |Δ|	Verdict	Case
Bayani RPCA-SC – the author’s own code, vendored verbatim (vendor/bayani_rpca_synth: FPCA.R + RPCA_2.py)	—	9	0	exact — matches to display precision	clustersc_rpca_germany
scpi_pkg scest(w_constr={‘name’:’ridge’}) + df_EST	scpi_germany.csv (10b150fbcc2c…)	3	0	exact — matches to display precision	scpi_ridge_germany

ClusterSC

Reference	Dataset	#	max |Δ|	Verdict	Case
jehangiramjad/tslib RobustSyntheticControl (live run, captured), modelType=’svd’, kSingularValuesToKeep=3	smoking_data.csv (a13dd4d5d6e4…)	8	0.036	tight	pcr_rsc_ref
deshen24/panel-data-regressions var.var_est (homoskedastic + jackknife)	—	6	0	exact — matches to display precision	rsc_shen_coverage

DPSC

Reference	Dataset	#	max |Δ|	Verdict	Case
srho1/dpsc PrivateSC (differentially private SC)	—	4	0	exact — matches to display precision	dpsc_prop99

FDID

Reference	Dataset	#	max |Δ|	Verdict	Case
Kathleen T. Li’s Fun_FDID.R (MKSC replication, live run, captured)	GDP.csv (487c7007cad0…)	7	0.00032	exact — matches to display precision	fdid_hongkong

GEOLIFT

Reference	Dataset	#	max |Δ|	Verdict	Case
R package augsynth (via Rscript)	—	15	0.026	exact — matches to display precision	geolift_augsynth_ref
R GeoLiftMarketSelection (via Rscript)	—	15	0.005	exact — matches to display precision	geolift_marketselection_ref

LINF

Reference	Dataset	#	max |Δ|	Verdict	Case
LinfinitySC our(method=’inf’|’l1-inf’) (Wang, Xing & Ye 2025), BioAlgs/LinfinitySC	—	40	0.00041	tight	linf_crossval_ref
LinfinitySC our(method=’inf’) (Wang, Xing & Ye 2025), BioAlgs/LinfinitySC, lambda via param_selector(method=’inf’, n_folds=10)	smoking_data.csv (a13dd4d5d6e4…)	43	0.39	close	linf_prop99

MAREX

Reference	Dataset	#	max |Δ|	Verdict	Case
jinglongzhao2/SCDesign (cardinality-K design, open quadprog, live run)	walmart_weekly_sales_covariates.csv (906fb3cd9e2f…)	7	0.016	tight	marex_walmart

MASC

Reference	Dataset	#	max |Δ|	Verdict	Case
maxkllgg/masc masc(…, nogurobi=TRUE) (LowRankQP), live run, captured	basque_jasa.csv (b3f957771c8e…)	8	4.6e-05	exact — matches to display precision	masc_crossval

MCNNM

Reference	Dataset	#	max |Δ|	Verdict	Case
susanathey/MCPanel R (mcnnm_cv, defaults)	smoking_data.csv (a13dd4d5d6e4…)	13	0.81	tight	mcnnm_prop99

MLSC

Reference	Dataset	#	max |Δ|	Verdict	Case
leabottmer/multi-level-sc-estimator (mlSC_estimator, cvxpy+SCS)	—	4	1e-06	exact — matches to display precision	mlsc_bottmer

MicroSynth

Reference	Dataset	#	max |Δ|	Verdict	Case
R package microsynth (via Rscript)	—	6	0.097	tight	microsynth_seattle

NSC

Reference	Dataset	#	max |Δ|	Verdict	Case
Tian (2023) NSC.R (vendored, live run, captured), a*=0.3, b*=0.7	smoking_data.csv (a13dd4d5d6e4…)	23	1.9	close	nsc_prop99

ORTHSC

Reference	Dataset	#	max |Δ|	Verdict	Case
Fry GMM-SCE.R GMMSC() (R, live run, captured)	carbontax_fullsample_data.dta.txt (0df075d00fcc…)	15	0.037	close	gmmsce_carbontax
Fry OrthogonalizedSyntheticControl (R, live run, captured)	carbontax_fullsample_data.dta.txt (0df075d00fcc…)	5	0	exact — matches to display precision	orthsc_carbontax

PDA

Reference	Dataset	#	max |Δ|	Verdict	Case
Authors’ Fun/L2relax.R (ishwang1/L2relax-PDA), per UK firm, reproduced via cvxpy/ECOS, live run captured	brexit_long.parquet (1e5997075c1e…)	4	1e-06	exact — matches to display precision	pda_brexit
R package pampe (pampe(), live run, captured)	HongKong.csv (ad5b35ff563a…)	6	3.6e-05	exact — matches to display precision	pda_hcw_hongkong
Authors’ Fun/L2relax.R (ishwang1/L2relax-PDA) reproduced via cvxpy/ECOS, live run captured	HongKong.csv (ad5b35ff563a…)	24	0	exact — matches to display precision	pda_hongkong
Shi & Huang fsPDA application script (zhentaoshi/fsPDA, live run, captured)	china_watches_long.csv (1ce8146af9a9…)	4	0.056	close	pda_luxurywatch
Authors’ Fun/L2relax.R (ishwang1/L2relax-PDA) reproduced via cvxpy/ECOS, live run captured	china_ppi_long.csv (cc4cda27e17b…)	64	1e-06	exact — matches to display precision	pda_ppi

PPSCM

Reference	Dataset	#	max |Δ|	Verdict	Case
R augsynth::multisynth (live run, captured)	Teachingaugsynth.scv (59573a2dd46f…)	49	0.0022	tight	ppscm_paglayan

PROPSC

Reference	Dataset	#	max |Δ|	Verdict	Case
R package propsdid (via Rscript)	—	6	0	exact — matches to display precision	propsc_spain

PROXIMAL

Reference	Dataset	#	max |Δ|	Verdict	Case
R gmm (authors’ analysis.Rmd, commit 3bcb5ec, reltol=1e-13)	—	4	41	tight	dr_proximal_brazil
KenLi93/proximal_sc_manuscript NC_nocov + NC_nocov_gmm (over-identified, Newey-West q=10), live run, captured	scpi_germany.csv (10b150fbcc2c…)	11	1e-06	exact — matches to display precision	proximal_germany_oid
authors’ proximal code (freshtaste/proximal, cloned)	—	3	0.014	tight	proximal_panic1907

RESCM

Reference	Dataset	#	max |Δ|	Verdict	Case
scmrelax L2RelaxationCV (Liao-Shi-Zheng; github.com/metricshilab/scmrelax = github.com/YapengZheng/Relaxed_SC; MOSEK->CLARABEL; live run, captured)	balanced_gdp.csv (26fee37d55d9…)	6	0.0013	tight	rescm_balanced_gdp
scmrelax L2RelaxationCV (Liao-Shi-Zheng; github.com/metricshilab/scmrelax = github.com/YapengZheng/Relaxed_SC; MOSEK->CLARABEL; live run, captured)	—	10	0.00036	tight	rescm_relax_ref

ROLLDID

Reference	Dataset	#	max |Δ|	Verdict	Case
lwdid.lwdid (Lee & Wooldridge DiD, live run, captured): prop99 common-timing (d, post, vce=None); castle staggered (gvar, control_group=’never_treated’, aggregate=’overall’, vce=None/hc3)	smoking_data.csv (a13dd4d5d6e4…)	9	0	exact — matches to display precision	rolldid_lw

SBC

Reference	Dataset	#	max |Δ|	Verdict	Case
authors’ SBC_HK.R (lsq detrend + trend_predict + Synth::synth ipop), live run, captured	hong_kong_handover.csv (4f3fea9b93ba…)	6	1e+06	close	sbc_hongkong

SCD

Reference	Dataset	#	max |Δ|	Verdict	Case
base-R SCD (point estimator + corrected RC variance + in_C projection QP), reproduced on public CPS microdata	cps_lawa_arizona.parquet (87a228da0307…)	5	0	exact — matches to display precision	scd_cps

SCMO

Reference	Dataset	#	max |Δ|	Verdict	Case
Tian-Lee-Panchenko Germany.R (fn_W solve.QP, live run, captured)	repgermany.csv (61a624e307e6…)	6	0.011	tight	scmo_germany

SCUL

Reference	Dataset	#	max |Δ|	Verdict	Case
authors’ SCUL() (R, via Rscript + glmnet)	—	3	0.14	tight	scul_prop99

SDID

Reference	Dataset	#	max |Δ|	Verdict	Case
synth-inference/synthdid R (synthdid_estimate)	smoking_data.csv (a13dd4d5d6e4…)	1	0.0016	tight	sdid_prop99

SI

Reference	Dataset	#	max |Δ|	Verdict	Case
authors’ SI code (INFORMS opre.2025.1590.cd), vendored benchmarks/reference/synth_iv_OR25	—	20	0	exact — matches to display precision	si_prop99

SNN

Reference	Dataset	#	max |Δ|	Verdict	Case
deshen24/syntheticNN (live run, captured), SyntheticNearestNeighbors(n_neighbors=1)	smoking_data.csv (a13dd4d5d6e4…)	14	0	exact — matches to display precision	snn_prop99

SPILLSYNTH

Reference	Dataset	#	max |Δ|	Verdict	Case
Melnychuk-Andrii/Spillover-SCM inclusive SCM (scm_weights/runInclusiveSCM), transcribed to NumPy	—	4	7.6	tight	spillsynth_iscm_xval
jcao0/synthetic-control-spillover MATLAB spillover.csv (CA row)	—	13	5.7e-05	exact — matches to display precision	spillsynth_prop99
Mendez tutorial Rcpp sc_spillover (cmg777)	california_panel.csv (9d6a73e21f1a…)	4	3.4	documented — see notes	spillsynth_prop99_sar
Sakaguchi-Tagawa RcppArmadillo sc_spillover (method=sar, live run on the nonproprietary panel, captured)	sudan_panel.csv (722471a42b6c…)	5	0.41	close	spillsynth_sudan

SPSC

Reference	Dataset	#	max |Δ|	Verdict	Case
qkrcks0218/SPSC R (single-proxy synthetic control)	—	4	0	exact — matches to display precision	spsc_panic
qkrcks0218/SPSC R (single-proxy synthetic control)	—	31	0.0007	exact — matches to display precision	spsc_prop99

SSC

Reference	Dataset	#	max |Δ|	Verdict	Case
jcao0/staggered_synthetic_control (committed results_ssc.csv / Table1_eigenvalue.csv)	—	364	0.001	tight	ssc_guanajuato

SpSyDiD

Reference	Dataset	#	max |Δ|	Verdict	Case
authors’ functions_ssdid fit_unit_weights / fit_time_weights under the canonical SDID convention (1/T_post post weights, 1/N_sp affected weights)	—	2	0	exact — matches to display precision	spsydid_lawa_diff
authors’ SDID weight functions (serenini/spatial_SDID functions_ssdid) + the notebook’s spatial WLS, via benchmarks.reference.spsydid_ref	—	20	0.094	close	spsydid_state_mc

TASC

Reference	Dataset	#	max |Δ|	Verdict	Case
srho1/tasc TimeAwareSC (live run, captured; em_pre, naive init, set_seed(1))	smoking_data.csv (a13dd4d5d6e4…)	15	25	documented — see notes	tasc_prop99

VanillaSC

Reference	Dataset	#	max |Δ|	Verdict	Case
R package augsynth (live run, Kansas study)	kansas_ascm.csv (b026c651760c…)	8	0.0091	tight	ascm_kansas
R package scinference (sc.cf t-test, live run, captured)	carbontax_data.dta (815787c1e448…)	3	0	exact — matches to display precision	cwz_ttest
Malo et al. scm.corner (SCM-Debug, live run, captured)	basque_mscmt.csv (3aca35dc9b55…)	3	0.00048	tight	malo_basque
Malo et al. scm.corner (SCM-Debug, live run, captured)	augmented_cali_long.csv (974ae6ad6ab7…)	6	0.0048	tight	malo_prop99
R package MSCMT (live run, captured)	basque_mscmt.csv (3aca35dc9b55…)	4	4e-05	exact — matches to display precision	mscmt_basque
authors’ wsoll1 (R, via Rscript + LowRankQP)	—	3	0.00098	exact — matches to display precision	pensynth_prop99
scpi_pkg scdata(cointegrated_data=True)+scpi CI_all_gaussian	scpi_germany.csv (10b150fbcc2c…)	13	0.11	close	scpi_germany_pi
Python package scpi_pkg	—	15	4.3e-05	exact — matches to display precision	scpi_staggered
R package Synth::synth	california_panel.csv (9d6a73e21f1a…)	8	0.02	tight	synth_prop99
Andersson (2019) AEJ:EP 11(4), Section III reported values	carbontax_data.dta (815787c1e448…)	4	0.028	close	vanillasc_carbontax

Pending capture

These cross-validation cases are wired up but their reference had not been captured when this page was last generated; the daily action records them once its toolchain provisions.

Case	Reference
brazil_vaccine_scm_vs_proximal	—
lto_refined_placebo	independent reproduction of tsudijon/LeaveTwoOutSCI LTO pair loop (outcome-only SC via LowRankQP), all three empirical applications