Comparing counterfactuals across methods

Comparing counterfactuals across methods#

When to use#

Every synthetic-control estimator in mlsynth produces the same observable: a counterfactual path for the treated unit – what the outcome would have looked like without the intervention – and, when the estimator carries inference, a per-period prediction interval around that path. Once you have fit more than one estimator on the same panel, the natural next question is how their counterfactuals compare, both to each other and to what was actually observed.

compare_counterfactuals() answers that question in one call. It reads each method’s counterfactual, its prediction interval (when present), and the stored att and pre-treatment fit (pre_rmse) off the standardized result object, lines them all up on a common time axis, and hands back a small container you can either inspect as a table or draw as a single overlay. It saves you the per-method plotting loop and the second loop that digs the interval arrays out of inference.details by hand.

The helper is deliberately undemanding about its inputs. A method can be a fitted BaseEstimatorResults (the common case), a plain array of counterfactual values, or a small dictionary spec. The last two let you compare results that do not expose the standardized surface – for instance the Spillover-Aware Synthetic Control (SPILLSYNTH) dispatcher, whose spillover-adjusted counterfactual is assembled per method – alongside ones that do.

What it returns#

compare_counterfactuals() returns a CounterfactualComparison with three faces:

  • curves – a tidy long table, one row per method and period, with columns counterfactual, lower and upper (the last two empty where a method has no interval at that period);

  • summary – one row per method, holding the stored att and pre_rmse, read off the result rather than recomputed, plus a window_rmse column when a fit_window is given (the fit loss over just those periods);

  • observed – the observed treated series, when one was supplied or could be taken from a standardized result;

  • weights – a tidy method / donor / weight frame, the donor weights read off each method’s weights.donor_weights (empty for methods that expose none);

a plot method that overlays the observed series and every method’s counterfactual, drawing prediction intervals as per-period error bars where they exist; and a plot_weights method that draws the donor weights as a grouped bar chart, one bar per method per donor – the same side-by-side comparison, on the weights rather than the paths.

Example: comparing different estimators on one panel#

The clearest use is to put several different estimators side by side. Here three of them – the classical synthetic control, a cluster-denoised variant, and the synthetic difference-in-differences estimator – are fit on Abadie, Diamond and Hainmueller’s California tobacco panel, then compared. Only the first carries a prediction interval, and the helper simply omits intervals for the methods that do not:

import pandas as pd
from mlsynth import VanillaSC, CLUSTERSC, SDID
from mlsynth.utils.counterfactual_compare import compare_counterfactuals

url = ("https://raw.githubusercontent.com/jgreathouse9/mlsynth/"
       "main/basedata/augmented_cali_long.csv")
df = pd.read_csv(url)
df = df[df["year"] <= 2000].copy()
df["treated"] = ((df["state"] == "California")
                 & (df["year"] >= 1989)).astype(int)
base = dict(df=df, outcome="cigsale", treat="treated", unitid="state",
            time="year", display_graphs=False)

sc   = VanillaSC({**base, "inference": "scpi", "alpha": 0.10}).fit()
clus = CLUSTERSC({**base}).fit()
sdid = SDID({**base}).fit()

cmp = compare_counterfactuals(
    {"VanillaSC": sc, "CLUSTERSC": clus, "SDID": sdid})

print(cmp.summary.round(2))
#                att  pre_rmse
# method
# VanillaSC   -19.51      1.66
# CLUSTERSC   -21.39      1.50
# SDID        -15.61     24.81

cmp.plot()

The summary table reads the stored effect and fit straight off each result, so the numbers match what each estimator reports on its own; the plot overlays the three counterfactuals against observed cigarette sales, with the prediction interval shown only for the method that produced one.

To compare fit over a specific window rather than the whole pre-period, pass fit_window=(low, high); the summary then gains a window_rmse column holding each method’s RMSE of observed minus counterfactual over those periods, alongside the stored all-pre pre_rmse. Observed is read off the results, so nothing extra need be supplied:

cmp = compare_counterfactuals(
    {"VanillaSC": sc, "CLUSTERSC": clus, "SDID": sdid},
    fit_window=(1980, 1988))      # fit over the last nine pre-treatment years
print(cmp.summary[["att", "pre_rmse", "window_rmse"]].round(2))
#                att  pre_rmse  window_rmse
# method
# VanillaSC   -19.51      1.66         1.37
# CLUSTERSC   -21.39      1.50         1.74
# SDID        -15.61     24.81        25.30

Example: two solvers of the same estimator, with prediction intervals#

The comparison need not be across estimators. A common use is to hold the estimator fixed and vary one setting – here the predictor-weight solver behind Vanilla Synthetic Control (VanillaSC) – so the disagreement is attributable to that setting alone. Both solvers carry the Cattaneo-Feng-Titiunik prediction interval, which the helper draws as per-period error bars, dodged so the two methods’ bars stay legible:

import pandas as pd
from mlsynth import VanillaSC
from mlsynth.utils.counterfactual_compare import compare_counterfactuals

url = ("https://raw.githubusercontent.com/jgreathouse9/mlsynth/"
       "main/basedata/augmented_cali_long.csv")
df = pd.read_csv(url)
df = df[df["year"] <= 2000].copy()
df["treated"] = ((df["state"] == "California")
                 & (df["year"] >= 1989)).astype(int)
# Three lagged outcomes as predictors, so the covariate-matching solvers run.
for L in (1975, 1980, 1988):
    df[f"cig{L}"] = df["state"].map(
        df[df["year"] == L].set_index("state")["cigsale"])
base = dict(df=df, outcome="cigsale", treat="treated", unitid="state",
            time="year", covariates=["cig1975", "cig1980", "cig1988"],
            inference="scpi", alpha=0.10, display_graphs=False)

mscmt = VanillaSC({**base, "backend": "mscmt", "seed": 42}).fit()
malo  = VanillaSC({**base, "backend": "malo", "seed": 0}).fit()

cmp = compare_counterfactuals({"MSCMT nested": mscmt, "Malo bilevel": malo})
cmp.plot(dodge=0.4,
         colors={"MSCMT nested": "C0", "Malo bilevel": "C3"},
         styles={"MSCMT nested": "--", "Malo bilevel": "-."})

The colors and styles mappings name the per-method line appearance; the dodge argument offsets each method’s error bars horizontally so overlapping intervals do not collide.

Two solvers can fit the pre-period equally well yet rest on different donor weights, and plot_weights shows that directly from the same object – no second loop to pull weights.donor_weights out of each result:

cmp.plot_weights(threshold=1e-3,          # drop donors negligible everywhere
                 colors={"MSCMT nested": "C0", "Malo bilevel": "C3"})

Each donor becomes a group of bars, one per method. threshold drops donors that are negligible in every method, max_donors keeps only the largest, and label maps each donor key to its axis label (e.g. lambda d: d.split(" (")[0] to strip a parenthetical tag).

Example: results outside the standardized contract#

Some results – the Spillover-Aware Synthetic Control (SPILLSYNTH) dispatcher among them – do not expose the standardized time_series surface, because their counterfactual is built per method. For those, pass the counterfactual yourself: either a plain array, or a dictionary spec that may also carry a prediction interval (with periods when the interval covers only part of the series), the effect, and a time axis. Mixed inputs are allowed, so a hand-built counterfactual can sit next to a standardized result in the same comparison:

import numpy as np
from mlsynth.utils.counterfactual_compare import compare_counterfactuals

years = np.arange(2000, 2010)
cmp = compare_counterfactuals(
    {
        # a dict spec: counterfactual, a post-period interval, and the att
        "method A": {"counterfactual": np.linspace(10, 5, 10),
                     "att": -2.1, "time": years,
                     "periods": years[5:],
                     "lower": np.linspace(4, 3, 5),
                     "upper": np.linspace(6, 5, 5)},
        # a bare array: just the counterfactual, no interval
        "method B": np.linspace(10, 6, 10),
    },
    observed=np.linspace(10, 4, 10), time=years)

print(cmp.summary)        # att for A, NaN for B; pre_rmse NaN for both
cmp.plot()

This is the same path the spillover illustration in the paper uses: each method’s full counterfactual is assembled, then handed to compare_counterfactuals with its effect, so one overlay and one effect table cover all of the methods at once.

Accepted inputs#

Each value in the methods mapping is one of:

  • a standardized result – the counterfactual, time axis, observed series, att, pre_rmse, and any inference.details interval are read off it;

  • a mapping with at least counterfactual, and optionally time, lower, upper, periods (to align a partial interval), att, pre_rmse and observed;

  • a plain array, treated as the counterfactual with no interval.

A prediction interval is always two-sided: supplying only lower or only upper is an error, as is an interval whose length matches neither the curve nor a supplied periods list. These are surfaced as MlsynthConfigError or MlsynthEstimationError, never silently dropped.

Core API#

Cross-method counterfactual comparison on a common time axis.

Different synthetic-control estimators all emit the same observable – a counterfactual path for the treated unit, and (when the estimator carries inference) a per-period prediction interval around it. When several are fit on one panel, the analyst wants to read them against each other and against the observed series. Doing that by hand means one ax.plot per method plus a second loop to pull the per-period bounds out of inference.details; this module collapses both into one call.

compare_counterfactuals() normalizes each method to (time, counterfactual, lower, upper, att, pre_rmse) – reading those straight off a standardized BaseEstimatorResults (so the stored effects.att and pre_rmse are used, not recomputed) or off an explicit spec for results that do not expose the standard surface (e.g. the SPILLSYNTH dispatcher, whose counterfactual is assembled per method). It returns a CounterfactualComparison with a tidy curves frame (data form), a summary frame (one row per method), and .plot() (plot form), mirroring the data/plot split of mlsynth.utils.design_compare.

class mlsynth.utils.counterfactual_compare.CounterfactualComparison(curves: DataFrame, summary: DataFrame, observed: Series | None = None, weights: DataFrame = None)#

Bases: object

Result of compare_counterfactuals().

Parameters:

  • curves (pd.DataFrame) – Long form, one row per (method, period): method, period, counterfactual, lower, upper (the last two NaN where the method has no prediction interval at that period).

  • summary (pd.DataFrame) – One row per method (indexed by label), columns att and pre_rmse read from the stored result fields (NaN when a method does not carry them).

  • observed (pd.Series, optional) – Observed treated series indexed by period, when one was supplied or could be read off a standardized result.

  • weights (pd.DataFrame) – Long form, one row per (method, donor): method, donor, weight. Empty when no method exposes donor weights.

plot(ax: Any = None, **kwargs: Any) → Any#

Overlay the counterfactuals (the plot form). See plot_counterfactual_comparison().

plot_weights(ax: Any = None, **kwargs: Any) → Any#

Grouped bar chart of each method’s donor weights (the plot form). See plot_weights_comparison().

mlsynth.utils.counterfactual_compare.compare_counterfactuals(methods: Mapping[str, Any], *, observed: Any = None, time: Any = None, fit_window: Tuple[Any, Any] | None = None) → CounterfactualComparison#

Line up several methods’ counterfactuals on a common time axis.

Parameters:

  • methods (mapping {label: spec}) – Ordered mapping (label preserved as the method name). Each spec is one of: a standardized BaseEstimatorResults (the counterfactual, time, effects.att, pre_rmse and any inference.details band are read off it); a mapping with at least counterfactual and optionally time/lower/upper/ periods/att/pre_rmse/observed; or a plain array treated as the counterfactual.

  • observed (array-like or pd.Series, optional) – Observed treated series. A bare array is paired with time (or the first method’s time). When omitted, it is taken from the first standardized result that carries time_series.observed_outcome.

  • time (array-like, optional) – Default x-axis for specs that do not carry their own time index.

  • fit_window ((low, high), optional) – When given, add a window_rmse column to summary: the RMSE of observed minus counterfactual over the periods low <= t <= high (NaN for a method with no period in the window). Needs an observed series – supplied or read off a standardized result – else a MlsynthConfigError is raised. This is the fit loss over the matching window, a tighter figure than the stored all-pre pre_rmse.

Returns:

CounterfactualComparison – .curves and .summary (data form) and .plot() (plot form).

mlsynth.utils.counterfactual_compare.plot_counterfactual_comparison(comparison: CounterfactualComparison, ax: Any = None, *, colors: Mapping[str, Any] | None = None, styles: Mapping[str, Any] | None = None, dodge: float = 0.0, band: str = 'errorbar', capsize: float = 1.5, elinewidth: float = 0.7, fill_alpha: float = 0.18, observed_color: str = 'black', legend: bool = True) → Any#

Overlay each method’s counterfactual, with its prediction interval.

Draws the observed series (when present) plus one line per method; where a method carries a prediction interval, its bounds are shown either as per-period error bars (band="errorbar", optionally dodged by dodge so overlapping bars stay legible) or as a shaded region (band="fill"). Renders in the in-house mlsynth style and returns the axis.

Parameters:

  • comparison (CounterfactualComparison) – The object returned by compare_counterfactuals().

  • colors, styles (mapping {label: value}, optional) – Per-method line colour and linestyle overrides; defaults cycle a palette and a solid line.

  • dodge (float, default 0.0) – Horizontal offset applied to each method’s error bars, centred on zero (band="errorbar" only).

  • band ({“errorbar”, “fill”}, default “errorbar”) – How to render each prediction interval: per-period error bars, or a shaded band between the bounds.

  • capsize, elinewidth (float) – Error-bar cosmetics (band="errorbar" only).

  • fill_alpha (float, default 0.18) – Opacity of the shaded band (band="fill" only).

  • observed_color (str, default “black”) – Colour of the observed series.

  • legend (bool, default True) – Whether to draw the legend.

mlsynth.utils.counterfactual_compare.plot_weights_comparison(comparison: ~mlsynth.utils.counterfactual_compare.CounterfactualComparison, ax: ~typing.Any = None, *, colors: ~typing.Mapping[str, ~typing.Any] | None = None, threshold: float = 0.001, max_donors: int | None = None, sort: bool = True, label: ~typing.Any = <class 'str'>, bar_width: float | None = None, ylabel: str = 'donor weight', legend: bool = True) → Any#

Grouped bar chart of each method’s donor weights.

One group of bars per donor, one bar per method, so the methods’ weight vectors can be read side by side (the same comparison the counterfactual overlay makes, on the weights rather than the paths). Donors with negligible weight in every method are dropped.

Parameters:

  • comparison (CounterfactualComparison) – The object returned by compare_counterfactuals(); its weights frame supplies the donor weights.

  • colors (mapping {label: value}, optional) – Per-method bar colour; defaults cycle the palette.

  • threshold (float, default 1e-3) – Drop a donor whose |weight| is below this in every method.

  • max_donors (int, optional) – Keep only the max_donors donors with the largest total weight.

  • sort (bool, default True) – Order donors by descending total weight (else by first appearance).

  • label (callable, default str) – Maps a donor key to its axis label (e.g. to strip a parenthetical).

  • bar_width (float, optional) – Total width shared by a donor’s bars; defaults to 0.8.

  • ylabel (str, default “donor weight”) – Y-axis label.

  • legend (bool, default True) – Whether to draw the legend.

Raises:

MlsynthConfigError – When no method exposes donor weights above threshold.