Calibrating a cumulative band

Calibrating a cumulative band#

A pointwise conformal band answers “how big was the effect in week 7”. A great many applied questions are instead about a total: how many extra units did the campaign sell over the quarter, how much did the policy add over its first year. The two are not related by arithmetic on the endpoints. Summing the per-period half-widths assumes every period’s error moves in lockstep; dividing one by the horizon assumes the errors are independent. mlsynth therefore builds a separate band for the total, by resampling an out-of-sample error series into synthetic paths of the right length and reading the half-width off the accumulated totals (mlsynth.utils.conformal.block_error_paths(), described on Panel Data Approach (PDA) and Vanilla Synthetic Control (VanillaSC)).

This page records what happens when that band is measured against known truth, what limits it, and which of the fixes offered by the time-series conformal literature apply. The runnable study is under benchmarks/studies/cumulative_calibration/; the durable, self-contained part of it is the benchmark case conformal_window_count.

Why coverage is bounded by windows, not periods#

A total over \(H\) periods is \(H\) times the mean error over those periods:

\[\sum_{t=T_0+1}^{T_0+H} e_t \;=\; H \cdot \bar e , \qquad \bar e \;=\; \frac{1}{H}\sum_{t=T_0+1}^{T_0+H} e_t .\]

The term that sizes a total is therefore a level, and a level is observed once per calibration window, not once per period. Split the calibration series into \(m\) windows of length \(H\) and the calibration set for the dominant term has size \(m\). The other \(m(H-1)\) degrees of freedom describe the fluctuation about each level, which contributes to the total only through cancellation.

That is why the draw separates the two components: a path takes one window’s level, sign-flipped, plus blocks of the residuals about it. Pooling them would dilute the level, since a resampled block usually straddles a window boundary and averages two windows’ levels toward zero.

The consequence is a hard ceiling. The study’s window_curve grants every assumption a conformal band is usually asked to worry about — the level and the fluctuation are Gaussian, independent, stationary and exchangeable across windows, and the variance-components estimator inside the draw is handed data generated by exactly the model it assumes — and varies only \(m\):

\(m\)

coverage

\(m\)

coverage

2

0.809

10

0.920

3

0.852

13

0.926

4

0.875

19

0.930

5

0.887

26

0.947

6

0.899

40

0.947

8

0.918

80

0.950

Nominal is 0.95 and the Monte Carlo standard error is about 0.004. Coverage climbs monotonically in \(m\) and does not reach nominal until \(m\) is in the dozens. Since nothing is violated in this simulation, no repair aimed at exchangeability, autocorrelation or non-stationarity can move these numbers. The binding constraint is the window count.

The same arithmetic appears in the finite-sample rank. Split conformal takes the \(\lceil (n+1)(1-\alpha)\rceil\)-th order statistic of \(n\) scores, which exists only when that index is at most \(n\), so 19 scores are needed at \(\alpha = 0.05\). Nineteen windows of a 13-week horizon is 247 pre-periods.

What this means for each estimator#

The number of windows an estimator can supply is a property of its design, not of its resampler.

Estimator

Calibration series

Windows at \(H = 13\)

MAREX

the blank window, withheld from the fit

\(T_b / H\); three at the study’s \(T_b = 47\)

VanillaSC, PDA

rolling origins across the pre-period, refitting at each

five in the study’s configuration

An estimator whose calibration window supports three sub-windows cannot deliver a nominal cumulative band however clean its errors are. The remedies are more pre-periods, more windows out of the same pre-periods, or observations of the same quantity from somewhere other than time.

Is the failure the one the literature describes?#

The standard charge against split conformal on time series is that exchangeability fails, and that the resulting failure is not marginal but conditional: coverage averages out to its nominal level while the misses arrive in bursts, so a marginal average hides runs of consecutive failures. The study’s burstiness arm tests that directly, recording MAREX’s per-period violation indicator over 100 draws of a panel built from real factor paths, loadings and residuals:

pointwise marginal coverage  0.9392
P(miss)                      0.0600
P(miss | prev miss)          0.1667   (12 of 72)
P(miss | prev ok)            0.0559
lag-1 corr of misses         +0.1087
runs  observed 167  expected 173.5  z = -1.26
longest miss-run per draw: max 2 of 13; 11 percent of draws have a run >= 2

The signature is present in direction — a violation roughly triples the odds of the next period violating — and modest in magnitude. The runs test does not reject, the worst observed burst is two consecutive periods out of thirteen, and the conditional gap is about two and a half standard errors. This is consistent with the panel’s measured serial persistence, whose lag-1 autocorrelation runs 0.06 to 0.07.

So the cumulative shortfall and the clustering are separate phenomena. The first is a marginal problem in \(m\) and survives full exchangeability; the second is a conditional problem in time and is mild here. A cumulative band is in any case the wrong patient for clustering: a total over \(H\) periods has no within-window pattern left to cluster, and pricing serial dependence into the total is what the block length in the draw already does.

Two routes to more calibration observations#

Ensemble calibration#

EnbPI (Xu and Xie 2021) manufactures out-of-sample residuals without extra data: fit an ensemble of models, each on a bootstrap sample of the training periods, and score period \(t\) using only the members whose sample excluded it. Ported to MAREX in loo_ensemble, that means holding the design’s treated set \(S\) and treated weights \(w\) fixed — the mixed-integer program, solved once — and refitting only the control weights per member, which is a least-squares fit on the probability simplex. design_cost measures the difference: 12.4 seconds for the design solve against 33 milliseconds per simplex fit, so a 200-member ensemble is affordable and the whole placebo sweep below costs under five percent of one design solve.

Every pre-period then carries an out-of-sample residual, and the window count rises from \(T_b/H = 3\) to \(T_0/H = 8\). Over 100 draws:

Band

\(m\)

coverage

s.e.

median half-width

ratio vs shipped

shipped

3

0.920

0.027

0.301

1.000

ensemble, blank window

3

0.960

0.020

0.282

0.940

ensemble, whole pre-period

8

0.910

0.029

0.261

0.814

as above, leak-corrected

8

0.940

0.024

0.279

0.877

Paired on the same draws, the ensemble on the blank window beats what ships four discordant pairs to nil (\(z = +2.00\)) while being six percent narrower, and adding the five extra windows loses to it nil to five (\(z = -2.24\)).

The extra windows lose because of a leak the design cannot avoid. \(S\) was chosen having seen the fitting periods, so a residual there is out of sample for the control weights but not for the choice of \(S\); the blank periods carry no such leak, having been withheld from the design entirely. Measured, the residual spread on fitted periods is 0.0259 against 0.0316 on withheld ones, a ratio of 0.818. Those five windows are systematically 18 percent easier than the three honest ones, and pooling them narrows the band on a false premise. Rescaling the fitted stretch onto the withheld stretch’s spread recovers most of the gap but still does not beat the three honest windows. Bootstrapping \(S\) itself would remove the leak and costs a mixed-integer solve per member.

The finding is that the ensemble helps and the extra windows do not. The gain comes from aggregating 200 members into a more stable prediction, which also lowers the point error slightly, and not from the window count that motivated the construction.

Cross-sectional calibration#

The other direction is to stop looking for observations in time. In an in-space placebo each donor takes a turn as the treated unit, fitted from the remaining donors, and its \(H\)-period cumulative discrepancy is an observation of exactly the quantity the band must cover — on the same \(H\) periods, needing no extra pre-period. With \(J-1\) donors that is \(J-1\) observations, and the exchangeability assumption moves from time windows to units.

The rank arithmetic follows the design across. At \(\alpha = 0.05\) a finite band needs 19 usable donors, so a design that treats \(|S|\) units needs \(J \ge 19 + |S|\). In the study’s configuration, \(J = 20\) with \(|S| = 3\) leaves 17 eligible donors and the band is infinite. Nineteen usable donors is an easier requirement to meet in a geo design than 19 disjoint horizon-length windows, and it is not free.

One caveat is measured, not assumed. A placebo contrast that was never optimised is not exchangeable with a design-optimised one: an unstandardised VanillaSC placebo came out fourteen times too wide, and standardising each placebo by its own pre-period fit — Abadie’s remedy — brought coverage to 0.973.

Can an existing conformal library do this?#

A synthetic control is a constrained regressor, so it wraps as a scikit-learn estimator and drops into a conformal library unchanged. external_libraries does exactly that with MAPIE: the simplex wrapper is accepted, TimeSeriesRegressor(method="enbpi", cv=BlockBootstrap(...)) fits 200 members, and its conformity-score spread of 0.0311 over 104 training periods sits close to the hand-rolled leave-one-out residuals (0.023 on fitted periods, 0.035 on withheld ones), which is an independent check on the implementation above.

What comes back is one interval per period:

predict returns pred (13,), intervals (13, 2, 1)
per-period half-widths: min 0.0701  max 0.0701
stacking them gives 0.9119

against cumulative half-widths of 0.547 for the shipped band and 0.325 for the ensemble on the same draw. Stacking the endpoints is the lockstep assumption this whole construction exists to replace, and it costs a factor of 1.7 to 2.8 here. There is no call in the library that returns an interval for a functional of the \(H\) predictions.

puncc (deel-ai) is the same picture. Its EnbPI and AdaptiveEnbPI are the ensemble machinery, but every score in api/nonconformity_scores and every set in api/prediction_sets is per-observation, and its only multiplicity tool is a Bonferroni correction, which gives simultaneous per-period bands — wider than stacking, still not an interval for the total.

Both libraries own the ensemble plumbing and neither owns the estimand. The online-adaptation family (ACI, AgACI, conformal PID) is unavailable for a different reason: refreshing residuals sequentially requires the realised outcome at each step, and in a causal design the counterfactual is never revealed, because it is the estimand.

Reproducing the study#

The self-contained part runs anywhere:

python benchmarks/run_benchmarks.py --case conformal_window_count

The real-data arms need a wide panel of positive outcomes. The one measured here was 211 media markets by 128 weeks of retail sales and is proprietary, so the study reads whatever panel MLSYNTH_CAL_PANEL points at and rebuilds the same structure from it — real factor paths, real loadings, real unit means and real idiosyncratic residuals, with a circular shift common to all units, so that cross-sectional dependence among residuals survives:

cd benchmarks/studies/cumulative_calibration
export MLSYNTH_CAL_PANEL=/path/to/panel.csv

python window_curve.py                             # the m table above
python design_cost.py                              # MIQP against simplex QP
python burstiness.py 100 0 out.jsonl               # clustered violations
python loo_ensemble.py 100 0 out.jsonl             # the four bands
python loo_ensemble.py --summarise out.jsonl
python external_libraries.py                       # MAPIE, if installed

The runs behind the tables on this page are committed under benchmarks/studies/cumulative_calibration/results/, so the summaries can be regenerated without the panel.

References#

Xu, C. and Y. Xie (2021). Conformal prediction interval for dynamic time-series. ICML.

Abadie, A. (2021). Using synthetic controls: feasibility, data requirements, and methodological aspects. Journal of Economic Literature 59(2), 391-425.