Geo Experiment Design (GEOX)

When to Use This Estimator

GEOX decides which geographic markets to treat in a marketing geo experiment, before the experiment runs. You have daily or weekly sales for a few dozen cities, you can turn a campaign on in some of them, and you must pick which ones. GEOX answers that by rehearsing the experiment on history you already have: it slides a pretend treatment window backwards through the past, injects a lift of known size, and counts how often the estimator would have caught it. The region with the smallest lift it can reliably detect is the one to test in.

Use it when:

• you can assign treatment at the geo level, and geos are the unit of analysis;

• you have pre-period history for every candidate market, with no gaps;

• the number of markets is in the tens, as with DMAs or metros;

• you want the design chosen by the same estimator that will analyse the result, so the power calculation is not about a different model than the one you will report.

Use something else when the design question is different in kind. Parallel-Trends Supergeo Design (PANGEO) pairs geos into balanced supergeos and randomises within pairs, which suits a rollout where every market must be used. GEOX picks a small test region and leaves the rest as donors. Synthetic Principal Component Design (SPCD) scores a design from holdout residuals when the estimator is a fixed linear contrast.

What Distinguishes It

The market-selection loop follows GeoLift, Meta’s geo-experiment package: anchor a candidate region at each market, add that market’s most correlated neighbours, sweep effect sizes over several backward backtests, and rank by a composite score. That loop is the estimator. The thing that scores a candidate inside it is a choice.

engine makes the choice. Everything else in the design – nomination, the backtest windows, effect injection, power, the minimum detectable effect, the composite rank, the constraint layer, the plots – is shared code and does not know which engine ran.

engine="augsynth" is the ridge-augmented synthetic control of Ben-Michael, Feller and Rothstein (2021), which is what GeoLift itself scores with. Pick it to reproduce GeoLift, or when the analysis that will be reported is a synthetic control fit.

engine="sdid", the default, is synthetic difference-in-differences (Arkhangelsky, Athey, Hirshberg, Imbens and Wager, 2021), which weights donor markets and pre-periods at the same time. Two consequences follow.

The first is that no single pre-period has to be matched. Synthetic control asks the donors to reproduce the treated path everywhere before treatment; synthetic DiD asks only that a weighted average of pre-periods line up, and lets the weights decide which periods carry the comparison. On a 105-day panel of 40 US metros, the fitted design puts weight on 7 of 91 pre-days and none on the rest. Days that resemble the post-period window count; days that do not are set aside.

The second is that a level difference between the treated region and its donors is absorbed. Synthetic DiD differences it out, so the donor pool does not have to contain the treated region’s scale, only its shape. A two-city test region in a pool of small markets remains estimable.

The two engines report different imbalance measures, because each reports its own estimator’s. pre_rmspe on the SDID path is the root-mean-square pre-period gap; scaled_l2 on the augsynth path is augsynth’s ratio of the fitted imbalance to the imbalance uniform donor weights would leave. Compare either across candidates within one design; neither is comparable across engines.

The name. This page was sdidgeo while synthetic DiD was the only thing that could score a design, and the estimator, its config, its result class and its plotter were SDIDGEO, SDIDGEOConfig, SDIDGEOResults and plot_sdidgeo_design. Those spellings are gone, not deprecated. Code written against them raises ImportError at the import line; substitute GEOX for SDIDGEO throughout, which is the whole of the change.

Notation

Let \(i = 1, \dots, N\) index markets and \(t = 1, \dots, T\) index periods, with outcome \(Y_{it}\). A candidate test region \(\mathcal{T}\) is a set of \(N_{\mathrm{tr}}\) markets; the donors are the rest, \(\mathcal{C}\). Write \(y_t = N_{\mathrm{tr}}^{-1}\sum_{i \in \mathcal{T}} Y_{it}\) for the treated average and \(\mathbf{Y}_{0}\) for the \(T \times |\mathcal{C}|\) donor matrix.

A pseudo-experiment is a pre/post split of the observed history. For a duration \(D\) and a backtest \(s = 1, \dots, S\), the pretend treatment runs over periods

\[[\,T_0,\; T_0 + D - 1\,], \qquad T_0 = T - D - s + 1 ,\]

so \(s = 1\) ends flush with the last observed period and each increment slides the window one period earlier. Everything before \(T_0\) is the pre-period.

What the engine supplies is a counterfactual path for \(y_t\) fitted on the pre-period alone, and \(\hat\tau\) is the average gap against it over the window. The default engine is synthetic DiD, which chooses unit weights \(\omega\) over donors and time weights \(\lambda\) over pre-periods, and estimates

\[\hat\tau \;=\; \Big( \bar y_{\mathrm{post}} - \lambda' y_{\mathrm{pre}} \Big) \;-\; \Big( \omega' \bar{\mathbf{Y}}_{0,\mathrm{post}} - \lambda' \mathbf{Y}_{0,\mathrm{pre}}\, \omega \Big).\]

Both weight vectors are non-negative and sum to one. Rearranged, the counterfactual path is \(\mathbf{Y}_{0}\omega + \big(\lambda' y_{\mathrm{pre}} - \lambda' \mathbf{Y}_{0,\mathrm{pre}}\omega\big)\).

The augsynth engine has no \(\lambda\). It weights donors alone, and every pre-period counts equally: simplex synthetic-control weights first, then a ridge augmentation that corrects the pre-period imbalance those weights leave, which is what takes \(\omega\) off the simplex. With fixed_effects (its default, augsynth’s fixedeff) every unit is demeaned by its own pre-period mean before fitting and the level comes back as an intercept, so its counterfactual path is \(\alpha + \mathbf{Y}_{0}\omega\). Everything downstream of the path is the same for both engines.

An effect of size \(e\) is injected multiplicatively, \(y_t \mapsto (1 + e)\, y_t\) on the window, matching GeoLift’s convention that an effect size reads as a percentage lift on the treated markets’ own volume.

Assumptions

1. Balanced panel, no gaps.

   Every market is observed in every period. Ingestion goes through geoex_dataprep, which raises when the panel is ragged.

   Remark. A market that starts reporting halfway through cannot serve as a donor for a backtest that begins before it exists, and silently dropping it would change the donor pool between backtests.

2. The history is untreated.

   The simulation reuses observed periods as pretend post-periods, so those periods must carry no real treatment effect. If a campaign ran in month three, backtests overlapping it inherit its effect and report power that the design will not reproduce.

   Remark. Where a genuine post-period exists, name it with post_col and it is held out of the simulation.

3. No interference between markets.

   Treating one market does not move another’s outcome. Neighbouring metros that share media markets or commuters violate this, and the donor pool then contains partially treated units.

   Remark. Where the interference is known, declare it: cluster_col or adjacency makes the affected pairs conflict, which keeps them out of the same test region and out of each other’s donor pool. Where it is only suspected, not_to_be_treated bars a market from treatment while leaving it a donor. Neither detects interference the panel does not declare.

4. The pre-period relationship persists into the test.

   Weights fit on history are used for the future. A market that tracked the test region for a year and then diverges breaks the design regardless of how the design was chosen.

   Remark. scaled_l2 and pre_rmspe in the shortlist report how well the fit held historically, which is the available evidence for this assumption and not proof of it.

5. The reference distribution represents the null.

   Detection compares the estimate to a distribution built from the panel itself, and which one depends on inference. Placebo reassignment presumes the donors are exchangeable enough that a donor-based null describes what a no-effect test region would look like. Conformal presumes the pre-period residuals are exchangeable with the post-period ones, so a permutation of the pre-period residuals describes what no effect would have produced.

   Remark. Each has its own failure mode. With few donors the placebo draws overlap heavily and the standard error is optimistic, so the design needs a donor pool comfortably larger than the test region. Conformal is the one that breaks under a trending or seasonally-structured residual, since permuting then compares periods that were never comparable; conformal_type="block" permutes in contiguous blocks, which preserves short-run dependence and not a trend.

Inference and Diagnostics

inference chooses the null a detection is taken against, and it varies separately from engine. Left unset, each engine takes its own default: placebo for sdid, conformal for augsynth, which is GeoLift’s choice.

Placebo reassignment is Arkhangelsky et al.’s Algorithm 4: reassign \(N_{\mathrm{tr}}\) donors as pretend-treated, drop them from the pool, refit, and take the standard deviation of the resulting estimates. Detection is the two-sided normal test \(2\big(1 - \Phi(|\hat\tau| / \hat\sigma)\big)\) against alpha. The procedure needs only a donor-built counterfactual, so it runs on either engine. It is the only one available on sdid, because the other two variance procedures those authors give – jackknife and bootstrap – are undefined for the single treated series a candidate region collapses to.

Conformal inference permutes the pre-period residuals and asks where the observed post-period residual falls among the permuted ones. Its argument needs those residuals to be exchangeable across the matching window, which is what synthetic DiD’s time weights exist to deny – \(\lambda\) says pre-periods are not interchangeable. So conformal is available on augsynth and refused on sdid, with the reason stated in the error. ns sets the number of permutations and conformal_type the scheme. "block" is the default and takes the panel’s cyclic shifts, which preserve serial dependence; "iid" permutes residuals freely and assumes an exchangeability a trending or seasonal panel denies. "iid" is also the scheme whose p-value can reach exactly zero, since a free permutation can be beaten by the observed statistic every time; finite_sample_p=True reports (1 + #{stat >= observed}) / (1 + ns) instead, which cannot. That correction is off by default so the GeoLift reproduction keeps augsynth’s convention; turn it on for inference you intend to report.

Holding one of the two fixed and varying the other separates the two sources of a difference between designs. Scoring one panel with both engines under placebo isolates the objective; scoring it with augsynth under both nulls isolates the inference.

Power at an effect size is the detection rate across backtests. The minimum detectable effect is the smallest magnitude whose power exceeds power_threshold, taking the smaller of the detectable positive and negative effects. Candidates are then ranked on a composite of three dense ranks: \(|\mathrm{MDE}|\), the power at the MDE, and how far the recovered lift sits from the injected one.

Under placebo inference the shortlist also carries mde_exact_up and mde_exact_down, the effect sizes at which the design starts detecting in each direction. These solve in closed form, so they do not inherit the effect grid’s step: a design that detects 0.1014 reports that instead of the 0.150 the grid rounds it to. They also read as a validity check. The rejection region is \(e < \mathrm{down}\) or \(e > \mathrm{up}\), and nothing forces \(\mathrm{down} < 0 < \mathrm{up}\); a design whose two crossings have the same sign already rejects at zero injected effect, which means it is firing on its own drift. Under conformal both columns are nan, because a conformal p-value re-permutes against the injected series and is not analytic in the effect size.

Two properties keep the sweep cheap, and both are consequences of where the fit gets its information. Every weight program reads the pre-period only – SDID’s unit weights read the treated pre-period, its time weights and ridge read donors and counts, augsynth’s ridge reads the pre-period matching matrices – so none of them sees the treated post block. Injecting an effect there cannot move the weights, which makes

\[\hat\tau(e) = \hat\tau(0) + e \cdot \bar y_{\mathrm{post}}\]

exact. The placebo draws reassign control markets, so \(\hat\sigma\) does not depend on \(e\) either. One fit and one placebo run therefore cover the whole grid of effect sizes.

Conformal gives up the second property and keeps the first. Its p-value tests the injected series against permutations of the pre-period residuals, so it moves with \(e\) and is recomputed at every grid point. That, and not the fit, is where the augsynth path’s cost sits.

Example

import numpy as np
import pandas as pd
from mlsynth import GEOX
from mlsynth.config_models import GEOXConfig

df = pd.read_csv("basedata/geolift_test_data.csv")
df["date"] = pd.to_datetime(df["date"])

design = GEOX(GEOXConfig(
    df=df, unitid="location", time="date", outcome="Y",
    treatment_size=2,
    durations=[14],
    effect_sizes=[round(x, 2) for x in np.arange(-0.30, 0.35, 0.05)],
    n_backtests=5,
    n_draws=100,
    seed=0,
    n_jobs=-1,
)).fit()

print(design.selected_units)
print(design.metadata["winner_mde_optimistic"])   # 0.15
print(design.metadata["winner_mde_planning"])     # 0.10
print(design.power.head())

On the 40-market, 105-day panel this selects atlanta and nashville with a minimum detectable effect of 0.15, out of 31 candidate regions:

                 candidate  duration   mde  power  scaled_l2  rank
       atlanta + nashville        14  0.15    1.0      0.324   1.0
jacksonville + minneapolis        14  0.20    1.0      0.529   2.0
       milwaukee + orlando        14  0.15    1.0      0.263   2.0
        cleveland + denver        14 -0.15    1.0      0.223   4.0
    detroit + new orleans        14 -0.15    1.0      0.530   5.0

Read that as: a 15% lift in Atlanta and Nashville together would be detected at the 10% level in essentially every backtest tried. Anything smaller would not be, so an experiment expecting a 5% lift needs a longer test, a bigger region, or a different metric.

The two MDEs in metadata are the same quantity measured on different backtests. winner_mde_optimistic is what the scan produced, and it is the smallest MDE in a field of candidates – the region most likely to be picked is the one whose estimate happened to come out low, so selection biases it downward. winner_mde_planning re-scores the winning region on backtests deeper in history that took no part in choosing it, which is why it is the one to plan against.

That correction is a tendency across panels, not an inequality on any one. Here it comes out lower, 0.10 against 0.15, because the held-out windows are different windows and carry their own noise. The bias it corrects, and how it shrinks as n_backtests grows, are measured directly in GEOX: does the minimum detectable effect mean what it says?. n_validation_backtests sets how many held-out backtests the re-scoring gets; zero turns it off and leaves winner_mde_planning as None.

design.design_weights carries both weight vectors, donor_weights over markets and time_weights over pre-period dates. The time weights are sparse: on this panel 7 of 91 pre-days carry any weight.

design.report is None. It is the slot for the realized effect and stays empty until the experiment has run and post-treatment outcomes exist.

Scanning several region sizes

treatment_size takes a list, so one run can score two-market regions against five-market ones (GeoLift’s N = c(2, 3, 4, 5)). Candidates are nominated once per size, pooled, and ranked together, so the shortlist answers how large a test region has to be and which markets it should contain at the same time. A treatment_size column carries each candidate’s size, and metadata["treatment_sizes"] the sizes scanned.

Each candidate is fit with its own treated count. Under engine="sdid" that count enters the ridge as \((N_{\mathrm{tr}} T_{\mathrm{post}})^{1/4}\), so a five-market region is regularised more strongly than a two-market one on the same panel.

Scanning sizes 2 through 5 on the test panel gives 123 candidates, and the best of each size:

size  candidate                                                          mde  scaled_l2
   5  columbus + jacksonville + milwaukee + minneapolis + new orleans   0.10      0.370
   4  columbus + jacksonville + milwaukee + minneapolis                 0.15      0.290
   2  atlanta + nashville                                               0.15      0.324
   3  atlanta + chicago + nashville                                     0.15      0.325

Bigger regions detect smaller lifts, which is the usual trade: five markets carry more volume than two, so the same proportional effect is easier to see. Set against that, a larger test region costs more to run and holds out more of the country from the control pool. The scan prices that choice instead of assuming it.

Design constraints

to_be_treated and not_to_be_treated name individual markets. The constraint fields express rules instead, and each one narrows where the search may look.

Interference. Two markets interfere when treating both contaminates the comparison, either because they share a media market or because one spills into the other. cluster_col names a per-market column (a DMA, a state) and makes markets sharing a value conflict. adjacency takes a square DataFrame of pairwise spillover strengths, and any off-diagonal entry above spillover_threshold is a conflict. Supplying both takes the union.

A conflict binds twice. No candidate region may hold two conflicting markets, so the treated set is an independent set of the conflict graph. And a treated market’s conflicting partners are dropped from its own donor pool, since a market contaminated by the treatment cannot serve as its own control. The second half is the exclusion restriction, and it applies to the backtests that score the candidate as well as to the deployed fit, so the reported MDE reflects the pool the experiment will actually have.

Coverage. stratum_col names a grouping the test region has to represent, with min_per_stratum requiring at least that many treated markets in every stratum holding an eligible market, and max_per_stratum capping any one stratum. Use this when the region has to span regions or store formats instead of concentrating wherever the correlations happen to be highest.

Size band. size_col with min_size and max_size bounds which markets may be treated, both ends inclusive. The floor is a power or operational minimum. The ceiling encodes synthesizability: a market far larger than every donor cannot sit inside their convex hull, and the scaled \(L^2\) imbalance grows accordingly. Markets outside the band stay available as donors, since the band restricts treatment eligibility alone.

config = GEOXConfig(
    df=df, unitid="location", time="date", outcome="Y",
    treatment_size=[2, 3],
    durations=[14], effect_sizes=[0.05, 0.10, 0.15, 0.20],
    cluster_col="dma",             # no two treated markets in one DMA
    stratum_col="region", min_per_stratum=1,   # every region represented
    size_col="volume", min_size=5_000,         # skip markets too small to power
)

When no combination of markets satisfies the constraints, the failure names which constraint bound the search, in have-versus-need form, so a design that cannot be run says why:

MlsynthConfigError: GEOX design is infeasible -- the binding constraint(s):
  - spillover/cluster: the largest conflict-free treated set is 2 <
    treatment_size=3. Relax the cluster/adjacency constraint, widen the
    candidate pool, or reduce treatment_size.

Every constraint is off by default, and with none configured the search runs exactly as it does above.

Reading out the experiment

fit() chooses a region before the experiment runs, so report is None on the returned design. Once outcomes exist, name the post-treatment periods with post_col and the same call fills it with the realized effect: the ATT over the post window, the observed, counterfactual and gap paths across the whole panel, both weight vectors, and the pre-period fit diagnostics.

The readout uses the configured engine and the configured null, the same two the design scored itself with. A minimum detectable effect computed one way and a readout computed another would leave the reported power describing a test nobody ran, and the promise that the design is chosen by the estimator that will analyse the result is the reason to use GEOX at all. For the same reason the readout inherits the design’s donor pool: where a spillover constraint barred a treated market’s conflict-neighbours, they stay barred here.

report.weights.summary_stats carries whatever the engine has to say about its own fit alongside the shared keys: SDID’s ridge and bias correction, augsynth’s intercept, its penalty and the augmentation by name. Under augment=None there is no penalty and lambda_ is None, which is the answer and not a missing value.

design = GEOX(GEOXConfig(
    df=df, unitid="location", time="date", outcome="Y",
    post_col="post",              # 1 on the periods the campaign ran
    treatment_size=2, durations=[14],
    effect_sizes=[0.05, 0.10, 0.15],
)).fit()

design.report.effects.att          # realized effect
design.report.inference.p_value    # tested against the same null

The design itself is unaffected by post_col: ingestion truncates to the pre-period before any candidate is scored, so a design fit on a pre-only panel and one fit on the full panel choose the same region. Only the readout sees the post periods.

how sets the scale the readout is written in. The fit always runs on the per-market mean, which keeps the target at donor scale, so how changes reported units and neither the region chosen nor the p-value: "mean" gives the per-market effect, "sum" the summed incremental across the treated markets, which is GeoLift’s convention and the one to use when the number is going next to a spend figure. Cost from cpic is computed off the summed incremental either way.

Plots

plot_geox_design draws the two views GeoLift shows for a market selection: the power curve, with the detection threshold and the minimum detectable effect marked and the other candidates drawn faintly behind, and the fit the design rests on.

from mlsynth import plot_mde_ranking, plot_geox_design

plot_geox_design(design, power_threshold=0.8)
plot_mde_ranking(design, top=12)

The power curve is U-shaped by construction: power falls to zero at no injected effect and rises to one as the lift grows in either direction. Where it crosses the threshold is the minimum detectable effect. plot_mde_ranking gives the market-selection view instead, ranking the shortlisted regions by the smallest lift each can detect.

Verification

The default pairing – GeoLift’s market-selection loop scored with synthetic DiD – exists in no published implementation, so GEOX has no replication path in the sense the other estimators do: nothing to reproduce, no simulation table to match, no reference to agree with. Its validation is assembled from three pieces instead, and they are not equally strong.

Each engine is cross-validated. mlsynth’s synthetic DiD matches the authors’ synthdid R package on Proposition 99 and Stata sdid on the EU ETS panel, and tests/test_geox.py asserts that the ATT GEOX scores a backtest with matches SDID(...).fit() on the same panel and window. The augsynth engine is the shared BilevelSCM, which carries its own validation, reached here through the adapter the seam tests pin. The two structural properties the effect sweep relies on are checked against brute-force recomputation in tests/test_geox.py, so the shortcut is proved and not assumed.

The harness does not perturb the engine. Force one market as the treated region and hand GEOX the real post-period, and its readout equals SDID(...).fit() on Proposition 99 exactly – the ATT and all 38 donor weights – over six treated units and seven design-knob settings (benchmarks/cases/geox_sdid_equivalence.py). That is what makes the engine’s validation the design’s: mlsynth’s SDID sits 1.6e-3 packs from the authors’ synthdid R, and GEOX sits zero from mlsynth’s SDID. See GEOX — does the harness change the estimator it wraps?.

The harness is cross-validated. With engine="augsynth" selected, GEOX reproduces the market selection GeoLift itself publishes: on the walkthrough panel, all five of its top-ranked designs come back with the same rank, minimum detectable effect, CPIC investment and abs_lift_in_zero, fourteen quantities value for value (benchmarks/cases/geox_augsynth_geolift.py). This reaches further than the engine: nomination, the backtest windows, effect injection, the power sweep, the MDE rule and the composite rank are shared code, so a divergence anywhere in that stack would move a rank or an investment. It is also what licenses reading the two engines against each other, since the harness around them is the same.

The composition is self-validated. Whether an MDE from SDID-scored backtests means what it claims has no external referent, so benchmarks/cases/geox_mc.py imposes ground truth on a constructed factor DGP and measures size at the null, out-of-sample power at the reported MDE, and the gap between the selected winner’s MDE and a region fixed in advance. See GEOX: does the minimum detectable effect mean what it says?. This is the weakest of the three, because the DGP and the claim come from the same place.

Core API

GEOX: geo experiment design by simulated power.

GEOX chooses which geographic markets to treat in a marketing geo experiment, before the experiment runs. It scores every candidate test region by simulated power: slide a pseudo-treatment window back through the observed history, inject a known lift, and ask how often the estimator would have detected it. The region with the smallest reliably detectable effect wins.

The market-selection loop follows GeoLift (Meta’s geo-experiment package): anchor a candidate at each market and take its most correlated neighbours, sweep effect sizes over backward backtests, then rank by a composite of the minimum detectable effect, the power at that effect, and how faithfully the estimator recovers the injected lift.

Which estimator does the scoring is a setting. engine="sdid" (the default) is synthetic difference-in-differences (Arkhangelsky, Athey, Hirshberg, Imbens & Wager 2021), which reweights donor markets and pre-periods, so no single pre-period has to be matched exactly and a level gap between the test region and its donors is differenced out. engine="augsynth" is the ridge-augmented synthetic control (Ben-Michael, Feller & Rothstein 2021) GeoLift itself scores with. Everything around the fit – nomination, backtests, injection, power, the MDE, the rank, the constraints – is the same either way.

inference varies separately. Placebo reassignment (SDID’s Algorithm 4) needs only a donor-built counterfactual and runs on either engine; conformal permutation runs on augsynth alone, because SDID’s time weights exist to say pre-periods are not exchangeable.

The output is a design – which markets to treat, the donor weights the analysis will use, and the power table behind the choice – not a treatment effect. The report slot stays None until the experiment has run.

class mlsynth.estimators.geox.GEOX(config: GEOXConfig)

Bases: object

Geo experiment design by simulated power.

Parameters:

config (GEOXConfig) – Panel, candidate-region constraints, scoring engine, simulation grid, and decision rule.

Examples

>>> import pandas as pd
>>> from mlsynth import GEOX
>>> from mlsynth.config_models import GEOXConfig
>>> df = pd.read_csv("basedata/geolift_test_data.csv")
>>> design = GEOX(GEOXConfig(
...     df=df, unitid="location", time="date", outcome="Y",
...     treatment_size=2, durations=[14],
...     effect_sizes=[-0.1, 0.0, 0.1, 0.2], n_backtests=5,
... )).fit()
>>> design.selected_units
['chicago', 'portland']

fit() → GEOXResults

Run the design search and return the chosen test region.

Returns:

GEOXResults – A DesignResult carrying selected_units, assignment, design_weights, the ranked power table, and the full search detail.

class mlsynth.config_models.GEOXConfig(*, df: DataFrame, outcome: str, unitid: str, time: str, treatment_size: int | List[int], to_be_treated: List | None = None, not_to_be_treated: List | None = None, cluster_col: str | None = None, adjacency: Any | None = None, spillover_threshold: float = 0.0, stratum_col: str | None = None, min_per_stratum: int | None = None, max_per_stratum: int | None = None, size_col: str | None = None, min_size: float | None = None, max_size: float | None = None, run_stochastic: bool = False, stochastic_mode: str = 'global', how: str = 'mean', post_col: str | None = None, engine: str = 'sdid', inference: str | None = None, ns: int = 1000, conformal_type: str = 'block', finite_sample_p: bool = False, fixed_effects: bool | None = None, augment: str | None = 'ridge', durations: List[int], effect_sizes: List[float], n_backtests: int = 5, alpha: float = 0.1, power_threshold: float = 0.8, n_validation_backtests: int = 8, n_draws: int = 200, cpic: float | None = None, budget: float | None = None, seed: int = 0, n_jobs: int = 1)

Bases: BaseMAREXConfig

Configuration for the GEOX market-selection design.

adjacency: Any | None

alpha: float

augment: str | None

budget: float | None

cluster_col: str | None

conformal_type: str

cpic: float | None

durations: List[int]

effect_sizes: List[float]

engine: str

finite_sample_p: bool

fixed_effects: bool | None

how: str

inference: str | None

max_per_stratum: int | None

max_size: float | None

min_per_stratum: int | None

min_size: float | None

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

n_backtests: int

n_draws: int

n_jobs: int

n_validation_backtests: int

not_to_be_treated: List | None

ns: int

post_col: str | None

power_threshold: float

run_stochastic: bool

seed: int

size_col: str | None

spillover_threshold: float

stochastic_mode: str

stratum_col: str | None

to_be_treated: List | None

treatment_size: int | List[int]

property treatment_sizes: List[int]

The requested region sizes, sorted and de-duplicated.

A scalar treatment_size reads as a one-element scan, so the rest of the pipeline never has to branch on which form was given.

The SDID scoring engine for GEOX.

One synthetic difference-in-differences fit on a pseudo-experiment, plus the placebo standard error its p-value is read against.

SDID (Arkhangelsky, Athey, Hirshberg, Imbens & Wager 2021) estimates the ATT as a doubly weighted difference in differences: unit weights omega match the donors to the treated pre-period trajectory, and time weights lambda match the pre-periods to the post-period donor average. The estimate is

\[\hat\tau = \big(\bar y_{post} - \lambda' y_{pre}\big) - \big(\omega' \bar Y_{0,post} - \lambda' Y_{0,pre}\omega\big),\]

which rearranges to a per-period path: the counterfactual is Y0 @ omega + bias_correction with bias_correction = lambda' y_pre - lambda' Y0_pre omega, and the effect at t is the gap against it. That is the same decomposition mlsynth.utils.sdid_helpers.cohort.estimate_cohort_sdid_effects() uses, so the design is scored by the estimator that will analyse the experiment.

Two structural properties make the power sweep cheap, and both are consequences of where the weights get their information:

• unit_weights() reads the treated pre-period trajectory, fit_time_weights() reads only donors, and compute_regularization() reads only donors and counts. No program sees the treated post block, so injecting an effect there cannot move omega or lambda and

  \[\hat\tau(e) = \hat\tau(0) + e \cdot \bar y_{post}\]

  holds exactly. The effect-size sweep is arithmetic, not refitting.

• The placebo draws reassign control units as pseudo-treated, so the injected effect never enters them. One placebo run covers the whole sweep.

Both are pinned in tests/test_geox.py against brute-force recomputation.

class mlsynth.utils.geox_helpers.engine.SDIDFit(omega: ndarray, lam: ndarray, bias_correction: float, counterfactual: ndarray, pre_rmspe: float, scaled_l2: float, zeta: float)

One SDID fit on a pseudo-experiment’s pre-period.

omega

Unit (donor) weights; non-negative and summing to one.

Type:

np.ndarray, shape (J,)

lam

Time weights over the backtest’s pre-period.

Type:

np.ndarray, shape (T0,)

bias_correction

lambda' y_pre - lambda' Y0_pre omega, the difference-in-differences level term that turns Y0 @ omega into a counterfactual path.

Type:

float

counterfactual

Y0 @ omega + bias_correction over every period of the panel.

Type:

np.ndarray, shape (T,)

pre_rmspe

Root mean squared pre-period gap against the counterfactual.

Type:

float

scaled_l2

pre_rmspe as a fraction of the treated series’ own pre-period standard deviation, so it compares across candidates of different size.

Type:

float

zeta

The unit-weight ridge (compute_regularization()). SDID’s analogue of augmented SCM’s penalty, but closed form instead of cross-validated.

Type:

float

bias_correction: float

counterfactual: ndarray

lam: ndarray

omega: ndarray

pre_rmspe: float

scaled_l2: float

tau_path(y) → ndarray

Per-period effect y - counterfactual over every period.

zeta: float

mlsynth.utils.geox_helpers.engine.normal_p_value(tau: float, sigma: float | None) → float

Two-sided p-value 2 (1 - Phi(|tau| / sigma)) for the null of no effect.

SDID’s variance procedures deliver a standard error, so detection is a normal-approximation test, not the conformal permutation test an augmented SCM design would use.

mlsynth.utils.geox_helpers.engine.placebo_sigma(y, Y0, n_pre: int, start: int, end: int, *, n_draws: int = 200, n_tr: int = 1, seed: int = 0) → float

Placebo standard error of the ATT (Arkhangelsky et al., Algorithm 4).

Reassigns n_tr donors as pseudo-treated, drops them from the donor pool, reruns the full SDID fit, and takes the standard deviation of the resulting ATTs. This is the only one of the paper’s three variance procedures defined for a single treated series, which is what a candidate test region collapses to, and it is what synthdid defaults to for that case.

The treated series is never read, so the returned sigma is the null distribution’s scale and does not depend on any injected effect.

Returns:

float – The placebo standard error, or nan when the donor pool is too small to draw from or too few draws converge.

mlsynth.utils.geox_helpers.engine.sdid_att(fit: SDIDFit, y, start: int, end: int) → float

The SDID ATT over [start, end]: the mean per-period effect.

mlsynth.utils.geox_helpers.engine.sdid_fit_once(y, Y0, n_pre: int, start: int, end: int, n_tr: int = 1) → SDIDFit

Fit SDID once on the backtest [:n_pre] / [start:end+1].

Parameters:

• y (array-like, shape (T,)) – Treated (aggregated candidate) outcomes over the full panel.

• Y0 (array-like, shape (T, J)) – Donor outcomes over the full panel.

• n_pre (int) – Number of pre-treatment periods.

• start, end (int) – Inclusive 0-indexed bounds of the pseudo-treatment block.

• n_tr (int) – Number of treated units, which enters the ridge as (n_tr * T_post)^(1/4) (synthdid’s eta.omega).

Returns:

SDIDFit

Raises:

• MlsynthConfigError – If the window is malformed or a weight program fails to converge.

• MlsynthDataError – If the donor pool is empty.

One pseudo-experiment for GEOX market-selection scoring.

A single backtest: fit SDID on the backtest’s pre-period, take the placebo standard error once, then sweep the effect sizes. Because neither SDID weight program reads the treated post block (see engine), the sweep is exact arithmetic on the ATT and the placebo sigma is shared across it – so the whole grid of effect sizes costs one fit and one placebo run.

The row schema matches what compute_power() consumes, one row per effect size.

mlsynth.utils.geox_helpers.simulate.inject_effect(treated, start: int, end: int, es: float) → ndarray

Scale the block [start, end] of treated by (1 + es).

The multiplicative injection GeoLift uses (Y[D == 1] *= 1 + es), so an effect size reads as a percentage lift on the treated markets’ own volume. Returns a new array; the input is never mutated.

Raises:

MlsynthConfigError – If [start, end] is not a valid in-bounds, non-empty window.

mlsynth.utils.geox_helpers.simulate.simulate_backtest(treated, donors, n_periods: int, duration: int, sim: int, effect_sizes, *, n_draws: int = 200, n_tr: int = 1, seed: int = 0, cpic: float | None = None, treated_total: ndarray | None = None, analytic: bool = True, engine: str = 'sdid', engine_kwargs: dict | None = None, alpha: float = 0.1) → List[dict]

Simulate one backtest across a grid of effect sizes.

Parameters:

• treated (array-like, shape (n_periods,)) – Aggregated treated series over the full panel.

• donors (array-like, shape (n_periods, J)) – Donor pool over the full panel.

• n_periods, duration, sim (int) – Panel length, pseudo-treatment duration, and backtest index.

• effect_sizes (iterable of float) – Effect sizes to sweep.

• n_draws (int) – Placebo draws behind the standard error.

• n_tr (int) – Number of treated markets in the candidate.

• cpic (float, optional) – Cost per incremental conversion. When given, each row reports the required cpic * effect_size * summed-treated-volume investment.

• treated_total (np.ndarray, optional) – Summed treated series for the investment volume, which stays a total even when the fit runs on the per-market mean.

• analytic (bool, default True) – Use the closed-form tau(es) = tau(0) + es * mean(y_post). Setting this to False re-injects and recomputes for each effect size, which gives the same answer and exists so the tests can prove the shortcut.

Returns:

list of dict – One row per effect size, carrying sim, duration, effect_size, p_value, placebo_mean_effect (the SDID ATT), detected_lift (ATT over the counterfactual post mean), scaled_l2, pre_rmspe and investment.

Raises:

MlsynthConfigError – If the backtest runs off the start of the panel, or treated / donors do not have n_periods rows.

mlsynth.utils.geox_helpers.simulate.simulate_recast_panel(scenario: str, effect_pct: float, seed: int, *, n_control: int = 20, total_days: int = 105, pre_days: int = 90, baseline_mean: float = 4000.0, baseline_spread: float = 0.6, trend_slope: float = 0.001, seasonality_amplitude: float = 0.1, autocorrelation: float = 0.3, noise_level: float = 0.2, outlier_multiplier: float = 5.0)

One panel from the Recast study’s DGP.

Y_cf[i,t] = baseline_i * trend_t * season_t * exp(noise_level * scale_i * ar_noise[i,t]) with scale_i = sqrt(noise_baseline_i / mean(noise_baselines)), and the treated geo’s post window multiplied by 1 + effect_pct. Every geo shares the trend and the weekly profile; only the baseline level and the noise draw differ, which is the study’s stated cross-geo structure (and its stated caveat – donors move in lockstep up to noise).

exp keeps the panel strictly positive whatever the noise draw, so a multiplicative lift is always well defined.

The treated geo is the one closest to the median baseline, chosen before any scenario modification; A2’s inflation is applied to that geo afterwards and deliberately does not feed the noise scaling, matching their noise_baselines argument.

Returns:

(wide, treated, cf) – wide – (total_days, n_geos) observed outcomes, columns City 1 ...; treated – the treated column’s name; cf – the treated geo’s untreated counterfactual, so the true ATT is exact and not approximated.

Batch driver for GEOX market-selection scoring.

Loops the single-backtest simulate_backtest() over the full candidates x durations x backtests grid and stacks the rows into one long table, ready for the power -> MDE -> rank aggregation.

mlsynth.utils.geox_helpers.batch.run_simulations(Ywide: DataFrame, candidates: Iterable[frozenset], durations: Iterable[int], n_backtests: int, effect_sizes: Iterable[float], *, n_draws: int = 200, seed: int = 0, cpic: float | None = None, n_jobs: int = 1, excluded: Mapping[frozenset, Iterable] | None = None, engine: str = 'sdid', engine_kwargs: Mapping[str, object] | None = None, alpha: float = 0.1) → DataFrame

Run the simulation grid and stack the results into one long table.

For each candidate test region, each treatment duration, and each backtest backtest sim = 1 .. n_backtests, fit SDID once and sweep the effect sizes, tagging every row with its candidate.

excluded maps a candidate to the markets barred from its donor pool (its spillover conflict-neighbours). Absent, every candidate keeps the full complement as donors.

Returns:

pd.DataFrame – Long-form table with one row per (candidate, duration, sim, effect size). Empty (with the right columns) when there are no candidates.

Raises:

MlsynthConfigError – If n_backtests is not a positive integer, or a backtest runs off the start of the panel.

Aggregation of the simulation cube for GEOX market selection.

Pure array/groupby reductions on the long p-value cube from run_simulations(), faithful to GeoLiftMarketSelection:

1. compute_power() – collapse the backtest dimension: power = detection rate, plus the backtest-averaged metrics.

2. compute_mde() – the minimum detectable effect per (candidate, duration), with GeoLift’s signed positive/negative selection.

(The composite rank is built on top of these.)

mlsynth.utils.geox_helpers.aggregate.compute_exact_mde(cube: DataFrame, *, power_threshold: float = 0.8) → DataFrame

The effect a design actually detects, per (candidate, duration).

compute_mde reports the smallest simulated effect whose power clears the threshold, so its resolution is the effect grid’s step: a design that truly detects 0.1014 reports 0.150 on a 0.05 grid, because 0.1014 misses the 0.10 point. This reports the crossing itself.

The per-backtest boundaries come from the engine (closed form where the p-value is analytic in the effect). Power above the threshold means at least k = floor(power_threshold * n_sims) + 1 backtests detect, so the design’s boundary is the k-th smallest of them – an order statistic, which is why this works on the pre-aggregation cube instead of riding through compute_power(), which averages.

Both directions are kept. The backtest’s own placebo effect is generally nonzero, so the detection interval sits off centre and a design can need less of a push downward than upward.

Returns nan for a design whose boundaries are not finite, which is what an engine reports when its p-value is not analytic in the effect.

mlsynth.utils.geox_helpers.aggregate.compute_mde(power_table: DataFrame, *, power_threshold: float = 0.8) → DataFrame

Minimum detectable effect per (candidate, duration).

Faithful to GeoLift: among effect sizes whose power exceeds power_threshold, take the smallest-magnitude detectable positive and negative effects and keep the smaller magnitude (ties -> positive). If nothing is detectable, the MDE is nan.

Parameters:

• power_table (pd.DataFrame) – Output of compute_power() (needs effect_size and power).

• power_threshold (float, default 0.8) – Power a candidate must exceed to “detect” an effect.

Returns:

pd.DataFrame – One row per (candidate, duration) with the signed mde.

mlsynth.utils.geox_helpers.aggregate.compute_power(cube: DataFrame, *, alpha: float = 0.1) → DataFrame

Collapse the backtest (sim) dimension into power + averaged metrics.

Power is the detection rate mean(p_value < alpha) over the backtest backtests; the other quantities are averaged over the same backtests (scaled_l2 / pre_rmspe are constant across sim only if the panel is, so they are averaged for generality).

Parameters:

• cube (pd.DataFrame) – Long simulation table from run_simulations().

• alpha (float, default 0.1) – Significance level for the detection test.

Returns:

pd.DataFrame – One row per (candidate, duration, effect_size) with power, placebo_mean_effect, scaled_l2, pre_rmspe.

mlsynth.utils.geox_helpers.aggregate.compute_rank(power_table: DataFrame, *, power_threshold: float = 0.8, budget: float | None = None) → DataFrame

Rank candidate designs, haircut-faithful to GeoLiftMarketSelection.

Builds the per-(candidate, duration) MDE row, then the GeoLift composite rank: the mean of three dense_rank components – |mde|, power (at the MDE; ascending, as in GeoLift), and abs_lift_in_zero (the recovery error |AvgDetectedLift - mde| at the MDE). scaled_l2 / pre_rmspe are carried for reporting but are not ranked. Candidates with no detectable effect (mde NaN) are dropped. Lower rank = better.

Parameters:

• power_table (pd.DataFrame) – Output of compute_power().

• power_threshold (float, default 0.8) – Forwarded to compute_mde().

Returns:

pd.DataFrame – One row per ranked (candidate, duration), sorted by rank.

End-to-end assembly for the GEOX design.

run_design wires the pipeline together:

geoex_dataprep -> rank_markets_by_correlation -> generate_candidate_markets -> run_simulations -> compute_power -> compute_rank -> design_fit -> GEOXResults

mlsynth.utils.geox_helpers.orchestration.design_fit(Ywide: DataFrame, candidate, n_pre: int, duration: int, exclude=None, how: str = 'mean', engine: str = 'sdid', engine_kwargs: dict | None = None) → CandidateDesign

Deployable SDID design for one candidate, fit on the full pre-period.

The scoring stage fits each backtest; this fits the design the experiment will actually deploy, with the pseudo-treatment window sitting at the end of the observed history.

exclude drops the candidate’s conflict-neighbours from its donor pool (the spillover exclusion restriction).

mlsynth.utils.geox_helpers.orchestration.engine_settings(config: GEOXConfig) → dict

The engine-specific settings bundle for config.

One bundle whichever engine runs: each engine reads what applies to it and ignores the rest, so the pipeline never branches on which is active.

mlsynth.utils.geox_helpers.orchestration.planning_mde(Ywide: DataFrame, candidate, config: GEOXConfig, power_table: DataFrame, exclude=None) → float | None

The winner’s MDE re-scored on backtests that did not choose it.

compute_rank hands back the smallest MDE in the field, and the smallest of many noisy estimates is optimistic: the region most likely to be picked is the one whose estimate happened to come out low. Re-scoring the winner on backtests held back from the search removes that, because those backtests played no part in selecting it – the same reason a region fixed in advance is calibrated at any backtest count.

Backtests n_backtests + 1 .. n_backtests + n_validation_backtests sit deeper in history, so their pseudo-treatment windows differ from every window the search used. Returns None when the panel cannot carry them.

mlsynth.utils.geox_helpers.orchestration.run_design(config: GEOXConfig) → GEOXResults

Run the full GEOX market-selection design from a config.

Result containers for GEOX.

GEOX produces a design, not a treatment effect, so its result is a DesignResult: which markets to treat, the donor weights the analysis will use, the power table behind the choice, and a report slot that stays None until the experiment has actually run.

class mlsynth.utils.geox_helpers.structures.CandidateDesign(units: List, weights: WeightsResults, observed: ndarray, counterfactual: ndarray, n_pre: int, pre_rmspe: float, scaled_l2: float, mde: float | None = None, power: float | None = None, rank: float | None = None, mde_planning: float | None = None)

One candidate test region’s deployable design.

units

The markets in the candidate region.

Type:

list

weights

Both SDID weight vectors: donor_weights over markets and time_weights over the pre-period dates.

Type:

WeightsResults

observed

The test region’s aggregated outcome over the panel.

Type:

np.ndarray

counterfactual

Synthetic path over the panel.

Type:

np.ndarray

n_pre

Pre-period length the design was fit on.

Type:

int

pre_rmspe, scaled_l2

Pre-period fit quality.

Type:

float

mde, power, rank

Stitched in from the shortlist: the minimum detectable effect, the power at that effect, and the composite rank. None when the candidate cleared no effect size.

Type:

float or None

mde_planning

The winner’s MDE re-scored on backtests that took no part in choosing it. mde is the smallest in a field of candidates and so is optimistic; this one is not selected on. Set only on the winner, and None when n_validation_backtests is 0 or the panel is too short.

Type:

float or None

counterfactual: ndarray

mde: float | None = None

mde_planning: float | None = None

n_pre: int

observed: ndarray

power: float | None = None

pre_rmspe: float

rank: float | None = None

scaled_l2: float

units: List

weights: WeightsResults

class mlsynth.utils.geox_helpers.structures.GEOXResults(*, report: BaseEstimatorResults | None = None, assignment: Any | None = None, selected_units: Any | None = None, design_weights: WeightsResults | None = None, power: Any | None = None, metadata: Dict[str, Any] | None = None, search: MarketSearch | None = None, **extra_data: Any)

Top-level result of the GEOX design.

A DesignResult front door (selected_units / assignment / design_weights / power / metadata / report) plus the grouped MarketSearch detail.

class Config

arbitrary_types_allowed = True

extra = 'allow'

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'json_encoders': {<class 'numpy.ndarray'>: <function MlsynthResult.Config.<lambda>>}}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

search: MarketSearch | None

class mlsynth.utils.geox_helpers.structures.MarketSearch(shortlist: ~pandas.DataFrame, power_table: ~pandas.DataFrame = <factory>, candidates: ~typing.List[~mlsynth.utils.geox_helpers.structures.CandidateDesign] = <factory>, winner: ~mlsynth.utils.geox_helpers.structures.CandidateDesign | None = None)

The design search: every candidate region and the winner.

shortlist

Ranked table, one row per (candidate, duration) that cleared the power threshold.

Type:

pd.DataFrame

power_table

Power at every (candidate, duration, effect size), before the MDE collapses it. This is what the power curves are drawn from.

Type:

pd.DataFrame

candidates

Every candidate’s deployable design.

Type:

list of CandidateDesign

winner

The top-ranked design, or None when nothing cleared the threshold.

Type:

CandidateDesign or None

candidates: List[CandidateDesign]

power_table: DataFrame

shortlist: DataFrame

winner: CandidateDesign | None = None

Plots for an GEOX design.

Two views, matching what GeoLift shows for a market selection:

• the power curve, power against injected effect size, with the detection threshold and the minimum detectable effect marked, and every other candidate drawn faintly behind so the winner is placed among the alternatives;

• the fit being bought, the test region’s aggregate against its synthetic counterfactual over the pre-period.

Both render in the mlsynth house style (mlsynth_style()); pass theme to override.

mlsynth.utils.geox_helpers.plotter.plot_geox_design(result, *, power_threshold: float = 0.8, theme=None, figsize=(12, 4.8), save_path: str | None = None, show: bool = False)

Plot an GEOX design: the power curve and the fit it rests on.

Parameters:

• result (GEOXResults) – The design returned by GEOX(...).fit().

• power_threshold (float, default 0.8) – Threshold line to draw; match the config used for the design.

• theme (dict or str, optional) – rcParams dict or named Matplotlib style overriding the house style.

• figsize (tuple, default (12, 4.8))

• save_path (str, optional) – Write the figure here.

• show (bool, default False) – Call plt.show() before returning.

Returns:

matplotlib.figure.Figure

Raises:

ValueError – If no candidate cleared the power threshold, so there is no design.

mlsynth.utils.geox_helpers.plotter.plot_mde_ranking(result, *, top: int = 12, theme=None, figsize=(8, 5), save_path: str | None = None, show: bool = False)

Rank the shortlisted test regions by minimum detectable effect.

The market-selection view: which regions can detect the smallest lift, with the chosen one highlighted.