From 2a4b617eced99420dbb2b0cd24344f2c607c494a Mon Sep 17 00:00:00 2001 From: igerber Date: Fri, 24 Apr 2026 12:41:55 -0400 Subject: [PATCH 1/3] Add R-parity fixtures + TestDCDHDynRParityByPath for by_path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wave 1 of the by_path follow-up sequence. Locks in the per-path SE convention against R DIDmultiplegtDYN 2.3.3 before downstream waves (inference completion, design variants, survey) build on it. Extends benchmarks/R/generate_dcdh_dynr_test_values.R with an extract_dcdh_by_path helper and two scenarios: - mixed_single_switch_by_path: 2 observed paths, by_path=2 - multi_path_reversible_by_path: 4 observed paths, by_path=3 The multi_path_reversible DGP is deterministic: path assignment is a function of F_g so each (D_{g,1}, F_g, S_g) cohort contains switchers from a single path. This keeps the cohort-recentered IF comparable to R's per-path re-run; cross-path cohort sharing is the documented SE divergence mechanism. New TestDCDHDynRParityByPath in tests/test_chaisemartin_dhaultfoeuille_parity.py matches paths by tuple label via set-equality (robust to R's undocumented frequency-tie tiebreak) and hard-asserts per-path switcher counts before SE comparison. Observed parity: point estimates and switcher counts match R exactly; per-path SE within 12% rtol (Phase 2 envelope) — scenario 13 max 10.2%, scenario 14 max 4.2%. REGISTRY.md Note (Phase 3 by_path...) updated with R-parity confirmation and a **Deviation from R (cross-path cohort-sharing SE)** bullet describing the mechanism under which full-panel cohort-centered plug-in (ours) and per-path re-run (R's) diverge. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 1 + benchmarks/R/generate_dcdh_dynr_test_values.R | 153 +++++++++++++- benchmarks/data/dcdh_dynr_golden_values.json | 193 ++++++++++++++++++ docs/methodology/REGISTRY.md | 2 +- ...test_chaisemartin_dhaultfoeuille_parity.py | 129 ++++++++++++ 5 files changed, 476 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9c503e6d..aaa2d43d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`stute_joint_pretest`, `joint_pretrends_test`, `joint_homogeneity_test` + `StuteJointResult`** (HeterogeneousAdoptionDiD Phase 3 follow-up). Joint Cramér-von Mises pretests across K horizons with shared-η Mammen wild bootstrap (preserves vector-valued empirical-process unit-level dependence per Delgado-Manteiga 2001 / Hlávka-Hušková 2020). The core `stute_joint_pretest` is residuals-in; two thin data-in wrappers construct per-horizon residuals for the two nulls the paper spells out: mean-independence (step 2 pre-trends, `OLS(Y_t − Y_base ~ 1)` per pre-period) and linearity (step 3 joint, `OLS(Y_t − Y_base ~ 1 + D)` per post-period). Sum-of-CvMs aggregation (`S_joint = Σ_k S_k`); per-horizon scale-invariant exact-linear short-circuit. Closes the paper Section 4.2 step-2 gap that Phase 3 `did_had_pretest_workflow` previously flagged with an "Assumption 7 pre-trends test NOT run" caveat. See `docs/methodology/REGISTRY.md` §HeterogeneousAdoptionDiD "Joint Stute tests" for algorithm, invariants, and scope exclusion of Eq 18 linear-trend detrending (deferred to Phase 4 Pierce-Schott replication). - **`did_had_pretest_workflow(aggregate="event_study")`**: multi-period dispatch on balanced ≥3-period panels. Runs QUG at `F` + joint pre-trends Stute across earlier pre-periods + joint homogeneity-linearity Stute across post-periods. Step 2 closure requires ≥2 pre-periods; with only a single pre-period (the base `F-1`) `pretrends_joint=None` and the verdict flags the skip. Reuses the Phase 2b event-study panel validator (last-cohort auto-filter under staggered timing with `UserWarning`; `ValueError` when `first_treat_col=None` and the panel is staggered). The data-in wrappers `joint_pretrends_test` and `joint_homogeneity_test` also route through that same validator internally, so direct wrapper calls inherit the last-cohort filter and constant-post-dose invariant. `HADPretestReport` extended with `pretrends_joint`, `homogeneity_joint`, and `aggregate` fields; serialization methods (`summary`, `to_dict`, `to_dataframe`, `__repr__`) preserve the Phase 3 output bit-exactly on `aggregate="overall"` — no `aggregate` key, no header row, no schema drift — and only surface the new fields on `aggregate="event_study"`. - **`ChaisemartinDHaultfoeuille.by_path`** — per-path event-study disaggregation, mirroring R `did_multiplegt_dyn(..., by_path=k)`. Passing `by_path=k` (positive int) to the estimator reports separate `DID_{path,l}` + SE + inference for the top-k most common observed treatment paths in the window `[F_g-1, F_g-1+L_max]`, answering the practitioner question "is a single pulse enough, or do you need sustained exposure?" across paths like `(0,1,0,0)` vs `(0,1,1,0)` vs `(0,1,1,1)`. The per-path SE follows the joiners-only / leavers-only IF precedent (switcher-side contribution zeroed for non-path groups; control pool and cohort structure unchanged; plug-in SE with path-specific divisor). Requires `drop_larger_lower=False` (multi-switch groups are the object of interest) and `L_max >= 1`. Binary treatment only in this release; combinations with `controls`, `trends_linear`, `trends_nonparam`, `heterogeneity`, `design2`, `honest_did`, `survey_design`, and `n_bootstrap > 0` raise `NotImplementedError` and are deferred to follow-up PRs. Results expose `results.path_effects: Dict[Tuple[int, ...], Dict[str, Any]]` and `results.to_dataframe(level="by_path")`; the summary grows a "Treatment-Path Disaggregation" block. Ties in path frequency are broken lexicographically on the path tuple for deterministic ranking. Overflow (`by_path > n_observed_paths`) returns all observed paths with a `UserWarning`. See `docs/methodology/REGISTRY.md` §ChaisemartinDHaultfoeuille `Note (Phase 3 by_path per-path event-study disaggregation)` for the full contract. +- **R-parity for `ChaisemartinDHaultfoeuille.by_path`** against `DIDmultiplegtDYN 2.3.3`. Two new scenarios in `benchmarks/data/dcdh_dynr_golden_values.json` generated from `did_multiplegt_dyn(..., by_path=k)`: `mixed_single_switch_by_path` (2 paths, `by_path=2`) and `multi_path_reversible_by_path` (4 observed paths, `by_path=3`, via a new deterministic multi-path DGP pattern in the R generator). Per-path point estimates and per-path switcher counts match R exactly; per-path SE matches within the Phase 2 multi-horizon SE envelope (observed rtol ≤ 10.2% on the 2-path scenario, ≤ 4.2% on the 4-path scenario). Parity tests live at `tests/test_chaisemartin_dhaultfoeuille_parity.py::TestDCDHDynRParityByPath`, matching paths by tuple label via set-equality (robust to R's undocumented frequency-tie tiebreak) and cross-checking per-path switcher counts before SE comparison. **Deviation documented:** cross-path cohort sharing — our full-panel cohort-centered plug-in vs R's per-path re-run diverges materially when a `(D_{g,1}, F_g, S_g)` cohort spans multiple observed paths; the two coincide when every cohort is single-path. The parity scenarios are constructed to keep cohorts single-path (scenario 13 by design, scenario 14 via path-assignment-deterministic-on-F_g). See `docs/methodology/REGISTRY.md` §ChaisemartinDHaultfoeuille `Note (Phase 3 by_path...)` for the full write-up. - **`target_parameter` block in BR/DR schemas (experimental; schema version bumped to 2.0)** — `BUSINESS_REPORT_SCHEMA_VERSION` and `DIAGNOSTIC_REPORT_SCHEMA_VERSION` bumped from `"1.0"` to `"2.0"` because the new `"no_scalar_by_design"` value on the `headline.status` / `headline_metric.status` enum (dCDH `trends_linear=True, L_max>=2` configuration) is a breaking change per the REPORTING.md stability policy. BusinessReport and DiagnosticReport now emit a top-level `target_parameter` block naming what the headline scalar actually represents for each of the 16 result classes. Closes BR/DR foundation gap #6 (target-parameter clarity). Fields: `name`, `definition`, `aggregation` (machine-readable dispatch tag), `headline_attribute` (raw result attribute), `reference` (citation pointer). BR's summary emits the short `name` right after the headline; DR's overall-interpretation paragraph does the same; both full reports carry a "## Target Parameter" section with the full definition. Per-estimator dispatch is sourced from REGISTRY.md and lives in the new `diff_diff/_reporting_helpers.py::describe_target_parameter`. A few branches read fit-time config (`EfficientDiDResults.pt_assumption`, `StackedDiDResults.clean_control`, `ChaisemartinDHaultfoeuilleResults.L_max` / `covariate_residuals` / `linear_trends_effects`); others emit a fixed tag (the fit-time `aggregate` kwarg on CS / Imputation / TwoStage / Wooldridge does not change the `overall_att` scalar — disambiguating horizon / group tables is tracked under gap #9). See `docs/methodology/REPORTING.md` "Target parameter" section. - SyntheticDiD coverage Monte Carlo calibration table added to `docs/methodology/REGISTRY.md` §SyntheticDiD — rejection rates at α ∈ {0.01, 0.05, 0.10} across `placebo` / `bootstrap` / `jackknife` on 3 representative DGPs (balanced / exchangeable, unbalanced, and Arkhangelsky et al. (2021) AER §6.3 non-exchangeable). Artifact at `benchmarks/data/sdid_coverage.json` (500 seeds × B=200), regenerable via `benchmarks/python/coverage_sdid.py`. diff --git a/benchmarks/R/generate_dcdh_dynr_test_values.R b/benchmarks/R/generate_dcdh_dynr_test_values.R index 9a0c0e3e..3da74e8d 100644 --- a/benchmarks/R/generate_dcdh_dynr_test_values.R +++ b/benchmarks/R/generate_dcdh_dynr_test_values.R @@ -29,6 +29,11 @@ library(DIDmultiplegtDYN) library(jsonlite) suppressMessages(library(polars)) # required by DIDmultiplegtDYN >= 2.x +# Pin DIDmultiplegtDYN min version because the by_path output slots +# (res$by_levels, res$by_level_i) were introduced in v2.3.3 and the +# structure is not version-stable per the R package's own docs. +stopifnot(packageVersion("DIDmultiplegtDYN") >= "2.3.3") + cat("Generating dCDH golden values via DIDmultiplegtDYN at l=1...\n") output_path <- file.path("benchmarks", "data", "dcdh_dynr_golden_values.json") @@ -51,7 +56,8 @@ gen_reversible <- function(n_groups, n_periods, pattern, seed, cycle_length = 2, treatment_effect = 2.0, heterogeneous_effects = FALSE, effect_sd = 0.5, group_fe_sd = 2.0, time_trend = 0.1, noise_sd = 0.5, - n_never_treated = 20, n_always_treated = 20) { + n_never_treated = 20, n_always_treated = 20, + L_max = 3) { # n_never_treated and n_always_treated add stable control cohorts so # both Python (AER 2020 zero-retention) and R DIDmultiplegtDYN (dynamic # paper, drop-cohort) implementations have controls available at every @@ -99,6 +105,76 @@ gen_reversible <- function(n_groups, n_periods, pattern, seed, D[g, seq_len(st - 1L)] <- 1L } } + } else if (pattern == "multi_path_reversible") { + # Deterministic multi-path DGP designed for by_path R-parity: + # - 4 distinct joiner-style target paths with unequal frequencies + # (so top-k ranking produces unique ranks with no ties) + # - path assignment is a DETERMINISTIC FUNCTION OF F_g, so each + # cohort (D_{g,1}, F_g, S_g) contains switchers from a single + # path. This avoids cross-path cohort sharing in the + # cohort-recentered influence function, which otherwise blows + # out SE parity with R's re-run-per-path convention. + # - post-window treatment is stable at path[L_max+1] (clean + # control-pool eligibility — no post-window contamination) + # + # Each group: + # - F_g in [2, n_periods - L_max] so the length-(L_max+1) window + # [F_g-1, F_g-1+L_max] fits the panel + # - path is determined by F_g: two F_g values per path (groups in + # {F_g in {2,3}} share path 1, {F_g in {4,5}} share path 2, + # {F_g in {6}} = path 3, {F_g in {7}} = path 4); within a path, + # F_g distribution yields n_groups * path_prop total groups + max_switch <- n_periods - L_max - 1L + stopifnot(max_switch >= 1L) + # With n_periods=10, L_max=3: max_switch=6, F_g in [2,7] (6 values). + + target_paths <- list( + c(0L, 1L, 1L, 1L), # sustained on (rank 1) + c(0L, 1L, 1L, 0L), # on then off (rank 2) + c(0L, 1L, 0L, 0L), # on briefly (rank 3) + c(0L, 1L, 0L, 1L) # on-off-on (rank 4, truncated under by_path=3) + ) + stopifnot(length(target_paths[[1]]) == L_max + 1L) + + # Per-F_g path assignment: 2 F_g values for paths 1 & 2, 1 for paths + # 3 & 4. This keeps each (D_{g,1}, F_g, S_g) cohort single-path. + f_g_to_path <- c(1L, 1L, 2L, 2L, 3L, 4L) + stopifnot(length(f_g_to_path) == max_switch) + # Group counts per F_g (rank 1 > rank 2 > rank 3 > rank 4 with unique + # ranks — no frequency ties, robust to R's undocumented tiebreak): + # F_g=2: 20 groups (path 1) + # F_g=3: 20 groups (path 1) → rank 1 has 40 switchers total + # F_g=4: 15 groups (path 2) + # F_g=5: 10 groups (path 2) → rank 2 has 25 switchers total + # F_g=6: 10 groups (path 3) → rank 3 has 10 switchers + # F_g=7: 5 groups (path 4) → rank 4 has 5 switchers (excluded by by_path=3) + counts_per_F_g <- c(20L, 20L, 15L, 10L, 10L, 5L) + stopifnot(sum(counts_per_F_g) == n_groups) + + # Build the group-to-(F_g, path) assignment, deterministic with seed + g_idx <- 1L + for (f_idx in seq_along(counts_per_F_g)) { + F_g <- f_idx + 1L # F_g in [2, 7] for f_idx in [1, 6] + path_idx <- f_g_to_path[f_idx] + target <- target_paths[[path_idx]] + n_here <- counts_per_F_g[f_idx] + for (k in seq_len(n_here)) { + g <- g_idx + # Pre-baseline [1 .. F_g-2]: initial state + if (F_g >= 3L) { + D[g, 1:(F_g - 2L)] <- target[1] + } + # Window [F_g-1 .. F_g-1+L_max]: exactly the target path + for (j in 0:L_max) { + D[g, F_g - 1L + j] <- target[j + 1L] + } + # Post-window [F_g+L_max .. n_periods]: stable at path[L_max+1] + if (F_g + L_max <= n_periods) { + D[g, (F_g + L_max):n_periods] <- target[L_max + 1L] + } + g_idx <- g_idx + 1L + } + } } else { stop(sprintf("Unknown pattern: %s", pattern)) } @@ -482,6 +558,81 @@ scenarios$joiners_only_controls_trends_lin <- list( results = extract_dcdh_multi(res12, n_effects = 2, n_placebos = 1) ) +# --------------------------------------------------------------------------- +# Phase 3 extension: by_path per-path event-study disaggregation +# --------------------------------------------------------------------------- + +# Helper: extract per-path by_path results. When did_multiplegt_dyn is +# called with by_path=k, the result object has no $results slot; instead +# per-path results live at res$by_level_1, res$by_level_2, ... in rank +# order (1 = most frequent observed path). res$by_levels is a character +# vector of comma-joined path labels (e.g. "0,1,1,1") in the same order. +extract_dcdh_by_path <- function(res, n_effects) { + by_levels <- res$by_levels + out <- list() + for (i in seq_along(by_levels)) { + slot <- res[[paste0("by_level_", i)]] + effects <- slot$results$Effects + horizons <- list() + for (h in seq_len(min(n_effects, nrow(effects)))) { + horizons[[as.character(h)]] <- list( + effect = as.numeric(effects[h, "Estimate"]), + se = as.numeric(effects[h, "SE"]), + ci_lo = as.numeric(effects[h, "LB CI"]), + ci_hi = as.numeric(effects[h, "UB CI"]), + n_switchers = as.numeric(effects[h, "Switchers"]), + n_obs = as.numeric(effects[h, "N"]) + ) + } + out[[i]] <- list( + path = by_levels[i], + frequency_rank = i, + horizons = horizons + ) + } + list(by_path = out) +} + +# Scenario 13: mixed_single_switch + by_path=2 (basic 2-path case). +# The mixed_single_switch DGP produces joiners (path 0,1,1,1) and +# leavers (path 1,0,0,0) as its only two observed paths at L_max=3, so +# by_path=2 captures both and tests core per-path parity. +cat(" Scenario 13: mixed_single_switch_by_path\n") +d13 <- gen_reversible(n_groups = N_GOLDEN, n_periods = 8, + pattern = "mixed_single_switch", seed = 113) +res13 <- did_multiplegt_dyn( + df = d13, outcome = "outcome", group = "group", time = "period", + treatment = "treatment", effects = 3, by_path = 2, ci_level = 95 +) +scenarios$mixed_single_switch_by_path <- list( + data = export_data(d13), + params = list(pattern = "mixed_single_switch", n_groups = N_GOLDEN, + n_periods = 8, seed = 113, effects = 3, by_path = 2, + ci_level = 95), + results = extract_dcdh_by_path(res13, n_effects = 3) +) + +# Scenario 14: multi_path_reversible + by_path=3 (top-k ranking case). +# The multi_path_reversible DGP produces 3+ distinct observed paths via +# random post-switch toggling. by_path=3 exercises the top-k selection +# when observed paths exceed k. n_periods=10 gives every switch_time a +# complete length-(L_max+1) window. +cat(" Scenario 14: multi_path_reversible_by_path\n") +d14 <- gen_reversible(n_groups = N_GOLDEN, n_periods = 10, + pattern = "multi_path_reversible", seed = 114, + p_switch = 0.35, L_max = 3) +res14 <- did_multiplegt_dyn( + df = d14, outcome = "outcome", group = "group", time = "period", + treatment = "treatment", effects = 3, by_path = 3, ci_level = 95 +) +scenarios$multi_path_reversible_by_path <- list( + data = export_data(d14), + params = list(pattern = "multi_path_reversible", n_groups = N_GOLDEN, + n_periods = 10, seed = 114, effects = 3, by_path = 3, + ci_level = 95, p_switch = 0.35), + results = extract_dcdh_by_path(res14, n_effects = 3) +) + # --------------------------------------------------------------------------- # Write output # --------------------------------------------------------------------------- diff --git a/benchmarks/data/dcdh_dynr_golden_values.json b/benchmarks/data/dcdh_dynr_golden_values.json index e5055728..a3186939 100644 --- a/benchmarks/data/dcdh_dynr_golden_values.json +++ b/benchmarks/data/dcdh_dynr_golden_values.json @@ -564,6 +564,199 @@ } } } + }, + "mixed_single_switch_by_path": { + "data": { + "group": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 30, 30, 30, 30, 30, 30, 30, 30, 31, 31, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32, 32, 32, 33, 33, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 43, 43, 43, 43, 43, 43, 43, 43, 44, 44, 44, 44, 44, 44, 44, 44, 45, 45, 45, 45, 45, 45, 45, 45, 46, 46, 46, 46, 46, 46, 46, 46, 47, 47, 47, 47, 47, 47, 47, 47, 48, 48, 48, 48, 48, 48, 48, 48, 49, 49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50, 50, 50, 50, 50, 51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52, 52, 52, 53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54, 54, 54, 54, 55, 55, 55, 55, 55, 55, 55, 55, 56, 56, 56, 56, 56, 56, 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 58, 58, 58, 58, 58, 58, 59, 59, 59, 59, 59, 59, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 61, 61, 61, 61, 61, 61, 61, 61, 62, 62, 62, 62, 62, 62, 62, 62, 63, 63, 63, 63, 63, 63, 63, 63, 64, 64, 64, 64, 64, 64, 64, 64, 65, 65, 65, 65, 65, 65, 65, 65, 66, 66, 66, 66, 66, 66, 66, 66, 67, 67, 67, 67, 67, 67, 67, 67, 68, 68, 68, 68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 69, 70, 70, 70, 70, 70, 70, 70, 70, 71, 71, 71, 71, 71, 71, 71, 71, 72, 72, 72, 72, 72, 72, 72, 72, 73, 73, 73, 73, 73, 73, 73, 73, 74, 74, 74, 74, 74, 74, 74, 74, 75, 75, 75, 75, 75, 75, 75, 75, 76, 76, 76, 76, 76, 76, 76, 76, 77, 77, 77, 77, 77, 77, 77, 77, 78, 78, 78, 78, 78, 78, 78, 78, 79, 79, 79, 79, 79, 79, 79, 79, 80, 80, 80, 80, 80, 80, 80, 80, 81, 81, 81, 81, 81, 81, 81, 81, 82, 82, 82, 82, 82, 82, 82, 82, 83, 83, 83, 83, 83, 83, 83, 83, 84, 84, 84, 84, 84, 84, 84, 84, 85, 85, 85, 85, 85, 85, 85, 85, 86, 86, 86, 86, 86, 86, 86, 86, 87, 87, 87, 87, 87, 87, 87, 87, 88, 88, 88, 88, 88, 88, 88, 88, 89, 89, 89, 89, 89, 89, 89, 89, 90, 90, 90, 90, 90, 90, 90, 90, 91, 91, 91, 91, 91, 91, 91, 91, 92, 92, 92, 92, 92, 92, 92, 92, 93, 93, 93, 93, 93, 93, 93, 93, 94, 94, 94, 94, 94, 94, 94, 94, 95, 95, 95, 95, 95, 95, 95, 95, 96, 96, 96, 96, 96, 96, 96, 96, 97, 97, 97, 97, 97, 97, 97, 97, 98, 98, 98, 98, 98, 98, 98, 98, 99, 99, 99, 99, 99, 99, 99, 99, 100, 100, 100, 100, 100, 100, 100, 100, 101, 101, 101, 101, 101, 101, 101, 101, 102, 102, 102, 102, 102, 102, 102, 102, 103, 103, 103, 103, 103, 103, 103, 103, 104, 104, 104, 104, 104, 104, 104, 104, 105, 105, 105, 105, 105, 105, 105, 105, 106, 106, 106, 106, 106, 106, 106, 106, 107, 107, 107, 107, 107, 107, 107, 107, 108, 108, 108, 108, 108, 108, 108, 108, 109, 109, 109, 109, 109, 109, 109, 109, 110, 110, 110, 110, 110, 110, 110, 110, 111, 111, 111, 111, 111, 111, 111, 111, 112, 112, 112, 112, 112, 112, 112, 112, 113, 113, 113, 113, 113, 113, 113, 113, 114, 114, 114, 114, 114, 114, 114, 114, 115, 115, 115, 115, 115, 115, 115, 115, 116, 116, 116, 116, 116, 116, 116, 116, 117, 117, 117, 117, 117, 117, 117, 117, 118, 118, 118, 118, 118, 118, 118, 118, 119, 119, 119, 119, 119, 119, 119, 119], + "period": [0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7], + "treatment": [0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + "outcome": [9.7161552081, 11.1802006167, 10.9600037688, 9.7928745607, 11.006079321, 12.5564522915, 12.7509093577, 13.7376208197, 9.3908092301, 9.3195095946, 12.4523838105, 12.2805691318, 13.3651138002, 11.9531462401, 12.5251711463, 11.9572701921, 8.8374050503, 7.90827238, 8.6604443732, 8.1493197048, 8.2578631452, 10.9656964642, 11.2297596107, 11.6919005548, 12.3791537444, 13.7718398651, 15.1488216688, 15.4562446592, 15.6924779796, 16.1414904774, 15.7719224382, 16.3837613367, 10.5096568247, 10.4784020729, 10.5440892258, 13.3558509546, 12.5070954419, 12.3411754156, 12.4306026836, 12.8082240839, 7.2586966093, 10.2174292173, 9.9755490293, 10.1527169102, 10.9091164482, 10.5760153045, 10.9012354786, 10.4151713817, 12.1210529476, 11.9362070207, 12.6407729559, 12.7551186363, 13.0642456273, 12.8357673681, 15.4209321361, 14.6877992987, 12.3427556839, 11.6546031407, 12.5586196846, 12.7334124329, 14.2100092944, 14.4835054715, 13.5962029267, 14.5282762948, 7.6666945772, 8.5210508983, 8.5676714428, 8.8446988108, 7.946565733, 9.6412064965, 10.3317408131, 9.9834021171, 11.1300485074, 10.6635146043, 11.4492130734, 14.0546376733, 13.0290772719, 14.267898747, 13.0959588246, 13.922983144, 8.7076102823, 8.391443521, 9.1179422812, 9.7490579396, 9.7437147455, 9.4297028652, 9.4833402573, 10.6871409627, 11.5174045088, 13.4019493752, 13.7845125404, 13.3687077289, 14.3667990032, 14.5887577143, 14.5979135301, 14.2741103318, 11.9944792383, 10.9680640403, 12.1030125187, 11.79491138, 11.5105074808, 12.1681417727, 11.7113332548, 14.1330831644, 11.7398328051, 13.4102721959, 14.2095864465, 13.1633700421, 14.6022789778, 13.9447271679, 13.3843932154, 14.3362127027, 8.9746668089, 9.5338721516, 8.3418566917, 9.2955363803, 11.7660679099, 10.4592688353, 11.4943757504, 10.2983811789, 7.6939202051, 7.7915885585, 10.5153626447, 10.5291039079, 9.947088045, 10.1617735466, 10.3309931072, 10.1111099713, 13.2114610934, 13.7676530053, 15.9274658227, 14.4859105597, 15.4189062959, 15.4926975485, 15.9153411444, 15.8630197101, 9.2040217263, 8.9558812018, 11.6541454643, 10.9084847069, 11.1868174929, 10.8639620428, 12.1179757996, 11.2752567819, 13.52555839, 12.5270558201, 13.2851611176, 13.8415372022, 13.5156636921, 12.4952603663, 13.4479577716, 15.2967359586, 9.6005383663, 9.2216063135, 8.8107568146, 9.8851273983, 11.8266714097, 10.4841389522, 11.0502824384, 11.6928417686, 11.2329529352, 13.4351923618, 14.2911260145, 14.6648330039, 14.6883309443, 13.3270230483, 13.4653478423, 13.9391648889, 4.6007432591, 4.7452604727, 4.7622488546, 6.4234372727, 6.5521457185, 6.7668320383, 8.1479133261, 8.9228072589, 10.8994781844, 10.3483613272, 9.7744597594, 9.1746015416, 13.1430116665, 12.1065298053, 13.1311987199, 12.3789524844, 11.6648489861, 11.2435644131, 12.6163324242, 13.0103194311, 13.7028276605, 13.2258804362, 13.7717780315, 13.5303061292, 10.5386739072, 12.8117376445, 13.1193080436, 12.3802059304, 13.5844552114, 13.2868724445, 13.7970966062, 12.7914049098, 4.2264771504, 5.3934019122, 4.7107012382, 4.417556889, 5.5893166839, 4.4336275642, 5.3338754553, 6.7119956548, 9.3411557797, 8.3081272469, 9.4095342356, 10.9850373734, 11.1242194326, 11.6328951409, 11.9387144261, 12.0582479207, 13.8587699494, 13.2154939869, 14.0585092746, 13.6050579078, 14.5516610777, 13.8288553776, 16.0405087871, 16.5296166199, 9.6548381473, 11.6846721737, 12.1768981764, 12.478226147, 11.5923946996, 12.2428900733, 12.2823321528, 12.1585508951, 8.8342227449, 8.4517511338, 9.7595259139, 9.1229367794, 9.1411884961, 11.569664542, 11.5002869664, 12.301476095, 6.5235636905, 6.1541387478, 6.7028334868, 7.322010165, 6.6158162386, 7.4479831101, 6.8702822643, 9.7489695813, 8.537571914, 9.0867487901, 9.919831965, 11.219925967, 12.1841599932, 11.8677195826, 11.8342486816, 11.6101853375, 9.517991626, 9.412927436, 10.9216887703, 11.3965454724, 10.471604647, 11.8778716658, 11.8287535079, 12.0408660053, 9.403707133, 9.5640774609, 10.3341526962, 11.9015833685, 10.9911164099, 11.6649425774, 11.7896168202, 12.5791076014, 12.8043659332, 12.9806487757, 15.6085953571, 14.8786485135, 15.2495890983, 15.8629759091, 14.741323768, 15.4019513882, 10.186476278, 10.5259265862, 12.3300719248, 12.1317070332, 13.2137937896, 12.0832773475, 13.3917472422, 12.5103232124, 13.8295161738, 13.2633529137, 13.8819127378, 15.3606143095, 15.7224216174, 16.1522533671, 14.5927747787, 16.6178566719, 10.6160861008, 11.6809217627, 13.9496657532, 13.0119899348, 13.3536160393, 13.5115036473, 14.1261005616, 14.3356969407, 12.9860095239, 14.3772338674, 14.3564416745, 13.8721612389, 13.8654878846, 16.0849537452, 16.2075209873, 16.8084029769, 6.9195746488, 8.0670541861, 9.0902569154, 9.7537142536, 10.554876458, 10.2216906592, 10.3747082866, 10.0367112039, 8.6481959993, 9.2634518382, 7.3883431499, 7.3394706429, 6.458104683, 7.1580794744, 7.2987971685, 6.5558418394, 7.8196853448, 7.2373552097, 6.1259269885, 6.1670357786, 7.5388321601, 6.3806911164, 6.2747425387, 6.8238263703, 10.6957697678, 10.8884625744, 9.0000991379, 9.1090019308, 10.0591592223, 10.1017011963, 9.9161608225, 9.8191353726, 12.8387670895, 13.3307151062, 13.6010701027, 13.4203175077, 13.1291852093, 13.4787560806, 11.7908378178, 11.2370221071, 12.9097565574, 13.0799694616, 12.5231627137, 10.3662414311, 10.7209955162, 10.3350072625, 11.5749973851, 11.6248996021, 12.7869568316, 12.251586705, 12.7057260843, 12.8875985847, 13.3564110335, 12.7630190259, 13.9958455345, 11.8952106546, 13.2776661266, 14.1118622343, 14.5298141687, 13.7243450724, 13.4044871968, 13.8040810644, 13.668514618, 13.3184942893, 11.8664877778, 11.0037970578, 11.6476448764, 11.3431562101, 11.1834192532, 10.2732343465, 10.878115999, 10.7222327055, 10.7788141535, 10.8925823585, 10.8023223182, 10.4187094262, 10.9457263747, 11.2742126539, 10.9908053682, 8.9294361857, 11.370386851, 11.2922819918, 11.9146722642, 10.6788328491, 12.2108402489, 12.1897762351, 9.5333753666, 9.7124684416, 11.9990300032, 12.8119094615, 13.6268523727, 12.6507748422, 10.4152789455, 10.6483795563, 10.7518414208, 10.1051508178, 12.1888526652, 11.4116074872, 10.9347307535, 8.8040446385, 8.6925502689, 9.5939448857, 8.7906889058, 9.4031520159, 7.5789713914, 7.6712462027, 8.59413007, 8.0003381872, 7.2422345201, 6.9389207884, 6.7229537297, 6.5058446677, 12.5660678798, 12.7187291035, 12.0671453075, 11.337695186, 12.0603250788, 11.7271991709, 11.4539256264, 11.5922695933, 8.4143851887, 8.2311182878, 8.8110408439, 8.7261200397, 8.9299982709, 9.0246870442, 7.1441606242, 7.2475345629, 14.3675897712, 13.703289504, 15.301955059, 14.5483965334, 13.0713488092, 13.0742706356, 12.8577930932, 12.0406318873, 10.5522304591, 10.0908297176, 10.1881687508, 10.7253227145, 9.8894052789, 8.8555808442, 9.4245912845, 8.6440993315, 14.2654198389, 13.7534522907, 13.3043735048, 14.1402138198, 13.6919919626, 13.3998875413, 13.743792009, 12.4982336746, 11.2367906425, 8.7512226444, 8.7752643848, 9.1424869406, 8.9908121927, 9.1445141389, 9.3294772435, 10.0339797712, 12.8922704764, 12.5671811695, 9.9156883275, 10.5663975107, 11.2459810511, 11.3619054798, 11.3100362232, 11.8930430999, 10.6830909347, 11.6180154486, 12.2088209702, 11.6319711721, 11.8947961208, 11.3867026614, 10.5050716835, 10.8843981543, 11.9604747612, 11.8327422466, 12.4061135603, 11.2518032646, 12.3057237705, 12.7477517358, 13.5264243571, 10.6413832226, 12.6303046452, 11.845737811, 11.8686124861, 13.3971806351, 13.0827856548, 12.7764299514, 10.6478321452, 10.4273941952, 14.8851445974, 14.0576442407, 14.8697095182, 13.5887642039, 13.304150634, 13.2265613202, 13.7339081159, 12.4757738519, 10.1804455868, 10.5562018459, 10.0615264038, 11.670772106, 10.9528121735, 11.4047635208, 11.506679009, 8.8030565504, 12.9065600988, 13.3132564195, 13.4336709625, 13.5754011927, 12.2461319023, 11.8587353646, 11.5933006052, 11.9069547303, 10.4023413119, 11.1366212092, 10.8472395383, 9.335168118, 9.4418002844, 9.5938302541, 10.123012754, 9.8538626382, 9.8232115759, 11.0861713826, 10.5072372708, 11.5818137987, 11.3407536784, 10.4247572269, 10.1256614385, 9.2387450324, 13.0893321281, 13.5994769534, 13.4607076266, 13.1283566048, 14.251857309, 13.2257748068, 12.2874045988, 11.896311176, 12.7336536207, 12.0083665083, 13.1578709289, 13.3275403702, 10.483417583, 10.7684915033, 11.1276212097, 10.3995480717, 11.4763151217, 13.0643915817, 11.8489983118, 12.0501409275, 12.2310949876, 12.7467131012, 11.6637965513, 10.4864658837, 13.1799412709, 13.3832997355, 12.8302913006, 12.6979349043, 13.3204671194, 13.699286027, 11.23074343, 12.6713650072, 8.7763080282, 7.7926241111, 8.0687697505, 7.3357677318, 6.8989503342, 7.2574142109, 6.4014564235, 8.0075522346, 12.3486559997, 11.4661065194, 10.9211686827, 11.1310026271, 11.9903222646, 10.7193134106, 11.2040947593, 11.5166352083, 13.0418930398, 10.9099495659, 10.816310034, 10.7038372212, 11.5381771456, 10.4683754208, 11.8228320167, 11.7153970694, 12.727840418, 12.1105035416, 11.5509482845, 12.3249754005, 12.3333899726, 10.426425778, 10.7982981565, 10.651751044, 12.0689654305, 13.2994706363, 11.952294334, 11.9528493527, 11.8925365604, 10.6234784291, 10.3783250708, 10.8593340414, 6.6456270238, 6.7390244868, 7.9285275076, 5.8870835709, 5.4258757201, 5.2377073104, 6.4873084929, 5.3518273686, 10.5612846465, 10.2529499462, 10.681491902, 10.4900770288, 10.6996718642, 8.943191401, 9.6548417667, 8.8951133881, 12.5648632858, 12.8481930988, 12.0357637912, 13.4481347054, 11.2933153039, 11.0737781153, 10.8476331876, 11.9434222244, 11.5496300535, 10.9854210266, 11.7480976722, 11.8030576762, 11.480287056, 11.4874631457, 12.7292064815, 12.1098546743, 7.5797923032, 8.0674637539, 7.4403695119, 7.7830586746, 7.7548148862, 7.3682139549, 8.5344725174, 7.5014453677, 9.676355355, 10.053148519, 9.5543658359, 9.6457316782, 10.4102290881, 9.4918269564, 9.9797442048, 11.0824766616, 6.1428300303, 6.3549847841, 6.4331561165, 7.1198805647, 7.5189660486, 6.6008235656, 7.2172225051, 8.5631250867, 9.2782188553, 8.3267165064, 10.2075148078, 9.4193144718, 9.5951433211, 9.5304745644, 10.6310193095, 9.5770319375, 5.0120172879, 5.8025076882, 5.2068915803, 5.8359569535, 5.6049660879, 5.7621413748, 5.2177392625, 6.3336102912, 7.2899731489, 8.4461591842, 8.6306972422, 7.9854717091, 8.4494034419, 8.8563482522, 8.8600082191, 9.6273950636, 7.2857573593, 7.6818475121, 7.9635421356, 6.8693662405, 9.4215673416, 8.0053392946, 7.7812601445, 8.2424032521, 12.1567105418, 11.9557560233, 12.7300185951, 13.988903814, 11.9995268049, 12.7791785013, 12.1990282277, 11.7981143416, 8.7560375816, 8.4069671081, 8.7001361206, 8.7575399627, 8.2385272386, 9.0910128252, 8.5688276743, 10.3248127897, 7.105042882, 7.5020900496, 6.5934397603, 7.1178551395, 7.8554271747, 6.6558758765, 7.0952484242, 7.0828285254, 8.835689708, 8.811993966, 10.4558443698, 9.5513776491, 9.2008376776, 9.1357290024, 8.6032942581, 9.7597673803, 6.9780077891, 7.6695645878, 7.7183849217, 8.0191563003, 8.1198896885, 7.7523871981, 7.6875200866, 7.7950031908, 8.2436833113, 8.9264688165, 8.6090370138, 8.1149153292, 8.1893322132, 8.3246526583, 8.8514735586, 8.7729234912, 9.1040227861, 9.9993231652, 8.9632234476, 9.2766031817, 8.9819593506, 9.4742876839, 9.4403685118, 9.713062652, 10.3860366805, 10.2240557123, 10.175735775, 10.3343564676, 10.4287776313, 10.9929890348, 10.3169841881, 10.0934269523, 10.2503562397, 11.6355509491, 12.0892984779, 11.1423795429, 11.7138602356, 10.7636301039, 12.2321294579, 11.6939697191, 11.4728812335, 10.6818146315, 11.2606339414, 11.0781892077, 9.890529068, 11.2246665662, 12.1292191299, 12.1827907115, 9.4860221894, 8.8253495635, 8.9572705937, 9.4560147156, 10.2497804552, 9.3778390648, 9.3141823288, 9.1952273204, 9.9872986981, 9.9321361269, 10.6215187196, 10.7563316729, 10.1378451718, 9.7166793943, 10.6783959373, 10.3675368592, 11.7915489344, 12.8125478784, 12.3169395825, 12.8620634768, 13.1862266354, 13.0082788711, 11.9668983997, 13.3691527424, 11.3917420743, 11.294497699, 11.5300938633, 10.992488505, 12.0487141484, 11.8416636557, 11.3789649218, 11.9746939461, 12.4092028441, 12.7718030745, 12.7714820936, 12.4567644761, 13.4599496511, 12.8873707042, 13.464197222, 13.1400103519, 10.9864409916, 10.9788181424, 11.3423241483, 10.8792568865, 10.8066281508, 11.2415457774, 11.1401032124, 11.6731082134, 10.4215785178, 10.5908212694, 9.4907378387, 11.1758231368, 11.0432678925, 10.3985169061, 11.0089350221, 10.8184982308, 12.0921139188, 11.6876909826, 12.6024262466, 12.0934021557, 12.0521335525, 11.9176282801, 13.0623332171, 12.3088658161, 13.9802491936, 12.4903755491, 13.5469037302, 13.6792890411, 13.6483989938, 13.3096217611, 14.2139618235, 13.772693719, 12.0792140098, 13.1330757849, 13.5573107825, 13.1236156476, 12.6782278098, 13.6078265493, 12.9111185923, 13.7258403183, 13.661009659, 12.922980366, 13.1142790406, 13.3321439591, 13.46982003, 13.8745622345, 13.0363589841, 14.6976999626, 14.9972778687, 16.1149713908, 15.9159727428, 16.0578077329, 15.5804020445, 16.0228987702, 15.8561246577, 15.8819214049, 10.6096521736, 10.9895361926, 11.6337122025, 10.8850326593, 12.0853323106, 11.7422274931, 11.1771993545, 11.1865207251, 15.8786912697, 16.7239938471, 16.9389438401, 16.2905304279, 15.9726827144, 15.5944019117, 15.7736511199, 16.3462461911, 11.3241798439, 10.6669080094, 10.7943954336, 11.2195739298, 10.6813770998, 10.9172244872, 11.6876554336, 12.1636319612, 12.3016718507, 12.8483336934, 12.4832550247, 13.4436977872, 13.0350687904, 12.0223363615, 12.8201953876, 14.1056479724, 10.0845679634, 9.8626879285, 10.2288229542, 10.7137270367, 10.3534561426, 10.1033382217, 10.350776315, 10.085186052, 13.3568657608, 13.6369570434, 13.7885751443, 13.4633475786, 13.6806067155, 12.8402828181, 14.2111853069, 13.574349992, 9.4139221591, 9.37873306, 10.2708998325, 9.596999306, 10.4007125153, 9.8817599656, 10.5900183893, 10.5304483674, 10.4723998781, 10.3518552951, 10.4205771139, 11.411488461, 11.2706373703, 11.0895258501, 11.3980480587, 11.5168940855, 11.8699923025, 13.1130323883, 13.7588319575, 13.0587150173, 13.7142234105, 14.0638619163, 13.4972439001, 13.9374389372, 13.3546086259, 13.7946024439, 13.8627380037, 14.916882817, 14.2959636032, 12.9976542384, 14.1538810421, 14.1992195797] + }, + "params": { + "pattern": "mixed_single_switch", + "n_groups": 80, + "n_periods": 8, + "seed": 113, + "effects": 3, + "by_path": 2, + "ci_level": 95 + }, + "results": { + "by_path": [ + { + "path": "0,1,1,1", + "frequency_rank": 1, + "horizons": { + "1": { + "effect": 1.8966017044, + "se": 0.15650747403, + "ci_lo": 1.589852692, + "ci_hi": 2.2033507168, + "n_switchers": 33, + "n_obs": 224 + }, + "2": { + "effect": 1.8381359536, + "se": 0.1248139863, + "ci_lo": 1.5935050357, + "ci_hi": 2.0827668715, + "n_switchers": 33, + "n_obs": 195 + }, + "3": { + "effect": 2.0655058753, + "se": 0.15107206298, + "ci_lo": 1.7694100728, + "ci_hi": 2.3616016778, + "n_switchers": 33, + "n_obs": 174 + } + } + }, + { + "path": "1,0,0,0", + "frequency_rank": 2, + "horizons": { + "1": { + "effect": 1.7094470289, + "se": 0.13438277264, + "ci_lo": 1.4460616344, + "ci_hi": 1.9728324234, + "n_switchers": 26, + "n_obs": 247 + }, + "2": { + "effect": 1.7615433882, + "se": 0.13891022732, + "ci_lo": 1.4892843455, + "ci_hi": 2.0338024308, + "n_switchers": 26, + "n_obs": 219 + }, + "3": { + "effect": 1.7478188062, + "se": 0.16267239825, + "ci_lo": 1.4289867644, + "ci_hi": 2.0666508481, + "n_switchers": 26, + "n_obs": 188 + } + } + } + ] + } + }, + "multi_path_reversible_by_path": { + "data": { + "group": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34, 34, 34, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 87, 87, 87, 87, 87, 87, 87, 87, 87, 87, 88, 88, 88, 88, 88, 88, 88, 88, 88, 88, 89, 89, 89, 89, 89, 89, 89, 89, 89, 89, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 91, 91, 91, 91, 91, 91, 91, 91, 91, 91, 92, 92, 92, 92, 92, 92, 92, 92, 92, 92, 93, 93, 93, 93, 93, 93, 93, 93, 93, 93, 94, 94, 94, 94, 94, 94, 94, 94, 94, 94, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 96, 96, 96, 96, 96, 96, 96, 96, 96, 96, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 102, 102, 102, 102, 102, 102, 102, 102, 102, 102, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 104, 104, 104, 104, 104, 104, 104, 104, 104, 104, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 106, 106, 106, 106, 106, 106, 106, 106, 106, 106, 107, 107, 107, 107, 107, 107, 107, 107, 107, 107, 108, 108, 108, 108, 108, 108, 108, 108, 108, 108, 109, 109, 109, 109, 109, 109, 109, 109, 109, 109, 110, 110, 110, 110, 110, 110, 110, 110, 110, 110, 111, 111, 111, 111, 111, 111, 111, 111, 111, 111, 112, 112, 112, 112, 112, 112, 112, 112, 112, 112, 113, 113, 113, 113, 113, 113, 113, 113, 113, 113, 114, 114, 114, 114, 114, 114, 114, 114, 114, 114, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 117, 117, 117, 117, 117, 117, 117, 117, 117, 117, 118, 118, 118, 118, 118, 118, 118, 118, 118, 118, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119], + "period": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], + "treatment": [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + "outcome": [9.8354210993, 12.0806822108, 12.7965331738, 12.7372392534, 13.126969971, 13.7156790946, 12.5648910991, 13.8399596958, 13.5480004228, 12.8973645822, 12.5518878186, 14.0544484841, 14.1530875179, 14.178357207, 14.8117190805, 14.6950049456, 14.629784452, 14.900433686, 14.5864681917, 14.9130676246, 8.6658410611, 11.1832690731, 11.8603032584, 10.9146836577, 11.3072081056, 11.0714699541, 11.809162478, 10.8427095182, 13.0716607261, 12.2744613191, 15.2298586052, 17.1289828374, 17.4528686316, 16.1086124233, 18.1115264329, 17.9740531622, 17.6586146549, 17.605457509, 17.6276005091, 17.7314739213, 10.701947838, 12.3116987832, 13.1510430191, 11.8810531259, 12.7515796228, 13.2845711001, 13.0658311134, 12.5852874024, 12.1998779604, 13.4585922385, 11.54063161, 12.7455347273, 12.7462829277, 13.4849429519, 12.8579857414, 13.8561649598, 13.8874497937, 13.6791137199, 13.1819724091, 12.5279312983, 8.6390307549, 10.8783417176, 10.0957981986, 11.0958765006, 10.2486283725, 11.341799121, 11.8003648668, 12.1038733092, 11.1425283794, 11.0885561554, 8.2437101981, 10.6601308436, 10.7245965593, 11.195753082, 10.5906766121, 10.8381456872, 12.2637852302, 11.287441121, 11.0785190376, 11.4204204144, 6.9259624285, 8.1488074043, 8.401202155, 8.8230937133, 8.8782944678, 8.9389038879, 8.4803428531, 8.2841422713, 9.3341328219, 9.1311334371, 10.7575203156, 12.83297154, 13.267334581, 14.054053455, 13.8082528014, 13.8230441768, 13.4237515957, 13.8211709907, 13.9981137259, 13.5445940561, 10.6461767235, 11.4943256652, 12.7740724038, 13.6825196384, 12.7544005264, 13.0956996355, 12.5997333402, 13.6385000504, 13.1788921364, 13.8014839853, 9.6335881247, 11.9093465678, 11.9310369925, 12.2165499395, 12.6188271966, 12.5570393445, 13.1121029798, 12.8258269228, 12.9296138498, 12.3567918841, 5.4096788068, 7.2274487756, 7.1760724943, 7.5317953183, 7.8044973365, 8.8293195758, 7.813074066, 7.3737908478, 7.6820952517, 7.9267570054, 8.7788356208, 10.2194123051, 11.1888782986, 10.9928255679, 10.1110534692, 10.2235324468, 11.1398730593, 10.4992264419, 11.583404993, 11.203208343, 12.9800416009, 15.1067399313, 14.7823822921, 14.7525549678, 14.5626727272, 14.3945539923, 15.0475323, 15.0439493658, 15.7752220625, 15.7574801422, 11.9190194698, 13.2472891342, 13.1205354253, 13.7611774784, 14.2307399992, 14.7078795402, 14.844674152, 13.9847454906, 13.7822485634, 15.6694927606, 10.4722131094, 11.0018750182, 12.1171175773, 12.5742464849, 12.6487480314, 11.2706909989, 12.7950630429, 12.9741468744, 12.6148330308, 13.1099715689, 9.4204273133, 11.3970661505, 11.5721125999, 11.375139908, 12.5776151854, 10.598895125, 11.616196248, 11.5646437552, 13.2982880345, 11.3358193562, 8.90337567, 10.8329846721, 10.6744538654, 10.4823739121, 11.7198600189, 12.0584180594, 11.1910936512, 11.5746595098, 12.1044611298, 11.3313102415, 8.8963515448, 9.7190263129, 11.4413678054, 11.4677453024, 10.4935101637, 10.938496104, 12.0385934922, 11.4785656306, 11.3234499051, 12.1214740913, 10.3638874261, 10.2168318588, 12.1990615619, 13.1561755774, 12.8122965911, 11.9310211959, 13.169545956, 13.0683907075, 12.4045989988, 13.2543595145, 8.6987572717, 9.1780255598, 10.9440788829, 10.8327976276, 11.2320594806, 10.8590990763, 10.6353697337, 11.2617207082, 11.367264286, 11.3860863828, 9.9649099048, 10.5392474244, 12.2877155567, 11.4901093102, 12.2223725404, 12.3073493707, 13.1918633578, 12.860628616, 12.7300554787, 13.5635714393, 10.906015828, 9.7336465733, 11.9516251222, 12.5898877738, 12.4609108367, 11.8825069826, 12.2273677212, 12.5970285214, 14.0229090131, 12.811366401, 14.8549718789, 13.6182217089, 15.7368419616, 15.6743430131, 14.7456863385, 17.5602374498, 16.1919563, 17.075873175, 16.9764065407, 18.6948733644, 10.4358754506, 10.1452010233, 13.1745655203, 12.5644010744, 12.674668887, 12.7633560323, 13.4168012582, 13.0435688037, 13.2413674052, 13.3823996145, 11.3404427888, 12.3778346803, 13.7847234005, 14.3514229415, 14.7214294525, 14.2130322746, 15.3329722256, 14.9425588237, 14.7429195496, 13.9266144326, 6.1152908808, 5.243813566, 7.357314103, 7.3164510488, 7.5348721813, 8.1573487876, 7.6569834096, 8.0167648073, 8.4405289699, 7.5358320716, 8.2504296152, 7.6447659522, 9.1400592136, 9.6361729119, 10.1451185682, 9.9323318476, 10.1519527845, 10.3729843789, 10.6217064095, 9.869362625, 10.0805026334, 10.1345963756, 13.2107598409, 11.7733965869, 11.7637185274, 12.932118721, 12.8466186873, 13.453663001, 13.1405968432, 13.3750994203, 9.1484915627, 9.2913535222, 11.1558859441, 11.5430261504, 11.9939249555, 12.0679247278, 11.4158989902, 12.4930298745, 12.0775883852, 12.640359491, 11.2650606528, 11.2866396194, 13.5392716974, 12.4342681029, 13.9876161379, 13.3216469024, 14.3216947101, 14.5903864609, 14.416206949, 14.2616664372, 8.5267027033, 8.1066725006, 11.8175716, 11.101942502, 11.6416439089, 11.5790854804, 11.485104838, 10.9395597317, 11.64305561, 12.1596722064, 9.4491705333, 9.868304176, 11.4610149599, 12.2915654152, 12.371685549, 12.2241749103, 11.98747107, 12.3886056073, 12.9955635201, 12.5151944775, 13.8904552884, 13.060872202, 15.2577155655, 15.6630050416, 16.3644236831, 16.1110288069, 15.4284866694, 16.6026216833, 16.0567803881, 15.694351601, 9.9306824262, 9.7172863509, 13.167320194, 12.7867239175, 13.4800922943, 13.661817392, 13.8265493646, 12.451779465, 12.315028571, 12.6294073659, 9.7758962139, 9.651029351, 10.9459810953, 11.5507799549, 11.9285306116, 11.2065023805, 11.4875077842, 11.7952373363, 12.7238057953, 12.5565875218, 7.4785200908, 9.3235028255, 10.2870847891, 10.16116006, 9.1869055811, 9.9206675903, 9.8700398088, 11.4835743131, 10.1213498441, 11.1228486342, 12.6487693284, 13.1919337454, 14.0552922289, 13.9648140821, 14.9235339789, 15.1681128009, 15.1060779033, 15.7312116709, 15.0735274358, 16.0735951849, 9.586634456, 10.6615311352, 11.6695168934, 11.5723167838, 12.1343396608, 12.4108155723, 12.7518341401, 12.7223863537, 12.6309365816, 12.9884978898, 9.7454756146, 10.6107862524, 9.7298484595, 13.2737466772, 13.463006172, 10.1020388969, 10.4259772525, 11.2851815495, 11.5023626118, 11.4941722435, 7.9118336587, 8.532874951, 9.110553386, 11.2811755441, 11.0508701695, 8.4355782251, 9.8917672468, 9.6199414022, 8.5665169879, 9.3415179703, 8.2380889066, 8.5077951272, 7.0183772727, 11.2243679395, 10.3185103765, 8.7049508646, 8.4160528111, 9.8377981455, 10.2794960978, 9.7305318605, 7.76868793, 7.7918470024, 8.4476544654, 10.3769370043, 11.261310187, 8.7902012211, 9.3077766698, 9.5127505322, 9.5660146669, 9.12300928, 9.786899601, 8.9601192938, 9.3236359423, 11.0317448221, 11.7742570352, 8.9687465475, 9.7410558798, 10.4205761103, 10.1504068157, 9.3445645934, 10.3021305878, 10.2142722977, 10.8320905749, 12.0359007612, 11.8069927656, 9.2772475005, 11.6037297833, 10.5945152047, 9.641904168, 11.4864443519, 10.0626276297, 8.7551526469, 9.2014826895, 12.9427316477, 12.1335416007, 10.1048280788, 9.883553045, 11.2195431892, 10.7570465691, 11.1162038697, 14.2728814678, 14.1798863291, 13.971252126, 17.0431722926, 16.0246157685, 14.2456375931, 13.7611754334, 14.3955750303, 15.1780508876, 14.3336062798, 12.794925815, 12.3234326, 11.7535809131, 15.3993285971, 16.0938827191, 12.8829818372, 13.3604629303, 13.6039581458, 13.5921486912, 13.351703891, 12.4813239423, 11.2035362924, 12.8164476369, 14.5258335472, 14.8587380928, 12.469810632, 12.3942351835, 13.3645304345, 13.2036741783, 13.8124313878, 8.222523421, 9.6845182724, 10.2081156291, 11.710756972, 12.2379945541, 9.8389861123, 9.7156867681, 10.9607392337, 9.4857518365, 10.3080897618, 12.1498114696, 11.1025692312, 12.0647030703, 14.7220271268, 14.8557049244, 13.4703633909, 13.2930889711, 13.6245004742, 13.4095737504, 12.8825546345, 12.9987960655, 12.4559570269, 11.5579759996, 15.0431443816, 15.2126142697, 13.2881007483, 11.772154701, 13.5215637909, 12.7857252356, 12.9055729741, 12.8141149766, 11.6409851772, 12.4455741888, 14.8436724825, 15.4535988549, 11.9608621218, 13.2435681829, 12.9298441866, 13.6121231833, 12.2593911926, 10.3843612022, 10.0275014576, 10.3804943034, 12.6992044814, 12.9417610206, 11.1196151344, 11.1626968395, 10.9342362379, 10.5926949163, 10.244638648, 10.4030413489, 10.1003126247, 11.3259411023, 11.5327932877, 12.8215194858, 12.1793118181, 11.4776740694, 10.6965381351, 11.7885493617, 11.3553052933, 9.3053630859, 9.6066810553, 9.197147542, 9.0919691197, 12.073211928, 10.934490018, 9.8174814523, 8.7327768169, 9.8477980234, 9.4096010459, 8.2884196123, 7.9999565762, 7.5363448692, 9.0895463138, 11.0660814861, 10.9498622831, 8.8587031172, 9.3853870511, 9.219315007, 8.1405225262, 6.5576464308, 7.3652049852, 6.3293965745, 7.2133814406, 8.4568538096, 9.0761238151, 8.4633836541, 6.3774654537, 7.815125999, 7.6448997663, 11.7463876253, 12.2531617528, 11.9017657423, 11.593441665, 14.2005488605, 13.3099330551, 12.6483022939, 12.7699873532, 13.1591154988, 13.1250127394, 8.2110079371, 8.1955540649, 8.4465612094, 7.3807904372, 9.7560319043, 10.7554139504, 8.666619514, 8.4430945366, 9.0803597652, 9.1668822573, 11.4257864046, 11.1401460193, 12.2791159719, 11.3275515843, 14.2137760603, 14.0864482844, 12.1702071464, 11.9030287937, 11.79127211, 10.7822666792, 12.0514708576, 11.2397181222, 11.6496865753, 12.0699646887, 14.3801875002, 14.4552930798, 11.8886497138, 12.4451378863, 12.8446409542, 12.831722639, 14.8038078891, 14.7375374494, 14.0775308144, 15.4628414901, 16.9115219547, 17.4194781326, 15.1412816673, 15.247114005, 14.7830343433, 14.7201068243, 8.3727700714, 8.4571339418, 8.3878803928, 8.3260961244, 11.7219444862, 9.6719306123, 9.6249089565, 9.7797263174, 8.7235450169, 9.6032545175, 10.921425431, 10.1111538065, 10.891361561, 10.3091857665, 11.218790958, 13.5092795745, 10.600993651, 11.8488228562, 10.8714660285, 11.756075363, 11.4361820469, 11.9164807832, 12.1898557583, 12.0383588111, 12.699677102, 13.983899351, 12.3608310745, 12.3934234627, 12.5273676727, 13.3876886061, 12.2279013951, 12.2081790951, 12.784025939, 11.2246226336, 11.8878233387, 14.5023870925, 11.8412949219, 12.3559195031, 11.7040696236, 12.8594750022, 8.3845329869, 8.0452046853, 8.1567737118, 7.7964931208, 8.780834895, 10.6396379381, 8.3658418028, 9.7192670099, 7.9756068685, 8.3212986186, 10.8035690609, 10.9981030602, 11.8930160164, 11.1589972701, 11.0361672888, 13.8547852878, 11.4197194158, 11.6459452341, 12.110938239, 10.9411596455, 12.0253838182, 13.5424504345, 12.2249342278, 11.6757712868, 12.7113344729, 15.2337832661, 12.1854784859, 13.4545903968, 12.6678937483, 12.9879335256, 13.5671020817, 12.0204116365, 13.8768521526, 13.503628243, 12.1879095443, 15.3308166199, 13.2401583793, 12.999475737, 13.2258245946, 12.6071498695, 8.2905567763, 9.194255239, 8.4116738676, 8.9815471824, 8.2830465587, 10.6103791299, 8.9738845211, 9.7365395368, 9.5608498786, 9.6800702267, 10.8623173404, 11.174576026, 10.9476370955, 10.6033687274, 11.4797384872, 13.3903370962, 11.5909901608, 12.0396315599, 11.9712152256, 11.3377623772, 14.317475048, 13.4737345771, 13.8141225046, 14.3689317341, 14.5550712786, 16.373793072, 14.9629782515, 14.629496745, 14.7778617004, 16.1470409771, 9.4257726724, 9.8479479325, 9.1886054836, 9.1083372619, 9.0684956348, 10.2655688441, 11.7282297191, 11.5029709575, 10.8227217234, 11.7680619936, 8.6019160359, 8.356060754, 9.1015909307, 8.5266191525, 9.3959057748, 9.1553823257, 11.1127282072, 10.0220963273, 11.1696307173, 10.9830898535, 10.823292466, 11.910973436, 10.4735271472, 11.6373240701, 11.6784231387, 10.5540489043, 13.2352726522, 12.3380762807, 14.0075441891, 13.9509265344, 8.9006290905, 9.6399605374, 10.2749219082, 10.3985941495, 9.5264508775, 10.378886325, 11.4591737083, 10.4394536637, 11.9442782604, 12.747633476, 8.0953435018, 8.7524658752, 6.9277947438, 7.8016760595, 7.0719854544, 7.2403577588, 9.816865347, 8.4620129416, 10.0824821499, 10.1233692341, 11.5505630718, 11.8372360492, 11.9313071277, 11.4986566405, 11.6219397369, 11.7201660272, 12.0328703649, 13.1598259174, 12.151476613, 12.681906451, 7.6162797198, 6.8098291532, 7.1360447766, 7.6015952471, 7.4137539989, 7.6442995435, 7.5344155775, 7.70436533, 8.0615963875, 8.0676320734, 11.1935878826, 11.3098930301, 10.7732738778, 12.4070568574, 11.8726259619, 12.3284719578, 11.7626038351, 12.3317680921, 12.1257887187, 12.0167601026, 12.0027494117, 11.7158033693, 12.2754990511, 11.6169543564, 12.2256309599, 12.5072862561, 13.5077042356, 12.6878157704, 11.5804416055, 12.8575389519, 10.6788323305, 10.1979921591, 11.1003840411, 10.757623584, 11.8188528735, 11.0819994308, 12.09152934, 11.4911110199, 11.3002939844, 11.4620841219, 9.8856355529, 9.9620775447, 9.8933515761, 9.9936192952, 9.5610990183, 10.4226367658, 10.9583478773, 10.1377426744, 11.0165155649, 11.4714546663, 7.3578991752, 7.9831955077, 7.6256116801, 7.9629795365, 8.2947633578, 7.685654897, 8.2810953124, 7.7154389106, 8.2256053603, 8.5288003137, 8.5892922942, 8.3474585414, 9.5069493426, 8.4620659789, 8.5982039101, 10.1176660416, 8.6088449435, 9.3879123861, 9.7555376982, 9.2088330473, 12.5338368348, 12.5143978701, 13.8935221211, 13.5703964987, 12.7344953451, 12.2588536495, 13.9702423234, 13.4584677444, 14.0682875165, 13.2131243752, 6.4376510607, 6.0455320548, 6.3956923599, 5.7039710978, 6.5605174965, 6.6970674195, 6.5672600961, 6.2881222344, 7.2641050751, 7.2477565083, 10.5822705874, 10.3250991332, 10.0049368712, 10.2254910803, 10.9881943397, 11.5956704094, 10.8067362653, 11.1212539409, 12.3883563517, 10.6886674522, 7.3132051912, 7.3318042377, 7.6874272438, 7.6965488577, 8.5162880437, 8.2274852087, 8.341924296, 8.1197253457, 8.3354926555, 8.0791471397, 11.1104775654, 11.2672385994, 11.6084317975, 11.2985194734, 11.8677170427, 12.2736412408, 12.1478842008, 12.0252957877, 11.6987303716, 12.0901704479, 9.0949116246, 9.0057459805, 9.2818641082, 8.6238422464, 10.0918372569, 9.3620845548, 9.7523669456, 9.9321780388, 10.6263274075, 9.5047765273, 8.4984539002, 7.7893698392, 9.0530419034, 8.3962152265, 8.1664109048, 9.6586373279, 9.9306597611, 8.5551379892, 8.4737727116, 9.313462362, 5.7293269226, 6.2473971762, 5.9237867153, 6.1322151744, 5.6662551148, 7.110153003, 6.5010155893, 6.631901997, 7.4745884916, 7.3529392422, 13.4714268431, 13.9895281446, 14.0265007141, 14.6360923891, 14.3859295631, 13.1857865539, 15.199365294, 13.7761371633, 14.4484812509, 14.8064526911, 13.1836650266, 12.5451388966, 12.6623399839, 13.8766344974, 13.0443801346, 14.0600455475, 13.5721630722, 13.347312577, 13.4836219081, 12.9909603139, 7.7243402917, 7.271659473, 7.0404498069, 7.37738704, 8.544569771, 8.2821179397, 9.1066572394, 8.004322788, 8.7607487913, 8.2343384288, 9.4655801199, 9.0787644938, 8.8043556044, 9.7944629452, 9.776766949, 10.3181233686, 10.0614353634, 9.4523057826, 10.1125091534, 10.9899074432, 8.3201178959, 9.6029645211, 8.2298681425, 8.7534092092, 9.3794087929, 9.4739828924, 9.0750337261, 9.621456032, 9.8258887365, 10.3866044978, 9.9531457829, 10.452152672, 10.3677757112, 11.410208438, 10.1698015744, 9.3720008366, 10.5054436833, 12.0314639424, 11.0210986235, 10.9490921174, 10.8450246717, 11.7509423159, 11.1769608087, 11.2786117395, 11.090561311, 12.1768555215, 11.3909899411, 12.3207460187, 12.1188634443, 13.1909649293, 12.1874899544, 13.0462154093, 12.8241427371, 12.7586023022, 13.3612391736, 12.5412492747, 13.3611518265, 12.5952173411, 13.111113636, 12.6938423433, 13.6263196993, 13.4565128775, 13.2182851663, 12.4008893832, 13.220023502, 12.5556436248, 13.1588321331, 12.9255427478, 12.8860529681, 14.0601676034, 13.1905528783, 13.868258604, 13.3944241572, 13.2340717819, 14.3714871837, 13.6169868696, 14.1057692802, 14.4487141452, 13.7176583574, 14.8813642806, 17.8260890845, 18.1289972708, 18.8999817963, 18.6873517732, 18.7763064068, 19.6063100928, 18.4024320811, 19.1178370418, 18.7660294627, 19.143962921, 12.001782294, 12.086869413, 12.0227115734, 12.387670962, 12.6674714555, 13.4581334697, 13.1109547034, 12.5310741066, 12.9404328034, 13.9648532084, 12.0836690726, 12.8804706224, 11.2848708831, 12.0969800743, 12.3748590112, 12.1494101781, 12.8661917278, 13.172505969, 12.7819255098, 12.2642574458, 11.2470754101, 11.5554447328, 11.3339371194, 10.6401672942, 10.6195652733, 11.0891605218, 10.5898605615, 10.8840534202, 11.8212268836, 11.8696868155, 11.4502047367, 11.4986840218, 11.111816971, 11.7400220333, 11.2104750791, 12.2829517836, 11.6487659793, 11.2893234023, 13.3367851406, 12.2079599326, 13.4586665282, 14.266106423, 13.5407569732, 13.6794145809, 15.0375504037, 13.8574197827, 14.5379829431, 13.6469624983, 14.7184230062, 15.3484553921, 11.4409540052, 11.9424718075, 11.5036119504, 12.6216592473, 11.9409572524, 11.9126316372, 13.1138970991, 13.0732245863, 12.5964343419, 12.6685824279, 8.4959204258, 9.2249454874, 10.2192163783, 9.6162787335, 10.0276903447, 9.0621151486, 9.9103265055, 9.3317387492, 10.1295943519, 10.6140940207, 11.8961083725, 11.6628363609, 11.2386713801, 11.2884623684, 12.7282563204, 12.0943642242, 11.9819283863, 11.9571063231, 12.3541656875, 11.714276324, 13.9607316001, 13.9722248266, 14.4537752168, 14.6027598815, 14.2622286354, 14.1695238293, 14.2407802156, 13.9472782055, 14.3601905126, 14.489298268, 11.9689522611, 10.5822337271, 12.0295602583, 12.1765809778, 11.8797244781, 11.2913084859, 11.9447359601, 11.9436308126, 12.1575836383, 12.5240693858, 10.6956983226, 10.8257172282, 10.9896370508, 11.1483326086, 11.2341085974, 11.8482064481, 12.0529801634, 11.2288797849, 12.3815778436, 12.5453338875, 12.3403356652, 13.4867714769, 12.1957212402, 12.6290277932, 12.8697885367, 12.4024833426, 12.8278874397, 13.6410009454, 13.3545187468, 13.6105621875, 11.6596840759, 11.1857787256, 10.7827165954, 10.6429209245, 10.7585514848, 10.9599683394, 11.8682870865, 11.2726987696, 11.6459320278, 11.9858587049] + }, + "params": { + "pattern": "multi_path_reversible", + "n_groups": 80, + "n_periods": 10, + "seed": 114, + "effects": 3, + "by_path": 3, + "ci_level": 95, + "p_switch": 0.35 + }, + "results": { + "by_path": [ + { + "path": "0,1,1,1", + "frequency_rank": 1, + "horizons": { + "1": { + "effect": 1.8114539175, + "se": 0.12764687185, + "ci_lo": 1.561270646, + "ci_hi": 2.0616371891, + "n_switchers": 40, + "n_obs": 180 + }, + "2": { + "effect": 1.9188094304, + "se": 0.12567851933, + "ci_lo": 1.6724840589, + "ci_hi": 2.1651348019, + "n_switchers": 40, + "n_obs": 145 + }, + "3": { + "effect": 1.9949137818, + "se": 0.14028924146, + "ci_lo": 1.7199519212, + "ci_hi": 2.2698756425, + "n_switchers": 40, + "n_obs": 120 + } + } + }, + { + "path": "0,1,1,0", + "frequency_rank": 2, + "horizons": { + "1": { + "effect": 2.3846181783, + "se": 0.19653307228, + "ci_lo": 1.9994204348, + "ci_hi": 2.7698159217, + "n_switchers": 25, + "n_obs": 105 + }, + "2": { + "effect": 2.175435323, + "se": 0.18913679101, + "ci_lo": 1.8047340244, + "ci_hi": 2.5461366215, + "n_switchers": 25, + "n_obs": 85 + }, + "3": { + "effect": -0.12035710021, + "se": 0.19731196128, + "ci_lo": -0.50708143804, + "ci_hi": 0.26636723763, + "n_switchers": 25, + "n_obs": 70 + } + } + }, + { + "path": "0,1,0,0", + "frequency_rank": 3, + "horizons": { + "1": { + "effect": 2.0332465047, + "se": 0.23535763505, + "ci_lo": 1.5719540165, + "ci_hi": 2.4945389929, + "n_switchers": 10, + "n_obs": 35 + }, + "2": { + "effect": -0.37906683362, + "se": 0.22135191364, + "ci_lo": -0.81290861225, + "ci_hi": 0.054774945019, + "n_switchers": 10, + "n_obs": 30 + }, + "3": { + "effect": 0.41937632622, + "se": 0.19837696072, + "ci_lo": 0.030564627837, + "ci_hi": 0.8081880246, + "n_switchers": 10, + "n_obs": 30 + } + } + } + ] + } } }, "generator": "generate_reversible_did_data v1", diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 3d7d535b..44da7543 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -638,7 +638,7 @@ The guard is fired by `_survey_se_from_group_if` (analytical and replicate) and - **Note (Phase 3 Design-2 switch-in/switch-out):** Convenience wrapper for Web Appendix Section 1.6 (Assumption 16). Identifies groups with exactly 2 treatment changes (join then leave), reports switch-in and switch-out mean effects. This is a descriptive summary, not a full re-estimation with specialized control pools as described in the paper. **Always uses raw (unadjusted) outcomes** regardless of active `controls`, `trends_linear`, or `trends_nonparam` options - those adjustments apply to the main estimator surface but not to the Design-2 descriptive block. For full adjusted Design-2 estimation with proper control pools, the paper recommends "running the command on a restricted subsample and using `trends_nonparam` for the entry-timing grouping." Activated via `design2=True` in `fit()`, requires `drop_larger_lower=False` to retain 2-switch groups. -- **Note (Phase 3 `by_path` per-path event-study disaggregation):** Per-path disaggregation of the multi-horizon event study, mirroring R `did_multiplegt_dyn(..., by_path=k)`. Activated via `ChaisemartinDHaultfoeuille(by_path=k, drop_larger_lower=False)` where `k` is a positive integer (top-k most common observed paths by switcher-group frequency). **Window convention:** the path tuple for a switcher group `g` is `(D_{g, F_g-1}, D_{g, F_g}, ..., D_{g, F_g-1+L_max})` — length `L_max + 1`, matching R's window `[F_{g-1}, F_{g-1+l}]`. **Ranking:** paths are ranked by descending frequency; ties are broken lexicographically on the path tuple for deterministic ordering, so every selected path has a unique `frequency_rank`. If `by_path` exceeds the number of observed paths, all observed paths are returned with a `UserWarning`. **Per-path SE convention (joiners/leavers precedent):** the per-path influence function follows the joiners-only / leavers-only IF construction at `chaisemartin_dhaultfoeuille.py:5495-5504`: the switcher-side contribution `+S_g * (Y_{g,out} - Y_{g,ref})` is zeroed for groups whose observed trajectory is NOT the selected path; control contributions and the full cohort structure `(D_{g,1}, F_g, S_g)` are unchanged. After applying the singleton-baseline eligible mask and cohort-recentering with the original cohort IDs, the plug-in SE uses the path-specific divisor `N_l_path` (count of path switchers eligible at horizon `l`) — same pattern as `joiners_se` using `joiner_total`. This gives the **within-path mean** estimand `DID_{path,l}` as the within-path average of `DID_{g,l}`. **Degenerate-cohort behavior per path:** when a path's centered IF at some horizon is identically zero (every variance-eligible path switcher forms its own `(D_{g,1}, F_g, S_g)` cohort, or the path has a single contributing group), SE / t_stat / p_value / conf_int are NaN-consistent and a `UserWarning` is emitted scoped to `(path, horizon)`. This mirrors the overall-path degenerate-cohort surface and is common for rare paths with few contributing groups. **Empty-state contract:** `results.path_effects` distinguishes "not requested" (`None`) from "requested but empty" (`{}` — all switchers have windows outside the panel or unobserved cells). The empty-dict case emits a `UserWarning` at fit-time and renders as an explicit "no observed paths" notice in `summary()`; `to_dataframe(level="by_path")` returns an empty DataFrame with the canonical column set (mirrors the `linear_trends` pattern when `trends_linear=True` but no horizons survive). **Requirements:** `drop_larger_lower=False` (multi-switch groups are the object of interest; default `True` filters them out) and `L_max >= 1` (path window depends on the horizon). **Scope (initial release):** binary treatment only; combinations with `controls`, `trends_linear`, `trends_nonparam`, `heterogeneity`, `design2`, `honest_did`, `survey_design`, and `n_bootstrap > 0` are gated behind explicit `NotImplementedError` for a strict first PR. **Placebos and TWFE diagnostic** remain sample-level summaries (not computed per path) in this release. Results are exposed on `results.path_effects` as `Dict[Tuple[int, ...], Dict[str, Any]]` with nested `horizons` dicts per horizon `l`, and on `results.to_dataframe(level="by_path")` as a long-format table with columns `[path, frequency_rank, n_groups, horizon, effect, se, t_stat, p_value, conf_int_lower, conf_int_upper, n_obs]`. Gated tests live in `tests/test_chaisemartin_dhaultfoeuille.py::TestByPathGates` / `::TestByPathBehavior` / `::TestByPathEdgeCases`. **R-parity** tests are deferred to a follow-up PR (the SE convention against R's `did_multiplegt_dyn` is to be confirmed in the parity work). +- **Note (Phase 3 `by_path` per-path event-study disaggregation):** Per-path disaggregation of the multi-horizon event study, mirroring R `did_multiplegt_dyn(..., by_path=k)`. Activated via `ChaisemartinDHaultfoeuille(by_path=k, drop_larger_lower=False)` where `k` is a positive integer (top-k most common observed paths by switcher-group frequency). **Window convention:** the path tuple for a switcher group `g` is `(D_{g, F_g-1}, D_{g, F_g}, ..., D_{g, F_g-1+L_max})` — length `L_max + 1`, matching R's window `[F_{g-1}, F_{g-1+l}]`. **Ranking:** paths are ranked by descending frequency; ties are broken lexicographically on the path tuple for deterministic ordering, so every selected path has a unique `frequency_rank`. If `by_path` exceeds the number of observed paths, all observed paths are returned with a `UserWarning`. **Per-path SE convention (joiners/leavers precedent):** the per-path influence function follows the joiners-only / leavers-only IF construction at `chaisemartin_dhaultfoeuille.py:5495-5504`: the switcher-side contribution `+S_g * (Y_{g,out} - Y_{g,ref})` is zeroed for groups whose observed trajectory is NOT the selected path; control contributions and the full cohort structure `(D_{g,1}, F_g, S_g)` are unchanged. After applying the singleton-baseline eligible mask and cohort-recentering with the original cohort IDs, the plug-in SE uses the path-specific divisor `N_l_path` (count of path switchers eligible at horizon `l`) — same pattern as `joiners_se` using `joiner_total`. This gives the **within-path mean** estimand `DID_{path,l}` as the within-path average of `DID_{g,l}`. **Degenerate-cohort behavior per path:** when a path's centered IF at some horizon is identically zero (every variance-eligible path switcher forms its own `(D_{g,1}, F_g, S_g)` cohort, or the path has a single contributing group), SE / t_stat / p_value / conf_int are NaN-consistent and a `UserWarning` is emitted scoped to `(path, horizon)`. This mirrors the overall-path degenerate-cohort surface and is common for rare paths with few contributing groups. **Empty-state contract:** `results.path_effects` distinguishes "not requested" (`None`) from "requested but empty" (`{}` — all switchers have windows outside the panel or unobserved cells). The empty-dict case emits a `UserWarning` at fit-time and renders as an explicit "no observed paths" notice in `summary()`; `to_dataframe(level="by_path")` returns an empty DataFrame with the canonical column set (mirrors the `linear_trends` pattern when `trends_linear=True` but no horizons survive). **Requirements:** `drop_larger_lower=False` (multi-switch groups are the object of interest; default `True` filters them out) and `L_max >= 1` (path window depends on the horizon). **Scope (initial release):** binary treatment only; combinations with `controls`, `trends_linear`, `trends_nonparam`, `heterogeneity`, `design2`, `honest_did`, `survey_design`, and `n_bootstrap > 0` are gated behind explicit `NotImplementedError` for a strict first PR. **Placebos and TWFE diagnostic** remain sample-level summaries (not computed per path) in this release. Results are exposed on `results.path_effects` as `Dict[Tuple[int, ...], Dict[str, Any]]` with nested `horizons` dicts per horizon `l`, and on `results.to_dataframe(level="by_path")` as a long-format table with columns `[path, frequency_rank, n_groups, horizon, effect, se, t_stat, p_value, conf_int_lower, conf_int_upper, n_obs]`. Gated tests live in `tests/test_chaisemartin_dhaultfoeuille.py::TestByPathGates` / `::TestByPathBehavior` / `::TestByPathEdgeCases`. **R-parity** against `DIDmultiplegtDYN 2.3.3` is confirmed at `tests/test_chaisemartin_dhaultfoeuille_parity.py::TestDCDHDynRParityByPath` via two scenarios: `mixed_single_switch_by_path` (2 paths, `by_path=2`) and `multi_path_reversible_by_path` (4 paths, `by_path=3`; path-assignment deterministic on `F_g` so each `(D_{g,1}, F_g, S_g)` cohort contains switchers from a single path). Per-path point estimates and per-path switcher counts match R exactly; per-path SE matches within the Phase 2 multi-horizon SE envelope (observed rtol ≤ 10.2% on the 2-path mixed scenario, ≤ 4.2% on the 4-path cohort-clean scenario). **Deviation from R (cross-path cohort-sharing SE):** our analytical SE is the marginal variance of the path-contribution estimator cohort-centered on the *full-panel* cohort structure (joiners/leavers precedent — non-path switchers contribute to cohort means via their zeroed switcher row). R's `did_multiplegt_dyn(..., by_path=k)` re-runs the estimator per path, so cohort means are computed over the path's own switchers only. When a cohort `(D_{g,1}, F_g, S_g)` spans multiple observed paths, Python and R SE diverge materially (our empirical probes with random post-window toggling saw rtol > 100%); when every cohort is single-path (scenario 13 by design, scenario 14 by construction), the two approaches coincide up to the documented Phase 2 envelope. Practitioners with cohort structures that mix paths should interpret the per-path SE as a within-full-panel marginal variance, not a per-path conditional variance. **Reference implementation(s):** - R: [`DIDmultiplegtDYN`](https://cran.r-project.org/package=DIDmultiplegtDYN) (CRAN, maintained by the paper authors). The Python implementation matches `did_multiplegt_dyn(..., effects=1)` at horizon `l = 1`. Parity tests live in `tests/test_chaisemartin_dhaultfoeuille_parity.py`. diff --git a/tests/test_chaisemartin_dhaultfoeuille_parity.py b/tests/test_chaisemartin_dhaultfoeuille_parity.py index 4fc7767d..b7f9588c 100644 --- a/tests/test_chaisemartin_dhaultfoeuille_parity.py +++ b/tests/test_chaisemartin_dhaultfoeuille_parity.py @@ -464,3 +464,132 @@ def test_parity_joiners_only_controls_trends_lin(self, golden_values): controls=["X1"], trends_linear=True, point_rtol=self.POINT_RTOL, ) + + +# --------------------------------------------------------------------------- +# Phase 3 extension: by_path per-path event-study disaggregation parity +# --------------------------------------------------------------------------- + + +class TestDCDHDynRParityByPath: + """ + Parity tests for ``by_path`` against R DIDmultiplegtDYN. + + R's ``did_multiplegt_dyn(..., by_path=k)`` returns ``k`` per-path + result slots (``res$by_level_1``, ``res$by_level_2``, ...) keyed by + frequency rank with path labels in ``res$by_levels``. The generator + script captures these under ``results.by_path`` as an ordered list. + + Paths are matched by **tuple label via set equality**, not by + frequency rank: R's tied-frequency tiebreak convention is + undocumented, while Python's is lexicographic. Set-equality catches + silent divergence where Python and R select non-identical top-k + path sets under equal frequencies, and the symmetric-difference + diagnostic in the assertion message makes the failure actionable. + + Per-path switcher counts are hard-asserted **before** SE + comparison so that window-eligibility divergences surface + explicitly rather than as apples-to-oranges SE gaps. + """ + + # Point-estimate tolerances mirror the Phase 1/2 parity classes. + POINT_RTOL = 1e-4 + MIXED_POINT_RTOL = 0.025 + # SE tolerance: 12% sits between Phase 2 multi-horizon SE_RTOL + # (0.10 for short panels) and the long-panel widening (0.15). Per- + # path slices have ~10-40 switchers - comparable to Phase 2's long- + # panel pure-direction slices where the cell-count vs obs-count + # weighting envelope (documented in REGISTRY.md) widens from 0.10 + # to 0.15 as horizon length compounds the small-sample correction. + SE_RTOL = 0.12 + + def _path_key_from_r_label(self, r_label: str): + return tuple(int(x) for x in r_label.split(",")) + + def _compare_by_path(self, scenario, by_path, L_max, point_rtol, se_rtol): + import warnings + + df = _golden_to_df(scenario["data"]) + est = ChaisemartinDHaultfoeuille(drop_larger_lower=False, by_path=by_path) + # Suppress the per-(path, horizon) degenerate-cohort UserWarning + # emitted by the foundation when a path subset produces an + # identically-zero centered IF. + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UserWarning) + results = est.fit( + df, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=L_max, + ) + + r_by_path = scenario["results"]["by_path"] + assert results.path_effects is not None + + py_keys = set(results.path_effects.keys()) + r_keys = {self._path_key_from_r_label(e["path"]) for e in r_by_path} + assert py_keys == r_keys, ( + f"Path-set mismatch between Python and R.\n" + f" Python only: {py_keys - r_keys}\n" + f" R only: {r_keys - py_keys}" + ) + + for r_path_entry in r_by_path: + path_key = self._path_key_from_r_label(r_path_entry["path"]) + py_path = results.path_effects[path_key] + for h_str, r_h in r_path_entry["horizons"].items(): + h = int(h_str) + assert ( + h in py_path["horizons"] + ), f"path={path_key}: horizon {h} missing from Python path_effects" + py_h = py_path["horizons"][h] + + # Per-path switcher count sanity check BEFORE SE comparison: + # if R and Python disagree on which groups land in this + # path's window-eligible set at horizon h, SE gaps become + # apples-to-oranges and point-estimate gaps are masked. + assert py_h["n_obs"] == int(r_h["n_switchers"]), ( + f"path={path_key} h={h}: switcher-count mismatch " + f"py={py_h['n_obs']} vs r={int(r_h['n_switchers'])} " + f"- window-eligibility divergence; investigate before " + f"comparing SE." + ) + + assert py_h["effect"] == pytest.approx(r_h["effect"], rel=point_rtol), ( + f"path={path_key} h={h}: " + f"py={py_h['effect']:.4f} vs r={r_h['effect']:.4f}" + ) + + if py_h["se"] > 0 and r_h["se"] > 0: + assert py_h["se"] == pytest.approx(r_h["se"], rel=se_rtol), ( + f"path={path_key} h={h} SE: " + f"py={py_h['se']:.4f} vs r={r_h['se']:.4f}" + ) + + def test_parity_mixed_single_switch_by_path(self, golden_values): + """2-path basic case: mixed_single_switch at L_max=3, by_path=2.""" + scenario = golden_values.get("mixed_single_switch_by_path") + if scenario is None: + pytest.skip("scenario 'mixed_single_switch_by_path' not in golden values") + self._compare_by_path( + scenario, + by_path=2, + L_max=3, + point_rtol=self.MIXED_POINT_RTOL, + se_rtol=self.SE_RTOL, + ) + + def test_parity_multi_path_reversible_by_path(self, golden_values): + """Top-k ranking case: 4 observed paths, by_path=3 selects top-3.""" + scenario = golden_values.get("multi_path_reversible_by_path") + if scenario is None: + pytest.skip("scenario 'multi_path_reversible_by_path' not in golden values") + self._compare_by_path( + scenario, + by_path=3, + L_max=3, + point_rtol=self.MIXED_POINT_RTOL, + se_rtol=self.SE_RTOL, + ) From c9dfc8223bcb846a6535b87e711f4ce1c0bcc528 Mon Sep 17 00:00:00 2001 From: igerber Date: Fri, 24 Apr 2026 13:33:37 -0400 Subject: [PATCH 2/3] Address PR #360 AI review: tighten POINT_RTOL, assert SE state, pin R version Three findings from CI review round 2 (1 P3 + 2 P2): - P2: `POINT_RTOL` tightened from 0.025 (MIXED_POINT_RTOL) to 1e-9. Observed rtol on all 15 (path, horizon) cells is ~1e-11 across both scenarios, consistent with R's 10-digit JSON rounding. The prior 2.5% tolerance would silently accept ~6 orders of magnitude of point-estimate regression while the docs claim exact R parity. - P2: `_compare_by_path()` now asserts matching finite/positive state on `py_se` vs `r_se` BEFORE the `pytest.approx` numeric check. Previously the `if py_se > 0 and r_se > 0:` guard silently skipped SE parity when one side degraded to 0/NaN/inf while the other stayed finite; the regression now fails explicitly with a diagnostic citing both values and suggesting a variance-identifiability check. - P3: `DIDmultiplegtDYN` version pin moved from `>= 2.3.3` to `== 2.3.3` so a future release with changed `by_path` slot semantics cannot silently regenerate the fixture. Comment documents the update protocol: bump the pin AND re-run the parity class when moving to a newer known-compatible version, or extend to an explicit allowlist once a second version is verified. Tests still pass (2/2 in TestDCDHDynRParityByPath); ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- benchmarks/R/generate_dcdh_dynr_test_values.R | 13 +++++-- ...test_chaisemartin_dhaultfoeuille_parity.py | 39 +++++++++++++++---- 2 files changed, 40 insertions(+), 12 deletions(-) diff --git a/benchmarks/R/generate_dcdh_dynr_test_values.R b/benchmarks/R/generate_dcdh_dynr_test_values.R index 3da74e8d..58775255 100644 --- a/benchmarks/R/generate_dcdh_dynr_test_values.R +++ b/benchmarks/R/generate_dcdh_dynr_test_values.R @@ -29,10 +29,15 @@ library(DIDmultiplegtDYN) library(jsonlite) suppressMessages(library(polars)) # required by DIDmultiplegtDYN >= 2.x -# Pin DIDmultiplegtDYN min version because the by_path output slots -# (res$by_levels, res$by_level_i) were introduced in v2.3.3 and the -# structure is not version-stable per the R package's own docs. -stopifnot(packageVersion("DIDmultiplegtDYN") >= "2.3.3") +# Pin DIDmultiplegtDYN to an exact version because the `by_path` output +# slots (res$by_levels, res$by_level_i) were introduced in v2.3.3 and +# the structure is not version-stable per the R package's own docs. A +# floor constraint (`>= 2.3.3`) could silently drift the fixture schema +# when regenerated against a future release. Update this pin *and* re- +# run TestDCDHDynRParityByPath when bumping to a newer known-compatible +# release; extend to an explicit allowlist (e.g. `%in% c("2.3.3", +# "2.3.4")`) once a second version is verified. +stopifnot(packageVersion("DIDmultiplegtDYN") == "2.3.3") cat("Generating dCDH golden values via DIDmultiplegtDYN at l=1...\n") diff --git a/tests/test_chaisemartin_dhaultfoeuille_parity.py b/tests/test_chaisemartin_dhaultfoeuille_parity.py index b7f9588c..f643bb24 100644 --- a/tests/test_chaisemartin_dhaultfoeuille_parity.py +++ b/tests/test_chaisemartin_dhaultfoeuille_parity.py @@ -492,9 +492,15 @@ class TestDCDHDynRParityByPath: explicitly rather than as apples-to-oranges SE gaps. """ - # Point-estimate tolerances mirror the Phase 1/2 parity classes. - POINT_RTOL = 1e-4 - MIXED_POINT_RTOL = 0.025 + # Point-estimate tolerance: 1e-9 locks in the observed exact-match + # property on the committed goldens (measured rtol ~1e-11 across + # all (path, horizon) cells in both scenarios, well inside R's + # 10-digit JSON rounding envelope). This matches the claim in + # REGISTRY.md and CHANGELOG.md that per-path point estimates agree + # with R exactly. Any material regression on point parity will + # trip this; loosening to 2.5% would silently accept a ~6 orders + # of magnitude regression. + POINT_RTOL = 1e-9 # SE tolerance: 12% sits between Phase 2 multi-horizon SE_RTOL # (0.10 for short panels) and the long-panel widening (0.15). Per- # path slices have ~10-40 switchers - comparable to Phase 2's long- @@ -562,10 +568,27 @@ def _compare_by_path(self, scenario, by_path, L_max, point_rtol, se_rtol): f"py={py_h['effect']:.4f} vs r={r_h['effect']:.4f}" ) - if py_h["se"] > 0 and r_h["se"] > 0: - assert py_h["se"] == pytest.approx(r_h["se"], rel=se_rtol), ( + # Assert matching finite/missing state BEFORE the numeric + # tolerance check so that a silent regression to 0/NaN on + # one side (while R stays finite, or vice versa) fails the + # test rather than skipping this cell. + import math + + py_se = py_h["se"] + r_se = r_h["se"] + py_finite_positive = math.isfinite(py_se) and py_se > 0.0 + r_finite_positive = math.isfinite(r_se) and r_se > 0.0 + assert py_finite_positive == r_finite_positive, ( + f"path={path_key} h={h} SE state mismatch " + f"(py_se={py_se}, r_se={r_se}): one side is " + f"finite+positive while the other is 0/NaN/inf. " + f"Verify the path subset produces matching " + f"variance-identifiability on both sides." + ) + if py_finite_positive and r_finite_positive: + assert py_se == pytest.approx(r_se, rel=se_rtol), ( f"path={path_key} h={h} SE: " - f"py={py_h['se']:.4f} vs r={r_h['se']:.4f}" + f"py={py_se:.4f} vs r={r_se:.4f}" ) def test_parity_mixed_single_switch_by_path(self, golden_values): @@ -577,7 +600,7 @@ def test_parity_mixed_single_switch_by_path(self, golden_values): scenario, by_path=2, L_max=3, - point_rtol=self.MIXED_POINT_RTOL, + point_rtol=self.POINT_RTOL, se_rtol=self.SE_RTOL, ) @@ -590,6 +613,6 @@ def test_parity_multi_path_reversible_by_path(self, golden_values): scenario, by_path=3, L_max=3, - point_rtol=self.MIXED_POINT_RTOL, + point_rtol=self.POINT_RTOL, se_rtol=self.SE_RTOL, ) From daefda78e347896ba56562f30935315bac3c5c95 Mon Sep 17 00:00:00 2001 From: igerber Date: Fri, 24 Apr 2026 13:49:20 -0400 Subject: [PATCH 3/3] Address PR #360 AI review round 3: frequency_rank assert + scenario 14 doc cleanup Two findings (1 P2 + 1 P3): - P2: `_compare_by_path` now asserts `py_path["frequency_rank"] == r_path_entry["frequency_rank"]` for every committed path. Both scenarios are constructed with unique path frequencies (scenario 13 via the mixed_single_switch pattern, scenario 14 via deterministic counts 40/25/10/5), so rank ordering is unambiguous and any regression in top-k tiebreak handling now fails explicitly instead of passing silently as long as the selected path set and per-path effects remain correct. - P3: scenario 14 generator docstring and recorded params still described the old stochastic `p_switch`-driven DGP (the pre-PR variant that blew out SE parity via cross-path cohort mixing). The `multi_path_reversible` pattern is now DETERMINISTIC: path assignment is a fixed function of F_g with counts 20/20/15/10/10/5 across the 6 F_g values. `p_switch = 0.35` dropped from both the scenario call and the `params` block in the fixture; comment block rewritten to describe the deterministic design and cite the REGISTRY note for the rationale behind the design choice. Fixture regenerated; scenario 14 params no longer carry the stale `p_switch` entry. Point and SE parity numbers unchanged (deterministic DGP produces the same treatment matrix as before). Tests pass (2/2); ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- benchmarks/R/generate_dcdh_dynr_test_values.R | 18 ++++++++++++------ benchmarks/data/dcdh_dynr_golden_values.json | 3 +-- .../test_chaisemartin_dhaultfoeuille_parity.py | 14 ++++++++++++++ 3 files changed, 27 insertions(+), 8 deletions(-) diff --git a/benchmarks/R/generate_dcdh_dynr_test_values.R b/benchmarks/R/generate_dcdh_dynr_test_values.R index 58775255..692992f4 100644 --- a/benchmarks/R/generate_dcdh_dynr_test_values.R +++ b/benchmarks/R/generate_dcdh_dynr_test_values.R @@ -618,14 +618,20 @@ scenarios$mixed_single_switch_by_path <- list( ) # Scenario 14: multi_path_reversible + by_path=3 (top-k ranking case). -# The multi_path_reversible DGP produces 3+ distinct observed paths via -# random post-switch toggling. by_path=3 exercises the top-k selection -# when observed paths exceed k. n_periods=10 gives every switch_time a -# complete length-(L_max+1) window. +# The `multi_path_reversible` pattern is a DETERMINISTIC multi-path DGP: +# path assignment is a fixed function of F_g (so every (D_{g,1}, F_g, +# S_g) cohort contains switchers from a single path), path proportions +# are fixed at 20/20/15/10/10/5 across the 6 F_g values, and +# post-window treatment is stable at path[L_max+1]. by_path=3 exercises +# top-k selection when observed paths exceed k (4 observed paths, top-3 +# selected). n_periods=10 gives every switch_time a complete length- +# (L_max+1) window. The old `p_switch`-driven random-toggle variant +# (pre-PR) blew out SE parity with R via cross-path cohort mixing; +# see the REGISTRY.md `Note (Phase 3 by_path ...)` Deviation bullet. cat(" Scenario 14: multi_path_reversible_by_path\n") d14 <- gen_reversible(n_groups = N_GOLDEN, n_periods = 10, pattern = "multi_path_reversible", seed = 114, - p_switch = 0.35, L_max = 3) + L_max = 3) res14 <- did_multiplegt_dyn( df = d14, outcome = "outcome", group = "group", time = "period", treatment = "treatment", effects = 3, by_path = 3, ci_level = 95 @@ -634,7 +640,7 @@ scenarios$multi_path_reversible_by_path <- list( data = export_data(d14), params = list(pattern = "multi_path_reversible", n_groups = N_GOLDEN, n_periods = 10, seed = 114, effects = 3, by_path = 3, - ci_level = 95, p_switch = 0.35), + ci_level = 95), results = extract_dcdh_by_path(res14, n_effects = 3) ) diff --git a/benchmarks/data/dcdh_dynr_golden_values.json b/benchmarks/data/dcdh_dynr_golden_values.json index a3186939..ccdf9281 100644 --- a/benchmarks/data/dcdh_dynr_golden_values.json +++ b/benchmarks/data/dcdh_dynr_golden_values.json @@ -660,8 +660,7 @@ "seed": 114, "effects": 3, "by_path": 3, - "ci_level": 95, - "p_switch": 0.35 + "ci_level": 95 }, "results": { "by_path": [ diff --git a/tests/test_chaisemartin_dhaultfoeuille_parity.py b/tests/test_chaisemartin_dhaultfoeuille_parity.py index f643bb24..47d27332 100644 --- a/tests/test_chaisemartin_dhaultfoeuille_parity.py +++ b/tests/test_chaisemartin_dhaultfoeuille_parity.py @@ -545,6 +545,20 @@ def _compare_by_path(self, scenario, by_path, L_max, point_rtol, se_rtol): for r_path_entry in r_by_path: path_key = self._path_key_from_r_label(r_path_entry["path"]) py_path = results.path_effects[path_key] + + # Assert the public frequency_rank contract matches R. Both + # committed scenarios are constructed with unique path + # frequencies (scenario 13 via mixed_single_switch pattern, + # scenario 14 via deterministic counts 40/25/10/5) so rank + # ordering is unambiguous and must agree; a regression in + # path ranking or top-k tiebreak handling should fail here + # even if the selected path set and per-path effects remain + # correct. + assert py_path["frequency_rank"] == r_path_entry["frequency_rank"], ( + f"path={path_key}: frequency_rank mismatch " + f"py={py_path['frequency_rank']} vs r={r_path_entry['frequency_rank']}" + ) + for h_str, r_h in r_path_entry["horizons"].items(): h = int(h_str) assert (