From e9acb1c93ace375cbb166e05179107e58f922bc0 Mon Sep 17 00:00:00 2001 From: Michael Walshe <62556482+michaelwalshe@users.noreply.github.com> Date: Tue, 10 Mar 2026 10:04:46 +0000 Subject: [PATCH 01/13] Change references to SAS engine to remove braces, make non-executable --- Comp/r-east_gsd_tte.qmd | 4 +-- Comp/r-sas-python_survey-stats-summary.qmd | 4 +-- Comp/r-sas-summary-stats.qmd | 2 +- Comp/r-sas_anova.qmd | 4 +-- Comp/r-sas_chi-sq.qmd | 2 +- Comp/r-sas_friedman.qmd | 2 +- Comp/r-sas_gee.qmd | 4 +-- Comp/r-sas_kruskalwallis.qmd | 2 +- Comp/r-sas_logistic-regr.qmd | 8 +++--- Comp/r-sas_mcnemar.qmd | 2 +- Comp/r-sas_mmrm.qmd | 18 ++++++------- Comp/r-sas_negbin.qmd | 2 +- Comp/r-sas_psmatch.qmd | 2 +- Comp/r-sas_survival.qmd | 8 +++--- SAS/SAS_Friedmantest.qmd | 4 +-- SAS/ancova.qmd | 4 +-- SAS/anova.qmd | 6 ++--- SAS/association.qmd | 4 +-- SAS/binomial_test.qmd | 18 ++++++------- SAS/ci_for_2indep_prop.qmd | 6 ++--- SAS/ci_for_paired_prop.qmd | 6 ++--- SAS/ci_for_prop.qmd | 10 ++++---- SAS/cmh.qmd | 16 ++++++------ SAS/correlation.qmd | 6 ++--- SAS/count_data_regression.qmd | 4 +-- SAS/gee.qmd | 8 +++--- SAS/gsd-tte.qmd | 2 +- SAS/jonchkheere_terpstra.qmd | 12 ++++----- SAS/kruskal_wallis.qmd | 4 +-- SAS/linear-regression.qmd | 8 +++--- SAS/logistic-regr.qmd | 30 +++++++++++----------- SAS/manova.qmd | 2 +- SAS/mcnemar.qmd | 2 +- SAS/mi_mar_regression.qmd | 10 ++++---- SAS/mmrm.qmd | 2 +- SAS/nparestimate.qmd | 4 +-- SAS/ranksum.qmd | 6 ++--- SAS/rbmi_continuous_joint_SAS.qmd | 22 ++++++++-------- SAS/recurrent_events.qmd | 22 ++++++++-------- SAS/rmst.qmd | 12 ++++----- SAS/sample_s_StatXact_test_of_trends.qmd | 6 ++--- SAS/sample_s_equivalence.qmd | 8 +++--- SAS/sample_s_noninferiority.qmd | 4 +-- SAS/sample_s_superiority.qmd | 4 +-- SAS/summary-stats.qmd | 2 +- SAS/summary_skew_kurt.qmd | 6 ++--- SAS/survey-stats-summary.qmd | 14 +++++----- SAS/survival.qmd | 8 +++--- SAS/survival_cif.qmd | 4 +-- SAS/survival_csh.qmd | 6 ++--- SAS/tipping_point.qmd | 18 ++++++------- SAS/tobit regression SAS.qmd | 8 +++--- SAS/ttest_1Sample.qmd | 6 ++--- SAS/ttest_2Sample.qmd | 8 +++--- SAS/ttest_Paired.qmd | 8 +++--- SAS/wilcoxonsr_HL.qmd | 12 ++++----- templates/multi_language_template.qmd | 6 ++--- templates/single_language_template.qmd | 2 +- 58 files changed, 212 insertions(+), 212 deletions(-) diff --git a/Comp/r-east_gsd_tte.qmd b/Comp/r-east_gsd_tte.qmd index 04b693882..c63f18357 100644 --- a/Comp/r-east_gsd_tte.qmd +++ b/Comp/r-east_gsd_tte.qmd @@ -548,7 +548,7 @@ pfs_rpact_sum |> - SAS code to reproduce the above rpact results appears below. -```{sas} +```sas PROC SEQDESIGN BOUNDARYSCALE=MLE ERRSPEND; DESIGN NSTAGES=2 INFO=CUM(0.748936170212766 1.0) @@ -585,7 +585,7 @@ Please note that the `BOUNDARYSCALE=MLE | SCORE | STDZ | PVALUE` options display SAS doesn't provide a boundary information with HR, so the HR boundaries is obtained from the MLE boundaries (as MLE $=\hat{\theta}=-log(\text{HR})$, see [SAS User's Guide: Test for Two Survival Distributions with a Log-Rank Test](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/statug/statug_seqdesign_details52.htm#statug.seqdesign.cseqdlogrank)) via the following code. -```{sas} +```sas DATA BHR; SET BMLE; Bound_UA_HR=exp(-Bound_UA); diff --git a/Comp/r-sas-python_survey-stats-summary.qmd b/Comp/r-sas-python_survey-stats-summary.qmd index 192b684d6..f1b5c5411 100644 --- a/Comp/r-sas-python_survey-stats-summary.qmd +++ b/Comp/r-sas-python_survey-stats-summary.qmd @@ -125,7 +125,7 @@ print(list( ## SAS -```{sas} +```sas * Mean, sum quantile of HI_CHOL; proc surveymeans data=nhanes mean sum clm quantile=(0.025 0.5 0.975); cluster SDMVPSU; @@ -377,7 +377,7 @@ In SAS, PROC SURVEYMEANS will calculate quantiles of specific probabilities as y The method and results from SAS are as follows: -```{sas} +```sas proc surveymeans data=apisrs total=6194 quantile=(0.025 0.5 0.975); var growth; run; diff --git a/Comp/r-sas-summary-stats.qmd b/Comp/r-sas-summary-stats.qmd index ebb2aa500..82c6bd5fc 100644 --- a/Comp/r-sas-summary-stats.qmd +++ b/Comp/r-sas-summary-stats.qmd @@ -15,7 +15,7 @@ c(10, 20, 30, 40, 150, 160, 170, 180, 190, 200) Assuming the data above is stored in the variable `aval` within the dataset `adlb`, the 25th and 40th percentiles could be calculated using the following code. -```{sas} +```sas proc univariate data=adlb; var aval; output out=stats pctlpts=25 40 pctlpre=p; diff --git a/Comp/r-sas_anova.qmd b/Comp/r-sas_anova.qmd index 599b52d95..71519e61c 100644 --- a/Comp/r-sas_anova.qmd +++ b/Comp/r-sas_anova.qmd @@ -56,7 +56,7 @@ lm_model |> In SAS, all contrasts must be manually defined, but the syntax is largely similar in both. -```{sas} +```sas proc glm data=work.mycsv; class drug; model post = pre drug / solution; @@ -119,7 +119,7 @@ Provided below is a detailed comparison of the results obtained from both SAS an Note, however, that there are some cases where the scale of the parameter estimates between SAS and R is off, though the test statistics and p-values are identical. In these cases, we can adjust the SAS code to include a divisor. As far as we can tell, this difference only occurs when using the predefined Base R contrast methods like `contr.helmert`. -```{sas} +```sas proc glm data=work.mycsv; class drug; model post = pre drug / solution; diff --git a/Comp/r-sas_chi-sq.qmd b/Comp/r-sas_chi-sq.qmd index 5a161b1e5..a9915a4b7 100644 --- a/Comp/r-sas_chi-sq.qmd +++ b/Comp/r-sas_chi-sq.qmd @@ -42,7 +42,7 @@ stats::chisq.test(bronch) To run a chi-squared test in SAS you used the following code. -```{sas} +```sas proc freq data=proj1.bronchitis; tables Cough*Bronchitis / chisq; run; diff --git a/Comp/r-sas_friedman.qmd b/Comp/r-sas_friedman.qmd index c995cab25..2cd2d0edc 100644 --- a/Comp/r-sas_friedman.qmd +++ b/Comp/r-sas_friedman.qmd @@ -52,7 +52,7 @@ res.fried In SAS, **CMH2** option of PROC FREQ is used to perform Friedman's test. -```{sas} +```sas proc freq data=data_bp; tables patient*dos*bp / cmh2 scores=rank noprint; diff --git a/Comp/r-sas_gee.qmd b/Comp/r-sas_gee.qmd index 4ee708f05..524800261 100644 --- a/Comp/r-sas_gee.qmd +++ b/Comp/r-sas_gee.qmd @@ -55,7 +55,7 @@ SAS results were stored in a dataset and reformatted to align with R output in t ### Ordinal variable - Example -```{sas} +```sas proc genmod data=resp ; class trtp(ref="P") avisitn(ref='1') usubjid ; model respord=trtp avisitn trtp*avisitn / dist=multinomial link=cumlogit ; @@ -104,7 +104,7 @@ summary(model) ### Nominal variable - Example -```{sas} +```sas proc gee data=resp ; class trtp(ref="P") avisitn(ref='1') usubjid respnom(ref='Lung'); model respnom=trtp avisitn trtp*avisitn/ dist=multinomial link=glogit ; diff --git a/Comp/r-sas_kruskalwallis.qmd b/Comp/r-sas_kruskalwallis.qmd index 8250c1583..8d3ad878a 100644 --- a/Comp/r-sas_kruskalwallis.qmd +++ b/Comp/r-sas_kruskalwallis.qmd @@ -14,7 +14,7 @@ stats::kruskal.test(Sepal_Width ~ Species, data = iris_sub) and in SAS using: -```{sas} +```sas proc npar1way data=iris_sub wilcoxon; class Species; var Sepal_Width; diff --git a/Comp/r-sas_logistic-regr.qmd b/Comp/r-sas_logistic-regr.qmd index 1bb9a12d0..5decc1868 100644 --- a/Comp/r-sas_logistic-regr.qmd +++ b/Comp/r-sas_logistic-regr.qmd @@ -165,7 +165,7 @@ cbind(est = coef(m1), confint.default(m1)) ### `PROC LOGISTIC` in SAS (without firth option) -```{sas} +```sas PROC LOGISTIC DATA=LUNG2; # import lung MODEL WT_GRP(EVENT="weight_gain") = AGE SEX PH_ECOG MEAL_CAL; ods output ESTIMATEs=estimates; @@ -249,7 +249,7 @@ cbind(est = coef(firth_mod), confint(firth_mod)) - `XCONV` specifies relative parameter convergence criterion, which should correspond to the `xconv` in `logistf` function in R. We specify `XCONV = 0.00000001` so it should be consistent with the R code above. -```{sas} +```sas PROC LOGISTIC DATA=LUNG2; MODEL WT_GRP(EVENT="weight gain") = AGE SEX PH_ECOG MEAL_CAL / firth clodds=PL clparm=PL xconv = 0.00000001; @@ -310,7 +310,7 @@ fit1$marginal_se We now use the SAS [`%Margins`](https://support.sas.com/kb/63/038.html) macro to perform the Ge et al. (2011) method on `trial01` to estimate the marginal risk difference and it's standard error. -```{sas} +```sas %Margins(data = myWork.trial01, class = trtp, classgref = first, /*Set reference to first level*/ @@ -337,7 +337,7 @@ run; ### `%LR` macro in SAS (Ge et al, 2011) -```{sas} +```sas %LR(data = myWork.trial01, /* input data set */ var1 = bl_cov, /* continuous covariates in the logistic regression */ var2 = trtp, /* categorical covariates in the logistic regression */ diff --git a/Comp/r-sas_mcnemar.qmd b/Comp/r-sas_mcnemar.qmd index 2626cf701..41fbb4982 100644 --- a/Comp/r-sas_mcnemar.qmd +++ b/Comp/r-sas_mcnemar.qmd @@ -45,7 +45,7 @@ confint(cohen_kappa, level = 0.95) The FREQ procedure can be used in SAS with the AGREE option to run the McNemar test, with OR, and RISKDIFF options stated for production of odds ratios and risk difference. These options were added as `epibasix::mcNemar` outputs the odds ratio and risk difference with confidence limits as default. In contrast to R, SAS outputs the Kappa coefficients with confident limits as default. -```{sas} +```sas proc freq data=colds; tables age12*age14 / agree or riskdiff; run; diff --git a/Comp/r-sas_mmrm.qmd b/Comp/r-sas_mmrm.qmd index 70e553550..5682fe25b 100644 --- a/Comp/r-sas_mmrm.qmd +++ b/Comp/r-sas_mmrm.qmd @@ -144,7 +144,7 @@ Code for fitting MMRMs to the FEV data using each of the considered functions an ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq; @@ -176,7 +176,7 @@ mmrm::mmrm( ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq; @@ -209,7 +209,7 @@ nlme::gls( ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = ARMCD|AVISIT / ddfm=satterthwaite solution chisq; @@ -251,7 +251,7 @@ glmmTMB::glmmTMB( ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq; @@ -294,7 +294,7 @@ glmmTMB::glmmTMB( ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq; @@ -326,7 +326,7 @@ nlme::gls( ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq; @@ -370,7 +370,7 @@ glmmTMB::glmmTMB( ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq; @@ -401,7 +401,7 @@ glmmTMB::glmmTMB( ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq; @@ -422,7 +422,7 @@ mmrm::mmrm( ### `PROC GLIMMIX` -```{sas} +```sas PROC GLIMMIX DATA = fev_data; CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID; MODEL FEV1 = ARMCD|AVISIT / ddfm=satterthwaite solution chisq; diff --git a/Comp/r-sas_negbin.qmd b/Comp/r-sas_negbin.qmd index a837828ba..44d5484bc 100644 --- a/Comp/r-sas_negbin.qmd +++ b/Comp/r-sas_negbin.qmd @@ -235,7 +235,7 @@ Exact match (at 0.001 level) can be obtained using `glm.nb` in R vs `PROC GENMOD After importing the dummy dataset we can run the negative binomial regression in SAS using \`PROC GENMOD. We estimate the model parameters and lsmeans for the treatment arms using both the default and OM weights. -```{sas} +```sas proc genmod data=df; class GRPC (ref='Plb') X2 (ref='A'); model y = GRPC x1 x2 / dist=negbin link=log offset=logtime; diff --git a/Comp/r-sas_psmatch.qmd b/Comp/r-sas_psmatch.qmd index c65fde747..4211e20be 100644 --- a/Comp/r-sas_psmatch.qmd +++ b/Comp/r-sas_psmatch.qmd @@ -164,7 +164,7 @@ Characteristic (N = 120) (N = 180) Mean Diff. ### SAS -```{sas} +```sas proc psmatch data=data region=cs(extend=0); class trtp sex bmi_cat; psmodel trtp(Treated="trt")= sex weight age bmi_cat; diff --git a/Comp/r-sas_survival.qmd b/Comp/r-sas_survival.qmd index 5e7b34c6a..fd0fee860 100644 --- a/Comp/r-sas_survival.qmd +++ b/Comp/r-sas_survival.qmd @@ -56,7 +56,7 @@ fit.cox <- survival::coxph( - SAS: change method for ties to use "efron" -```{sas} +```sas proc phreg data=dat; class afb; model lenfol*fstat(0) = afb/rl ties = efron; @@ -88,7 +88,7 @@ fit.km <- survival::survfit( - SAS: change to "log" -```{sas} +```sas proc lifetest data=dat conftype=log; time lenfoly*fstat(0); strata afb; @@ -136,7 +136,7 @@ fit.cox <- survival::coxph(survival::Surv(LENFOLY, FSTAT) ~ AFB, data = dat) - SAS: adjust convergence criterion -```{sas} +```sas proc phreg data=dat; class afb; model lenfol*fstat(0) = afb / rl fconv = 1e-9; @@ -188,7 +188,7 @@ summary(fit.km, times = c(80, 100, 120), extend = T) Below is the SAS code: -```{sas} +```sas proc lifetest data=dat outsurv=_SurvEst timelist= 80 100 120 reduceout stderr; time lenfoly*fstat(0); run; diff --git a/SAS/SAS_Friedmantest.qmd b/SAS/SAS_Friedmantest.qmd index bb18eadf7..30fcbab49 100644 --- a/SAS/SAS_Friedmantest.qmd +++ b/SAS/SAS_Friedmantest.qmd @@ -20,7 +20,7 @@ Simulated dataset of 10 subjects(blocks) with continuous endpoints are generated ## Data source -```{sas} +```sas data one_way_repeat; do subject = 1 to 10; do timepoint = 1 to 4; @@ -50,7 +50,7 @@ When the data contains missing response, the procedure discards the correspondin ## Example Code for Friedman Chi-square test -```{sas} +```sas proc freq data=one_way_repeat; tables subject*timepoint*response / cmh2 scores=rank noprint; diff --git a/SAS/ancova.qmd b/SAS/ancova.qmd index 7e28ad903..b23adfe28 100644 --- a/SAS/ancova.qmd +++ b/SAS/ancova.qmd @@ -20,7 +20,7 @@ In SAS, there are several ways to perform ANCOVA analysis. One common way is to The following data was used in this example. -```{sas} +```sas data DrugTest; input Drug $ PreTreatment PostTreatment @@; datalines; @@ -37,7 +37,7 @@ data DrugTest; The following code was used to test the effects of a drug pre and post treatment: -```{sas} +```sas proc glm data=DrugTest; class Drug; model PostTreatment = Drug PreTreatment / solution; diff --git a/SAS/anova.qmd b/SAS/anova.qmd index e383b3404..e44de67c3 100644 --- a/SAS/anova.qmd +++ b/SAS/anova.qmd @@ -12,7 +12,7 @@ To demonstrate the various types of sums of squares, we'll create a data frame c For this example, we're testing for a significant difference in `stem_length` using ANOVA. -```{sas} +```sas proc glm data = disease; class drug disease; model y=drug disease drug*disease; @@ -30,7 +30,7 @@ knitr::include_graphics("../images/linear/sas-f-table.png") SAS has four types of sums of squares calculations. To get these calculations, the sum of squares option needs to be added (`/ ss1 ss2 ss3 ss4`) to the model statement. -```{sas} +```sas proc glm; class drug disease; model y=drug disease drug*disease / ss1 ss2 ss3 ss4; @@ -81,7 +81,7 @@ knitr::include_graphics("../images/linear/sas-ss-type-4.png") To get contrasts in SAS, we use the `estimate` statement. For looking at contrast we are going to fit a different model on new data, that doesn't include an interaction term as it is easier to calculate contrasts without an interaction term. For this dataset we have three different drugs A, C, and E. -```{sas} +```sas proc glm data=testdata; class drug; model post = drug pre / solution; diff --git a/SAS/association.qmd b/SAS/association.qmd index c5a917c6b..8a37e9bdc 100644 --- a/SAS/association.qmd +++ b/SAS/association.qmd @@ -10,7 +10,7 @@ In SAS, association analysis methods for count data/contingency tables is typica The following tabulation was used for the SAS Chi-Square and Fisher's testing. This tabulation was derived from the same `lung` dataset used for the R function testing. The dataset is defined as follows: -```{sas} +```sas data test_case; input treatment $ Count Weight $; @@ -26,7 +26,7 @@ data test_case; The following SAS code produces both the Chi-Square and Fisher's Exact tests of association. Note that the results contain many statistics not produced by the corresponding R function. The relevant sections of the output have been outlined in red. -```{sas} +```sas proc freq data = test_case; weight Count; tables treatment * Weight / chisq fisher; diff --git a/SAS/binomial_test.qmd b/SAS/binomial_test.qmd index 5c4dae584..9306562ee 100644 --- a/SAS/binomial_test.qmd +++ b/SAS/binomial_test.qmd @@ -8,7 +8,7 @@ execute: Set the seed for reproducibility and simulate 1000 coin flips using a Bernoulli distribution. -```{sas} +```sas /* Set the seed for reproducibility */ %let seed = 19; @@ -29,7 +29,7 @@ run; Use SQL to count how many heads and tails were observed in the simulation. -```{sas} +```sas proc sql; select sum(result = "H") as heads_count, @@ -44,7 +44,7 @@ quit; Print the counts using `%put` statements. -```{sas} +```sas %put Heads Count: &heads_count; %put Tails Count: &tails_count; %put Total Flips: &total_flips; @@ -54,7 +54,7 @@ Print the counts using `%put` statements. Use `proc freq` to check if the observed results differ significantly from the expected probability of 0.5. -```{sas} +```sas proc freq data=coin_flips; tables result / binomial(p=0.5); run; @@ -66,7 +66,7 @@ We load a clinical dataset and test if the observed death proportion is signific ### Import Dataset -```{sas} +```sas proc import datafile='/home/u63532805/CAMIS/lung_cancer.csv' out=lung_cancer dbms=csv @@ -78,7 +78,7 @@ run; ### Create Binary Flag for Deaths -```{sas} +```sas data lung_cancer; set lung_cancer; death_flag = (status = 1); @@ -87,7 +87,7 @@ run; ### Perform Asymptotic Binomial Test -```{sas} +```sas #| eval: false proc freq data=lung_cancer; tables death_flag / binomial(p=0.19 level='1'); @@ -97,7 +97,7 @@ run; ### Perform Exact Binomial Test -```{sas} +```sas proc freq data=lung_cancer; tables death_flag / binomial(p=0.19 level='1'); exact binomial; @@ -107,7 +107,7 @@ run; ### Perform mid-P adjusted Exact Binomial Test -```{sas} +```sas #| eval: false proc freq data=lung_cancer; tables death_flag / binomial(p=0.19 level='1'); diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 368378dc1..600faf754 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -18,7 +18,7 @@ Caution is required if there are no responders (or all responders) in both group The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. -```{sas} +```sas proc import datafile = 'data/adcibc.csv' out = adcibc dbms = csv @@ -44,7 +44,7 @@ run; The below shows that for the Active Treatment, there are 36 responders out of 154 subjects, p1 = 0.2338 (23.38% responders), while for the placebo treatment p2 = 12/77 = 0.1558, giving a risk difference of 0.0779, relative risk 1.50, and odds ratio 1.6525. -```{sas} +```sas proc freq data=adcibc2; table trt*resp/ nopct nocol; run; @@ -147,7 +147,7 @@ Similarly for relrisk, although the output does not state that the RR is calcula SAS output often rounds to 3 or 4 decimal places in the output window, however the full values can be obtained using SAS ODS statements. -```{sas} +```sas ****************************; *** Risk Difference examples; diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd index 4adcad917..3cb5f9a70 100644 --- a/SAS/ci_for_paired_prop.qmd +++ b/SAS/ci_for_paired_prop.qmd @@ -12,7 +12,7 @@ execute: The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. -```{sas} +```sas data adcibc2 (keep=trt resp) ; set adcibc; if aval gt 4 then resp="Yes"; @@ -24,7 +24,7 @@ run; The below shows that for the Active Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). -```{sas} +```sas proc freq data=adcibc2; table trt*resp/ nopct nocol; run; @@ -177,7 +177,7 @@ The default method is Mantel-Haenszel confidence limits. SAS can also Score (Mie As you can see below, this is not a CI for difference in proportions of 0.196, it is a CI for the risk difference of 0.0260. So must be interpreted with much consideration. -```{sas} +```sas data dat_used; input ID$ PBO ACT @@; datalines; diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index 60d0166e3..c684750d6 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -12,7 +12,7 @@ See separate page for general introductory information on confidence intervals f The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. -```{sas} +```sas data adcibc2 (keep=trt resp) ; set adcibc; if aval gt 4 then resp="Yes"; @@ -24,7 +24,7 @@ run; The below shows that for the Active Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). -```{sas} +```sas proc freq data=adcibc2; table trt*resp/ nopct nocol; run; @@ -168,7 +168,7 @@ Caution is required if there are no responders in a group (aside from any issues The output consists of the proportion of resp=Yes, the Asymptotic SE, 95% CIs using normal-approximation method, 95% CI using the Clopper-Pearson method, and then a Binomial test statistic and p-value for the null hypothesis of H0: Proportion = 0.5. -```{sas} +```sas proc sort data=adcibc2; by trt; run; @@ -197,7 +197,7 @@ By adding the option `BINOMIAL(LEVEL="Yes" CL=)`, the other C - `BINOMIAL(LEVEL="Yes" CL=WILSON(CORRECT) WALD(CORRECT));`will return Wilson (with continuity correction) and Wald (with continuity correction) -```{sas} +```sas proc freq data=adcibc2; table resp/ nopct nocol BINOMIAL(LEVEL="Yes" @@ -216,7 +216,7 @@ run; knitr::include_graphics("../images/ci_for_prop/binomial_prop_all_act.png") ``` -```{sas} +```sas proc freq data=adcibc2; table resp/ nopct nocol BINOMIAL(LEVEL="Yes" diff --git a/SAS/cmh.qmd b/SAS/cmh.qmd index eb742248f..b6123ba55 100644 --- a/SAS/cmh.qmd +++ b/SAS/cmh.qmd @@ -16,7 +16,7 @@ When the design of the contingency table is 2 x 2 x K (i.e, X == 2 levels, Y == Below is the syntax to conduct a CMH analysis in SAS: -```{sas} +```sas proc freq data = filtered_data; tables K * X * Y / cmh; * the order of K, X, and Y appearing on the line is important!; @@ -31,7 +31,7 @@ The adcibc data described [here](https://psiaims.github.io/CAMIS/R/cmh.html) is The code used is always the same, however, we can limit the number of levels in each example to show a 2x2x2 case, 2x3xK case etc. -```{sas} +```sas proc freq data = adcibc; tables agegr1 * trtp * sex / cmh; run; @@ -70,7 +70,7 @@ Risk differences within each strata (for 2 x 2 x K tables) can be obtained by ad The individual treatment comparisons within strata can be useful to explore if the treatment effect is in the same direction and size for each strata, such as to determine if pooling them is in fact sensible. -```{sas} +```sas # Default method is: Wald asymptotic confidence limits proc freq data = adcibc (where=(trtpn ne 54 and agegr1 ne ">80")); tables agegr1 * trtp * sex / cmh riskdiff; @@ -87,7 +87,7 @@ knitr::include_graphics("../images/cmh/saspage_output3.png") Note above that exact CI's are not output for the difference betweeen the treatments. You can request SAS output other CI methods as shown below. This outputs the risk difference between the treatments and 95% CI, calculated for each age group strata separately using the Miettinen-Nurminen (score) (MN) method. -```{sas} +```sas # You can change the confidence limits derivation using (cl=xxx) option proc freq data = adcibc (where=(trtpn ne 54 and agegr1 ne "\>80")); tables agegr1 * trtp * sex / cmh riskdiff(cl=mn); @@ -109,7 +109,7 @@ PROC FREQ in the SAS Viya platform has introduced a new COMMONRISKDIFF(CL=MN/MNM See the next section on Common risk differences available in SAS. -```{sas} +```sas # Specifying the Miettinen-Nurminen (score) method proc freq data = adcibc (where=(trtpn ne 54 and agegr1 ne ">80")); tables agegr1 * trtp * sex / cmh riskdiff (common cl=mn); @@ -130,7 +130,7 @@ Including either column=1 or column=2 tells SAS which is your outcome of interes Note that if the data are such that there are no responses in either treatment group, the cross-tabulation will only have 1 column, and SAS PROC FREQ will fail to produce a confidence interval. In this unusual situation however, a valid confidence interval can (and should) still be produced, which may be obtained using the %SCORECI macro available at [https://github.com/petelaud/ratesci-sas/tree/main](#0). -```{sas} +```sas proc freq data = adcibc (where=(trtpn ne 54 and agegr1 ne "\>80")); tables agegr1 * trtp * sex / cmh riskdiff(common column=2); run; @@ -145,7 +145,7 @@ Note that SAS (since v9.3M2 / STAT 12.1) PROC FREQ will produce the Miettinen-Nu Stratified Miettinen-Nurminen CIs have more recently been added to PROC FREQ, but only within the SAS Viya platform (using CL=MN or CL=MNMH in the COMMONRISKDIFF option). The only other known way to have SAS produce these CIs is to use this publicly available macro: [https://github.com/petelaud/ratesci-sas/tree/main](#0). Note that the macro (currently) does not implement Miettinen & Nurminen's proposed iterative weights, but instead the simpler (and similar) MH weights are used. -```{sas} +```sas proc freq data = adcibc (where=(trtpn ne 54 and agegr1 ne ">80")); tables agegr1 * trtp * sex / cmh commonriskdiff(CL=SCORE TEST=SCORE); run; @@ -160,7 +160,7 @@ run; knitr::include_graphics("../images/cmh/saspage_output4.png") ``` -```{sas} +```sas proc freq data = adcibc (where=(trtpn ne 54 and agegr1 ne ">80")); tables agegr1 * trtp * sex / cmh commonriskdiff(CL= newcombe); run; diff --git a/SAS/correlation.qmd b/SAS/correlation.qmd index 03dd2528b..d1ce1995d 100644 --- a/SAS/correlation.qmd +++ b/SAS/correlation.qmd @@ -38,7 +38,7 @@ In contrast, if the data set contains missing values for the analysis variables ## **Pearson Correlation** -```{sas} +```sas proc corr data=lung pearson; var age mealcal; run; @@ -48,7 +48,7 @@ run; ## **Spearman Correlation** -```{sas} +```sas proc corr data=lung spearman; var age mealcal; run; @@ -58,7 +58,7 @@ run; ## Kendall's rank correlation -```{sas} +```sas proc corr data=lung kendall; var age mealcal; run; diff --git a/SAS/count_data_regression.qmd b/SAS/count_data_regression.qmd index 4749975ae..410f65f9f 100644 --- a/SAS/count_data_regression.qmd +++ b/SAS/count_data_regression.qmd @@ -40,7 +40,7 @@ It is generally good practice to apply the OM option on the lsmeans statement. T You can use exponential of the maximum likelihood parameter estimate (for treat and age in this example), and the exponential of the Wald 95% Confidence Limits to obtain the odds ratios and 95% CIs. Estimates of the least squares means and CI's for each treatment are output using the lsmeans option. These estimates also have to be back transformed using exponential distribution to get the mean count back onto the original scale. Proc Genmod uses GLM parameterization. -```{sas} +```sas ods output ParameterEstimates=ORs lsmeans=lsm; proc genmod data=polyps; class treat (ref="placebo"); @@ -83,7 +83,7 @@ In SAS, we can use proc genmod to perform negative binomial regression. The belo Model parameterization is very similar to poisson -```{sas} +```sas ods output ParameterEstimates=ORs lsmeans=lsm; proc genmod data=polyps; class treat (ref="placebo"); diff --git a/SAS/gee.qmd b/SAS/gee.qmd index c75ab079b..8707a84af 100644 --- a/SAS/gee.qmd +++ b/SAS/gee.qmd @@ -18,7 +18,7 @@ To uniquely identify subjects, a new variable USUBJID was created by concatenati Additionally, two variables were created using randomly generated values to simulate variables with more than two categories. One was an ordinal variable with values 1, 2, and 3; the other was a nominal variable with categories 'liver', 'lung', and 'bone'. The resulting dataset is saved here: /data/resp.xlsx The following SAS code was used: -```{sas} +```sas proc format; value respmulti 1='Liver' @@ -53,7 +53,7 @@ Predicted probabilities and Odds Ratios (OR) can be obtained in SAS using the `l - `cl` computes confidence intervals. -```{sas} +```sas proc gee data=resp; class trtp(ref="P") avisitn(ref='1') usubjid; model outcome(event='1') = trtp avisitn trtp*avisitn/ dist=bin link=logit; @@ -112,7 +112,7 @@ The estimated parameters for each model are detailed below. ORs can be obtained [Ordinal variable:]{.underline} -```{sas} +```sas proc gee data=resp; class trtp(ref="A") avisitn(ref='1') usubjid; model respord=trtp avisitn trtp*avisitn/ dist=multinomial link=cumlogit; @@ -124,7 +124,7 @@ run; [Nominal variable:]{.underline} -```{sas} +```sas proc gee data=resp ; class trtp(ref="A") avisitn(ref='1') usubjid; model respnom(event='Liver')=trtp avisitn trtp*avisitn/ dist=multinomial link=glogit; diff --git a/SAS/gsd-tte.qmd b/SAS/gsd-tte.qmd index 5f0d39365..563207eb2 100644 --- a/SAS/gsd-tte.qmd +++ b/SAS/gsd-tte.qmd @@ -18,7 +18,7 @@ A GSD will be utilized for progression-free survival (PFS). PFS will be tested a The SAS code is shown below: -```{sas} +```sas PROC SEQDESIGN; DESIGN NSTAGES=2 INFO=CUM(0.75 1.0) diff --git a/SAS/jonchkheere_terpstra.qmd b/SAS/jonchkheere_terpstra.qmd index de441dfe4..ea5242da0 100644 --- a/SAS/jonchkheere_terpstra.qmd +++ b/SAS/jonchkheere_terpstra.qmd @@ -14,7 +14,7 @@ The JT test is particularly well-suited for dose-response or trend analysis with To request Jonckheere-Terpstra test, specify the **JT** option in the Table statement like below: -```{sas} +```sas proc freq; table Var1 * Var2 / JT; Quit; @@ -28,7 +28,7 @@ PROC FREQ also provides exact p-values for the Jonckheere-Terpstra test. You can This dataset has been generated using example data which aligned with the specifications outlined in the section on the Jonckheere–Terpstra test from reference \[5\]. It represents the duration of hospital stays for a randomly selected group of patients across three distinct ICU departments: cardiothoracic, medical, and neurosurgical. -```{sas} +```sas data ICU_Stay; input ICU $ Stay; label Stay = 'Length of Stay in Days'; @@ -62,7 +62,7 @@ run; The code performs a frequency analysis on the 'ICU_Stay' dataset, examining the relationship between 'ICU' and 'Stay' variables. It applies the Jonckheere-Terpstra test using JT option to identify trends in the ordered categorical 'Stay' variable. The output is streamlined by omitting percentages and totals for columns and rows with the 'nopercent nocol norow' options, emphasizing the Jonckheere-Terpstra test outcomes. -```{sas} +```sas proc freq data=ICU_Stay; table ICU * Stay / JT nopercent nocol norow; run; @@ -78,7 +78,7 @@ Comparing this with a standard Normal distribution gives a P value of 0.005, ind This dataset incorporates illustrative data extracted from reference \[3\]. It encapsulates the responses of subjects randomly assigned to one of four treatment arms: placebo, low dosage(20mg), medium dosage(60mg), and high dosage(180mg). The variable of interest is a continuous measure. The variable 'groupn' is used to provide an order of 'group'. -```{sas} +```sas data contin; input groupn group $ subject response; cards; @@ -114,7 +114,7 @@ run; The code is performing a Jonckheere-Terpstra trend test on a continuous 'response' variable, categorized by a 'group' variable, using the 'proc freq' procedure. The analysis is applied to the dataset named 'contin'. The result is presented with a title "Jonckheere-Terpstra Trend Test for Continuous Data", indicating the specific nature of the test being conducted. The 'JT' option is used to specify the Jonckheere-Terpstra test. -```{sas} +```sas proc freq data=contin; tables group * response/JT; title "Jonckheere-Terpstra Trend Test for Continuous Data"; @@ -131,7 +131,7 @@ There is a significant trend across different groups in the response gives a P v With EXACT statement, the exact version and it Monte Carlo approximation can be also conducted. However, it should be noted that the exact test, i.e., a permuation test takes a long time to compelete the task even for a small dataset. -```{sas} +```sas proc freq data = inds; title "Asymptotic p-value calculation"; table ICU * Stay / jt; diff --git a/SAS/kruskal_wallis.qmd b/SAS/kruskal_wallis.qmd index 3b65ee7ae..f205163a7 100644 --- a/SAS/kruskal_wallis.qmd +++ b/SAS/kruskal_wallis.qmd @@ -8,7 +8,7 @@ execute: The Kruskal-Wallis test is a non-parametric equivalent to the one-way ANOVA. For this example, the data used is a subset of R's datasets::iris, testing for difference in sepal width between species of flower. This data was subset in R and input manually to SAS with a data step. -```{sas} +```sas data iris_sub; input Species $ Sepal_Width; datalines; @@ -38,7 +38,7 @@ run; The Kruskal-Wallis test can be implemented in SAS using the NPAR1WAY procedure with WILCOXON option. Below, the test is defined with the indicator variable (Species) by the CLASS statement, and the response variable (Sepal_Width) by the VAR statement. Adding the EXACT statement outputs the exact p-value in addition to the asymptotic result. The null hypothesis is that the samples are from identical populations. -```{sas} +```sas proc npar1way data=iris_sub wilcoxon; class Species; var Sepal_Width; diff --git a/SAS/linear-regression.qmd b/SAS/linear-regression.qmd index 262d8f178..e7d77b992 100644 --- a/SAS/linear-regression.qmd +++ b/SAS/linear-regression.qmd @@ -12,7 +12,7 @@ To demonstrate the use of linear regression we examine a dataset that illustrate The first step is to obtain the simple descriptive statistics for the numeric variables of htwt data, and one-way frequencies for categorical variables. This is accomplished by employing proc means and proc freq procedures There are 237 participants who are from 13.9 to 25 years old. It is a cross-sectional study, with each participant having one observation. We can use this data set to examine the relationship of participants' height to their age and sex. -```{sas} +```sas proc means data=htwt; run; ``` @@ -29,7 +29,7 @@ WEIGHT WEIGHT 237 101.3080169 19.4406980 50.5000000 171.5000000 ---------------------------------------------------------------------------- ``` -```{sas} +```sas proc freq data=htwt; tables sex; run; @@ -48,7 +48,7 @@ m 126 53.16 237 100.00 In order to create a regression model to demonstrate the relationship between age and height for females, we first need to create a flag variable identifying females and an interaction variable between age and female gender flag. -```{sas} +```sas data htwt2; set htwt; if sex="f" then female=1; @@ -63,7 +63,7 @@ run; Next, we fit a regression model, representing the relationships between gender, age, height and the interaction variable created in the datastep above. We again use a where statement to restrict the analysis to those who are less than or equal to 19 years old. We use the clb option to get a 95% confidence interval for each of the parameters in the model. The model that we are fitting is ***height = b0 + b1 x female + b2 x age + b3 x fem_age + e*** -```{sas} +```sas proc reg data=htwt2; where age <=19; model height = female age fem_age / clb; diff --git a/SAS/logistic-regr.qmd b/SAS/logistic-regr.qmd index e848f8dfb..c965f8d15 100644 --- a/SAS/logistic-regr.qmd +++ b/SAS/logistic-regr.qmd @@ -38,7 +38,7 @@ Below we are fitting trt01pn and sex as categorical variables, age, ph_ecog2 and You can use exponential of the maximum likelihood parameter estimate and the exponential of the Wald 95% Confidence Limits to obtain the odds ratios and 95% CIs. Proc Genmod uses GLM parameterization. -```{sas} +```sas Example data: . = missing, trt01pn (1 or 2), sex (1 or 2), ph_ecog2 (0,1,2,3) wt_gain (1=gain, 0=no gain) ``` @@ -51,7 +51,7 @@ wt_gain trt01pn age sex ph_ecog2 meal_caln 1 2 60 1 2 . ``` -```{sas} +```sas proc genmod data=lung; class trt01pn (ref="1") sex (ref="1"); model wt_gain (event="1") = trt01pn age sex ph_ecog2 meal_caln / @@ -106,7 +106,7 @@ Proc Logistic is often preferred above Proc Genmod as it outputs the Odds Ratios NOTE: that the 95% confidence limits are being calculated using the Wald method. This assumes symmetric intervals around the maximum likelihood estimate using a normal distribution assumption (MLE +/-1.96\* SE). Alternative confidence interval estimation methods exist such as the profile likelihood method but SAS does not calculate these. -```{sas} +```sas proc logistic data=lung; class trt01pn (ref="1") sex (ref="1") /param=glm; model wt_gain(event="1") = trt01pn age sex ph_ecog2 meal_caln; @@ -166,7 +166,7 @@ meal_cal 1.001 1.000 1.002 To compare two logistic models, the -2 \* Log Likelihood from each model can be compared against a $\chi^2$-distribution with degrees of freedom calculated using the difference in the two models' parameters. -```{sas} +```sas Model 1: model wt_gain(event="1") = trt01pn age sex ph_ecog2 meal_caln; ``` @@ -194,7 +194,7 @@ SC 207.491 216.477 SAS also allows us to fit forward or backwards stepwise selection. Below we specify to stop when we have 4 variables left in the model. This is not commonly done in practice but is included to highlight the difference in using a selection procedure compared to doing the difference betweeen the -2 \* log likelihood models using a $\chi^2$-distribution. -```{sas} +```sas proc logistic data=lung; class trt01pn (ref="1") sex (ref="1") /param=glm; model wt_gain(event="1") = trt01pn age sex ph_ecog2 meal_caln/ @@ -229,7 +229,7 @@ This is the SAS default such that if you do not specify the `/param` option, SAS With the EFFECT option, dose_id has 3 levels, and so needs 2 design variables (β1 and β2). Sex has 2 levels so uses just 1 design variable (β1). For dose_id, the reference level (Placebo) is given values of "-1" for both of the β1 and β2 parameters. General model: Y= α + β1x1 + β2x1 {+β3x3} etc, representing each parameter in the model. -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=effect; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -274,7 +274,7 @@ To compare 10mg Active vs Placebo, we would do the following: This equates to a contrast statement as follows: -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=effect; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -292,7 +292,7 @@ To compare the average of (10mg Active and 20mg Active) vs Placebo, we would do This equates to a contrast statement as follows: -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=effect; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -313,7 +313,7 @@ Active (10mg +20mg) vs Placebo 1 1.1610 0.2813 Now let's look at the `param=glm` option. GLM parameterization has a design variable for each level of a parameter. Hence for dose with 3 levels, we have 3 design variables (β1, β2 and β3). -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=glm; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -358,7 +358,7 @@ To compare 10mg Active vs Placebo, we would do the following: This equates to a contrast statement as follows: -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=glm; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -378,7 +378,7 @@ To compare the average of (10mg Active and 20mg Active) vs Placebo, we would do This equates to a contrast statement as follows: -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=glm; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -399,7 +399,7 @@ Active (10mg +20mg) vs Placebo 1 1.1610 0.2813 Now let's look at the `param=ref` option. Similar to param=effect, ref parameterization uses 1 less design variable compared to the number of levels each parameter has, but the parameterization is different. For dose with 3 levels, we have 2 design variables (β1 and β2). -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=ref; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -442,7 +442,7 @@ To compare 10mg Active vs Placebo, we would do the following: This equates to a contrast statement as follows: -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=glm; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -460,7 +460,7 @@ To compare the average of (10mg Active and 20mg Active) vs Placebo, we would do This equates to a contrast statement as follows: -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=glm; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; @@ -481,7 +481,7 @@ Active (10mg +20mg) vs Placebo 1 1.1610 0.2813 The Contrast statement, only outputs the p-value for the contrast, but it is common to also require an estimate of the difference between the treatments, with associated 95% CI. You can do this by changing `contrast` to an `estimate` statement. Note that the parameterization of the contrast remains the same as when using a contrast statement as shown below. These estimates and 95% CI's can be back transformed to give you the Odds ratio of the contrast and associated 95% CI. The estimate coefficients table should be checked for accuracy versus the contrast you are trying to do. -```{sas} +```sas proc logistic data=lung; class dose_id (ref="3") sex (ref="1") /param=glm; model wt_gain(event="1") = dose_id age sex ph_ecog2 meal_caln; diff --git a/SAS/manova.qmd b/SAS/manova.qmd index f7d35b941..b247bddc8 100644 --- a/SAS/manova.qmd +++ b/SAS/manova.qmd @@ -10,7 +10,7 @@ This example employs multivariate analysis of variance (MANOVA) to measure diffe For each of 26 samples of pottery, the percentages of oxides of five metals are measured. The following statements create the data set and invoke the GLM procedure to perform a one-way MANOVA. Additionally, it is of interest to know whether the pottery from one site in Wales (Llanederyn) differs from the samples from other sites; a CONTRAST statement is used to test this hypothesis. -```{sas} +```sas # Example code title "Romano-British Pottery"; data pottery; diff --git a/SAS/mcnemar.qmd b/SAS/mcnemar.qmd index ff3501648..e203bb997 100644 --- a/SAS/mcnemar.qmd +++ b/SAS/mcnemar.qmd @@ -12,7 +12,7 @@ To demonstrate McNemar's test in SAS, data concerning the presence or absence of Testing for a significant difference in cold symptoms between ages, using McNemar's test in SAS, can be performed as below. The AGREE option is stated within the FREQ procedure to produce agreement tests and measures, including McNemar's test. -```{sas} +```sas proc freq data=colds; tables age12*age14 / agree; run; diff --git a/SAS/mi_mar_regression.qmd b/SAS/mi_mar_regression.qmd index 2b7e50c30..2ec36ea60 100644 --- a/SAS/mi_mar_regression.qmd +++ b/SAS/mi_mar_regression.qmd @@ -15,7 +15,7 @@ execute: As PROC MI requires a horizontal, one record per subject data set. More often than not, the data we impute will come from a vertical ADaM BDS data set. So we need to first transpose the aval with the avisitn as ID (assuming avisitn = 1 to 5),creating transposed variable v1-v5. -```{sas} +```sas data dummy; length USUBJID $4; do i=1 to 10; @@ -61,7 +61,7 @@ The pattern can be checked using the following code, missing data pattern could - "Arbitrary" : The missingness of data does not follow any specific order or predictable sequence. Data can be missing at random points without a discernible pattern. -```{sas} +```sas ods select MissPattern; proc mi data=dummyt nimpute=0; var v1 - v5; @@ -82,7 +82,7 @@ knitr::include_graphics( ## FCS Regression for non-monotone missing pattern -```{sas} +```sas proc mi data=dummyt out=outdata nimpute=10 seed=123; class sex1n; var sex1n v1 - v5; @@ -119,7 +119,7 @@ knitr::include_graphics("../images/mi_mar_linear_sas/mi_mar_reg_fcs2.PNG") Let's update above SAS code to generate a dummy dataset with monotone missing pattern -```{sas} +```sas data dummy; length USUBJID $4; do i=1 to 10; @@ -167,7 +167,7 @@ knitr::include_graphics( In this case we will use `monotone` statement instead of `FCS` for the imputation, example code as below: -```{sas} +```sas proc mi data=dummyt out=outdata nimpute=10 seed=123; class sex1n; var sex1n v1 - v5; diff --git a/SAS/mmrm.qmd b/SAS/mmrm.qmd index bafc7de25..14e4f7922 100644 --- a/SAS/mmrm.qmd +++ b/SAS/mmrm.qmd @@ -10,7 +10,7 @@ execute: In SAS the following code was used (assessments at `avisitn=0` should also be removed from the response variable): -```{sas} +```sas proc mixed data=adlbh; where base ne . and avisitn not in (., 99); class usubjid trtpn(ref="0") avisitn; diff --git a/SAS/nparestimate.qmd b/SAS/nparestimate.qmd index eca179f2c..1422a47ad 100644 --- a/SAS/nparestimate.qmd +++ b/SAS/nparestimate.qmd @@ -14,7 +14,7 @@ PROC NPAR1WAY provides these estimates in a flexible manner. # Case study -```{sas} +```sas # Hollander-Wolfe-Chicken Example data all; input group $ value; @@ -45,7 +45,7 @@ run; Hodges-Lehmann estimate and Moses confidence interval for the 2-sample case will be generated when putting HL as an option. The direction of the comparison can be controlled via refclass. If the exact confidence interval is required additionally then the exact statement together with the option HL needs to be defined. The Hodges-Lehmann point estimate and confidence interval can be addressed via the HodgesLehmann option under the ODS statement. -```{sas} +```sas proc npar1way hl (refclass = "B") data = all; class group; var value; diff --git a/SAS/ranksum.qmd b/SAS/ranksum.qmd index 79073889d..d25c91d3e 100644 --- a/SAS/ranksum.qmd +++ b/SAS/ranksum.qmd @@ -14,7 +14,7 @@ To perform a Wilcoxon rank-sum test in SAS, you can use the PROC NPAR1WAY proced 1. **Create the Dataset**: If there are two groups (smoker and non-smoker) with their respective measurements birth weight, you can input the data as follows: -```{sas} +```sas /* Create dataset */ data bw; input bw grp $; @@ -54,7 +54,7 @@ run; 2. **Perform the Wilcoxon rank-sum Test**: Use the PROC NPAR1WAY procedure to perform the test. The wilcoxon option specifies that you want to perform the Wilcoxon rank-sum test. When computing the asymptotic Wilcoxon two-sample test, PROC NPAR1WAY uses a continuity correction by default. If specify the CORRECT=NO option in the PROC NPAR1WAY statement, the procedure does not use a continuity correction. Typically, we will also want the Hodges-Lehman confidence intervals. To get these you will need to add `hl` to the pro npar1way statement. -```{sas} +```sas /* Perform Wilcoxon rank-sum test - with continuity correction by default*/ proc npar1way data=BW wilcoxon hl; class grp; @@ -110,7 +110,7 @@ The correction does not effect the Hodges-Lehman CI. The Location shift is the H For sufficiently small sample size, the large-sample normal approximation used by the asymptotic Wilcoxon might not be appropriate, so the exact statement is needed. -```{sas} +```sas /* Perform Wilcoxon rank-sum test - with continuity correction by default*/ proc npar1way data=BW wilcoxon CORRECT=NO hl; class grp; diff --git a/SAS/rbmi_continuous_joint_SAS.qmd b/SAS/rbmi_continuous_joint_SAS.qmd index 800e15a3c..d1ef5ff27 100644 --- a/SAS/rbmi_continuous_joint_SAS.qmd +++ b/SAS/rbmi_continuous_joint_SAS.qmd @@ -44,7 +44,7 @@ A publicly available example [dataset](https://r-packages.io/datasets/antidepres The relevant endpoint is the Hamilton 17-item depression rating scale (HAMD17) which was assessed at baseline and at weeks 1, 2, 4, and 6 (visits 4-7). Study drug discontinuation occurred in 24% (20/84) of subjects from the active drug and 26% (23/88) of subjects from placebo. All data after study drug discontinuation are missing. -```{sas} +```sas proc print data=dat (obs=10); var PATIENT GENDER THERAPY RELDAYS VISIT BASVAL HAMDTL17 CHANGE; run; @@ -60,7 +60,7 @@ knitr::include_graphics("../images/rbmi/SAS_ExploreData1.PNG") The number of patients per visit and arm are: -```{sas} +```sas proc freq data=dat; table VISIT*THERAPY / norow nocol nopercent nocum; run; @@ -76,7 +76,7 @@ knitr::include_graphics("../images/rbmi/SAS_ExploreData2.PNG") The mean change from baseline of the endpoint (Hamilton 17-item depression rating scale, HAMD17) per visit per treatment group using only the complete cases are: -```{sas} +```sas proc means data=dat n mean nonobs; class VISIT THERAPY; var CHANGE; @@ -93,7 +93,7 @@ knitr::include_graphics("../images/rbmi/SAS_ExploreData3.PNG") The missingness pattern is show below. The incomplete data is primarily monotone in nature. 128 patients have complete data for all visits (all 1’s at each visit). 20, 10 and 13 patients have 1, 2 or 3 monotone missing data, respectively. Further, there is a single additional intermittent missing observation (patient 3618). -```{sas} +```sas proc transpose data=dat out=HAMD_wide(drop=_NAME_) prefix=CHG; by PATIENT THERAPY BASVAL; id VISIT; @@ -117,7 +117,7 @@ knitr::include_graphics("../images/rbmi/SAS_ExploreData4.PNG") A complete case analysis is performed using mixed model for repeated measures (MMRM) with covariates: treatment \[THERAPY\], gender \[GENDER\], visit \[VISIT\] as factors; baseline score \[BASVAL\] as continuous; and visit-by-treatment \[THERAPY \* VISIT\] interaction, and visit-by-baseline \[BASVAL \* VISIT\] interaction. An unstructured covariance matrix is used. The **MIXED** procedure is used. -```{sas} +```sas proc mixed data=dat method=reml; class THERAPY(ref="PLACEBO") VISIT(ref="4") PATIENT GENDER(ref="F"); model CHANGE = THERAPY GENDER VISIT BASVAL THERAPY*VISIT BASVAL*VISIT /s ddfm=satterthwaite; @@ -178,7 +178,7 @@ Most of the computation time is spent in the Part1B macro where the MCMC procedu To perform reference-based multiple imputation using MAR approach to following code is used -```{sas} +```sas %part1A(jobname = HAMD, Data=dat, Subject=PATIENT, @@ -207,7 +207,7 @@ To perform reference-based multiple imputation using MAR approach to following c To print the results of the contrast at week 7 -```{sas} +```sas proc print data=HAMD_MAR_OUT; where VISIT = "7"; var VISIT THERAPY _THERAPY Diff SE_Diff df Probt LCL_Diff UCL_Diff; @@ -226,7 +226,7 @@ knitr::include_graphics("../images/rbmi/SAS_MAR_contrast.PNG") To perform reference-based multiple imputation using Copy Reference (CR) approach the following changes are needed in part2A of the 5 macros -```{sas} +```sas %part2A(jobname = HAMD_CR, inname = HAMD, method = CR, @@ -247,7 +247,7 @@ knitr::include_graphics("../images/rbmi/SAS_MNAR_CR_contrast.PNG") To perform reference-based multiple imputation using Jump to Reference (J2R) approach the following changes are needed in part2A of the 5 macros -```{sas} +```sas %part2A(jobname = HAMD_J2R, inname = HAMD, method = J2R, @@ -268,7 +268,7 @@ knitr::include_graphics("../images/rbmi/SAS_MNAR_J2R_contrast.PNG") To perform reference-based multiple imputation using Copy Increments in Reference (CIR) approach the following changes are needed in part2A of the 5 macros -```{sas} +```sas %part2A(jobname = HAMD_CIR, inname = HAMD, method = CIR, @@ -317,7 +317,7 @@ Part 3 of the 5 macros carries out a univariate ANOVA analysis at selected time Since, all imputed datasets are readily available (after part2B), another possibility is to analyse each imputed dataset using the analysis model of your choice, and combining the results using `PROC MIANALYZE`. For example, suppose an MMRM should be fit on each imputed dataset: -```{sas} +```sas data HAMD_CR_DATAFULL; set HAMD_CR_DATAFULL; _Imputation_ = draw; diff --git a/SAS/recurrent_events.qmd b/SAS/recurrent_events.qmd index 6984bdf08..79a384f89 100644 --- a/SAS/recurrent_events.qmd +++ b/SAS/recurrent_events.qmd @@ -351,7 +351,7 @@ And the data structure must be: We will use the `bladder2` data for this. -```{sas} +```sas proc phreg data=bladder2 covs(aggregate); class rx (ref='1'); model (tstart, tstop) * event(0) = rx size number /rl; @@ -373,7 +373,7 @@ By including the `covs(aggregate)` option and setting `id subjid;`, SAS will com The original Andersen-Gill model of 1989 can be fitted by changing `covs(aggregate)` to `covs` in the procedure, while excluding `id subjid;`. -```{sas} +```sas proc phreg data=bladder2 covs; class rx (ref='1'); model (tstart, tstop) * event(0) = rx size number /rl; @@ -415,7 +415,7 @@ And the data structure must be: We will use the `bladder2` data for this. -```{sas} +```sas proc phreg data=bladder2 covs(aggregate); class rx (ref='1'); model (tstart, tstop) * event(0) = rx size number /rl; @@ -457,7 +457,7 @@ And the data structure must be: This data structure can be achieved in `bladder2` by adding a `gtime` variable. -```{sas} +```sas data bladder2; set bladder2; gtime = tstop - tstart; @@ -466,7 +466,7 @@ run; We artificially set start = 0 for each gap time interval by including `gtime` instead of `(start, stop)` in the `model` statement. -```{sas} +```sas proc phreg data=bladder2 covs(aggregate); class rx (ref='1'); model gtime * event(0) = rx size number/rl; @@ -520,7 +520,7 @@ And the data structure must be: We will use the `bladder` data for this. -```{sas} +```sas proc phreg data=bladder covs(aggregate); class rx (ref='1'); model tstop * event(0) = rx size number /rl; @@ -608,7 +608,7 @@ For the Prentice-Williams-Peterson and Wei-Lin-Weissfeld models we can incorpora To get event-specific estimates for the treatment effect (**rx**), we first need to introduce four new **rx** variables to the `bladder2` and `bladder` datasets, one for each stratum. -```{sas} +```sas data bladder2; set bladder2; rx_enum1 = rx*(enum=1); @@ -618,7 +618,7 @@ data bladder2; run; ``` -```{sas} +```sas data bladder; set bladder; rx_enum1 = rx*(enum=1); @@ -634,7 +634,7 @@ With these four interaction variables, we need to specify `rx_enum1-rx_enum4` in **Total time model** -```{sas} +```sas proc phreg data=bladder2 covs(aggregate); class enum / param=glm; model (tstart, tstop) * event(0) = rx_enum1-rx_enum4 size number /rl; @@ -654,7 +654,7 @@ knitr::include_graphics("../images/recurrent_events/SAS_PWPtt_stratified.png") **Gap time model** -```{sas} +```sas proc phreg data=bladder2 covs(aggregate); class enum / param=glm; model gtime * event(0) = rx_enum1-rx_enum4 size number/rl; @@ -674,7 +674,7 @@ knitr::include_graphics("../images/recurrent_events/SAS_PWPgt_stratified.png") #### Wei-Lin-Weissfeld model -```{sas} +```sas proc phreg data=bladder covs(aggregate); class enum / param=glm; model tstop * event(0) = rx_enum1-rx_enum4 size number /rl; diff --git a/SAS/rmst.qmd b/SAS/rmst.qmd index 99c2a98fd..eaafe9c9c 100644 --- a/SAS/rmst.qmd +++ b/SAS/rmst.qmd @@ -77,7 +77,7 @@ It is good practice to first view the shape of your Kaplan-Meier curves. As you It is very important to pre-specify your approach for selection of `tau`. As you can see from the curves, if we compared the period 0 to 6 months, vs 0 to 18 months, we would get very different results for the treatment comparison. -```{sas} +```sas proc lifetest data=adcibc conftype=log; time time*cnsr(1); strata trt01pn; @@ -96,7 +96,7 @@ knitr::include_graphics("../images/rmst/kmplot.png") The code below calculates `tau` as the minimum time of the last event observed in each treatment group. This will be the period of time, our AUC will be calculated over. As cnsr=0 are events, we only select these observations. Below, the maximum event in treatment 1 = 883 days and in treatment 2 = 350 days. We therefore set `tau` = 350. This method avoids including in the AUC a period of time where events are no longer occurring in both treatments. You can see why setting `tau` is very important as we are likely to get very different AUCs calculating over the 350 day as opposed to the 883 day period! -```{sas} +```sas proc sort data=lung_cancer (where=(cnsr=0)) out=tau1; by trt01pn time; run; @@ -133,7 +133,7 @@ In the output, its important to check that your event/censoring flag is the righ ### Linear link model - provides estimates of treatment differences -```{sas} +```sas proc rmstreg data=adcibc tau=&_tau; class trt01pn sex; model time*cnsr(1) =trt01pn sex age /link=linear method=ipcw (strata=trt01pn); @@ -166,7 +166,7 @@ The code is similar to above. We include the option `exp` on the lsmeans row, si Similar to the linear model, we obtain results of a `restricted mean survival time` estimate of 255.21 days on treatment 1 vs 264.75 days on treatment 2. The difference (Active-Placebo) on the log scale is -0.03667 (95% CI: -0.1493 to 0.07596, p=0.5234) but this is hard to interpret. Hence, once back transformed, the treatment ratio (Active/Placebo) is 0.9640 (95% CI: 0.8613 to 1.0789, p=0.5234). -```{sas} +```sas proc rmstreg data=adcibc tau=&_tau; class trt01pn sex; model time*cnsr(1) =trt01pn sex age /link=log method=ipcw (strata=trt01pn); @@ -187,7 +187,7 @@ knitr::include_graphics("../images/rmst/rmstreg_output3.png") The pseudo-observations method [Andersen, Hansen and Klein 2004](https://pubmed.ncbi.nlm.nih.gov/15690989/), is available in SAS using the method=pv option. You use the link=linear or link=log options and output is similarly interpreted as described for Method 1 IPCW method. -```{sas} +```sas proc rmstreg data=adcibc tau=&_tau; class trt01pn sex ; model time*cnsr(1) =trt01pn sex age /link=linear method=pv; @@ -200,7 +200,7 @@ run; A non-parametric method to calculate the RMST is available using the AUC Kaplan-Meier curves. -```{sas} +```sas proc lifetest data=adcibc plots=(rmst s) rmst (tau=&_tau); time time*cnsr(1) ; strata trt01pn / diff=control('2') ; diff --git a/SAS/sample_s_StatXact_test_of_trends.qmd b/SAS/sample_s_StatXact_test_of_trends.qmd index 179fbe6fa..eadd40a69 100644 --- a/SAS/sample_s_StatXact_test_of_trends.qmd +++ b/SAS/sample_s_StatXact_test_of_trends.qmd @@ -53,7 +53,7 @@ knitr::include_graphics("../images/samplesize/StatXact3.PNG") SAS code: -```{sas} +```sas proc sxpowerbin; tr/ex; palpha 0.025; @@ -97,7 +97,7 @@ knitr::include_graphics("../images/samplesize/StatXact5.PNG") SAS code: -```{sas} +```sas proc sxpowerbin; tr/ex dist_file=tr; palpha 0.05; @@ -140,7 +140,7 @@ What is the required sample size to achieve the power of 80% with the significan SAS code: -```{sas} +```sas proc sxpowerbin ti =15; tr/ex; palpha 0.05; diff --git a/SAS/sample_s_equivalence.qmd b/SAS/sample_s_equivalence.qmd index 162fc517b..592171986 100644 --- a/SAS/sample_s_equivalence.qmd +++ b/SAS/sample_s_equivalence.qmd @@ -30,7 +30,7 @@ For a mean $\mu_1$, we are testing if $\mu_1$ is equivalent to some value $\thet A reformulation of a treatment pill, needs to have a weight equivalent to a target value of $\theta$ =130 mg. Weight is assumed normally distributed and an acceptable weight is between 110 mg and 150 mg, hence $\delta=20mg$. The standard deviation of the weight is 50 mg. What sample size is needed assuming an alpha level of 5% with 80% power to conclude the weight is within the margin $\delta$ (the tablet weight is equivalent to 130 milligram). The below shows a sample size of 55 pills is required. -```{sas} +```sas PROC POWER ; onesamplemeans test=equiv lower = 110 @@ -69,7 +69,7 @@ It is anticipated that patients will have the same mean diastolic BP of 96 mmHg A total sample size of 90 is recommended, which equates to a sample size of 45 patients per treatment group. Notice how SAS asks for the `lower` and `upper` bounds, these are derived by using the meandiff $\theta$+/- the acceptable equivalence limit $\delta$ (which is stated as 5 mmHg above). -```{sas} +```sas PROC POWER ; twosamplemeans test=equiv_diff lower = -5 @@ -117,7 +117,7 @@ A client is interested in conducting a clinical trial to compare two cholesterol Below shows a sample size of 140 patients in Total (70 per treatment group). -```{sas} +```sas PROC POWER ; twosamplemeans test=equiv_diff lower = -0.04 @@ -156,7 +156,7 @@ Let's consider a standard standard two-sequence, two period crossover design for The below shows a sample size of 8 patients is required. -```{sas} +```sas PROC POWER; pairedmeans test=equiv_diff lower = -35 diff --git a/SAS/sample_s_noninferiority.qmd b/SAS/sample_s_noninferiority.qmd index 06186f7eb..86bfae55c 100644 --- a/SAS/sample_s_noninferiority.qmd +++ b/SAS/sample_s_noninferiority.qmd @@ -20,7 +20,7 @@ This example is a sample size calculation for the following hypotheses: $H_0:\mu A client is interested in conducting a clinical trial to compare two cholesterol lowering agents for treatment of hypercholesterolemic patients through a parallel design. The primary efficacy parameter is a low-density lipidprotein cholesterol (LDL-C). We will consider the situation where the intended trial is for testing noninferiority. For establishing it, suppose the true mean difference is 0 and the noninferiority margin is chosen to be -0.05 (-5%). Assuming SD = 0.1, how many patients are required for an 80% power and an overall significance level of 5%? -```{sas} +```sas PROC POWER; twosamplemeans test=equiv_diff @@ -56,7 +56,7 @@ Let's consider a standard two-sequence, two period crossover design. Suppose tha Alpha = 0.025 is used below, instead of 0.05 because you are doing non-inferiority (a one sided test). Note that this is still the sample size for alpha=0.05. The below shows a sample size of 13 patients is required. -```{sas} +```sas pairedmeans test=equiv_diff lower = -0.3 diff --git a/SAS/sample_s_superiority.qmd b/SAS/sample_s_superiority.qmd index 6d89a3605..2036fc96b 100644 --- a/SAS/sample_s_superiority.qmd +++ b/SAS/sample_s_superiority.qmd @@ -39,7 +39,7 @@ A client is interested in conducting a clinical trial to compare two cholesterol The code below estimates the sample size in SAS. NOTE: you can either specify the MEANDIFF=8 or if you know the separate group means X and Y, you can use GROUPMEANS =X\|Y code instead. SAS also assume a default alpha level of 0.05, a 1:1 balanced randomization and a Normal distribution. -```{sas} +```sas PROC POWER ; TWOSAMPLEMEANS TEST=DIFF MEANDIFF=8 @@ -70,7 +70,7 @@ Variance of the difference = 2x Variance within patient. Vardiff=2∗Varpatient We wish to run an AB/BA single dose crossover to compare two brochodilators. The primary outcome is peak expiratory flow, and a clinically relevant difference of 30 l/min is sought with 80% power, the significance level is 5% and the best estimate of the within patient standard deviation is 32 l/min. What size of trial do we require? (After recalculating: 32∗2=45 and assuming no period effect and assuming between each pair of measurements on the same subject that we have a 0.5 correlation) -```{sas} +```sas PROC POWER ; PAIREDMEANS TEST=DIFF NPAIRS=. diff --git a/SAS/summary-stats.qmd b/SAS/summary-stats.qmd index 0939d8073..c344e2361 100644 --- a/SAS/summary-stats.qmd +++ b/SAS/summary-stats.qmd @@ -8,7 +8,7 @@ Percentiles can be calculated in SAS using the UNIVARIATE procedure. The procedu This is how the 25th and 40th percentiles of `aval` in the dataset `adlb` could be calculated, using the default option for `PCTLDEF`. For quantiles, Q1= 25%, Q2=50%, Q3 = 75%, Q4=100%. -```{sas} +```sas proc univariate data=adlb; var aval; output out=stats pctlpts=25 40 pctlpre=p; diff --git a/SAS/summary_skew_kurt.qmd b/SAS/summary_skew_kurt.qmd index af473a6bc..c6cac61b8 100644 --- a/SAS/summary_skew_kurt.qmd +++ b/SAS/summary_skew_kurt.qmd @@ -20,7 +20,7 @@ In SAS, Skewness and Kurtosis are usually calculated using `PROC MEANS`. The pro The following data was used in this example. -```{sas} +```sas data dat; input team $ points assists; datalines; @@ -77,7 +77,7 @@ knitr::include_graphics("../images/summarystats/sas_kurtosis.png") Skewness and Kurtosis are commonly calculated in SAS as follows: -```{sas} +```sas proc means data=dat SKEWNESS KURTOSIS; var points; run; @@ -99,7 +99,7 @@ The above results correspond to the Type 2 methodology in R. The N option produces the following results -```{sas} +```sas proc means data=dat SKEWNESS KURTOSIS vardef = N; var points; run; diff --git a/SAS/survey-stats-summary.qmd b/SAS/survey-stats-summary.qmd index 84731fe7d..e444ba33d 100644 --- a/SAS/survey-stats-summary.qmd +++ b/SAS/survey-stats-summary.qmd @@ -41,7 +41,7 @@ head(apisrs) |> If we want to calculate a mean of a variable in a dataset which has been obtained from a **s**imple **r**andom **s**ample such as `apisrs`, in SAS we can do the following (*nb. here `total=6194` is obtained from the constant `fpc` column, and provides the finite population correction*): -```{sas} +```sas proc surveymeans data=apisrs total=6194 mean; var growth; run; @@ -68,7 +68,7 @@ run; To calculate population totals, we can request the `sum`. However SAS requires the user to specify the weights, otherwise the totals will be incorrect. These weights in this case are equivalent to the total population size divided by the sample size: -```{sas} +```sas data apisrs; set apisrs nobs=n; weight = fpc / n; @@ -102,7 +102,7 @@ growth 197589 12949 To perform ratio analysis for means or proportions of analysis variables in SAS, we can use the following: -```{sas} +```sas proc surveymeans data=apisrs total=6194; ratio api00 / api99; run; @@ -139,7 +139,7 @@ api00 api99 200 1.051066 0.003604 1.04395882 To calculate a proportion in SAS, we use the `PROC SURVEYFREQ`, in the simplest case below: -```{sas} +```sas proc surveyfreq data=apisrs total=6194; table 'sch.wide'n / cl; run; @@ -168,7 +168,7 @@ run; To calculate quantiles in SAS, we can use the `quantile` option to request specific quantiles, or can use keywords to request common quantiles (e.g. quartiles or the median). This will use Woodruff's method for confidence intervals, and a custom quantile method [@SAS_2018, pp. 9834]. -```{sas} +```sas proc surveymeans data=apisrs total=6194 quantile=(0.025 0.5 0.975); var growth; run; @@ -210,7 +210,7 @@ head(nhanes) |> To produce means and standard quartiles for this sample, taking account of sample design, we can use the following: -```{sas} +```sas proc surveymeans data=nhanes mean quartiles; cluster SDMVPSU; strata SDMVSTRA; @@ -252,7 +252,7 @@ run; To produce an analysis of separate subpopulations in SAS we can use the `DOMAIN` statement (note: do not use the `BY` statement as it will not give statistically valid analysis), here we also request the design effect: -```{sas} +```sas proc surveymeans data=nhanes mean deff; cluster SDMVPSU; strata SDMVSTRA; diff --git a/SAS/survival.qmd b/SAS/survival.qmd index d54e26a34..ca6d74dc6 100644 --- a/SAS/survival.qmd +++ b/SAS/survival.qmd @@ -52,7 +52,7 @@ The data include 500 subjects from the Worcester Heart Attack Study. This study - gender: males = 0, females = 1 - stratification factor -```{sas} +```sas libname mylib "..\data"; data dat; @@ -69,7 +69,7 @@ The KM estimators and log-rank test are from `PROC LIFETEST`, and Cox PH model i ### KM estimators and log-rank test -```{sas} +```sas proc lifetest data=dat outsurv=_SurvEst timelist= 1 3 5 reduceout stderr; time lenfoly*fstat(0); strata afb; @@ -98,7 +98,7 @@ knitr::include_graphics("../images/survival/sas_logrank.png") ### Cox PH model -```{sas} +```sas proc phreg data = dat; class afb; model lenfol*fstat(0) = afb/rl; @@ -119,7 +119,7 @@ knitr::include_graphics("../images/survival/sas_cox.png") In a stratified model, the Kaplan-Meier estimators remain the same as those in the non-stratified model. To implement stratified log-rank tests and Cox proportional hazards models, simply add the `STRATA` option in both `PROC LIFETEST` and `PROC PHREG`. -```{sas} +```sas # KM estimators and log-rank test proc lifetest data=dat; time lenfoly*fstat(0); diff --git a/SAS/survival_cif.qmd b/SAS/survival_cif.qmd index d72174b5c..c44542702 100644 --- a/SAS/survival_cif.qmd +++ b/SAS/survival_cif.qmd @@ -32,7 +32,7 @@ The bone marrow transplant (BTM) dataset as presented by Guo & So (2018) is used SAS code to prepare the data: -```{sas} +```sas proc format; value DiseaseGroup 1='ALL' 2='AML-Low Risk' @@ -56,7 +56,7 @@ run; PROC LIFETEST is used to estimate the CIFs in SAS. For illustration, we model the time to relapse. -```{sas} +```sas ods graphics on; proc lifetest data=bmt plots=cif(test) diff --git a/SAS/survival_csh.qmd b/SAS/survival_csh.qmd index 60761f36a..93a8234c1 100644 --- a/SAS/survival_csh.qmd +++ b/SAS/survival_csh.qmd @@ -38,7 +38,7 @@ The bone marrow transplant (BTM) dataset as presented by Guo & So (2018) is used - For illustration, a categorical variable `waitCat` is created from `waitTime` as `waitCat = TRUE` if `waitTime > 200`, and `FALSE` otherwise. -```{sas} +```sas proc format; value DiseaseGroup 1='ALL' 2='AML-Low Risk' @@ -65,7 +65,7 @@ run; Starting in SAS/STAT 14.3, all competing events can be estimated together. However, currently this syntax does not allow the `strata` statement. -```{sas} +```sas proc phreg data=Bmt; title 'Cause-Specific Hazard Regression for Relapse and Death without strata'; class Group (order=internal ref=first); @@ -100,7 +100,7 @@ For more information, please see [Guo C and So Y. (2018)](https://support.sas.co We use `Relapse` as an example. -```{sas} +```sas ods output ParameterEstimates=p1; proc phreg data=bmt; title 'Cause-Specific Hazard Regression for Relapse with strata'; diff --git a/SAS/tipping_point.qmd b/SAS/tipping_point.qmd index 75c19d0c4..67d0f1f37 100644 --- a/SAS/tipping_point.qmd +++ b/SAS/tipping_point.qmd @@ -46,7 +46,7 @@ The same publicly available [dataset](https://r-packages.io/datasets/antidepress The relevant endpoint for the antidepressant trial was assessed using the Hamilton 17-item depression rating scale (HAMD17), which was measured at baseline and subsequently at weeks 1, 2, 3, 4 and 6 (visits 4-7). Study drug discontinuation occurred in 24% (20/84) of subjects in the active drug group, compared to 26% (23/88) of subjects in the placebo group. Importantly, all data after study drug discontinuation are missing and there is a single intermittent missing observation. -```{sas} +```sas proc print data=dat (obs=10); var PATIENT GENDER THERAPY RELDAYS VISIT BASVAL HAMDTL17 CHANGE; run; @@ -63,7 +63,7 @@ knitr::include_graphics("../images/tipping_point/SAS_data_exploration_1.PNG") The number of patients per visit and treatment group are: -```{sas} +```sas proc freq data=dat; table VISIT*THERAPY / norow nocol nopercent nocum; run; @@ -80,7 +80,7 @@ knitr::include_graphics("../images/tipping_point/SAS_data_exploration_2.PNG") The mean change from baseline of the HAMD17 endpoint per visit and treatment group using only the complete cases are: -```{sas} +```sas proc means data=dat n mean nonobs; class VISIT THERAPY; var CHANGE; @@ -98,7 +98,7 @@ knitr::include_graphics("../images/tipping_point/SAS_data_exploration_3.PNG") The missingness pattern is: -```{sas} +```sas proc transpose data=dat out=HAMD_wide(drop=_NAME_) prefix=CHG; by PATIENT THERAPY BASVAL; id VISIT; @@ -141,7 +141,7 @@ As mentioned, we will illustrate the use of the so-called `five macros` in SAS f To conduct a tipping point analysis under the MAR assumption, we simply specify `method = MAR` under `Part2A()` of the `five macros`. Generally, the rest of `Part1` and `Part2` are the same as in the scenario without any delta adjustment. -```{sas} +```sas %part1A(jobname = HAMD, Data=dat, Subject=PATIENT, @@ -174,7 +174,7 @@ A clear description of these arguments is given in the documentation: To automate the tipping point analysis, you can create a new macro like shown below. The first part of this macro prints all results, while the second part prints the non-significant and significant results separately by filtering on `Probt`. -```{sas} +```sas data all_results; length DELTA 8 VISIT $10 THERAPY $20 _THERAPY $20 Diff SE_Diff df Probt LCL_Diff UCL_Diff 8; stop; @@ -209,7 +209,7 @@ run; knitr::include_graphics("../images/tipping_point/SAS_MAR_all_results.png") ``` -```{sas} +```sas proc sql noprint; create table delta_ge_05 as @@ -376,7 +376,7 @@ Let's now consider the antidepressant data again. Suppose we apply a delta adjus To program this, we would define `Delta`, `DLag` and `DGroups` in `Part3()` of the `five macros` as follows: -```{sas} +```sas %part3(Jobname = HAMD_MAR, anref=PLACEBO, Delta = 2 2 2 2, @@ -400,7 +400,7 @@ As already illustrated in the tipping point analysis assuming MAR above, you may To apply this delta = 5 to both groups we leave `DGroups` unspecified. -```{sas} +```sas %part3(Jobname = HAMD_MAR, anref=PLACEBO, Delta = 5 5 5 5, diff --git a/SAS/tobit regression SAS.qmd b/SAS/tobit regression SAS.qmd index 1c7bb9b38..465b716fc 100644 --- a/SAS/tobit regression SAS.qmd +++ b/SAS/tobit regression SAS.qmd @@ -26,7 +26,7 @@ $$ with $\epsilon_{i} \sim N(0,\sigma^2)$. But we only observe $y = max(\tau, y^ We assume two equally sized groups (n=10 in each group). The data is censored on the left at a value of $\tau=8.0$. In group A 4/10 records are censored, and 1/10 in group B. -```{sas} +```sas data dat_used; input ID$ ARM$ Y CENS; cards; @@ -60,7 +60,7 @@ The analysis will be based on a Tobit analysis of variance with $Y$, rounded to First a data manipulation step needs to be performed in which the censored values are set to missing for a new variable called *lower*. -```{sas} +```sas data dat_used; set dat_used; if Y <= 8.0 then lower=.; else lower=Y; @@ -69,7 +69,7 @@ run; The data are sorted to make sure the intercept will correspond to the mean of ARM A. -```{sas} +```sas proc sort data=dat_used; by descending ARM; run; @@ -83,7 +83,7 @@ The **LIFEREG** procedure is used for tobit regression. The following model synt Here, if the *lower* value is missing, then the *upper* value is used as a left-censored value. -```{sas} +```sas proc lifereg data=dat_used order=data; class ARM; model (lower, Y) = ARM / d=normal; diff --git a/SAS/ttest_1Sample.qmd b/SAS/ttest_1Sample.qmd index 4071f9eb3..175d41596 100644 --- a/SAS/ttest_1Sample.qmd +++ b/SAS/ttest_1Sample.qmd @@ -18,7 +18,7 @@ In SAS, a one sample t-test is usually performed using PROC TTEST. The one sampl The following data was used in this example. -```{sas} +```sas data read; input score count @@; datalines; @@ -39,7 +39,7 @@ By default, SAS PROC TTEST t-test assumes normality in the data and uses a class The following code was used to test the comparison of a reading scores against a baseline hypothesis value of 30: -```{sas} +```sas proc ttest data=read h0=30; var score; run; @@ -63,7 +63,7 @@ The SAS one sample t-test also supports lognormal analysis for a one sample t-te Using the same data as above, we will set the "DIST" option to "lognormal" to perform this analysis: -```{sas} +```sas proc ttest data=read h0=30 dist=lognormal; var score; run; diff --git a/SAS/ttest_2Sample.qmd b/SAS/ttest_2Sample.qmd index 93e575a8f..79e9dffcd 100644 --- a/SAS/ttest_2Sample.qmd +++ b/SAS/ttest_2Sample.qmd @@ -16,7 +16,7 @@ knitr::opts_chunk$set(echo = TRUE) The following data was used in this example. -```{sas} +```sas data d1; length trt_grp $ 9; input trt_grp $ WtGain @@; @@ -42,7 +42,7 @@ Both the Student's t-test and Welch's t-test (the Satterthwaite approximation is For this example, we're testing the significant difference in mean of Weight gain (*WtGain*) between treatment and placebo (*trt_grp*) using PROC TTEST procedure in SAS. -```{sas} +```sas proc ttest data=d1; class trt_grp; var WtGain; @@ -71,7 +71,7 @@ Note: Before entering straight into the t-test we need to check whether the assu 1. Normality: You can check for data to be normally distributed by plotting a histogram of the data by treatment. Alternatively, you can use the Shapiro-Wilk test or the Kolmogorov-Smirnov test. If the test is \<0.05 and your sample is quite small then this suggests you should not use the t-test. However, if your sample in each treatment group is large (say \>30 in each group), then you do not need to rely so heavily on the assumption that the data have an underlying normal distribution in order to apply the two-sample t-test. This is where plotting the data using histograms can help to support investigation into the normality assumption. We have checked the normality of the observations using the code below. Here for both the treatment groups we have P value greater than 0.05 (Shapiro-Wilk test is used), therefore the normality assumption is there for our data. -```{sas} +```sas proc univariate data=d1 normal; qqplot WtGain; by trt_grp; @@ -125,7 +125,7 @@ knitr::include_graphics("../images/ttest/variance_sas.png") For this example, it is important to use the Welch's t-test (the Satterthwaite approximation is used to calculate the effective degrees of freedom and variance) results. -```{sas} +```sas data d2; length trt_grp $ 9; input trt_grp $ WtGain @@; diff --git a/SAS/ttest_Paired.qmd b/SAS/ttest_Paired.qmd index ac2962c5e..d5791bcc4 100644 --- a/SAS/ttest_Paired.qmd +++ b/SAS/ttest_Paired.qmd @@ -25,7 +25,7 @@ By default, SAS PROC TTEST t-test assumes normality in the data and uses a class The following data was used in this example. -```{sas} +```sas data pressure; input SBPbefore SBPafter @@; datalines; @@ -39,7 +39,7 @@ data pressure; The following code was used to test the comparison of two paired samples of Systolic Blood Pressure before and after a procedure. -```{sas} +```sas proc ttest data=pressure; paired SBPbefore*SBPafter; run; @@ -61,7 +61,7 @@ The SAS paired t-test also supports analysis of lognormal data. Here is the data ### Data -```{sas} +```sas data auc; input TestAUC RefAUC @@; datalines; @@ -75,7 +75,7 @@ data auc; For cases when the data is lognormal, SAS offers the "DIST" option to chose between a normal and lognormal distribution. The procedure also offers the TOST option to specify the equivalence bounds. -```{sas} +```sas proc ttest data=auc dist=lognormal tost(0.8, 1.25); paired TestAUC*RefAUC; run; diff --git a/SAS/wilcoxonsr_HL.qmd b/SAS/wilcoxonsr_HL.qmd index 60e490f72..19eef4ec0 100644 --- a/SAS/wilcoxonsr_HL.qmd +++ b/SAS/wilcoxonsr_HL.qmd @@ -20,7 +20,7 @@ Again, wilcoxon signed rank test was applied to analyse the time to return to ba Let's consider a case where the dataset has no ties and N (number of observations) = 240. -```{sas} +```sas data TTR; set TTR; diff = TRT_B - TRT_A; @@ -37,7 +37,7 @@ knitr::include_graphics("../images/wilcoxonsr/wsr_data.PNG") In SAS Wilcoxon Signed-Rank test is available using PROC UNIVARIATE. -```{sas} +```sas proc univariate data=TTR alpha=0.1; var diff; @@ -55,14 +55,14 @@ knitr::include_graphics("../images/wilcoxonsr/wsr_240.PNG") Now let's consider a smaller dataset, created by selecting first 19 observations from our main data. -```{sas} +```sas data TTR_19; set TTR; if _N_ <= 19; run; ``` -```{sas} +```sas proc univariate data=TTR_19 alpha=0.1; var diff; @@ -98,7 +98,7 @@ StatXact® PROCs for SAS users is a clinical trial analysis software from Cytel ### Dataset without ties and N \> 20 {.unnumbered} -```{sas} +```sas /* Wilxocon S-R test - p values */ PROC PAIRED DATA=WilcoxonSignedRank_TTR ALPHA=0.9; @@ -115,7 +115,7 @@ RUN; knitr::include_graphics("../images/wilcoxonsr/wsrSX_240a.PNG") ``` -```{sas} +```sas /* Wilcoxon S-R - H-L estimator and CI */ PROC PAIRED DATA=WilcoxonSignedRank_TTR ALPHA=0.9; diff --git a/templates/multi_language_template.qmd b/templates/multi_language_template.qmd index a72a9ccc7..67b0b1f3a 100644 --- a/templates/multi_language_template.qmd +++ b/templates/multi_language_template.qmd @@ -64,7 +64,7 @@ example_data <- data.frame( ## SAS -```{sas} +```sas /* Set random seed for reproducibility */ data example_data; call streaminit(123); @@ -110,7 +110,7 @@ summary(example_model) ### SAS -```{sas} +```sas /* SAS code for basic Poisson Regression */ proc genmod data=example_data; class predictor; @@ -144,7 +144,7 @@ summary(alternative_model) ### SAS -```{sas} +```sas /* SAS code for handling overdispersion */ proc genmod data=example_data; class predictor; diff --git a/templates/single_language_template.qmd b/templates/single_language_template.qmd index 282ed4d6c..a136831aa 100644 --- a/templates/single_language_template.qmd +++ b/templates/single_language_template.qmd @@ -39,7 +39,7 @@ exData <- tibble::tribble( ## SAS -```{sas} +```sas data ExData; input Var1 $ Var2 ...; datalines; From 5fd660ac6e00890c7016a29f9b00a11f51055e27 Mon Sep 17 00:00:00 2001 From: Michael Walshe <62556482+michaelwalshe@users.noreply.github.com> Date: Tue, 10 Mar 2026 10:22:51 +0000 Subject: [PATCH 02/13] Remove eval: false options in headers, eval: true options in code blocks --- Comp/r-east_gsd_tte.qmd | 16 ---------------- Comp/r-sas-python_survey-stats-summary.qmd | 9 --------- Comp/r-sas-summary-stats.qmd | 3 --- Comp/r-sas_anova.qmd | 5 ----- Comp/r-sas_chi-sq.qmd | 5 ----- Comp/r-sas_friedman.qmd | 6 ------ Comp/r-sas_gee.qmd | 7 ------- Comp/r-sas_kruskalwallis.qmd | 2 -- Comp/r-sas_logistic-regr.qmd | 8 -------- Comp/r-sas_mcnemar.qmd | 6 ------ Comp/r-sas_mmrm.qmd | 16 ++++------------ Comp/r-sas_negbin.qmd | 12 ++---------- Comp/r-sas_psmatch.qmd | 2 -- Comp/r-sas_survival.qmd | 10 ---------- SAS/SAS_Friedmantest.qmd | 2 -- SAS/ancova.qmd | 4 ---- SAS/anova.qmd | 7 ------- SAS/association.qmd | 3 --- SAS/binomial_test.qmd | 2 -- SAS/ci_for_2indep_prop.qmd | 2 -- SAS/ci_for_paired_prop.qmd | 2 -- SAS/ci_for_prop.qmd | 6 ------ SAS/cmh.qmd | 7 ------- SAS/correlation.qmd | 2 -- SAS/count_data_regression.qmd | 4 ---- SAS/gee.qmd | 7 ------- SAS/gsd-tte.qmd | 3 --- SAS/jonchkheere_terpstra.qmd | 2 -- SAS/kruskal_wallis.qmd | 3 --- SAS/linear-regression.qmd | 2 -- SAS/logistic-regr.qmd | 2 -- SAS/manova.qmd | 11 ----------- SAS/mcnemar.qmd | 3 --- SAS/mi_mar_regression.qmd | 7 ------- SAS/mmrm.qmd | 2 -- SAS/nparestimate.qmd | 2 -- SAS/ranksum.qmd | 2 -- SAS/rbmi_continuous_joint_SAS.qmd | 14 -------------- SAS/recurrent_events.qmd | 21 --------------------- SAS/rmst.qmd | 9 --------- SAS/rounding.qmd | 14 -------------- SAS/sample_s_StatXact_test_of_trends.qmd | 9 --------- SAS/sample_s_equivalence.qmd | 7 ------- SAS/sample_s_noninferiority.qmd | 5 ----- SAS/sample_s_superiority.qmd | 5 ----- SAS/summary-stats.qmd | 2 -- SAS/summary_skew_kurt.qmd | 7 ------- SAS/survey-stats-summary.qmd | 5 ----- SAS/survival.qmd | 6 ------ SAS/survival_cif.qmd | 5 ----- SAS/survival_csh.qmd | 4 ---- SAS/tipping_point.qmd | 11 ----------- SAS/tobit regression SAS.qmd | 4 ---- SAS/ttest_1Sample.qmd | 5 ----- SAS/ttest_2Sample.qmd | 8 -------- SAS/ttest_Paired.qmd | 5 ----- SAS/wilcoxonsr_HL.qmd | 6 ------ 57 files changed, 6 insertions(+), 340 deletions(-) diff --git a/Comp/r-east_gsd_tte.qmd b/Comp/r-east_gsd_tte.qmd index c63f18357..884c14616 100644 --- a/Comp/r-east_gsd_tte.qmd +++ b/Comp/r-east_gsd_tte.qmd @@ -2,8 +2,6 @@ title: "R vs EAST vs SAS: Group sequential design" editor_options: chunk_output_type: console -execute: - eval: false --- ## Introduction @@ -30,7 +28,6 @@ We assume that a GSD is utilized for progression-free survival (PFS) endpoint. I Further design assumptions are as follows: ```{r} -#| eval: true # PFS HR=0.6 hr1_pfs <- 0.6 # median PFS of 9.4 months in the control arm @@ -62,7 +59,6 @@ Note that, in EAST the number of target events is reported as an integer, howeve For ease of comparison the results from EAST are summarized below: ```{r} -#| eval: true #| echo: false #| warning: false library(flextable) @@ -106,7 +102,6 @@ pfs_east |> - gsDesign code to reproduce the above EAST results: ```{r} -#| eval: true #| warning: false library(gsDesign) @@ -136,7 +131,6 @@ pfs_gsDesign |> - gsDesign vs EAST comparison using absolute differences: ```{r} -#| eval: true #| echo: false digit_comp <- 4 pfs_gsDesign |> @@ -191,7 +185,6 @@ pfs_gsDesign |> - Note that, here `gsDesign2::gs_power_ahr()` is used given the number of target events for each analysis based on EAST results. ```{r} -#| eval: true #| echo: false #helper function to align the gsDesign2 summary with gsDesign summary as_gs <- function(xnph) { @@ -318,7 +311,6 @@ as_gs <- function(xnph) { ``` ```{r} -#| eval: true #| warning: false #| message: false library(gsDesign2) @@ -365,7 +357,6 @@ pfs_gsDesign2 |> - gsDesign2 vs EAST comparison using absolute differences: ```{r} -#| eval: true #| echo: false pfs_gsDesign2 |> as_gs() |> @@ -418,7 +409,6 @@ pfs_gsDesign2 |> - rpact code to reproduce the above EAST results appears below. ```{r} -#| eval: true #| warning: false library(rpact) @@ -451,7 +441,6 @@ kable(summary(pfs_rpact)) ```{r} #| echo: false -#| eval: true pcross_h1_eff <- cumsum(pfs_rpact$rejectPerStage) pcross_h1_fut <- pfs_rpact$futilityPerStage[1] @@ -572,7 +561,6 @@ RUN; The following shows the events (D) and required sample sizes (N) for IA and FA. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -599,7 +587,6 @@ RUN; The HR boundaries are shown below. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -611,7 +598,6 @@ knitr::include_graphics( The results calculated by SAS are presneted in the table below. Please note that SAS doesn't report the probablities $P(Cross | HR=1)$ and $P(Cross | HR=0.6)$, resulting in empty cells for these results in the table. ```{r} -#| eval: true #| echo: false #| warning: false library(flextable) @@ -648,7 +634,6 @@ pfs_sas |> - SAS vs rapct comparison using absolute differences: ```{r} -#| eval: true #| echo: false #| warning: false pfs_rpact_sum |> @@ -694,7 +679,6 @@ pfs_rpact_sum |> - SAS vs EAST comparison using absolute differences: ```{r} -#| eval: true #| echo: false pfs_sas_sum <- tibble::tibble( sas_eff = pfs_sas$eff_125, diff --git a/Comp/r-sas-python_survey-stats-summary.qmd b/Comp/r-sas-python_survey-stats-summary.qmd index f1b5c5411..b69bee4af 100644 --- a/Comp/r-sas-python_survey-stats-summary.qmd +++ b/Comp/r-sas-python_survey-stats-summary.qmd @@ -1,8 +1,6 @@ --- title: "R vs SAS vs Python Survey Summary Statistics" bibliography: survey-stats-summary.bib -execute: - eval: false --- This document will compare the survey summary statistics functionality in SAS (available through SAS/STAT), R (available from the [`{survey}`](%5B%60%7Bsurvey%7D%60%5D(https://r-survey.r-forge.r-project.org/survey/html/api.html)) package), and Python (available from the [`samplics`](https://samplics-org.github.io/samplics/) package), highlighting differences in methods and results. Only the default Taylor series linearisation method for calculating variances is used in all languages. A more detailed comparison between R and SAS for specific methods and use-cases is available in [@2017_YRBS], [@so2020modelling], or [@adamico_2009]. For a general guide to survey statistics, which has companion guides for both R and SAS, see [@Lohr_2022]. @@ -31,7 +29,6 @@ For the full R, SAS, and Python code and results used for this comparison, see b ## R ```{r} -#| eval: true #| message: false #| warning: false library(survey) @@ -277,7 +274,6 @@ run; ## Python ```{python} -#| eval: true import pandas as pd from samplics import TaylorEstimator from samplics.utils.types import PopParam @@ -363,7 +359,6 @@ print( `samplics` in Python does not have a method for calculating quantiles, and in R and SAS the available methods lead to different results. To demonstrate the differences in calculating quantiles, we will use the `apisrs` dataset from the `survey` package in R [@API_2000]. ```{r} -#| eval: true #| message: false library(survey) @@ -407,7 +402,6 @@ run; If in R we use the default `qrule="math"` (equivalent to `qrule="hf1"` and matches `type=1` in the `quantile` function for unweighted data) along with the default `interval.type="mean"`, we get the following results: ```{r} -#| eval: true srs_design <- survey::svydesign(data = apisrs, id = ~1, fpc = ~fpc, ) survey::svyquantile( @@ -422,7 +416,6 @@ survey::svyquantile( Here we can see that the quantiles, confidence intervals, and standard errors do not match SAS. From testing, none of the available `qrule` methods match SAS for the quantile values, so it is recommended to use the default values unless you have need of some of the other properties of different quantile definitions - see [`vignette("qrule", package="survey")`](https://cran.r-project.org/web/packages/survey/vignettes/qrule.pdf) for more detail. If an exact match to SAS is required, then the `svyquantile` function allows for passing a custom function to the `qrule` argument to define your own method for calculating quantiles. Below is an example that will match SAS: ```{r} -#| eval: true sas_qrule <- function(x, w, p) { # Custom qrule to match SAS, based on survey::oldsvyquantile's internal method if (any(is.na(x))) { @@ -459,7 +452,6 @@ sas_quants Note that although the quantiles and standard errors match, the confidence intervals still do not match SAS. For this another custom calculation is required, based on the formula used in SAS: ```{r} -#| eval: true sas_quantile_confint <- function(newsvyquantile, level = 0.05, df = Inf) { q <- coef(newsvyquantile) se <- survey::SE(newsvyquantile) @@ -503,7 +495,6 @@ In contrast, the `samplics` package in Python is still early in development, and ::: {.callout-note collapse="true" title="Session Info"} ```{r} -#| eval: true #| echo: false si <- sessioninfo::session_info("survey", dependencies = FALSE) # If reticulate is used, si will include python info. However, this doesn't diff --git a/Comp/r-sas-summary-stats.qmd b/Comp/r-sas-summary-stats.qmd index 82c6bd5fc..441981562 100644 --- a/Comp/r-sas-summary-stats.qmd +++ b/Comp/r-sas-summary-stats.qmd @@ -1,6 +1,5 @@ --- title: "Deriving Quantiles or Percentiles in R vs SAS" -eval: false --- ### Data @@ -45,7 +44,6 @@ This gives the following output. ```{r} #| echo: false -#| eval: true adlb <- data.frame(aval = c(10, 20, 30, 40, 150, 160, 170, 180, 190, 200)) quantile(adlb$aval, probs = c(0.25, 0.4)) ``` @@ -66,7 +64,6 @@ This gives the following output. ```{r} #| echo: false -#| eval: true quantile(adlb$aval, probs = c(0.25, 0.4), type = 2) ``` diff --git a/Comp/r-sas_anova.qmd b/Comp/r-sas_anova.qmd index 71519e61c..c659f6d37 100644 --- a/Comp/r-sas_anova.qmd +++ b/Comp/r-sas_anova.qmd @@ -1,7 +1,5 @@ --- title: "R vs SAS Linear Models" -execute: - eval: false --- # R vs. SAS ANOVA @@ -31,7 +29,6 @@ The following table provides an overview of the support and results comparabilit In order to get the ANOVA model fit and sum of squares you can use the `anova` function in the `stats` package. ```{r} -#| eval: true library(emmeans) drug_trial <- read.csv("../data/drug_trial.csv") @@ -43,7 +40,6 @@ lm_model |> It is recommended to use the `emmeans` package to get the contrasts between R. ```{r} -#| eval: true lm_model |> emmeans("drug") |> contrast( @@ -66,7 +62,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% diff --git a/Comp/r-sas_chi-sq.qmd b/Comp/r-sas_chi-sq.qmd index a9915a4b7..4c30d119d 100644 --- a/Comp/r-sas_chi-sq.qmd +++ b/Comp/r-sas_chi-sq.qmd @@ -1,7 +1,5 @@ --- title: "R/SAS Chi-Squared and Fisher's Exact Comparision" -execute: - eval: false --- # Chi-Squared Test @@ -26,7 +24,6 @@ For an r x c table (where r is the number of rows and c the number of columns), For this example we will use data about cough symptoms and history of bronchitis. ```{r} -#| eval: true bronch <- matrix(c(26, 247, 44, 1002), ncol = 2) row.names(bronch) <- c("cough", "no cough") colnames(bronch) <- c("bronchitis", "no bronchitis") @@ -36,7 +33,6 @@ bronch To a chi-squared test in R you will use the following code. ```{r} -#| eval: true stats::chisq.test(bronch) ``` @@ -51,7 +47,6 @@ run; The result in the "Chi-Square" section of the results table in SAS will not match R, in this case it will be 12.1804 with a p-value of 0.0005. This is because by default R does a Yate's continuity adjustment. To change this set `correct` to false. ```{r} -#| eval: true stats::chisq.test(bronch, correct = FALSE) ``` diff --git a/Comp/r-sas_friedman.qmd b/Comp/r-sas_friedman.qmd index 2cd2d0edc..cc78c365e 100644 --- a/Comp/r-sas_friedman.qmd +++ b/Comp/r-sas_friedman.qmd @@ -1,11 +1,8 @@ --- title: "R vs SAS Non-parametric Analysis - Friedman test" -execute: - eval: false --- ```{r} -#| eval: true #| label: review-setup #| message: false #| warning: false @@ -19,7 +16,6 @@ library(ggpubr) Friedman's test is used when you have one within-subjects independent variable with two or more levels and a dependent variable that is not interval and normally distributed (but at least ordinal). To build such unreplicated blocked data, we'll create a data frame called  `df_bp` from random number. In  `df_bp` : dependent variable `bp` is randomly generated; Block: `subjid` ; Group: `time_point`. ```{r} -#| eval: true set.seed(123) df_bp = data.frame(bp = runif(n = 50, 138, 200)) |> @@ -42,7 +38,6 @@ ggpubr::ggboxplot(df_bp, x = "time_point", y = "bp", add = "jitter") In R, **friedman_test** can be used to compare multiple means of rank in `bp` grouped by `time_point`, stratified by `subjid`. ```{r} -#| eval: true res.fried <- df_bp |> friedman_test(bp ~ time_point | subjid) res.fried @@ -87,7 +82,6 @@ SAS `PROC FREQ` documentation: mutate( @@ -105,7 +103,6 @@ glimpse(lung2) We use the `trial01` dataset provided with {beeca} R package. Initial data preparation involves setting the treatment indicator as a categorical variable and removing any incomplete cases. ```{r} -#| eval: true data("trial01") trial01$trtp <- factor(trial01$trtp) ## set treatment to a factor @@ -145,7 +142,6 @@ Note, the default fitting method in `glm` is consistent with the default fitting - Default fitting method for `PROC LOGISTIC` procedure is Fisher's scoring method, which is reported as part of the SAS default output, and it is equivalent to "Iteratively reweighted least squares" method as reported in this [documentation](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_logistic_sect033.htm). ```{r} -#| eval: true m1 <- stats::glm( wt_grp ~ age + sex + ph.ecog + meal.cal, data = lung2, @@ -159,7 +155,6 @@ summary(m1)$coefficients Note, function `confint.default` gives the Wald confidence limits, which is the default option in SAS `PROC LOGISTIC` procedure; whereas `confint` gives the profile-likelihood limits. Conditional odds ratio is calculated by taking the exponential of the model parameters. ```{r} -#| eval: true cbind(est = coef(m1), confint.default(m1)) ``` @@ -219,7 +214,6 @@ Note that while Firth logistic regression is not required for our example datase - By default, `logistf` function in R computes the confidence interval estimates and hypothesis tests (including p-value) for each parameter based on profile likelihood, which is also reported in the output below. However, Wald method (confidence interval and tests) can be specified by specifying the `control` argument with [`pl = FALSE`](https://cran.r-project.org/web/packages/logistf/logistf.pdf). ```{r} -#| eval: true firth_mod <- logistf( wt_grp ~ age + sex + ph.ecog + meal.cal, data = lung2, @@ -239,7 +233,6 @@ summary(firth_mod)$coefficients Note, function `confint` gives the profile-likelihood limits. Given the parameters from Firth's bias-reduced logistic regression is estimated using penalized maximum likelihood, `confint` function is used. Conditional odds ratio is calculated by taking the exponential of the model parameters. ```{r} -#| eval: true cbind(est = coef(firth_mod), confint(firth_mod)) ``` @@ -286,7 +279,6 @@ We compare two implementions of g-computation in SAS: We fit a logistic regression model with covariate adjustment to estimate the marginal treatment effect using the delta method for variance estimation: as outlined in Ge et al (2011). ```{r} -#| eval: true #| label: glm ## fit the model including model based variance estimation with delta method fit1 <- stats::glm(aval ~ trtp + bl_cov, family = "binomial", data = trial01) |> diff --git a/Comp/r-sas_mcnemar.qmd b/Comp/r-sas_mcnemar.qmd index 41fbb4982..fe5f0cf36 100644 --- a/Comp/r-sas_mcnemar.qmd +++ b/Comp/r-sas_mcnemar.qmd @@ -1,7 +1,5 @@ --- title: "R v SAS McNemar's test" -execute: - eval: false --- ## Introduction @@ -20,7 +18,6 @@ The following table provides an overview of the support and results comparabilit In R,the {stats} or the {coin} package can be used to calculate McNemar. The {coin} package has the same defaults as SAS. But, using either of these packages, the first step is to calculate a frequency table, using the table function. ```{r} -#| eval: true library(coin) colds <- read.csv( @@ -35,7 +32,6 @@ coin::mh_test(freq_tbl) In order to get Cohen's Kappa an additional package is needed. ```{r} -#| eval: true library(vcd) cohen_kappa <- vcd::Kappa(freq_tbl) @@ -52,7 +48,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 40% @@ -69,7 +64,6 @@ There is another R package that is sometimes used to calculate McNemar's, called ::: {.callout-note collapse="true" title="Session Info"} ```{r} -#| eval: true #| echo: false si <- sessioninfo::session_info( c("coin", "stats"), #Vector of packages used diff --git a/Comp/r-sas_mmrm.qmd b/Comp/r-sas_mmrm.qmd index 5682fe25b..714f13d1d 100644 --- a/Comp/r-sas_mmrm.qmd +++ b/Comp/r-sas_mmrm.qmd @@ -1,14 +1,13 @@ --- title: "R vs SAS MMRM" -message: false -warning: false -echo: true -eval: false +execute: + message: false + warning: false + echo: true --- ```{r} #| include: false -#| eval: true knitr::opts_chunk$set( collapse = TRUE, comment = "#>", @@ -21,7 +20,6 @@ knitr::opts_chunk$set( ```{r} #| label: review-setup #| include: false -#| eval: true library(dplyr) library(purrr) @@ -484,7 +482,6 @@ The `mmrm`, `PROC GLIMMIX`, `gls`, `lmer`, and `glmmTMB` functions are applied t ```{r} #| echo: false -#| eval: true # format table in markdown cached_mmrm_results$conv_time_fev |> @@ -509,7 +506,6 @@ The MMRM implementations are now applied to the BCVA dataset 10 times. The conve ```{r} #| echo: false -#| eval: true # format table in markdown cached_mmrm_results$conv_time_bcva |> arrange(median) |> @@ -536,7 +532,6 @@ We next estimate the marginal mean treatment effects for each visit in the FEV a ```{r} #| label: review-treatment-fev #| echo: false -#| eval: true # plot estimates ggplot( cached_mmrm_results$rel_diff_ests_tbl_fev, @@ -559,7 +554,6 @@ The R procedures' estimates are very similar to those output by `PROC GLIMMIX`, ```{r} #| label: review-treatment-bcva #| echo: false -#| eval: true # plot estimates ggplot( cached_mmrm_results$rel_diff_ests_tbl_bcva, @@ -604,7 +598,6 @@ Ten datasets of 200 patients are generated each of the following levels of missi ```{r} #| label: review-missingness-table #| echo: false -#| eval: true ## construct the table cached_mmrm_results$df_missingness |> kable(caption = "Number of patients per visit") @@ -615,7 +608,6 @@ The convergence rates of all implementations for stratified by missingness level ```{r} #| label: review-convergence-rate-missingness #| echo: false -#| eval: true ## plot the convergence rates cached_mmrm_results$conv_rate |> mutate( diff --git a/Comp/r-sas_negbin.qmd b/Comp/r-sas_negbin.qmd index 44d5484bc..feb2d91a6 100644 --- a/Comp/r-sas_negbin.qmd +++ b/Comp/r-sas_negbin.qmd @@ -2,8 +2,8 @@ title: "R vs SAS: Negative Binomial Regression" format: html toc: true -echo: true -eval: false +execute: + echo: true keep-hidden: true --- @@ -96,7 +96,6 @@ In order to run these analyses we need to load a few packages. ```{r} #| output: false -#| eval: true #| message: false library(MASS) library(dplyr) @@ -108,7 +107,6 @@ We also define the `glm_nb_cov` function to obtain the SAS variance-covariance m ```{r} #| output: false #| message: false -#| eval: true ## Helper function to compute the variance from negative binomial regression ## This matches with variance estimated from SAS glm_nb_cov <- function(mod) { @@ -183,7 +181,6 @@ A dummy dataset is simulated, including The dummy dataset is saved as a csv file, and then the csv file is read into SAS. ```{r} -#| eval: true N = 100 # set seed for replication @@ -253,7 +250,6 @@ Below is a screenshot of output tables summarizing coefficient estimates and lsm Lets now try to reproduce the results in R using `MASS::glm.nb`. ```{r} -#| eval: true fit <- glm.nb(y ~ grpc + x1 + x2 + offset(logtime), data = df, x = TRUE) # model coefficients summary @@ -263,7 +259,6 @@ summary(fit)$coefficients We can see that while the estimates are exactly matching those in SAS, the standard errors are slightly smaller. This is a result of the difference in covariance estimation mentioned above. To obtain exactly the same results as in SAS we need to re-estimate the covariance matrix using the `glm_nb_cov` function we defined earlier. Note that to use this function with the fitted results we needed to specify `x = TRUE` in the `glm.nb` function so that the design matrix is available. ```{r} -#| eval: true sigma_hat <- glm_nb_cov(fit) ## recalculate confidence intervals, and p-values @@ -293,7 +288,6 @@ new_summary Now the estimates, standard errors, 95% confidence interval limits and p-values are exactly matching those in SAS up to the 4th digit. We can also provide an estimate and CI for the dispersion parameter: ```{r} -#| eval: true # estimate and 95%-CI for k = 1/theta theta_est <- fit$theta theta_se <- sqrt(sigma_hat[6, 6]) @@ -311,7 +305,6 @@ We see that while the point estimate is the same as in SAS, the CI for the dispe Finally we can replicate the estimation of lsmeans in SAS via the emmeans package. Note that we need to supply the re-estimated covariance matrix, but only provide the rows and columns for the model coefficients without the dispersion parameter as emmeans does not need the latter. ```{r} -#| eval: true # lsmeans with weights = equal, equivalent to SAS default lsmean1 <- emmeans( fit, @@ -338,7 +331,6 @@ lsmean2 Estimates and CIs are exactly matching those in SAS for both of the options. Finally we can also obtain the z statistic and corresponding p-values: ```{r} -#| eval: true test(lsmean1) test(lsmean2) ``` diff --git a/Comp/r-sas_psmatch.qmd b/Comp/r-sas_psmatch.qmd index 4211e20be..a347467e2 100644 --- a/Comp/r-sas_psmatch.qmd +++ b/Comp/r-sas_psmatch.qmd @@ -1,7 +1,5 @@ --- title: "Propensity Score Matching" -execute: - eval: false --- # Introduction diff --git a/Comp/r-sas_survival.qmd b/Comp/r-sas_survival.qmd index fd0fee860..f337663b0 100644 --- a/Comp/r-sas_survival.qmd +++ b/Comp/r-sas_survival.qmd @@ -1,7 +1,5 @@ --- title: "R vs SAS - Kaplan Meier and Cox-proportion hazards modelling" -execute: - eval: false --- # Comparison of SAS vs R @@ -29,7 +27,6 @@ Results from the examples shown for R [here](https://psiaims.github.io/CAMIS/R/s Comparing the non-stratified model results side-by-side, the CIs for the quartile estimates and landmark estimates are different between R and SAS. HR and CI also have slight differences. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -102,7 +99,6 @@ From a [reference](https://myweb.uiowa.edu/pbreheny/7210/f15/notes/9-10.pdf): Th Now if we change the confidence interval type in SAS to "log" and tie handling to "efron", the results will be identical to the results in R. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -112,7 +108,6 @@ knitr::include_graphics("../images/survival/r_sas_chg_default.png") Below is the side-by-side comparison for stratified analysis with default methods in SAS matched to R's, the results are also identical. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -156,7 +151,6 @@ The convergence criterion details are described in their documentation: Now we look at other cases when the data has some special type which causes a mismatch between SAS and R. Suppose a dataset has 10 observations, and the first 5 are all events, and the last 5 are all censored. ```{r} -#| eval: true test <- tibble( time = c(54, 75, 77, 84, 87, 92, 103, 105, 112, 118), status = c(1, 1, 1, 1, 1, 0, 0, 0, 0, 0) @@ -197,7 +191,6 @@ run; Below is the side-by-side comparison: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -215,7 +208,6 @@ The kth quantile for a survival curve S(t) is the location at which a horizontal For example, using the data above, the survival probability is exactly 0.5 at time=87 and remains at 0.5 until the last censored observation at 118. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -240,7 +232,6 @@ For the 120-day event-free estimate, SAS considers that 120 days is beyond the m If we change the last observation in the dataset to be an event (instead of censored), R and SAS will both give 0 for the event-free survival estimate, because it is for sure that all subjects did not survive beyond 120 days. ```{r} -#| eval: true test <- tibble( time = c(54, 75, 77, 84, 87, 92, 103, 105, 112, 118), status = c(1, 1, 1, 1, 1, 0, 0, 0, 0, 1) @@ -250,7 +241,6 @@ test ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% diff --git a/SAS/SAS_Friedmantest.qmd b/SAS/SAS_Friedmantest.qmd index 30fcbab49..9d5b8eb08 100644 --- a/SAS/SAS_Friedmantest.qmd +++ b/SAS/SAS_Friedmantest.qmd @@ -1,7 +1,5 @@ --- title: "Friedman Chi-Square test using SAS" -execute: - eval: false --- # Introduction diff --git a/SAS/ancova.qmd b/SAS/ancova.qmd index b23adfe28..d93ffa7e4 100644 --- a/SAS/ancova.qmd +++ b/SAS/ancova.qmd @@ -1,12 +1,9 @@ --- title: "Ancova" date: "2024-02-20" -execute: - eval: false --- ```{r} -#| eval: true #| label: setup #| include: false knitr::opts_chunk$set(echo = TRUE) @@ -50,7 +47,6 @@ run; Output: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% diff --git a/SAS/anova.qmd b/SAS/anova.qmd index e44de67c3..2181e2323 100644 --- a/SAS/anova.qmd +++ b/SAS/anova.qmd @@ -1,7 +1,5 @@ --- title: "ANOVA" -execute: - eval: false --- ### **Getting Started** @@ -40,7 +38,6 @@ run; #### Type I ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -50,7 +47,6 @@ knitr::include_graphics("../images/linear/sas-ss-type-1.png") #### Type II {.unnumbered} ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -60,7 +56,6 @@ knitr::include_graphics("../images/linear/sas-ss-type-2.png") #### Type III {.unnumbered} ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -70,7 +65,6 @@ knitr::include_graphics("../images/linear/sas-ss-type-3.png") #### Type IV {.unnumbered} ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -91,7 +85,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% diff --git a/SAS/association.qmd b/SAS/association.qmd index 8a37e9bdc..b552e94d3 100644 --- a/SAS/association.qmd +++ b/SAS/association.qmd @@ -1,7 +1,5 @@ --- title: "Association Analysis for Count Data Using SAS" -execute: - eval: false --- In SAS, association analysis methods for count data/contingency tables is typically performed using the `PROC FREQ` procedure. This procedure has options for Chi-Square and Fisher's Exact tests. @@ -37,7 +35,6 @@ run; Output: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% diff --git a/SAS/binomial_test.qmd b/SAS/binomial_test.qmd index 9306562ee..25baf78bc 100644 --- a/SAS/binomial_test.qmd +++ b/SAS/binomial_test.qmd @@ -1,7 +1,5 @@ --- title: "Binomial Test on Coin Flips and Clinical Data" -execute: - eval: false --- ## Simulating Coin Flips diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 600faf754..8e17e69de 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -2,8 +2,6 @@ title: "Confidence Intervals for Independent Proportions in SAS" bibliography: references.bib csl: nature.csl -execute: - eval: false --- ## Introduction diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd index 3cb5f9a70..e9a4bb248 100644 --- a/SAS/ci_for_paired_prop.qmd +++ b/SAS/ci_for_paired_prop.qmd @@ -1,7 +1,5 @@ --- title: "Confidence intervals for Paired Proportions in SAS" -execute: - eval: false --- ## Introduction diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index c684750d6..d7f48343f 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -1,7 +1,5 @@ --- title: "Confidence intervals for a Proportion in SAS" -execute: - eval: false --- ## Introduction @@ -31,7 +29,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -181,7 +178,6 @@ run; ```{r} #| echo: false -#| eval: true #| fig-align: center #| out-width: 50% knitr::include_graphics("../images/ci_for_prop/binomial_prop_pbo.png") @@ -210,7 +206,6 @@ run; ```{r} #| echo: false -#| eval: true #| fig-align: center #| out-width: 50% knitr::include_graphics("../images/ci_for_prop/binomial_prop_all_act.png") @@ -228,7 +223,6 @@ run; ```{r} #| echo: false -#| eval: true #| fig-align: center #| out-width: 50% knitr::include_graphics("../images/ci_for_prop/binomial_prop_cc_act.png") diff --git a/SAS/cmh.qmd b/SAS/cmh.qmd index b6123ba55..577b2d66d 100644 --- a/SAS/cmh.qmd +++ b/SAS/cmh.qmd @@ -1,7 +1,5 @@ --- title: "CMH Test" -execute: - eval: false --- # Cochran-Mantel-Haenszel Test @@ -56,7 +54,6 @@ Let's test if there is a difference between 3 treatments (Placebo, Xanomeline lo #| echo: false #| fig-align: center #| out-width: 50% -#| eval: true knitr::include_graphics("../images/cmh/saspage_output2.png") ``` @@ -81,7 +78,6 @@ run; #| echo: false #| fig-align: center #| out-width: 50% -#| eval: true knitr::include_graphics("../images/cmh/saspage_output3.png") ``` @@ -120,7 +116,6 @@ run; #| echo: false #| fig-align: center #| out-width: 50% -#| eval: true knitr::include_graphics("../images/cmh/saspage_output3c.png") ``` @@ -156,7 +151,6 @@ run; #| echo: false #| fig-align: center #| out-width: 50% -#| eval: true knitr::include_graphics("../images/cmh/saspage_output4.png") ``` @@ -170,7 +164,6 @@ run; #| echo: false #| fig-align: center #| out-width: 50% -#| eval: true knitr::include_graphics("../images/cmh/saspage_output4b.png") ``` diff --git a/SAS/correlation.qmd b/SAS/correlation.qmd index d1ce1995d..422aad448 100644 --- a/SAS/correlation.qmd +++ b/SAS/correlation.qmd @@ -1,7 +1,5 @@ --- title: "Correlation Analysis using SAS" -execute: - eval: false --- # **Example: Lung Cancer Data** diff --git a/SAS/count_data_regression.qmd b/SAS/count_data_regression.qmd index 410f65f9f..f50418289 100644 --- a/SAS/count_data_regression.qmd +++ b/SAS/count_data_regression.qmd @@ -2,8 +2,6 @@ title: "Poisson and Negative Binomial Regression in SAS" date: last-modified date-format: D MMMM, YYYY -execute: - eval: false --- This page serves as an introduction to performing Poisson and Negative Binomial regression in SAS. For detail on how results compare between R and SAS see [RvsSAS](R%20vs%20SAS:%20Negative%20Binomial%20Regression). @@ -60,7 +58,6 @@ run; ```{r} #| echo: false -#| eval: true #| fig-align: center #| out-width: 50% knitr::include_graphics("../images/count_data_regression/poisson1.png") @@ -104,7 +101,6 @@ run; ```{r} #| echo: false -#| eval: true #| fig-align: center #| out-width: 50% knitr::include_graphics("../images/count_data_regression/negbin1.png") diff --git a/SAS/gee.qmd b/SAS/gee.qmd index 8707a84af..a9df37e47 100644 --- a/SAS/gee.qmd +++ b/SAS/gee.qmd @@ -1,7 +1,5 @@ --- title: "Generalized Estimating Equations (GEE) methods in SAS" -execute: - eval: false --- # INTRODUCTION @@ -72,7 +70,6 @@ Results were extracted into a SAS dataset using the `ODS OUTPUT` statement and s [Estimated Parameters:]{.underline} ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -82,7 +79,6 @@ knitr::include_graphics("../images/gee/1_estimated_parameters.png") [Probability of event:]{.underline} ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -92,7 +88,6 @@ knitr::include_graphics("../images/gee/2_probability_of_event.png") [ODDS RATIO (OR):]{.underline} ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -138,7 +133,6 @@ run; [Ordinal variable:]{.underline} ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -148,7 +142,6 @@ knitr::include_graphics("../images/gee/4_ordinal_variable.png") [Nominal variable:]{.underline} ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% diff --git a/SAS/gsd-tte.qmd b/SAS/gsd-tte.qmd index 563207eb2..daebe5f65 100644 --- a/SAS/gsd-tte.qmd +++ b/SAS/gsd-tte.qmd @@ -2,8 +2,6 @@ title: "Group Sequential Design in Survival Endpoints Using SAS" date: last-modified date-format: D MMMM, YYYY -execute: - eval: false --- # Introduction @@ -41,7 +39,6 @@ RUN; As shown below, a total sample size of 398 is recommended, which equates to 199 in each group. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% diff --git a/SAS/jonchkheere_terpstra.qmd b/SAS/jonchkheere_terpstra.qmd index ea5242da0..2322f45ae 100644 --- a/SAS/jonchkheere_terpstra.qmd +++ b/SAS/jonchkheere_terpstra.qmd @@ -1,7 +1,5 @@ --- title: "SAS Jonckheere-Terpstra Test" -execute: - eval: false --- ## Background diff --git a/SAS/kruskal_wallis.qmd b/SAS/kruskal_wallis.qmd index f205163a7..2a23fcd1e 100644 --- a/SAS/kruskal_wallis.qmd +++ b/SAS/kruskal_wallis.qmd @@ -1,7 +1,5 @@ --- title: "Kruskal Wallis SAS" -execute: - eval: false --- ## Introduction @@ -49,7 +47,6 @@ run; ## Results ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 90% diff --git a/SAS/linear-regression.qmd b/SAS/linear-regression.qmd index e7d77b992..95645e35f 100644 --- a/SAS/linear-regression.qmd +++ b/SAS/linear-regression.qmd @@ -2,8 +2,6 @@ title: "Linear Regression" date: last-modified date-format: D MMMM, YYYY -execute: - eval: false --- To demonstrate the use of linear regression we examine a dataset that illustrates the relationship between Height and Weight in a group of 237 teen-aged boys and girls. The dataset is available at (../data/htwt.csv) and is imported to sas using proc import procedure. diff --git a/SAS/logistic-regr.qmd b/SAS/logistic-regr.qmd index c965f8d15..84130d649 100644 --- a/SAS/logistic-regr.qmd +++ b/SAS/logistic-regr.qmd @@ -2,8 +2,6 @@ title: "Logistic Regression in SAS" date: last-modified date-format: D MMMM, YYYY -execute: - eval: false --- For a brief description of what is logistic regression see [here](../R/logistic_regr.html). diff --git a/SAS/manova.qmd b/SAS/manova.qmd index b247bddc8..4b16969e1 100644 --- a/SAS/manova.qmd +++ b/SAS/manova.qmd @@ -1,7 +1,5 @@ --- title: "Multivariate Analysis of Variance in SAS" -execute: - eval: false --- **Example 39.6 Multivariate Analysis of Variance** from [SAS MANOVA User Guide](https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_glm_sect051.htm) @@ -58,7 +56,6 @@ After the summary information (1), PROC GLM produces the univariate analyses for **1 Summary Information about Groups** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -68,7 +65,6 @@ knitr::include_graphics("../images/manova/manova1_class.jpg") **2 Univariate Analysis of Variance for Aluminum Oxide (AI)** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -78,7 +74,6 @@ knitr::include_graphics("../images/manova/manova2_anova_ao.jpg") **3 Univariate Analysis of Variance for Iron Oxide (Fe)** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -88,7 +83,6 @@ knitr::include_graphics("../images/manova/manova3_anova_fe.jpg") **4 Univariate Analysis of Variance for Calcium Oxide (Ca)** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -98,7 +92,6 @@ knitr::include_graphics("../images/manova/manova4_anova_ca.jpg") **5 Univariate Analysis of Variance for Magnesium Oxide (Mg)** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -108,7 +101,6 @@ knitr::include_graphics("../images/manova/manova5_anova_mg.jpg") **6 Analysis of Variance for Sodium Oxide (Na)** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -122,7 +114,6 @@ The PRINTE option also displays the partial correlation matrix (7) associated wi **7 Error SSCP Matrix and Partial Correlations** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -136,7 +127,6 @@ Four multivariate tests are computed, all based on the characteristic roots and **8 Hypothesis SSCP Matrix and Multivariate Tests for Overall Site Effect** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -146,7 +136,6 @@ knitr::include_graphics("../images/manova/manova8_hyp_tests.jpg") **9 Hypothesis SSCP Matrix and Multivariate Tests for Differences between Llanederyn and the Other Sites** ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% diff --git a/SAS/mcnemar.qmd b/SAS/mcnemar.qmd index e203bb997..7136b94e9 100644 --- a/SAS/mcnemar.qmd +++ b/SAS/mcnemar.qmd @@ -1,7 +1,5 @@ --- title: "McNemar's test in SAS" -execute: - eval: false --- ### Performing McNemar's test in SAS @@ -21,7 +19,6 @@ run; #### Results ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 40% diff --git a/SAS/mi_mar_regression.qmd b/SAS/mi_mar_regression.qmd index 2ec36ea60..f5d329d4b 100644 --- a/SAS/mi_mar_regression.qmd +++ b/SAS/mi_mar_regression.qmd @@ -1,7 +1,5 @@ --- title: "Multiple Imputaton: Linear Regression in SAS" -execute: - eval: false --- ## Input dataset preparation before multiple imputation @@ -46,7 +44,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -71,7 +68,6 @@ run; As below figure shows the missingness dose not follow any specific order, obviously the missing pattern is arbitrary and non-monotone missing pattern. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -100,7 +96,6 @@ run; - The `DETAILS` option displays the regression coefficients in the regression model used in each imputation. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -108,7 +103,6 @@ knitr::include_graphics("../images/mi_mar_linear_sas/mi_mar_reg_fcs.PNG") ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -156,7 +150,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% diff --git a/SAS/mmrm.qmd b/SAS/mmrm.qmd index 14e4f7922..bf5c0d230 100644 --- a/SAS/mmrm.qmd +++ b/SAS/mmrm.qmd @@ -1,7 +1,5 @@ --- title: "MMRM in SAS" -execute: - eval: false --- # Mixed Models diff --git a/SAS/nparestimate.qmd b/SAS/nparestimate.qmd index 1422a47ad..b4b4eb932 100644 --- a/SAS/nparestimate.qmd +++ b/SAS/nparestimate.qmd @@ -1,7 +1,5 @@ --- title: "Non-parametric point estimation in SAS" -execute: - eval: false --- # Introduction diff --git a/SAS/ranksum.qmd b/SAS/ranksum.qmd index d25c91d3e..9167b5ebd 100644 --- a/SAS/ranksum.qmd +++ b/SAS/ranksum.qmd @@ -1,7 +1,5 @@ --- title: "Wilcoxon Rank Sum /Mann-Whitney U test" -execute: - eval: false --- # Wilcoxon Rank Sum / Mann-Whitney U test diff --git a/SAS/rbmi_continuous_joint_SAS.qmd b/SAS/rbmi_continuous_joint_SAS.qmd index d1ef5ff27..6c615330d 100644 --- a/SAS/rbmi_continuous_joint_SAS.qmd +++ b/SAS/rbmi_continuous_joint_SAS.qmd @@ -1,7 +1,5 @@ --- title: "Reference-Based Multiple Imputation (joint modelling): Continuous Data" -execute: - eval: false --- ## Reference-based multiple imputation (rbmi) @@ -51,7 +49,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 80% @@ -67,7 +64,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 80% @@ -84,7 +80,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 80% @@ -106,7 +101,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 80% @@ -129,7 +123,6 @@ run; The parameter estimates of the fixed effects are: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 80% @@ -139,7 +132,6 @@ knitr::include_graphics("../images/rbmi/SAS_CompleteCase_fixedEstimates.PNG") The estimated unstructured covariance matrix parameters are: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 80% @@ -151,7 +143,6 @@ knitr::include_graphics( The treatment difference at visit 7 is of interest, and is estimated to be -2.829 (se=1.117) with 95% CI of \[-5.033 to -0.624\] (p=0.0122). ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 90% @@ -215,7 +206,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 90% @@ -236,7 +226,6 @@ To perform reference-based multiple imputation using Copy Reference (CR) approac The results for M=500 imputations are ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 90% @@ -257,7 +246,6 @@ To perform reference-based multiple imputation using Jump to Reference (J2R) app The results for M=500 imputations are ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 90% @@ -278,7 +266,6 @@ To perform reference-based multiple imputation using Copy Increments in Referenc The results for M=500 imputations are ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 90% @@ -342,7 +329,6 @@ run; The results for M=500 imputations are ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 90% diff --git a/SAS/recurrent_events.qmd b/SAS/recurrent_events.qmd index 79a384f89..8d45f83c3 100644 --- a/SAS/recurrent_events.qmd +++ b/SAS/recurrent_events.qmd @@ -1,18 +1,14 @@ --- title: "SAS Recurrent Events" -execute: - eval: false --- ```{r} -#| eval: true #| label: setup #| include: FALSE knitr::opts_chunk$set(echo = TRUE) ``` ```{r} -#| eval: true #| label: libaries #| include: FALSE library(dplyr) @@ -89,7 +85,6 @@ For both versions of the Andersen-Gill model, the data must be structured as fol This can be visually represented: ```{r} -#| eval: true #| label: AG_lineplot #| echo: FALSE #| fig-align: 'center' @@ -140,7 +135,6 @@ For the total time model, the data must be structured as follows: This can be visually represented: ```{r} -#| eval: true #| label: PWPtt_lineplot #| echo: FALSE #| fig-align: 'center' @@ -173,7 +167,6 @@ For the gap time model, the data must be structured as follows: This can be visually represented: ```{r} -#| eval: true #| label: PWPgt_lineplot #| echo: FALSE #| fig-align: 'center' @@ -244,7 +237,6 @@ Note that, because the ordering of events is not important in the Andersen-Gill A nice visual representation of the stratification and time interval structure of each model is given below. The correct data structure is pivotal when modelling recurrent events and depends on the methodology you want to use, as illustrated in the figure. ```{r} -#| eval: true #| label: combined_lineplot #| echo: FALSE #| fig-align: 'center' @@ -289,14 +281,12 @@ Importantly, both datasets collect the data in a **counting process** structure. Let's look more closely at the `bladder2` and `bladder` data: ```{r} -#| eval: true #| label: bladder2 bladder2 <- survival::bladder2 gt(head(bladder2, 6)) ``` ```{r} -#| eval: true #| label: bladder2_enum bladder2 %>% group_by(enum) %>% summarise(n = n()) %>% gt() @@ -305,14 +295,12 @@ bladder2 %>% In `bladder2`, in the Andersen-Gill format, each subject has a variable amount of records, depending on the amount of events that subject experienced. ```{r} -#| eval: true #| label: bladder bladder <- survival::bladder gt(head(bladder, 20)) ``` ```{r} -#| eval: true #| label: bladder_enum bladder %>% group_by(enum) %>% summarise(n = n()) %>% gt() @@ -381,7 +369,6 @@ run; ``` ```{r} -#| eval: true #| label: AG_original #| echo: FALSE #| fig-align: 'center' @@ -425,7 +412,6 @@ run; ``` ```{r} -#| eval: true #| label: PWPtt #| echo: FALSE #| fig-align: 'center' @@ -476,7 +462,6 @@ run; ``` ```{r} -#| eval: true #| label: PWPgt #| echo: FALSE #| fig-align: 'center' @@ -530,7 +515,6 @@ run; ``` ```{r} -#| eval: true #| label: WLW #| echo: FALSE #| fig-align: 'center' @@ -551,7 +535,6 @@ Importantly, the strata of the Wei-Lin-Weissfeld model as set by `strata enum;` Summary for Prentice-Williams-Peterson models: ```{r} -#| eval: true #| label: PWP_structure #| echo: FALSE #| fig-align: 'center' @@ -562,7 +545,6 @@ knitr::include_graphics("../images/recurrent_events/SAS_PWP_structure.png") Summary for Wei-Lin-Weissfeld model: ```{r} -#| eval: true #| label: WLW_structure #| echo: FALSE #| fig-align: 'center' @@ -644,7 +626,6 @@ run; ``` ```{r} -#| eval: true #| label: PWPtt_stratified #| echo: FALSE #| fig-align: 'center' @@ -664,7 +645,6 @@ run; ``` ```{r} -#| eval: true #| label: PWPgt_stratified #| echo: FALSE #| fig-align: 'center' @@ -684,7 +664,6 @@ run; ``` ```{r} -#| eval: true #| label: WLW_stratified #| echo: FALSE #| fig-align: 'center' diff --git a/SAS/rmst.qmd b/SAS/rmst.qmd index eaafe9c9c..09ffa670a 100644 --- a/SAS/rmst.qmd +++ b/SAS/rmst.qmd @@ -2,8 +2,6 @@ title: "Restricted Mean Survival Time (RMST) in SAS" date: last-modified date-format: D MMMM, YYYY -execute: - eval: false --- Under the situation where you time to event outcome has non-proportional hazards over time, the commonly used Cox Proportional Hazards regression analysis and the log-rank test can be invalid - especially when the survival curves are crossing. One alternative is to analyse the Restricted Mean Survival Time (RMST). @@ -26,7 +24,6 @@ A common mistake is to get the (0) or (1) indicator the wrong way around. For ex Throughout this page, we will use cnsr(1) approach. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -38,7 +35,6 @@ knitr::include_graphics("../images/rmst/SASrmstreg2.png") The selection of `tau` for RMST analysis is very important. It's the period of time the average (or AUC for proc lifetest) is calculated over. If events are no longer occurring on both treatment groups, then you may not be looking at the key time period of interest. Therefore, it is better practice, to calculate `tau` yourself and set this as an option in the SAS code, (commonly based on the minimum time of the last event observed in each treatment group). ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -85,7 +81,6 @@ Run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -143,7 +138,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -153,7 +147,6 @@ knitr::include_graphics("../images/rmst/rmstreg_output1.png") The above model results in a `restricted mean survival time` estimate of 257.16 days on treatment 1 vs 267.04 days on treatment 2. The difference (Active-Placebo) is -9.88 days (95% CI: -39.0 to 19.25, p=0.5061). Hence, there is no evidence of a difference between the treatments with respect to RMST when we look over the Kaplan-Meier plot from 0 to 350 days. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -176,7 +169,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -235,7 +227,6 @@ knitr::include_graphics("../images/rmst/rmstreg_output4.png") ### Version ```{r} -#| eval: true #| echo: false si <- sessioninfo::session_info("rmst", dependencies = FALSE) si$external <- structure( diff --git a/SAS/rounding.qmd b/SAS/rounding.qmd index 715dd5cf2..697614b19 100644 --- a/SAS/rounding.qmd +++ b/SAS/rounding.qmd @@ -1,7 +1,5 @@ --- title: "Rounding in SAS" -execute: - eval: false --- SAS provides two distinct rounding functions that handle tie-breaking (values exactly halfway between two numbers) differently: @@ -21,7 +19,6 @@ sasquatch::sas_connect() ```{r} #| include: false -#| eval: true sas_code_01 <- " @@ -58,7 +55,6 @@ saveRDS(widget_01, "SAS/rds/rounding/widget_01.rds") ```{r} #| echo: false -#| eval: true #| results: asis cat("```txt\n") cat(sas_code_01) @@ -67,7 +63,6 @@ cat("```") ```{r} #| echo: false -#| eval: true readRDS(file = "rds/rounding/widget_01.rds") ``` @@ -77,7 +72,6 @@ These precision issues are inherent to how computers store decimal numbers, not ```{r} #| include: false -#| eval: true sas_code_02 <- " @@ -115,7 +109,6 @@ saveRDS(widget_02, file = "SAS/rds/rounding/widget_02.rds") ```{r} #| echo: false -#| eval: true #| results: asis cat("```txt\n") cat(sas_code_02) @@ -124,7 +117,6 @@ cat("```") ```{r} #| echo: false -#| eval: true readRDS(file = "rds/rounding/widget_02.rds") ``` @@ -132,7 +124,6 @@ The following analysis systematically identifies cases where `round` fails by te ```{r} #| include: false -#| eval: true sas_code_03 <- " @@ -172,7 +163,6 @@ saveRDS(widget_03, file = "SAS/rds/rounding/widget_03.rds") ```{r} #| echo: false -#| eval: true #| results: asis cat("```txt\n") cat(sas_code_03) @@ -181,7 +171,6 @@ cat("```") ```{r} #| echo: false -#| eval: true readRDS("rds/rounding/widget_03.rds") ``` @@ -189,7 +178,6 @@ This example demonstrates rounding behavior with results from common arithmetic ```{r} #| include: false -#| eval: true sas_code_04 <- " @@ -237,7 +225,6 @@ saveRDS(widget_04, file = "SAS/rds/rounding/widget_04.rds") ```{r} #| echo: false -#| eval: true #| results: asis cat("```txt\n") cat(sas_code_04) @@ -246,7 +233,6 @@ cat("```") ```{r} #| echo: false -#| eval: true readRDS(file = "rds/rounding/widget_04.rds") ``` diff --git a/SAS/sample_s_StatXact_test_of_trends.qmd b/SAS/sample_s_StatXact_test_of_trends.qmd index eadd40a69..9563df2cf 100644 --- a/SAS/sample_s_StatXact_test_of_trends.qmd +++ b/SAS/sample_s_StatXact_test_of_trends.qmd @@ -1,13 +1,10 @@ --- title: "Sample size for K ordered binomial populations - Cochran-Armitage trend test" -execute: - eval: false --- ```{r} #| echo: false #| include: false -#| eval: true knitr::opts_chunk$set(echo = TRUE) ``` @@ -30,7 +27,6 @@ The below parameters are need for the calculations: In StatXact we can directly specify all of the response probabilities, or we can specify the baseline probability and use the below logit model with prespecified slope lambda to derive the rest: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -44,7 +40,6 @@ Let's consider an example of a dose-finding phase I clinical trial of patients w Design parameter are as below: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -72,7 +67,6 @@ run; Output from StatXact and results: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -88,7 +82,6 @@ Let's consider an example of a long-term follow-up study of subjects exposed to Design parameter are as below: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -115,7 +108,6 @@ run; Output from StatXact and results: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -129,7 +121,6 @@ Asymptotic power of the test is 77%, a considerable overestimate of the actual p Let's consider an example of the study where the design parameters are as below: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% diff --git a/SAS/sample_s_equivalence.qmd b/SAS/sample_s_equivalence.qmd index 592171986..d327038cb 100644 --- a/SAS/sample_s_equivalence.qmd +++ b/SAS/sample_s_equivalence.qmd @@ -2,8 +2,6 @@ title: "Sample Size for Equivalence Trials in SAS" date: last-modified date-format: D MMMM, YYYY -execute: - eval: false --- # Introduction @@ -44,7 +42,6 @@ RUN; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -83,7 +80,6 @@ RUN; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -131,7 +127,6 @@ RUN; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -171,7 +166,6 @@ RUN; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -188,7 +182,6 @@ knitr::include_graphics("../images/sample_s_equivalence/SAS2crossovermeans.png") ```{r} #| echo: false -#| eval: true si <- sessioninfo::session_info("sample_s_equivalence", dependencies = FALSE) si$external <- structure( list("SAS" = "9.04.01M7P08062020"), diff --git a/SAS/sample_s_noninferiority.qmd b/SAS/sample_s_noninferiority.qmd index 86bfae55c..7dca29eda 100644 --- a/SAS/sample_s_noninferiority.qmd +++ b/SAS/sample_s_noninferiority.qmd @@ -2,8 +2,6 @@ title: "Sample Size for Non-Inferiority Trials in SAS" date: last-modified date-format: D MMMM, YYYY -execute: - eval: false --- # Introduction @@ -37,7 +35,6 @@ RUN; As shown below, a total sample size of 102 is recommended, which equates to 51 in each group. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -70,7 +67,6 @@ test=equiv_diff ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -88,7 +84,6 @@ knitr::include_graphics( ### Version ```{r} -#| eval: true #| echo: false si <- sessioninfo::session_info("sample_s_noninferiority", dependencies = FALSE) si$external <- structure( diff --git a/SAS/sample_s_superiority.qmd b/SAS/sample_s_superiority.qmd index 2036fc96b..918a85c13 100644 --- a/SAS/sample_s_superiority.qmd +++ b/SAS/sample_s_superiority.qmd @@ -3,8 +3,6 @@ title: "Sample Size for Superiority Trials in SAS" output: html_document date: last-modified date-format: D MMMM, YYYY -execute: - eval: false --- SAS has 2 procedures for doing Sample size. A basic summary is provided here based on Jenny Cody's paper^1^ , but see the paper itself for more details. There are also many available options to best to consult SAS online support for [PROC POWER](PROC%20POWER:%20Simple%20AB/BA%20Crossover%20Designs%20::%20SAS/STAT(R)%209.3%20User's%20Guide)^2^ and [PROC GLMPOWER](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_glmpower_a0000000154.htm)^3^. @@ -53,7 +51,6 @@ RUN; As shown below, a total sample size of 114 is recommended, which equates to 57 in each group. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -83,7 +80,6 @@ RUN; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -100,7 +96,6 @@ knitr::include_graphics("../images/sample_s_superiority/SAScrossovermeans.png") ### Version ```{r} -#| eval: true #| echo: false si <- sessioninfo::session_info("sample_s_superiority", dependencies = FALSE) si$external <- structure( diff --git a/SAS/summary-stats.qmd b/SAS/summary-stats.qmd index c344e2361..d4b0b647e 100644 --- a/SAS/summary-stats.qmd +++ b/SAS/summary-stats.qmd @@ -1,7 +1,5 @@ --- title: "Calculating Quantiles (percentiles) in SAS" -execute: - eval: false --- Percentiles can be calculated in SAS using the UNIVARIATE procedure. The procedure has the option `PCTLDEF` which allows for five different percentile definitions to be used. The default is `PCTLDEF=5`, which uses the empirical distribution function to find percentiles. diff --git a/SAS/summary_skew_kurt.qmd b/SAS/summary_skew_kurt.qmd index c6cac61b8..cb89ded1d 100644 --- a/SAS/summary_skew_kurt.qmd +++ b/SAS/summary_skew_kurt.qmd @@ -1,12 +1,9 @@ --- title: "Skewness/Kurtosis" output: html_document -execute: - eval: false --- ```{r} -#| eval: true #| label: setup #| include: false knitr::opts_chunk$set(echo = TRUE) @@ -54,7 +51,6 @@ The following shows the SAS documentation for the two measures. The [SAS documentation for Skewness](https://documentation.sas.com/doc/en/vdmmlcdc/8.1/casfedsql/p04x27b92gon3gn10e5y5ybxbvmi.htm) is provided here for convenience: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -66,7 +62,6 @@ knitr::include_graphics("../images/summarystats/sas_skewness.png") The SAS documentation for Kurtosis is as follows: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -86,7 +81,6 @@ run; Output: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 30% @@ -108,7 +102,6 @@ run; Output: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 30% diff --git a/SAS/survey-stats-summary.qmd b/SAS/survey-stats-summary.qmd index e444ba33d..cb51b10ba 100644 --- a/SAS/survey-stats-summary.qmd +++ b/SAS/survey-stats-summary.qmd @@ -1,8 +1,6 @@ --- title: "Survey Summary Statistics using SAS" bibliography: ../Comp/survey-stats-summary.bib -execute: - eval: false --- When conducting large-scale trials on samples of the population, it can be necessary to use a more complex sampling design than a simple random sample. @@ -24,7 +22,6 @@ For survey summary statistics in SAS, we can use the `SURVEYMEANS` and `SURVEYFR We will use the [API]((https://r-survey.r-forge.r-project.org/survey/html/api.html)) dataset [@API_2000], which contains a number of datasets based on different samples from a dataset of academic performance. Initially we will just cover the methodology with a simple random sample and a finite population correction to demonstrate functionality. ```{r} -#| eval: true #| echo: false #| message: false library(survey) @@ -200,7 +197,6 @@ run; Much of the previous examples and notes still stand for more complex survey designs, here we will demonstrate using a dataset from NHANES [@NHANES_2010], which uses both stratification and clustering: ```{r} -#| eval: true #| echo: false data("nhanes") @@ -295,7 +291,6 @@ race Variable Mean of Mean Effect ::: {.callout-note collapse="true" title="Session Info"} ```{r} -#| eval: true #| echo: false si <- sessioninfo::session_info("survey", dependencies = FALSE) si$external <- structure( diff --git a/SAS/survival.qmd b/SAS/survival.qmd index ca6d74dc6..6beeac0f7 100644 --- a/SAS/survival.qmd +++ b/SAS/survival.qmd @@ -1,7 +1,5 @@ --- title: "Survival Analysis Using SAS" -execute: - eval: false --- The most commonly used survival analysis methods in clinical trials include: @@ -31,7 +29,6 @@ While these models may be explored in a separate document, this particular docum Below is a standard mock-up for survival analysis in clinical trials. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -79,7 +76,6 @@ run; The landmark estimates and quartile estimates for AFB = 0 group are as shown in below: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -89,7 +85,6 @@ knitr::include_graphics("../images/survival/sas_km_afib0.png") The logrank test result is in below: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -108,7 +103,6 @@ run; The hazard ratio and confidence intervals are shown as below: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% diff --git a/SAS/survival_cif.qmd b/SAS/survival_cif.qmd index c44542702..40373c0b0 100644 --- a/SAS/survival_cif.qmd +++ b/SAS/survival_cif.qmd @@ -1,7 +1,5 @@ --- title: "Estimating Cumulative Incidence Functions Using SAS" -execute: - eval: false --- ## Objective @@ -9,7 +7,6 @@ execute: In this document we present how to estimate the cumulative incidence function (CIF) in SAS (version 9.4). We focus on the competing risks model where each subject experiences only one out of *k* possible events as depicted in the figure below. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 25% @@ -76,7 +73,6 @@ Below are selected outputs for comparison with the R outputs in the companion do CIF estimates for time to relapse at selected timepoints for 'AML-Low Risk' patients: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -86,7 +82,6 @@ knitr::include_graphics("../images/survival_competing_risks/cifSAS.jpg") CIF estimates for time to relapses: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% diff --git a/SAS/survival_csh.qmd b/SAS/survival_csh.qmd index 93a8234c1..802577645 100644 --- a/SAS/survival_csh.qmd +++ b/SAS/survival_csh.qmd @@ -1,7 +1,5 @@ --- title: "Estimating and Testing Cause Specific Hazard Ratio Using SAS" -execute: - eval: false --- ## Objective @@ -9,7 +7,6 @@ execute: In this document we present how to estimate and test cause specific hazard ratio for the probability of experiencing a certain event at a given time in a competing risks model in SAS (version 9.4). We focus on the basic model where each subject experiences only one out of *k* possible events as depicted in the figure below. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 25% @@ -76,7 +73,6 @@ run; The results for both events are given below: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% diff --git a/SAS/tipping_point.qmd b/SAS/tipping_point.qmd index 67d0f1f37..506b3668b 100644 --- a/SAS/tipping_point.qmd +++ b/SAS/tipping_point.qmd @@ -1,12 +1,9 @@ --- title: "SAS Tipping Point (Delta Adjustment): Continuous Data" -execute: - eval: false --- ```{r} #| include: false -#| eval: true knitr::opts_chunk$set(echo = TRUE) ``` @@ -55,7 +52,6 @@ run; ```{r} #| label: data_exploration_1_png #| echo: false -#| eval: true #| fig-align: center #| out-width: 80% knitr::include_graphics("../images/tipping_point/SAS_data_exploration_1.PNG") @@ -113,7 +109,6 @@ run; ```{r} #| label: data_exploration_4_png #| echo: false -#| eval: true #| fig-align: center #| out-width: 80% knitr::include_graphics("../images/tipping_point/SAS_data_exploration_4.PNG") @@ -249,7 +244,6 @@ run; ```{r} #| label: MAR_non_sig_results_png #| echo: false -#| eval: true #| fig-align: center #| out-width: 80% knitr::include_graphics("../images/tipping_point/SAS_MAR_non_sig_results.png") @@ -258,7 +252,6 @@ knitr::include_graphics("../images/tipping_point/SAS_MAR_non_sig_results.png") ```{r} #| label: MAR_sig_results_png #| echo: false -#| eval: true #| fig-align: center #| out-width: 80% knitr::include_graphics("../images/tipping_point/SAS_MAR_sig_results.png") @@ -269,7 +262,6 @@ To determine the **exact** tipping point between the last "significant" delta an ```{r} #| label: MAR_TP_png #| echo: false -#| eval: true #| fig-align: center #| out-width: 80% knitr::include_graphics("../images/tipping_point/SAS_MAR_TP.png") @@ -284,7 +276,6 @@ A nice visualization of this tipping point analysis for the MAR approach is show ```{r} #| label: MAR_est_pval_plot_png #| echo: false -#| eval: true #| fig-align: center #| out-width: 80% knitr::include_graphics("../images/tipping_point/SAS_MAR_est_pval.png") @@ -316,7 +307,6 @@ Of all considered approaches, the MAR approach yields the largest delta adjustme ```{r} #| label: comparison_est_plot_png #| echo: false -#| eval: true #| fig-align: 'center' #| out-width: '80%' knitr::include_graphics("../images/tipping_point/SAS_comparison_est.png") @@ -325,7 +315,6 @@ knitr::include_graphics("../images/tipping_point/SAS_comparison_est.png") ```{r} #| label: comparison_pval_plot_png #| echo: false -#| eval: true #| fig-align: 'center' #| out-width: '80%' knitr::include_graphics("../images/tipping_point/SAS_comparison_pval.png") diff --git a/SAS/tobit regression SAS.qmd b/SAS/tobit regression SAS.qmd index 465b716fc..bcee028d3 100644 --- a/SAS/tobit regression SAS.qmd +++ b/SAS/tobit regression SAS.qmd @@ -1,7 +1,5 @@ --- title: "Tobit Regression" -execute: - eval: false --- # Tobit regression @@ -95,7 +93,6 @@ run; The fit statistics, type 3 analysis of effects and parameter estimated are shown here. The output provides an estimate of difference between groups A and B (B-A), namely 1.8225 (se=0.8061). The presented p-value is a two-sided p-value based on the Z-test. The scale parameter is an estimate for $\sigma$. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 100% @@ -105,7 +102,6 @@ knitr::include_graphics("../images/tobit/SAS_tobit_1.PNG") The p-value and confidence intervals of the contrast B-A are shown here. The p-value is the same as above. ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 80% diff --git a/SAS/ttest_1Sample.qmd b/SAS/ttest_1Sample.qmd index 175d41596..b0061afdd 100644 --- a/SAS/ttest_1Sample.qmd +++ b/SAS/ttest_1Sample.qmd @@ -1,12 +1,9 @@ --- title: "One Sample t-test in SAS" output: html_document -execute: - eval: false --- ```{r} -#| eval: true #| label: setup #| include: false knitr::opts_chunk$set(echo = TRUE) @@ -48,7 +45,6 @@ run; Output: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -72,7 +68,6 @@ run; Output: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 60% diff --git a/SAS/ttest_2Sample.qmd b/SAS/ttest_2Sample.qmd index 79e9dffcd..aed78b195 100644 --- a/SAS/ttest_2Sample.qmd +++ b/SAS/ttest_2Sample.qmd @@ -1,12 +1,9 @@ --- title: "Independent Two-Sample t-test" output: html_document -execute: - eval: false --- ```{r} -#| eval: true #| label: setup #| include: false knitr::opts_chunk$set(echo = TRUE) @@ -56,7 +53,6 @@ Output: ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -85,7 +81,6 @@ Output: ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 30% @@ -97,7 +92,6 @@ knitr::include_graphics("../images/ttest/trt_sas.png") ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 30% @@ -113,7 +107,6 @@ Output: ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 30% @@ -154,7 +147,6 @@ Output: ```{r} #| echo: false -#| eval: true #| fig-align: center #| out-width: 50% knitr::include_graphics("../images/ttest/test2.png") diff --git a/SAS/ttest_Paired.qmd b/SAS/ttest_Paired.qmd index d5791bcc4..d8b34ad9b 100644 --- a/SAS/ttest_Paired.qmd +++ b/SAS/ttest_Paired.qmd @@ -1,11 +1,8 @@ --- title: "Paired t-test" -execute: - eval: false --- ```{r} -#| eval: true #| label: setup #| include: false knitr::opts_chunk$set(echo = TRUE) @@ -48,7 +45,6 @@ run; Output: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 50% @@ -84,7 +80,6 @@ run; Output: ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 70% diff --git a/SAS/wilcoxonsr_HL.qmd b/SAS/wilcoxonsr_HL.qmd index 19eef4ec0..c1de66ebc 100644 --- a/SAS/wilcoxonsr_HL.qmd +++ b/SAS/wilcoxonsr_HL.qmd @@ -1,7 +1,5 @@ --- title: "Wilcoxon signed-rank test in SAS & StatXact®" -execute: - eval: false --- ### **Introduction** @@ -28,7 +26,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -70,7 +67,6 @@ run; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -108,7 +104,6 @@ RUN; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% @@ -125,7 +120,6 @@ RUN; ``` ```{r} -#| eval: true #| echo: false #| fig-align: center #| out-width: 75% From 16ee0bb87dc75946d571f419a2567dbc49847e12 Mon Sep 17 00:00:00 2001 From: Michael Walshe <62556482+michaelwalshe@users.noreply.github.com> Date: Tue, 10 Mar 2026 10:27:31 +0000 Subject: [PATCH 03/13] Add back eval: false to SAS/rounding.qmd --- SAS/rounding.qmd | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/SAS/rounding.qmd b/SAS/rounding.qmd index 697614b19..a72a652fb 100644 --- a/SAS/rounding.qmd +++ b/SAS/rounding.qmd @@ -11,7 +11,7 @@ The key difference appears when rounding values that are exactly halfway between ```{r} #| include: false - +#| eval: false library(sasquatch) sasquatch::sas_connect() @@ -48,6 +48,8 @@ run; ```{r} #| include: false +#| eval: false + widget_01 <- sasquatch::sas_run_string(sas_code_01, capture = "listing") saveRDS(widget_01, "SAS/rds/rounding/widget_01.rds") @@ -102,6 +104,7 @@ run; ```{r} #| include: false +#| eval: false widget_02 <- sasquatch::sas_run_string(sas_code_02, capture = "listing") saveRDS(widget_02, file = "SAS/rds/rounding/widget_02.rds") @@ -155,7 +158,7 @@ quit; ```{r} #| include: false -#| +#| eval: false widget_03 <- sasquatch::sas_run_string(sas_code_03, capture = "listing") saveRDS(widget_03, file = "SAS/rds/rounding/widget_03.rds") @@ -217,7 +220,7 @@ run; ```{r} #| include: false - +#| eval: false widget_04 <- sasquatch::sas_run_string(sas_code_04, capture = "listing") saveRDS(widget_04, file = "SAS/rds/rounding/widget_04.rds") From 9a10cc583c6d6ae77dac967134bc55728f1d8f29 Mon Sep 17 00:00:00 2001 From: Michael Walshe <62556482+michaelwalshe@users.noreply.github.com> Date: Tue, 10 Mar 2026 10:27:44 +0000 Subject: [PATCH 04/13] Remove sasquatch requirement --- renv.lock | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/renv.lock b/renv.lock index 3571f9a63..e0665dd00 100644 --- a/renv.lock +++ b/renv.lock @@ -9,7 +9,7 @@ ] }, "Python": { - "Version": "3.12.6", + "Version": "3.12.7", "Type": "virtualenv", "Name": "./renv/python/virtualenvs/renv-python-3.12" }, @@ -4461,26 +4461,6 @@ ], "Hash": "2f8e8e88b3267f0d61319e77876eaf90" }, - "sasquatch": { - "Package": "sasquatch", - "Version": "0.1.0.9003", - "Source": "Repository", - "Repository": "https://ropensci.r-universe.dev", - "RemoteUrl": "https://github.com/ropensci/sasquatch", - "RemoteRef": "main", - "RemoteSha": "e13f0a8b3c7a0a4cd15c37d9d0442b2b1200425e", - "Requirements": [ - "R", - "cli", - "evaluate", - "htmlwidgets", - "knitr", - "reticulate", - "rlang", - "rstudioapi" - ], - "Hash": "b6b484cd8852e4e22f5a02b428e11ec8" - }, "sass": { "Package": "sass", "Version": "0.4.10", From 0c40c2f82b47b35fa0e366f15c3c3a6899ffb5bc Mon Sep 17 00:00:00 2001 From: Michael Walshe <62556482+michaelwalshe@users.noreply.github.com> Date: Tue, 10 Mar 2026 11:25:37 +0000 Subject: [PATCH 05/13] Revert sessioninfo to 1.2.2 as 1.2.3 errors on windows (fix in dev version) --- renv.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/renv.lock b/renv.lock index e0665dd00..04f1d0180 100644 --- a/renv.lock +++ b/renv.lock @@ -4563,16 +4563,16 @@ }, "sessioninfo": { "Package": "sessioninfo", - "Version": "1.2.3", + "Version": "1.2.2", "Source": "Repository", - "Repository": "CRAN", + "Repository": "RSPM", "Requirements": [ "R", "cli", "tools", "utils" ], - "Hash": "bf169c6e52cdbded916e448dc1254913" + "Hash": "3f9796a8d0a0e8c6eb49a4b029359d1f" }, "sf": { "Package": "sf", From 0795c69b663670ea8a8116ca62258fe38dc2e752 Mon Sep 17 00:00:00 2001 From: Michael Walshe <62556482+michaelwalshe@users.noreply.github.com> Date: Tue, 10 Mar 2026 11:27:26 +0000 Subject: [PATCH 06/13] Re-add eval: false to R/SAS files that need it --- Comp/r-sas-summary-stats.qmd | 2 ++ Comp/r-sas_kruskalwallis.qmd | 1 + Comp/r-sas_mmrm.qmd | 10 +++++++++- Comp/r-sas_psmatch.qmd | 2 ++ 4 files changed, 14 insertions(+), 1 deletion(-) diff --git a/Comp/r-sas-summary-stats.qmd b/Comp/r-sas-summary-stats.qmd index 441981562..9c8315416 100644 --- a/Comp/r-sas-summary-stats.qmd +++ b/Comp/r-sas-summary-stats.qmd @@ -37,6 +37,7 @@ The procedure has the option `PCTLDEF` which allows for five different percentil The 25th and 40th percentiles of `aval` can be calculated using the `quantile` function. ```{r} +#| eval: false quantile(adlb$aval, probs = c(0.25, 0.4)) ``` @@ -57,6 +58,7 @@ The default percentile definition used by the UNIVARIATE procedure in SAS finds It is possible to get the quantile function in R to use the same definition as the default used in SAS, by specifying `type=2`. ```{r} +#| eval: false alquantile(adlb$aval, probs = c(0.25, 0.4), type = 2) ``` diff --git a/Comp/r-sas_kruskalwallis.qmd b/Comp/r-sas_kruskalwallis.qmd index 63690bb74..559b1e2a3 100644 --- a/Comp/r-sas_kruskalwallis.qmd +++ b/Comp/r-sas_kruskalwallis.qmd @@ -7,6 +7,7 @@ title: "Kruskal Wallis R v SAS" From the individual R and SAS pages, performing the Kruskal-Wallis test in R using: ```{r} +#| eval: false stats::kruskal.test(Sepal_Width ~ Species, data = iris_sub) ``` diff --git a/Comp/r-sas_mmrm.qmd b/Comp/r-sas_mmrm.qmd index 714f13d1d..32e9b2a25 100644 --- a/Comp/r-sas_mmrm.qmd +++ b/Comp/r-sas_mmrm.qmd @@ -4,6 +4,7 @@ execute: message: false warning: false echo: true + eval: false --- ```{r} @@ -20,7 +21,7 @@ knitr::opts_chunk$set( ```{r} #| label: review-setup #| include: false - +#| eval: true library(dplyr) library(purrr) library(microbenchmark) @@ -185,6 +186,7 @@ RUN; ### `mmrm` ```{r} + mmrm::mmrm( formula = FEV1 ~ ARMCD * AVISIT + ar1h(VISITN | USUBJID), data = fev_data @@ -482,6 +484,7 @@ The `mmrm`, `PROC GLIMMIX`, `gls`, `lmer`, and `glmmTMB` functions are applied t ```{r} #| echo: false +#| eval: true # format table in markdown cached_mmrm_results$conv_time_fev |> @@ -506,6 +509,7 @@ The MMRM implementations are now applied to the BCVA dataset 10 times. The conve ```{r} #| echo: false +#| eval: true # format table in markdown cached_mmrm_results$conv_time_bcva |> arrange(median) |> @@ -532,6 +536,7 @@ We next estimate the marginal mean treatment effects for each visit in the FEV a ```{r} #| label: review-treatment-fev #| echo: false +#| eval: true # plot estimates ggplot( cached_mmrm_results$rel_diff_ests_tbl_fev, @@ -554,6 +559,7 @@ The R procedures' estimates are very similar to those output by `PROC GLIMMIX`, ```{r} #| label: review-treatment-bcva #| echo: false +#| eval: true # plot estimates ggplot( cached_mmrm_results$rel_diff_ests_tbl_bcva, @@ -598,6 +604,7 @@ Ten datasets of 200 patients are generated each of the following levels of missi ```{r} #| label: review-missingness-table #| echo: false +#| eval: true ## construct the table cached_mmrm_results$df_missingness |> kable(caption = "Number of patients per visit") @@ -608,6 +615,7 @@ The convergence rates of all implementations for stratified by missingness level ```{r} #| label: review-convergence-rate-missingness #| echo: false +#| eval: true ## plot the convergence rates cached_mmrm_results$conv_rate |> mutate( diff --git a/Comp/r-sas_psmatch.qmd b/Comp/r-sas_psmatch.qmd index a347467e2..ad0c525d9 100644 --- a/Comp/r-sas_psmatch.qmd +++ b/Comp/r-sas_psmatch.qmd @@ -1,5 +1,7 @@ --- title: "Propensity Score Matching" +execute: + eval: false --- # Introduction From b72d8c39c624b73a4ffb941da060e4ff5baae4ad Mon Sep 17 00:00:00 2001 From: Michael Walshe <62556482+michaelwalshe@users.noreply.github.com> Date: Tue, 10 Mar 2026 12:46:29 +0000 Subject: [PATCH 07/13] Update GLMM to use SAS as markdown renderer, not {r} with eval: false --- SAS/glmm.qmd | 41 ++++++++++++++++------------------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/SAS/glmm.qmd b/SAS/glmm.qmd index 13c017fe2..ae4ba2fe4 100644 --- a/SAS/glmm.qmd +++ b/SAS/glmm.qmd @@ -60,8 +60,7 @@ To uniquely identify subjects, a new variable USUBJID was created by concatenati Additionally, two variables were created using randomly generated values to simulate variables with more than two categories. One was an ordinal variable with values 1, 2, and 3; the other was a nominal variable with categories 'liver', 'lung', and 'bone'. The following SAS code was used: -```{r} -#| eval: false +```sas proc format; value respmulti 1='Liver' @@ -83,8 +82,7 @@ GLMM with GHQ approximation can be fitted using `PROC GLIMMIX` by specifying `qu Unlike in GEE models, which computed the robust Sandwich S.E. by default, the GLIMMIX procedure displays the model-based S.E. (also called naïve S.E. in R) by default. -```{r} -#| eval: false +```sas proc glimmix data=resp method=quad(qpoints=5); class trtp(ref="P") avisitn(ref='1') usubjid; model outcome=trtp avisitn trtp*avisitn/dist=bin link=logit solution ddfm=BW;/*1*/; @@ -108,8 +106,7 @@ knitr::include_graphics("../images/glmm/SAS_Image_1.png") Laplace is a particular GHQ where only one point is used. In SAS, it can be obtained in the method statement using either `method=quad(qpoints=)` or `method=Laplace`. Both approaches return similar results with slight differences in later decimal places (See Appendix 1). -```{r} -#| eval: false +```sas proc glimmix data=resp method=laplace; class trtp(ref="A") avisitn(ref='1') usubjid; model outcome=trtp avisitn trtp*avisitn/dist=bin link=logit solution ddfm=BW; @@ -131,8 +128,7 @@ The PQL approach uses linear approximations instead of likelihood, making it **l In SAS, this is implemented by default using the Residual Pseudo-Likelihood method (`method=RSPL)`, which is a refinement of PQL which incorporates residual adjustments to better approximate the marginal likelihood, in the GLIMMIX procedure. -```{r} -#| eval: false +```sas proc glimmix data=resp method=rspl; class trtp(ref="A") avisitn(ref='1') usubjid; model outcome=trtp avisitn trtp*avisitn/ dist=bin link=logit solution ddfm=residual; @@ -156,8 +152,7 @@ Additionally, FDA advises "*sponsors to consider using of robust SE method such The example below is done using GHQ with n=5 points, but it also works for Laplace approximation. -```{r} -#| eval: false +```sas proc glimmix data=resp method=quad(qpoints=5) empirical; class trtp(ref="A") avisitn(ref='1') usubjid; model outcome=trtp avisitn trtp*avisitn/dist=bin link=logit solution ddfm=BW; @@ -186,15 +181,14 @@ Predicted probabilities and odds ratios (ORs) can be obtained in SAS using the ` The example below is done using GHQ with n=5 points, but it also works for Laplace approximation. -```{r} -#| eval: false - proc glimmix data=resp method=quad(qpoints=5) empirical; - class trtp(ref="P") avisitn(ref='1') usubjid; - model outcome=trtp avisitn trtp*avisitn / dist=bin link=logit solution ddfm=betwithin ; - lsmeans trtp*avisitn /cl ilink oddsratio diff; - random intercept /subject=usubjid ; - ods output LSMeans = Lsmeans Diffs=Diffs1; - run; +```sas +proc glimmix data=resp method=quad(qpoints=5) empirical; +class trtp(ref="P") avisitn(ref='1') usubjid; +model outcome=trtp avisitn trtp*avisitn / dist=bin link=logit solution ddfm=betwithin ; +lsmeans trtp*avisitn /cl ilink oddsratio diff; +random intercept /subject=usubjid ; +ods output LSMeans = Lsmeans Diffs=Diffs1; +run; ``` @@ -224,8 +218,7 @@ One notable limitation is that the `LSMEANS` statement does not work as expected ### Ordinal variable -```{r} -#| eval: false +```sas proc glimmix data=resp method=quad(qpoints=5) empirical; class trtp(ref="P") avisitn(ref='1') usubjid; model respord=trtp avisitn trtp*avisitn base / dist=multinomial link=cumlogit solution oddsratio ddfm=betwithin; @@ -244,8 +237,7 @@ knitr::include_graphics("../images/glmm/SAS_Image_7.png") ### Nominal variable -```{r} -#| eval: false +```sas proc glimmix data=resp method=quad(qpoints=5) ; class trtp(ref="P") avisitn(ref='1') usubjid respnom; model respnom(order=freq ref=first)=trtp avisitn trtp*avisitn base / dist=multinomial link=glogit solution ddfm=betwithin ; @@ -267,8 +259,7 @@ knitr::include_graphics("../images/glmm/SAS_Image_8.png") ### Laplace: GLIMMIX with method=Laplace vs method=quad(qpoints=1) -```{r} -#| eval: false +```sas proc glimmix data=resp method=laplace empirical; class trtp(ref="P") avisitn(ref='1') usubjid; model outcome=trtp avisitn trtp*avisitn/ dist=bin link=logit solution ddfm=BW; From 458fd375a59121594c4ffaa91df30c61cd69ae1c Mon Sep 17 00:00:00 2001 From: Michael Walshe Date: Wed, 11 Mar 2026 09:22:52 +0000 Subject: [PATCH 08/13] Render freeze on WSL --- .../r-east_gsd_tte/execute-results/html.json | 4 +- .../execute-results/html.json | 4 +- .../execute-results/html.json | 8 +- .../execute-results/html.json | 2 +- .../r-sas_anova/execute-results/html.json | 8 +- .../r-sas_chi-sq/execute-results/html.json | 8 +- .../r-sas_friedman/execute-results/html.json | 8 +- .../figure-html/unnamed-chunk-2-1.png | Bin 0 -> 42709 bytes .../Comp/r-sas_gee/execute-results/html.json | 8 +- .../execute-results/html.json | 4 +- .../execute-results/html.json | 8 +- .../r-sas_mcnemar/execute-results/html.json | 4 +- .../Comp/r-sas_mmrm/execute-results/html.json | 8 +- .../review-convergence-rate-missingness-1.png | Bin 0 -> 94566 bytes .../figure-html/review-treatment-bcva-1.png | Bin 0 -> 60506 bytes .../figure-html/review-treatment-bcva-2.png | Bin 0 -> 68563 bytes .../figure-html/review-treatment-fev-1.png | Bin 0 -> 55478 bytes .../r-sas_negbin/execute-results/html.json | 8 +- .../r-sas_psmatch/execute-results/html.json | 8 +- .../r-sas_survival/execute-results/html.json | 8 +- .../execute-results/html.json | 2 +- .../execute-results/html.json | 2 +- .../execute-results/html.json | 2 +- _freeze/R/ancova/execute-results/html.json | 2 +- .../ancova/figure-html/unnamed-chunk-14-1.png | Bin 43729 -> 20930 bytes .../execute-results/html.json | 15 ++++ .../execute-results/html.json | 15 ++++ .../R/ci_for_prop/execute-results/html.json | 8 +- .../R/friedman_test/execute-results/html.json | 2 +- .../figure-html/visualization-1.png | Bin 66768 -> 39334 bytes .../execute-results/html.json | 2 +- .../R/logistic_regr/execute-results/html.json | 2 +- .../execute-results/html.json | 2 +- .../execute-results/html.json | 2 +- .../figure-html/explore data 2-1.png | Bin 47614 -> 30156 bytes .../execute-results/html.json | 2 +- _freeze/R/rounding/execute-results/html.json | 2 +- .../execute-results/html.json | 8 +- .../R/summary-stats/execute-results/html.json | 2 +- .../execute-results/html.json | 2 +- .../execute-results/html.json | 2 +- _freeze/R/survival/execute-results/html.json | 2 +- .../figure-html/unnamed-chunk-8-1.png | Bin 144072 -> 142982 bytes .../R/survival_cif/execute-results/html.json | 2 +- .../R/survival_cif/figure-html/cr.plot-1.png | Bin 94727 -> 77448 bytes .../R/survival_cif/figure-html/cr.plot-2.png | Bin 95835 -> 79022 bytes .../R/survival_csh/execute-results/html.json | 2 +- .../R/tipping_point/execute-results/html.json | 2 +- .../MAR_v1_tipping_point_visualization-1.png | Bin 104652 -> 76012 bytes .../MAR_v2_tipping_point_visualization-1.png | Bin 71879 -> 48933 bytes .../figure-html/data_exploration_4-1.png | Bin 47614 -> 30156 bytes .../execute-results/html.json | 2 +- .../R/ttest_1Sample/execute-results/html.json | 2 +- .../R/ttest_2Sample/execute-results/html.json | 6 +- .../R/ttest_Paired/execute-results/html.json | 2 +- _freeze/R/xgboost/execute-results/html.json | 2 +- _freeze/SAS/ancova/execute-results/html.json | 8 +- _freeze/SAS/anova/execute-results/html.json | 8 +- .../SAS/association/execute-results/html.json | 8 +- .../execute-results/html.json | 15 ++++ .../execute-results/html.json | 15 ++++ .../SAS/ci_for_prop/execute-results/html.json | 4 +- _freeze/SAS/cmh/execute-results/html.json | 8 +- .../execute-results/html.json | 8 +- _freeze/SAS/gee/execute-results/html.json | 4 +- _freeze/SAS/glmm/execute-results/html.json | 4 +- _freeze/SAS/gsd-tte/execute-results/html.json | 8 +- .../kruskal_wallis/execute-results/html.json | 8 +- _freeze/SAS/manova/execute-results/html.json | 8 +- _freeze/SAS/mcnemar/execute-results/html.json | 8 +- .../execute-results/html.json | 8 +- .../execute-results/html.json | 8 +- .../execute-results/html.json | 8 +- _freeze/SAS/rmst/execute-results/html.json | 4 +- .../SAS/rounding/execute-results/html.json | 6 +- .../execute-results/html.json | 8 +- .../execute-results/html.json | 4 +- .../execute-results/html.json | 4 +- .../execute-results/html.json | 4 +- .../execute-results/html.json | 8 +- .../execute-results/html.json | 4 +- .../SAS/survival/execute-results/html.json | 8 +- .../survival_cif/execute-results/html.json | 8 +- .../survival_csh/execute-results/html.json | 8 +- .../tipping_point/execute-results/html.json | 8 +- .../execute-results/html.json | 4 +- .../ttest_1Sample/execute-results/html.json | 8 +- .../ttest_2Sample/execute-results/html.json | 4 +- .../ttest_Paired/execute-results/html.json | 8 +- .../wilcoxonsr_HL/execute-results/html.json | 8 +- .../execute-results/html.json | 2 +- .../execute-results/html.json | 4 +- .../execute-results/html.json | 4 +- data/quarto_pkg_dependencies.csv | 84 ++++++++++-------- 94 files changed, 333 insertions(+), 199 deletions(-) create mode 100644 _freeze/Comp/r-sas_friedman/figure-html/unnamed-chunk-2-1.png create mode 100644 _freeze/Comp/r-sas_mmrm/figure-html/review-convergence-rate-missingness-1.png create mode 100644 _freeze/Comp/r-sas_mmrm/figure-html/review-treatment-bcva-1.png create mode 100644 _freeze/Comp/r-sas_mmrm/figure-html/review-treatment-bcva-2.png create mode 100644 _freeze/Comp/r-sas_mmrm/figure-html/review-treatment-fev-1.png create mode 100644 _freeze/R/ci_for_2indep_prop/execute-results/html.json create mode 100644 _freeze/R/ci_for_paired_prop/execute-results/html.json create mode 100644 _freeze/SAS/ci_for_2indep_prop/execute-results/html.json create mode 100644 _freeze/SAS/ci_for_paired_prop/execute-results/html.json diff --git a/_freeze/Comp/r-east_gsd_tte/execute-results/html.json b/_freeze/Comp/r-east_gsd_tte/execute-results/html.json index 4ab24d44f..cf8e9f310 100644 --- a/_freeze/Comp/r-east_gsd_tte/execute-results/html.json +++ b/_freeze/Comp/r-east_gsd_tte/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "70409931020d2271604c50b8dc5f9ef5", + "hash": "8836906a81288cda8e91a752b4916545", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs EAST vs SAS: Group sequential design\"\neditor_options: \n chunk_output_type: console\nexecute: \n eval: false\n---\n\n## Introduction\n\nIn this vignette, we briefly compare sample size/power calculations for a group sequential design (GSD) for time to event endpoints between EAST and [gsDesign](https://keaven.github.io/gsDesign/), [gsDesign2](https://merck.github.io/gsDesign2/), and [rpact](https://www.rpact.org/). Note that, a comparison between rpact and gsDesign has been previously reported [here](https://www.rpact.org/vignettes/planning/rpact_vs_gsdesign_examples/#comparison-analysis-time-of-rpact-vs.-gsdesign). Additionally, we present comparative results between SAS `PROC SEQDESIGN` and rpact to provide a comprehensive evaluation framework.\n\nThere are two main methods that are generally used for GSD sample-size/power calculations for time to event endpoints under proportional hazard assumption:\n\n- Lachin & Foulkes (LF) Method (1986)\n- Kim & Tsiatis (KT) Method (1990)\n\nThe main difference between the two methods is that LF method requires specification of accrual duration as well as study duration, while KT method calculates study duration iteratively given accrual rates and accrual duration. In general, these two methods produce similar, but not identical results.\n\nBoth LF and KT methods are implemented in gsDesign and [SAS](https://documentation.sas.com/doc/en/pgmsascdc/v_067/statug/statug_seqdesign_details42.htm#statug.seqdesign.cseqdinputd), while KT method is implemented in EAST and rpact. gsDesign2 uses a modification of the LF method while applying an average hazard ratio (AHR) approach for non-proportional hazards (Schemper, Wakounig, and Heinze, 2009, Yung and Liu 2020). gsDesign2 also enables use of the sample size method of Yung and Liu (2020).\n\nOne additional computational difference to note for EAST vs gsDesign/gsDesign2 is the usage of different log hazard ratio variance assumptions. By default, EAST uses the variance under the null hypothesis and provides an option for using the variance under the alternative hypothesis. gsDesign, on the other hand, is using both of these variances as suggested by Lachin and Foulkes (1986). gsDesign2 has `info_scale` argument in `gsDesign2::gs_power_ahr()`, `gsDesign2::gs_design_ahr()`, which could be set to variance under the null or alternative hypothesis or to the combination of variances.\n\nBelow we provide an example of reproducing EAST results from [this vignette](https://psiaims.github.io/CAMIS/East/gsd-tte.html) using gsDesign/gsDesign2/rpact. As shown in the example, gsDesign2 and rpact can reproduce EAST calculations for GSD boundaries, while gsDesign results have minor differences. Similarly, our comparison between SAS `PROC SEQDESIGN` and rpact shows good agreement in the calculation, with only minimal numerical differences observed. gsDesign has an option under development to support a complete concordance with EAST.\n\n## Design example\n\nWe assume that a GSD is utilized for progression-free survival (PFS) endpoint. It will be tested at one interim analysis (IA) for both efficacy and non-binding futility and then at final analysis (FA). O'Brien-Fleming spending function will be used for efficacy testing and Hwang-Shih-DeCani spending function with $\\gamma = -10$ will be used for futility.\n\nFurther design assumptions are as follows:\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# PFS HR=0.6\nhr1_pfs <- 0.6\n# median PFS of 9.4 months in the control arm\nmed_pfs <- 9.4\n# minimum follow-up of 10 months for PFS\nminfu_pfs <- 10\n# Monthly exponential dropout of 0.019 for PFS\ndo_rate_pfs <- 0.019\n# IA timing for PFS is at approximately 75% information fraction, and is derived\n# using the number of events that was calculated by EAST which sets integer event counts to approximate targeted information\ntiming_pfs_rpact <- c(176 / 235, 1)\ntiming_pfs_gs <- c(0.75, 1)\n\n# power of approximately 95% for PFS, EAST reported power will be used\npower_pfs <- 0.9505021\n\n# Enrollment period of 24 months\nenroll_dur <- 24\n# 1:1 randomization ratio\nrand_ratio <- 1\n# alpha level of 1.25%\nalphal <- 0.0125\n```\n:::\n\n\nWe assume that EAST was initially used to calculate the target number of events and the total sample size, and we will use gsDesign/gsDesign2/rpact to reproduce those.\n\nNote that, in EAST the number of target events is reported as an integer, however, gsDesign/gsDesign2/rpact by default provide non-integer values which match *exactly* the specified information fraction. Both gsDesign/gsDesign2 can facilitate computations using integer number of events with `gsDesign::toInteger()` and `gsDesign2::to_integer()` as shown below. In order to reproduce EAST results with rpact, we will use the number of events that was calculated in EAST for `informationRates` argument in `rpact::getDesignGroupSequential()`: 176 and 235 PFS events for IA and FA respectively (please see the `timing_pfs_rpact` object in the code above).\n\nFor ease of comparison the results from EAST are summarized below:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

-2.6606

-0.7379

N=398

p (1-sided)

0.0039

0.2303

Events: 176

HR at bound

0.6696

0.8947

Month: 25

P(Cross) if HR=1

0.0039

0.7697

P(Cross) if HR=0.60

0.7666

0.0040

FA

Z

-2.2798

N=398

p (1-sided)

0.0113

Events: 235

HR at bound

0.7427

Month: 34

P(Cross) if HR=1

0.0125

P(Cross) if HR=0.60

0.9505

\n```\n\n:::\n:::\n\n\n- The comparison between EAST and gsDesign/gsDesign/rpact results is presented below using absolute difference in efficacy/futility boundaries and crossing probabilities up to 4 decimals. Non-zero values are highlighted.\n- Note that, in gsDesign/gsDesign Efficacy/Futility bounds refer to upper/lower bounds respectively, while in EAST these refer to the opposite directions, i.e., lower/upper bounds respectively. For the comparison purposes, we will assume that Efficacy/Futility bounds refer to upper/lower bounds respectively.\n\n## Code to reproduce EAST results\n\n### gsDesign code\n\n- gsDesign code to reproduce the above EAST results:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(gsDesign)\n\npfs_gsDesign <- gsDesign::gsSurv(\n k = length(timing_pfs_gs),\n timing = timing_pfs_gs,\n R = enroll_dur,\n eta = do_rate_pfs,\n minfup = minfu_pfs,\n T = enroll_dur + minfu_pfs,\n lambdaC = log(2) / med_pfs,\n hr = hr1_pfs,\n beta = 1 - power_pfs,\n alpha = alphal,\n sfu = sfLDOF,\n sfl = sfHSD,\n sflpar = -10,\n test.type = 4\n) |>\n toInteger()\n\n\npfs_gsDesign |>\n gsDesign::gsBoundSummary()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Analysis Value Efficacy Futility\n IA 1: 75% Z 2.6606 0.7422\n N: 400 p (1-sided) 0.0039 0.2290\n Events: 176 ~HR at bound 0.6696 0.8941\n Month: 25 P(Cross) if HR=1 0.0039 0.7710\n P(Cross) if HR=0.6 0.7679 0.0040\n Final Z 2.2798 2.2798\n N: 400 p (1-sided) 0.0113 0.0113\n Events: 235 ~HR at bound 0.7427 0.7427\n Month: 34 P(Cross) if HR=1 0.0125 0.9875\n P(Cross) if HR=0.6 0.9510 0.0490\n```\n\n\n:::\n:::\n\n\n- gsDesign vs EAST comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

0.0000

0.0043

N=398

p (1-sided)

0.0000

0.0013

Events: 176

HR at bound

0.0000

0.0006

Month: 25

P(Cross) if HR=1

0.0000

0.0013

P(Cross) if HR=0.60

0.0013

0.0000

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

0.0000

P(Cross) if HR=0.60

0.0005

\n```\n\n:::\n:::\n\n\n### gsDesign2 code\n\n- gsDesign2 code to reproduce the above EAST results appears below.\n- Note that, here `gsDesign2::gs_power_ahr()` is used given the number of target events for each analysis based on EAST results.\n\n\n::: {.cell}\n\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(gsDesign2)\nlibrary(tibble)\n\nenroll_rate <- tibble(\n stratum = \"All\",\n duration = enroll_dur,\n rate = 398 / enroll_dur\n)\nfail_rate_pfs <- tibble(\n stratum = \"All\",\n duration = Inf, #could be set to Inf when proportional hazard is assumed\n fail_rate = log(2) / med_pfs,\n hr = hr1_pfs,\n dropout_rate = do_rate_pfs\n)\n\npfs_gsDesign2 <- gs_power_ahr(\n enroll_rate = enroll_rate,\n fail_rate = fail_rate_pfs,\n ratio = rand_ratio,\n event = c(176, 235),\n upper = gs_spending_bound,\n upar = list(\n sf = gsDesign::sfLDOF,\n total_spend = alphal\n ),\n lower = gs_spending_bound,\n lpar = list(\n sf = gsDesign::sfHSD,\n total_spend = 1 - power_pfs,\n param = -10\n ),\n info_scale = \"h0_info\"\n) |>\n to_integer()\n\npfs_gsDesign2 |>\n summary() |>\n gsDesign2::as_gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n \n \n \n \n \n \n \n \n
Bound summary for AHR design
AHR approximations of ~HR at bound
BoundZNominal p1~HR at bound2\n
Cumulative boundary crossing probability
\n
Alternate hypothesisNull hypothesis
Analysis: 1 Time: 25.4 N: 398 Events: 176 AHR: 0.6 Information fraction: 0.75
Futility0.740.23030.89470.00400.7697
Efficacy2.660.00390.66960.76660.0039
Analysis: 2 Time: 34.1 N: 398 Events: 235 AHR: 0.6 Information fraction: 1
Futility2.280.01130.74270.04950.9875
Efficacy2.280.01130.74270.95050.0125
1 One-sided p-value for experimental vs control treatment. Value < 0.5 favors experimental, > 0.5 favors control.
2 Approximate hazard ratio to cross bound.
\n
\n```\n\n:::\n:::\n\n\n- gsDesign2 vs EAST comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

0.0000

0.0000

N=398

p (1-sided)

0.0000

0.0000

Events: 176

HR at bound

0.0000

0.0000

Month: 25

P(Cross) if HR=1

0.0000

0.0000

P(Cross) if HR=0.60

0.0000

0.0000

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

0.0000

P(Cross) if HR=0.60

0.0000

\n```\n\n:::\n:::\n\n\n### rpact code\n\n- rpact code to reproduce the above EAST results appears below.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(rpact)\n\npfs_rpact_gsd <- rpact::getDesignGroupSequential(\n sided = 1,\n alpha = alphal,\n informationRates = timing_pfs_rpact,\n typeOfDesign = \"asOF\",\n beta = 1 - power_pfs,\n typeBetaSpending = \"bsHSD\",\n gammaB = -10,\n bindingFutility = FALSE\n)\n\npfs_rpact <- rpact::getSampleSizeSurvival(\n design = pfs_rpact_gsd,\n accrualTime = enroll_dur,\n followUpTime = minfu_pfs,\n lambda2 = log(2) / med_pfs,\n hazardRatio = hr1_pfs,\n dropoutRate1 = 0.2,\n dropoutRate2 = 0.2,\n dropoutTime = 12\n)\n\nkable(summary(pfs_rpact))\n```\n\n::: {.cell-output-display}\n*Sample size calculation for a survival endpoint*\n\nSequential analysis with a maximum of 2 looks (group sequential design), \none-sided overall significance level 1.25%, power 95.1%.\nThe results were calculated for a two-sample logrank test, \nH0: hazard ratio = 1, H1: hazard ratio = 0.6, control lambda(2) = 0.074, \naccrual time = 24, accrual intensity = 16.6, follow-up time = 10, \ndropout rate(1) = 0.2, dropout rate(2) = 0.2, dropout time = 12.\n\n| Stage | 1 | 2 |\n| ----- | ----- | ----- |\n| Planned information rate | 74.9% | 100% |\n| Cumulative alpha spent | 0.0039 | 0.0125 |\n| Cumulative beta spent | 0.0040 | 0.0495 |\n| Stage levels (one-sided) | 0.0039 | 0.0113 |\n| Efficacy boundary (z-value scale) | 2.661 | 2.280 |\n| Futility boundary (z-value scale) | 0.738 | |\n| Efficacy boundary (t) | 0.670 | 0.743 |\n| Futility boundary (t) | 0.895 | |\n| Cumulative power | 0.7666 | 0.9505 |\n| Number of subjects | 397.9 | 397.9 |\n| Expected number of subjects under H1 | | 397.9 |\n| Cumulative number of events | 176.0 | 235.0 |\n| Expected number of events under H1 | | 189.5 |\n| Analysis time | 25.34 | 34.00 |\n| Expected study duration under H1 | | 27.32 |\n| Overall exit probability (under H0) | 0.7736 | |\n| Overall exit probability (under H1) | 0.7707 | |\n| Exit probability for efficacy (under H0) | 0.0039 | |\n| Exit probability for efficacy (under H1) | 0.7666 | |\n| Exit probability for futility (under H0) | 0.7697 | |\n| Exit probability for futility (under H1) | 0.0040 | |\n\nLegend:\n\n* *(t)*: treatment effect scale\n\n:::\n:::\n\n\n- rpact vs EAST comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

0.0000

0.0000

N=398

p (1-sided)

0.0000

0.0000

Events: 176

HR at bound

0.0000

0.0000

Month: 25

P(Cross) if HR=1

0.0000

0.0000

P(Cross) if HR=0.60

0.0000

0.0000

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

0.0000

P(Cross) if HR=0.60

0.0000

\n```\n\n:::\n:::\n\n\n### SAS code\n\n- SAS code to reproduce the above rpact results appears below.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC SEQDESIGN BOUNDARYSCALE=MLE ERRSPEND;\n DESIGN NSTAGES=2 \n INFO=CUM(0.748936170212766 1.0) \n ALT=UPPER \n ALPHA=0.0125 \n BETA=0.05\n METHOD(ALPHA)=ERRFUNCOBF \n METHOD(BETA)=ERRFUNCGAMMA(GAMMA=-10) \n STOP=BOTH(BETABOUNDARY=NONBINDING);\n SAMPLESIZE MODEL=TWOSAMPLESURVIVAL(\n NULLMEDSURVTIME=9.4\n HAZARDRATIO=0.6\n ACCTIME=24 \n FOLTIME=10\n LOSS=EXP(HAZARD=0.018595295942851)\n WEIGHT=1);\n ODS OUTPUT Boundary=BMLE SampleSize=SS SampleSizeSummary=SSS;\nRUN;\n```\n:::\n\n\nThe following shows the events (D) and required sample sizes (N) for IA and FA.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/gsd-tte/result-pfs-sas-comp.png){fig-align='center' width=100%}\n:::\n:::\n\n\nPlease note that the `BOUNDARYSCALE=MLE | SCORE | STDZ | PVALUE` options display the boundary values in the MLE, standardize Z, score, and p-value scales, respectively. SAS will provide a boundary information table based on the specified `BOUNDARYSCALE`. In the information table, Alpha indicates the efficacy boundaries, and Beta indicates futility boundaries.\n\nSAS doesn't provide a boundary information with HR, so the HR boundaries is obtained from the MLE boundaries (as MLE $=\\hat{\\theta}=-log(\\text{HR})$, see [SAS User's Guide: Test for Two Survival Distributions with a Log-Rank Test](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/statug/statug_seqdesign_details52.htm#statug.seqdesign.cseqdlogrank)) via the following code.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nDATA BHR;\n SET BMLE;\n Bound_UA_HR=exp(-Bound_UA);\n Bound_UB_HR=exp(-Bound_UB);\n LABEL BOUND_UA_HR=\"Upper Alpha (HR)\" BOUND_UA_HR=\"Upper Beta (HR)\";\nPROC PRINT LABEL;\n VAR _Stage_ _InfoProp_ Bound_UA Bound_UB Bound_UA_HR Bound_UB_HR;\nRUN;\n```\n:::\n\n\nThe HR boundaries are shown below.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/gsd-tte/result-pfs-sas-bhr.png){fig-align='center' width=100%}\n:::\n:::\n\n\nThe results calculated by SAS are presneted in the table below. Please note that SAS doesn't report the probablities $P(Cross | HR=1)$ and $P(Cross | HR=0.6)$, resulting in empty cells for these results in the table.\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

2.6606

0.7379

N=398

p (1-sided)

0.0039

0.2303

Events: 176

HR at bound

0.6696

0.8947

Month: 25

P(Cross) if HR=1

P(Cross) if HR=0.6

FA

Z

2.2798

N=398

p (1-sided)

0.0113

Events: 235

HR at bound

0.7427

Month: 34

P(Cross) if HR=1

P(Cross) if HR=0.6

\n```\n\n:::\n:::\n\n\n- SAS vs rapct comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

0.0000

0.0000

N=398

p (1-sided)

0.0000

0.0000

Events: 176

HR at bound

0.0000

0.0000

Month: 25

P(Cross) if HR=1

P(Cross) if HR=0.6

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

P(Cross) if HR=0.6

\n```\n\n:::\n:::\n\n\n- SAS vs EAST comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

diff_eff_sas

diff_fut_sas

IA1: 75%

Z

0.0000

0.0000

N=398

p (1-sided)

0.0000

0.0000

Events: 176

HR at bound

0.0000

0.0000

Month: 25

P(Cross) if HR=1

P(Cross) if HR=0.60

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

P(Cross) if HR=0.60

\n```\n\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsessionInfo()\n```\n:::\n\n\n## References\n\n- Lachin JM and Foulkes M. Evaluation of sample size and power for analyses of survival with allowance for nonuniform patient entry, losses to follow-up, non-compliance, and stratification. Biometrics 1986;42:507-19.\n- Kim K and Tsiatis AA. Study duration for clinical trials with survival response and early stopping rule. Biometrics 1990(46): 81-92.\n- Schemper M, Wakounig S and Heinze G. The estimation of average hazard ratios by weighted cox regression. Statistics in Medicine 2009; 28(19): 2473-2489.\n- Yung G and Liu Y. Sample size and power for the weighted log-rank test and Kaplan-Meier based tests with allowance for nonproportional hazards. Biometrics 2020;76:939-50.", + "markdown": "---\ntitle: \"R vs EAST vs SAS: Group sequential design\"\neditor_options: \n chunk_output_type: console\n---\n\n## Introduction\n\nIn this vignette, we briefly compare sample size/power calculations for a group sequential design (GSD) for time to event endpoints between EAST and [gsDesign](https://keaven.github.io/gsDesign/), [gsDesign2](https://merck.github.io/gsDesign2/), and [rpact](https://www.rpact.org/). Note that, a comparison between rpact and gsDesign has been previously reported [here](https://www.rpact.org/vignettes/planning/rpact_vs_gsdesign_examples/#comparison-analysis-time-of-rpact-vs.-gsdesign). Additionally, we present comparative results between SAS `PROC SEQDESIGN` and rpact to provide a comprehensive evaluation framework.\n\nThere are two main methods that are generally used for GSD sample-size/power calculations for time to event endpoints under proportional hazard assumption:\n\n- Lachin & Foulkes (LF) Method (1986)\n- Kim & Tsiatis (KT) Method (1990)\n\nThe main difference between the two methods is that LF method requires specification of accrual duration as well as study duration, while KT method calculates study duration iteratively given accrual rates and accrual duration. In general, these two methods produce similar, but not identical results.\n\nBoth LF and KT methods are implemented in gsDesign and [SAS](https://documentation.sas.com/doc/en/pgmsascdc/v_067/statug/statug_seqdesign_details42.htm#statug.seqdesign.cseqdinputd), while KT method is implemented in EAST and rpact. gsDesign2 uses a modification of the LF method while applying an average hazard ratio (AHR) approach for non-proportional hazards (Schemper, Wakounig, and Heinze, 2009, Yung and Liu 2020). gsDesign2 also enables use of the sample size method of Yung and Liu (2020).\n\nOne additional computational difference to note for EAST vs gsDesign/gsDesign2 is the usage of different log hazard ratio variance assumptions. By default, EAST uses the variance under the null hypothesis and provides an option for using the variance under the alternative hypothesis. gsDesign, on the other hand, is using both of these variances as suggested by Lachin and Foulkes (1986). gsDesign2 has `info_scale` argument in `gsDesign2::gs_power_ahr()`, `gsDesign2::gs_design_ahr()`, which could be set to variance under the null or alternative hypothesis or to the combination of variances.\n\nBelow we provide an example of reproducing EAST results from [this vignette](https://psiaims.github.io/CAMIS/East/gsd-tte.html) using gsDesign/gsDesign2/rpact. As shown in the example, gsDesign2 and rpact can reproduce EAST calculations for GSD boundaries, while gsDesign results have minor differences. Similarly, our comparison between SAS `PROC SEQDESIGN` and rpact shows good agreement in the calculation, with only minimal numerical differences observed. gsDesign has an option under development to support a complete concordance with EAST.\n\n## Design example\n\nWe assume that a GSD is utilized for progression-free survival (PFS) endpoint. It will be tested at one interim analysis (IA) for both efficacy and non-binding futility and then at final analysis (FA). O'Brien-Fleming spending function will be used for efficacy testing and Hwang-Shih-DeCani spending function with $\\gamma = -10$ will be used for futility.\n\nFurther design assumptions are as follows:\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# PFS HR=0.6\nhr1_pfs <- 0.6\n# median PFS of 9.4 months in the control arm\nmed_pfs <- 9.4\n# minimum follow-up of 10 months for PFS\nminfu_pfs <- 10\n# Monthly exponential dropout of 0.019 for PFS\ndo_rate_pfs <- 0.019\n# IA timing for PFS is at approximately 75% information fraction, and is derived\n# using the number of events that was calculated by EAST which sets integer event counts to approximate targeted information\ntiming_pfs_rpact <- c(176 / 235, 1)\ntiming_pfs_gs <- c(0.75, 1)\n\n# power of approximately 95% for PFS, EAST reported power will be used\npower_pfs <- 0.9505021\n\n# Enrollment period of 24 months\nenroll_dur <- 24\n# 1:1 randomization ratio\nrand_ratio <- 1\n# alpha level of 1.25%\nalphal <- 0.0125\n```\n:::\n\n\nWe assume that EAST was initially used to calculate the target number of events and the total sample size, and we will use gsDesign/gsDesign2/rpact to reproduce those.\n\nNote that, in EAST the number of target events is reported as an integer, however, gsDesign/gsDesign2/rpact by default provide non-integer values which match *exactly* the specified information fraction. Both gsDesign/gsDesign2 can facilitate computations using integer number of events with `gsDesign::toInteger()` and `gsDesign2::to_integer()` as shown below. In order to reproduce EAST results with rpact, we will use the number of events that was calculated in EAST for `informationRates` argument in `rpact::getDesignGroupSequential()`: 176 and 235 PFS events for IA and FA respectively (please see the `timing_pfs_rpact` object in the code above).\n\nFor ease of comparison the results from EAST are summarized below:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

-2.6606

-0.7379

N=398

p (1-sided)

0.0039

0.2303

Events: 176

HR at bound

0.6696

0.8947

Month: 25

P(Cross) if HR=1

0.0039

0.7697

P(Cross) if HR=0.60

0.7666

0.0040

FA

Z

-2.2798

N=398

p (1-sided)

0.0113

Events: 235

HR at bound

0.7427

Month: 34

P(Cross) if HR=1

0.0125

P(Cross) if HR=0.60

0.9505

\n```\n\n:::\n:::\n\n\n- The comparison between EAST and gsDesign/gsDesign/rpact results is presented below using absolute difference in efficacy/futility boundaries and crossing probabilities up to 4 decimals. Non-zero values are highlighted.\n- Note that, in gsDesign/gsDesign Efficacy/Futility bounds refer to upper/lower bounds respectively, while in EAST these refer to the opposite directions, i.e., lower/upper bounds respectively. For the comparison purposes, we will assume that Efficacy/Futility bounds refer to upper/lower bounds respectively.\n\n## Code to reproduce EAST results\n\n### gsDesign code\n\n- gsDesign code to reproduce the above EAST results:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(gsDesign)\n\npfs_gsDesign <- gsDesign::gsSurv(\n k = length(timing_pfs_gs),\n timing = timing_pfs_gs,\n R = enroll_dur,\n eta = do_rate_pfs,\n minfup = minfu_pfs,\n T = enroll_dur + minfu_pfs,\n lambdaC = log(2) / med_pfs,\n hr = hr1_pfs,\n beta = 1 - power_pfs,\n alpha = alphal,\n sfu = sfLDOF,\n sfl = sfHSD,\n sflpar = -10,\n test.type = 4\n) |>\n toInteger()\n\n\npfs_gsDesign |>\n gsDesign::gsBoundSummary()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Analysis Value Efficacy Futility\n IA 1: 75% Z 2.6606 0.7422\n N: 400 p (1-sided) 0.0039 0.2290\n Events: 176 ~HR at bound 0.6696 0.8941\n Month: 25 P(Cross) if HR=1 0.0039 0.7710\n P(Cross) if HR=0.6 0.7679 0.0040\n Final Z 2.2798 2.2798\n N: 400 p (1-sided) 0.0113 0.0113\n Events: 235 ~HR at bound 0.7427 0.7427\n Month: 34 P(Cross) if HR=1 0.0125 0.9875\n P(Cross) if HR=0.6 0.9510 0.0490\n```\n\n\n:::\n:::\n\n\n- gsDesign vs EAST comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

0.0000

0.0043

N=398

p (1-sided)

0.0000

0.0013

Events: 176

HR at bound

0.0000

0.0006

Month: 25

P(Cross) if HR=1

0.0000

0.0013

P(Cross) if HR=0.60

0.0013

0.0000

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

0.0000

P(Cross) if HR=0.60

0.0005

\n```\n\n:::\n:::\n\n\n### gsDesign2 code\n\n- gsDesign2 code to reproduce the above EAST results appears below.\n- Note that, here `gsDesign2::gs_power_ahr()` is used given the number of target events for each analysis based on EAST results.\n\n\n::: {.cell}\n\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(gsDesign2)\nlibrary(tibble)\n\nenroll_rate <- tibble(\n stratum = \"All\",\n duration = enroll_dur,\n rate = 398 / enroll_dur\n)\nfail_rate_pfs <- tibble(\n stratum = \"All\",\n duration = Inf, #could be set to Inf when proportional hazard is assumed\n fail_rate = log(2) / med_pfs,\n hr = hr1_pfs,\n dropout_rate = do_rate_pfs\n)\n\npfs_gsDesign2 <- gs_power_ahr(\n enroll_rate = enroll_rate,\n fail_rate = fail_rate_pfs,\n ratio = rand_ratio,\n event = c(176, 235),\n upper = gs_spending_bound,\n upar = list(\n sf = gsDesign::sfLDOF,\n total_spend = alphal\n ),\n lower = gs_spending_bound,\n lpar = list(\n sf = gsDesign::sfHSD,\n total_spend = 1 - power_pfs,\n param = -10\n ),\n info_scale = \"h0_info\"\n) |>\n to_integer()\n\npfs_gsDesign2 |>\n summary() |>\n gsDesign2::as_gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n \n \n \n \n \n \n \n \n
Bound summary for AHR design
AHR approximations of ~HR at bound
BoundZNominal p1~HR at bound2\n
Cumulative boundary crossing probability
\n
Alternate hypothesisNull hypothesis
Analysis: 1 Time: 25.4 N: 398 Events: 176 AHR: 0.6 Information fraction: 0.75
Futility0.740.23030.89470.00400.7697
Efficacy2.660.00390.66960.76660.0039
Analysis: 2 Time: 34.1 N: 398 Events: 235 AHR: 0.6 Information fraction: 1
Futility2.280.01130.74270.04950.9875
Efficacy2.280.01130.74270.95050.0125
1 One-sided p-value for experimental vs control treatment. Value < 0.5 favors experimental, > 0.5 favors control.
2 Approximate hazard ratio to cross bound.
\n
\n```\n\n:::\n:::\n\n\n- gsDesign2 vs EAST comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

0.0000

0.0000

N=398

p (1-sided)

0.0000

0.0000

Events: 176

HR at bound

0.0000

0.0000

Month: 25

P(Cross) if HR=1

0.0000

0.0000

P(Cross) if HR=0.60

0.0000

0.0000

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

0.0000

P(Cross) if HR=0.60

0.0000

\n```\n\n:::\n:::\n\n\n### rpact code\n\n- rpact code to reproduce the above EAST results appears below.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(rpact)\n\npfs_rpact_gsd <- rpact::getDesignGroupSequential(\n sided = 1,\n alpha = alphal,\n informationRates = timing_pfs_rpact,\n typeOfDesign = \"asOF\",\n beta = 1 - power_pfs,\n typeBetaSpending = \"bsHSD\",\n gammaB = -10,\n bindingFutility = FALSE\n)\n\npfs_rpact <- rpact::getSampleSizeSurvival(\n design = pfs_rpact_gsd,\n accrualTime = enroll_dur,\n followUpTime = minfu_pfs,\n lambda2 = log(2) / med_pfs,\n hazardRatio = hr1_pfs,\n dropoutRate1 = 0.2,\n dropoutRate2 = 0.2,\n dropoutTime = 12\n)\n\nkable(summary(pfs_rpact))\n```\n\n::: {.cell-output-display}\n*Sample size calculation for a survival endpoint*\n\nSequential analysis with a maximum of 2 looks (group sequential design), \none-sided overall significance level 1.25%, power 95.1%.\nThe results were calculated for a two-sample logrank test, \nH0: hazard ratio = 1, H1: hazard ratio = 0.6, control lambda(2) = 0.074, \naccrual time = 24, accrual intensity = 16.6, follow-up time = 10, \ndropout rate(1) = 0.2, dropout rate(2) = 0.2, dropout time = 12.\n\n| Stage | 1 | 2 |\n| ----- | ----- | ----- |\n| Planned information rate | 74.9% | 100% |\n| Cumulative alpha spent | 0.0039 | 0.0125 |\n| Cumulative beta spent | 0.0040 | 0.0495 |\n| Stage levels (one-sided) | 0.0039 | 0.0113 |\n| Efficacy boundary (z-value scale) | 2.661 | 2.280 |\n| Futility boundary (z-value scale) | 0.738 | |\n| Efficacy boundary (t) | 0.670 | 0.743 |\n| Futility boundary (t) | 0.895 | |\n| Cumulative power | 0.7666 | 0.9505 |\n| Number of subjects | 397.9 | 397.9 |\n| Expected number of subjects under H1 | | 397.9 |\n| Cumulative number of events | 176.0 | 235.0 |\n| Expected number of events under H1 | | 189.5 |\n| Analysis time | 25.34 | 34.00 |\n| Expected study duration under H1 | | 27.32 |\n| Overall exit probability (under H0) | 0.7736 | |\n| Overall exit probability (under H1) | 0.7707 | |\n| Exit probability for efficacy (under H0) | 0.0039 | |\n| Exit probability for efficacy (under H1) | 0.7666 | |\n| Exit probability for futility (under H0) | 0.7697 | |\n| Exit probability for futility (under H1) | 0.0040 | |\n\nLegend:\n\n* *(t)*: treatment effect scale\n\n:::\n:::\n\n\n- rpact vs EAST comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

0.0000

0.0000

N=398

p (1-sided)

0.0000

0.0000

Events: 176

HR at bound

0.0000

0.0000

Month: 25

P(Cross) if HR=1

0.0000

0.0000

P(Cross) if HR=0.60

0.0000

0.0000

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

0.0000

P(Cross) if HR=0.60

0.0000

\n```\n\n:::\n:::\n\n\n### SAS code\n\n- SAS code to reproduce the above rpact results appears below.\n\n```sas\nPROC SEQDESIGN BOUNDARYSCALE=MLE ERRSPEND;\n DESIGN NSTAGES=2 \n INFO=CUM(0.748936170212766 1.0) \n ALT=UPPER \n ALPHA=0.0125 \n BETA=0.05\n METHOD(ALPHA)=ERRFUNCOBF \n METHOD(BETA)=ERRFUNCGAMMA(GAMMA=-10) \n STOP=BOTH(BETABOUNDARY=NONBINDING);\n SAMPLESIZE MODEL=TWOSAMPLESURVIVAL(\n NULLMEDSURVTIME=9.4\n HAZARDRATIO=0.6\n ACCTIME=24 \n FOLTIME=10\n LOSS=EXP(HAZARD=0.018595295942851)\n WEIGHT=1);\n ODS OUTPUT Boundary=BMLE SampleSize=SS SampleSizeSummary=SSS;\nRUN;\n```\n\nThe following shows the events (D) and required sample sizes (N) for IA and FA.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/gsd-tte/result-pfs-sas-comp.png){fig-align='center' width=100%}\n:::\n:::\n\n\nPlease note that the `BOUNDARYSCALE=MLE | SCORE | STDZ | PVALUE` options display the boundary values in the MLE, standardize Z, score, and p-value scales, respectively. SAS will provide a boundary information table based on the specified `BOUNDARYSCALE`. In the information table, Alpha indicates the efficacy boundaries, and Beta indicates futility boundaries.\n\nSAS doesn't provide a boundary information with HR, so the HR boundaries is obtained from the MLE boundaries (as MLE $=\\hat{\\theta}=-log(\\text{HR})$, see [SAS User's Guide: Test for Two Survival Distributions with a Log-Rank Test](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/statug/statug_seqdesign_details52.htm#statug.seqdesign.cseqdlogrank)) via the following code.\n\n```sas\nDATA BHR;\n SET BMLE;\n Bound_UA_HR=exp(-Bound_UA);\n Bound_UB_HR=exp(-Bound_UB);\n LABEL BOUND_UA_HR=\"Upper Alpha (HR)\" BOUND_UA_HR=\"Upper Beta (HR)\";\nPROC PRINT LABEL;\n VAR _Stage_ _InfoProp_ Bound_UA Bound_UB Bound_UA_HR Bound_UB_HR;\nRUN;\n```\n\nThe HR boundaries are shown below.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/gsd-tte/result-pfs-sas-bhr.png){fig-align='center' width=100%}\n:::\n:::\n\n\nThe results calculated by SAS are presneted in the table below. Please note that SAS doesn't report the probablities $P(Cross | HR=1)$ and $P(Cross | HR=0.6)$, resulting in empty cells for these results in the table.\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

2.6606

0.7379

N=398

p (1-sided)

0.0039

0.2303

Events: 176

HR at bound

0.6696

0.8947

Month: 25

P(Cross) if HR=1

P(Cross) if HR=0.6

FA

Z

2.2798

N=398

p (1-sided)

0.0113

Events: 235

HR at bound

0.7427

Month: 34

P(Cross) if HR=1

P(Cross) if HR=0.6

\n```\n\n:::\n:::\n\n\n- SAS vs rapct comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

Efficacy

Futility

IA1: 75%

Z

0.0000

0.0000

N=398

p (1-sided)

0.0000

0.0000

Events: 176

HR at bound

0.0000

0.0000

Month: 25

P(Cross) if HR=1

P(Cross) if HR=0.6

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

P(Cross) if HR=0.6

\n```\n\n:::\n:::\n\n\n- SAS vs EAST comparison using absolute differences:\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n```{=html}\n

Analysis

Value

diff_eff_sas

diff_fut_sas

IA1: 75%

Z

0.0000

0.0000

N=398

p (1-sided)

0.0000

0.0000

Events: 176

HR at bound

0.0000

0.0000

Month: 25

P(Cross) if HR=1

P(Cross) if HR=0.60

FA

Z

0.0000

N=398

p (1-sided)

0.0000

Events: 235

HR at bound

0.0000

Month: 34

P(Cross) if HR=1

P(Cross) if HR=0.60

\n```\n\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsessionInfo()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nR version 4.5.2 (2025-10-31)\nPlatform: x86_64-pc-linux-gnu\nRunning under: Ubuntu 24.04.3 LTS\n\nMatrix products: default\nBLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 \nLAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0\n\nlocale:\n [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C \n [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 \n [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 \n [7] LC_PAPER=en_US.UTF-8 LC_NAME=C \n [9] LC_ADDRESS=C LC_TELEPHONE=C \n[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C \n\ntime zone: Europe/London\ntzcode source: system (glibc)\n\nattached base packages:\n[1] stats graphics grDevices datasets utils methods base \n\nother attached packages:\n[1] rpact_4.3.0 tibble_3.3.1 gsDesign2_1.1.8 gsDesign_3.9.0 \n[5] flextable_0.9.11\n\nloaded via a namespace (and not attached):\n [1] gt_1.3.0 rappdirs_0.3.4 sass_0.4.10 \n [4] generics_0.1.4 tidyr_1.3.2 fontLiberation_0.1.0 \n [7] renv_1.0.10 xml2_1.5.2 r2rtf_1.3.0 \n[10] digest_0.6.39 magrittr_2.0.4 evaluate_1.0.5 \n[13] grid_4.5.2 RColorBrewer_1.1-3 fastmap_1.2.0 \n[16] jsonlite_2.0.0 zip_2.3.3 purrr_1.2.1 \n[19] scales_1.4.0 fontBitstreamVera_0.1.1 textshaping_1.0.4 \n[22] cli_3.6.5 rlang_1.1.7 fontquiver_0.2.1 \n[25] withr_3.0.2 yaml_2.3.12 otel_0.2.0 \n[28] gdtools_0.5.0 tools_4.5.2 officer_0.7.3 \n[31] uuid_1.2-2 dplyr_1.2.0 ggplot2_4.0.2 \n[34] vctrs_0.7.1 R6_2.6.1 lifecycle_1.0.5 \n[37] fs_1.6.6 htmlwidgets_1.6.4 ragg_1.5.0 \n[40] pkgconfig_2.0.3 pillar_1.11.1 gtable_0.3.6 \n[43] data.table_1.18.2.1 glue_1.8.0 Rcpp_1.1.1 \n[46] systemfonts_1.3.1 xfun_0.56 tidyselect_1.2.1 \n[49] knitr_1.51 xtable_1.8-4 farver_2.1.2 \n[52] htmltools_0.5.9 patchwork_1.3.2 rmarkdown_2.30 \n[55] compiler_4.5.2 S7_0.2.1 askpass_1.2.1 \n[58] openssl_2.3.4 \n```\n\n\n:::\n:::\n\n\n## References\n\n- Lachin JM and Foulkes M. Evaluation of sample size and power for analyses of survival with allowance for nonuniform patient entry, losses to follow-up, non-compliance, and stratification. Biometrics 1986;42:507-19.\n- Kim K and Tsiatis AA. Study duration for clinical trials with survival response and early stopping rule. Biometrics 1990(46): 81-92.\n- Schemper M, Wakounig S and Heinze G. The estimation of average hazard ratios by weighted cox regression. Statistics in Medicine 2009; 28(19): 2473-2489.\n- Yung G and Liu Y. Sample size and power for the weighted log-rank test and Kaplan-Meier based tests with allowance for nonproportional hazards. Biometrics 2020;76:939-50.", "supporting": [ "r-east_gsd_tte_files" ], diff --git a/_freeze/Comp/r-sas-python_survey-stats-summary/execute-results/html.json b/_freeze/Comp/r-sas-python_survey-stats-summary/execute-results/html.json index 6268494ad..2ba655cbd 100644 --- a/_freeze/Comp/r-sas-python_survey-stats-summary/execute-results/html.json +++ b/_freeze/Comp/r-sas-python_survey-stats-summary/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "886e364bd8434d6a5b9e3197d09afd06", + "hash": "2fdbf7bb2e5745b861a647c4d512ac44", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS vs Python Survey Summary Statistics\"\nbibliography: survey-stats-summary.bib\nexecute: \n eval: false\n---\n\nThis document will compare the survey summary statistics functionality in SAS (available through SAS/STAT), R (available from the [`{survey}`](%5B%60%7Bsurvey%7D%60%5D(https://r-survey.r-forge.r-project.org/survey/html/api.html)) package), and Python (available from the [`samplics`](https://samplics-org.github.io/samplics/) package), highlighting differences in methods and results. Only the default Taylor series linearisation method for calculating variances is used in all languages. A more detailed comparison between R and SAS for specific methods and use-cases is available in [@2017_YRBS], [@so2020modelling], or [@adamico_2009]. For a general guide to survey statistics, which has companion guides for both R and SAS, see [@Lohr_2022].\n\n# Result Comparison\n\nThe following table shows different survey summary statistics, the capabilities of each language, and whether or not the results match. Each analysis also includes calculating the standard error and confidence intervals.\n\n| Analysis | Supported in R | Supported in SAS | Supported in Python | Results Match\\* | Notes |\n|------------|------------|------------|------------|------------|------------|\n| Mean | [Yes](../R/survey-stats-summary.html#Mean) | [Yes](../SAS/survey-stats-summary.html#Mean) | [Yes](../Python/survey-stats-summary.html#Mean) | Yes | Must specify degrees of freedom in R for confidence limits |\n| Total | [Yes](../R/survey-stats-summary.html#Total) | [Yes](../SAS/survey-stats-summary.html#Total) | [Yes](../Python/survey-stats-summary.html#Total) | Yes | Must specify degrees of freedom in R for confidence limits |\n| Ratios | [Yes](../R/survey-stats-summary.html#Ratios) | [Yes](../SAS/survey-stats-summary.html#Ratios) | [Yes](../Python/survey-stats-summary.html#Ratios) | Yes | Must specify degrees of freedom in R for confidence limits |\n| Proportions | [Yes](../R/survey-stats-summary.html#Proportions) | [Yes](../SAS/survey-stats-summary.html##Proportions) | [Yes](../Python/survey-stats-summary.html#Proportions) | Yes\\*\\* | In Python, the confidence limits of proportions only match to 1 or 2 s.f. This is due to a different method being used, which is undocumented. |\n| Quantiles | [Yes](../R/survey-stats-summary.html#Quantiles) | [Yes](../SAS/survey-stats-summary.html#Quantiles) | No | [No](#Quantiles) | Different methods for calculating quantiles |\n| Domain Analysis | [Yes](../R/survey-stats-summary.html#Summary%20Statistics%20on%20Complex%20Survey%20Designs) | [Yes](../SAS/survey-stats-summary.html##Summary%20Statistics%20on%20Complex%20Survey%20Designs) | [Yes](../Python/survey-stats-summary.html#Domain%20Estimations) | Yes | |\n| Design Effect | [Yes](../R/survey-stats-summary.html#Summary%20Statistics%20on%20Complex%20Survey%20Designs) | [Yes](../SAS/survey-stats-summary.html##Summary%20Statistics%20on%20Complex%20Survey%20Designs) | No | Yes | Set `deff=\"replace\"` in R to match SAS exactly |\n\n*\\*Results match where feature is available*\n\n*\\*\\*For confidence limits of proportions near 0 and 1, `survey::svyciprop` can be more accurate than `confint` in R, but does not match other software.*\n\nFor the full R, SAS, and Python code and results used for this comparison, see below:\n\n::: {.callout-note collapse=\"true\" appearance=\"minimal\" title=\"Show Code\"}\n## R\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(survey)\n\ndata(\"nhanes\")\n\nnhanes_design <- survey::svydesign(\n data = nhanes,\n id = ~SDMVPSU, # Specify the PSU/cluster column\n strata = ~SDMVSTRA, # The stratification column\n weights = ~WTMEC2YR, # The weighting column\n nest = TRUE # Allows for PSUs with the same name nested within different strata\n)\n\n# Mean of HI_CHOL\nhi_chol_mean <- survey::svymean(~HI_CHOL, nhanes_design, na.rm = TRUE)\n\n# Sum of HI_CHOL\nhi_chol_sum <- survey::svytotal(~HI_CHOL, nhanes_design, na.rm = TRUE)\n\n# Ratio of HI_CHOL / RIAGENDR\nhi_chol_ratio <- survey::svyratio(\n numerator = ~HI_CHOL,\n denominator = ~RIAGENDR,\n nhanes_design,\n na.rm = TRUE,\n ci = TRUE,\n se = TRUE,\n separate = FALSE\n)\n\n# Proportion of different AGECAT values\nagecat_props <- survey::svymean(~agecat, nhanes_design, na.rm = TRUE)\n\n# Quantiles of HI_CHOL\nhi_chol_quart <- survey::svyquantile(\n ~HI_CHOL,\n nhanes_design,\n quantiles = c(0.025, 0.5, 0.975),\n na.rm = TRUE,\n ci = TRUE\n)\n\n# Domain analysis of mean of HI_CHOL by race, with design effect\nhi_chol_mean_by_race <- survey::svyby(\n ~HI_CHOL,\n ~race,\n nhanes_design,\n svymean,\n na.rm = TRUE,\n deff = \"replace\"\n)\n\nprint(list(\n \"Mean of HI_CHOL\" = coef(hi_chol_mean),\n \"SE of Mean HI_CHOL\" = survey::SE(hi_chol_mean),\n \"CL of Mean HI_CHOL\" = confint(\n hi_chol_mean,\n df = survey::degf(nhanes_design)\n ),\n \"Sum of HI_CHOL\" = coef(hi_chol_sum),\n \"SE of Sum HI_CHOL\" = survey::SE(hi_chol_sum),\n \"CL of Sum HI_CHOL\" = confint(hi_chol_sum, df = survey::degf(nhanes_design)),\n \"Ratio of HI_CHOL / RIAGENDR\" = coef(hi_chol_ratio),\n \"SE of Ratio HI_CHOL / RIAGENDR\" = survey::SE(hi_chol_ratio),\n \"CL of Ratio HI_CHOL / RIAGENDR\" = confint(\n hi_chol_ratio,\n df = survey::degf(nhanes_design)\n ),\n \"Proportion of AGECAT\" = coef(agecat_props),\n \"SE of Proportion AGECAT\" = survey::SE(agecat_props),\n \"CL of Proportion AGECAT\" = confint(\n agecat_props,\n df = survey::degf(nhanes_design)\n ),\n \"Quantiles of HI_CHOL\" = coef(hi_chol_quart),\n \"SE of Quantiles HI_CHOL\" = survey::SE(hi_chol_quart),\n \"CL of Quantiles HI_CHOL\" = confint(\n hi_chol_quart,\n df = survey::degf(nhanes_design)\n ),\n \"Mean of HI_CHOL by race\" = coef(hi_chol_mean_by_race),\n \"SE of HI_CHOL by race\" = survey::SE(hi_chol_mean_by_race),\n \"CL of HI_CHOL by race\" = confint(\n hi_chol_mean_by_race,\n df = survey::degf(nhanes_design)\n ),\n \"Design Effect of HI_CHOL by race\" = hi_chol_mean_by_race$DEff.HI_CHOL\n))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n$`Mean of HI_CHOL`\n HI_CHOL \n0.112143 \n\n$`SE of Mean HI_CHOL`\n HI_CHOL\nHI_CHOL 0.00544584\n\n$`CL of Mean HI_CHOL`\n 2.5 % 97.5 %\nHI_CHOL 0.1005983 0.1236876\n\n$`Sum of HI_CHOL`\n HI_CHOL \n28635245 \n\n$`SE of Sum HI_CHOL`\n HI_CHOL\nHI_CHOL 2020711\n\n$`CL of Sum HI_CHOL`\n 2.5 % 97.5 %\nHI_CHOL 24351530 32918961\n\n$`Ratio of HI_CHOL / RIAGENDR`\nHI_CHOL/RIAGENDR \n 0.07422209 \n\n$`SE of Ratio HI_CHOL / RIAGENDR`\nHI_CHOL/RIAGENDR \n 0.003714728 \n\n$`CL of Ratio HI_CHOL / RIAGENDR`\n 2.5 % 97.5 %\nHI_CHOL/RIAGENDR 0.06634722 0.08209696\n\n$`Proportion of AGECAT`\n agecat(0,19] agecat(19,39] agecat(39,59] agecat(59,Inf] \n 0.2077495 0.2934079 0.3032896 0.1955530 \n\n$`SE of Proportion AGECAT`\n agecat(0,19] agecat(19,39] agecat(39,59] agecat(59,Inf] \n 0.006129950 0.009560692 0.004519463 0.008092578 \n\n$`CL of Proportion AGECAT`\n 2.5 % 97.5 %\nagecat(0,19] 0.1947546 0.2207444\nagecat(19,39] 0.2731401 0.3136756\nagecat(39,59] 0.2937088 0.3128704\nagecat(59,Inf] 0.1783975 0.2127085\n\n$`Quantiles of HI_CHOL`\nHI_CHOL.0.025 HI_CHOL.0.5 HI_CHOL.0.975 \n 0 0 1 \n\n$`SE of Quantiles HI_CHOL`\nHI_CHOL.0.025 HI_CHOL.0.5 HI_CHOL.0.975 \n 0.2358596 0.2358596 0.0000000 \n\n$`CL of Quantiles HI_CHOL`\n l u\nHI_CHOL.0.025 0 1\nHI_CHOL.0.5 0 1\nHI_CHOL.0.975 1 1\n\n$`Mean of HI_CHOL by race`\n 1 2 3 4 \n0.10149167 0.12164921 0.07864006 0.09967861 \n\n$`SE of HI_CHOL by race`\n[1] 0.006245843 0.006604134 0.010384645 0.024666227\n\n$`CL of HI_CHOL by race`\n 2.5 % 97.5 %\n1 0.08825107 0.1147323\n2 0.10764907 0.1356493\n3 0.05662560 0.1006545\n4 0.04738854 0.1519687\n\n$`Design Effect of HI_CHOL by race`\n[1] 1.082734 1.407822 2.091156 3.098290\n```\n\n\n:::\n:::\n\n\n## SAS\n\n\n::: {.cell}\n\n```{.sas .cell-code}\n* Mean, sum quantile of HI_CHOL;\nproc surveymeans data=nhanes mean sum clm quantile=(0.025 0.5 0.975);\n cluster SDMVPSU;\n strata SDMVSTRA;\n weight WTMEC2YR;\n var HI_CHOL;\nrun;\n\n* Ratio of HI_CHOL / RIAGENDR;\nproc surveymeans data=nhanes;\n cluster SDMVPSU;\n strata SDMVSTRA;\n weight WTMEC2YR;\n ratio HI_CHOL / RIAGENDR;\nrun;\n\n* Proportions of agecat;\nproc surveyfreq data=nhanes;\n cluster SDMVPSU;\n strata SDMVSTRA;\n weight WTMEC2YR;\n table agecat / cl;\nrun;\n\n* Mean and DEFF of HI_CHOL by race;\nproc surveymeans data=nhanes mean deff;\n cluster SDMVPSU;\n strata SDMVSTRA;\n weight WTMEC2YR;\n domain race;\n var HI_CHOL;\nrun;\n```\n:::\n\n\n``` default\n\n The SURVEYMEANS Procedure\n\n Data Summary\n\n Number of Strata 15\n Number of Clusters 31\n Number of Observations 8591\n Sum of Weights 276536446\n\n\n Statistics\n\n Std Error Std Error\n Variable Mean of Mean 95% CL for Mean Sum of Sum 95% CL for Sum\n --------------------------------------------------------------------------------------------------------------------------\n HI_CHOL 0.112143 0.005446 0.10059829 0.12368762 28635245 2020711 24351529.8 32918960.7\n --------------------------------------------------------------------------------------------------------------------------\n\n\n Quantiles\n\n Std\n Variable Percentile Estimate Error 95% Confidence Limits\n ---------------------------------------------------------------------------------\n HI_CHOL 2.5 0 0.024281 -0.0514730 0.05147298\n 50 Median 0 0.024281 -0.0514730 0.05147298\n 97.5 0.777070 0.024281 0.7255973 0.82854324\n ---------------------------------------------------------------------------------\n\n The SURVEYMEANS Procedure\n\n Data Summary\n\n Number of Strata 15\n Number of Clusters 31\n Number of Observations 8591\n Sum of Weights 276536446\n\n\n Statistics\n\n Std Error\n Variable N Mean of Mean 95% CL for Mean\n ---------------------------------------------------------------------------------\n HI_CHOL 7846 0.112143 0.005446 0.10059829 0.12368762\n RIAGENDR 8591 1.512019 0.005302 1.50077977 1.52325807\n ---------------------------------------------------------------------------------\n\n\n Ratio Analysis\n\n Std\n Numerator Denominator N Ratio Error 95% CL for Ratio\n ----------------------------------------------------------------------------------------------\n HI_CHOL RIAGENDR 7846 0.074222 0.003715 0.06634722 0.08209696\n ----------------------------------------------------------------------------------------------\n\n The SURVEYFREQ Procedure\n\n Data Summary\n\n Number of Strata 15\n Number of Clusters 31\n Number of Observations 8591\n Sum of Weights 276536446\n\n\n Table of agecat\n\n Weighted Std Err of Std Err of 95% Confidence Limits\n agecat Frequency Frequency Wgt Freq Percent Percent for Percent\n -------------------------------------------------------------------------------------------------------\n (0,19] 2532 57450307 3043819 20.7749 0.6130 19.4755 22.0744\n (19,39] 2033 81137975 3692818 29.3408 0.9561 27.3140 31.3676\n (39,59] 2021 83870623 4853936 30.3290 0.4519 29.3709 31.2870\n (59,Inf] 2005 54077541 4284296 19.5553 0.8093 17.8398 21.2709\n\n Total 8591 276536446 13935730 100.0000 \n -------------------------------------------------------------------------------------------------------\n\n The SURVEYMEANS Procedure\n\n Data Summary\n\n Number of Strata 15\n Number of Clusters 31\n Number of Observations 8591\n Sum of Weights 276536446\n\n\n Statistics\n\n Std Error Design\n Variable Mean of Mean Effect\n --------------------------------------------------------\n HI_CHOL 0.112143 0.005446 2.336725\n --------------------------------------------------------\n\n The SURVEYMEANS Procedure\n\n Statistics for race Domains\n\n Std Error Design\n race Variable Mean of Mean Effect\n ------------------------------------------------------------------------\n 1 HI_CHOL 0.101492 0.006246 1.082734\n 2 HI_CHOL 0.121649 0.006604 1.407822\n 3 HI_CHOL 0.078640 0.010385 2.091156\n 4 HI_CHOL 0.099679 0.024666 3.098290\n ------------------------------------------------------------------------\n```\n\n## Python\n\n\n::: {.cell}\n\n```{.python .cell-code}\nimport pandas as pd\nfrom samplics import TaylorEstimator\nfrom samplics.utils.types import PopParam\n\nnhanes = pd.read_csv(\"../data/nhanes.csv\")\n\nnhanes_design_kwargs = dict(\n psu=nhanes[\"SDMVPSU\"],\n stratum=nhanes[\"SDMVSTRA\"],\n samp_weight=nhanes[\"WTMEC2YR\"],\n remove_nan=True,\n)\n\n# Mean of HI_CHOL\nmean_estimator = TaylorEstimator(PopParam.mean)\nmean_estimator.estimate(nhanes[\"HI_CHOL\"], **nhanes_design_kwargs)\nhi_chol_means = mean_estimator.to_dataframe()\n\n# Sum of HI_CHOL\ntotal_estimator = TaylorEstimator(PopParam.total)\ntotal_estimator.estimate(nhanes[\"HI_CHOL\"], **nhanes_design_kwargs)\nhi_chol_totals = total_estimator.to_dataframe()\n\n# Ratio of HI_CHOL / RIAGENDR\nratio_estimator = TaylorEstimator(PopParam.ratio)\nratio_estimator.estimate(\n y=nhanes[\"HI_CHOL\"], x=nhanes[\"RIAGENDR\"], **nhanes_design_kwargs\n)\nhi_chol_ratio = ratio_estimator.to_dataframe()\n\n# Proportion of different AGECAT values\nprop_estimator = TaylorEstimator(PopParam.prop)\nprop_estimator.estimate(nhanes[\"agecat\"], **nhanes_design_kwargs)\nagecat_prop = prop_estimator.to_dataframe()\n\n# Quantiles of HI_CHOL\n# NA\n\n# Domain analysis of mean of HI_CHOL by race, with design effect\nmean_estimator = TaylorEstimator(PopParam.mean)\nmean_estimator.estimate(\n nhanes[\"HI_CHOL\"],\n **nhanes_design_kwargs,\n domain=nhanes[\"race\"],\n deff=True, # Design effect param currently has no effect\n)\nhi_chol_domain_means = mean_estimator.to_dataframe()\n\n\nag_dict = agecat_prop.set_index(\"_level\").to_dict()\nhc_dict = hi_chol_domain_means.set_index(\"_domain\").to_dict()\n\nprint(\n f\"\"\"\n Mean of HI_CHOL: {hi_chol_means[\"_estimate\"][0]}\n SE of Mean HI_CHOL: {hi_chol_means[\"_stderror\"][0]}\n CL of Mean HI_CHOL: {(hi_chol_means[\"_lci\"][0], hi_chol_means[\"_uci\"][0])}\n Sum of HI_CHOL: {hi_chol_totals[\"_estimate\"][0]}\n SE of Sum HI_CHOL: {hi_chol_totals[\"_stderror\"][0]}\n CL of Sum HI_CHOL: {(hi_chol_totals[\"_lci\"][0], hi_chol_totals[\"_uci\"][0])}\n Ratio of HI_CHOL / RIAGENDR: {hi_chol_ratio[\"_estimate\"][0]}\n SE of Ratio HI_CHOL / RIAGENDR: {hi_chol_ratio[\"_stderror\"][0]}\n CL of Ratio HI_CHOL / RIAGENDR: {(hi_chol_ratio[\"_lci\"][0], hi_chol_ratio[\"_uci\"][0])}\n Proportion of AGECAT: {ag_dict[\"_estimate\"]}\n SE of Proportion AGECAT: {ag_dict[\"_stderror\"]}\n LCL of Proportion AGECAT: {ag_dict[\"_lci\"]}\n UCL of Proportion AGECAT: {ag_dict[\"_uci\"]}\n Quantiles of HI_CHOL: Not available\n Mean of HI_CHOL by race: {hc_dict[\"_estimate\"]}\n SE of HI_CHL by race: {hc_dict[\"_stderror\"]}\n LCL of HI_CHOL by race: {hc_dict[\"_lci\"]}\n UCL of HI_CHOL by race: {hc_dict[\"_uci\"]}\n Design Effect of HI_CHOL by race: Not available\n \"\"\"\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n Mean of HI_CHOL: 0.11214295634969222\n SE of Mean HI_CHOL: 0.005445839698954557\n CL of Mean HI_CHOL: (np.float64(0.1005982919131703), np.float64(0.12368762078621415))\n Sum of HI_CHOL: 28635245.254672\n SE of Sum HI_CHOL: 2020710.7436996205\n CL of Sum HI_CHOL: (np.float64(24351529.84091034), np.float64(32918960.668433655))\n Ratio of HI_CHOL / RIAGENDR: 0.07422209323594066\n SE of Ratio HI_CHOL / RIAGENDR: 0.0037147278931070065\n CL of Ratio HI_CHOL / RIAGENDR: (np.float64(0.06634722189017901), np.float64(0.0820969645817023))\n Proportion of AGECAT: {'(0,19]': 0.2077494937870972, '(19,39]': 0.29340788818591346, '(39,59]': 0.30328958320385285, '(59,Inf]': 0.19555303482313666}\n SE of Proportion AGECAT: {'(0,19]': 0.006129950336419631, '(19,39]': 0.009560691634608896, '(39,59]': 0.004519462827363183, '(59,Inf]': 0.008092578243976422}\n LCL of Proportion AGECAT: {'(0,19]': 0.19505410930097866, '(19,39]': 0.27355685874096586, '(39,59]': 0.2937950591158628, '(59,Inf]': 0.1789647230500222}\n UCL of Proportion AGECAT: {'(0,19]': 0.2210442684297426, '(19,39]': 0.3140766293472951, '(39,59]': 0.31295496708023285, '(59,Inf]': 0.21327950895208636}\n Quantiles of HI_CHOL: Not available\n Mean of HI_CHOL by race: {1: 0.10149166545397208, 2: 0.12164920535593333, 3: 0.07864006039908408, 4: 0.09967860947712034}\n SE of HI_CHL by race: {1: 0.006245843308749599, 2: 0.006604133623532979, 3: 0.010384645000548863, 4: 0.024666226871851268}\n LCL of HI_CHOL by race: {1: 0.0882510691256497, 2: 0.10764906749064211, 3: 0.056625596431891564, 4: 0.04738854441969514}\n UCL of HI_CHOL by race: {1: 0.11473226178229445, 2: 0.13564934322122454, 3: 0.1006545243662766, 4: 0.15196867453454554}\n Design Effect of HI_CHOL by race: Not available\n \n```\n\n\n:::\n:::\n\n:::\n\n# Differences\n\n## Quantiles {#quantiles}\n\n`samplics` in Python does not have a method for calculating quantiles, and in R and SAS the available methods lead to different results. To demonstrate the differences in calculating quantiles, we will use the `apisrs` dataset from the `survey` package in R [@API_2000].\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(survey)\n\ndata(\"api\")\n\nhead(apisrs) |>\n gt::gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n \n
cdsstypenamesnamesnumdnamednumcnamecnumflagpcttestapi00api99targetgrowthsch.widecomp.impbothawardsmealsellyr.rndmobilityacs.k3acs.46acs.corepct.respnot.hsghsgsome.colcol.gradgrad.schavg.edfullemerenrollapi.stupwfpc
15739081534155HMcFarland HighMcFarland High1039McFarland Unified432Kern14NA984624481814NoYesNoNo4431NA6NANA2482443412731.91713547742930.976194
19642126066716EStowers (Cecil Stowers (Cecil B.) Elementary1124ABC Unified1Los Angeles18NA100878831NA47YesYesYesYes825NA151930NA974102343213.66901047842030.976194
30664493030640HBrea-Olinda HigBrea-Olinda High2868Brea-Olinda Unified79Orange29NA987347423-8NoNoNoNo1010NA7NANA2895592141243.7183181410128730.976194
19644516012744EAlameda ElementAlameda Elementary1273Downey Unified187Los Angeles18NA997726577115YesYesYesYes7025NA2323NANA100374014811.96851834229130.976194
40688096043293ESunnyside ElemeSunnyside Elementary4926San Luis Coastal Unified640San Luis Obispo39NA99739719420YesYesYesYes4312NA122029NA918212734103.17100021718930.976194
19734456014278ELos Molinos EleLos Molinos Elementary2463Hacienda la Puente Unif284Los Angeles18NA93835822NA13YesYesYesNo1619NA131929NA71182038343.96752025821130.976194
\n
\n```\n\n:::\n:::\n\n\nIn SAS, PROC SURVEYMEANS will calculate quantiles of specific probabilities as you request them, using Woodruff's method for intervals and a custom quantile method [@SAS_2018, pp. 9834]. The quantile method does not match any of the available `qrules` in R, and although the default `interval.types` in the R `survey::svyquantile` function also uses Woodruff's method, it is a different implementation.\n\nThe method and results from SAS are as follows:\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc surveymeans data=apisrs total=6194 quantile=(0.025 0.5 0.975);\n var growth;\nrun;\n```\n:::\n\n\n``` default\n The SURVEYMEANS Procedure\n\n Data Summary\n\n Number of Observations 200\n\n\n\n\n Quantiles\n\n Std\n Variable Percentile Estimate Error 95% Confidence Limits\n ---------------------------------------------------------------------------------\n growth 2.5 -16.500000 1.755916 -19.962591 -13.037409\n 50 Median 26.500000 1.924351 22.705263 30.294737\n 97.5 99.000000 16.133827 67.184794 130.815206\n ---------------------------------------------------------------------------------\n```\n\nIf in R we use the default `qrule=\"math\"` (equivalent to `qrule=\"hf1\"` and matches `type=1` in the `quantile` function for unweighted data) along with the default `interval.type=\"mean\"`, we get the following results:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsrs_design <- survey::svydesign(data = apisrs, id = ~1, fpc = ~fpc, )\n\nsurvey::svyquantile(\n ~growth,\n srs_design,\n quantiles = c(0.025, 0.5, 0.975),\n ci = TRUE,\n se = TRUE\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n$growth\n quantile ci.2.5 ci.97.5 se\n0.025 -16 -21 -12 2.281998\n0.5 27 24 31 1.774887\n0.975 103 93 189 24.341307\n\nattr(,\"hasci\")\n[1] TRUE\nattr(,\"class\")\n[1] \"newsvyquantile\"\n```\n\n\n:::\n:::\n\n\nHere we can see that the quantiles, confidence intervals, and standard errors do not match SAS. From testing, none of the available `qrule` methods match SAS for the quantile values, so it is recommended to use the default values unless you have need of some of the other properties of different quantile definitions - see [`vignette(\"qrule\", package=\"survey\")`](https://cran.r-project.org/web/packages/survey/vignettes/qrule.pdf) for more detail. If an exact match to SAS is required, then the `svyquantile` function allows for passing a custom function to the `qrule` argument to define your own method for calculating quantiles. Below is an example that will match SAS:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsas_qrule <- function(x, w, p) {\n # Custom qrule to match SAS, based on survey::oldsvyquantile's internal method\n if (any(is.na(x))) {\n return(NA * p)\n }\n w <- rowsum(w, x, reorder = TRUE)\n x <- sort(unique(x))\n cum.w <- cumsum(w) / sum(w)\n cdf <- approxfun(\n cum.w,\n x,\n method = \"linear\",\n f = 1,\n yleft = min(x),\n yright = max(x),\n ties = min\n )\n cdf(p)\n}\n\n\nsas_quants <- survey::svyquantile(\n ~growth,\n srs_design,\n quantiles = c(0.025, 0.5, 0.975),\n qrule = sas_qrule,\n ci = TRUE,\n se = TRUE\n)\n\nsas_quants\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n$growth\n quantile ci.2.5 ci.97.5 se\n0.025 -16.5 -22.00000 -15.07482 1.755916\n0.5 26.5 23.03563 30.62510 1.924351\n0.975 99.0 83.70616 147.33657 16.133827\n\nattr(,\"hasci\")\n[1] TRUE\nattr(,\"class\")\n[1] \"newsvyquantile\"\n```\n\n\n:::\n:::\n\n\nNote that although the quantiles and standard errors match, the confidence intervals still do not match SAS. For this another custom calculation is required, based on the formula used in SAS:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsas_quantile_confint <- function(newsvyquantile, level = 0.05, df = Inf) {\n q <- coef(newsvyquantile)\n se <- survey::SE(newsvyquantile)\n ci <- cbind(\n q,\n q + se * qt(level / 2, df),\n q - se * qt(1 - level / 2, df),\n se\n )\n colnames(ci) <- c(\n \"quantile\",\n paste0(\"ci.\", c(100 * level / 2, 100 * (1 - level / 2))),\n \"se\"\n )\n\n ci\n}\n\nsas_quantile_confint(sas_quants, df = survey::degf(srs_design))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n quantile ci.2.5 ci.97.5 se\ngrowth.0.025 -16.5 -19.96259 -19.96259 1.755916\ngrowth.0.5 26.5 22.70526 22.70526 1.924351\ngrowth.0.975 99.0 67.18479 67.18479 16.133827\n```\n\n\n:::\n:::\n\n\n## Other considerations\n\n### Degrees of Freedom\n\nSome of the functions in R require the degrees of freedom to be specified when calculating confidence intervals, otherwise it assumes a normal distribution. This can be done easily by using the `survey::degf` function, which calculates the degrees of freedom for a survey design object.\n\n### Single PSU Strata\n\nAlthough it was not apparent with the examples used here, if there is only one PSU from a stratum then R will by default error, whereas SAS will remove that stratum from the variance calculation. This can be changed in R by setting the `options(survey.lonely.psu=\"certainty\")` to match SAS and have it make no contribution to the variance. In `samplics`, this behaviour can be configured using the `single_psu` argument to the `estimate` method, and can be set to to match SAS using `SinglePSUEst.certainty`. This should be considered carefully however, in R and Python there are additional methods of handling single PSUs that may be more appropriate for your use-case.\n\n### Documentation Differences\n\nOne key consideration when choosing a statistical package is the documentation available. In this case, both the `survey` package in R and the survey procedures in SAS have a much more comprehensive set of documentation and examples than `samplics` in Python. This includes both detailed examples, as well as the underlying theory and methods used in the calculations including references to the literature.\n\n# Summary and Recommendations\n\nThe `{survey}` package in R and the survey procedures in SAS/STAT both provide similar functionality for calculating survey summary statistics. In most cases in both our tests and others, the results are identical ([@2017_YRBS], [@so2020modelling], [@adamico_2009]). Where differences do occur, primarily in calculating quantiles, the methods in R are more varied and well-documented.\n\nIn contrast, the `samplics` package in Python is still early in development, and although it does provide some functionality there are still major limitations in both basic statistics (i.e. quantiles) and in more complex methods that were beyond the scope of this document, and the methods are much less well-documented.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-24\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P survey * 4.4-8 2025-08-28 [?] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n\n::: {.cell}\n\n:::\n\n:::", + "markdown": "---\ntitle: \"R vs SAS vs Python Survey Summary Statistics\"\nbibliography: survey-stats-summary.bib\n---\n\nThis document will compare the survey summary statistics functionality in SAS (available through SAS/STAT), R (available from the [`{survey}`](%5B%60%7Bsurvey%7D%60%5D(https://r-survey.r-forge.r-project.org/survey/html/api.html)) package), and Python (available from the [`samplics`](https://samplics-org.github.io/samplics/) package), highlighting differences in methods and results. Only the default Taylor series linearisation method for calculating variances is used in all languages. A more detailed comparison between R and SAS for specific methods and use-cases is available in [@2017_YRBS], [@so2020modelling], or [@adamico_2009]. For a general guide to survey statistics, which has companion guides for both R and SAS, see [@Lohr_2022].\n\n# Result Comparison\n\nThe following table shows different survey summary statistics, the capabilities of each language, and whether or not the results match. Each analysis also includes calculating the standard error and confidence intervals.\n\n| Analysis | Supported in R | Supported in SAS | Supported in Python | Results Match\\* | Notes |\n|------------|------------|------------|------------|------------|------------|\n| Mean | [Yes](../R/survey-stats-summary.html#Mean) | [Yes](../SAS/survey-stats-summary.html#Mean) | [Yes](../Python/survey-stats-summary.html#Mean) | Yes | Must specify degrees of freedom in R for confidence limits |\n| Total | [Yes](../R/survey-stats-summary.html#Total) | [Yes](../SAS/survey-stats-summary.html#Total) | [Yes](../Python/survey-stats-summary.html#Total) | Yes | Must specify degrees of freedom in R for confidence limits |\n| Ratios | [Yes](../R/survey-stats-summary.html#Ratios) | [Yes](../SAS/survey-stats-summary.html#Ratios) | [Yes](../Python/survey-stats-summary.html#Ratios) | Yes | Must specify degrees of freedom in R for confidence limits |\n| Proportions | [Yes](../R/survey-stats-summary.html#Proportions) | [Yes](../SAS/survey-stats-summary.html##Proportions) | [Yes](../Python/survey-stats-summary.html#Proportions) | Yes\\*\\* | In Python, the confidence limits of proportions only match to 1 or 2 s.f. This is due to a different method being used, which is undocumented. |\n| Quantiles | [Yes](../R/survey-stats-summary.html#Quantiles) | [Yes](../SAS/survey-stats-summary.html#Quantiles) | No | [No](#Quantiles) | Different methods for calculating quantiles |\n| Domain Analysis | [Yes](../R/survey-stats-summary.html#Summary%20Statistics%20on%20Complex%20Survey%20Designs) | [Yes](../SAS/survey-stats-summary.html##Summary%20Statistics%20on%20Complex%20Survey%20Designs) | [Yes](../Python/survey-stats-summary.html#Domain%20Estimations) | Yes | |\n| Design Effect | [Yes](../R/survey-stats-summary.html#Summary%20Statistics%20on%20Complex%20Survey%20Designs) | [Yes](../SAS/survey-stats-summary.html##Summary%20Statistics%20on%20Complex%20Survey%20Designs) | No | Yes | Set `deff=\"replace\"` in R to match SAS exactly |\n\n*\\*Results match where feature is available*\n\n*\\*\\*For confidence limits of proportions near 0 and 1, `survey::svyciprop` can be more accurate than `confint` in R, but does not match other software.*\n\nFor the full R, SAS, and Python code and results used for this comparison, see below:\n\n::: {.callout-note collapse=\"true\" appearance=\"minimal\" title=\"Show Code\"}\n## R\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(survey)\n\ndata(\"nhanes\")\n\nnhanes_design <- survey::svydesign(\n data = nhanes,\n id = ~SDMVPSU, # Specify the PSU/cluster column\n strata = ~SDMVSTRA, # The stratification column\n weights = ~WTMEC2YR, # The weighting column\n nest = TRUE # Allows for PSUs with the same name nested within different strata\n)\n\n# Mean of HI_CHOL\nhi_chol_mean <- survey::svymean(~HI_CHOL, nhanes_design, na.rm = TRUE)\n\n# Sum of HI_CHOL\nhi_chol_sum <- survey::svytotal(~HI_CHOL, nhanes_design, na.rm = TRUE)\n\n# Ratio of HI_CHOL / RIAGENDR\nhi_chol_ratio <- survey::svyratio(\n numerator = ~HI_CHOL,\n denominator = ~RIAGENDR,\n nhanes_design,\n na.rm = TRUE,\n ci = TRUE,\n se = TRUE,\n separate = FALSE\n)\n\n# Proportion of different AGECAT values\nagecat_props <- survey::svymean(~agecat, nhanes_design, na.rm = TRUE)\n\n# Quantiles of HI_CHOL\nhi_chol_quart <- survey::svyquantile(\n ~HI_CHOL,\n nhanes_design,\n quantiles = c(0.025, 0.5, 0.975),\n na.rm = TRUE,\n ci = TRUE\n)\n\n# Domain analysis of mean of HI_CHOL by race, with design effect\nhi_chol_mean_by_race <- survey::svyby(\n ~HI_CHOL,\n ~race,\n nhanes_design,\n svymean,\n na.rm = TRUE,\n deff = \"replace\"\n)\n\nprint(list(\n \"Mean of HI_CHOL\" = coef(hi_chol_mean),\n \"SE of Mean HI_CHOL\" = survey::SE(hi_chol_mean),\n \"CL of Mean HI_CHOL\" = confint(\n hi_chol_mean,\n df = survey::degf(nhanes_design)\n ),\n \"Sum of HI_CHOL\" = coef(hi_chol_sum),\n \"SE of Sum HI_CHOL\" = survey::SE(hi_chol_sum),\n \"CL of Sum HI_CHOL\" = confint(hi_chol_sum, df = survey::degf(nhanes_design)),\n \"Ratio of HI_CHOL / RIAGENDR\" = coef(hi_chol_ratio),\n \"SE of Ratio HI_CHOL / RIAGENDR\" = survey::SE(hi_chol_ratio),\n \"CL of Ratio HI_CHOL / RIAGENDR\" = confint(\n hi_chol_ratio,\n df = survey::degf(nhanes_design)\n ),\n \"Proportion of AGECAT\" = coef(agecat_props),\n \"SE of Proportion AGECAT\" = survey::SE(agecat_props),\n \"CL of Proportion AGECAT\" = confint(\n agecat_props,\n df = survey::degf(nhanes_design)\n ),\n \"Quantiles of HI_CHOL\" = coef(hi_chol_quart),\n \"SE of Quantiles HI_CHOL\" = survey::SE(hi_chol_quart),\n \"CL of Quantiles HI_CHOL\" = confint(\n hi_chol_quart,\n df = survey::degf(nhanes_design)\n ),\n \"Mean of HI_CHOL by race\" = coef(hi_chol_mean_by_race),\n \"SE of HI_CHOL by race\" = survey::SE(hi_chol_mean_by_race),\n \"CL of HI_CHOL by race\" = confint(\n hi_chol_mean_by_race,\n df = survey::degf(nhanes_design)\n ),\n \"Design Effect of HI_CHOL by race\" = hi_chol_mean_by_race$DEff.HI_CHOL\n))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n$`Mean of HI_CHOL`\n HI_CHOL \n0.112143 \n\n$`SE of Mean HI_CHOL`\n HI_CHOL\nHI_CHOL 0.00544584\n\n$`CL of Mean HI_CHOL`\n 2.5 % 97.5 %\nHI_CHOL 0.1005983 0.1236876\n\n$`Sum of HI_CHOL`\n HI_CHOL \n28635245 \n\n$`SE of Sum HI_CHOL`\n HI_CHOL\nHI_CHOL 2020711\n\n$`CL of Sum HI_CHOL`\n 2.5 % 97.5 %\nHI_CHOL 24351530 32918961\n\n$`Ratio of HI_CHOL / RIAGENDR`\nHI_CHOL/RIAGENDR \n 0.07422209 \n\n$`SE of Ratio HI_CHOL / RIAGENDR`\nHI_CHOL/RIAGENDR \n 0.003714728 \n\n$`CL of Ratio HI_CHOL / RIAGENDR`\n 2.5 % 97.5 %\nHI_CHOL/RIAGENDR 0.06634722 0.08209696\n\n$`Proportion of AGECAT`\n agecat(0,19] agecat(19,39] agecat(39,59] agecat(59,Inf] \n 0.2077495 0.2934079 0.3032896 0.1955530 \n\n$`SE of Proportion AGECAT`\n agecat(0,19] agecat(19,39] agecat(39,59] agecat(59,Inf] \n 0.006129950 0.009560692 0.004519463 0.008092578 \n\n$`CL of Proportion AGECAT`\n 2.5 % 97.5 %\nagecat(0,19] 0.1947546 0.2207444\nagecat(19,39] 0.2731401 0.3136756\nagecat(39,59] 0.2937088 0.3128704\nagecat(59,Inf] 0.1783975 0.2127085\n\n$`Quantiles of HI_CHOL`\nHI_CHOL.0.025 HI_CHOL.0.5 HI_CHOL.0.975 \n 0 0 1 \n\n$`SE of Quantiles HI_CHOL`\nHI_CHOL.0.025 HI_CHOL.0.5 HI_CHOL.0.975 \n 0.2358596 0.2358596 0.0000000 \n\n$`CL of Quantiles HI_CHOL`\n l u\nHI_CHOL.0.025 0 1\nHI_CHOL.0.5 0 1\nHI_CHOL.0.975 1 1\n\n$`Mean of HI_CHOL by race`\n 1 2 3 4 \n0.10149167 0.12164921 0.07864006 0.09967861 \n\n$`SE of HI_CHOL by race`\n[1] 0.006245843 0.006604134 0.010384645 0.024666227\n\n$`CL of HI_CHOL by race`\n 2.5 % 97.5 %\n1 0.08825107 0.1147323\n2 0.10764907 0.1356493\n3 0.05662560 0.1006545\n4 0.04738854 0.1519687\n\n$`Design Effect of HI_CHOL by race`\n[1] 1.082734 1.407822 2.091156 3.098290\n```\n\n\n:::\n:::\n\n\n## SAS\n\n```sas\n* Mean, sum quantile of HI_CHOL;\nproc surveymeans data=nhanes mean sum clm quantile=(0.025 0.5 0.975);\n cluster SDMVPSU;\n strata SDMVSTRA;\n weight WTMEC2YR;\n var HI_CHOL;\nrun;\n\n* Ratio of HI_CHOL / RIAGENDR;\nproc surveymeans data=nhanes;\n cluster SDMVPSU;\n strata SDMVSTRA;\n weight WTMEC2YR;\n ratio HI_CHOL / RIAGENDR;\nrun;\n\n* Proportions of agecat;\nproc surveyfreq data=nhanes;\n cluster SDMVPSU;\n strata SDMVSTRA;\n weight WTMEC2YR;\n table agecat / cl;\nrun;\n\n* Mean and DEFF of HI_CHOL by race;\nproc surveymeans data=nhanes mean deff;\n cluster SDMVPSU;\n strata SDMVSTRA;\n weight WTMEC2YR;\n domain race;\n var HI_CHOL;\nrun;\n```\n\n``` default\n\n The SURVEYMEANS Procedure\n\n Data Summary\n\n Number of Strata 15\n Number of Clusters 31\n Number of Observations 8591\n Sum of Weights 276536446\n\n\n Statistics\n\n Std Error Std Error\n Variable Mean of Mean 95% CL for Mean Sum of Sum 95% CL for Sum\n --------------------------------------------------------------------------------------------------------------------------\n HI_CHOL 0.112143 0.005446 0.10059829 0.12368762 28635245 2020711 24351529.8 32918960.7\n --------------------------------------------------------------------------------------------------------------------------\n\n\n Quantiles\n\n Std\n Variable Percentile Estimate Error 95% Confidence Limits\n ---------------------------------------------------------------------------------\n HI_CHOL 2.5 0 0.024281 -0.0514730 0.05147298\n 50 Median 0 0.024281 -0.0514730 0.05147298\n 97.5 0.777070 0.024281 0.7255973 0.82854324\n ---------------------------------------------------------------------------------\n\n The SURVEYMEANS Procedure\n\n Data Summary\n\n Number of Strata 15\n Number of Clusters 31\n Number of Observations 8591\n Sum of Weights 276536446\n\n\n Statistics\n\n Std Error\n Variable N Mean of Mean 95% CL for Mean\n ---------------------------------------------------------------------------------\n HI_CHOL 7846 0.112143 0.005446 0.10059829 0.12368762\n RIAGENDR 8591 1.512019 0.005302 1.50077977 1.52325807\n ---------------------------------------------------------------------------------\n\n\n Ratio Analysis\n\n Std\n Numerator Denominator N Ratio Error 95% CL for Ratio\n ----------------------------------------------------------------------------------------------\n HI_CHOL RIAGENDR 7846 0.074222 0.003715 0.06634722 0.08209696\n ----------------------------------------------------------------------------------------------\n\n The SURVEYFREQ Procedure\n\n Data Summary\n\n Number of Strata 15\n Number of Clusters 31\n Number of Observations 8591\n Sum of Weights 276536446\n\n\n Table of agecat\n\n Weighted Std Err of Std Err of 95% Confidence Limits\n agecat Frequency Frequency Wgt Freq Percent Percent for Percent\n -------------------------------------------------------------------------------------------------------\n (0,19] 2532 57450307 3043819 20.7749 0.6130 19.4755 22.0744\n (19,39] 2033 81137975 3692818 29.3408 0.9561 27.3140 31.3676\n (39,59] 2021 83870623 4853936 30.3290 0.4519 29.3709 31.2870\n (59,Inf] 2005 54077541 4284296 19.5553 0.8093 17.8398 21.2709\n\n Total 8591 276536446 13935730 100.0000 \n -------------------------------------------------------------------------------------------------------\n\n The SURVEYMEANS Procedure\n\n Data Summary\n\n Number of Strata 15\n Number of Clusters 31\n Number of Observations 8591\n Sum of Weights 276536446\n\n\n Statistics\n\n Std Error Design\n Variable Mean of Mean Effect\n --------------------------------------------------------\n HI_CHOL 0.112143 0.005446 2.336725\n --------------------------------------------------------\n\n The SURVEYMEANS Procedure\n\n Statistics for race Domains\n\n Std Error Design\n race Variable Mean of Mean Effect\n ------------------------------------------------------------------------\n 1 HI_CHOL 0.101492 0.006246 1.082734\n 2 HI_CHOL 0.121649 0.006604 1.407822\n 3 HI_CHOL 0.078640 0.010385 2.091156\n 4 HI_CHOL 0.099679 0.024666 3.098290\n ------------------------------------------------------------------------\n```\n\n## Python\n\n\n::: {.cell}\n\n```{.python .cell-code}\nimport pandas as pd\nfrom samplics import TaylorEstimator\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nMatplotlib is building the font cache; this may take a moment.\n```\n\n\n:::\n\n```{.python .cell-code}\nfrom samplics.utils.types import PopParam\n\nnhanes = pd.read_csv(\"../data/nhanes.csv\")\n\nnhanes_design_kwargs = dict(\n psu=nhanes[\"SDMVPSU\"],\n stratum=nhanes[\"SDMVSTRA\"],\n samp_weight=nhanes[\"WTMEC2YR\"],\n remove_nan=True,\n)\n\n# Mean of HI_CHOL\nmean_estimator = TaylorEstimator(PopParam.mean)\nmean_estimator.estimate(nhanes[\"HI_CHOL\"], **nhanes_design_kwargs)\nhi_chol_means = mean_estimator.to_dataframe()\n\n# Sum of HI_CHOL\ntotal_estimator = TaylorEstimator(PopParam.total)\ntotal_estimator.estimate(nhanes[\"HI_CHOL\"], **nhanes_design_kwargs)\nhi_chol_totals = total_estimator.to_dataframe()\n\n# Ratio of HI_CHOL / RIAGENDR\nratio_estimator = TaylorEstimator(PopParam.ratio)\nratio_estimator.estimate(\n y=nhanes[\"HI_CHOL\"], x=nhanes[\"RIAGENDR\"], **nhanes_design_kwargs\n)\nhi_chol_ratio = ratio_estimator.to_dataframe()\n\n# Proportion of different AGECAT values\nprop_estimator = TaylorEstimator(PopParam.prop)\nprop_estimator.estimate(nhanes[\"agecat\"], **nhanes_design_kwargs)\nagecat_prop = prop_estimator.to_dataframe()\n\n# Quantiles of HI_CHOL\n# NA\n\n# Domain analysis of mean of HI_CHOL by race, with design effect\nmean_estimator = TaylorEstimator(PopParam.mean)\nmean_estimator.estimate(\n nhanes[\"HI_CHOL\"],\n **nhanes_design_kwargs,\n domain=nhanes[\"race\"],\n deff=True, # Design effect param currently has no effect\n)\nhi_chol_domain_means = mean_estimator.to_dataframe()\n\n\nag_dict = agecat_prop.set_index(\"_level\").to_dict()\nhc_dict = hi_chol_domain_means.set_index(\"_domain\").to_dict()\n\nprint(\n f\"\"\"\n Mean of HI_CHOL: {hi_chol_means[\"_estimate\"][0]}\n SE of Mean HI_CHOL: {hi_chol_means[\"_stderror\"][0]}\n CL of Mean HI_CHOL: {(hi_chol_means[\"_lci\"][0], hi_chol_means[\"_uci\"][0])}\n Sum of HI_CHOL: {hi_chol_totals[\"_estimate\"][0]}\n SE of Sum HI_CHOL: {hi_chol_totals[\"_stderror\"][0]}\n CL of Sum HI_CHOL: {(hi_chol_totals[\"_lci\"][0], hi_chol_totals[\"_uci\"][0])}\n Ratio of HI_CHOL / RIAGENDR: {hi_chol_ratio[\"_estimate\"][0]}\n SE of Ratio HI_CHOL / RIAGENDR: {hi_chol_ratio[\"_stderror\"][0]}\n CL of Ratio HI_CHOL / RIAGENDR: {(hi_chol_ratio[\"_lci\"][0], hi_chol_ratio[\"_uci\"][0])}\n Proportion of AGECAT: {ag_dict[\"_estimate\"]}\n SE of Proportion AGECAT: {ag_dict[\"_stderror\"]}\n LCL of Proportion AGECAT: {ag_dict[\"_lci\"]}\n UCL of Proportion AGECAT: {ag_dict[\"_uci\"]}\n Quantiles of HI_CHOL: Not available\n Mean of HI_CHOL by race: {hc_dict[\"_estimate\"]}\n SE of HI_CHL by race: {hc_dict[\"_stderror\"]}\n LCL of HI_CHOL by race: {hc_dict[\"_lci\"]}\n UCL of HI_CHOL by race: {hc_dict[\"_uci\"]}\n Design Effect of HI_CHOL by race: Not available\n \"\"\"\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n Mean of HI_CHOL: 0.11214295634969222\n SE of Mean HI_CHOL: 0.005445839698954557\n CL of Mean HI_CHOL: (np.float64(0.1005982919131703), np.float64(0.12368762078621415))\n Sum of HI_CHOL: 28635245.254672\n SE of Sum HI_CHOL: 2020710.7436996205\n CL of Sum HI_CHOL: (np.float64(24351529.84091034), np.float64(32918960.668433655))\n Ratio of HI_CHOL / RIAGENDR: 0.07422209323594066\n SE of Ratio HI_CHOL / RIAGENDR: 0.0037147278931070065\n CL of Ratio HI_CHOL / RIAGENDR: (np.float64(0.06634722189017901), np.float64(0.0820969645817023))\n Proportion of AGECAT: {'(0,19]': 0.2077494937870972, '(19,39]': 0.29340788818591346, '(39,59]': 0.30328958320385285, '(59,Inf]': 0.19555303482313666}\n SE of Proportion AGECAT: {'(0,19]': 0.006129950336419631, '(19,39]': 0.009560691634608896, '(39,59]': 0.004519462827363183, '(59,Inf]': 0.008092578243976422}\n LCL of Proportion AGECAT: {'(0,19]': 0.19505410930097866, '(19,39]': 0.27355685874096586, '(39,59]': 0.2937950591158628, '(59,Inf]': 0.1789647230500222}\n UCL of Proportion AGECAT: {'(0,19]': 0.2210442684297426, '(19,39]': 0.3140766293472951, '(39,59]': 0.31295496708023285, '(59,Inf]': 0.21327950895208636}\n Quantiles of HI_CHOL: Not available\n Mean of HI_CHOL by race: {1: 0.10149166545397208, 2: 0.12164920535593333, 3: 0.07864006039908408, 4: 0.09967860947712034}\n SE of HI_CHL by race: {1: 0.006245843308749599, 2: 0.006604133623532979, 3: 0.010384645000548863, 4: 0.024666226871851268}\n LCL of HI_CHOL by race: {1: 0.0882510691256497, 2: 0.10764906749064211, 3: 0.056625596431891564, 4: 0.04738854441969514}\n UCL of HI_CHOL by race: {1: 0.11473226178229445, 2: 0.13564934322122454, 3: 0.1006545243662766, 4: 0.15196867453454554}\n Design Effect of HI_CHOL by race: Not available\n \n```\n\n\n:::\n:::\n\n:::\n\n# Differences\n\n## Quantiles {#quantiles}\n\n`samplics` in Python does not have a method for calculating quantiles, and in R and SAS the available methods lead to different results. To demonstrate the differences in calculating quantiles, we will use the `apisrs` dataset from the `survey` package in R [@API_2000].\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(survey)\n\ndata(\"api\")\n\nhead(apisrs) |>\n gt::gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n \n
cdsstypenamesnamesnumdnamednumcnamecnumflagpcttestapi00api99targetgrowthsch.widecomp.impbothawardsmealsellyr.rndmobilityacs.k3acs.46acs.corepct.respnot.hsghsgsome.colcol.gradgrad.schavg.edfullemerenrollapi.stupwfpc
15739081534155HMcFarland HighMcFarland High1039McFarland Unified432Kern14NA984624481814NoYesNoNo4431NA6NANA2482443412731.91713547742930.976194
19642126066716EStowers (Cecil Stowers (Cecil B.) Elementary1124ABC Unified1Los Angeles18NA100878831NA47YesYesYesYes825NA151930NA974102343213.66901047842030.976194
30664493030640HBrea-Olinda HigBrea-Olinda High2868Brea-Olinda Unified79Orange29NA987347423-8NoNoNoNo1010NA7NANA2895592141243.7183181410128730.976194
19644516012744EAlameda ElementAlameda Elementary1273Downey Unified187Los Angeles18NA997726577115YesYesYesYes7025NA2323NANA100374014811.96851834229130.976194
40688096043293ESunnyside ElemeSunnyside Elementary4926San Luis Coastal Unified640San Luis Obispo39NA99739719420YesYesYesYes4312NA122029NA918212734103.17100021718930.976194
19734456014278ELos Molinos EleLos Molinos Elementary2463Hacienda la Puente Unif284Los Angeles18NA93835822NA13YesYesYesNo1619NA131929NA71182038343.96752025821130.976194
\n
\n```\n\n:::\n:::\n\n\nIn SAS, PROC SURVEYMEANS will calculate quantiles of specific probabilities as you request them, using Woodruff's method for intervals and a custom quantile method [@SAS_2018, pp. 9834]. The quantile method does not match any of the available `qrules` in R, and although the default `interval.types` in the R `survey::svyquantile` function also uses Woodruff's method, it is a different implementation.\n\nThe method and results from SAS are as follows:\n\n```sas\nproc surveymeans data=apisrs total=6194 quantile=(0.025 0.5 0.975);\n var growth;\nrun;\n```\n\n``` default\n The SURVEYMEANS Procedure\n\n Data Summary\n\n Number of Observations 200\n\n\n\n\n Quantiles\n\n Std\n Variable Percentile Estimate Error 95% Confidence Limits\n ---------------------------------------------------------------------------------\n growth 2.5 -16.500000 1.755916 -19.962591 -13.037409\n 50 Median 26.500000 1.924351 22.705263 30.294737\n 97.5 99.000000 16.133827 67.184794 130.815206\n ---------------------------------------------------------------------------------\n```\n\nIf in R we use the default `qrule=\"math\"` (equivalent to `qrule=\"hf1\"` and matches `type=1` in the `quantile` function for unweighted data) along with the default `interval.type=\"mean\"`, we get the following results:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsrs_design <- survey::svydesign(data = apisrs, id = ~1, fpc = ~fpc, )\n\nsurvey::svyquantile(\n ~growth,\n srs_design,\n quantiles = c(0.025, 0.5, 0.975),\n ci = TRUE,\n se = TRUE\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n$growth\n quantile ci.2.5 ci.97.5 se\n0.025 -16 -21 -12 2.281998\n0.5 27 24 31 1.774887\n0.975 99 84 189 26.623305\n\nattr(,\"hasci\")\n[1] TRUE\nattr(,\"class\")\n[1] \"newsvyquantile\"\n```\n\n\n:::\n:::\n\n\nHere we can see that the quantiles, confidence intervals, and standard errors do not match SAS. From testing, none of the available `qrule` methods match SAS for the quantile values, so it is recommended to use the default values unless you have need of some of the other properties of different quantile definitions - see [`vignette(\"qrule\", package=\"survey\")`](https://cran.r-project.org/web/packages/survey/vignettes/qrule.pdf) for more detail. If an exact match to SAS is required, then the `svyquantile` function allows for passing a custom function to the `qrule` argument to define your own method for calculating quantiles. Below is an example that will match SAS:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsas_qrule <- function(x, w, p) {\n # Custom qrule to match SAS, based on survey::oldsvyquantile's internal method\n if (any(is.na(x))) {\n return(NA * p)\n }\n w <- rowsum(w, x, reorder = TRUE)\n x <- sort(unique(x))\n cum.w <- cumsum(w) / sum(w)\n cdf <- approxfun(\n cum.w,\n x,\n method = \"linear\",\n f = 1,\n yleft = min(x),\n yright = max(x),\n ties = min\n )\n cdf(p)\n}\n\n\nsas_quants <- survey::svyquantile(\n ~growth,\n srs_design,\n quantiles = c(0.025, 0.5, 0.975),\n qrule = sas_qrule,\n ci = TRUE,\n se = TRUE\n)\n\nsas_quants\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n$growth\n quantile ci.2.5 ci.97.5 se\n0.025 -16.5 -22.00000 -15.07482 1.755916\n0.5 26.5 23.03563 30.62510 1.924351\n0.975 99.0 83.70616 147.33657 16.133827\n\nattr(,\"hasci\")\n[1] TRUE\nattr(,\"class\")\n[1] \"newsvyquantile\"\n```\n\n\n:::\n:::\n\n\nNote that although the quantiles and standard errors match, the confidence intervals still do not match SAS. For this another custom calculation is required, based on the formula used in SAS:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsas_quantile_confint <- function(newsvyquantile, level = 0.05, df = Inf) {\n q <- coef(newsvyquantile)\n se <- survey::SE(newsvyquantile)\n ci <- cbind(\n q,\n q + se * qt(level / 2, df),\n q - se * qt(1 - level / 2, df),\n se\n )\n colnames(ci) <- c(\n \"quantile\",\n paste0(\"ci.\", c(100 * level / 2, 100 * (1 - level / 2))),\n \"se\"\n )\n\n ci\n}\n\nsas_quantile_confint(sas_quants, df = survey::degf(srs_design))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n quantile ci.2.5 ci.97.5 se\ngrowth.0.025 -16.5 -19.96259 -19.96259 1.755916\ngrowth.0.5 26.5 22.70526 22.70526 1.924351\ngrowth.0.975 99.0 67.18479 67.18479 16.133827\n```\n\n\n:::\n:::\n\n\n## Other considerations\n\n### Degrees of Freedom\n\nSome of the functions in R require the degrees of freedom to be specified when calculating confidence intervals, otherwise it assumes a normal distribution. This can be done easily by using the `survey::degf` function, which calculates the degrees of freedom for a survey design object.\n\n### Single PSU Strata\n\nAlthough it was not apparent with the examples used here, if there is only one PSU from a stratum then R will by default error, whereas SAS will remove that stratum from the variance calculation. This can be changed in R by setting the `options(survey.lonely.psu=\"certainty\")` to match SAS and have it make no contribution to the variance. In `samplics`, this behaviour can be configured using the `single_psu` argument to the `estimate` method, and can be set to to match SAS using `SinglePSUEst.certainty`. This should be considered carefully however, in R and Python there are additional methods of handling single PSUs that may be more appropriate for your use-case.\n\n### Documentation Differences\n\nOne key consideration when choosing a statistical package is the documentation available. In this case, both the `survey` package in R and the survey procedures in SAS have a much more comprehensive set of documentation and examples than `samplics` in Python. This includes both detailed examples, as well as the underlying theory and methods used in the calculations including references to the literature.\n\n# Summary and Recommendations\n\nThe `{survey}` package in R and the survey procedures in SAS/STAT both provide similar functionality for calculating survey summary statistics. In most cases in both our tests and others, the results are identical ([@2017_YRBS], [@so2020modelling], [@adamico_2009]). Where differences do occur, primarily in calculating quantiles, the methods in R are more varied and well-documented.\n\nIn contrast, the `samplics` package in Python is still early in development, and although it does provide some functionality there are still major limitations in both basic statistics (i.e. quantiles) and in more complex methods that were beyond the scope of this document, and the methods are much less well-documented.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P survey * 4.4-8 2025-08-28 [?] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Python configuration ────────────────────────────────────────────────────────\n Python 3.12.3 (main, Mar 3 2026, 12:15:18) [GCC 13.3.0]\n samplics 0.4.22\n```\n\n\n:::\n:::\n\n:::", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/Comp/r-sas-summary-stats/execute-results/html.json b/_freeze/Comp/r-sas-summary-stats/execute-results/html.json index 238ae3190..13f363e05 100644 --- a/_freeze/Comp/r-sas-summary-stats/execute-results/html.json +++ b/_freeze/Comp/r-sas-summary-stats/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "d816264f52fcea7d8df6824cfcd3cba2", + "hash": "bc048ecc2f50f63adfadaa9f44329ef6", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Deriving Quantiles or Percentiles in R vs SAS\"\neval: false\n---\n\n### Data\n\nThe following data will be used show the differences between the default percentile definitions used by SAS and R:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nc(10, 20, 30, 40, 150, 160, 170, 180, 190, 200)\n```\n:::\n\n\n### SAS Code\n\nAssuming the data above is stored in the variable `aval` within the dataset `adlb`, the 25th and 40th percentiles could be calculated using the following code.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc univariate data=adlb;\n var aval;\n output out=stats pctlpts=25 40 pctlpre=p;\nrun;\n```\n:::\n\n\nThis procedure creates the dataset `stats` containing the variables `p25` and `p40`.\n\n\n::: {.cell layout-align=\"center\"}\n\n:::\n\n\nThe procedure has the option `PCTLDEF` which allows for five different percentile definitions to be used. The default is `PCTLDEF=5`.\n\n### R code\n\nThe 25th and 40th percentiles of `aval` can be calculated using the `quantile` function.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nquantile(adlb$aval, probs = c(0.25, 0.4))\n```\n:::\n\n\nThis gives the following output.\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n 25% 40% \n 32.5 106.0 \n```\n\n\n:::\n:::\n\n\nThe function has the argument `type` which allows for nine different percentile definitions to be used. The default is `type = 7`.\n\n### Comparison\n\nThe default percentile definition used by the UNIVARIATE procedure in SAS finds the 25th and 40th percentiles to be 30 and 95. The default definition used by R finds these percentiles to be 32.5 and 106.\n\nIt is possible to get the quantile function in R to use the same definition as the default used in SAS, by specifying `type=2`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nalquantile(adlb$aval, probs = c(0.25, 0.4), type = 2)\n```\n:::\n\n\nThis gives the following output.\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n25% 40% \n 30 95 \n```\n\n\n:::\n:::\n\n\nIt is not possible to get the UNIVARIATE procedure in SAS to use the same definition as the default used in R.\n\nRick Wicklin provided a [blog post](https://blogs.sas.com/content/iml/2017/05/24/definitions-sample-quantiles.html) showing how SAS has built in support for calculations using 5 of the 9 percentile definitions available in R, and also demonstrated how you can use a SAS/IML function to calculate percentiles using the other 4 definitions.\n\nMore information about quantile derivation can be found in the [SAS blog](https://blogs.sas.com/content/iml/2021/07/26/compare-quantiles-sas-r-python.html).\n\n### Key references:\n\n[Compare the default definitions for sample quantiles in SAS, R, and Python](https://blogs.sas.com/content/iml/2021/07/26/compare-quantiles-sas-r-python.html)\n\n[Sample quantiles: A comparison of 9 definitions](https://blogs.sas.com/content/iml/2017/05/24/definitions-sample-quantiles.html)\n\n[Hyndman, R. J., & Fan, Y. (1996). Sample quantiles in statistical packages. The American Statistician, 50(4), 361-365.](https://www.jstor.org/stable/2684934)\n", - "supporting": [], + "markdown": "---\ntitle: \"Deriving Quantiles or Percentiles in R vs SAS\"\n---\n\n### Data\n\nThe following data will be used show the differences between the default percentile definitions used by SAS and R:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nc(10, 20, 30, 40, 150, 160, 170, 180, 190, 200)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [1] 10 20 30 40 150 160 170 180 190 200\n```\n\n\n:::\n:::\n\n\n### SAS Code\n\nAssuming the data above is stored in the variable `aval` within the dataset `adlb`, the 25th and 40th percentiles could be calculated using the following code.\n\n```sas\nproc univariate data=adlb;\n var aval;\n output out=stats pctlpts=25 40 pctlpre=p;\nrun;\n```\n\nThis procedure creates the dataset `stats` containing the variables `p25` and `p40`.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/summarystats/sas-percentiles-output.jpg){fig-align='center' width=15%}\n:::\n:::\n\n\nThe procedure has the option `PCTLDEF` which allows for five different percentile definitions to be used. The default is `PCTLDEF=5`.\n\n### R code\n\nThe 25th and 40th percentiles of `aval` can be calculated using the `quantile` function.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nquantile(adlb$aval, probs = c(0.25, 0.4))\n```\n:::\n\n\nThis gives the following output.\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n 25% 40% \n 32.5 106.0 \n```\n\n\n:::\n:::\n\n\nThe function has the argument `type` which allows for nine different percentile definitions to be used. The default is `type = 7`.\n\n### Comparison\n\nThe default percentile definition used by the UNIVARIATE procedure in SAS finds the 25th and 40th percentiles to be 30 and 95. The default definition used by R finds these percentiles to be 32.5 and 106.\n\nIt is possible to get the quantile function in R to use the same definition as the default used in SAS, by specifying `type=2`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nalquantile(adlb$aval, probs = c(0.25, 0.4), type = 2)\n```\n:::\n\n\nThis gives the following output.\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n25% 40% \n 30 95 \n```\n\n\n:::\n:::\n\n\nIt is not possible to get the UNIVARIATE procedure in SAS to use the same definition as the default used in R.\n\nRick Wicklin provided a [blog post](https://blogs.sas.com/content/iml/2017/05/24/definitions-sample-quantiles.html) showing how SAS has built in support for calculations using 5 of the 9 percentile definitions available in R, and also demonstrated how you can use a SAS/IML function to calculate percentiles using the other 4 definitions.\n\nMore information about quantile derivation can be found in the [SAS blog](https://blogs.sas.com/content/iml/2021/07/26/compare-quantiles-sas-r-python.html).\n\n### Key references:\n\n[Compare the default definitions for sample quantiles in SAS, R, and Python](https://blogs.sas.com/content/iml/2021/07/26/compare-quantiles-sas-r-python.html)\n\n[Sample quantiles: A comparison of 9 definitions](https://blogs.sas.com/content/iml/2017/05/24/definitions-sample-quantiles.html)\n\n[Hyndman, R. J., & Fan, Y. (1996). Sample quantiles in statistical packages. The American Statistician, 50(4), 361-365.](https://www.jstor.org/stable/2684934)\n", + "supporting": [ + "r-sas-summary-stats_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas-wilcoxon-ranksum_hl/execute-results/html.json b/_freeze/Comp/r-sas-wilcoxon-ranksum_hl/execute-results/html.json index 2d46d3412..540bf973a 100644 --- a/_freeze/Comp/r-sas-wilcoxon-ranksum_hl/execute-results/html.json +++ b/_freeze/Comp/r-sas-wilcoxon-ranksum_hl/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "859f2601b50b8ddc83fcda0363baa68b", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS Wilcoxon Rank-Sum Test\"\n---\n\n\n\n## Introduction\n\nThis page compares the Wilcoxon rank-sum test, Hodges-Lehmann estimator, and estimation of the Mann-Whitney parameter in R and SAS.\n\n\n### Example Data\n\nFor this example we are using a dataset of birth weights for smoking and non-smoking mothers (*Data source: Table 30.4, Kirkwood BR. and Sterne JAC. Essentials of medical statistics. Second Edition. ISBN 978-0-86542-871-3*). This dataset is both small (so an exact test is recommended) and has ties in it.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nbw_ns <- c(3.99, 3.89, 3.6, 3.73, 3.31, \n 3.7, 4.08, 3.61, 3.83, 3.41, \n 4.13, 3.36, 3.54, 3.51, 2.71)\nbw_s <- c(3.18, 2.74, 2.9, 3.27, 3.65, \n 3.42, 3.23, 2.86, 3.6, 3.65, \n 3.69, 3.53, 2.38, 2.34)\n\nsmk_data <- data.frame(\n value = c(bw_ns, bw_s), \n smoke = as.factor(rep(c(\"non\", \"smoke\"), c(length(bw_ns), length(bw_s))))\n) \n# Relevel the factors to make it smoker - non-smokers \nsmk_data$smoke <- forcats::fct_relevel(smk_data$smoke, \"smoke\")\nhead(smk_data)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n value smoke\n1 3.99 non\n2 3.89 non\n3 3.60 non\n4 3.73 non\n5 3.31 non\n6 3.70 non\n```\n\n\n:::\n:::\n\n\nTo view the code implementations, see the [SAS](../SAS/ranksum.qmd) and [R](../R/nonpara_wilcoxon_ranksum.qmd) pages, respectively.\n\n\n## Comparison\n\n### Software Capabilities\n\nThe following table provides an overview of the supported analyses between R and SAS. A specific comparison of the results and whether they match are provided below.\n\n| Analysis | Supported in R {stats} | Supported in R {coin} | Supported in R {asht} | Supported in SAS | Notes |\n|----------|------------------------|-----------------------|-----------------------|------------------|-------|\n| Wilcoxon Rank-Sum -- Normal approximation **with** continuity correction | Yes | No | Yes | Yes | In {coin}, one can add `correct=TRUE`, but note that no error is given and the results of a normal approximation approach **without** continuity correction are provided.\n| Wilcoxon Rank-Sum -- Normal approximation **without** continuity correction | Yes | Yes | Yes | Yes |\n| Wilcoxon Rank-Sum -- Exact | Partly | Yes | Partly | Yes | In {stats}, one can only do the exact method when no ties are present.; In {asht}, exact test is possible but the run time is long for larger sample size.\n| Wilcoxon Rank-Sum -- Approximative (Monte Carlo simulation) | No | Yes | Yes | No |\n| Hodges-Lehmann estimator -- Asymptotic | Yes | No | No | Yes |\n| Hodges-Lehmann estimator -- Exact | Partly | Yes | No | Yes | In {stats}, one can only do the exact method when no ties are present.\n| Hodges-Lehmann estimator -- Approximative (Monte Carlo simulation) | No | Yes | No | No |\n| Mann-Whitney parameter | No | No | Yes | No | In {asht}, confidence intervals can be obtained using asymptotic approximation, Monte Carlo simulations, or exact methods (for small sample size)\n\n\n### Wilcoxon Rank Sum test\n\nIn the below table the p-values of the Wilcoxon Rank Sum Test with different options are compared.\n\n| Analysis | R {stats} | R {coin} | R {asht} | SAS | Match | Notes |\n|----------|-----------|----------|----------|-----|-------|-------|\n| Wilcoxon Rank-Sum -- Normal approximation **with** continuity correction | 0.0100 | / | 0.0100 | 0.0100 | Yes | Not possible with {coin}\n| Wilcoxon Rank-Sum -- Normal approximation **without** continuity correction | 0.0094 | 0.0094 | 0.0094 | 0.0094 | Yes | \n| Wilcoxon Rank-Sum -- Exact | / | 0.0082 | / | 0.0082 | Yes | Not possible with {stats} since there are ties.; In {asht} run-time very long. \n| Wilcoxon Rank-Sum -- Approximative (Monte Carlo simulation) | / | 0.0083 | 0.0083 | / | Yes | With 100,000 simulations\n\n\n### Hodges-Lehmann estimator\n\nIn the below table the Hodges-Lehmann estimate and 95% confidence intervals are compared.\n\n| Analysis | R {stats} | R {coin} | R {asht} | SAS | Match | Notes |\n|----------|-----------|----------|----------|-----|-------|-------|\n| Hodges-Lehmann estimator -- Asymptotic | -0.426 (-0.770 to -0.090) | -0.426 (-0.760 to -0.100) | / | -0.425 (-0.770 to -0.090) | No | In {coin}, the CI is the exact CI. The CIs match between {stats} and SAS.\n| Hodges-Lehmann estimator -- Exact | / | -0.425 (-0.760 to -0.100) | / | -0.425 (-0.760 to -0.100) | Yes | Not possible with {stats} since there are ties; In {asht} run-time very long.\n| Hodges-Lehmann estimator -- Approximative (Monte Carlo simulation) | / | -0.425 (-0.760 to -0.100) | / | /| / | With 500,000 simulations\n\n\n### Mann-Whitney Parameter\nThe estimation of the Mann-Whitney parameter is only possible in R `asht` package.\n\n\n\n## Special considerations for one-sided p-values\n\nIt is important to note that in SAS you can get an *unexpected* one-sided p-value. In the SAS documentation for [PROC NPAR1WAY](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_npar1way_details04.htm) it is stated that: \n\n*\"PROC NPAR1WAY computes one-sided and two-sided asymptotic p-values for each two-sample linear rank test. When the test statistic z is greater than its null hypothesis expected value of 0, PROC NPAR1WAY computes the right-sided p-value, which is the probability of a larger value of the statistic occurring under the null hypothesis. When the test statistic is less than or equal to 0, PROC NPAR1WAY computes the left-sided p-value, which is the probability of a smaller value of the statistic occurring under the null hypothesis\"* (similar for the exact p-value). \n\nThus SAS reports the one-sided p-value in the direction of the test statistic. This can cause an *unexpected* one-sided p-value, if your data provides a test statistic in the other direction of the pre-specified one-sided hypothesis.\n\nConsider the following data example to showcase this:\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_used <- data.frame(\n ID = c(\"001\", \"002\", \"003\", \"004\", \"005\", \"006\", \"007\", \"008\", \"009\", \"010\",\n \"011\", \"012\", \"013\", \"014\", \"015\", \"016\", \"017\", \"018\", \"019\", \"020\",\n \"021\", \"022\", \"023\", \"024\", \"025\", \"026\", \"027\", \"028\", \"029\", \"030\"),\n ARM = c(\"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\",\n \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\",\n \"High\", \"High\", \"High\", \"High\", \"High\", \"High\", \"High\", \"High\", \"High\", \"High\"),\n Y = c(8.5, 8.9, 8.2, 8.1, 7.1, 7.4, 6.0, 6.5, 7.0, 7.0,\n 6.5, 9.4, 8.9, 8.8, 9.6, 8.3, 8.9, 7.0, 9.1, 6.9,\n 8.0, 7.3, 7.1, 6.2, 4.7, 4.7, 4.2, 4.1, 3.4, 3.9)\n)\ndat_used\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n ID ARM Y\n1 001 Placebo 8.5\n2 002 Placebo 8.9\n3 003 Placebo 8.2\n4 004 Placebo 8.1\n5 005 Placebo 7.1\n6 006 Placebo 7.4\n7 007 Placebo 6.0\n8 008 Placebo 6.5\n9 009 Placebo 7.0\n10 010 Placebo 7.0\n11 011 Low 6.5\n12 012 Low 9.4\n13 013 Low 8.9\n14 014 Low 8.8\n15 015 Low 9.6\n16 016 Low 8.3\n17 017 Low 8.9\n18 018 Low 7.0\n19 019 Low 9.1\n20 020 Low 6.9\n21 021 High 8.0\n22 022 High 7.3\n23 023 High 7.1\n24 024 High 6.2\n25 025 High 4.7\n26 026 High 4.7\n27 027 High 4.2\n28 028 High 4.1\n29 029 High 3.4\n30 030 High 3.9\n```\n\n\n:::\n:::\n\n\nSuppose we would have the following two hypothesis, where for both Low Dose and High Dose we expect smaller values (Y) than Placebo:\n\n- $H_{0}$: No difference between Placebo and Low Dose, vs $H_{1}$: Placebo has higher values (Y) than Low Dose\n\n- $H_{0}$: No difference between Placebo and High Dose, vs $H_{1}$: Placebo has higher values (Y) than High Dose\n\n\n### Asymptotic results without continuity correction\n\n**Placebo and High Dose group**\n\nLet us the {coin} package in R to compare the Placebo and High Dose group:\n\n::: {.cell}\n\n```{.r .cell-code}\n# Note: greater implies that H1 is Y1 - Y2 = Placebo - High > 0\ncoin::wilcox_test(\n Y ~ factor(ARM, levels = c(\"Placebo\", \"High\")),\n distribution = \"asymptotic\",\n alternative = \"greater\",\n data = dat_used %>% dplyr::filter(ARM %in% c(\"Placebo\", \"High\")))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Wilcoxon-Mann-Whitney Test\n\ndata: Y by\n\t factor(ARM, levels = c(\"Placebo\", \"High\")) (Placebo, High)\nZ = 2.5352, p-value = 0.005619\nalternative hypothesis: true mu is greater than 0\n```\n\n\n:::\n:::\n\n\nIn SAS, the following results is obtained. As can be seen in both R and SAS the one-sided p-value is 0.0056.\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/ranksum/SAS_one_sided_pvalue.png){fig-align='center' width=90%}\n:::\n:::\n\n\n\n**Placebo and Low Dose group**\n\nLet us the {coin} package in R to compare the Placebo and Low Dose group:\n\n::: {.cell}\n\n```{.r .cell-code}\n# Note: greater implies that H1 is Y1 - Y2 = Placebo - High > 0\ncoin::wilcox_test(\n Y ~ factor(ARM, levels = c(\"Placebo\", \"Low\")),\n distribution = \"asymptotic\",\n alternative = \"greater\",\n data = dat_used %>% dplyr::filter(ARM %in% c(\"Placebo\", \"Low\")))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Wilcoxon-Mann-Whitney Test\n\ndata: Y by\n\t factor(ARM, levels = c(\"Placebo\", \"Low\")) (Placebo, Low)\nZ = -1.7066, p-value = 0.9561\nalternative hypothesis: true mu is greater than 0\n```\n\n\n:::\n:::\n\n\nIn SAS, the following results is obtained. The one-sided p-values clearly do not match ({coin} p-value = 0.9561; SAS p-value = 0.0439). As mentioned above, SAS reports the p-value in the direction of the test statistic. This can cause an *unexpected* one-sided p-value, if your data provides a test statistic in the other directiont than the pre-specified one-sided hypothesis. Do note that $1 - 0.9561 = 0.0439$.\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/ranksum/SAS_one_sided_pvalue2.png){fig-align='center' width=90%}\n:::\n:::\n\n\n\n### Exact results\n\n**Placebo and High Dose group**\n\nLet us the {coin} package in R to compare the Placebo and High Dose group:\n\n::: {.cell}\n\n```{.r .cell-code}\n# Note: greater implies that H1 is Y1 - Y2 = Placebo - High > 0\ncoin::wilcox_test(\n Y ~ factor(ARM, levels = c(\"Placebo\", \"High\")),\n distribution = \"exact\",\n alternative = \"greater\",\n data = dat_used %>% dplyr::filter(ARM %in% c(\"Placebo\", \"High\")))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tExact Wilcoxon-Mann-Whitney Test\n\ndata: Y by\n\t factor(ARM, levels = c(\"Placebo\", \"High\")) (Placebo, High)\nZ = 2.5352, p-value = 0.004682\nalternative hypothesis: true mu is greater than 0\n```\n\n\n:::\n:::\n\n\nIn SAS (see above), the same one-sided p-value of 0.0047 is obtained.\n\n\n**Placebo and Low Dose group**\n\nLet us the {coin} package in R to compare the Placebo and Low Dose group:\n\n::: {.cell}\n\n```{.r .cell-code}\n# Note: greater implies that H1 is Y1 - Y2 = Placebo - High > 0\ncoin::wilcox_test(\n Y ~ factor(ARM, levels = c(\"Placebo\", \"Low\")),\n distribution = \"exact\",\n alternative = \"greater\",\n data = dat_used %>% dplyr::filter(ARM %in% c(\"Placebo\", \"Low\")))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tExact Wilcoxon-Mann-Whitney Test\n\ndata: Y by\n\t factor(ARM, levels = c(\"Placebo\", \"Low\")) (Placebo, Low)\nZ = -1.7066, p-value = 0.9574\nalternative hypothesis: true mu is greater than 0\n```\n\n\n:::\n:::\n\n\nPlease see above for the SAS result. The one-sided p-values clearly do not match ({coin} p-value = 0.9574; SAS p-value = 0.0455).\n\n\n\n## Summary and Recommendation\n\nWilcoxon Rank Sum test and the associated Hodges-Lehmann CI are able to be consistently computed in both SAS and R. The user needs to be aware of some small differences:\n\n- In SAS the `exact wilcoxon hl` statement is needed to get both the exact p-value and CI.\n\n- In {stats} exact values are only possible when there are no ties and the exact parameter is set to true (`exact = TRUE`). This will give the exact p-value and CI.\n\n- In {coin} it is not possible to do a normal approximation **with** continuity correction.\n\n- For the asymptotic Hodges-Lehmann estimator, {stats} and {coin} use an algorithm to define the estimate, whereas SAS provides the *traditional* Hodges-Lehmann estimator. \n\nIf you have a study where you would like to use R for the exact Wilcoxon Rank Sum test and there is the risk of ties, {coin} would be recommended.\n\n\n\n## Ties\n\nIn all presented R packages and SAS, when there are tied values, the average score method (mid-ranks) is used. This is done by first sorting the observations in ascending order and assigning ranks as if there were no ties. The procedure averages the scores for tied observations and assigns this average score to each of the tied observations. Thus, all tied data values have the same score value.\n\n\n## Additional References\n\nProvided are references and additional reading materials for both R and SAS documentation related to the analysis.\n\n**R Documentation:**\n\n- `wilcox.test` function: \n\n- `wilcox_test` function: \n\n**SAS Documentation:**\n\n- `PROC npar1way`: \n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-24\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P coin * 1.4-3 2023-09-27 [?] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::\n", + "markdown": "---\ntitle: \"R vs SAS Wilcoxon Rank-Sum Test\"\n---\n\n\n\n## Introduction\n\nThis page compares the Wilcoxon rank-sum test, Hodges-Lehmann estimator, and estimation of the Mann-Whitney parameter in R and SAS.\n\n\n### Example Data\n\nFor this example we are using a dataset of birth weights for smoking and non-smoking mothers (*Data source: Table 30.4, Kirkwood BR. and Sterne JAC. Essentials of medical statistics. Second Edition. ISBN 978-0-86542-871-3*). This dataset is both small (so an exact test is recommended) and has ties in it.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nbw_ns <- c(3.99, 3.89, 3.6, 3.73, 3.31, \n 3.7, 4.08, 3.61, 3.83, 3.41, \n 4.13, 3.36, 3.54, 3.51, 2.71)\nbw_s <- c(3.18, 2.74, 2.9, 3.27, 3.65, \n 3.42, 3.23, 2.86, 3.6, 3.65, \n 3.69, 3.53, 2.38, 2.34)\n\nsmk_data <- data.frame(\n value = c(bw_ns, bw_s), \n smoke = as.factor(rep(c(\"non\", \"smoke\"), c(length(bw_ns), length(bw_s))))\n) \n# Relevel the factors to make it smoker - non-smokers \nsmk_data$smoke <- forcats::fct_relevel(smk_data$smoke, \"smoke\")\nhead(smk_data)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n value smoke\n1 3.99 non\n2 3.89 non\n3 3.60 non\n4 3.73 non\n5 3.31 non\n6 3.70 non\n```\n\n\n:::\n:::\n\n\nTo view the code implementations, see the [SAS](../SAS/ranksum.qmd) and [R](../R/nonpara_wilcoxon_ranksum.qmd) pages, respectively.\n\n\n## Comparison\n\n### Software Capabilities\n\nThe following table provides an overview of the supported analyses between R and SAS. A specific comparison of the results and whether they match are provided below.\n\n| Analysis | Supported in R {stats} | Supported in R {coin} | Supported in R {asht} | Supported in SAS | Notes |\n|----------|------------------------|-----------------------|-----------------------|------------------|-------|\n| Wilcoxon Rank-Sum -- Normal approximation **with** continuity correction | Yes | No | Yes | Yes | In {coin}, one can add `correct=TRUE`, but note that no error is given and the results of a normal approximation approach **without** continuity correction are provided.\n| Wilcoxon Rank-Sum -- Normal approximation **without** continuity correction | Yes | Yes | Yes | Yes |\n| Wilcoxon Rank-Sum -- Exact | Partly | Yes | Partly | Yes | In {stats}, one can only do the exact method when no ties are present.; In {asht}, exact test is possible but the run time is long for larger sample size.\n| Wilcoxon Rank-Sum -- Approximative (Monte Carlo simulation) | No | Yes | Yes | No |\n| Hodges-Lehmann estimator -- Asymptotic | Yes | No | No | Yes |\n| Hodges-Lehmann estimator -- Exact | Partly | Yes | No | Yes | In {stats}, one can only do the exact method when no ties are present.\n| Hodges-Lehmann estimator -- Approximative (Monte Carlo simulation) | No | Yes | No | No |\n| Mann-Whitney parameter | No | No | Yes | No | In {asht}, confidence intervals can be obtained using asymptotic approximation, Monte Carlo simulations, or exact methods (for small sample size)\n\n\n### Wilcoxon Rank Sum test\n\nIn the below table the p-values of the Wilcoxon Rank Sum Test with different options are compared.\n\n| Analysis | R {stats} | R {coin} | R {asht} | SAS | Match | Notes |\n|----------|-----------|----------|----------|-----|-------|-------|\n| Wilcoxon Rank-Sum -- Normal approximation **with** continuity correction | 0.0100 | / | 0.0100 | 0.0100 | Yes | Not possible with {coin}\n| Wilcoxon Rank-Sum -- Normal approximation **without** continuity correction | 0.0094 | 0.0094 | 0.0094 | 0.0094 | Yes | \n| Wilcoxon Rank-Sum -- Exact | / | 0.0082 | / | 0.0082 | Yes | Not possible with {stats} since there are ties.; In {asht} run-time very long. \n| Wilcoxon Rank-Sum -- Approximative (Monte Carlo simulation) | / | 0.0083 | 0.0083 | / | Yes | With 100,000 simulations\n\n\n### Hodges-Lehmann estimator\n\nIn the below table the Hodges-Lehmann estimate and 95% confidence intervals are compared.\n\n| Analysis | R {stats} | R {coin} | R {asht} | SAS | Match | Notes |\n|----------|-----------|----------|----------|-----|-------|-------|\n| Hodges-Lehmann estimator -- Asymptotic | -0.426 (-0.770 to -0.090) | -0.426 (-0.760 to -0.100) | / | -0.425 (-0.770 to -0.090) | No | In {coin}, the CI is the exact CI. The CIs match between {stats} and SAS.\n| Hodges-Lehmann estimator -- Exact | / | -0.425 (-0.760 to -0.100) | / | -0.425 (-0.760 to -0.100) | Yes | Not possible with {stats} since there are ties; In {asht} run-time very long.\n| Hodges-Lehmann estimator -- Approximative (Monte Carlo simulation) | / | -0.425 (-0.760 to -0.100) | / | /| / | With 500,000 simulations\n\n\n### Mann-Whitney Parameter\nThe estimation of the Mann-Whitney parameter is only possible in R `asht` package.\n\n\n\n## Special considerations for one-sided p-values\n\nIt is important to note that in SAS you can get an *unexpected* one-sided p-value. In the SAS documentation for [PROC NPAR1WAY](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_npar1way_details04.htm) it is stated that: \n\n*\"PROC NPAR1WAY computes one-sided and two-sided asymptotic p-values for each two-sample linear rank test. When the test statistic z is greater than its null hypothesis expected value of 0, PROC NPAR1WAY computes the right-sided p-value, which is the probability of a larger value of the statistic occurring under the null hypothesis. When the test statistic is less than or equal to 0, PROC NPAR1WAY computes the left-sided p-value, which is the probability of a smaller value of the statistic occurring under the null hypothesis\"* (similar for the exact p-value). \n\nThus SAS reports the one-sided p-value in the direction of the test statistic. This can cause an *unexpected* one-sided p-value, if your data provides a test statistic in the other direction of the pre-specified one-sided hypothesis.\n\nConsider the following data example to showcase this:\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_used <- data.frame(\n ID = c(\"001\", \"002\", \"003\", \"004\", \"005\", \"006\", \"007\", \"008\", \"009\", \"010\",\n \"011\", \"012\", \"013\", \"014\", \"015\", \"016\", \"017\", \"018\", \"019\", \"020\",\n \"021\", \"022\", \"023\", \"024\", \"025\", \"026\", \"027\", \"028\", \"029\", \"030\"),\n ARM = c(\"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\", \"Placebo\",\n \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\", \"Low\",\n \"High\", \"High\", \"High\", \"High\", \"High\", \"High\", \"High\", \"High\", \"High\", \"High\"),\n Y = c(8.5, 8.9, 8.2, 8.1, 7.1, 7.4, 6.0, 6.5, 7.0, 7.0,\n 6.5, 9.4, 8.9, 8.8, 9.6, 8.3, 8.9, 7.0, 9.1, 6.9,\n 8.0, 7.3, 7.1, 6.2, 4.7, 4.7, 4.2, 4.1, 3.4, 3.9)\n)\ndat_used\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n ID ARM Y\n1 001 Placebo 8.5\n2 002 Placebo 8.9\n3 003 Placebo 8.2\n4 004 Placebo 8.1\n5 005 Placebo 7.1\n6 006 Placebo 7.4\n7 007 Placebo 6.0\n8 008 Placebo 6.5\n9 009 Placebo 7.0\n10 010 Placebo 7.0\n11 011 Low 6.5\n12 012 Low 9.4\n13 013 Low 8.9\n14 014 Low 8.8\n15 015 Low 9.6\n16 016 Low 8.3\n17 017 Low 8.9\n18 018 Low 7.0\n19 019 Low 9.1\n20 020 Low 6.9\n21 021 High 8.0\n22 022 High 7.3\n23 023 High 7.1\n24 024 High 6.2\n25 025 High 4.7\n26 026 High 4.7\n27 027 High 4.2\n28 028 High 4.1\n29 029 High 3.4\n30 030 High 3.9\n```\n\n\n:::\n:::\n\n\nSuppose we would have the following two hypothesis, where for both Low Dose and High Dose we expect smaller values (Y) than Placebo:\n\n- $H_{0}$: No difference between Placebo and Low Dose, vs $H_{1}$: Placebo has higher values (Y) than Low Dose\n\n- $H_{0}$: No difference between Placebo and High Dose, vs $H_{1}$: Placebo has higher values (Y) than High Dose\n\n\n### Asymptotic results without continuity correction\n\n**Placebo and High Dose group**\n\nLet us the {coin} package in R to compare the Placebo and High Dose group:\n\n::: {.cell}\n\n```{.r .cell-code}\n# Note: greater implies that H1 is Y1 - Y2 = Placebo - High > 0\ncoin::wilcox_test(\n Y ~ factor(ARM, levels = c(\"Placebo\", \"High\")),\n distribution = \"asymptotic\",\n alternative = \"greater\",\n data = dat_used %>% dplyr::filter(ARM %in% c(\"Placebo\", \"High\")))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Wilcoxon-Mann-Whitney Test\n\ndata: Y by\n\t factor(ARM, levels = c(\"Placebo\", \"High\")) (Placebo, High)\nZ = 2.5352, p-value = 0.005619\nalternative hypothesis: true mu is greater than 0\n```\n\n\n:::\n:::\n\n\nIn SAS, the following results is obtained. As can be seen in both R and SAS the one-sided p-value is 0.0056.\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/ranksum/SAS_one_sided_pvalue.png){fig-align='center' width=90%}\n:::\n:::\n\n\n\n**Placebo and Low Dose group**\n\nLet us the {coin} package in R to compare the Placebo and Low Dose group:\n\n::: {.cell}\n\n```{.r .cell-code}\n# Note: greater implies that H1 is Y1 - Y2 = Placebo - High > 0\ncoin::wilcox_test(\n Y ~ factor(ARM, levels = c(\"Placebo\", \"Low\")),\n distribution = \"asymptotic\",\n alternative = \"greater\",\n data = dat_used %>% dplyr::filter(ARM %in% c(\"Placebo\", \"Low\")))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Wilcoxon-Mann-Whitney Test\n\ndata: Y by\n\t factor(ARM, levels = c(\"Placebo\", \"Low\")) (Placebo, Low)\nZ = -1.7066, p-value = 0.9561\nalternative hypothesis: true mu is greater than 0\n```\n\n\n:::\n:::\n\n\nIn SAS, the following results is obtained. The one-sided p-values clearly do not match ({coin} p-value = 0.9561; SAS p-value = 0.0439). As mentioned above, SAS reports the p-value in the direction of the test statistic. This can cause an *unexpected* one-sided p-value, if your data provides a test statistic in the other directiont than the pre-specified one-sided hypothesis. Do note that $1 - 0.9561 = 0.0439$.\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/ranksum/SAS_one_sided_pvalue2.png){fig-align='center' width=90%}\n:::\n:::\n\n\n\n### Exact results\n\n**Placebo and High Dose group**\n\nLet us the {coin} package in R to compare the Placebo and High Dose group:\n\n::: {.cell}\n\n```{.r .cell-code}\n# Note: greater implies that H1 is Y1 - Y2 = Placebo - High > 0\ncoin::wilcox_test(\n Y ~ factor(ARM, levels = c(\"Placebo\", \"High\")),\n distribution = \"exact\",\n alternative = \"greater\",\n data = dat_used %>% dplyr::filter(ARM %in% c(\"Placebo\", \"High\")))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tExact Wilcoxon-Mann-Whitney Test\n\ndata: Y by\n\t factor(ARM, levels = c(\"Placebo\", \"High\")) (Placebo, High)\nZ = 2.5352, p-value = 0.004682\nalternative hypothesis: true mu is greater than 0\n```\n\n\n:::\n:::\n\n\nIn SAS (see above), the same one-sided p-value of 0.0047 is obtained.\n\n\n**Placebo and Low Dose group**\n\nLet us the {coin} package in R to compare the Placebo and Low Dose group:\n\n::: {.cell}\n\n```{.r .cell-code}\n# Note: greater implies that H1 is Y1 - Y2 = Placebo - High > 0\ncoin::wilcox_test(\n Y ~ factor(ARM, levels = c(\"Placebo\", \"Low\")),\n distribution = \"exact\",\n alternative = \"greater\",\n data = dat_used %>% dplyr::filter(ARM %in% c(\"Placebo\", \"Low\")))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tExact Wilcoxon-Mann-Whitney Test\n\ndata: Y by\n\t factor(ARM, levels = c(\"Placebo\", \"Low\")) (Placebo, Low)\nZ = -1.7066, p-value = 0.9574\nalternative hypothesis: true mu is greater than 0\n```\n\n\n:::\n:::\n\n\nPlease see above for the SAS result. The one-sided p-values clearly do not match ({coin} p-value = 0.9574; SAS p-value = 0.0455).\n\n\n\n## Summary and Recommendation\n\nWilcoxon Rank Sum test and the associated Hodges-Lehmann CI are able to be consistently computed in both SAS and R. The user needs to be aware of some small differences:\n\n- In SAS the `exact wilcoxon hl` statement is needed to get both the exact p-value and CI.\n\n- In {stats} exact values are only possible when there are no ties and the exact parameter is set to true (`exact = TRUE`). This will give the exact p-value and CI.\n\n- In {coin} it is not possible to do a normal approximation **with** continuity correction.\n\n- For the asymptotic Hodges-Lehmann estimator, {stats} and {coin} use an algorithm to define the estimate, whereas SAS provides the *traditional* Hodges-Lehmann estimator. \n\nIf you have a study where you would like to use R for the exact Wilcoxon Rank Sum test and there is the risk of ties, {coin} would be recommended.\n\n\n\n## Ties\n\nIn all presented R packages and SAS, when there are tied values, the average score method (mid-ranks) is used. This is done by first sorting the observations in ascending order and assigning ranks as if there were no ties. The procedure averages the scores for tied observations and assigns this average score to each of the tied observations. Thus, all tied data values have the same score value.\n\n\n## Additional References\n\nProvided are references and additional reading materials for both R and SAS documentation related to the analysis.\n\n**R Documentation:**\n\n- `wilcox.test` function: \n\n- `wilcox_test` function: \n\n**SAS Documentation:**\n\n- `PROC npar1way`: \n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P coin * 1.4-3 2023-09-27 [?] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::\n", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/Comp/r-sas_anova/execute-results/html.json b/_freeze/Comp/r-sas_anova/execute-results/html.json index 0ba37e414..18ed8c366 100644 --- a/_freeze/Comp/r-sas_anova/execute-results/html.json +++ b/_freeze/Comp/r-sas_anova/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "79157f1cc7d0b4eae8e7d97d52ab4baf", + "hash": "413f591182708dfadf5464b6fb2b5ba2", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS Linear Models\"\nexecute: \n eval: false\n---\n\n# R vs. SAS ANOVA\n\n## Introduction\n\nThis section compares the implementation of analysis of variance (ANOVA) in R and SAS. ANOVA compares the mean of two or more groups to determine if at least one group is significantly different from the others.\n\nR and SAS give the same result for the linear model. But, there some differences with calculating sums of squares. If you are looking for type I sum of square that is available in base R `stats` package using the `anova()` function. Type II and Type III sum of squares are available in the `car` and the `rstatix` packages. `rstatix` uses the `car` package to calculate the sum of square, but can be considered easier to use as it handles the contrast for type III automatically.\n\n## General Comparison Table\n\nThe following table provides an overview of the support and results comparability between R and SAS for the new analysis point.\n\n+----------+----------------+------------------+---------------+------------------------------------------+\n| Analysis | Supported in R | Supported in SAS | Results Match | Notes |\n+==========+================+==================+===============+==========================================+\n| ANOVA | Yes ✅ | Yes ✅ | Mostly yes | R can't calculate type IV Sum of Squares |\n+----------+----------------+------------------+---------------+------------------------------------------+\n\n### Matching Contrasts: R and SAS {.unnumbered}\n\n### Scenario 1: Basic Functionality\n\n#### R Code Example\n\nIn order to get the ANOVA model fit and sum of squares you can use the `anova` function in the `stats` package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(emmeans)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nWelcome to emmeans.\nCaution: You lose important information if you filter this package's results.\nSee '? untidy'\n```\n\n\n:::\n\n```{.r .cell-code}\ndrug_trial <- read.csv(\"../data/drug_trial.csv\")\n\nlm_model <- lm(formula = post ~ pre + drug, data = drug_trial)\nlm_model |>\n anova()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnalysis of Variance Table\n\nResponse: post\n Df Sum Sq Mean Sq F value Pr(>F) \npre 1 802.94 802.94 50.0393 1.639e-07 ***\ndrug 2 68.55 34.28 2.1361 0.1384 \nResiduals 26 417.20 16.05 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\nIt is recommended to use the `emmeans` package to get the contrasts between R.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlm_model |>\n emmeans(\"drug\") |>\n contrast(\n method = list(\n \"C vs A\" = c(-1, 1, 0),\n \"E vs CA\" = c(-1, -1, 2)\n )\n )\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df t.ratio p.value\n C vs A 0.109 1.80 26 0.061 0.9521\n E vs CA 6.783 3.28 26 2.067 0.0488\n```\n\n\n:::\n:::\n\n\nIn SAS, all contrasts must be manually defined, but the syntax is largely similar in both.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc glm data=work.mycsv;\n class drug;\n model post = pre drug / solution;\n estimate 'C vs A' drug -1 1 0;\n estimate 'E vs CA' drug -1 -1 2;\nrun;\n```\n:::\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/linear/clipboard-2025781059.png){fig-align='center' width=75%}\n:::\n\n::: {.cell-output-display}\n![](../images/linear/clipboard-1394032862.png){fig-align='center' width=75%}\n:::\n:::\n\n\n#### Results Comparison\n\nProvided below is a detailed comparison of the results obtained from both SAS and R.\n\n##### Sums of Squares\n\n+------------------------+-------------+-------------+-------------+\n| Statistic | R Result | SAS Result | Match |\n+========================+=============+=============+=============+\n| Sum of Square (Type I) | 802.94 | 802.94 | Yes |\n| | | | |\n| | 68.55 | 68.55 | |\n+------------------------+-------------+-------------+-------------+\n| Degrees of Freedom | 1 | 1 | Yes |\n| | | | |\n| | 2 | 2 | |\n+------------------------+-------------+-------------+-------------+\n| Mean Square | 802.94 | 802.94 | Yes |\n| | | | |\n| | 34.28 | 34.28 | |\n+------------------------+-------------+-------------+-------------+\n| F Value | 50.04 | 50.04 | Yes |\n| | | | |\n| | 2.14 | 2.14 | |\n+------------------------+-------------+-------------+-------------+\n| p-value | \\<0.0001 | \\<0.0001 | Yes |\n| | | | |\n| | 0.1384 | 0.1384 | |\n+------------------------+-------------+-------------+-------------+\n\n##### Contrasts\n\n| Statistic | R Result | SAS Result | Match |\n|---------------------------------|----------|------------|-------|\n| ***contrast estimate C vs A*** | 0.109 | 0.109 | Yes |\n| SE | 1.80 | 1.80 | Yes |\n| t-ratio | 0.06 | 0.06 | Yes |\n| p-value | 0.9521 | 0.9521 | Yes |\n| ***contrast estimate E vs CA*** | 6.783 | 6.783 | Yes |\n| SE | 3.28 | 3.28 | Yes |\n| t-ratio | 2.07 | 2.07 | Yes |\n| p-value | 0.0488 | 0.0488 | Yes |\n\nNote, however, that there are some cases where the scale of the parameter estimates between SAS and R is off, though the test statistics and p-values are identical. In these cases, we can adjust the SAS code to include a divisor. As far as we can tell, this difference only occurs when using the predefined Base R contrast methods like `contr.helmert`.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc glm data=work.mycsv;\n class drug;\n model post = pre drug / solution;\n estimate 'C vs A' drug -1 1 0 / divisor = 2;\n estimate 'E vs CA' drug -1 -1 2 / divisor = 6;\nrun;\n```\n:::\n\n\n## Summary and Recommendation\n\nThere were no major differences between the R emmeans package and the SAS PROC GLM step in conducting ANOVA on the clinical trial data. Both are robust software tools that generate mostly same results. Scaling for parameter coefficients need to be handled with care however as contrast estimates between R and S differed by a sign.\n\n## Additional References\n\nProvide references and additional reading materials for both R and SAS documentation related to the analysis.\n\n**R Documentation:**\n\n- `lm` function: \n- `emmeans` package: \n\n**SAS Documentation:**\n\n- `PROC GLM`: ", - "supporting": [], + "markdown": "---\ntitle: \"R vs SAS Linear Models\"\n---\n\n# R vs. SAS ANOVA\n\n## Introduction\n\nThis section compares the implementation of analysis of variance (ANOVA) in R and SAS. ANOVA compares the mean of two or more groups to determine if at least one group is significantly different from the others.\n\nR and SAS give the same result for the linear model. But, there some differences with calculating sums of squares. If you are looking for type I sum of square that is available in base R `stats` package using the `anova()` function. Type II and Type III sum of squares are available in the `car` and the `rstatix` packages. `rstatix` uses the `car` package to calculate the sum of square, but can be considered easier to use as it handles the contrast for type III automatically.\n\n## General Comparison Table\n\nThe following table provides an overview of the support and results comparability between R and SAS for the new analysis point.\n\n+----------+----------------+------------------+---------------+------------------------------------------+\n| Analysis | Supported in R | Supported in SAS | Results Match | Notes |\n+==========+================+==================+===============+==========================================+\n| ANOVA | Yes ✅ | Yes ✅ | Mostly yes | R can't calculate type IV Sum of Squares |\n+----------+----------------+------------------+---------------+------------------------------------------+\n\n### Matching Contrasts: R and SAS {.unnumbered}\n\n### Scenario 1: Basic Functionality\n\n#### R Code Example\n\nIn order to get the ANOVA model fit and sum of squares you can use the `anova` function in the `stats` package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(emmeans)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nWelcome to emmeans.\nCaution: You lose important information if you filter this package's results.\nSee '? untidy'\n```\n\n\n:::\n\n```{.r .cell-code}\ndrug_trial <- read.csv(\"../data/drug_trial.csv\")\n\nlm_model <- lm(formula = post ~ pre + drug, data = drug_trial)\nlm_model |>\n anova()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnalysis of Variance Table\n\nResponse: post\n Df Sum Sq Mean Sq F value Pr(>F) \npre 1 802.94 802.94 50.0393 1.639e-07 ***\ndrug 2 68.55 34.28 2.1361 0.1384 \nResiduals 26 417.20 16.05 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\nIt is recommended to use the `emmeans` package to get the contrasts between R.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlm_model |>\n emmeans(\"drug\") |>\n contrast(\n method = list(\n \"C vs A\" = c(-1, 1, 0),\n \"E vs CA\" = c(-1, -1, 2)\n )\n )\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df t.ratio p.value\n C vs A 0.109 1.80 26 0.061 0.9521\n E vs CA 6.783 3.28 26 2.067 0.0488\n```\n\n\n:::\n:::\n\n\nIn SAS, all contrasts must be manually defined, but the syntax is largely similar in both.\n\n```sas\nproc glm data=work.mycsv;\n class drug;\n model post = pre drug / solution;\n estimate 'C vs A' drug -1 1 0;\n estimate 'E vs CA' drug -1 -1 2;\nrun;\n```\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/linear/clipboard-2025781059.png){fig-align='center' width=75%}\n:::\n\n::: {.cell-output-display}\n![](../images/linear/clipboard-1394032862.png){fig-align='center' width=75%}\n:::\n:::\n\n\n#### Results Comparison\n\nProvided below is a detailed comparison of the results obtained from both SAS and R.\n\n##### Sums of Squares\n\n+------------------------+-------------+-------------+-------------+\n| Statistic | R Result | SAS Result | Match |\n+========================+=============+=============+=============+\n| Sum of Square (Type I) | 802.94 | 802.94 | Yes |\n| | | | |\n| | 68.55 | 68.55 | |\n+------------------------+-------------+-------------+-------------+\n| Degrees of Freedom | 1 | 1 | Yes |\n| | | | |\n| | 2 | 2 | |\n+------------------------+-------------+-------------+-------------+\n| Mean Square | 802.94 | 802.94 | Yes |\n| | | | |\n| | 34.28 | 34.28 | |\n+------------------------+-------------+-------------+-------------+\n| F Value | 50.04 | 50.04 | Yes |\n| | | | |\n| | 2.14 | 2.14 | |\n+------------------------+-------------+-------------+-------------+\n| p-value | \\<0.0001 | \\<0.0001 | Yes |\n| | | | |\n| | 0.1384 | 0.1384 | |\n+------------------------+-------------+-------------+-------------+\n\n##### Contrasts\n\n| Statistic | R Result | SAS Result | Match |\n|---------------------------------|----------|------------|-------|\n| ***contrast estimate C vs A*** | 0.109 | 0.109 | Yes |\n| SE | 1.80 | 1.80 | Yes |\n| t-ratio | 0.06 | 0.06 | Yes |\n| p-value | 0.9521 | 0.9521 | Yes |\n| ***contrast estimate E vs CA*** | 6.783 | 6.783 | Yes |\n| SE | 3.28 | 3.28 | Yes |\n| t-ratio | 2.07 | 2.07 | Yes |\n| p-value | 0.0488 | 0.0488 | Yes |\n\nNote, however, that there are some cases where the scale of the parameter estimates between SAS and R is off, though the test statistics and p-values are identical. In these cases, we can adjust the SAS code to include a divisor. As far as we can tell, this difference only occurs when using the predefined Base R contrast methods like `contr.helmert`.\n\n```sas\nproc glm data=work.mycsv;\n class drug;\n model post = pre drug / solution;\n estimate 'C vs A' drug -1 1 0 / divisor = 2;\n estimate 'E vs CA' drug -1 -1 2 / divisor = 6;\nrun;\n```\n\n## Summary and Recommendation\n\nThere were no major differences between the R emmeans package and the SAS PROC GLM step in conducting ANOVA on the clinical trial data. Both are robust software tools that generate mostly same results. Scaling for parameter coefficients need to be handled with care however as contrast estimates between R and S differed by a sign.\n\n## Additional References\n\nProvide references and additional reading materials for both R and SAS documentation related to the analysis.\n\n**R Documentation:**\n\n- `lm` function: \n- `emmeans` package: \n\n**SAS Documentation:**\n\n- `PROC GLM`: ", + "supporting": [ + "r-sas_anova_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_chi-sq/execute-results/html.json b/_freeze/Comp/r-sas_chi-sq/execute-results/html.json index f0649beee..4f95c5b0a 100644 --- a/_freeze/Comp/r-sas_chi-sq/execute-results/html.json +++ b/_freeze/Comp/r-sas_chi-sq/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "b44dc66ec37f1168acc5738171523446", + "hash": "bc6c174b1f2a154f1248b71fdfb86a23", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R/SAS Chi-Squared and Fisher's Exact Comparision\"\nexecute: \n eval: false\n---\n\n# Chi-Squared Test\n\nChi-Squared test is a hypothesis test for independent contingency tables, dependent on rows and column totals. The test assumes:\n\n- observations are independent of each other\n\n- all values are 1 or more and at least 80% of the cells are greater than 5.\n\n- data should be categorical\n\nThe Chi-Squared statistic is found by:\n\n$$\n\\chi^2=\\frac{\\sum(O-E)^2}{E}\n$$\n\nWhere O is the observed and E is the expected.\\\nFor an r x c table (where r is the number of rows and c the number of columns), the Chi-squared distribution's degrees of freedom is (r-1)\\*(c-1). The resultant statistic with correct degrees of freedom follows this distribution when its expected values are aligned with the assumptions of the test, under the null hypothesis. The resultant p value informs the magnitude of disagreement with the null hypothesis and not the magnitude of association\n\nFor this example we will use data about cough symptoms and history of bronchitis.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nbronch <- matrix(c(26, 247, 44, 1002), ncol = 2)\nrow.names(bronch) <- c(\"cough\", \"no cough\")\ncolnames(bronch) <- c(\"bronchitis\", \"no bronchitis\")\nbronch\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n bronchitis no bronchitis\ncough 26 44\nno cough 247 1002\n```\n\n\n:::\n:::\n\n\nTo a chi-squared test in R you will use the following code.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::chisq.test(bronch)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tPearson's Chi-squared test with Yates' continuity correction\n\ndata: bronch\nX-squared = 11.145, df = 1, p-value = 0.0008424\n```\n\n\n:::\n:::\n\n\nTo run a chi-squared test in SAS you used the following code.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc freq data=proj1.bronchitis;\ntables Cough*Bronchitis / chisq;\nrun;\n```\n:::\n\n\nThe result in the \"Chi-Square\" section of the results table in SAS will not match R, in this case it will be 12.1804 with a p-value of 0.0005. This is because by default R does a Yate's continuity adjustment. To change this set `correct` to false.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::chisq.test(bronch, correct = FALSE)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tPearson's Chi-squared test\n\ndata: bronch\nX-squared = 12.18, df = 1, p-value = 0.0004829\n```\n\n\n:::\n:::\n\n\nAlternatively, SAS also produces the correct chi-square value by default. It is the \"Continuity Adj. Chi-Square\" value in the results table.\n\n# Fisher's Exact Test\n\nComparison between the Fisher's Exact Test in both R and SAS shows that the two software match on the p-value and confidence intervals. The odd ratio does not match. The reason the odds ratio does not match is because R uses an \"exact\" odds ratio based on the hypergeomtric distribution, while SAS uses a standard AD/BC odds ratio. Note that R always uses an \"exact\" Fisher test. Therefore, when trying to match SAS, you must use the \"exact\" statement on the PROC FREQ.", - "supporting": [], + "markdown": "---\ntitle: \"R/SAS Chi-Squared and Fisher's Exact Comparision\"\n---\n\n# Chi-Squared Test\n\nChi-Squared test is a hypothesis test for independent contingency tables, dependent on rows and column totals. The test assumes:\n\n- observations are independent of each other\n\n- all values are 1 or more and at least 80% of the cells are greater than 5.\n\n- data should be categorical\n\nThe Chi-Squared statistic is found by:\n\n$$\n\\chi^2=\\frac{\\sum(O-E)^2}{E}\n$$\n\nWhere O is the observed and E is the expected.\\\nFor an r x c table (where r is the number of rows and c the number of columns), the Chi-squared distribution's degrees of freedom is (r-1)\\*(c-1). The resultant statistic with correct degrees of freedom follows this distribution when its expected values are aligned with the assumptions of the test, under the null hypothesis. The resultant p value informs the magnitude of disagreement with the null hypothesis and not the magnitude of association\n\nFor this example we will use data about cough symptoms and history of bronchitis.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nbronch <- matrix(c(26, 247, 44, 1002), ncol = 2)\nrow.names(bronch) <- c(\"cough\", \"no cough\")\ncolnames(bronch) <- c(\"bronchitis\", \"no bronchitis\")\nbronch\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n bronchitis no bronchitis\ncough 26 44\nno cough 247 1002\n```\n\n\n:::\n:::\n\n\nTo a chi-squared test in R you will use the following code.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::chisq.test(bronch)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tPearson's Chi-squared test with Yates' continuity correction\n\ndata: bronch\nX-squared = 11.145, df = 1, p-value = 0.0008424\n```\n\n\n:::\n:::\n\n\nTo run a chi-squared test in SAS you used the following code.\n\n```sas\nproc freq data=proj1.bronchitis;\ntables Cough*Bronchitis / chisq;\nrun;\n```\n\nThe result in the \"Chi-Square\" section of the results table in SAS will not match R, in this case it will be 12.1804 with a p-value of 0.0005. This is because by default R does a Yate's continuity adjustment. To change this set `correct` to false.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::chisq.test(bronch, correct = FALSE)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tPearson's Chi-squared test\n\ndata: bronch\nX-squared = 12.18, df = 1, p-value = 0.0004829\n```\n\n\n:::\n:::\n\n\nAlternatively, SAS also produces the correct chi-square value by default. It is the \"Continuity Adj. Chi-Square\" value in the results table.\n\n# Fisher's Exact Test\n\nComparison between the Fisher's Exact Test in both R and SAS shows that the two software match on the p-value and confidence intervals. The odd ratio does not match. The reason the odds ratio does not match is because R uses an \"exact\" odds ratio based on the hypergeomtric distribution, while SAS uses a standard AD/BC odds ratio. Note that R always uses an \"exact\" Fisher test. Therefore, when trying to match SAS, you must use the \"exact\" statement on the PROC FREQ.", + "supporting": [ + "r-sas_chi-sq_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_friedman/execute-results/html.json b/_freeze/Comp/r-sas_friedman/execute-results/html.json index 8525a8f85..55fa31a73 100644 --- a/_freeze/Comp/r-sas_friedman/execute-results/html.json +++ b/_freeze/Comp/r-sas_friedman/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "bf0d6a74a589870b80a36f17c7982235", + "hash": "c310d2c8900a292c24f3bbe1ff0d3865", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS Non-parametric Analysis - Friedman test\"\nexecute: \n eval: false\n---\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(tidyverse)\nlibrary(rstatix)\nlibrary(ggpubr)\n```\n:::\n\n\n## Data used\n\nFriedman's test is used when you have one within-subjects independent variable with two or more levels and a dependent variable that is not interval and normally distributed (but at least ordinal). To build such unreplicated blocked data, we'll create a data frame called  `df_bp` from random number. In  `df_bp` : dependent variable `bp` is randomly generated; Block: `subjid` ; Group: `time_point`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nset.seed(123)\n\ndf_bp = data.frame(bp = runif(n = 50, 138, 200)) |>\n mutate(\n subjid = as.factor(row_number() %% 5),\n time_point = as.factor((row_number() - 1) %/% 5 + 1)\n )\n\nhead(df_bp)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n bp subjid time_point\n1 155.8298 1 1\n2 186.8749 2 1\n3 163.3566 3 1\n4 192.7471 4 1\n5 196.3090 0 1\n6 140.8245 1 2\n```\n\n\n:::\n:::\n\n\nLet's see distribution of `df_bp`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nggpubr::ggboxplot(df_bp, x = \"time_point\", y = \"bp\", add = \"jitter\")\n```\n:::\n\n\n## Example Code using {rstatix}\n\nIn R, **friedman_test** can be used to compare multiple means of rank in `bp` grouped by `time_point`, stratified by `subjid`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nres.fried <- df_bp |>\n friedman_test(bp ~ time_point | subjid)\nres.fried\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 6\n .y. n statistic df p method \n* \n1 bp 5 10.9 9 0.284 Friedman test\n```\n\n\n:::\n:::\n\n\n## Example Code using {PROC FREQ}\n\nIn SAS, **CMH2** option of PROC FREQ is used to perform Friedman's test.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc freq data=data_bp;\n tables patient*dos*bp / \n cmh2 scores=rank noprint;\nrun;\n```\n:::\n\n\n## Comparison\n\nThe Row Mean Scores Differ statistic of SAS result is compared with statistic of R result, together with *p*-value.\n\n+---------------+----------------------------------------------------------------------------------+---------------------------------------------------------------------------+---------------+\n| Analysis | Supported in R | Supported in SAS | Results Match |\n+===============+==================================================================================+===========================================================================+===============+\n| Friedman Test | Yes | Yes![](../images/friedman/Friedman_SAS.png){fig-align=\"left\" width=\"216\"} | Yes |\n| | | | |\n| | ![](../images/friedman/Friedman_R.png){fig-align=\"left\" width=\"221\" height=\"31\"} | | |\n+---------------+----------------------------------------------------------------------------------+---------------------------------------------------------------------------+---------------+\n\n## Comparison Results from more data\n\nFriedman's chi-suqare approximation varies when the number of blocks or the number of groups in the randomized block design differs. Similar comparison is done when number of block `subjid` ranges from 4 to 20 and number of group `time_point` ranges from 2 to 6. All results yield exact match (Comparison criterion is set to the tenth decimal place).\n\n# Summary and Recommendation\n\nThe R friedman test is comparable to SAS. Comparison between SAS and R show identical results for the datasets tried. The **rstatix** package `friedman_test()` function is very similar to SAS in the output produced.\n\n# References\n\nR `friedman_test()` documentation: \n\nSAS `PROC FREQ` documentation: \n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-24\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P rstatix * 0.7.3 2025-10-18 [?] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::", - "supporting": [], + "markdown": "---\ntitle: \"R vs SAS Non-parametric Analysis - Friedman test\"\n---\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(tidyverse)\nlibrary(rstatix)\nlibrary(ggpubr)\n```\n:::\n\n\n## Data used\n\nFriedman's test is used when you have one within-subjects independent variable with two or more levels and a dependent variable that is not interval and normally distributed (but at least ordinal). To build such unreplicated blocked data, we'll create a data frame called  `df_bp` from random number. In  `df_bp` : dependent variable `bp` is randomly generated; Block: `subjid` ; Group: `time_point`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nset.seed(123)\n\ndf_bp = data.frame(bp = runif(n = 50, 138, 200)) |>\n mutate(\n subjid = as.factor(row_number() %% 5),\n time_point = as.factor((row_number() - 1) %/% 5 + 1)\n )\n\nhead(df_bp)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n bp subjid time_point\n1 155.8298 1 1\n2 186.8749 2 1\n3 163.3566 3 1\n4 192.7471 4 1\n5 196.3090 0 1\n6 140.8245 1 2\n```\n\n\n:::\n:::\n\n\nLet's see distribution of `df_bp`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nggpubr::ggboxplot(df_bp, x = \"time_point\", y = \"bp\", add = \"jitter\")\n```\n\n::: {.cell-output-display}\n![](r-sas_friedman_files/figure-html/unnamed-chunk-2-1.png){width=672}\n:::\n:::\n\n\n## Example Code using {rstatix}\n\nIn R, **friedman_test** can be used to compare multiple means of rank in `bp` grouped by `time_point`, stratified by `subjid`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nres.fried <- df_bp |>\n friedman_test(bp ~ time_point | subjid)\nres.fried\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 6\n .y. n statistic df p method \n* \n1 bp 5 10.9 9 0.284 Friedman test\n```\n\n\n:::\n:::\n\n\n## Example Code using {PROC FREQ}\n\nIn SAS, **CMH2** option of PROC FREQ is used to perform Friedman's test.\n\n```sas\nproc freq data=data_bp;\n tables patient*dos*bp / \n cmh2 scores=rank noprint;\nrun;\n```\n\n## Comparison\n\nThe Row Mean Scores Differ statistic of SAS result is compared with statistic of R result, together with *p*-value.\n\n+---------------+----------------------------------------------------------------------------------+---------------------------------------------------------------------------+---------------+\n| Analysis | Supported in R | Supported in SAS | Results Match |\n+===============+==================================================================================+===========================================================================+===============+\n| Friedman Test | Yes | Yes![](../images/friedman/Friedman_SAS.png){fig-align=\"left\" width=\"216\"} | Yes |\n| | | | |\n| | ![](../images/friedman/Friedman_R.png){fig-align=\"left\" width=\"221\" height=\"31\"} | | |\n+---------------+----------------------------------------------------------------------------------+---------------------------------------------------------------------------+---------------+\n\n## Comparison Results from more data\n\nFriedman's chi-suqare approximation varies when the number of blocks or the number of groups in the randomized block design differs. Similar comparison is done when number of block `subjid` ranges from 4 to 20 and number of group `time_point` ranges from 2 to 6. All results yield exact match (Comparison criterion is set to the tenth decimal place).\n\n# Summary and Recommendation\n\nThe R friedman test is comparable to SAS. Comparison between SAS and R show identical results for the datasets tried. The **rstatix** package `friedman_test()` function is very similar to SAS in the output produced.\n\n# References\n\nR `friedman_test()` documentation: \n\nSAS `PROC FREQ` documentation: \n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P rstatix * 0.7.3 2025-10-18 [?] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::", + "supporting": [ + "r-sas_friedman_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_friedman/figure-html/unnamed-chunk-2-1.png b/_freeze/Comp/r-sas_friedman/figure-html/unnamed-chunk-2-1.png new file mode 100644 index 0000000000000000000000000000000000000000..d5bf94aea00d05d8c02269d8cf74705f5e08051e GIT binary patch literal 42709 zcmc$`1yoh-+Acf|43JPkK}wXC200ETj zx##lT|9{Rm&N*Y8efIeOIo|i(YcJ+}=A2L5_jO;_^{iJ4a`*9ZuizpGf-m{tt`dUa zh#|;{sx!yo8@H2p-@yNzefmJt4nZjW(SMHxPp%jt$W=u0?rjz4n1!Jy?)WB0H7oNh zZ(q{V7G7n%u=nupC)0b2TB)H@s;mW!)I-91b4>q2Sg zrL&TTq76sKWbMKx4f(m>hqdgzZQjp*kExq?DD~jBGggsERD=gW5bqzdtWMZZNC-Es z5cU(U*t65<2eD&}%h)d%5gaP)2b_ee6X=HogxDGT1+n*WLiB_8afQ)i=m!Pl|DchF z#V3eKT+2n+*yPSUGBPqUF=23`z&7{eDUR;7NOnC(TlBbc>uJKECi(}Y{?7kL&-NdA z!GAR(cEibAyF>KVwq1>NsSo>b#nz}zx@w)R>U`1iTzBbw#$;BWR>UTw1!l@yhbGeM zKHWe>#>43XtApJ#%^oKImiQu7?CTTrgtwW#6ZEM&7c3qTM7>-00&nHT<;_%!45!$< z$G;~WNw{Xas~J9y1d|K7929HZdC`*Ie(NS>-m*Uqrt8Mr*}Bz2X6do9v5mQ|i{k4` zLv9Tbti@XA(8d{8-`U}nuhhlI+ne%jPUEedqEUJ1U%#RC)v9lXz9$N7-Cmt>l9JSrZnbo_cHLQl6 zgZjLV$vo7~u_7!jG($nu@rVT)$gJtW^^pVyUR^`&vGiAgfCpKUmI zlo?;Rj3G``h@7opG_4bKpp5g-wVX}<>2)TiuU?lf(8k}`vx1v<9J&u((Fnq=B=L-I zKi_G3Rwa?>12aaJTx~unrR#|v8Hb(yof|a1Y>&6HrH^#4@D1YkKV=+d^et?DIg3#{ z9BDCAb(;8k7|tV3N@a26G7@%FF&+2M@J5Ak;g&767fTBy5}DQT2^4;5wh-6K2shy` zAQE?4>(82w91T)*d;W%w+^@BJu_I-yLB1&8vZLreP7qNwL*;2Isck3oe|3XeJ>xQy zBB24k_!-&BIn_L8%@`^3B*n8dzPFDsILNWATZTIzxB8 zjv+oq$c1Y>!uMnaJ0ZeozwUYvEPD9Oak2j*gWTujn!#}bw9A5pX2>SgCd1XC(#-Sx%JVoiCl*2j$ zf^Ed-&!5?JN_=0uFfMT#uMTo8_v}cP-PfZ;5I_HGc<%L!9(eSD(d8RT`}yXRY?d`_ z8UhE4{d9*WQby(`85hJ!!@|Pyo%KpN`T2h{`VysI*rZe0(b)JzD7xP|Fd#S{0T0_0 z#rg9W;TgocxG7##G-LBnMoi2*I9MLuithDE1j$SJdfYltxZJ(bEzGN!$b?8db-`>S z=9b&$T}=Nb`{gGNk~N`bH0#w9Gczd?Z~O39UPRsJJVoQH!2ix(VbY-MwSd#o{*Q>| z<)@H45<)mg&q+tH)Y(|HeY%GRP+Y=$!ncJd>*X=UMOVrX#ba(K*_a8;4ERL;@QiM| zk-y32+p1!dWtDY7eCDY5Kz^#|=R9V(IODQ-Dy8|i2-Rr=5(JU59sA*TnviPZ)+Gvx zm=7PAZ$n1Y$DLf4Kgfz7akE=)Q4bAdo=+b#UKDh4Z|db^CTTGClWpqn;t@Y{(3L+u z&P~oSDD1>mf7Q3Oq$?|}sR!nyVXnvC=GaevLTVBBgWZkfoD#323MK=A*yeX}kQYCH ze0Mx2ScBKpzKSwdak6#LYwuMRlP*b??Zd>ZjzU><(S8oOL-UFq!zUuLThqq_OUihF zPrGyHAr9irB7WzNdZtG1$B&ac#oFd$`sFJnQu{T6zph2Mifoe|2T+48W>-4ynJ27*D*mz!l>crnAOEZVom zXh7C8bm;P9+8-wWWNWkauZ({8oa2ZXr_?*fwiNkzikedd6pXh?p?k00Q)&I$;YAU| zedc3(`t`!*-Ac>8ru2qFDuEvvKh{EXviMw=UdETQltkG&Ob+DK$nE`TDb-ARoO}L0 zulq*de)$%@xY5X;@ji^|&dA8w2}>ABTrRt*##lk8>R-PMZz_~JF6J84NRgd|!H_?T zGrW9NX#b<`nMX1rrDL!2`v}eF4?adM%4C{@|1KH1C)=m!E#Y z!GE8Kqxe(g}uFrx1grNWt?J$okXw&}jVz0%;8p`JrUOl)Xwe#3(UK|)CAwD0l? zecQXbcDdyJuac!@){kB+lwG-wOpBG6Xl0TcJ-u%2yCESY<@wG8I%p(LpvclHBEZLA z94N{O@A8s=QBY7o$!*+N7sfC&IB3{-3PE&}eQzvZ>A7uAVE@2##m4r%a(L>)wWRxl zX${VAv`b7TelWZ(vKk4JD`1^gD#*+CkPFUj`TMjF3(TTIqdgSdwch_Cca5I$eFfPv7^|u z_Mbs|-fBx|YhuGXm7WK`UR_*SS($8zv>qxcfN(ST3aQe@0!xL@t*r{8qLnbYKU=q% z6;xDI*iARH>Q$5len+ZgzWJe`vp(ObAwn(dp^*hf{7kG|k=0PiR8zEGg}Y|9KB6yz zMOciD#dg!D&XI=1#Kg=F3py>WFZ6LUF-iVWLG-QuIb)(OEFUuskW#nAKF!bt5xe4k znhIH+S)FW%=Cc-YTGD>}`0-?YxS-=gZ(1MUoZ9i0df@|LfnMh!-o%hvULb4ZY{57zrM^~Xf@<|ba)UK7uS9|Ur0S$yYI`R zJEo?l?(Xit14zot%HRX6E*d8`1ZNiJ4J0X#&1s02=H~MhkwPB3o$s!bpL^l!>#LL` zHF<*?A*P47u8q~qH#X`Yj+c~?Sr{n7zz|qkS|-UDJ+c=V`YaiCiCs6dW44{|aJ7LQ zvW8Z%tr0wwQRArueX?^e=6iBP_ww6Jd5k3C8Dz=w^YatSPkiAJ`bB`)j?H$YtWCGH zc+zum*+QqaPvky0<@3F!#ss1)*}xt*q`3LE#QTsaSiU2~7i2&1FzJ-ozcH|`*hxP5 z2oaM)pU>~aX@V%*d1D2Ikdu#Cx|~KlieRiWH?N#~@!jY2NV%JO{R0G-Buh=uYVaFH z4ZU0p8dgFsKrv!FHrJJr9C+{U-6+9VNnE7o@{OD7J*(gCT}3nsgGF|0hlYfspV^F8 z7npTP&BcM_K0ND;WE#pSziFMW@NPudI{p^U>kq1C;(f#G0 zFrWsHA2X|E_7++WK;q>%$lXNrAHfm{!PC{%rIU^b3<_%UY#bQS`XJ<|H&Wr@gc)Y_ z)qqzEp}Aul!p$m&9QEP7kJx^C9w z0&I~cf>4#9@UiNI*peyF2Z(BNA9&IrJ+UvC4VUe#jMqf-Jh>*#(E!h3+#K`%K&CIx zs3w@2T^81{_YWUT)g7vjA3s(eYz}l~s0T?TW@Ka(6*;X#{??=SG+MBrd$+)$6!Iv>P*Ri4m+?5_j*Gm6_MPp`@asLf=F^J7yi`Gp_-B#i1{z zpr8OzH;h4ETU)yy2D;EvD^3SW=KvOH;->vfs|+lKo*X@DA?M{cl!BiFPbRe5+S^my zdOBM5?8GDiPyq^zPbp8d^Ly;B!{Fz|_dy_gb&=t79k$HI6@TV$fBeJ=yrOWva$j|zqGVudmsBgKVBavbm4(9gScgzf<6)Qd-Rek z_QU@j(SXeXP(u7WY(;B5plJlDEr5VWE$qrePhVYEH(Su7Yu=jh3{7SWf0J1#w@Gs# z8TV9gt^paJtE(&E@~@I%^tMUIk+Uc{;wR_sz7JRtGgPWx=DboJL=H1UZ|N?A`+BI< zsdA%ROHxvj-9prJf7?8#ybvNGtcQjR=MY+yjv>BgV`CHVxjP#`azjZ;shyA4tkV>N zt>r*rI47z1k5Nd9P*kuNZJC6IhC;sh@#9CjS{Abn2NH6MF!uyPdKusQujzO%s>_!no5ZxW2jMk72b@5Z#!)=30C*gx=UGC+7W*9C z9BM&Fi^cu|NJ_K>r-*UhzI}WC{Q0~X0l4`=3jkp@wM-4QFo9SrNGbeOPhoW*>}~a! z*=^+v0#;GY)ZkQ?M#NlUM8rORdD4-5nM!iwc)2cSdfI7oL9_1Kv3oV}Dhv?oop9SA zDDP9#a&qS0@h2$ODX{3vdwc!<)X@T>uLcc#c?wB*ZwnJSzo4k7*m6j8=~5v82n+hl zEE*x>ZO{V|Y&yI3H*zMJePDg0q?lw!z(^8{6FXP6QwY1-!~4U-UkOTvxn1tMVb=dO zZSdpO6qw3R^WCyb#E+o@d3}ks{>Y{}kSuI~F6&`|fbGPCLwS5+oXyQm5@wYU!Fy9s zMxsIfeWqgK3yDNdqgBWwu-cmR*pLtc9I7uANBe6nZkTc>2)}{kCQTo@N}VvU00j8? z>qyQa3B-^Y`@Y#E4l?CU%+8WsxDa|E19!EJk?hGgX-P;(aGySoyn-o&cb<%F7S13e z+gZ(q@T*(q%z5ipZ(Ezcpi#Lrx=2U&}@9R{s2k z9=|m8nmI(m)HL;>&I7%xyJP;;Y?^trdt^6mWB`J@=tWZSq=TNCI{x`fW5pjoe=^8K zeSs9U&T3Xr0znz?5=*LD)(60J%IkAo^9NfaN6R)*FJG={{p>W$ZGCh)%KX??FlL{is_upV~}gxQqhiQrsS?12O@8S3kM6 zAH``<4ap*p?3>roVWghtllr}Vr0=HHpz~~d5*qN*6wx^G5>jz6m2kz*R5YN31Be3f zSO6e4H#Q*Yc)w!z3;hkF-`?JC$nPK{)0i*%>-TS?wbHV8@7^^<^G!Q{1sEDk%6YAb zP)Z?9<)&F@ZPx`TT~0%!W@cnGg**z)Izi3?U!IqBcEI%1lySXA(}$Z) zu_tRQKio8D(<+o%A}(IrozEF5agZK9N6u%}c;v&J_~{cmzXnucaMIG!c7$$1ggig; z%>VAE#KeIj>k*i_-rn9XU%rHzJBHuRc(}|37Rl75Z~JSTo6R;7NlF_Feag?Fq(YF! zXE&BZ#NmMT*b>9PwwgULF#&xwV_yQ}DM{VolCERg`V`*85Xk ze}Bo-iqsZU9(o3bXf92NtH(v@fyyL;cO6@65pDTep0YQS7z{~iZf=fQleS^4LtfOy z*?F+oj-jL6VZM7(?zGr(Xc%Hk(jXI;I-q#zNH!zU@!46Ig_bU+2{7V2p#zzN_J5U{DHzr%uGAeS0# zsz*ljp3?%lu>N*D$2nHOVRt-O)R4bGH^*z!AEH}dZ?EA^V-8Xoms@Kn5C)_w^y{1J zM*8d5(o~lo#^I7MFZAUz0E5KHI0MrSf|ha92jG&dI#b^QRaI-lDfshjC#3+C z6`weccrQG~8MbB7_>%Cnf_9-k{06|fW}C*LA&;Lg@RO;#o0^*RIVP@OMu-)mG3$|v z@wQ3sfW_FH0%QQMmOuLW;?bi=eZlxwcv7EE0k8rX>8&+GBjPGn_f(#l~;NEA-ptq!KHgn@#& zZ9JH&mf_*yAs;KCnka!^{2Xn?``R+4^Me$a0x(7bE~|!6hQZAITZS>Kq+W#@3=UN0 zpiNTC(E4f$IRX#xHhS%kr+C8X2jqXFpp4QuVo!M;8WG`mbm&p8tM^eH2kp9XV$$p? zEbe3Z_51Z@crthjog}GnSZVwM0-F#!0o~h9Gn4*>Gv0LDD>c-BxuJfm{8dvUIuQEq z9Wg2C%+gYY{`cc9`6Oo`f5RcR0u<*>)7AtC5Rp!^8@qrovjM&Ds^9?PLeEergcS#Y z9x5*jB+r$!gTuzv5Ja zGe8i)Xg#Jp@0VhQTu`)6`$lE-pGKuQ9{unIrs1pgh)8*LM%P@AuG`|)h*!L@TVYur zuJ2#U0?@%ZdHEJt!NQO=VV${_4|jE`&dSHNy!_i$Ul`F5>CSvV$b?H()QTOig#U~j z_P4wLJ(T*N73ThI@DKB!&)})$;Ll>_EkWDt)n9_IU7*GYU}7sHsc_n*lCim z+&*i6cz6h;2rucmmO|}PN7h@noWDQEfoT^l;E*nr6v{_~Ja~Z)hPQaVDG$LF9yFt| zW29tdV|gtf0(@21^PmH=E(8ZVF4z&kw3djE`k+8dUKAcGb6JD+KQ%V?wrHGz zj&9PD2C03HZE?B~;*rOmW3lpt9B@|?pweC#V_HX_A3?dXDF;{!kS}`x6aXS{baZ6Z zD%{^3um*gIz6Bh(cb@r^!MB<4elV{~Jr7)g?hzON0R=K3%aJdSegbn-+<}cYp$emq z*r3cs(Wsf(Ttn%#YuC!k%9aL;uX~_{E!2lpk4^YD7y8y_+O8KX3tc8B-` z=>sLkIjDGBQaG*zx#;4d=Lg(*Rzx9+B2LjG7N-F1TV97{@$vC!gst*&-CQVk-?1%L z-k<*S`!xi(`t=tTCKMDD1k@r63kxwt0rEFxxxGHRear~{X^g&EO<2!6S+$h7Sb6w{Of_91j6 z(9JhcNtv-z*Xv;8%a<=9At6wmin^}X_VpD*HU^?dF---n2PLr)LZLUUYYqzWLW@47 zrK`wQ2HdjArw$Hi_YN|}y#`)nMHbWD-MzCjgc1;Ig+yp4ynA`vnm<`S{&1-jQy=?B z6tF$+xpR$XIZ{de0JNb{jtkCdGd*|ePFy6m^2~!$IjW(=VLq>hU{;)-KCS&$e%t4i+vdW1o+qK&-_iSty#lVq*wE5a*q}O)NjaGU zc#7^UEnZ$;d~)7Qz$$~l*TZQpjLRH?z}Z2mUmuJ50FZY3vbhm!F9A8hHJzzCX#C)6Xlrhf@o5FsEJFfcGcZ7QaQjjm#f6Z7+K zpg(19QVF}R1G0(WBJ`%fj-dCoX~66#a`T)j#Z;8qO~2uBL87qB4I$=-k*%|#XJut& zWPAW-cp$ZBURE=s zW41BsoEV!8_N~4c!q}WWgB~vf0;EPX%3U3lySq~re_9kH6MrW)qb-be#CV`o5|`0A zZ^OSY@fO&S7U&LR(GEZ*FxRESPJ1)`^S8t{Q(mhs*ucgEnqyKxWHh>l7*aHZnt0pM;g5%#aSo}xh_`gG*{IAo}-)WlgfX{mP92|s% z^0Km7P?7@i1Of)V6C>hH2c;+iY`K(F08CCG>Kz1hcB{#_jqMkDZNR_);4_qy9I1M4 zgTgK`HV{MY=Q^L%h0>uEG=wn+C#OmnXF%b|rC>C^hu|rv$Q{7^%8KHTWhJ9g~YWp-VezXJ)_wZsXNt!C$N647TvXCg~|<9lw=y#;P4 zJ!}jtEEY`SJ<{psdd_@=^PUTrrbr($<5TwPbQ&_WZw zHad3-Sw$DEf<6=idU=MD#m!b6q|beP^5gpe7C^9+4rdNO>YxKyaCEh*0m#U;@29Cr z&%L-@I095F^f*cuA02E>*5V+m=nPfRmx~i(a`KIVARx7LMl1q|Cy7AP6TSPocK#@vNP+TW**4aNO-QZ&B!;2p&pSZ z`P*~PIK%VH!>dTWjJ7^ZidqR0*&a^$^zxYBpPmcSoo_X7eYsVpo7-fp)4t)65t-Hh z^~OZ|VyfW)c#_Jx7h?$}@XfURPA5Q?k|w62id(_)^$U|riRLzW16V@r6fWDc&1qhY zt6}ZRMbb|MEuPj@({7m0$(8a9JKux2_pD<2w$?uFluuh}2q$Ywb~Vh0Hs?^RZu$mA z#vPWjjBWa*4qQGQ{JLNIy`hM0E$xx&Q)(=H@}^)ctYYwIjip@XSeuSE3?UXsF$r#Y zQ$yyfK$0`~$&R7rgVAfcAz?4)EsP2EXL_9>t_t%{8*K@Xtd914I0g@Q%9|4l@6H8^ z8|_KWQ|Sz5^#0Pp8_pfiDP_uilre(&#OeZbq5b{3Croi(i{3BHs_C_)qczlCWl+TO zo&Y!j${x-t^ESY>7O*@V)!cs63Zeb{b&1+oaWINNjSW1zv1qbd#%cU3hDNF(>dLTa z^t=JLa4SB3Chy^FJn*OP;?xV;spX>5Ry`fgv4)UaKJWrt-{l7`m#Z$0xQztKx8z*u zwCs$~D)Dzqk?R}ePVm|}>i@<$SHwhZ8PrLp`g?Hr;L%pSlbMiS1W{q8S5NN1b>(Yn zLGe3s^M01Nqw%5$g2x470;HpQ(yE@G9+175nVB=5fd$1AGzJ{oY!Ekkb5B)?yPZJfAupkRgJSL@&&@{)JiyJnQ;FOveI7c;`5n1K4H90z`r`Nx;Yee2VkdkaRA zikxWfzIt)JhlPCf(|HM1s7b!WyfEZ3w=^L6zRZ25-Z=^zMrTKAmg%0+LR4Xlx)&@o8H~7`IOzb1swcE~u z`KMgqW`T=MPP$5|RhES+xLVltz9mbKWD|~*Y`A|=Otp9zyxW@p3-7g*(8xfW+UU=cNO3qe3d<%q7=S^^x~HeZu9Y41PE) zL*XBtOn#X1fXzu9td8M**0u$_ye%S?x;gSArr&nqoaF><6mf1lx@Y` zqT5MFJ>s#vQv1yPJk=LkD2q1N)*7~*#6i{pzyVJx?6fop^?ZF4K8FslwFgkW@9$3# z9H00NG71EVwb_(-dU|@}L0#Z)Kv)4HQ4W;ztAdln&f#}nf7JW>;7HzJy+6Ofj&Zp^ z{bst^FPGS{c!T>nb6PzvV@4}XDMvypos7Oxds0EnvwKSHv89<8#Z8%Sn(H#9>>eXRZT3GYwUEiElI zHPb2ap11@AZvq0CLF5C$V-v8tTB^eBB~w88lf>IXt-S7AK58Rxj;^z@+8_F_4Okzf zzQ40OX?v&hFFNG4WV?>QW3aFKql~NG9P#(!BnR3O~d}#`L(ACWMS+%WL&E$ zUV5|H%#YN?>3lip8R$%G9EPIS-6PX|Jzh0dB9*|Dnje`1Av(1k=uP(o$eY_Dz(m}n zn*ahxPr-yj)U&ME>oeT_3+*NC(T+D>l?yp{6BYW(c#T$G3h=TosfVw}luZ0|eMobT zvozw+ezH$?bFSS;I>V2sy7Tmw0hjgiZmJvMhWMF}jx38&JA|!J&+Z=C+So9q5}wlF z2zPhgTGC0DdA~3Y;!HX4v^WkDPdu@=B#qD-#Xs_WG2Dem^L0bOA@jq|G)wmvX@BM` zG=U{+Ua^wkXZ)D0(oo?HG-MAdRUd(0jnQPVu21KwBJJU4-yq=%9^aH?Wv~@+D zr}5$b4nQW*PvDJo^^0+aqau>s91}}DB=<7S3j3SC>q0+Wad^@);F{ZyZ)WaCw5#Um4o9Hgh3cyDSO3chc&PxgLd*2U2^R1qI@y%JNJ6%QKUwtiKg| zjogccJ2VLv-S~;)T$Lvi#W4n|DbU1)d3mmdzw7JMfua5W{X3w{+_FBnkHRmw(dEq? zC-*Fv;s-;rBUaxGY7MU~u702sGyo5)bkY3GA?Czn!3I#~21ysHw>=uXzGb>Sl5N+! zp%(jq6IazHxTr)mca)l1!LYcr6w_aDLqK2`WB@>guiv~G&Nm76_xHyZ>I$8jPfw_- z&P~p8%9M`?G(09|PJY|2UlTn2?sAk(CPkJQDlE+Mm?|uOXy`MY3 zMYCS+k&YjqP%od+uRpH^U>%Rdu6jD)9nK!R7h7KL0aO(@YpkuU*`2^0iin+|Vc8WN zBv>@gdQa58IM01{#mj5DL%$)@F?PgMrAsk-eR(eJgxg`D^Cv#sk8DGZQ&*0v$@FHj znK9hfx>YNL&zD5`S2atgYEI$L_GK#N_P)Fx-D7d)jSO#kruK;5z858h;hsL+3Tn`r zPk3bt{1u!gm^d!#GQR6G5=wJFzYNX)a5Ndra!eJ~|GxX%hrGYyi-MLxYstvt;f)>p zo~)GfrI+QKzepLWZw>uEChdA_Z39c_dE3Lf$D`>q(vrnJhw++-3>UQ;aH8e;vy}La zXI^QNxyQyU(Jub`oe|dpnJLlZs!W7O2_1_?ncG&2IeRU-mE^HzTFt++G)>|^CcbdA zxk0l68d#MI4E#SdVNeh&BnK!iMBKNRK|QXE1&(VX@-bdn`g(#+osSGfq0JNF?W2c; zhn3oKcNRNm7V8>D;$j9Ype&gNx?Xt5;w%d@BMrMn*=V`@jWBe*qt+%z#@7Oc^Rc zM;S=-NJ161t~#mbv7!2g8>UkDRS2dW7p*>?hQ9?;@VA4ta(9iMjt)3S?wVZ4*k6Io zPmuc%8mgq=58yf(t&kP;p=6kN6GG|porr>h#R(TG)%*WX;r|q94e$MjCH7Cm#hR?*MeKYkNJ4qE1{5E5v?>YF((>pi zyz;22xC|da#Yk5NSo<%1t$%iKXecWyE1&GCqvIh+WwZ?t)p03ME=5`aoJ!!;(bCj3 zDk)M)Qv!i08ko*sX)tq-p=Fdl4ah&JcnBw6E_9k{9fL}6b$J=(`k>>mW!kUHdhVRg z%V5@-xm*YeR_&@IcsdlmparYOa~KvcFygqDA5QAP$b)_p92^`ef?h6$z5F&PFuZPn z!;id1*|5Kpbv}{;NlP9&1U)uRPIzWym_NP5Cz_UK$Y!%mkY{xlps2y z)F?d=Fg-sXJst>Q0-Q@I1K^F7B^1hl4>a;RuL};Qcu_rcUi^I&#<=isRnVP-uH57r z0*e+(t^!h*ih=PZ28$4aL?pm$0IvfnuSFD`TfGRfNaEik#GaBvP6RZJ1{bU zCn_GijbI@;+L@+4ecB&t-|jT!`%Bs|5%p0X0jo{S_2s8v1-0+v|G#@@{@3)+KYGr8 zp*_)Q_0<*?`V<1ceftKcg#Lj6bwGt#Q{rb3F{OQKAaTT+!F=}zj1XANP6?(J&|l>s zhX5~c3swnY8g$-x#|)J?+@PiXrOzR>*8=<_i}trKFi_CjA$+7r!VLD=CfdNA0&~C> zL|#JBFa@D#h7=D-LDlJwY}<*qM%RZf?f$bR;AUh9z}e zlm(nL=*$=61h_vAD)%HA6t3|G&oGePGSspfZ5;DGbEuGnTiB2ik_1d7ia`3#-MV!P zdPMD0}mmBMAiRB zk9&Z*l5XfT#gTHFaK4HZP!h{ESV6Pr3 z-GboJQ3PJQtF*L+;LrfC!o*lRbSR_Hx8sBfGq{9=TdS+}y~7>C?%Q7=T7!!(42?8M z2ounBj4Vqdm7-t-u9Gw2xdO(hOE9&%#H6I8z>PsK6P3_r+IW5x3~W=kSl}8K5*#YA zM_|d}2bCBk6?8Dbuk@GLCwl&`_rXpmIQ@Sj2?KqopwT-pz-QU7lGl-xoD2?LAetM2nI?b+3`IC*!i%Sa?VXT1tamIRzM9>r60i!jSK7Vb~2I$|G2ZlAnr;0axmwdGy&ydrXwZzB*SrU@p*{Vz$!>Rc)$pz_R>&yCf)B3TLCKkbr<-6zw!js!3boiK=jz%MHh z0bnhIn+s&Yh0e4vfL((hk%ZKzHa0LeK+ZG)0R@DVva&Mlfa-%02dWOdbrMVV5F!DX z5G(On4Uz&H#?z~0(JEX3OOXFLvTN$hpbcbZX1+l8A#FdU`30bAabaOX?j*uk0$GJI z5P%70Y3XqFXBaB^5m3%yFznKB)gMy~g1CTi#-R!e%GvqbNd_rN$=SaAX&_1Ru0)A? zdHUg!!2T?YZc?zP8Vid+zj|v!{7&u%Apu5jLT)9dvQo5X43>oVwIqE!ciV|v*7*~g z6leJo&gBHTK9_xIe*ah0!FTjBq)Hk$p^7|y;b10~?Ed3(;;9zt z4y$@Adw%}8ZEt(n;s`5kZ!q%;{8=VcE-CI$_vQFS72h@DwPnBznHY}Zu%qKSm3fb= zOSE9KVbVy{PR)CrH4*20xhM1MdlLn8=3mLSjTC0a_!sL_{aQzMW{=dgyQlXuu~!o_ z?Hm%6jZHOMnzRRs>!gpzID~1vYuqPaV)i#a6m1l4U-qF4?l>gqcUVj8zv+BJ_?u;lP^gL^OzOc0s|U(I`#0B&&2Ga<-P%dp2O3B@Hh3NK^6*pnBI zKMQF+OXqEWR!893Ri(QEKj6P7@1+G?Rl0Zn_S^Jp_ulGWQ@W;f$GY83@(i6Xv69nt zg_zy>P}zalx&=?s90Km6*y+nl#Z+GJO^lBrwI$k@w>#$}vs3ED>qpW@qUmJ9ZDTRf z)uy_rm zcwlkCjo=FZLOxA#+wr)ezQ#T~X&isS?Wo(Y!)wGf^co%w?cO!I^^ftJXFX=Gl|0X?Zdsge34;x% zaC~Op(!1`o>aTdcn^wf>u3S*DhZ$y3nLVhUY3vcXntzhr`PiIkZ+;X@!@(80$NJX> zQ}xHzD?a2uA`6t)@s&M>{S}X|G=PDB!5^?K1iT+ zaXOxFso!gRVZ)6u@gnVIryf*NT7v2-Y@Rl&SU-8QgLtED^nH+8_K(_cO%9>#@}R0y=c2y>~hib+mptP ze`dxz?aR(TyFx6zQfkGSTg~Qz4Nl-bn3a>nnR|2a#Px<#Y)CwnM(G9{{@;#SsP%YO z>9Uc0`wWv;M$riKlR2}J3)TlY4xW;=b&<_FoD+_kf4f5}8?P+pT@)kU$1OH3dOi9? zHluy_b@AF<&fGz$is85_Xsa{2aW8gqLos=*QYk>K#vE^}44U zd*?51-K`^&scYM#uiP4%RQY&g*EKj|$F7|D@9DqKx%Dj`+|Sf#Pbo9_^NRiXoKtM! z&fxe_#iwQ>k4N)?OB(}nqPr^H#eC5mA7kBibJDR-jgWn57;w3SG%TYSt$WyYa{GKTaax5 z-hrdl+hX{c)56zP$v>-s!zz*YLWhTGoYb)=RaHZ)c(JiQ29BaArUVP_E9~0wCILZw z7zzk*jY47%GPw^D`De3hNb&C>dpNDCp9Kgi;H+T3WDB+XbgUEjvV^RMg#c|8JU>>8 zzl4@eh?)`D!(i*2`^R!T8beK85V3*7#6RW3{2iG{_mJZH(dPzIn52y%FrBunmg~>^SFJuuVreK+_;|GUj*33A=H{zIypG6|fIn zPg1)k19b-!vPr$R-UZll5Q!k6q%1e6At+GZzptSY71}^@Lrtb`5;6b%5A(p|EeM4v z81;mNg)6uJ;Js$!gsptw3YV#YGDGv*(?_rA&x?cg+6dIi2wj{nl-NfUqsh+Bwyr!V zychWmJQ@H?Wm`J4b@qM|iheZ>G=VJ!Ov|-F;ARD!1N&uqhla|5M*+&)j6sR)Z^W7B zgm?|RH$ITI&f3FnTLl)v5N_Bd0uE3pd7r}Vz5eg?eFXWlI{rXp6T4O>S600YVfbEsth_o%2SkoLR4A^?>(v$68!zn#wv zJ_vsTO4Ln(S0nw7aRqW1Q@#sTEo{8#1sxWMIo6?k*vSh1A02!9uRqdO&%|MuWTq^b zOQ9;U1B(T?BJSV4JJAwf2}VLNUOPN}svvkJ$vY%(b{emxAyCWJY zu&E7@!ktCpFBiP_SMGxeH%a}ikqdb5-@jo6KQ>sVLTNPxTGoKAdu)ywNfb%YBY z6`PE#HN$5oCa40o-o5mIt8e${Pynio@Och|&sh0zopD zZ2?A1fQ6;emaH9iKLCPf`l4_t*N>|925grF`V4o+et+0H&B1OcsL>E2($|vY0fY}ZrJ z%vBxqV%Q}D2QEuzo>2qPLD6w%2?(;ai}Q{@f>V%*n{ZA}6u;e+p4S2J3?Xe|u!|Oc zuOQWs)(FbwZs`2>4Diy&3pg-$a7u`-?NbgR=uniq0@Vfu~?{0OP?&cOAW26L|2|YkJtDKv^Ri z^gMrG-!fPpCe1M}J7|so5E zMHYR3rlyju(8{RlB;=$otUwlmF9O1k3b?wUPXaEn&SB>PoO0C;VmfbqQ&W)E68zs* zqQZYn^T+;ymOTG57M}Wt6mEsh6>uhkT=~}zQJp~fO*l;d^+Qzn|6-Q^okqN4`_WE8 z#JFLrBv=j$z<3PQ0(j71dtxrE5co<-@Uq2mY&b{qXjxbmV9!Ar_`>_&!hC1f%=-+* zI;6wZ)m0@^ugAg4Dg<9uNX`!e^37VW>+sl4GsJ$Y{enN zA1rgR15bpM#B=<|c*1!!0s^j(#vv=@<>zOB))Gc93(o9L0+is;7=%4RTgzj2I=DeS z1Y^1@>=+Awb&~w5^H`3+ z!v{M^(_`JkAhWUOm7*GIxUz-UDxmw=AN?%?YY2PECL#yx9Z^xFMyqfx;%)hdplstCusjg3EjJs0 zu@LQe4>yJ-0u)H~LI#R``c`3|Dr~O(aFB;7TWbauo||%SE+gmRSJOwEUECKgU8>H& zSmNRgyMn*Y7-(~|ED$n`t*w3VyIWkh>Mw$+{RP&POYr+2uEwycqzavSuP#-Y$3;g+gHLfXn;JGf!W^27 zLVq10F>&&7R6@s!8(O((3_^|uh(QLw0my?dqKbb{v-+4AtaE?7J*cR9jAKkEB7ocEc#bK-)N2#E(MOA+1Mv2q3ZfY@IUV? zeqEc$=7Lkj(z|1j{pj|F^|ZFI&S_Sa6b(;osutXL^st-4;Yn zqi^Zp8OmXwCAvioRt-$X0T5{52P@#GAHaQNhTX`GL|0Kd8;1M)*FZqp#Eei=QF)?# z_CO)1Y8s0Q=xgfV_>V5>|7=_!e?ui)0{Q^xPg8)u05*XX0+7_q|2aNZxm z^R5zN=#CF?WFLTYfKKm{!qNKqckXaZf_D|5Ap2J*nqM%+AT!=mXvRk31n|gcvJ8EzpWiEv;<+kU6%bZaxTihp!6Xi}^M|9Y;@7m z@3|q5hVUCx=`HOhf38~(e>=F^_B-QlTBx6n^9(vRNWyXa9>txusbIHtk3FvrClM}V zftV{RJw?}by^>Ph-YOjyoS5rQotfACZ066I(fq5HjGf4BDRM=}>@xd=PN7t+cYk=p zC01>&gn<-tHqo2Y!G8npCDWoRa!jU9#SOOc-Gjl2a>|PjJ1s=k=bKEZlK1zH@$aSD}6j(*i^~#K_FF;^R#5`iWmFBz*@9) z8oT?MUE*nU3*V;l}oh#+&ZA zXOUr4I{cf{5hNpxy8Askcc#`PYm$$wSD!IR)Mcv_83{&gGEc8BI4krV;kz4{CK8P3 zJb1HXK4+peta~W&>?>hXOf=tIbpETnRuU^=2K%8e_027BiZI=!9TSDD43sZ}`nGoN zivbqw`6 zcm<{@RSkrmcvic^Ig!^NhfQW$fLZ9w_z=m(=~&4Kzu1w{q>+>YCmLTOrH{Pa(+Ae( z^BAiuR2I6$2X*8+N0!TydzE{e%gUM$teWUqNms4i;PN!Fq1+LVZ^g~O4C)jIrrGvN zF$}8t>r~i0U2h6fD^viDZn7pO(-4yy$ksz(BMG}sNoTX1pA``EB9oW>0(k6C^G|*+ zXc*(vw8n^&VlwqA=3lYUPQUm$Z2f1meaAiMwEUrdFUW?wYIbiV-BN{TtNLY~&WX?9 z7Ls?C|9b6pn?Q9J-*kcH?Az6?5(lsRAZ)SgYF)n34O}5)rS%^+lA&O+DRNmeR_zzG z2yr}gqjx#g)T=agq<4#Yt)@n}QYiHCpNp&IBYggJb>gkYewhd$C5Z3mDmjp+))C#N{0Bv}PtRYQb1qypQZ; zM5AGoA7Y=5Va(yz@I2ms=n~Rh!idjEccm_+3A(pLxy}wIsf^8sblpIwdTaJ^9NcrT zl?tjUv(wKO+ZCR*u#NjPL?(g_ur!aPzW?mor+cGQBfLu@Hmui}>*_j+++&S%oN&{X z8&@jhxduzzZz^6ApYf31O+NB29WMGDtUJhkg<7{_Rl`b+wmbrjJBYWYrropPZ+0z= zz9AICc=XiUpE?FE@@c#54xU&S_1N2q;<6Nd_{G#+8gpblXKP{Km@iMl9MgOXei5+M z<B~+pwJKjssFerG1`2Ugysr)?}Ts4**u~j+Y`0fKjkUv9b*a?SAAYe*AP;j z5%d^8jP=3WEC0iMmVakX{=~@6id=ugN@!SoIjMa7%4^yU$D=`ec`2E^g%-3$q>5Y3 z{j~0k{n4STIHto$aweVX_mNkAX}S8)GwDJ%+OFBc)i*TE|E;~ZimN*MzC}?iFc6TE zZUrPnN+hHkk!~cVMVgI((jiDUsFbwQ-6h=((jnaqcW(Us&p8kGe$K-^_neozUj*Fy zyT7s4Tyu^&##s7I9HNL0hXJiy@*=9_Q5Cg-dE^qkyd`goqu%Z4xP z-K<*4%Uj==RDV_Y3xB_i(wkT588_=vUWfO>C?P+!b2vWe^_;G)ya}w*u-tcVhK#8H zR7qqj6H$}j2^Ou(NgJAy@AM${kM#Vap`ETe8S{ty@=3G^W|h{ADH`Lcs4>C!6-lRs z^u>xj-#W=b+#sh)Q@g^pnn)a4p;nYVONJKgWK!Rh$;5FuH%nMN{tTW<a25YFb9oveOVB3%z;NB>=VyeW$~KE3$kk>jv|hI+b6^`@lCfxr5Cn(oEg zrk6K@Z)0n+Of2b3+)4oc(8u#jUEeFeaK8E{IB$7OkGz|&8W9Cy{9!2-OzGOe$)3{I zz_U89n;2GrNiuR&alV&OxeUJ8$GYYhj81I&J)*BX(fd%HjR|RA;5K zZUAy6I%~w(P%hf8MmAVL^RK%{tw7iPwzVjGedE-!wK=!QaSYdQ;JYzSZ@-+4e4sqBqGQot<_3mTjs()4Q zT%m{l8|{ha-@8h7AA)I;h7#ATpUn-~x)N2tr+q5Yx}jJhFtLzpZ>F3UrRSBnZSult zwlu+;Jx8N_r@wGh^f#kcl}jP&pTD>&6aL%(?nsp3?TQvs&JPHk!RGi1Q+wxJyXMtw za;}rXm3_HNi;aGe6J5)|gfH=z&! z#lS201c0T&`gyM#0LLR`Mw|}aTwlNO^VS(3abhtr+V%7o#2I}9t z%9e%$p11KJ3pdgwqod0J7zYHbPut#ey*u7DE{uT6>?e{{GW-R!T6Cr3u+%gM5G2j{ z!Idcm{HIIv-z#OEOHYEY<~=<18VJrUb5UP!NIL zCna3a+E=Yy0Cp|_ECZWDh4velK1Vu&Kv~euDee{(7B)DJu?K4|S6A13hDcOEfn3jT z|GiRk3=|fKM4+pgo}UHX=ySMX04Ae@3ycIiN(8c`Cxm9cMSQ23B_A@Og^nESbM+su zz5zBH_DHTyqs?Fl{JVMqy(m<=iE>>k;lEr31Gtez80E^$t%Zsjb2H?LijeBmw zNbVNVe2mlWg4q={?31zV?DD^m4UV)kq$xzQ7iZyLqMZbF&DP8PWBmVi%_K=_aI{E{A zR>OeOoIw~`aPE5`6p33;`?%=#M%c zDfnVhgGM0R0&!Ev?q5%+0O>Ib*tHs941gd3VR56szaPB2u3f*r2h( zPNPl|fPRxuoB=q21mD4nYiDPN9#%h~K841_WFY&yy)QqC9_y=4<{3!B^{znI1=2E5 zJvxJQ56BiKKoqQbVYfc63cwY(R~#K(*do~$pv?gEm>3&72?@#e3V#%R4w8Hr^u56C zK>BY1Tmpoj3CQ`lH!9?-oR1(2&xpT=;x_)=_1qcQK0rUT8= z4={Bk3bBm42a7(yE}!mUpmdkP!{uXVA8e?BK@CnuSt^jv=6#ABo)!jbML9T9i5LMj z3FI4&TQkDoc0nzF1%=E5!Vl+%V;e>$ECPa3U@-x#3cxx@S{B}CYs?rsqtxs?+B^(G z0$Ey>qr2^ZgMvRL2@z426r^9h?SS}#tg$r<=ix&&;PMy@UPU3J9~KC@xP^umY{1p` z3o7-;NV7lW0Op1xfriY`U=`A8fm>kLX>>3RB%HS+Dkvyp5qz#LFu1KXpKx)F?4oXV zE!jK=ULv$+j4YFfR`$T$ZK3sxrk&j>Hz8=4gD$}SuKierANg|g1mP_iP|x6YP*`iO z0WEBDauRGLcU7P-0(c}27FI8eV{#!7`hvvLI0r)R9E$qV7AZJHs1jrXteX-F!+{*4 zB@{xEsd@lGE0Mvw0Pk#e1$U^j(o%!xgW%$Ly$<<4AlT5oPWJKrcm?&wGk_v~LHVK& zE)f^g(BHRPRq11e$8h(~9R(W5gCkynn-GL+n9Up$`4E9)z*nX&7&s^ou9GkU-)a{K zKG8j@a7TdZc?pt8{($a9G^ojyRHj{{;M*e=|>dDC|G3d-hF^cPym zealL@T09^I#@p{*u_*(F6zOxDmfZwF6qJHGXg~e!PY3)4C|(r+_5r905fsw?uU<+8 zLjb8>`U2Q7!ozGsM4dCAGYfg>Lr95CFlQE&rf=7T_8ND0rA z3-CIHulygOrmlvuMU~lgUtb))CfiJ>@*v9Sm4 zZeINdppa!kKSA=kQYoO9&|}N45g4ueZ=J`>p4k6gFshMUMV>Z_Dl8cBOxjiB8nrsi zpjH!%FzC~xmI#A}NlS-7DR4k@HOo^iycUsh9{K)4K*T`c#}kPc4G|IaqGq}}AhLv| zqJ5sO6b}y%?6Xqs{4SdRodZbSz~c|XLqI1e{A~a_m00Mx)R{w$AqO%8fFdDwK2iA5 zz0<6zsYxpxQveV%ve}zAZ}2Z(qdbxhNk;Ol9zI(Fz7K>LXr2M{%M6Ax7 z($2u*VYC)XXqiObGYF^hOnAt5%~IQ;xS2UX)&WX#-qX$cMIazjFfe36*Ap5jg=)2M zTe3)yYv^6n8+V3bqLrluh|B^}2ATxl-yw@VLeRjW=nnu=3p3h4mp@`2`pcl9rNf29 zu*MF$i;&0woFN|a{&Wjzto7U)mQ#Qz0W0Ke&Ww@-_5UcGGfKNJJW+NksISqpdQ+td zuYd zj4Q<6UxNtz5d%xh+}1Mqk_DuL5k0MV2$1|$8QDN~0+&inMHLlg3~n_5=*R3g7i_Q2 z%;@GITWUqf#=ct?uuR!L+yv95Pe9pHW`N+)4c#d_@QN8|Zf=IetGSuhe27G93Yl~g zdfg!iiHhQM+I{O@yIGeGPAq_*S3wgz;~FZ8x;2mlk=}mgcG3~^FwB7Il_M5RVd8x? zkBXf94Rj2EeJ#a}MM6>qvNnk1K*!Ys=oL8ObWLq-$gF7f2s-~@8wv;ngiu%{(ua`d z#ebYuvJWN^UEpT~Yq-uC7B+S>s6k!NPoWeoa0l9{px2#fpqIgt<2T|aMew-^+&mz5 zkx%5qcLoFnX+vLCrrBc42AR%|ENpC4n~gjr7Xn!dV4GUG$}qNu{lJA7L=HfuzVMs{ zx-3!}qoOhl{Vo`<&{oigQBVIAT8T)FDEtibLx~~8Yg7F$5c|2fx%(!s@QVXl>I$@7 z&;{4Pf*jtCsy_rx;ppXd$N_Q0z6<{+S(%1=2Oe2$Jo#|dTJ1a4_?Oz2Qov(m$cPgF zM-_;z&_e4xvjrw75IWIUf(@GWsxnnEHD(2S0u_JYn2GMKf|$$f7C`iHe^JPg!ASb+O} zL604zL{Ha(0M&O@nu15FjaVxjZ$^2qSLcC-1iE2)efsc`M#(0`zASyJyVIs~a4Zz_ zw~$Uiwh1H+$l2+sNnc4qAc9qI3LKW!RUsb$%^Mz2%%>I9kkJ5^xR)Wx0GUCV#GG^m z{vUpp3`y3fBP$0Py(zASZ&AlnbhuwltVc$NOuh%=nNCr8zUB zv`!H~K8nIm$P123HqZhCHN-f0O+jMuK;-gDpOL|&g&-T<=?n(Lt#entzW6tSt^j!Z z8Dv>D3{+G?ADK}U{@s~pQ&5Pg5+IJw>`=09{Hs@=)%5hR(Ca~ym5GQ8Mry6jeSs3- zEW`d1*Z)1vyFWBa-zznNJI!`oFf@17$?l=x{QP&T+@LLccy#pN@}O7$Y_Ymg#6w^$ zf8&%20~4Cf|HG->g`ULeT&V2R=PUE`g+Rbz;HAUh_y1QV8X`&SFmIvr%tQpE>|$S#JC)FDC$Pc_%LBO!2A~$pVbz~2!OE$} z0lT}qV1Z=;`MykZ#~l=1sOzFTrDEB$ft?H72C2(V{)#>551^@N54Bns>F&WnB8R0G zuwS5^R)Ym0&cG=0a}1Pr$f{7P`TTUJ3r6)^;h*Eirlt(yD}I;6E&2m!4uhc#>?gJE zISW;(=|IK+ltu8FBnf7eh%e#n38;Ati7y;`v6OG*F2BSp9YvukaFqZG+jSUOyhxXmv zoPM7kU z%j)#>H-0JIl#WOf_bnsnyj}XazsQzQOzvIia$YfIyB*iwz|e4+%d0qqe6^xmUAP zfVKno%)$vQtjdbk2W(YZ)}0Vq?`I$pFpV+vCTy6_-SftO!u|goes1 z(SSz5c$pbxI~589aEEQ%ZHLo=(k)Yp#MBo2>7XBjTnS+X2fKVC3$rCFW7}ch0})(xqJiL0t#_0-aQB@ zNb{nt3-l>SPf==OK0~;H&*d@1MiFq^W$D5_V1w$St92N1i>J>AAiDyc0y;Smv`mXs zlp5YX+g}-CO3K}NAOQNJ-IbvYkjDUdP~t`(#bq60ItWwc-ZX@g~22n0&jz!A<%S`4I~y>Nl$wkt)D%){jaQSMKD zJtZ00SEz9y+f(2sMhOS>#u!KqkXX_ff@c&Yv>7T?^wt14WCcLBPn(3qI*^K>``^Kx z$yG?Spg%|#k#>+0hyVmjLnyDmKM(rbuaTj zvd&LqB-_FR4g0B6%mNrVlY%uxU2v8-Wa{}(mB717lJ?hCSmu9Fsh?!xA&{3MEqj5%w-^Mu$0n$ShJlL#ss*)?0?+>xss4nJl0gy<7%liR zQ@hu`f~*Jz&~r}CAMuis|8hJOYLJY>AHhE8)f$=F|MtaewI5d6QY<_?sI&5b#|G&k z7}qEPeGJM4NEbt?#Mj=!({SU01mW#4+zkLJZGmZ6KnassrOMe6=tY1)XaY9~Cm)#+ zE+DT#tq=;*nwpwLNM&H~0Y<_}M<;QlfIN~XB>it+LlO@n3IJ7~J}iMJ0hL}j(qtG# z1wauunkPVtc>1(=w;eh5Pxj6KQF|gF75XYpPEH=(35+5@qv$WerB|?My3rto@$`D| zfPcQb569srhhY>0J4L;1Vu4@DHlR}NP6qtX$RG>Ga)b<0*!9T$Dh7WjM8#|=BN)f9 zKBVyDBR`D9RAU}IcmR`wvX2#MybR1Z;5Wld5_uph@elweBB2yEc-9T13gEZVUov2R zK7#czdwYNX-tF5f6XnG+2Z^p>U=t09{GvCh%NHu~Y%@5B+sH^dSOL7xfl`kTVA`<9 z@<=N%7`j}XoWR*i0VD{?6S<_pAW>5O9oV-86OcDMboh8p-v1Ja^nW*B?tgi56M>{& z#>M@+DJU^OPGp#Y*bnWJ-Ti$#*yzBPCnD~l z&b0&XF$i{wJtI&EK}!aT2ccY!Jq%em#i)3%wjWzfWnB4@>v+;8AGY7H-#+d8`Iras z!G+=_-SZ!lKLgM!h(73!41b%AB(8`ID>ju4Hxo?3U6gBUJyH9>0{?KNmX=F}fSnAm2~`4Xd;4M=EHa~W;4ebWCe@5w zW|4p{%9P{7C!1a;+_ZO&kIn8KmlN$ZFJ0N`q>atln5Mt6cev?ALq?>QE^i-%=nWijte;xE>O=S3;VO4j zMk6}1UTQ6tTn25}QN5@(k!uE0_SLgpk{?tGEdE)u;sao>2%_KclRocEwF?`+nT&9+sTMCBV~WZ*}53Vz|w zSXiM601^B#E?P?jfaefjVN|PDAl)k7V(!E$6QL7 zmRw3$RVhFFC3ow?R%fMpNu`4l-L}XdevJ70UgfcaBvN^1`fqz7^8?fUnDxivuxDM| zFl2r(7>i{s2^sB&A9Z278x?q_Gjl(GT z9!w*b)$UH>Lk$NfeUpo>drfF*%DgHsN`03tkDiWKxo`{P3iRU9YcZTz3{>%A@jY zy@H-2;3}E%SNSImnDHUWx7quvs2TxWMZEj>zYgt~k)x=5v4z9eE*#u+aitcXvwUj2 z^}|H9%#^C_!sh+sdc290&-i)!AAZA`C<#Z*U%!ou>>fjR#rP(&6%7Rlh{weVF4P~O zkZ}N!gs|R~nBCz>+I_C){on7?=h3cDLhcKQG3GGBKQa~hE2Pl4m!&e|9=dSXk&X8rt^FdOkpZF(IalN()dm`^ zStt}Yk$WqGILp|{5Kuh;h5(Li6}pNt_NMSU(3)IbaxLWPeawGp(U+GP$vB>PFPo?hYJ*b7sBjt3%-xP-)0 zp3tbM5&*FQL88Nk>$%DCMO$CH&}HH@1U9pMbJCT15Lo7sbP@OcT016aBXGFyaf)K+R)TwXutqlJN1gA)Z9=t|A;4Gv@sQR z<>t+^{KzhGrxVwpVd!9xAFcRZ_F4ajulq%y!U?cJzBa=+m)_{iCPCx5($xhAs{(=K2(V+{ddbpMPP8UH zVp#mWgjB;`W-$phzW(oTViOry#nQ93`P!7UrTt5LTWPlPo>dAHW|78~mYLJonn?k09sz$SbD759^)nfHjdSI#bjc6Z zky4jC932ms3>L)TX%oY`Vx|JK9Lg5pcN9ZTRrxLhfCs4VGoT3q3dWkh;&}|2@wGHC z`g0&r7w~}$DOEtQx8%MFctW74fW8J8G$8ZtOyQ5pw=NN0 zlrMxYP+>&F^SvmvPrM?hVNU3l$eMq#)D-V7MM_-PwY2 z^`BS;#qACmbk#0j|NkG6x8Z-hA#Qcu*H0JWnIe5ay>gIM9K7zV7XNXP&m71-ccKx0 z-KdD#B_t|}0p=JmenBMTm4nQIkndxH9W+Nrj|a4a$aiyn%gaxoawB~9R%c1z-y8TN z-rYPe^a~2=>gZsz!EwrmIu^ntwjAIOAnLMQdLQsgK|uj(K6CDkW%WCEtW=A|X9<~o%9S}4wKy`Y*2M{4h`1Rm8-9Y~V{K}^V zBoBaVcriBLN=ZpMIy%B}`LHth1iU-y2lc{6i!DiUaD;^bA$Z%H?E9^aMF3EiNdRXs zkvbY^aUeekQ(zsMiWJ{rj6?0*-#xz+>g|2AIeez*`9e*#5>js5fRTEz;&s$8j?{IgbMkR3qYh+^6AA2i>8J~cMHz+NI?#)MM<9YpJ0ogX*}4Y z>Fu=NeF6g2Lr-5{G_?|n2l1M~Xk!bPFsRWD2!EZ2=(KKd5V-r8 zFz0VqGjniU1Qj!T{nKm#`Qd*y+;uRyA8~^hOdVS<^4gcw0z%cY8O`3Xva$*e=@vk( z9v&VpEF6(|<;(;&&=W}_rvPFL{8t=kgl}`BgQN!e3-Zg=5BaLhUN_3og?m1c62`^7 zeg?Z3M90Pw;;8Op?Ph|K*0cNcjza)&g^(||J5x_-|Le#9 z`Roi+eyOG*Wh3=I@Y~A2PelusIB;EactrZxxDUa`^TH%>^s}q}ak6w-g>6s6mF&NW z@=^5+g$TP&wpbgCBi*&=5@*3kL!xaxL1mZc1+lJojS!OI>184Aq7#E>P3>b36SCNB zj7207_Qtg82`u?vrANGSg!iAlAw@hMmVBHc%bU$&6EE_W?EEy9_&cbrj5#Wt*R5;H@#-lJU!hwZK@W#zvfvdS5_Al8%4}b4A-XQayx3! z{11fmzbfttNmYUbT=eC~DvR*!_I`F5tv$7yvS&v%nncU3S-LBhT7gT1D*FA1bJVs>N z?%_x+k56YN1@+|3hmjUq3&WEP|G@s{{Fru z?1Pm^1%h6y^yc+PjI1`bb@i$t-`bdIUT@b1A?}dFhEg1@58S4e z^=h8?iyj*Eu<2CtILBOVA}mu<4GGx~+~6{s;Q1^RP`qz4QrM?HdC%|&tyth-od-LV z@pRY}_KTKS!;Pwr@Du>7uk#x&b0TRa4d481rH&hj_o{Cvl&6=_e;CzGES(CRNAcm4Wn+*cWqyzpC$PNe)uQt8#wz{=4a` zD&3(i*my%{iT)Ouyl!WGpJ@|EzUy-|RQsttL8-BmkMy!a4Ge$p{lSlx*H>RZBc46Y z;-sMHd{dL!T#})Vd$eeDNQaMUb#KimQHsIImi=Hdm7GnQ0Fx!(@HVBDwY2Ng`qTk= z9&XfT;=}E>BkP}Rl)4KqW1rs9^bhVGR~sIW!^AYgJN*5vWHIUew%&d~wq=u#PdaL~ zps3u-hW>iJ3Y-9k#h0-p1Y70>=u}M?lS6@&6sf!TKW9V?gQXo0itimLvQX$QWJioQ z%4o0)w6|KY6JfG=n&xhFr9BvtwNA2=-p*}+Z|O515L~Uz&4791qc+$RgNd2{7LxuX zqw-%-SnQdK(>a#=gR7Y$&+p&8LLwpT!^dggP{Y_^!j)0_UAX~sYEM8)WuoH~e9YoR zP=Cv6ysrL_6Rw>E%}4(WZ%TpXc~aupI&4g%p@RdDnMnS;bCR`M>^JSHyuQYwO@j(g zbQi1f zbh1>E<8Jj@{0&}xWeP^6NZ+WRdH!(f%C;@q#*GhIILzKlU&jp2`|YgZMHhP77h35^ zq1*agzTS_t-=n0%J!4y_L7C8%q6=m3x43GZ@Z^>i$qP&t+nQ5Creqs(LtA_9Kp)QX zmt(K#%beV6DC}O*y`G7d&g9tOBi`Z^ZFs9v*~V>6#h^|xV^-#JsP%b9Fo!epCAPb! z1BJNta?9^$&R0{Y$M16QsF3MKtlijq$H4ojufnl-Fr1SDYkj=TboKVv3X<2`RYBfo z??vnB?9Uw>Q14LoW30I&1anYY70{!u0U;^r8inDY-MEL_Onm`Y+Sll$6?$KfDB{@q?K`i&1%#1n%q33l zFA9G2$zSGu6P3H2NXRwxGW162wvcXX?Gx{SZ8L&2`B!x99FfXsN7ln3`WF6~AN#S1@2s{5E5OVg=? zdsioWTjP4LL$=?)|H*H8hL6F<+-0h&2;LRF-PK64avj zz1REp+JL+FEb?yU8QmoI*N8U-B3=eADr97$MzG;KlQ)D%`iogy-S5BEI8Cgp> z%YA2b{)_F_<}pQz!Ccf2g%_C$9hDyWxQ}URl_@B6Q!{4TEb6LX9e+4AS~~gEz%ahD z{P#4O)_$GJSE!-fu`J1+fr(||0E2Jz0sUrYE#K*Gk>rAaAf7%$f!x9O0-*-P;O1OP zR@7};8Lg_wFVxXvihCyg--UV7T-CplXfa;A9&U8@6_d+X^v}Tmap*L!8OOK2c%s*f;m&FeLBx@7CqB6nd_w zjo!CnFJdPWMiXdXYmwS6c8uTqEty{L8D@227{D)t{)fgzsciI5vRt26YxvFW#sD96 z3{Q*6CcEm}7jLgj;-g=14dpLXtm3?$`Bks{OsHXGD049YyRBE(jxClwQ?*x#Wov54 zKX5_3LF(@vBA@BW`Q*xK=Z1!A63L2_6UpigWl!H|ubh^*bHWp@R9Gym14&PkVwf}# z?BAwFh@Um7I@uF)gQfI2uS1{!0dnNciwU*&mq(<1@@ZI&+7T7B6ylS?YhRmizN&V; z|EJ>lO76#}_bDP}G;G)s+hb&+PXccTTFuv#?h^Ls*)0_bO`rSuYVQtn6id$=tvxo- zPv`2T%3ud$GTlE(bJ;4~ESrmhg8ckV9DhQYos#)@y~9Tzst)%P7PY*W+$*2o zZ445_DjBw(kn=pBD}R6U=lmnB2kVn%nbg-DP?*6%t!B~so}suu==!4}21o zm~zJ-9YeLpANfYx+qtqn+U+itM&R1i-EGgiulrs18pX_N#`9lM?%F|TOJh6-`3V@T zhBPu)1bI&^I$Kir#iE?3sm&6&Hf)L-lu6m`1DKAdwx{DhMVkBiV8mD-|8Bed!s*uoSPkGWY1cOLADOF4$nVPP6=%>6idnbE+lWIOj*>p_0vi{F$_i{uS=gHX1= ze&&0@`gSwpM9r78c``q>khu(v=LQ*#+1ZbJk8UGbN<)drPF7`x*ujyB;j-f)2By=o zFI_T*U-AaM=@;w^g#G&I*WAm&wj*BGuT|Rn!~U|*WQU2OoT)sC22a`EV(IN-4OzTC zMn;lTm%V(1=yFW$T_)OjeFI6zn?lZsdbiVvdq!OooFuPrr9}oABTiRi1T$}zC4JnZ zIhh-0I+wzYwjEmTQ`1EeKVm6Y%0K5_Va;3lvS&}KJ)E~x7eO)hAm=6DnmX(EKO!lDa}-RNg)VX8+HZ6II9k4qt!QX9&FsC zGG9&L3SHO;swL4-@+%{CIm;>k6ddL9-l?3O(q-J9jh#|EHWAZ)cGK^7{MiX(dHlu2 z-L9SCq`eF>4&^UktRy8F*O~n6aRtYP+18_^PsHd`(y1Suot8II5)COM;-z~fq?UUf zJFDt%kLtbAI-44ACi&go)+QtCS4vQ3-tz-j2Hm_j#TJ{p;PkC)$kyILu4kBwvtf(9 z@=izZQedTEneHk)3D`d3!jXEA zkR9KkXu@@}G}4iH{pR4ipUfHOb7PZP_SUq)4!Pf74?3?qc zQZ2Y#1inUp%VfDRlSE9ZW_z@ododp(oXwkfW;Z8dre^U{OXPGdl+f_zP`=?228L@N zH?uvnu8lh82eUx?|s)qT+-}{FInMHy(~KY zxo?#B+uyzks&_koCxI<5Ul?AQX{poQpdFZ?N<7yd8ix1F`Sg_a`o;O|=GEojAJgky z{2r0*6!WSc`Brxt?Y-pMG8`-dZ+49#5P@R{_H##Vs^=SxmkHvTs=1wAprjD2GkA+Mz{+ipz#Y80Hm)rGk{#l#u>e`~Pd%*f7V zFf)}}xgb^t9dUu#7DRfQ^(&nWGF*!{ft6~BoYvN?B0{e%EVR&geul-$m-t3U9>$>% z)SqA5yg}zP$y%v75kZ=bCW}Lzy1)9=ygtQ8zWR=9+r6D9^dWUT4xK7|)R<^;2z%{; zhh-!qzq!vFe<&Ul>W`fIE6NkHbia37b|d1 zcJzHt6AE@PQeMRBEbEBZG~;T;ZRihE$cmEFit*1>)@+aT$z`9KP{J=x!WfyxvoMBC zWVt&fVO_C|>!5SuuQVD@E~YM%rXO~!-jpEMy9LJ2<&}Mho*TI1l^?XpC?*RdI8sBG zaW)kDBx@-OU%za3=M1j!+o*WGUmrz_h55L`IYaLj4~hcA+S!16xin80nOI!mA8$fN zt;y%a%bK#uae{qnH=Y57vvH{djry~b0g*PRQ-tfWwi}0U7!I)u_;ao9mX<%JzBhpz zN=oZ5)Ig_b#lInEQh~^RXaa-FNlo1J*6GE=77glH3C>`;v{P2X;>jVbt%OSV$)|O! z=BQt;Ws8LQ_^7Idm$bhyGB%Jakxr^*4nhBl#)lcfh-DRiin^&Fbf!C4WjpRb(K}vt zKOt!aXEYNp*My3~a9O>%{+2hNj`vjucOML;fP?}LIjYW_j&b+$)Whj3Oim39nM3bp z_C?TQR`BYqMU@9^lSC!_O5z`3VQLR#CnGFhQG|}Z423Pkd2zyCcV%lkGRTk#cWBL? z96s_Do0iAh$}4yu)NaOSrk&G(kiA#=i;RD$l5K4b`y1L-O!*TIOOF={PKF)a11lugsRzASWzhQI=+duZ@4AoT$u2IP|tPXw&D7!Nt;~7 zqlWRo_V35xHkD%5i-SBLeWKF+Gf^1@!(;YGLtH4&9jM+FRYt5Ps8%p_@>Du5qJH$r z?60yXloV8W+y3D0^Pk!r~t>rd&Lt z_w`YD8Jl-kGXS%No2BZZtWeyC4cnk=g35XxPxtTG&iMG`uN5j8p)Pv5&KtG*_}CMi z*{_@CcKtBka2`wkS-e!v($?MB617=P{X~iY(<<=8VB^xTf8*hkT17{w`A)KA?`yky zitsyEu@G!x=)h5pjApXh|BQQUvv@O)^+h(ZDQ%e=4{=^v&VnGeEg%)8R-+vM#7qkV{xyZ&}RpqGcT4{ zl#2J^4d-mcN-r~9CVsoX4Ss=Dg`rn{48#l_P&9Lj%_=iUxH&UfqtDo{ca`;_J zD~!9`=|9Pn(&f!f;@|KfLC~mT#Ve;wqd@t9H~_~d%0P$P6)s(G#9$`MDK#K@^=H?~ z9Zk%H{h*YG@7FB||5&#tic_$74t4)N%lw-32z7;#^p4H6Z^PkTJ+E^I>a!!~*b0O? z@9M$ttvkM))8F6Avw*)ccOn}gKx=X50t}y?1_AhnT(v8>}RFw}bm$tWA2|{sM zFX9F|A9-I_eDv6Hqs5Jve#4kbH_j8mpSFL_7N31oF?XTKn|Glizdi66vyL6SnEUP}HF|1uvBAPvLiC-FYx<9!Dc`JvpIC!K?8EHxf`gqr zjwfP{@AeKqCUJzIL-^=%IIgO+ zLl{es$(qgOeBx^`@s7MOOp8rpPV1=1I+lR8PpE06f0@-xY|-f+!}3BfpOSfHTyl6+ z*>pc@^Fq;2_Ra;4-TNqC*zSBAN;|C-T+=$56CN-UO#OB{BE6!VQ1sc)1)GDV_PqV_ zw)(!$?`y(8zPVM&J3leDZ$)o}t%+cn&06t1wY$ypz7#EJU)FN@M|V$))cm0Bdb<8D z_B;(u)~cBjXQ#t!m5-25J=b+Q%-uLv3wu|h(j8ToOi)L&3zQzbOPpJA=c-7|(Lkbx>b{_?#0vREP_cte_FkR9VHFYzJ4R!-OCIx2%C~N`ioPrsp9?SWH5IuZ)GZa zjActax;-y1-+q>T;@CR?zq{n=z4iHK#kCg)ONCXj$~nh+i#?fgR=-F5BoxO@IaY=j zD|^V}U$D$cVZ85(FM;vEqNvHIy<2hIl=Uz`#XjYr#GHP}jPKoyb3=V{fpbduhk=JK zPq_CTT^-(88_5esK0++qZ8F2KEVN$6$h0RZy?^vbFwgFN?&MeV>I3iUKS?=HBI>er zG{Wy$2Wl*DHRl!=JG{QV@CO3Kk~aF zIWzG3as>+3e=manV}Crtq*yqLWo~*fDc@;wM>Amc>mG3EQXhDMW)u{g6e1T#d&M`g zg(g{-{C}^}Lu&@Lahmliirdgld!#^UQbl*BRJsi|!6wC z9_45{uFUmU>tofqEEB}Eik8}29=P(yL`Cp_bKa^$%lj6@g39E||NeaE8{8Xat3bBY{;4D>6IzF7NBve=YfLcA1q*6bbV;^U`e^Cxb+0nxKnCt zfk4mip?nu4I1M)?)uM3KMR^eNUkLQt3y$m$JXgmEX$o?+Mg}_vUmNHe$`6zq{|Smr z6nAWv%~l#KJAP&{ zftVX|)e!6dyL{J_seM!KX_GR6)ervGieDB2k!3Q25x9ml5{lHcU0%dq?HV|^7a=&a zjk|>0cry6i8G)kV5+>oL=89d%=I*plMs2y}n{nJ+&3S_JEVC24O9SPp=_L!(dmF|F z3uQ82$}4*`;yG4d>=VGvULwA1ui+CuKIyGsq~ybpMm!wytSC4{EZQ+XuXa@)KE}=* zo4hEKz2N%h+qG`7$ajnH=V|LVVsAXsKRh`_%eSkgvGYwwzfSWdNISW3ykMVsZPt*2 z?V5dR@uo$Vu*$VrG`o6KK1jy6$FgT&HO*D3MON0QEablJe9Li)q-gf=)`u@A{1Cl_yf=2nP0RL-X)zXPe~$=})4+`bW8ntEB;x75tRu(yo1%uWB3|>d!t4#Rh?Aw`tus+e z8a$u$NLPaKmkNohqmAp1p07uHey^Q4yRs@ni(vhVX8E;a&gg|EuwFl?IU0`pbeNQn zuwmf5|2)@oJ~Nz*W-WfVGku3Zf344a^Q+9~afN6W4gveqsueXqvKQwkp& zoYW-298*3hFf+yBAtJg?Z)%j=vCCDxX_??Iq+Tp-=drAQ=h5D#2;~JIpXedNUn~@v zEi^1U3uWJ`An9)ZK0hOt?J{Zn=0hhBZaiZ9^sfK%kkZ|pZ@NMa4ne^;dm9#0E_j;c zYa#7sOZZ8j(GnZ-e%+i{^A9vxC#%pjw1&tQ**eTjBH$jwIf{Z4RYy-FoWw3 zLj7YX)Fnx^BN*YYK@n}XiHb_k8LC(hd)vaj?H2rD*w zKApr4b?`ixb2(NSs@Kk^>>N<}HjLPwo#&>wo8S_2buu{rIkR2l$YvlASAFB}${C$j zJoACpVuZ8j&;Ah?dVU6Bp5ehMyHWrZ`EgLA`pmiOJH&W|@XGfBs1XnsyNmWbJbw2s zIClOX%m#WA(^*j&uV-oH(H|)e)$19avvM#juNl6cL(mNTxiHZuzK)58LHgoB>k$cD zLFwQ!HjXT|vVL>co3b?!?I(e%+&mX~e{y?%KZ-F74- z6tkHhxlK(mnIa!~T(&rsd9|#laNZ|*wQbR|LLcXZB5Y6X#_-Y)H|0m~Mq&1Qc`7Qw z7znsGtA)iD-m0u_DdX^9M~sSs#Zj7LASSX}2`iB*%|~Bz-J^}`s6Qi_wHo`e^WVYO zuL{Tv8=6ujlXbbatd=QwD!H`km=v5&6c!(gKforTz9p2qIat~n7#`ez1!d~92kHa( znk5RuR3o2@^E`pqE?VcAoh+^dhHs4J1YBqhRe#4A2~im)r}zu$8j3_5?s~UNo|Dr{ z2KrD*i-@deDz@N}euhJAiqv2tDf%|ku+2871oi_q5%Xcr46i`0O<`tM+MvklnE|m8 z)e-7Ov&Xdb>VUMRcMN2S4M*m&=2Q0)pQgmEn)&+RTUzAMJ)VcftOfOw7PM5SXq8hZ zmqljCGWL9LC6f8JR#(HJPs_7(RRc{qbwWqV*OEd~^Go_|n8A+Wbc00HexnG?gMzsq z)(#c>T5}iByHL!Tam$Ah_=W3o9YxV}$fY?CBnpml_}h?;6%o}{TB6%p0=!`v4X5Yw zOs)#8+?9))*zpHbx9jZJK9JoH&U@*W=;tFg^3#|STZF2Ch}S{T-t%bj-~tL=mO^cQ zw3Trf=Pft>ow9y!fu*c+f*4{aGTQV!jgriCJ)(atAN5 z>&bcXdhWUN5%CjD`J)4{`oP3wISH&(k71=)%bx#Z_L-jdjkfd8_1HqWtI+TqO3My+ zp@ZSUjQ?+ONKARAcxHuH_LZT>}GxW|H; z;3L*reZJ;a%cm`aIB!nQQeW{th_ZR4NHr68?0TB8*tdh%$>z{IX7Pe@Yby}L-H^!V z$)j!m?eBJh8?4LxeqvIpv+hCeM05@}D&Fuy;&v!oYsQRRqlWsL^Gz+Y5({feW!dCY zM9X=liL}S&?W+cd!*?*Q2l7P4Axzg^Re$(XxH0)HIPHn#=PJxH#73jXRbTuKAeUoo zlrE*ybPg7p)Fws6>!H>u*{t^DxF{-iCap9dkd$q6YE$S+1lhZ)b+@fMJ}CanX@9g7 z%HtZ?BFkO=HFje^f~WlEF=q^{DV)C$vMM%3P{I&OsqPtkjdnq2=&VTu2ZYneO9o5D zuLlFDr}WkwGmW3P)eSw&$R`mJD!D@s?}fARjk|b|vg>Rwv}AosUO5@m_@k^(Z={OY zoRMzh-RvJby0*bsKd=qCo@x_4KWqtNAh&3T(|pL=w^1_<{H&=CgAs$;Kx#pCg<@9u zO`3%wMiNB{=P5@|mRyeavjbzz%I$ZSXBOx6GuZ+C&6qh;iX>0W=CWFuDZ}`?WW0YO zuQ6YgAE?P!s(U=8RK>WDo0=LCKCx2dXmp&X$HA0^KbNA%;yhWpQv5AAvZ$SOjH!F5 zqgDcvy|pE+;=QlF{`Q*woV$+nC+Oz+9G7YA#Vs1~s9Yp_%qyjmImsg55mU{eKDVB! zy?@LDnZ74|`j6ZTOoiN#0TcNmL%jv!)yvm^{&@VQ`h@O5sk-ik{wmujYx+kf~^vv6S9 decimal position) in results obtained with PROC GENMOD and PROC GEE. Similarly, differences in later decimals may be found across R functions.\n\n# OUTCOME WITH MORE THAN TWO CATEGORIES\n\nModels with cumulative logit link functions apply to ordinal data and generalized logit models are fit to nominal data.\n\nIn SAS, similar syntax used for GEE models can be applied by specifying a multinomial distribution and selecting the appropriate link function. In R, the `multgee`package provides two functions for estimating GEE models when the outcome has more than two categories: `ordLORgee` for ordinal variables and `nomLORgee` for nominal variables.\n\n| **Procedure/Function** | multgee:ordLORgee | multgee:nomLORgee | PROC GEE/PROC GENMOD | PROC GEE/GENMOD |\n|---------------|---------------|---------------|---------------|---------------|\n| Correlation matrix (default) | Exchangeable | Independence | Independence | |\n| Correlation matrix (options) | independence, uniform, exchangeable, time.exch, fixed | independence, exchangeable, RC or fixed | Independence^(1)^ | |\n| Link function | clogit (implicit - not user configurable) | glogit (implicit - not user configurable) | glogit, glogit^(2)^ | |\n| Model-based (naive) SE | No | By default | `modelse` option in `repeated` statement | |\n\n(1) For multinomial responses, SAS limits the correlation matrix type to `independent,` so other correlation matrix options are not supported.\n\n(2) Generalized logit is available in PROC GEE, but not in PROC GENMOD.\n\nThe same data analyzed in the corresponding section for SAS and R are analyzed to compare results using equivalent settings, i.e.: specifying 'independence' correlation matrix in R to match SAS.\n\nSAS results were stored in a dataset and reformatted to align with R output in terms of order and decimal precision, making visual comparison easier.\n\n### Ordinal variable - Example\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc genmod data=resp ;\nclass trtp(ref=\"P\") avisitn(ref='1') usubjid ; \nmodel respord=trtp avisitn trtp*avisitn / dist=multinomial link=cumlogit ; \nrepeated subject=usubjid /corr=ind; \nlsmeans trtp*avisitn/cl exp ilink oddsratio diff;\nods output GEEEmpPEst=GEEEmpPEst_ord ;\nrun;\n```\n:::\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/gee/r_sas_1_ordinal_variable.png){fig-align='center' width=50%}\n:::\n:::\n\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel <- multgee::ordLORgee(formula = respord ~ trtp + avisitn + trtp*avisitn,\n data = resp,\n id = usubjid,\n repeated = avisitn,\n LORstr = \"independence\")\nsummary(model)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nGEE FOR ORDINAL MULTINOMIAL RESPONSES \nversion 1.6.0 modified 2017-07-10 \n\nLink : Cumulative logit \n\nLocal Odds Ratios:\nStructure: independence\n\ncall:\nmultgee::ordLORgee(formula = respord ~ trtp + avisitn + trtp * \n avisitn, data = resp, id = usubjid, repeated = avisitn, LORstr = \"independence\")\n\nSummary of residuals:\n Min. 1st Qu. Median Mean 3rd Qu. Max. \n-0.4698929 -0.3568933 -0.2853896 0.0002563 0.6339985 0.7277428 \n\nNumber of Iterations: 1 \n\nCoefficients:\n Estimate san.se san.z Pr(>|san.z|) \nbeta10 -0.55049 0.24303 -2.2652 0.02350 *\nbeta20 0.62726 0.25615 2.4488 0.01433 *\ntrtpA 0.42992 0.34205 1.2569 0.20880 \navisitn2 0.00108 0.36663 0.0030 0.99765 \navisitn3 0.11733 0.32268 0.3636 0.71616 \navisitn4 -0.15048 0.33263 -0.4524 0.65099 \ntrtpA:avisitn2 -0.27197 0.50959 -0.5337 0.59355 \ntrtpA:avisitn3 -0.58563 0.46208 -1.2674 0.20503 \ntrtpA:avisitn4 -0.31782 0.44813 -0.7092 0.47819 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\nLocal Odds Ratios Estimates:\n [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]\n[1,] 0 0 1 1 1 1 1 1\n[2,] 0 0 1 1 1 1 1 1\n[3,] 1 1 0 0 1 1 1 1\n[4,] 1 1 0 0 1 1 1 1\n[5,] 1 1 1 1 0 0 1 1\n[6,] 1 1 1 1 0 0 1 1\n[7,] 1 1 1 1 1 1 0 0\n[8,] 1 1 1 1 1 1 0 0\n\np-value of Null model: 0.68672 \n```\n\n\n:::\n:::\n\n\n### Nominal variable - Example\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc gee data=resp ;\nclass trtp(ref=\"P\") avisitn(ref='1') usubjid respnom(ref='Lung'); \nmodel respnom=trtp avisitn trtp*avisitn/ dist=multinomial link=glogit ; \nrepeated subject=usubjid /corr=ind /*modelse*/; \nlsmeans trtp*avisitn/cl exp ilink oddsratio diff;\nods output GEEEmpPEst=GEEEmpPEst_nom ;\nrun;\n```\n:::\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/gee/r_sas_2_nominal_variable.png){fig-align='center' width=50%}\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel <- multgee::nomLORgee(formula = respnom ~ trtp + avisitn + trtp*avisitn,\n data = resp,\n id = usubjid,\n repeated = avisitn,\n LORstr = \"independence\")\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nWarning in vglm.fitter(x = x, y = y, w = w, offset = offset, Xm2 = Xm2, : some\nquantities such as z, residuals, SEs may be inaccurate due to convergence at a\nhalf-step\n```\n\n\n:::\n\n```{.r .cell-code}\nsummary(model)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nGEE FOR NOMINAL MULTINOMIAL RESPONSES \nversion 1.6.0 modified 2017-07-10 \n\nLink : Baseline Category Logit \n\nLocal Odds Ratios:\nStructure: independence\nHomogenous scores: TRUE\n\ncall:\nmultgee::nomLORgee(formula = respnom ~ trtp + avisitn + trtp * \n avisitn, data = resp, id = usubjid, repeated = avisitn, LORstr = \"independence\")\n\nSummary of residuals:\n Min. 1st Qu. Median Mean 3rd Qu. Max. \n-0.4444 -0.3684 -0.3333 0.0000 0.6111 0.7778 \n\nNumber of Iterations: 1 \n\nCoefficients:\n Estimate san.se san.z Pr(>|san.z|)\nbeta10 0.22314 0.33541 0.6653 0.5059\ntrtpA:1 -0.62861 0.50139 -1.2537 0.2099\navisitn2:1 0.18232 0.43095 0.4231 0.6722\navisitn3:1 0.01325 0.47862 0.0277 0.9779\navisitn4:1 -0.06899 0.48140 -0.1433 0.8860\ntrtpA:avisitn2:1 0.16252 0.63421 0.2563 0.7977\ntrtpA:avisitn3:1 0.95184 0.69786 1.3639 0.1726\ntrtpA:avisitn4:1 0.64631 0.70063 0.9225 0.3563\nbeta20 0.27193 0.33184 0.8195 0.4125\ntrtpA:2 0.01575 0.45535 0.0346 0.9724\navisitn2:2 0.18005 0.42536 0.4233 0.6721\navisitn3:2 0.15551 0.46598 0.3337 0.7386\navisitn4:2 -0.27193 0.50787 -0.5354 0.5924\ntrtpA:avisitn2:2 -0.25642 0.60253 -0.4256 0.6704\ntrtpA:avisitn3:2 0.11642 0.64694 0.1800 0.8572\ntrtpA:avisitn4:2 0.15610 0.63994 0.2439 0.8073\n\nLocal Odds Ratios Estimates:\n [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]\n[1,] 0 0 1 1 1 1 1 1\n[2,] 0 0 1 1 1 1 1 1\n[3,] 1 1 0 0 1 1 1 1\n[4,] 1 1 0 0 1 1 1 1\n[5,] 1 1 1 1 0 0 1 1\n[6,] 1 1 1 1 0 0 1 1\n[7,] 1 1 1 1 1 1 0 0\n[8,] 1 1 1 1 1 1 0 0\n\np-value of Null model: 0.28295 \n```\n\n\n:::\n:::\n\n\n# REFERENCES\n\n\\[1\\] [SAS Institute Inc.. SAS Help Center. The GEE procedure.](https://documentation.sas.com/doc/en/statug/15.2/statug_gee_examples01.htm)\n\n\\[2\\] [SAS/STAT® 13.1 User's Guide The GEE Procedure.](https://support.sas.com/documentation/onlinedoc/stat/141/gee.pdf)\n\n\\[3\\] [SAS/STAT® 13.1 User's Guide The GENMOD Procedure.](https://support.sas.com/documentation/onlinedoc/stat/131/genmod.pdf)\n\n\\[4\\] [Generalized Estimating Equation Package](https://cran.r-project.org/web/packages/geepack/geepack.pdf)\n\n\\[5\\] [Generalized Estimation Equation Solver](https://cran.r-project.org/web/packages/gee/gee.pdf)\n\n\\[6\\] [Touloumis A. (2015). \"R Package multgee: A Generalized Estimating Equations Solver for Multinomial Responses.\" Journal of Statistical Software.](https://www.jstatsoft.org/article/view/v064i08)", - "supporting": [], + "markdown": "---\ntitle: \"Generalized Estimating Equations (GEE) methods\"\n---\n\n# Comparison of SAS vs R\n\n# BINARY OUTCOME\n\nFor dichotomous response variables, the link functions is the probit (in case of rare events complementary log-log may be preferable). For outcomes with more than two categories, the cumulative link function is used in case of ordinal variables and generalized logit for nominal variables.\n\nIn SAS, PROC GEE or PROC GENMOD can be used to compute GEE models. In R, GEE models can be fitted using `geepack::geeglm` or `gee::gee.`\n\nEstimated probabilities and odds ratios (OR) can be obtained in SAS by adding an `LSMEANS` statement, and in R by using an additional function with results from `geepack::geeglm`. For models fitted with `gee::gee`, the `emmeans` package is not supported.\n\nThe table below summarizes options and details for each procedure/function:\n\n| **Procedure/Function** | geepack::geeglm | **gee:gee** | PROC GEE/GENMOD |\n|------------------|------------------|-------------------|------------------|\n| Outcome variable | Numeric (0, 1) | Factor | Numeric or character in `class` statement |\n| Correlation matrix (default) | Independence | Independence | Independence |\n| Correlation matrix (options) | independence, exchangeable, ar1, unstructured and user defined. | independence, fixed, stat_M_dep, non_stat_M_dep, exchangeable, AR-M and unstructured. | independence, ar, exchangeable, mdep, unstructured and user defined. |\n| Outcome variable | Numeric (0, 1) | Factor | Numeric or factor, in `class` statement |\n| Sandwich SE | By default | By default | By default |\n| Model-based (naive) SE | No | By default | `modelse` option in `repeated` statement |\n| Link functions | probit, logit | probit, logit | probit, logit, clogit and glogit |\n| Estimated probability of event | additional function `emmeans::emmeans` | `emmeans::emmeans` not supported | `LSMEANS` statement with `ilink` option |\n| Odds Ratio (OR) | additional function `emmeans::emmeans` | `emmeans::emmeans` not supported | `LSMEANS` statement with `exp` or `oddsraio`option |\n\nA comparison between SAS (using PROC GEE) and the R functions using data available in [\"Gee Model for Binary Data\"](https://documentation.sas.com/doc/en/statug/15.2/statug_code_genmex5.htm), and similar results were found across software using similar options (See R and SAS sections).\n\nNote small differences may be found in later decimal places (\\>9 decimal position) in results obtained with PROC GENMOD and PROC GEE. Similarly, differences in later decimals may be found across R functions.\n\n# OUTCOME WITH MORE THAN TWO CATEGORIES\n\nModels with cumulative logit link functions apply to ordinal data and generalized logit models are fit to nominal data.\n\nIn SAS, similar syntax used for GEE models can be applied by specifying a multinomial distribution and selecting the appropriate link function. In R, the `multgee`package provides two functions for estimating GEE models when the outcome has more than two categories: `ordLORgee` for ordinal variables and `nomLORgee` for nominal variables.\n\n| **Procedure/Function** | multgee:ordLORgee | multgee:nomLORgee | PROC GEE/PROC GENMOD | PROC GEE/GENMOD |\n|---------------|---------------|---------------|---------------|---------------|\n| Correlation matrix (default) | Exchangeable | Independence | Independence | |\n| Correlation matrix (options) | independence, uniform, exchangeable, time.exch, fixed | independence, exchangeable, RC or fixed | Independence^(1)^ | |\n| Link function | clogit (implicit - not user configurable) | glogit (implicit - not user configurable) | glogit, glogit^(2)^ | |\n| Model-based (naive) SE | No | By default | `modelse` option in `repeated` statement | |\n\n(1) For multinomial responses, SAS limits the correlation matrix type to `independent,` so other correlation matrix options are not supported.\n\n(2) Generalized logit is available in PROC GEE, but not in PROC GENMOD.\n\nThe same data analyzed in the corresponding section for SAS and R are analyzed to compare results using equivalent settings, i.e.: specifying 'independence' correlation matrix in R to match SAS.\n\nSAS results were stored in a dataset and reformatted to align with R output in terms of order and decimal precision, making visual comparison easier.\n\n### Ordinal variable - Example\n\n```sas\nproc genmod data=resp ;\nclass trtp(ref=\"P\") avisitn(ref='1') usubjid ; \nmodel respord=trtp avisitn trtp*avisitn / dist=multinomial link=cumlogit ; \nrepeated subject=usubjid /corr=ind; \nlsmeans trtp*avisitn/cl exp ilink oddsratio diff;\nods output GEEEmpPEst=GEEEmpPEst_ord ;\nrun;\n```\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/gee/r_sas_1_ordinal_variable.png){fig-align='center' width=50%}\n:::\n:::\n\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel <- multgee::ordLORgee(formula = respord ~ trtp + avisitn + trtp*avisitn,\n data = resp,\n id = usubjid,\n repeated = avisitn,\n LORstr = \"independence\")\nsummary(model)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nGEE FOR ORDINAL MULTINOMIAL RESPONSES \nversion 1.6.0 modified 2017-07-10 \n\nLink : Cumulative logit \n\nLocal Odds Ratios:\nStructure: independence\n\ncall:\nmultgee::ordLORgee(formula = respord ~ trtp + avisitn + trtp * \n avisitn, data = resp, id = usubjid, repeated = avisitn, LORstr = \"independence\")\n\nSummary of residuals:\n Min. 1st Qu. Median Mean 3rd Qu. Max. \n-0.4698929 -0.3568933 -0.2853896 0.0002563 0.6339985 0.7277428 \n\nNumber of Iterations: 1 \n\nCoefficients:\n Estimate san.se san.z Pr(>|san.z|) \nbeta10 -0.55049 0.24303 -2.2652 0.02350 *\nbeta20 0.62726 0.25615 2.4488 0.01433 *\ntrtpA 0.42992 0.34205 1.2569 0.20880 \navisitn2 0.00108 0.36663 0.0030 0.99765 \navisitn3 0.11733 0.32268 0.3636 0.71616 \navisitn4 -0.15048 0.33263 -0.4524 0.65099 \ntrtpA:avisitn2 -0.27197 0.50959 -0.5337 0.59355 \ntrtpA:avisitn3 -0.58563 0.46208 -1.2674 0.20503 \ntrtpA:avisitn4 -0.31782 0.44813 -0.7092 0.47819 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\nLocal Odds Ratios Estimates:\n [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]\n[1,] 0 0 1 1 1 1 1 1\n[2,] 0 0 1 1 1 1 1 1\n[3,] 1 1 0 0 1 1 1 1\n[4,] 1 1 0 0 1 1 1 1\n[5,] 1 1 1 1 0 0 1 1\n[6,] 1 1 1 1 0 0 1 1\n[7,] 1 1 1 1 1 1 0 0\n[8,] 1 1 1 1 1 1 0 0\n\np-value of Null model: 0.68672 \n```\n\n\n:::\n:::\n\n\n### Nominal variable - Example\n\n```sas\nproc gee data=resp ;\nclass trtp(ref=\"P\") avisitn(ref='1') usubjid respnom(ref='Lung'); \nmodel respnom=trtp avisitn trtp*avisitn/ dist=multinomial link=glogit ; \nrepeated subject=usubjid /corr=ind /*modelse*/; \nlsmeans trtp*avisitn/cl exp ilink oddsratio diff;\nods output GEEEmpPEst=GEEEmpPEst_nom ;\nrun;\n```\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/gee/r_sas_2_nominal_variable.png){fig-align='center' width=50%}\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel <- multgee::nomLORgee(formula = respnom ~ trtp + avisitn + trtp*avisitn,\n data = resp,\n id = usubjid,\n repeated = avisitn,\n LORstr = \"independence\")\nsummary(model)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nGEE FOR NOMINAL MULTINOMIAL RESPONSES \nversion 1.6.0 modified 2017-07-10 \n\nLink : Baseline Category Logit \n\nLocal Odds Ratios:\nStructure: independence\nHomogenous scores: TRUE\n\ncall:\nmultgee::nomLORgee(formula = respnom ~ trtp + avisitn + trtp * \n avisitn, data = resp, id = usubjid, repeated = avisitn, LORstr = \"independence\")\n\nSummary of residuals:\n Min. 1st Qu. Median Mean 3rd Qu. Max. \n-0.4444 -0.3684 -0.3333 0.0000 0.6111 0.7778 \n\nNumber of Iterations: 1 \n\nCoefficients:\n Estimate san.se san.z Pr(>|san.z|)\nbeta10 0.22314 0.33541 0.6653 0.5059\ntrtpA:1 -0.62861 0.50139 -1.2537 0.2099\navisitn2:1 0.18232 0.43095 0.4231 0.6722\navisitn3:1 0.01325 0.47862 0.0277 0.9779\navisitn4:1 -0.06899 0.48140 -0.1433 0.8860\ntrtpA:avisitn2:1 0.16252 0.63421 0.2563 0.7977\ntrtpA:avisitn3:1 0.95184 0.69786 1.3639 0.1726\ntrtpA:avisitn4:1 0.64631 0.70063 0.9225 0.3563\nbeta20 0.27193 0.33184 0.8195 0.4125\ntrtpA:2 0.01575 0.45535 0.0346 0.9724\navisitn2:2 0.18005 0.42536 0.4233 0.6721\navisitn3:2 0.15551 0.46598 0.3337 0.7386\navisitn4:2 -0.27193 0.50787 -0.5354 0.5924\ntrtpA:avisitn2:2 -0.25642 0.60253 -0.4256 0.6704\ntrtpA:avisitn3:2 0.11642 0.64694 0.1800 0.8572\ntrtpA:avisitn4:2 0.15610 0.63994 0.2439 0.8073\n\nLocal Odds Ratios Estimates:\n [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]\n[1,] 0 0 1 1 1 1 1 1\n[2,] 0 0 1 1 1 1 1 1\n[3,] 1 1 0 0 1 1 1 1\n[4,] 1 1 0 0 1 1 1 1\n[5,] 1 1 1 1 0 0 1 1\n[6,] 1 1 1 1 0 0 1 1\n[7,] 1 1 1 1 1 1 0 0\n[8,] 1 1 1 1 1 1 0 0\n\np-value of Null model: 0.28295 \n```\n\n\n:::\n:::\n\n\n# REFERENCES\n\n\\[1\\] [SAS Institute Inc.. SAS Help Center. The GEE procedure.](https://documentation.sas.com/doc/en/statug/15.2/statug_gee_examples01.htm)\n\n\\[2\\] [SAS/STAT® 13.1 User's Guide The GEE Procedure.](https://support.sas.com/documentation/onlinedoc/stat/141/gee.pdf)\n\n\\[3\\] [SAS/STAT® 13.1 User's Guide The GENMOD Procedure.](https://support.sas.com/documentation/onlinedoc/stat/131/genmod.pdf)\n\n\\[4\\] [Generalized Estimating Equation Package](https://cran.r-project.org/web/packages/geepack/geepack.pdf)\n\n\\[5\\] [Generalized Estimation Equation Solver](https://cran.r-project.org/web/packages/gee/gee.pdf)\n\n\\[6\\] [Touloumis A. (2015). \"R Package multgee: A Generalized Estimating Equations Solver for Multinomial Responses.\" Journal of Statistical Software.](https://www.jstatsoft.org/article/view/v064i08)", + "supporting": [ + "r-sas_gee_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_kruskalwallis/execute-results/html.json b/_freeze/Comp/r-sas_kruskalwallis/execute-results/html.json index 9f4c7835c..a3bcc49b8 100644 --- a/_freeze/Comp/r-sas_kruskalwallis/execute-results/html.json +++ b/_freeze/Comp/r-sas_kruskalwallis/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "6b9825af98efb2959c72216e389e4422", + "hash": "c779ca1736a9b12ee359fbaafd26295a", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Kruskal Wallis R v SAS\"\nexecute: \n eval: false\n---\n\n## Kruskal-Wallis: R and SAS\n\nFrom the individual R and SAS pages, performing the Kruskal-Wallis test in R using:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::kruskal.test(Sepal_Width ~ Species, data = iris_sub)\n```\n:::\n\n\nand in SAS using:\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc npar1way data=iris_sub wilcoxon;\n class Species;\n var Sepal_Width;\n exact;\nrun;\n```\n:::\n\n\nproduced the same results for the test statistic and asymptotic p-value.\n\nThere is a difference between languages in that SAS provides the EXACT option to easily output the exact p-value, where R does not seem to have an equivalent. A Monte Carlo permutation test may offer an alternative to the exact test on R. The `coin` package could help in implementing this.", + "markdown": "---\ntitle: \"Kruskal Wallis R v SAS\"\n---\n\n## Kruskal-Wallis: R and SAS\n\nFrom the individual R and SAS pages, performing the Kruskal-Wallis test in R using:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::kruskal.test(Sepal_Width ~ Species, data = iris_sub)\n```\n:::\n\n\nand in SAS using:\n\n```sas\nproc npar1way data=iris_sub wilcoxon;\n class Species;\n var Sepal_Width;\n exact;\nrun;\n```\n\nproduced the same results for the test statistic and asymptotic p-value.\n\nThere is a difference between languages in that SAS provides the EXACT option to easily output the exact p-value, where R does not seem to have an equivalent. A Monte Carlo permutation test may offer an alternative to the exact test on R. The `coin` package could help in implementing this.", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/Comp/r-sas_logistic-regr/execute-results/html.json b/_freeze/Comp/r-sas_logistic-regr/execute-results/html.json index 2b15b538d..3f6e6ce10 100644 --- a/_freeze/Comp/r-sas_logistic-regr/execute-results/html.json +++ b/_freeze/Comp/r-sas_logistic-regr/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "dc6f14f93f09543b9e1d6ea7ef5048bf", + "hash": "94ad018c8f0f6965b2339f41392c3b94", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS: Logistic Regression\"\ntoc: true\necho: true\neval: false\nkeep-hidden: true\n---\n\n# Summary\n\n## Goal\n\nComparison of results between SAS vs R for different applications of logistic regression; where possible we try to ensure the same statistical method or algorithm is specified. However, there are some underlying differences between the algorithms in SAS vs R that cannot be (at least not easily) \"tweaked\". The document also provides some remarks on what parameters to look out for and what could have caused the numerical differences.\n\n## Scope\n\n::::::: columns\n:::: {.column width=\"45%\"}\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n## Methodologies\n\n- Logistic regression\n- Firth's bias-reduced logistic regression\n- g-computation / standardization with covariate adjustment\n:::\n::::\n\n:::: {.column width=\"55%\"}\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n## Technical implementations\n\n- SAS: `PROC LOGISTIC` (with and without firth option) and `%margins` macro\\\n- R: `stats::glm`, `logistf::logistf` and `beeca::get_marginal_effect`\n:::\n::::\n:::::::\n\n## Findings\n\nBelow are summary of findings from a numerical comparison using example data, where possible we specify the same algorithm in R and SAS.\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Logistic regression\n\nMaximum Likelihood Estimates and p-values for the Model Parameters have an exact match (at 0.001 level) using `glm` in R vs `PROC LOGISTIC` procedure (without Firth option) in SAS.\n\nWhen using GLM parameterization (see [SAS page](https://psiaims.github.io/CAMIS/SAS/logistic-regr.html) for explanation of SAS parameterization types), the parameters estimates (and 95% CIs) can be exponentiated to provide odds ratios and 95% CIs for odds ratios.\n\nAs default for categorical variables, R uses the first category as reference see [R page](https://psiaims.github.io/CAMIS/R/logistic-regr.html), and SAS uses the last category as reference group. Check your design matrix in SAS, and `contr.` options in R to ensure interpretation of estimates from the model is correct, then results align.\n\nAn exact match (at 0.001 level) is obtained for the Odds ratios and CIs when the same method and same parameterization is used, however SAS Proc Logistic can only calculate Wald CI's. Profile likelihood CIs are not available.\n\nR using glm() function, can use the confint() function to calculate CI's using the profile likelihood method or the confint.default() function to calculate CIs using the Wald method.\n:::\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Firth logistic regression\n\nExact match cannot be obtained for all estimates using `logistf` vs `PROC LOGISTIC` procedure (with Firth option). More specifically:\\\n- Coefficient estimate and 95% CI matched at 0.001 level;\\\n- Standard error are not the same (e.g., 0.02023 for age in R vs 0.02065 in SAS);\\\n- p-value is not the same (0.6288 in R for age vs 0.6348 in SAS);\\\n:::\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### g-computation with covariate adjustment\n\nExact match (at 0.001 level) can be obtained using `get_marginal_effect` in R vs `%margins` macro in SAS.\n:::\n\nIn the following sections, the parameterisation of logistic regression implementation (with an without Firth option) will be compared followed by numerical comparison using example data.\n\n# Prerequisites\n\n## R packages\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(tidyverse)\n```\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\n── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──\n✔ dplyr 1.2.0 ✔ readr 2.1.6\n✔ forcats 1.0.1 ✔ stringr 1.6.0\n✔ ggplot2 4.0.2 ✔ tibble 3.3.1\n✔ lubridate 1.9.5 ✔ tidyr 1.3.2\n✔ purrr 1.2.1 \n── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──\n✖ dplyr::filter() masks stats::filter()\n✖ dplyr::lag() masks stats::lag()\nℹ Use the conflicted package () to force all conflicts to become errors\n```\n\n\n:::\n\n```{.r .cell-code}\nlibrary(survival) # for example data\nlibrary(logistf) # for firth regression\nlibrary(beeca) # for covariate adjustment\n```\n:::\n\n\n## Data\n\n### Logistic regressions\n\nWe use the `lung` dataset provided with {survival} R package. Initial data preparation involves generating a new binary outcome based on the weight change.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# the lung dataset is available in ./data/lung_cancer.csv\nlung2 <- survival::lung |>\n mutate(\n wt_grp = factor(wt.loss > 0, labels = c(\"weight loss\", \"weight gain\"))\n )\nglimpse(lung2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nRows: 228\nColumns: 11\n$ inst 3, 3, 3, 5, 1, 12, 7, 11, 1, 7, 6, 16, 11, 21, 12, 1, 22, 16…\n$ time 306, 455, 1010, 210, 883, 1022, 310, 361, 218, 166, 170, 654…\n$ status 2, 2, 1, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, …\n$ age 74, 68, 56, 57, 60, 74, 68, 71, 53, 61, 57, 68, 68, 60, 57, …\n$ sex 1, 1, 1, 1, 1, 1, 2, 2, 1, 1, 1, 2, 2, 1, 1, 1, 1, 1, 2, 1, …\n$ ph.ecog 1, 0, 0, 1, 0, 1, 2, 2, 1, 2, 1, 2, 1, NA, 1, 1, 1, 2, 2, 1,…\n$ ph.karno 90, 90, 90, 90, 100, 50, 70, 60, 70, 70, 80, 70, 90, 60, 80,…\n$ pat.karno 100, 90, 90, 60, 90, 80, 60, 80, 80, 70, 80, 70, 90, 70, 70,…\n$ meal.cal 1175, 1225, NA, 1150, NA, 513, 384, 538, 825, 271, 1025, NA,…\n$ wt.loss NA, 15, 15, 11, 0, 0, 10, 1, 16, 34, 27, 23, 5, 32, 60, 15, …\n$ wt_grp NA, weight gain, weight gain, weight gain, weight loss, weig…\n```\n\n\n:::\n:::\n\n\n### g-computation\n\nWe use the `trial01` dataset provided with {beeca} R package. Initial data preparation involves setting the treatment indicator as a categorical variable and removing any incomplete cases.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndata(\"trial01\")\n\ntrial01$trtp <- factor(trial01$trtp) ## set treatment to a factor\n\ntrial01 <- trial01 |>\n filter(!is.na(aval)) ## remove missing data i.e complete cases analysis\n\n# save the dataset to be imported in SAS\n# write.csv(trial01, file = \"data/trial01.csv\", na = \".\")\n```\n:::\n\n\n# Logistic Regression\n\n## Parameterisation Comparison\n\nThe following set of tables compare how to configure particular parameters / attributes of the methodologies.\n\n| Attribute | SAS
`PROC LOGISTIC` | R
`stats::glm` | Description | Note |\n|:-------------:|:-------------:|:-------------:|:--------------|:--------------|\n| Likelihood optimization algorithm | Default | Default | Fisher's scoring method (i.e., iteratively reweighted least squares (IRLS)) | For logistic regression, parameter estimates and covariance matrices estimated should be the same for both Fisher's and Newton-Raphson algorithm for maximum likelihood. |\n| Convergence criteria | Default | NA | Specifies relative gradient convergence criterion (GCONV=1E--8) | In`PROC LOGISTIC` there are three other convergence criteria which can be specified. However, there is no exact criterion that matches the criteria in `stats::glm`. |\n| Convergence criteria | NA | Default | Specifies relative difference between deviance \\< 1E--8. | |\n| Confidence interval (CI) estimation method | Default | `confint.default()` | Wald CI | In `stats::glm` in R, function confint.default() gives the Wald confidence limits; whereas function confint() gives the profile-likelihood limits. |\n| Hypothesis tests for regression coefficients | Default | Default | Wald tests, which are based on estimates for the regression coefficients and its corresponding standard error. | |\n\n: Standard Logistic Regression in SAS vs R {#tbl-1}\n\n## Numerical Comparison {#sec-num-comp}\n\nEvery effort is made to ensure that the R code employs estimation methods/ optimization algorithms/ other components that closely match (as much as possible) those used in the SAS code.\n\n### `glm` in R\n\nNote, the default fitting method in `glm` is consistent with the default fitting method in `PROC LOGISTIC` procedure.\n\n- Default fitting method in `glm` is iteratively reweighted least squares, and the documentation can be found [here](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/glm).\n- Default fitting method for `PROC LOGISTIC` procedure is Fisher's scoring method, which is reported as part of the SAS default output, and it is equivalent to \"Iteratively reweighted least squares\" method as reported in this [documentation](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_logistic_sect033.htm).\n\n\n::: {.cell}\n\n```{.r .cell-code}\nm1 <- stats::glm(\n wt_grp ~ age + sex + ph.ecog + meal.cal,\n data = lung2,\n family = binomial(link = \"logit\")\n)\n\n# model coefficients summary\nsummary(m1)$coefficients\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error z value Pr(>|z|)\n(Intercept) 3.2631672833 1.6488206996 1.9790917 0.04780569\nage -0.0101717451 0.0208107243 -0.4887742 0.62500157\nsex -0.8717357187 0.3714041991 -2.3471348 0.01891841\nph.ecog 0.4179665342 0.2588653214 1.6146100 0.10639518\nmeal.cal -0.0008869427 0.0004467405 -1.9853642 0.04710397\n```\n\n\n:::\n:::\n\n\nNote, function `confint.default` gives the Wald confidence limits, which is the default option in SAS `PROC LOGISTIC` procedure; whereas `confint` gives the profile-likelihood limits. Conditional odds ratio is calculated by taking the exponential of the model parameters.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncbind(est = coef(m1), confint.default(m1))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n est 2.5 % 97.5 %\n(Intercept) 3.2631672833 0.031538095 6.494796e+00\nage -0.0101717451 -0.050960015 3.061653e-02\nsex -0.8717357187 -1.599674572 -1.437969e-01\nph.ecog 0.4179665342 -0.089400173 9.253332e-01\nmeal.cal -0.0008869427 -0.001762538 -1.134731e-05\n```\n\n\n:::\n:::\n\n\n### `PROC LOGISTIC` in SAS (without firth option)\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC LOGISTIC DATA=LUNG2; # import lung\n\tMODEL WT_GRP(EVENT=\"weight_gain\") = AGE SEX PH_ECOG MEAL_CAL;\n\tods output ESTIMATEs=estimates;\nrun;\n```\n:::\n\n\nBelow is screenshot of output tables summarizing coefficient estimates and confidence intervals\n\n![](../images/logistic_regression/sas_logistic_estimates.png){fig-align=\"left\"}\n\n![](../images/logistic_regression/sas_logistic_ci.png){fig-align=\"left\"}\n\n### Comment on model selection\n\nAs indicated in [Logistic regression in R](https://psiaims.github.io/CAMIS/R/logistic_regr.html) and [Logistic regression in SAS](https://psiaims.github.io/CAMIS/SAS/logistic-regr.html), the chi-Sq test statistics and p-values are different when performing model selections in R vs. SAS. The reason for this discrepancy is that the chi-Sq statistics from `anova()` in R is based on deviance test using residual deviance while the chi-Sq statistics from `PROC LOGISTIC` w/ `SELECTION` option in SAS is based on Wald test using z-values squared.\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Conclusion for logistic regression\n\nExact match (at 0.001 level) can be obtained using `glm` in R vs `PROC LOGISTIC` procedure (without Firth option) in SAS, for coefficient estimates, 95% CI, and for p-value.\n:::\n\n# Firth logistic regression\n\nThe following set of tables compare how to configure particular parameters / attributes of the methodologies.\n\n## Parameterisation Comparison\n\n| Attribute | SAS
`PROC LOGISTIC` w/ Firth option | R
`logistf::logistf` | Description | Note |\n|:-------------:|:--------------|:--------------|:--------------|:--------------|\n| Likelihood optimization algorithm | Default | `control =`
`logistf.control`
`(fit =“IRLS”)` | Fisher's scoring method (i.e., iteratively reweighted least squares (IRLS)) | |\n| Likelihood optimization algorithm | `TECHNIQUE = NEWTON` | Default | Newton-Raphson algorithm | |\n| Convergence criteria | Default | NA | Specifies relative gradient convergence criterion (GCONV=1E--8). | In`PROC LOGISTIC` there are three other convergence criteria which can be specified. If more than one convergence criterion is specified, the optimization is terminated as soon as one of the criteria is satisfied. |\n| Convergence criteria | NA | Default | Specifies three criteria that need to be met: the change in log likelihood is less than lconv (default is 1E-5), the maximum absolute element of the score vector is less than gconv (default is 1E-5), and the maximum absolute change in beta is less than xconv (default is 1E-5). | The gconv criteria in `logistif` is different from `GCONV` in SAS. The lconv criteria is also not exactly the same as the `ABSFCONV` or `FCONV` in `PROC LOGISTIC` in SAS, although the criteria use log likelihood. However, the `xconv` in R and `XCONV` in SAS seems to be consistent. |\n| Convergence criteria | `XCONV = 1E–8` | `control = logistf.control( xconv = 1E–8, lconv = 1, gconv = 1)` | Specifies the maximum absolute change in beta \\< 1E--8. | In `logistf`, three convergence criteria are checked at the same time. So here we use a large convergence criteria value for `lconv` and `gconv` to mimic the scenario where only `xconv` is checked. |\n| Confidence interval (CI) estimation method | Default | `pl= FALSE` | Wald CI | For `logistf`: \"Note that from version 1.24.1 on, the variance-covariance matrix is based on the second derivative of the likelihood of the augmented data rather than the original data, which proved to be a better approximation if the user chooses to set a higher value for the penalty strength.\" This could cause differences in standard error estimates in R vs SAS for Firth logistic regression, and consequently results in differences in the corresponding Wald CI estimates and hypothesis tests results (e.g., p-values). |\n| Confidence interval (CI) estimation method | `CLPARM = PL`
`CLODDS = PL` | Default | Profile likelihood-based CI | For Firth's bias-reduced logistic regression, it makes more sense to use penalized likelihood-based CI so it is consistent with the parameter estimation method which uses penalized maximum likelihood. |\n| Hypothesis tests for regression coefficients | Default | pl= FALSE | Wald tests, which are based on estimates for the regression coefficients and its corresponding standard error. | |\n| Hypothesis tests for regression coefficients | NA | Default | \"Likelihood ratio tests\", which are based on profile penalized log likelihood. | In SAS, when the model statement option `CLPARM = PL` is specified, the CI will be calculated based on profile likelihood. However, the hypothesis testing method is still a Wald method. This could cause results mismatch in the p-value. |\n\n: Firth's Bias-Reduced Logistic Regression in SAS vs R {#tbl-2}\n\n## Numerical Comparison\n\nNote that while Firth logistic regression is not required for our example dataset nonetheless we use it for demonstration purposes only.\n\n### `logistf` in R\n\n- By default, the [convergence criteria in `logistf`](https://cran.r-project.org/web/packages/logistf/logistf.pdf) specifies that three criteria need to be met at the same time, i.e., the change in log likelihood is less than lconv (default is 1E-5), the maximum absolute element of the score vector is less than gconv (default is 1E-5), and the maximum absolute change in beta is less than xconv (default is 1E-5). In SAS, the [default convergence criteria in `PROC LOGISTIC`](https://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_logistic_sect034.htm) specifies relative gradient convergence criterion (GCONV=1E--8); while SAS also support three other convergence criteria but when there are more than one convergence criterion specified, the optimization is terminated as soon as one of the criteria is satisfied. By looking at the R pacakge/SAS documentation, the `gconv` criteria in `logistif` function is different from the `GCONV` in SAS. The `lconv` criteria is also not exactly the same as the `ABSFCONV` or `FCONV` in PROC LOGISTIC in SAS, although the criteria use log likelihood. However, similar convergence criteria might be obtained by using the maximum absolute change in parameter estimates (i.e., `xconv` in R and SAS). Therefore, for comparison with the SAS output, in `logistf` function, we use a large convergence criteria value for `lconv` and `gconv` to mimic the scenario where only `xconv` is checked, i.e., specify `logistf.control(xconv = 0.00000001, gconv = 1, lconv = 1)` for the `control` argument.\n\n- By default, `logistf` function in R computes the confidence interval estimates and hypothesis tests (including p-value) for each parameter based on profile likelihood, which is also reported in the output below. However, Wald method (confidence interval and tests) can be specified by specifying the `control` argument with [`pl = FALSE`](https://cran.r-project.org/web/packages/logistf/logistf.pdf).\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfirth_mod <- logistf(\n wt_grp ~ age + sex + ph.ecog + meal.cal,\n data = lung2,\n control = logistf.control(\n fit = \"IRLS\",\n xconv = 0.00000001,\n gconv = 1,\n lconv = 1\n )\n)\nsummary(firth_mod)$coefficients\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nlogistf(formula = wt_grp ~ age + sex + ph.ecog + meal.cal, data = lung2, \n control = logistf.control(fit = \"IRLS\", xconv = 1e-08, gconv = 1, \n lconv = 1))\n\nModel fitted by Penalized ML\nCoefficients:\n coef se(coef) lower 0.95 upper 0.95 Chisq\n(Intercept) 3.1532937589 1.6031659729 0.051844703 6.410119e+00 3.9726447\nage -0.0098111679 0.0202315630 -0.050518148 2.974343e-02 0.2337368\nsex -0.8455619163 0.3632129422 -1.571158740 -1.356810e-01 5.4536777\nph.ecog 0.4018229715 0.2520090355 -0.090278518 9.093255e-01 2.5553004\nmeal.cal -0.0008495327 0.0004288525 -0.001722033 -7.098976e-06 3.9058205\n p method\n(Intercept) 0.04624509 2\nage 0.62876680 2\nsex 0.01952718 2\nph.ecog 0.10992492 2\nmeal.cal 0.04811912 2\n\nMethod: 1-Wald, 2-Profile penalized log-likelihood, 3-None\n\nLikelihood ratio test=10.54964 on 4 df, p=0.03212009, n=170\nWald test = 33.85701 on 4 df, p = 7.972359e-07\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n (Intercept) age sex ph.ecog meal.cal \n 3.1532937589 -0.0098111679 -0.8455619163 0.4018229715 -0.0008495327 \n```\n\n\n:::\n\n```{.r .cell-code}\n## Code below would give Wald CI and tests results by adding `pl = FALSE`\n# logistf(..., pl = FALSE)\n```\n:::\n\n\nNote, function `confint` gives the profile-likelihood limits. Given the parameters from Firth's bias-reduced logistic regression is estimated using penalized maximum likelihood, `confint` function is used. Conditional odds ratio is calculated by taking the exponential of the model parameters.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncbind(est = coef(firth_mod), confint(firth_mod))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n est Lower 95% Upper 95%\n(Intercept) 3.1532937589 0.051844703 6.410119e+00\nage -0.0098111679 -0.050518148 2.974343e-02\nsex -0.8455619163 -1.571158740 -1.356810e-01\nph.ecog 0.4018229715 -0.090278518 9.093255e-01\nmeal.cal -0.0008495327 -0.001722033 -7.098976e-06\n```\n\n\n:::\n:::\n\n\n### `PROC LOGISTIC` in SAS (with firth option)\n\n- Note, by default, SAS computes confidence interval based on Wald tests. Given the parameters from Firth's method is estimated using penalized maximum likelihood, below specifies CLODDS = PL CLPARM=PL (based on profile likelihood), which is consistent with the maximization method and the R code above. However, the [default hypothesis test for the regression coefficients](https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_logistic_details50.htm) is still a Wald test, and the Chi-square statistics is calculated based on coefficient estimate and its corresponding standard error.\n\n- `XCONV` specifies relative parameter convergence criterion, which should correspond to the `xconv` in `logistf` function in R. We specify `XCONV = 0.00000001` so it should be consistent with the R code above.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC LOGISTIC DATA=LUNG2;\n\tMODEL WT_GRP(EVENT=\"weight gain\") = AGE SEX PH_ECOG MEAL_CAL / firth \n clodds=PL clparm=PL xconv = 0.00000001;\n\tods output ESTIMATEs=estimates;\nrun;\n```\n:::\n\n\nBelow is screenshot of output tables summarizing coefficient estimates and it's 95% CI\n\n![](../images/logistic_regression/sas_logistic_firth_estimates.png){fig-align=\"left\"}\n\n![](../images/logistic_regression/sas_logistic_firth_ci.png){fig-align=\"left\"}\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Conclusion for Firth logistic regression\n\nExact match cannot be obtained for all estimates using `logistf` vs `PROC LOGISTIC` procedure with Firth option. More specifically:\\\n- Coefficient estimate and its 95% CI matched at 0.001 level;\\\n- Standard error are not the same (e.g., 0.02023 for age in R vs 0.02065 in SAS);\\\n- p-value is not the same (0.6288 in R for age vs 0.6348 in SAS);\\\n:::\n\n# g-computation with covariate adjustment\n\nWe compare two implementions of g-computation in SAS:\n\n1. The \"Predictive margins and average marginal effects\" [%margins](https://support.sas.com/kb/63/038.html#pur) macro. The %margins macro uses \"the delta method \\[...\\] to determine the standard errors for predictive margins and marginal effects\". Note that the %margins macro uses the `PROC GENMOD` procedure to implement the working logistic regression model and require another macro [%NLEST](https://support.sas.com/kb/58/775.html) to calculate contrasts that requires delta methodl such as risk ratio or odds ratio.\n2. The SAS code provided in the appendix of the [Ge et al. (2011)](https://journals.sagepub.com/doi/10.1177/009286151104500409) implements the method outlined in the associated paper and simulations. Note: the Ge et al. (2011) macro uses the `PROC LOGISTIC` procedure to implement the working logistic regression model. `PROC IML` is used to calculate the delta method to determine the standard errors.\n\n## Numerical Comparison\n\n### `get_marginal_effect` in R\n\nWe fit a logistic regression model with covariate adjustment to estimate the marginal treatment effect using the delta method for variance estimation: as outlined in Ge et al (2011).\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## fit the model including model based variance estimation with delta method\nfit1 <- stats::glm(aval ~ trtp + bl_cov, family = \"binomial\", data = trial01) |>\n beeca::get_marginal_effect(\n trt = \"trtp\",\n method = \"Ge\",\n contrast = \"diff\",\n reference = \"0\",\n type = \"model-based\"\n )\n\n\"Marginal treatment effect\"\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] \"Marginal treatment effect\"\n```\n\n\n:::\n\n```{.r .cell-code}\nfit1$marginal_est\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n diff: 1-0 \n-0.06836399 \nattr(,\"reference\")\n[1] \"0\"\nattr(,\"contrast\")\n[1] \"diff: 1-0\"\n```\n\n\n:::\n\n```{.r .cell-code}\n\"Standard error\"\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] \"Standard error\"\n```\n\n\n:::\n\n```{.r .cell-code}\nfit1$marginal_se\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n diff: 1-0 \n0.06071641 \nattr(,\"reference\")\n[1] \"0\"\nattr(,\"contrast\")\n[1] \"diff: 1-0\"\nattr(,\"type\")\n[1] \"Ge - model-based\"\n```\n\n\n:::\n:::\n\n\n### `%Margins` macro in SAS\n\nWe now use the SAS [`%Margins`](https://support.sas.com/kb/63/038.html) macro to perform the Ge et al. (2011) method on `trial01` to estimate the marginal risk difference and it's standard error.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\n%Margins(data = myWork.trial01,\n class = trtp,\n classgref = first, /*Set reference to first level*/\n response = avaln,\n roptions = event='1', /*Ensure event is set to 1 = Yes */\n dist = binomial, \n model = trtp bl_cov,\n margins = trtp, \n options = cl diff reverse, /*Specify risk difference contrast and \n direction of treatment effect is correct*/\n link = logit); /*Specify logit link function */\n\t\n** Store output data sets ; \ndata myWork.margins_trt_estimates;\n set work._MARGINS;\nrun;\n\ndata myWork.margins_trt_diffs;\n set work._DIFFSPM;\nrun;\n```\n:::\n\n\n![](../images/logistic_regression/sas_logistic_gcomp_margins.png){fig-align=\"left\"}\n\n### `%LR` macro in SAS (Ge et al, 2011)\n\n\n::: {.cell}\n\n```{.sas .cell-code}\n%LR(data = myWork.trial01, /* input data set */\n\tvar1 = bl_cov, /* continuous covariates in the logistic regression */\n\tvar2 = trtp, /* categorical covariates in the logistic regression */\n\tp1 = 1, /* number of continuous covariates in the logistic regression */\n\tp2 = 1, /* number of categorical covariates in the logistic regression */\n\tresp = avaln, /* binary response variable in the logistic regression */\n\tntrt = 1); /* position of the treatment variable in the categorical covariates */\n\t\ndata myWork.ge_macro_trt_diffs;\n set work.geout;\nrun;\n```\n:::\n\n\n![](../images/logistic_regression/sas_logistic_gcomp_ge.png){fig-align=\"left\"}\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Conclusion for g-computation with covariate adjustment\n\nExact match at the 0.001 level.\n:::\n\n# Final remarks\n\nIn summary, there are a few things to be aware of when comparing logistic regression results in R vs SAS. It is crucial to carefully manage the input parameters for each model to ensure they are configured similarly for logistic regression analyses. As highlighted also in [Logistic Regression in SAS](https://psiaims.github.io/CAMIS/SAS/logistic-regr.html), the variable parameterization is also important for modelling and interpretation, ensuring the types of variable (continuous vs. categorical) and reference values of categorical variable are applied as expected.\n\n1. **Likelihood optimization method**\n\n- The default likelihood optimization method in `glm` and `PROC LOGISTIC` is the same (i.e., Fisher's scoring method or iteratively reweighted least squares (IRLS)).\n\n- However, the default optimization method in `logistf` is Newton-Raphson, which can be modified into IRLS via `control = logistf.control(fit = “IRLS”)`. Alternatively, one could specify `technique = newton` in the model statement in SAS to modify the likelihood optimization method.\n\n2. **Convergence criteria**\n\n- Although both SAS and R allows options to modify the convergence criteria, the criteria does not seem to be exactly the same, which could cause results mismatch in some scenarios.\n\n- The [default convergence criteria in `PROC LOGISTIC`](https://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_logistic_sect034.htm) specifies the relative gradient convergence criterion; where the [default convergence criteria in `glm`](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/glm.control.html) specifies relative difference between deviance.\n\n- The default setting in logistf have checked more than one convergence criterion in its algorithm (i.e., [change in log likelihood, derivative of the log likelihood and parameter estimates](https://cran.r-project.org/web/packages/logistf/logistf.pdf)). One could specify a very large value for two of the criteria in order to mimic the scenario where only one criterion is checked (e.g., `control = logistf.control (xconv = 0.00000001, lconv = 1, gconv = 1`) in `logistf` in R should be consistent to the option of `xconv = 0.00000001` in SAS).\n\n3. **Confidence interval**\n\n- The `confint()` function in R will computes profile likelihood based CI for `glm` fitted model. However, in SAS, the default confidence interval is Wald CI. To match the default CI calculation in SAS for `glm` fitted model, use `confint.default()` function in R.\n\n- Nevertheless, Firth's biased-reduced logistic regression estimates parameter using penalized maximum likelihood, it makes more sense to use `confint()` function for `logistf` fitted model. In the meantime, in SAS, when fitting a Firth's logistic regression, it is also better to specify the model statement option `clparm = pl` which will also generate profile penalized likelihood CI.\n\n- We shall note that in the Firth logistic regression numerical example, the estimated standard errors does not match, but the CIs match at 0.001 level. This is because the CI was estimated based on profile penalized likelihood in R and SAS, and please see the next discussion point for potential reasons about differences between the estimated standard error. (I have compared Wald CIs estimated in R vs SAS, which could not match. This make sense as Wald CIs are calculated based on the estimated standard errors.)\n\n4. **Hypothesis test and p-value**\n\n- The default hypothesis tests for the regression coefficients are the same in `glm` and `PROC LOGISTIC`, which are both Wald tests and calculated based on estimates for the regression coefficients and its corresponding standard error.\n\n- As for `logistf` function, the default hypothesis testing method is based on profile penalized log likelihood (source code [here](https://github.com/georgheinze/logistf/blob/master/R/logistf.R)). And it was noted in the [R documentation](https://cran.r-project.org/web/packages/logistf/logistf.pdf) that, *\"from version 1.24.1 on, the variance-covariance matrix is based on the second derivative of the likelihood of the augmented data rather than the original data, which proved to be a better approximation if the user chooses to set a higher value for the penalty strength.\"* This could cause difference in the estimate of standard error in R vs SAS for Firth logistic regression, and consequently results in differences in the corresponding Wald CI estimates and hypothesis tests results (e.g., p-values).\n\n- Wald method can be used in a `logistf` function in R by specifying `pl = FALSE` in the `control` argument, which should correspond to the method used in SAS to calculate p-value. However, when specifying `pl = FALSE`, the CI is also calculated using Wald method.\n\n# Reference\n\n- A relevant blog [here](https://sas-and-r.blogspot.com/2010/11/example-815-firth-logistic-regression.html) (check comments in the blog).\n- [PROC LOGISTIC statement documentation in SAS](https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_logistic_sect004.htm).\n- [Reference manual for `logistf` package in R](https://cran.r-project.org/web/packages/logistf/logistf.pdf).\n- [GitHub repository for `logistf` package in R](https://github.com/georgheinze/logistf).\n- [GitHub repository for a SAS procedure about Firth logistic regression authored by the author of `logistf` R package](https://github.com/georgheinze/flicflac/tree/master/LogisticRegression), which was based on PROC IML instead of PROC LOGISTIC and was probably authored before the availability of Firth option in PROC LOGISTIC statement in SAS.\n- Ge, Miaomiao, et al. \"Covariate-adjusted difference in proportions from clinical trials using logistic regression and weighted risk differences.\" Drug information journal: DIJ/Drug Information Association 45 (2011): 481-493.\n- SAS Institute Inc. [\"Predictive margins and average marginal effects.\"](https://support.sas.com/kb/63/038.html) (Last Published: 13 Dec 2023)", - "supporting": [], + "markdown": "---\ntitle: \"R vs SAS: Logistic Regression\"\ntoc: true\necho: true\neval: false\nkeep-hidden: true\n---\n\n# Summary\n\n## Goal\n\nComparison of results between SAS vs R for different applications of logistic regression; where possible we try to ensure the same statistical method or algorithm is specified. However, there are some underlying differences between the algorithms in SAS vs R that cannot be (at least not easily) \"tweaked\". The document also provides some remarks on what parameters to look out for and what could have caused the numerical differences.\n\n## Scope\n\n::::::: columns\n:::: {.column width=\"45%\"}\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n## Methodologies\n\n- Logistic regression\n- Firth's bias-reduced logistic regression\n- g-computation / standardization with covariate adjustment\n:::\n::::\n\n:::: {.column width=\"55%\"}\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n## Technical implementations\n\n- SAS: `PROC LOGISTIC` (with and without firth option) and `%margins` macro\\\n- R: `stats::glm`, `logistf::logistf` and `beeca::get_marginal_effect`\n:::\n::::\n:::::::\n\n## Findings\n\nBelow are summary of findings from a numerical comparison using example data, where possible we specify the same algorithm in R and SAS.\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Logistic regression\n\nMaximum Likelihood Estimates and p-values for the Model Parameters have an exact match (at 0.001 level) using `glm` in R vs `PROC LOGISTIC` procedure (without Firth option) in SAS.\n\nWhen using GLM parameterization (see [SAS page](https://psiaims.github.io/CAMIS/SAS/logistic-regr.html) for explanation of SAS parameterization types), the parameters estimates (and 95% CIs) can be exponentiated to provide odds ratios and 95% CIs for odds ratios.\n\nAs default for categorical variables, R uses the first category as reference see [R page](https://psiaims.github.io/CAMIS/R/logistic-regr.html), and SAS uses the last category as reference group. Check your design matrix in SAS, and `contr.` options in R to ensure interpretation of estimates from the model is correct, then results align.\n\nAn exact match (at 0.001 level) is obtained for the Odds ratios and CIs when the same method and same parameterization is used, however SAS Proc Logistic can only calculate Wald CI's. Profile likelihood CIs are not available.\n\nR using glm() function, can use the confint() function to calculate CI's using the profile likelihood method or the confint.default() function to calculate CIs using the Wald method.\n:::\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Firth logistic regression\n\nExact match cannot be obtained for all estimates using `logistf` vs `PROC LOGISTIC` procedure (with Firth option). More specifically:\\\n- Coefficient estimate and 95% CI matched at 0.001 level;\\\n- Standard error are not the same (e.g., 0.02023 for age in R vs 0.02065 in SAS);\\\n- p-value is not the same (0.6288 in R for age vs 0.6348 in SAS);\\\n:::\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### g-computation with covariate adjustment\n\nExact match (at 0.001 level) can be obtained using `get_marginal_effect` in R vs `%margins` macro in SAS.\n:::\n\nIn the following sections, the parameterisation of logistic regression implementation (with an without Firth option) will be compared followed by numerical comparison using example data.\n\n# Prerequisites\n\n## R packages\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(tidyverse)\nlibrary(survival) # for example data\nlibrary(logistf) # for firth regression\nlibrary(beeca) # for covariate adjustment\n```\n:::\n\n\n## Data\n\n### Logistic regressions\n\nWe use the `lung` dataset provided with {survival} R package. Initial data preparation involves generating a new binary outcome based on the weight change.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# the lung dataset is available in ./data/lung_cancer.csv\nlung2 <- survival::lung |>\n mutate(\n wt_grp = factor(wt.loss > 0, labels = c(\"weight loss\", \"weight gain\"))\n )\nglimpse(lung2)\n```\n:::\n\n\n### g-computation\n\nWe use the `trial01` dataset provided with {beeca} R package. Initial data preparation involves setting the treatment indicator as a categorical variable and removing any incomplete cases.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndata(\"trial01\")\n\ntrial01$trtp <- factor(trial01$trtp) ## set treatment to a factor\n\ntrial01 <- trial01 |>\n filter(!is.na(aval)) ## remove missing data i.e complete cases analysis\n\n# save the dataset to be imported in SAS\n# write.csv(trial01, file = \"data/trial01.csv\", na = \".\")\n```\n:::\n\n\n# Logistic Regression\n\n## Parameterisation Comparison\n\nThe following set of tables compare how to configure particular parameters / attributes of the methodologies.\n\n| Attribute | SAS
`PROC LOGISTIC` | R
`stats::glm` | Description | Note |\n|:-------------:|:-------------:|:-------------:|:--------------|:--------------|\n| Likelihood optimization algorithm | Default | Default | Fisher's scoring method (i.e., iteratively reweighted least squares (IRLS)) | For logistic regression, parameter estimates and covariance matrices estimated should be the same for both Fisher's and Newton-Raphson algorithm for maximum likelihood. |\n| Convergence criteria | Default | NA | Specifies relative gradient convergence criterion (GCONV=1E--8) | In`PROC LOGISTIC` there are three other convergence criteria which can be specified. However, there is no exact criterion that matches the criteria in `stats::glm`. |\n| Convergence criteria | NA | Default | Specifies relative difference between deviance \\< 1E--8. | |\n| Confidence interval (CI) estimation method | Default | `confint.default()` | Wald CI | In `stats::glm` in R, function confint.default() gives the Wald confidence limits; whereas function confint() gives the profile-likelihood limits. |\n| Hypothesis tests for regression coefficients | Default | Default | Wald tests, which are based on estimates for the regression coefficients and its corresponding standard error. | |\n\n: Standard Logistic Regression in SAS vs R {#tbl-1}\n\n## Numerical Comparison {#sec-num-comp}\n\nEvery effort is made to ensure that the R code employs estimation methods/ optimization algorithms/ other components that closely match (as much as possible) those used in the SAS code.\n\n### `glm` in R\n\nNote, the default fitting method in `glm` is consistent with the default fitting method in `PROC LOGISTIC` procedure.\n\n- Default fitting method in `glm` is iteratively reweighted least squares, and the documentation can be found [here](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/glm).\n- Default fitting method for `PROC LOGISTIC` procedure is Fisher's scoring method, which is reported as part of the SAS default output, and it is equivalent to \"Iteratively reweighted least squares\" method as reported in this [documentation](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_logistic_sect033.htm).\n\n\n::: {.cell}\n\n```{.r .cell-code}\nm1 <- stats::glm(\n wt_grp ~ age + sex + ph.ecog + meal.cal,\n data = lung2,\n family = binomial(link = \"logit\")\n)\n\n# model coefficients summary\nsummary(m1)$coefficients\n```\n:::\n\n\nNote, function `confint.default` gives the Wald confidence limits, which is the default option in SAS `PROC LOGISTIC` procedure; whereas `confint` gives the profile-likelihood limits. Conditional odds ratio is calculated by taking the exponential of the model parameters.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncbind(est = coef(m1), confint.default(m1))\n```\n:::\n\n\n### `PROC LOGISTIC` in SAS (without firth option)\n\n```sas\nPROC LOGISTIC DATA=LUNG2; # import lung\n\tMODEL WT_GRP(EVENT=\"weight_gain\") = AGE SEX PH_ECOG MEAL_CAL;\n\tods output ESTIMATEs=estimates;\nrun;\n```\n\nBelow is screenshot of output tables summarizing coefficient estimates and confidence intervals\n\n![](../images/logistic_regression/sas_logistic_estimates.png){fig-align=\"left\"}\n\n![](../images/logistic_regression/sas_logistic_ci.png){fig-align=\"left\"}\n\n### Comment on model selection\n\nAs indicated in [Logistic regression in R](https://psiaims.github.io/CAMIS/R/logistic_regr.html) and [Logistic regression in SAS](https://psiaims.github.io/CAMIS/SAS/logistic-regr.html), the chi-Sq test statistics and p-values are different when performing model selections in R vs. SAS. The reason for this discrepancy is that the chi-Sq statistics from `anova()` in R is based on deviance test using residual deviance while the chi-Sq statistics from `PROC LOGISTIC` w/ `SELECTION` option in SAS is based on Wald test using z-values squared.\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Conclusion for logistic regression\n\nExact match (at 0.001 level) can be obtained using `glm` in R vs `PROC LOGISTIC` procedure (without Firth option) in SAS, for coefficient estimates, 95% CI, and for p-value.\n:::\n\n# Firth logistic regression\n\nThe following set of tables compare how to configure particular parameters / attributes of the methodologies.\n\n## Parameterisation Comparison\n\n| Attribute | SAS
`PROC LOGISTIC` w/ Firth option | R
`logistf::logistf` | Description | Note |\n|:-------------:|:--------------|:--------------|:--------------|:--------------|\n| Likelihood optimization algorithm | Default | `control =`
`logistf.control`
`(fit =“IRLS”)` | Fisher's scoring method (i.e., iteratively reweighted least squares (IRLS)) | |\n| Likelihood optimization algorithm | `TECHNIQUE = NEWTON` | Default | Newton-Raphson algorithm | |\n| Convergence criteria | Default | NA | Specifies relative gradient convergence criterion (GCONV=1E--8). | In`PROC LOGISTIC` there are three other convergence criteria which can be specified. If more than one convergence criterion is specified, the optimization is terminated as soon as one of the criteria is satisfied. |\n| Convergence criteria | NA | Default | Specifies three criteria that need to be met: the change in log likelihood is less than lconv (default is 1E-5), the maximum absolute element of the score vector is less than gconv (default is 1E-5), and the maximum absolute change in beta is less than xconv (default is 1E-5). | The gconv criteria in `logistif` is different from `GCONV` in SAS. The lconv criteria is also not exactly the same as the `ABSFCONV` or `FCONV` in `PROC LOGISTIC` in SAS, although the criteria use log likelihood. However, the `xconv` in R and `XCONV` in SAS seems to be consistent. |\n| Convergence criteria | `XCONV = 1E–8` | `control = logistf.control( xconv = 1E–8, lconv = 1, gconv = 1)` | Specifies the maximum absolute change in beta \\< 1E--8. | In `logistf`, three convergence criteria are checked at the same time. So here we use a large convergence criteria value for `lconv` and `gconv` to mimic the scenario where only `xconv` is checked. |\n| Confidence interval (CI) estimation method | Default | `pl= FALSE` | Wald CI | For `logistf`: \"Note that from version 1.24.1 on, the variance-covariance matrix is based on the second derivative of the likelihood of the augmented data rather than the original data, which proved to be a better approximation if the user chooses to set a higher value for the penalty strength.\" This could cause differences in standard error estimates in R vs SAS for Firth logistic regression, and consequently results in differences in the corresponding Wald CI estimates and hypothesis tests results (e.g., p-values). |\n| Confidence interval (CI) estimation method | `CLPARM = PL`
`CLODDS = PL` | Default | Profile likelihood-based CI | For Firth's bias-reduced logistic regression, it makes more sense to use penalized likelihood-based CI so it is consistent with the parameter estimation method which uses penalized maximum likelihood. |\n| Hypothesis tests for regression coefficients | Default | pl= FALSE | Wald tests, which are based on estimates for the regression coefficients and its corresponding standard error. | |\n| Hypothesis tests for regression coefficients | NA | Default | \"Likelihood ratio tests\", which are based on profile penalized log likelihood. | In SAS, when the model statement option `CLPARM = PL` is specified, the CI will be calculated based on profile likelihood. However, the hypothesis testing method is still a Wald method. This could cause results mismatch in the p-value. |\n\n: Firth's Bias-Reduced Logistic Regression in SAS vs R {#tbl-2}\n\n## Numerical Comparison\n\nNote that while Firth logistic regression is not required for our example dataset nonetheless we use it for demonstration purposes only.\n\n### `logistf` in R\n\n- By default, the [convergence criteria in `logistf`](https://cran.r-project.org/web/packages/logistf/logistf.pdf) specifies that three criteria need to be met at the same time, i.e., the change in log likelihood is less than lconv (default is 1E-5), the maximum absolute element of the score vector is less than gconv (default is 1E-5), and the maximum absolute change in beta is less than xconv (default is 1E-5). In SAS, the [default convergence criteria in `PROC LOGISTIC`](https://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_logistic_sect034.htm) specifies relative gradient convergence criterion (GCONV=1E--8); while SAS also support three other convergence criteria but when there are more than one convergence criterion specified, the optimization is terminated as soon as one of the criteria is satisfied. By looking at the R pacakge/SAS documentation, the `gconv` criteria in `logistif` function is different from the `GCONV` in SAS. The `lconv` criteria is also not exactly the same as the `ABSFCONV` or `FCONV` in PROC LOGISTIC in SAS, although the criteria use log likelihood. However, similar convergence criteria might be obtained by using the maximum absolute change in parameter estimates (i.e., `xconv` in R and SAS). Therefore, for comparison with the SAS output, in `logistf` function, we use a large convergence criteria value for `lconv` and `gconv` to mimic the scenario where only `xconv` is checked, i.e., specify `logistf.control(xconv = 0.00000001, gconv = 1, lconv = 1)` for the `control` argument.\n\n- By default, `logistf` function in R computes the confidence interval estimates and hypothesis tests (including p-value) for each parameter based on profile likelihood, which is also reported in the output below. However, Wald method (confidence interval and tests) can be specified by specifying the `control` argument with [`pl = FALSE`](https://cran.r-project.org/web/packages/logistf/logistf.pdf).\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfirth_mod <- logistf(\n wt_grp ~ age + sex + ph.ecog + meal.cal,\n data = lung2,\n control = logistf.control(\n fit = \"IRLS\",\n xconv = 0.00000001,\n gconv = 1,\n lconv = 1\n )\n)\nsummary(firth_mod)$coefficients\n\n## Code below would give Wald CI and tests results by adding `pl = FALSE`\n# logistf(..., pl = FALSE)\n```\n:::\n\n\nNote, function `confint` gives the profile-likelihood limits. Given the parameters from Firth's bias-reduced logistic regression is estimated using penalized maximum likelihood, `confint` function is used. Conditional odds ratio is calculated by taking the exponential of the model parameters.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncbind(est = coef(firth_mod), confint(firth_mod))\n```\n:::\n\n\n### `PROC LOGISTIC` in SAS (with firth option)\n\n- Note, by default, SAS computes confidence interval based on Wald tests. Given the parameters from Firth's method is estimated using penalized maximum likelihood, below specifies CLODDS = PL CLPARM=PL (based on profile likelihood), which is consistent with the maximization method and the R code above. However, the [default hypothesis test for the regression coefficients](https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_logistic_details50.htm) is still a Wald test, and the Chi-square statistics is calculated based on coefficient estimate and its corresponding standard error.\n\n- `XCONV` specifies relative parameter convergence criterion, which should correspond to the `xconv` in `logistf` function in R. We specify `XCONV = 0.00000001` so it should be consistent with the R code above.\n\n```sas\nPROC LOGISTIC DATA=LUNG2;\n\tMODEL WT_GRP(EVENT=\"weight gain\") = AGE SEX PH_ECOG MEAL_CAL / firth \n clodds=PL clparm=PL xconv = 0.00000001;\n\tods output ESTIMATEs=estimates;\nrun;\n```\n\nBelow is screenshot of output tables summarizing coefficient estimates and it's 95% CI\n\n![](../images/logistic_regression/sas_logistic_firth_estimates.png){fig-align=\"left\"}\n\n![](../images/logistic_regression/sas_logistic_firth_ci.png){fig-align=\"left\"}\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Conclusion for Firth logistic regression\n\nExact match cannot be obtained for all estimates using `logistf` vs `PROC LOGISTIC` procedure with Firth option. More specifically:\\\n- Coefficient estimate and its 95% CI matched at 0.001 level;\\\n- Standard error are not the same (e.g., 0.02023 for age in R vs 0.02065 in SAS);\\\n- p-value is not the same (0.6288 in R for age vs 0.6348 in SAS);\\\n:::\n\n# g-computation with covariate adjustment\n\nWe compare two implementions of g-computation in SAS:\n\n1. The \"Predictive margins and average marginal effects\" [%margins](https://support.sas.com/kb/63/038.html#pur) macro. The %margins macro uses \"the delta method \\[...\\] to determine the standard errors for predictive margins and marginal effects\". Note that the %margins macro uses the `PROC GENMOD` procedure to implement the working logistic regression model and require another macro [%NLEST](https://support.sas.com/kb/58/775.html) to calculate contrasts that requires delta methodl such as risk ratio or odds ratio.\n2. The SAS code provided in the appendix of the [Ge et al. (2011)](https://journals.sagepub.com/doi/10.1177/009286151104500409) implements the method outlined in the associated paper and simulations. Note: the Ge et al. (2011) macro uses the `PROC LOGISTIC` procedure to implement the working logistic regression model. `PROC IML` is used to calculate the delta method to determine the standard errors.\n\n## Numerical Comparison\n\n### `get_marginal_effect` in R\n\nWe fit a logistic regression model with covariate adjustment to estimate the marginal treatment effect using the delta method for variance estimation: as outlined in Ge et al (2011).\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## fit the model including model based variance estimation with delta method\nfit1 <- stats::glm(aval ~ trtp + bl_cov, family = \"binomial\", data = trial01) |>\n beeca::get_marginal_effect(\n trt = \"trtp\",\n method = \"Ge\",\n contrast = \"diff\",\n reference = \"0\",\n type = \"model-based\"\n )\n\n\"Marginal treatment effect\"\nfit1$marginal_est\n\n\"Standard error\"\nfit1$marginal_se\n```\n:::\n\n\n### `%Margins` macro in SAS\n\nWe now use the SAS [`%Margins`](https://support.sas.com/kb/63/038.html) macro to perform the Ge et al. (2011) method on `trial01` to estimate the marginal risk difference and it's standard error.\n\n```sas\n%Margins(data = myWork.trial01,\n class = trtp,\n classgref = first, /*Set reference to first level*/\n response = avaln,\n roptions = event='1', /*Ensure event is set to 1 = Yes */\n dist = binomial, \n model = trtp bl_cov,\n margins = trtp, \n options = cl diff reverse, /*Specify risk difference contrast and \n direction of treatment effect is correct*/\n link = logit); /*Specify logit link function */\n\t\n** Store output data sets ; \ndata myWork.margins_trt_estimates;\n set work._MARGINS;\nrun;\n\ndata myWork.margins_trt_diffs;\n set work._DIFFSPM;\nrun;\n```\n\n![](../images/logistic_regression/sas_logistic_gcomp_margins.png){fig-align=\"left\"}\n\n### `%LR` macro in SAS (Ge et al, 2011)\n\n```sas\n%LR(data = myWork.trial01, /* input data set */\n\tvar1 = bl_cov, /* continuous covariates in the logistic regression */\n\tvar2 = trtp, /* categorical covariates in the logistic regression */\n\tp1 = 1, /* number of continuous covariates in the logistic regression */\n\tp2 = 1, /* number of categorical covariates in the logistic regression */\n\tresp = avaln, /* binary response variable in the logistic regression */\n\tntrt = 1); /* position of the treatment variable in the categorical covariates */\n\t\ndata myWork.ge_macro_trt_diffs;\n set work.geout;\nrun;\n```\n\n![](../images/logistic_regression/sas_logistic_gcomp_ge.png){fig-align=\"left\"}\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Conclusion for g-computation with covariate adjustment\n\nExact match at the 0.001 level.\n:::\n\n# Final remarks\n\nIn summary, there are a few things to be aware of when comparing logistic regression results in R vs SAS. It is crucial to carefully manage the input parameters for each model to ensure they are configured similarly for logistic regression analyses. As highlighted also in [Logistic Regression in SAS](https://psiaims.github.io/CAMIS/SAS/logistic-regr.html), the variable parameterization is also important for modelling and interpretation, ensuring the types of variable (continuous vs. categorical) and reference values of categorical variable are applied as expected.\n\n1. **Likelihood optimization method**\n\n- The default likelihood optimization method in `glm` and `PROC LOGISTIC` is the same (i.e., Fisher's scoring method or iteratively reweighted least squares (IRLS)).\n\n- However, the default optimization method in `logistf` is Newton-Raphson, which can be modified into IRLS via `control = logistf.control(fit = “IRLS”)`. Alternatively, one could specify `technique = newton` in the model statement in SAS to modify the likelihood optimization method.\n\n2. **Convergence criteria**\n\n- Although both SAS and R allows options to modify the convergence criteria, the criteria does not seem to be exactly the same, which could cause results mismatch in some scenarios.\n\n- The [default convergence criteria in `PROC LOGISTIC`](https://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_logistic_sect034.htm) specifies the relative gradient convergence criterion; where the [default convergence criteria in `glm`](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/glm.control.html) specifies relative difference between deviance.\n\n- The default setting in logistf have checked more than one convergence criterion in its algorithm (i.e., [change in log likelihood, derivative of the log likelihood and parameter estimates](https://cran.r-project.org/web/packages/logistf/logistf.pdf)). One could specify a very large value for two of the criteria in order to mimic the scenario where only one criterion is checked (e.g., `control = logistf.control (xconv = 0.00000001, lconv = 1, gconv = 1`) in `logistf` in R should be consistent to the option of `xconv = 0.00000001` in SAS).\n\n3. **Confidence interval**\n\n- The `confint()` function in R will computes profile likelihood based CI for `glm` fitted model. However, in SAS, the default confidence interval is Wald CI. To match the default CI calculation in SAS for `glm` fitted model, use `confint.default()` function in R.\n\n- Nevertheless, Firth's biased-reduced logistic regression estimates parameter using penalized maximum likelihood, it makes more sense to use `confint()` function for `logistf` fitted model. In the meantime, in SAS, when fitting a Firth's logistic regression, it is also better to specify the model statement option `clparm = pl` which will also generate profile penalized likelihood CI.\n\n- We shall note that in the Firth logistic regression numerical example, the estimated standard errors does not match, but the CIs match at 0.001 level. This is because the CI was estimated based on profile penalized likelihood in R and SAS, and please see the next discussion point for potential reasons about differences between the estimated standard error. (I have compared Wald CIs estimated in R vs SAS, which could not match. This make sense as Wald CIs are calculated based on the estimated standard errors.)\n\n4. **Hypothesis test and p-value**\n\n- The default hypothesis tests for the regression coefficients are the same in `glm` and `PROC LOGISTIC`, which are both Wald tests and calculated based on estimates for the regression coefficients and its corresponding standard error.\n\n- As for `logistf` function, the default hypothesis testing method is based on profile penalized log likelihood (source code [here](https://github.com/georgheinze/logistf/blob/master/R/logistf.R)). And it was noted in the [R documentation](https://cran.r-project.org/web/packages/logistf/logistf.pdf) that, *\"from version 1.24.1 on, the variance-covariance matrix is based on the second derivative of the likelihood of the augmented data rather than the original data, which proved to be a better approximation if the user chooses to set a higher value for the penalty strength.\"* This could cause difference in the estimate of standard error in R vs SAS for Firth logistic regression, and consequently results in differences in the corresponding Wald CI estimates and hypothesis tests results (e.g., p-values).\n\n- Wald method can be used in a `logistf` function in R by specifying `pl = FALSE` in the `control` argument, which should correspond to the method used in SAS to calculate p-value. However, when specifying `pl = FALSE`, the CI is also calculated using Wald method.\n\n# Reference\n\n- A relevant blog [here](https://sas-and-r.blogspot.com/2010/11/example-815-firth-logistic-regression.html) (check comments in the blog).\n- [PROC LOGISTIC statement documentation in SAS](https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_logistic_sect004.htm).\n- [Reference manual for `logistf` package in R](https://cran.r-project.org/web/packages/logistf/logistf.pdf).\n- [GitHub repository for `logistf` package in R](https://github.com/georgheinze/logistf).\n- [GitHub repository for a SAS procedure about Firth logistic regression authored by the author of `logistf` R package](https://github.com/georgheinze/flicflac/tree/master/LogisticRegression), which was based on PROC IML instead of PROC LOGISTIC and was probably authored before the availability of Firth option in PROC LOGISTIC statement in SAS.\n- Ge, Miaomiao, et al. \"Covariate-adjusted difference in proportions from clinical trials using logistic regression and weighted risk differences.\" Drug information journal: DIJ/Drug Information Association 45 (2011): 481-493.\n- SAS Institute Inc. [\"Predictive margins and average marginal effects.\"](https://support.sas.com/kb/63/038.html) (Last Published: 13 Dec 2023)", + "supporting": [ + "r-sas_logistic-regr_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_mcnemar/execute-results/html.json b/_freeze/Comp/r-sas_mcnemar/execute-results/html.json index 43f5e6b5c..12c8991ec 100644 --- a/_freeze/Comp/r-sas_mcnemar/execute-results/html.json +++ b/_freeze/Comp/r-sas_mcnemar/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "8f8d04defa0a8b2c677cddade94f26f1", + "hash": "4461d11ad8af1cabbc46387fee603562", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R v SAS McNemar's test\"\nexecute: \n eval: false\n---\n\n## Introduction\n\nMcNemar's test is a test of marginal homogeneity. That is used with 2x2 contingency tables, when both x and y are binary factors.\n\n## General Comparison Table\n\nThe following table provides an overview of the support and results comparability between R and SAS for the new analysis point.\n\n| Analysis | Supported in R | Supported in SAS | Results Match | Notes |\n|---------------|---------------|---------------|---------------|---------------|\n| McNemar's Chi-Squared test | [Yes](../R/mcnemar.qmd) | [Yes](../SAS/mcnemar.qmd) | ✅ | By default SAS doesn't include the continuity correction. In R use {stats} or {coin} |\n| Cohen's Kappa CI | [Yes](../R/mcnemar.qmd) | [Yes](../SAS/mcnemar.qmd) | ✅ | In R use {vcd} |\n\nIn R,the {stats} or the {coin} package can be used to calculate McNemar. The {coin} package has the same defaults as SAS. But, using either of these packages, the first step is to calculate a frequency table, using the table function.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(coin)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nLoading required package: survival\n```\n\n\n:::\n\n```{.r .cell-code}\ncolds <- read.csv(\n file = \"../data/colds.csv\"\n)\nfreq_tbl <- table(\"age12\" = colds$age12, \"age14\" = colds$age14)\nfreq_tbl\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n age14\nage12 No Yes\n No 707 256\n Yes 144 212\n```\n\n\n:::\n\n```{.r .cell-code}\ncoin::mh_test(freq_tbl)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test\n\ndata: response by\n\t conditions (age12, age14) \n\t stratified by block\nchi-squared = 31.36, df = 1, p-value = 2.144e-08\n```\n\n\n:::\n:::\n\n\nIn order to get Cohen's Kappa an additional package is needed.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(vcd)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nLoading required package: grid\n```\n\n\n:::\n\n```{.r .cell-code}\ncohen_kappa <- vcd::Kappa(freq_tbl)\ncohen_kappa\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n value ASE z Pr(>|z|)\nUnweighted 0.2999 0.02733 10.97 5.07e-28\nWeighted 0.2999 0.02733 10.97 5.07e-28\n```\n\n\n:::\n\n```{.r .cell-code}\nconfint(cohen_kappa, level = 0.95)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n \nKappa lwr upr\n Unweighted 0.2463654 0.3534966\n Weighted 0.2463654 0.3534966\n```\n\n\n:::\n:::\n\n\nThe FREQ procedure can be used in SAS with the AGREE option to run the McNemar test, with OR, and RISKDIFF options stated for production of odds ratios and risk difference. These options were added as `epibasix::mcNemar` outputs the odds ratio and risk difference with confidence limits as default. In contrast to R, SAS outputs the Kappa coefficients with confident limits as default.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc freq data=colds;\n tables age12*age14 / agree or riskdiff;\nrun;\n```\n:::\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/mcnemar/sas-mcnemar.png){fig-align='center' width=40%}\n:::\n:::\n\n\n## Summary and Recommendation\n\nWhen calculating the odds ratio and risk difference confidence limits, SAS is not treating the data as matched-pairs. There is advice on the SAS blog and SAS support page to amend this, which requires a lot of additional coding.\n\n{stats} is using Edward's continuity correction by default, but this can be removed. In contrast, there is no option to include Edward's continuity correction in SAS, but this can be manually coded to agree with R. However, its use is controversial due to being seen as overly conservative.\n\nThere is another R package that is sometimes used to calculate McNemar's, called `epibasix`. This package is no longer being maintained, and there was no documentation available for certain methods used. Therefore, the use of the `epibasix` package is advised against and other packages may be more suitable.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-24\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P coin * 1.4-3 2023-09-27 [?] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::", + "markdown": "---\ntitle: \"R v SAS McNemar's test\"\n---\n\n## Introduction\n\nMcNemar's test is a test of marginal homogeneity. That is used with 2x2 contingency tables, when both x and y are binary factors.\n\n## General Comparison Table\n\nThe following table provides an overview of the support and results comparability between R and SAS for the new analysis point.\n\n| Analysis | Supported in R | Supported in SAS | Results Match | Notes |\n|---------------|---------------|---------------|---------------|---------------|\n| McNemar's Chi-Squared test | [Yes](../R/mcnemar.qmd) | [Yes](../SAS/mcnemar.qmd) | ✅ | By default SAS doesn't include the continuity correction. In R use {stats} or {coin} |\n| Cohen's Kappa CI | [Yes](../R/mcnemar.qmd) | [Yes](../SAS/mcnemar.qmd) | ✅ | In R use {vcd} |\n\nIn R,the {stats} or the {coin} package can be used to calculate McNemar. The {coin} package has the same defaults as SAS. But, using either of these packages, the first step is to calculate a frequency table, using the table function.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(coin)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nLoading required package: survival\n```\n\n\n:::\n\n```{.r .cell-code}\ncolds <- read.csv(\n file = \"../data/colds.csv\"\n)\nfreq_tbl <- table(\"age12\" = colds$age12, \"age14\" = colds$age14)\nfreq_tbl\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n age14\nage12 No Yes\n No 707 256\n Yes 144 212\n```\n\n\n:::\n\n```{.r .cell-code}\ncoin::mh_test(freq_tbl)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test\n\ndata: response by\n\t conditions (age12, age14) \n\t stratified by block\nchi-squared = 31.36, df = 1, p-value = 2.144e-08\n```\n\n\n:::\n:::\n\n\nIn order to get Cohen's Kappa an additional package is needed.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(vcd)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nLoading required package: grid\n```\n\n\n:::\n\n```{.r .cell-code}\ncohen_kappa <- vcd::Kappa(freq_tbl)\ncohen_kappa\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n value ASE z Pr(>|z|)\nUnweighted 0.2999 0.02733 10.97 5.07e-28\nWeighted 0.2999 0.02733 10.97 5.07e-28\n```\n\n\n:::\n\n```{.r .cell-code}\nconfint(cohen_kappa, level = 0.95)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n \nKappa lwr upr\n Unweighted 0.2463654 0.3534966\n Weighted 0.2463654 0.3534966\n```\n\n\n:::\n:::\n\n\nThe FREQ procedure can be used in SAS with the AGREE option to run the McNemar test, with OR, and RISKDIFF options stated for production of odds ratios and risk difference. These options were added as `epibasix::mcNemar` outputs the odds ratio and risk difference with confidence limits as default. In contrast to R, SAS outputs the Kappa coefficients with confident limits as default.\n\n```sas\nproc freq data=colds;\n tables age12*age14 / agree or riskdiff;\nrun;\n```\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/mcnemar/sas-mcnemar.png){fig-align='center' width=40%}\n:::\n:::\n\n\n## Summary and Recommendation\n\nWhen calculating the odds ratio and risk difference confidence limits, SAS is not treating the data as matched-pairs. There is advice on the SAS blog and SAS support page to amend this, which requires a lot of additional coding.\n\n{stats} is using Edward's continuity correction by default, but this can be removed. In contrast, there is no option to include Edward's continuity correction in SAS, but this can be manually coded to agree with R. However, its use is controversial due to being seen as overly conservative.\n\nThere is another R package that is sometimes used to calculate McNemar's, called `epibasix`. This package is no longer being maintained, and there was no documentation available for certain methods used. Therefore, the use of the `epibasix` package is advised against and other packages may be more suitable.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P coin * 1.4-3 2023-09-27 [?] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/Comp/r-sas_mmrm/execute-results/html.json b/_freeze/Comp/r-sas_mmrm/execute-results/html.json index 865ee598a..e85a82de0 100644 --- a/_freeze/Comp/r-sas_mmrm/execute-results/html.json +++ b/_freeze/Comp/r-sas_mmrm/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "402c5e7d123c00390fd87e00943d50a3", + "hash": "5737252e803780338f43a1dc0b2723d4", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS MMRM\"\nmessage: false\nwarning: false\necho: true\neval: false\n---\n\n\n\n\n\n# Introduction\n\nIn this vignette we briefly compare the `mmrm::mmrm`, SAS's `PROC GLIMMIX`, `nlme::gls`, `lme4::lmer`, and `glmmTMB::glmmTMB` functions for fitting mixed models for repeated measures (MMRMs). A primary difference in these implementations lies in the covariance structures that are supported \"out of the box\". In particular, `PROC GLIMMIX` and `mmrm` are the only procedures which provide support for many of the most common MMRM covariance structures. Most covariance structures can be implemented in `gls`, though users are required to define them manually. `lmer` and `glmmTMB` are more limited. We find that `mmmrm` converges more quickly than other R implementations while also producing estimates that are virtually identical to `PROC GLIMMIX`'s.\n\nNOTE: that factor parameterization in the model, and the default order that SAS and R choose reference levels for factors are different. Hence, the first thing to ensure when trying to replicate MMRMs in SAS and R is that you have these options aligned. See [parameterization in SAS](https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_mixed_sect023.htm) for more detail on SAS parameterization. This can be matched in R using the default `contr.treatment` option. It is reccommended to specify the level of any factors you want to use as the reference in both SAS and R.\n\nIn SAS this is done on the class row: 'class armcd(ref=\"ARM A\")' \nIn R, this is done using `relevel(ARMCD,ref=\"ARM A\")` in addition to adding the base option to the contrast() statement when selecting the contr.treatment parameterization. \n`contrasts(ARMCD) <- contr.treatment(levels(ARMCD), base=which(levels(ARMCD)==\"ARM A\"))`\n\n# Datasets\n\nTwo datasets are used to illustrate model fitting with the `mmrm`, `lme4`, `nlme`, `glmmTMB` R packages as well as `PROC GLIMMIX`. These data are also used to compare these implementations' operating characteristics.\n\n## FEV Data\n\nThe FEV dataset contains measurements of FEV1 (forced expired volume in one second), a measure of how quickly the lungs can be emptied. Low levels of FEV1 may indicate chronic obstructive pulmonary disease (COPD). It is summarized below.\n\n```default\n Stratified by ARMCD\n Overall PBO TRT\n n 800 420 380\n USUBJID (%)\n PT[1-200] 200 105 (52.5) 95 (47.5)\n AVISIT\n VIS1 200 105 95\n VIS2 200 105 95\n VIS3 200 105 95\n VIS4 200 105 95\n RACE (%)\n Asian 280 (35.0) 152 (36.2) 128 (33.7)\n Black or African American 300 (37.5) 184 (43.8) 116 (30.5)\n White 220 (27.5) 84 (20.0) 136 (35.8)\n SEX = Female (%) 424 (53.0) 220 (52.4) 204 (53.7)\n FEV1_BL (mean (SD)) 40.19 (9.12) 40.46 (8.84) 39.90 (9.42)\n FEV1 (mean (SD)) 42.30 (9.32) 40.24 (8.67) 44.45 (9.51)\n WEIGHT (mean (SD)) 0.52 (0.23) 0.52 (0.23) 0.51 (0.23)\n VISITN (mean (SD)) 2.50 (1.12) 2.50 (1.12) 2.50 (1.12)\n VISITN2 (mean (SD)) -0.02 (1.03) 0.01 (1.07) -0.04 (0.98)\n```\n\n## BCVA Data\n\nThe BCVA dataset contains data from a randomized longitudinal ophthalmology trial evaluating the change in baseline corrected visual acuity (BCVA) over the course of 10 visits. BCVA corresponds to the number of letters read from a visual acuity chart. A summary of the data is given below:\n\n```default\n Stratified by ARMCD\n Overall CTL TRT\n n 8605 4123 4482\n USUBJID (%)\n PT[1-1000] 1000 494 (49.4) 506 (50.6)\n AVISIT\n VIS1 983 482 501\n VIS2 980 481 499\n VIS3 960 471 489\n VIS4 946 458 488\n VIS5 925 454 471\n VIS6 868 410 458\n VIS7 816 388 428\n VIS8 791 371 420\n VIS9 719 327 392\n VIS10 617 281 336\n RACE (%)\n Asian 297 (29.7) 151 (30.6) 146 (28.9)\n Black or African American 317 (31.7) 149 (30.1) 168 (33.2)\n White 386 (38.6) 194 (39.3) 192 (37.9)\n BCVA_BL (mean (SD)) 75.12 (9.93) 74.90 (9.76) 75.40 (10.1)\n BCVA_CHG (mean (SD))\n VIS1 5.59 (1.31) 5.32 (1.23) 5.86 (1.33)\n VIS10 9.18 (2.91) 7.49 (2.58) 10.60 (2.36)\n```\n\n# Model Implementations {.tabset}\n\nListed below are some of the most commonly used covariance structures used when fitting MMRMs. We indicate which matrices are available \"out of the box\" for each implementation considered in this vignette. Note that this table is not exhaustive; `PROC GLIMMIX` and `glmmTMB` support additional spatial covariance structures.\n\n| Covariance structures | `mmrm` | `PROC GLIMMIX` | `gls` | `lmer` | `glmmTMB` |\n|:-------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Ante-dependence (heterogeneous) | X | X | | | |\n| Ante-dependence (homogeneous) | X | | | | |\n| Auto-regressive (heterogeneous) | X | X | X | | |\n| Auto-regressive (homogeneous) | X | X | X | | X |\n| Compound symmetry (heterogeneous) | X | X | X | | X |\n| Compound symmetry (homogeneous) | X | X | X | | |\n| Spatial exponential | X | X | X | | X |\n| Toeplitz (heterogeneous) | X | X | | | X |\n| Toeplitz (homogeneous) | X | X | | | |\n| Unstructured | X | X | X | X | X |\n\nCode for fitting MMRMs to the FEV data using each of the considered functions and covariance structures are provided below. Fixed effects for the visit number, treatment assignment and the interaction between the two are modeled.\n\n## Ante-dependence (heterogeneous)\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=ANTE(1);\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + adh(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n## Ante-dependence (homogeneous)\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + ad(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n## Auto-regressive (heterogeneous)\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=ARH(1);\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + ar1h(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corCAR1(form = ~ AVISIT | USUBJID),\n weights = nlme::varIdent(form = ~ 1 | AVISIT),\n na.action = na.omit\n)\n```\n:::\n\n\n## Auto-regressive (homogeneous)\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = ARMCD|AVISIT / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=AR(1);\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + ar1(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corCAR1(form = ~ AVISIT | USUBJID),\n na.action = na.omit\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + ar1(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n## Compound symmetry (heterogeneous)\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=CSH;\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + csh(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corCompSymm(form = ~ AVISIT | USUBJID),\n weights = nlme::varIdent(form = ~ 1 | AVISIT),\n na.action = na.omit\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + cs(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n## Compound symmetry (homogeneous)\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=CS;\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + cs(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corCompSymm(form = ~ AVISIT | USUBJID),\n na.action = na.omit\n)\n```\n:::\n\n\n## Spatial exponential\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM / subject=USUBJID type=sp(exp)(visitn) rcorr;\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + sp_exp(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = corExp(form = ~ AVISIT | USUBJID),\n weights = varIdent(form = ~ 1 | AVISIT),\n na.action = na.omit\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# NOTE: requires use of coordinates\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + exp(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n## Toeplitz (heterogeneous)\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=TOEPH;\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + toeph(AVISIT | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + toep(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n## Toeplitz (homogeneous)\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=TOEP;\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + toep(AVISIT | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n## Unstructured\n\n### `PROC GLIMMIX`\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = ARMCD|AVISIT / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=un;\nRUN;\n```\n:::\n\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + us(AVISIT | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corSymm(form = ~ AVISIT | USUBJID),\n weights = nlme::varIdent(form = ~ 1 | AVISIT),\n na.action = na.omit\n)\n```\n:::\n\n\n### `lmer`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlme4::lmer(\n FEV1 ~ ARMCD * AVISIT + (0 + AVISIT | USUBJID),\n data = fev_data,\n control = lme4::lmerControl(check.nobs.vs.nRE = \"ignore\"),\n na.action = na.omit\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + us(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n# Benchmarking\n\nNext, the MMRM fitting procedures are compared using the FEV and BCVA datasets. FEV1 measurements are modeled as a function of race, treatment arm, visit number, and the interaction between the treatment arm and the visit number. Change in BCVA is assumed to be a function of race, baseline BCVA, treatment arm, visit number, and the treatment--visit interaction. In both datasets, repeated measures are modeled using an unstructured covariance matrix. The implementations' convergence times are evaluated first, followed by a comparison of their estimates. Finally, we fit these procedures on simulated BCVA-like data to assess the impact of missingness on convergence rates.\n\n## Convergence Times\n\n### FEV Data\n\nThe `mmrm`, `PROC GLIMMIX`, `gls`, `lmer`, and `glmmTMB` functions are applied to the FEV dataset 10 times. The convergence times are recorded for each replicate and are reported in the table below.\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n\nTable: Comparison of convergence times: milliseconds\n\n|Implementation | Median| First Quartile| Third Quartile|\n|:--------------|------:|--------------:|--------------:|\n|mmrm | 56.15| 55.76| 56.30|\n|PROC GLIMMIX | 100.00| 100.00| 100.00|\n|lmer | 247.02| 245.25| 257.46|\n|gls | 687.63| 683.50| 692.45|\n|glmmTMB | 715.90| 708.70| 721.57|\n\n\n:::\n:::\n\n\nIt is clear from these results that `mmrm` converges significantly faster than other R functions. Though not demonstrated here, this is generally true regardless of the sample size and covariance structure used. `mmrm` is faster than `PROC GLIMMIX`.\n\n### BCVA Data\n\nThe MMRM implementations are now applied to the BCVA dataset 10 times. The convergence times are presented below.\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n\nTable: Comparison of convergence times: seconds\n\n|Implementation | Median| First Quartile| Third Quartile|\n|:--------------|------:|--------------:|--------------:|\n|mmrm | 3.36| 3.32| 3.46|\n|glmmTMB | 18.65| 18.14| 18.87|\n|PROC GLIMMIX | 36.25| 36.17| 36.29|\n|gls | 164.36| 158.61| 165.93|\n|lmer | 165.26| 157.46| 166.42|\n\n\n:::\n:::\n\n\nWe again find that `mmrm` produces the fastest convergence times on average.\n\n## Marginal Treatment Effect Estimates Comparison\n\nWe next estimate the marginal mean treatment effects for each visit in the FEV and BCVA datasets using the MMRM fitting procedures. All R implementations' estimates are reported relative to `PROC GLIMMIX`'s estimates. Convergence status is also reported.\n\n### FEV Data\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](../images/mmrm/review-treatment-fev-1.png){width=100%}\n:::\n:::\n\n\nThe R procedures' estimates are very similar to those output by `PROC GLIMMIX`, though `mmrm` and `gls` generate the estimates that are closest to those produced when using SAS. All methods converge using their default optimization arguments.\n\n### BCVA Data\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](../images/mmrm/review-treatment-bcva-1.png){width=100%}\n:::\n\n::: {.cell-output-display}\n![](../images/mmrm/review-treatment-bcva-2.png){width=100%}\n:::\n:::\n\n\n`mmrm`, `gls` and `lmer` produce estimates that are virtually identical to `PROC GLIMMIX`'s, while `glmmTMB` does not. This is likely explained by `glmmTMB`'s failure to converge. Note too that `lmer` fails to converge.\n\n## Impact of Missing Data on Convergence Rates\n\nThe results of the previous benchmark suggest that the amount of patients missing from later time points affect certain implementations' capacity to converge. We investigate this further by simulating data using a data-generating process similar to that of the BCVA datasets, though with various rates of patient dropout.\n\nTen datasets of 200 patients are generated each of the following levels of missingness: none, mild, moderate, and high. In all scenarios, observations are missing at random. The number patients observed at each visit is obtained for one replicated dataset at each level of missingness is presented in the table below.\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n\nTable: Number of patients per visit\n\n| | none| mild| moderate| high|\n|:-----|----:|-----:|--------:|-----:|\n|VIS01 | 200| 196.7| 197.6| 188.1|\n|VIS02 | 200| 195.4| 194.4| 182.4|\n|VIS03 | 200| 195.1| 190.7| 175.2|\n|VIS04 | 200| 194.1| 188.4| 162.8|\n|VIS05 | 200| 191.6| 182.5| 142.7|\n|VIS06 | 200| 188.2| 177.3| 125.4|\n|VIS07 | 200| 184.6| 168.0| 105.9|\n|VIS08 | 200| 178.5| 155.4| 82.6|\n|VIS09 | 200| 175.3| 139.9| 58.1|\n|VIS10 | 200| 164.1| 124.0| 39.5|\n\n\n:::\n:::\n\n\nThe convergence rates of all implementations for stratified by missingness level is presented in the plot below.\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](../images/mmrm/review-convergence-rate-missingness-1.png){width=100%}\n:::\n:::\n\n\n`mmrm`, `gls`, and `PROC GLIMMIX` are resilient to missingness, only exhibiting some convergence problems in the scenarios with the most missingness. These implementations converged in all the other scenarios' replicates. `glmmTMB`, on the other hand, has convergence issues in the no-, mild-, and high-missingness datasets, with the worst convergence rate occurring in the datasets with the most dropout. Finally, `lmer` is unreliable in all scenarios, suggesting that it's convergence issues stem from something other than the missing observations.\n\nNote that the default optimization schemes are used for each method; these schemes can be modified to potentially improve convergence rates.\n\nA more comprehensive simulation study using data-generating processes similar to the one used here is outlined in the [`simulations/missing-data-benchmarks`](https://github.com/openpharma/mmrm/tree/main/simulations/missing-data-benchmarks) subdirectory. In addition to assessing the effect of missing data on software convergence rates, we also evaluate these methods' fit times and empirical bias, variance, 95% coverage rates, type I error rates and type II error rates. `mmrm` is found to be the most most robust software for fitting MMRMs in scenarios where a large proportion of patients are missing from the last time points. Additionally, `mmrm` has the fastest average fit times regardless of the amount of missingness. All implementations considered produce similar empirical biases, variances, 95% coverage rates, type I error rates and type II error rates.\n", - "supporting": [], + "markdown": "---\ntitle: \"R vs SAS MMRM\"\nexecute:\n message: false\n warning: false\n echo: true\n eval: false\n---\n\n\n\n\n\n# Introduction\n\nIn this vignette we briefly compare the `mmrm::mmrm`, SAS's `PROC GLIMMIX`, `nlme::gls`, `lme4::lmer`, and `glmmTMB::glmmTMB` functions for fitting mixed models for repeated measures (MMRMs). A primary difference in these implementations lies in the covariance structures that are supported \"out of the box\". In particular, `PROC GLIMMIX` and `mmrm` are the only procedures which provide support for many of the most common MMRM covariance structures. Most covariance structures can be implemented in `gls`, though users are required to define them manually. `lmer` and `glmmTMB` are more limited. We find that `mmrm` converges more quickly than other R implementations while also producing estimates that are virtually identical to `PROC GLIMMIX`'s.\n\n::: {.callout-note}\nFactor parameterization in the model, and the default order that SAS and R choose reference levels for factors are different. Hence, the first thing to ensure when trying to replicate MMRMs in SAS and R is that you have these options aligned. See [parameterization in SAS](https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_mixed_sect023.htm) for more detail on SAS parameterization. This can be matched in R using the default `contr.treatment` option. It is recommended to specify the level of any factors you want to use as the reference in both SAS and R.\n\nIn SAS, this is done on the class row:\n\n```default\nclass armcd(ref=\"ARM A\")\n```\n\nIn R, this is done using `relevel(ARMCD, ref = \"ARM A\")` in addition to adding the base option to the `contrast()` statement when selecting the `contr.treatment` parameterization:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncontrasts(ARMCD) <- contr.treatment(levels(ARMCD), base = which(levels(ARMCD) == \"ARM A\"))\n```\n:::\n\n:::\n\n# Datasets\n\nTwo datasets are used to illustrate model fitting with the `mmrm`, `lme4`, `nlme`, `glmmTMB` R packages as well as `PROC GLIMMIX`. These data are also used to compare these implementations' operating characteristics.\n\n## FEV Data\n\nThe FEV dataset contains measurements of FEV1 (forced expired volume in one second), a measure of how quickly the lungs can be emptied. Low levels of FEV1 may indicate chronic obstructive pulmonary disease (COPD). It is summarized below.\n\n```default\n Stratified by ARMCD\n Overall PBO TRT\n n 800 420 380\n USUBJID (%)\n PT[1-200] 200 105 (52.5) 95 (47.5)\n AVISIT\n VIS1 200 105 95\n VIS2 200 105 95\n VIS3 200 105 95\n VIS4 200 105 95\n RACE (%)\n Asian 280 (35.0) 152 (36.2) 128 (33.7)\n Black or African American 300 (37.5) 184 (43.8) 116 (30.5)\n White 220 (27.5) 84 (20.0) 136 (35.8)\n SEX = Female (%) 424 (53.0) 220 (52.4) 204 (53.7)\n FEV1_BL (mean (SD)) 40.19 (9.12) 40.46 (8.84) 39.90 (9.42)\n FEV1 (mean (SD)) 42.30 (9.32) 40.24 (8.67) 44.45 (9.51)\n WEIGHT (mean (SD)) 0.52 (0.23) 0.52 (0.23) 0.51 (0.23)\n VISITN (mean (SD)) 2.50 (1.12) 2.50 (1.12) 2.50 (1.12)\n VISITN2 (mean (SD)) -0.02 (1.03) 0.01 (1.07) -0.04 (0.98)\n```\n\n## BCVA Data\n\nThe BCVA dataset contains data from a randomized longitudinal ophthalmology trial evaluating the change in baseline corrected visual acuity (BCVA) over the course of 10 visits. BCVA corresponds to the number of letters read from a visual acuity chart. A summary of the data is given below:\n\n```default\n Stratified by ARMCD\n Overall CTL TRT\n n 8605 4123 4482\n USUBJID (%)\n PT[1-1000] 1000 494 (49.4) 506 (50.6)\n AVISIT\n VIS1 983 482 501\n VIS2 980 481 499\n VIS3 960 471 489\n VIS4 946 458 488\n VIS5 925 454 471\n VIS6 868 410 458\n VIS7 816 388 428\n VIS8 791 371 420\n VIS9 719 327 392\n VIS10 617 281 336\n RACE (%)\n Asian 297 (29.7) 151 (30.6) 146 (28.9)\n Black or African American 317 (31.7) 149 (30.1) 168 (33.2)\n White 386 (38.6) 194 (39.3) 192 (37.9)\n BCVA_BL (mean (SD)) 75.12 (9.93) 74.90 (9.76) 75.40 (10.1)\n BCVA_CHG (mean (SD))\n VIS1 5.59 (1.31) 5.32 (1.23) 5.86 (1.33)\n VIS10 9.18 (2.91) 7.49 (2.58) 10.60 (2.36)\n```\n\n# Model Implementations {.tabset}\n\nListed below are some of the most commonly used covariance structures used when fitting MMRMs. We indicate which matrices are available \"out of the box\" for each implementation considered in this vignette. Note that this table is not exhaustive; `PROC GLIMMIX` and `glmmTMB` support additional spatial covariance structures.\n\n| Covariance structures | `mmrm` | `PROC GLIMMIX` | `gls` | `lmer` | `glmmTMB` |\n|:-------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Ante-dependence (heterogeneous) | X | X | | | |\n| Ante-dependence (homogeneous) | X | | | | |\n| Auto-regressive (heterogeneous) | X | X | X | | |\n| Auto-regressive (homogeneous) | X | X | X | | X |\n| Compound symmetry (heterogeneous) | X | X | X | | X |\n| Compound symmetry (homogeneous) | X | X | X | | |\n| Spatial exponential | X | X | X | | X |\n| Toeplitz (heterogeneous) | X | X | | | X |\n| Toeplitz (homogeneous) | X | X | | | |\n| Unstructured | X | X | X | X | X |\n\nCode for fitting MMRMs to the FEV data using each of the considered functions and covariance structures are provided below. Fixed effects for the visit number, treatment assignment and the interaction between the two are modeled.\n\n## Ante-dependence (heterogeneous)\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=ANTE(1);\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + adh(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n## Ante-dependence (homogeneous)\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + ad(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n## Auto-regressive (heterogeneous)\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=ARH(1);\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + ar1h(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corCAR1(form = ~ AVISIT | USUBJID),\n weights = nlme::varIdent(form = ~ 1 | AVISIT),\n na.action = na.omit\n)\n```\n:::\n\n\n## Auto-regressive (homogeneous)\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = ARMCD|AVISIT / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=AR(1);\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + ar1(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corCAR1(form = ~ AVISIT | USUBJID),\n na.action = na.omit\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + ar1(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n## Compound symmetry (heterogeneous)\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=CSH;\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + csh(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corCompSymm(form = ~ AVISIT | USUBJID),\n weights = nlme::varIdent(form = ~ 1 | AVISIT),\n na.action = na.omit\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + cs(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n## Compound symmetry (homogeneous)\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=CS;\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + cs(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corCompSymm(form = ~ AVISIT | USUBJID),\n na.action = na.omit\n)\n```\n:::\n\n\n## Spatial exponential\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM / subject=USUBJID type=sp(exp)(visitn) rcorr;\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + sp_exp(VISITN | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = corExp(form = ~ AVISIT | USUBJID),\n weights = varIdent(form = ~ 1 | AVISIT),\n na.action = na.omit\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# NOTE: requires use of coordinates\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + exp(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n## Toeplitz (heterogeneous)\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=TOEPH;\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + toeph(AVISIT | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + toep(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n## Toeplitz (homogeneous)\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = AVISIT|ARMCD / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=TOEP;\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + toep(AVISIT | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n## Unstructured\n\n### `PROC GLIMMIX`\n\n```sas\nPROC GLIMMIX DATA = fev_data;\n CLASS AVISIT(ref = 'VIS1') ARMCD(ref = 'PBO') USUBJID;\n MODEL FEV1 = ARMCD|AVISIT / ddfm=satterthwaite solution chisq;\n RANDOM AVISIT / subject=USUBJID type=un;\nRUN;\n```\n\n### `mmrm`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm::mmrm(\n formula = FEV1 ~ ARMCD * AVISIT + us(AVISIT | USUBJID),\n data = fev_data\n)\n```\n:::\n\n\n### `gls`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nnlme::gls(\n formula = FEV1 ~ ARMCD * AVISIT,\n data = fev_data,\n correlation = nlme::corSymm(form = ~ AVISIT | USUBJID),\n weights = nlme::varIdent(form = ~ 1 | AVISIT),\n na.action = na.omit\n)\n```\n:::\n\n\n### `lmer`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlme4::lmer(\n FEV1 ~ ARMCD * AVISIT + (0 + AVISIT | USUBJID),\n data = fev_data,\n control = lme4::lmerControl(check.nobs.vs.nRE = \"ignore\"),\n na.action = na.omit\n)\n```\n:::\n\n\n### `glmmTMB`\n\n\n::: {.cell}\n\n```{.r .cell-code}\nglmmTMB::glmmTMB(\n FEV1 ~ ARMCD * AVISIT + us(0 + AVISIT | USUBJID),\n dispformula = ~0,\n data = fev_data\n)\n```\n:::\n\n\n# Benchmarking\n\nNext, the MMRM fitting procedures are compared using the FEV and BCVA datasets. FEV1 measurements are modeled as a function of race, treatment arm, visit number, and the interaction between the treatment arm and the visit number. Change in BCVA is assumed to be a function of race, baseline BCVA, treatment arm, visit number, and the treatment--visit interaction. In both datasets, repeated measures are modeled using an unstructured covariance matrix. The implementations' convergence times are evaluated first, followed by a comparison of their estimates. Finally, we fit these procedures on simulated BCVA-like data to assess the impact of missingness on convergence rates.\n\n## Convergence Times\n\n### FEV Data\n\nThe `mmrm`, `PROC GLIMMIX`, `gls`, `lmer`, and `glmmTMB` functions are applied to the FEV dataset 10 times. The convergence times are recorded for each replicate and are reported in the table below.\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n\nTable: Comparison of convergence times: milliseconds\n\n|Implementation | Median| First Quartile| Third Quartile|\n|:--------------|------:|--------------:|--------------:|\n|mmrm | 56.15| 55.76| 56.30|\n|PROC GLIMMIX | 100.00| 100.00| 100.00|\n|lmer | 247.02| 245.25| 257.46|\n|gls | 687.63| 683.50| 692.45|\n|glmmTMB | 715.90| 708.70| 721.57|\n\n\n:::\n:::\n\n\nIt is clear from these results that `mmrm` converges significantly faster than other R functions. Though not demonstrated here, this is generally true regardless of the sample size and covariance structure used. `mmrm` is faster than `PROC GLIMMIX`.\n\n### BCVA Data\n\nThe MMRM implementations are now applied to the BCVA dataset 10 times. The convergence times are presented below.\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n\nTable: Comparison of convergence times: seconds\n\n|Implementation | Median| First Quartile| Third Quartile|\n|:--------------|------:|--------------:|--------------:|\n|mmrm | 3.36| 3.32| 3.46|\n|glmmTMB | 18.65| 18.14| 18.87|\n|PROC GLIMMIX | 36.25| 36.17| 36.29|\n|gls | 164.36| 158.61| 165.93|\n|lmer | 165.26| 157.46| 166.42|\n\n\n:::\n:::\n\n\nWe again find that `mmrm` produces the fastest convergence times on average.\n\n## Marginal Treatment Effect Estimates Comparison\n\nWe next estimate the marginal mean treatment effects for each visit in the FEV and BCVA datasets using the MMRM fitting procedures. All R implementations' estimates are reported relative to `PROC GLIMMIX`'s estimates. Convergence status is also reported.\n\n### FEV Data\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](r-sas_mmrm_files/figure-html/review-treatment-fev-1.png){width=672}\n:::\n:::\n\n\nThe R procedures' estimates are very similar to those output by `PROC GLIMMIX`, though `mmrm` and `gls` generate the estimates that are closest to those produced when using SAS. All methods converge using their default optimization arguments.\n\n### BCVA Data\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](r-sas_mmrm_files/figure-html/review-treatment-bcva-1.png){width=672}\n:::\n\n::: {.cell-output-display}\n![](r-sas_mmrm_files/figure-html/review-treatment-bcva-2.png){width=672}\n:::\n:::\n\n\n`mmrm`, `gls` and `lmer` produce estimates that are virtually identical to `PROC GLIMMIX`'s, while `glmmTMB` does not. This is likely explained by `glmmTMB`'s failure to converge. Note too that `lmer` fails to converge.\n\n## Impact of Missing Data on Convergence Rates\n\nThe results of the previous benchmark suggest that the amount of patients missing from later time points affect certain implementations' capacity to converge. We investigate this further by simulating data using a data-generating process similar to that of the BCVA datasets, though with various rates of patient dropout.\n\nTen datasets of 200 patients are generated each of the following levels of missingness: none, mild, moderate, and high. In all scenarios, observations are missing at random. The number patients observed at each visit is obtained for one replicated dataset at each level of missingness is presented in the table below.\n\n\n::: {.cell}\n::: {.cell-output-display}\n\n\nTable: Number of patients per visit\n\n| | none| mild| moderate| high|\n|:-----|----:|-----:|--------:|-----:|\n|VIS01 | 200| 196.7| 197.6| 188.1|\n|VIS02 | 200| 195.4| 194.4| 182.4|\n|VIS03 | 200| 195.1| 190.7| 175.2|\n|VIS04 | 200| 194.1| 188.4| 162.8|\n|VIS05 | 200| 191.6| 182.5| 142.7|\n|VIS06 | 200| 188.2| 177.3| 125.4|\n|VIS07 | 200| 184.6| 168.0| 105.9|\n|VIS08 | 200| 178.5| 155.4| 82.6|\n|VIS09 | 200| 175.3| 139.9| 58.1|\n|VIS10 | 200| 164.1| 124.0| 39.5|\n\n\n:::\n:::\n\n\nThe convergence rates of all implementations for stratified by missingness level is presented in the plot below.\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](r-sas_mmrm_files/figure-html/review-convergence-rate-missingness-1.png){width=672}\n:::\n:::\n\n\n`mmrm`, `gls`, and `PROC GLIMMIX` are resilient to missingness, only exhibiting some convergence problems in the scenarios with the most missingness. These implementations converged in all the other scenarios' replicates. `glmmTMB`, on the other hand, has convergence issues in the no-, mild-, and high-missingness datasets, with the worst convergence rate occurring in the datasets with the most dropout. Finally, `lmer` is unreliable in all scenarios, suggesting that it's convergence issues stem from something other than the missing observations.\n\nNote that the default optimization schemes are used for each method; these schemes can be modified to potentially improve convergence rates.\n\nA more comprehensive simulation study using data-generating processes similar to the one used here is outlined in the [`simulations/missing-data-benchmarks`](https://github.com/openpharma/mmrm/tree/main/simulations/missing-data-benchmarks) subdirectory. In addition to assessing the effect of missing data on software convergence rates, we also evaluate these methods' fit times and empirical bias, variance, 95% coverage rates, type I error rates and type II error rates. `mmrm` is found to be the most most robust software for fitting MMRMs in scenarios where a large proportion of patients are missing from the last time points. Additionally, `mmrm` has the fastest average fit times regardless of the amount of missingness. All implementations considered produce similar empirical biases, variances, 95% coverage rates, type I error rates and type II error rates.\n", + "supporting": [ + "r-sas_mmrm_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_mmrm/figure-html/review-convergence-rate-missingness-1.png b/_freeze/Comp/r-sas_mmrm/figure-html/review-convergence-rate-missingness-1.png new file mode 100644 index 0000000000000000000000000000000000000000..bab25b7b53b59621d2ca692d4984985822b9d135 GIT binary patch literal 94566 zcmc$`cTkhv*ESkZK}0}6M5!8@bP?%Ik=_*n=?DT+LnjcrAVrFRfYgX|klst^y@cK& zp?3&9lmt$Ee17luo|!Xq-gCY`zBR*e&)xRE_u6}{Yh8P-gleeCliZ=X0{{R>6cuFN z0syyO0suFfZ{y?6KyJ!J;{FjiDCjx^0QA9s{_vt!56l68#{fl{SMR_nJM-=@DJ>ZZ z_mPII5GJ0NZ)9ZYYAg+FsZKpy-@_98|M^QFSGwS7 zgp01uB#!EsQN(I8h)?v=rx&s~jCnjhOahMOY)h(as)Rw}xRnOTajIzTeO;<#q5nf^JUXZIMVv z`{zDiu^6tcYqkT1{h6cGRA*{=XLD>qdbH@$&Sy55K!xKuq9kTKkmje~4+oQJl)R=4 z&}umyBuQ-x+6>@`y$7?~%sUI0_nVmdkAty%(co%00N(iJAJpCylR1hRN z#c^>(!K+u7(wl|A{$f*?$)qb6>>KI8DdFJ>GlN8dhf~q@k#v1_D%7}41Y?xltkeJx ztk@Rw1!ov<^Y9kKvPxh20|=h+9i3j87o+WkJUue&Mi5%gi^XV5c#6nT>Ur|7ASK8o zTPTmyQ8s5*nryUu_-xgAMcvASe{^mAG=SvFY z9(OFLi>h(N-4;#VyI30v5wV+zZ}g^KuC^0Rw`f0jOGG4E3XZ9!KN}C-*9AI0kIIg( zyCtuh`)9HIBzSeFg4u8d&D+={rpvsPc@m6uS=lHaRmyY+#?;+X#aN?SK!o6Z8D_~j z0rZhX;d# zf|(CpJz>Wqny%p9T3mZyj3dBx6HxxPuNG`i*iYVYa_I=Tsr9s^D3 z$Id+BG^nXONgk&zd*m1vZ-!);mA20u3Vc^+2J7@uO{%mWZHO%Oez-Jmh`e@YHT?eK zu7|lbwN7M+(_Y$?>w@jdctJ0@6l^r(Y_oEPP=}OJ7{Vl|Yp&l_?UB$Zku0Jd@_r_d zN@_2ZO38a9-YO^&;rFFA0iLk z%)|9Dot26K33FI_C!;dbHm35*9<{T%jO3^>)A;;L{bwKKXX2?+W@Jc67WDk`$RikR zrhbp#vPD;IX}mlKZv0hMZQ%ZQH3kxlzWdJHL{~b+5@%nd`)TuVAKw9)(~7;TH@@h} z*2q}x#;HJY@jcX-<00+maD!WIBZHWuwxWmr!d9^#_@Q<-Hg6xLIA8NUfzwWAZQ0wC z!j7%#P0W(P@ik_nj#Nx)(^kj(1Uc)pKV)hc>ONwrNcgXA3mwgjeydOp?=e-q%x7Lx_~eBI6gri-dL$w}x%Gjlxg*=CDnRr&Gk zLn}Y1utC?0fp6J!BPE(UC94DeV)eceeU29vIZyW*lqYg-Pf0|s_<7A}%)-!Eps>>H zp!9am{?_y5!LOU=QFoP)$KkZNp4J;*vj=T(><81m7uF`VMZ*~Ug|OQRreZm&N#PtI51l(=*4s%Ob7 zyXd}d3%i6>A6UfmTqN5&0hhL$8j#Je^ZMEQ!wIM;Xb{FVr(9qiFtzdrIx5s==b%CFW zvQLr>AOvpXC~uM*&;)PJra18y?P?LeZy0LE0clp z+gA8&d0-~ z6!oa9PbvjZJ9#Nq!BIJDjF9)H-`+f9NQK$#mLmulq>H=iPq3#-2!&I}DQAWY|G4#| z+2vdUv7Rgu@Ci`7PargKW!_XqgGC z1Cq~3Prh^LA}JlYV{aM~@!+ni>B~g`^l9cA+gTKxMseR11G-@+0#qA5suv6BZ3lpi zAV~r`(|CvmRBA14SeTE!tJ{JS>hyO29%9VAc#n}L1Fw-o4`*PTabWF1%mwN;7VL-i z%xfR!jx~p-&LXs>8Wz9_+8Id0I;ZW%*Mw->3+Kf?EbV%ubi;P@>Tt)~*9h=;^!}0a zTIh1>=d{FYlrQzKz4@pJ@!Ip*1ku{{$TuSEA2=zTy5t|}vuzWKdd*Y2Q%SvRd1$8M zjY(yt00I$aBWnRB=(CN>LMQowln8Z{6%f3Jg!WE!xDdWb8 zrcx4lnUr^|tKH2tbC}kC>t+Mft;R{r5n?WVdMFY3d+rpu*ml`^rFZ7U2yO7R73-HW zUZJ~jRKWc;6wfFGYxga-k+wcUUw@@Mu%=0VqnyL>&~rWwdf8V;e;Gw5=JpAl-PGs_ zap9!Ko)e^uU1K$6rP!PYKCzHTw_l#T?%;f8(-L&aDGynTH=X>rVDt5Qt^vJYnnCFH z8+(vZTA1QY{H7w=w`b#Ei-I_fdcmWb;N;|0&8w(9=vA59c9G$S0-qF#EXe1E)aNM` z2gim{hxv+RsT;kN8Wq!4 z)aRY!bL^}bK0kMJ@?@GtQBWCvgD$fEj_p&9zt;BWZ26sAVE%YmlJBSRn`ik}ia5_( z{-jy|%V}$h_Qi*9m^{W_DK()(iv&qBxWp$b+B2v3wxw9SM+v@590`?tD2Wdf~5y-Gq>&Bo|9wRbAmBys%dF zN5Rt5@<^g@jxy~%G!d7(N~NzkOt&_93Bj9PiWJaCkgqBHiMWvwBZL^ud}>8@Y<~>j z1U7A12E38qC4sv<*3S2ullsbCI-r+CCdt?@FPUn>o`l5Ov=ei-y@K}?P z9-A2}seQncp~FHQ&d;e~r>jD=FXCPWo7MGYy@{GZUv>jEiK`p8I3{_Mkk?hr-!~RT zbP`U-2RTWSAf??klVCp~Pz$kADbdxL%)yZbzuKU6&h_DU&rT1%b37wtviK#hWUAgu zL`Q`)lB9sGXI^pm<63k>Tym991F^hEaO2k|kWmRM+yzw@%%`Z5lJ1V`pw0{mRNkyonwjxg z>pa|8Zgn8J_Q?0*$+z~)@f-O|<+)MtF{HiH$i_gMM{$_qb-2rfWkMA+g%2z&>s5li z#fUANBEb~y??2;hri^Y!#d&WMSinUvU#=`PDWF#MBT76W4(RGv9+n8|M+rK=pYhXv zh<&Nk?Rt&S1o96s{NW8zsQ)Au7jMc?I%<*Z^DZBq< ztv<~O2!va#AZ7-uI4-u-mMpN4=6MzBS(3I$_Ar4^yM0w z#M^Pm6z+J2>|y;~y@Z#RW%pyDmomCynlMM18O(Pu_~euhheJz`tD05r?yptm)Y_!e z;UU^lspqs|cYN$m+!pdTPWyKNpi}JvJ>O4v3C}pJ|7+D)Z#!S`;jJIfO;EH^j3TTD zl`qIijZqzjaz&IMrFYc5PtabmuJ6IHp`T?Y;v6;-sIry6}!_O3Rc#lP1reXW+Bq}@t;}TtqCRo^!FXV7J%YWS5 zHv^A)_AbX7dF*-d!I`#=@>zT(XZ0Fz`S8vI3;P>k2Yr{ECyGVrD0dQQU^+}#2fC0| zc(`{@r_{Rd?Y)J*pxEJeqs)>uzQ%JZjr{j$D{5i1Zi=W zX4{AgQPTJEnpha;2>3a==^~nrv24A66Rb9TQt*_CKa9Pgt7fBVjc3pvu6lDRL%{6# z($4mRx8C@pNFYRNyD2N>=!d|UU+HgH@V%^X(0>G)J;+|SeN=CSb6ck`+_}HgFQ{S= ziB0Lb3+vj>ghgTYgZ(7?R8!Vk>tqp&0Aty|ut6N#{#>{gh!z!B)z;2ax8eEfa5R<} z3?x4&dcp*G1Q{VU_T9LR{rT2?*9H8=pPl&xxN69gU=>8%7Z$$7qN`|V)KwZXNRu>s@x;g&)rsR8AU(=G|jgm7b2+$ zZ!ge~vLM*H|4S`RZCL7iv9IrI%6rjvXgmoeL}+f=9sW{0XE=jc&IcK0Vg|vfZnK)K zLMhnK3EY64ba>ch#_8Ivn#|Nge9xA;B-|@IjH3m1UbwV}kTZQD`?UKZSHW&Bk~>GM zKXC9lq{ZBSSFcwXIAVBdvDD5|4>Awwuvqd4l>tWYui28A@5)8l+J2OSdv~8*MbeR% zhVPH+oD}#wlRld7@@F#5r_^_2p|Zy3?-{J07;*K*GNYRyA|2mgz~oc9JgBC<|8d!g$>s&#QG5P(qSkaVJ08o)Em^ zrcvH_8@;HPMP;FP;||(MD&F0O>K*cIYyBJu($gJ}4@J}MFIq0&mAFj)At)wr)tN_6 zTV<#)VbQf?^ekU}BwMqIRN%+hiSwtCWz<|)qxZU+ib{X#1}(W|6kY8-sG%2~0nHJ$ z!N~EsZmiFvqe*&h&H0(*mW2`z*UxoQWQd&7Fy-TPC`h)jFqyQqme)z`~5gN=5< z4@OSdDw*565Y)M!zlR7Xe0u}iZL9iNvXh~IDt?CiKIcYh)+U&u!4E`ucemfmxpQAu zrk2PDJ0o8=`yqW@5;Z1*+D?QbsdnYb&;n-lW=0V0X}xEzZRV$6eg#1F=jm&d>S3WA z72{@o>{##$fV|x5xC+J@QbZ93jSrEK`{VNPYT~ z0eWFA>!hoftf;L$?Oulz_alep&9s-Mp4N2=@4SXu*zIgp#MW|lB<;aiq)#N3OLes! z3?rLk;%)gBI#(m0_IT<0Jk-1HF(Ax}Oj8ysrFJcF?J^A-bb9{WMWzF-=s+4tO0e}Y ziZN^JA-UrnYHEEm;?~-l+WVjkCC>f|f{z}I1;%`)>nv*uX=lr0Y>zcUexWdVm-LRL6B+Sa4)M!0x{b6S~`;@TTyx7W9C9g zh%>G57JGf+>~r0T#S&N)tJHQUdGy0_k2Wh3rlsw>(5cCm`mVGv-rVQYoxE3ntk}yy z0Ej(Bf7UI68B)Bhpd#XZm^~y^VgNblI8$YmIGTAyLcadYeiw^35E^nAK(2G)@D|uZ zZ?xe;K*p8m(^*tDTSHC~f9;?%Bx^VCWNzd@%TAsbxlUTJbE!t7dDia`A?JI+#OCba zUe4LI#IeH?;4h`j{3Q+6UqPcPLIDIVaCP-^{26sUyS*-@EICG2yO-u#`i?33gC0m& zLta;ckZWcH2GtBx*Q2@vuI55kAW*p?&LYPnFRlRyA#yIXe zZV9>h9bE`S1dpQ&0y3OPJaH>kY0LX?}k;QJv%v884 z>jQyV<>fIYaP)X z4U0k))I2l$b8DKyW4?ifN90-k7XH=|->PY+s^ODDsIScJb9I~=KEN>}!;9mY+62OkdG-WJ`w$H**o?-BWd8xPMd@0g>9!ui3wa6{dl zgR2s36kE6}g}vR3gIZq*Ga`e{g+K4+a>_--qlE_h0j5Q__n*!Jcnm+bzeNW!bepvI z%tkQ`=LQho9k3b9e^1Q%%=h)1VX~@sgjEioDYd+&ZMw-+Nk07nPe5SGkvV%<5oPCv zQb>qPG-o67B$uvDHonegKFvi=hWTJ~w9EJ!Lx<`_)PkH3@;lEO*mZzQvcRQ}B;MOv z6S9yn__Ex&LWlyyb-OAVKYDBGOW^@NIZ6^!+ahb<)-icx^>jT%L8XS;_*QUW97%)3 z$?iSZfene!u{Y7xUz@bYgC4a^9-YK*N}X<126s+0{_wJBGxo(CbTM+e84gyNRCX$C z3H;`K0;CvyJ9;PJ_tp>@GOgc4ra>85te!%wf7sj2qe`6PKB~1A;H(tox5s3R*)O0H z8|*@em1A^$Z^^ei@n8=}8*N7sN?s0|OA^aOlIu&;oO(cDp0Dz?^qYVJe(*bY+Q^Ws zYihS=p97PV1&xND0rQYkM5(Hch?a*rK4t{MM>4Ic-07AA@4g|;7Tyu{8GZKo6v8$Auibx;xp8<{U}!4>q}VG?IX_5y&eU)ZI~b*Z5`3BI zlhL`s81Q<;B2}#B5Jwo#RIi4V4uAep!n9NIg*>{#V=|Jep~dydVWgQnPCE(^kG_dd zPu4d&?GEsxGp{xjv0+A?eYGxIGtx>jOBqa;t>c}DZ!CK$Lav9-gAzjKYrVADcbh)i z)w`E1hbXvTHeLU^JD4Y0iB<@8t}3W|#w47hW_Tk@ru{{Z`;IDK!=dhByoja|_@IM2 zXTZQaw%VjpPSO10eQ$+GK4y@7FA%Ccf&2+t>EGnGy9>_~+&v&(;}M3GTo+6l`<$;P z7<(M)ye?Oq3&_D@&Tr$WtoCDCAr<-~n?Vr5sY@ZiGRkdj&oTDJ#@uxMerNm<&GWi? z5_#tAJrc3+43Fv-W8yV-hTADa?#Z-HBJ#O30bPI8|JF#GG3-N**7TD;qY#Y6|l zD9QIlGLEAf+KxyNTO&D&o0lUySqfFCX>uDzeUZ;h>}tGNd-;HAX~myO{-K9kjP443 zXM{D^6X1MD;cOB8Obz!d=9sg{7BSmUOnbL4fA zL9MlZz(`-FDE)SZTUVRK9AD zf(4H-o4V*#*P(e|JofNta+^0EJV#Hy$@^~dx(yDWXjcVlLNDz77b!oLsqv&WZKmV) zF|Q9O#U2c-rc5qG8-AyYK66mtDDxGP`xL4$bGUh6$8nN>JLk&mgh zK_1b+L$y|Esa1SS^9L+F_owx4vp`-;-t#KG2bbx2AO3NuG9Nbyhou&us}F3NKq2BO z(!Gs`~412%`jDbL#dB5^dx-LqpE$M$uFrEe*G*e>Ls^`>>h* zNaF#H((XLAHJ1sgsh2aR+BNBs`Y{fcR_Fji`F&RcFK-@LcQ5rQFXmZ;|CN51ND#(J z|LHkHsnn^q6K+JOW;W{WE^kdL`Ly&d!Z~(yy@|W)y(tYf@;F=q=y9R$mLmSadHx`vGm@CecH_6A{)=p%TvX2!bXLGa>T3m)spd`I8Q~q|> zUgsChtmkif%r;Q&VYdRq2*KGBM;~4;_Cd^Bj12@RUI*55%1!jp(>c5OU~_!0ARBva zU8E>d!-LiB8<-s@>Q^)1nMgv}uk7l6mRy%RIrPV%R>ir`Y~B}_YuAwsQVli()gUq} z>n?KCaBacW@L-G_f7Ao}jyw0YqY3WsVZ~;YuS@ArV@_ibWbwS~Mte_2{iOHx-m~jD zqn2s6vmNQ^dmnr%&G`hHiQSt}EQ4c$g17&Z3!o~>OU+Er=eYV*2YWloeOq{UmOBU3 z&XU7CU+-a$>XjXZTuwQ(A2vAlSRbN>ToyQqeaPmWS5`J-<=sHSVcy~Fc!>T#?7%x&zvAY9qMV3-C^AmEg)HuDugl4>GEDutCn7Q=%)T=J zD!9|culPUO_ab?mop$9}6ZNZ~c+_Lvjkmb=!&ylFn*GMHwp%AMAa-oCe*dF&sQ(+5 zn`hEyqQIs5^9gdaj2$6XmW?#;b@Y_dosgwXhMc9}R*Cn+t;~dAQ*#SPi6XPjLT-jv zwa|bNg_)&g(_p23p5LYv&^3Fvr_bJkyXF+0 zAKF;_0nd_E_gB$Hn`hz`64(%d5#W&C`p$$lA8+ z5qvM7fI&N`+Sjk^?xXk(wYydb11-vh0i?#eW9Nd>8#+GKmK}nptB+AfN7x37{0;4b z-v2wVHF;mR+THhPNz(`Zr1 zX)9-awwB78Aky7daIJunO!mpe_M9vqxU5V!NUH&F+>x-!yD>xOKtedB5CqlnmALGF zkKxamDf>aU(DA(~cr}&XWYn~?DlkBOpt1BSwNbLFBR>{W<-1n^;}Z!F&AUY0?l%6U zn|xl2mqt{&%__wGz3>hp*qPcO*#W$Hl)6o;V4I=}QycVZl$gm(rT4%yqS;@37D|_F&$yMKLuw3ge=`3F4IT9W{)e^z`;^v^+g)8UqZDyoUBD?-Lx==i#L#&Zb%P zYiZ#QYSVMWNi0rW7nJr+f1BbV=FEA|_BpQ7`5r9jYE>`;0DjZLogC*D^%_s=j>Xp1 zzaT@DJ9gDJZeC~*n1CtfpAG7S z!?`u1{s#BGje`&|BKA$1#0>d1KQ}`$!bgTG2YW@RYmr!U{>~I- zy&JSKF=k*^7$UeQYAd6UHXX2l2W*e*83@xdI38WKb?NHmrN-)GEedg4H@nLpw<4C` z&urp0-Xp2N=sn(sxoQS=nf!KT`6z=LS1ESG3iNMe=B}oC1Csi8DUX{y}FEPtG{^@qblFk zO1mj@#Fqwm1Dc6{InkToT<#ZSs7`SW zzNPPbjgt0GH#AFDqzEomW(!S6EEwsU%5s_H3{BoFVT}m|l1qiYI!3l#H&UIZ1Sg); ztN4@L`jdIVFMfE_nkc@?!&BqP&(Cj-9yyy}=Mc7%`hjgMJOsTlEOzg+Q^_+vi?&{- z>F^NeJBh-XXY2$T0ww9V#{mUj0A!Un*C#UEX&=cmOHS3<+S!IWi+8LTmN!kMw%}rT zxp0(~O(+fCo!gU>zs0jGaIE%1=;vN%!DsLAZJ**^(VYLJ`L*4X1K+FNVIy%II_%oM z!OAo3#{NW~8JMjI^R{lIUWU?KUvkqkAzT-77*9&sk9Gn16v~LYPv^4SIEp45EZE*N zaw0DBw^KG>Y!=cVeCUB78HCsBi4!l>1{~kk22AEvN*nR6+hW*?r_f!ah)(-0LXEih zhebh9`fjI>M3$`+s3^E8E|eW0mXfHaf{zu+O=;kI=r=Cy7d#Ur5&2#OovMC4<;(V( zYpzlxPx)(lvDb6d{J|@@u*5YeN4JB1~a3iPR!a4U`u&;G_N^;5Y2xAT(b4Pgq;qQ z4ax}!y0E$AlK$;UwxSIAqR-3A38b+a3OhzpSiG1xm&NeHs?d2VC+!2cy~l#Q88(bs(BiW_FSCTk@kwqR%6XJLLASA?Jibpj%Pa9-xvSbTft{% zKpgCKy0PEuOUgoGG&dD~tR^D_Lb$lQgXU^Vb)8?EQ`~%&GfEzmY=61sy7R;fM@PrWV&ip>#t&^@jEs!pIX?5b zt@S5XR8$-enhL<=y&PPQ(q*EP9(ukBnQh!C_XU&>h#iORV zz3Y?tEn7cM>uC2UHm#<$=6EX%)^S~pgvJ&~e)Yy|Ch%624M1qe$A?@O%k~)WqWPOn zF3>0`yO|f|mchL-seeROhLn`_PEhN}D&W3?lG4a^(F+h_Zhn6L&k2pZcZrFKG9N81 zEzQi#Hk|->GPAN|Wn^v#ETB-v#YPg!2{$wzGcheMb$k^~g%1u6&d$!_9<-aQaUA|7 zD>GK0Q|y0(P%r;(gq-tQ|D#xoyiGi#Y{_1LKI1A%@7{B(geP=>B!FVi1^fQaW0JIMR z=6PLmt8L3cM#dDArgi!DMq2XFaCayM0HAU@T8?ilX(7>Re)s>Zd z)vNhR*>6+SEk<{CpiM;eGcgDWbB5utSjqfN@4o0ZA-HD6Tj zqjtzfm~sM-&BX6gTRU8$-stJ>>})lt$!e6< z+mj9W=6)2N$Vjnq6N_^E`zk?jE4`#wjooY&?*8E%cc5u_s-(B)#!z~>#Xy;@ok^B8 z_)gr59Rc$?M8yYgI}(ONZF!@*B8!;LmfbnxCLXK%a}%E}H^~$BZm*_=yF0lpRUYs; z&NNKA67f0Gz=x-!*?=Q*AYE^R$q%hR9tcm<066A*V%h!>HJA;fA95Lq&&_+|ICb)y zH~@Z0JA3o>%Uw}}>heK+osyXSHr!zA_-X-UzQ@Eow;s#wiejYoa%~B?H6=i1k-LbO zcJyf+w2vF=FLxh3&yB5a+1_c-oVv8`oQ7Q1Q4gTKZXgLLOrw^T%5**0V$V+Yw%H=4 zE6(~(x{Bu;y|L|`#SiR5)@ae&^Y5hF-;(~Gi(cUOG9{x3T&H+bzc~K`h<}Z3wNe>V;!XJEN!pbk~cRd#eUBHH$b$ z4!wKCzRv9)8yoAsJEL`o3g;`-wsFwmd&iHB_#Uc|G5nn(1(5&B&F%CGd#MbyScj+T z>gs|`@!!W^$@K}+oz%hSy9+`{YDu~-ie3#$VknBo+hpTqF9 z-_y7Cw^TE~R%BAhijwR>#!`xO`odGqxC{%NA{CaVcgI0`B=t1pkB+|0dEw*V@J?Zt zVV;||cBu`biV!Z~EJP1%$;#khqN^8966`KCvP_cu$DJ(gyw5fLQK+ffbu}@O%-!!v|w^ zT$W>AOh%DGMA%?yF|xwkMAO}Xf($_Q4!&-+TsjK?oP*QWYs#X2FO<%jkguNibPkIC z=)DlK#Coy{Q1~L?W+cda-b0B2i%~zJ36nia|;NlGDCrrLHVPjP9b1R zYRf-;Z;v@GoQa$Jh>_wbc(=Fl$3R!Q1@Pl``d9U4&&lB(3+~iVD*SJeB>VmyE$x`9B=zqA4h574! zE}dJiTdKZm9l&SS`Tgp54FU$4EYz=_!A&+Hi~b)YH@qAiN}8H{E5~mH>7qhKCVbhL z5#_HWM7o~u%`0u1AC5Npw$@MJv!=R>1=RK(8POA+E2lZ&oZSA~KLBQ7^iU2gp4~_+ERf ztE-z8_kJu3BBs{1lmMI?L3RZg85zxr55=XEt+-r^YE>8YvT2DN40m#2TO zkx$RgqGl@lOui)at@b7GqDFJnPjqfg6g{{JhF3a6tK2f{YFzc-UBPbZcAl5aSKmUE zL!jD9nyj2(Bt*+R^=+|F1|{Pz&%p4G(bueqCYnUw$54p|ABi{bz+mCYfX zfAQfnzU=Q-BcZZ-6D(LX50$nzgwn<`_VdE4 z&UJZRov$k@D&jE3=;$aM4j+-lyGJe}CN>C_@;%*|#$BR`=o}rTBohb>%O$}}Q`i#Q z`b<{dq(|e)vXc|?42Yw6PU_Sks0fC=l%CjCN)$@BlDG2QnNn2renbECt)~5!m0)6> zy{pwxLm8Qr{_?hK$saRSb5X&BfTz^ql6U|Wb2Br|QyD<~&vzW9@(dhsPkeFXoVJiq zxbmwE;>rJE$j#tv}r0dicx8{L0s!TZ)I+Y2|&n4oo6cDGIlVT*f`p0u% z_=NBAoN*m8yvCDLa<<3^rvZ$F1EtYlxSsoV;{L*HREoxys)BG31=OL}nELp1sU^oA-_bck(0bGv`fk z-?yHRZf1P5tmsSs ze?T_XouI5fyfQ?k->dXn4s>j>J+`*CT!y#rnQ{3(w1yd*He4hK}1&ykYob-AijcmdAFXxG;6ZJP}CjaL(O!lUP>O8OR@fbCLmpe&9Wk#y( zX0;3TbnAeBRg-aVVUg%$7SuEVm6GD&;ko9;Sxuv|kBSNkA-4+t)>1tq$Qf#*4CWIw z>kLmY^2~GaWc!7~bR=}b`w7NArSCrceO1;=zhE{v4m~CC8r$!)bWUv-H~DOiWLfjz zQVVeMQ|0#le!FhPr#I}ycLGX2_QY6o68ycAg6l;O1vNDeWK@piOxL@EPVWDmFB(3# z_t%EAkWJo~K6UUBSHHY4L$v+SXF@V3C#U@MhyOh9cUXQ^KeWwr{oL-fD7b&)X3T}z zgKOaqV`*lDA^KFmz@C^O+X}Pejtob`=TlprqrNaoK?K&Gtgp>Bcu$@-l6IUrm$Vxy zTU-@DEH+y&)Z%KMYLMVG)SDv55!btVX#Moxo?>veIzbgUV5szjg-@2t##-w*UKUzd zv#&6OqFB%_w3j%KjglA8T2pU%4lg2stkaz`vMOSk0La4 zM(OYHJ|){G+aGDR>YGqECx~obm%$vg1bA=3)yZs`p8!oa6?;TfmiXOdz)2YcNFjG zA&^U)Bf!NurQfYs)l%Nuo%}O!fbW{z!h!wDRCml3P2X)8U$fL7r|`joY2|UT2`-zd zTTz^5tejo4RGMcoBqT65qifH3AkAkzh#|tz*4B^)eTFNR(BCKOij-ST0}nFL=~YS; zEe=_F);c}V|Csv?=kY{E9>Gr2b;~V4uoG?9iGQ`qQ2Dr-HZ#ni$yUC>tomAYcPE9q zV!TH&FWmX}vX#vX&+?QVI#x$V=Uxysyk|hGv+Q~K@j9XaC_M?)IR;z2XWEu>Ukn}3 zRE*u7t%i7b#CfqlefkN#H(ym%^~b0Ek(Zm+mVT26(n~u?irv1mW_|5Suqz)Q9^<9TNp?Ci)Zk%gf6{yIWdbrE9#e`$JUW zB*WU+&oWBA(1iQ|}=n3(waM@O^(55L9z5%WzBz@3r?+Aa%A%Wbqj{us`0ir|*g zpF4iO%~SHFqj*QQO>Zz>2jFerdGF4pI$^ujJs}H4bF;E#hqOCJ;$`Gx9>K^M;?PJwqW=TrK;&kXMc;sUb- zOg#)HK}7?2MdB}jFF@(ZI`$sN!o;TgN=F32bt&JY^m5H{{7MR<;wW-q$V{fGuA|6h zo%Yz*+aBWdY3i5a-uYqs^cYI`fXQkm`N*A(amI5evFZtrT&qZ5W66hQ^Giu*0*Ep$ zi=)2IZ2ZgTgkMX}bTlvQ<}Wub2?&MbQnY~4!lI(_UP5o~hlVUW0DU69UGH|#_$SFB zMTYs)vs)}Et)CyO=avqM*xcVXB(CIaF8|IXrY5l_PC)j_c!|e3B^I<@)yQJ8XseIj z)U!mbwdZaRODjOycAvD{8@oO$IH_>Nv6=5Z&kmy=>_sNjoZqU*JvuvGoo_N~q>+LQ ze=Yp#YoVbrgkv?78VzwIfvuh0{M=l>zEZBt`MU2C4LH{dXGo9P+^46f|M(spL4tF9 zo5dn!m6esHrR>h0e%81k>6lGwW@>6`dRn)R&JSO&P*6f5HRQWe)|IV63D@?#%ve5v zx07MYUA1j;ceq~GgIALch!~iU%aL)7RC+eYz;&{*RmkTW%(b=LLQXWSe>Fo3np|AH zg(9*2`8Ydly3Az4SnPTRalK8{NO7b5U##%Xz|J!~#bJG_v>qQdUi;NJ^zlI1uc+Or zDm!!#bLp9~spkwX3!=Q>>muyYR=4UlT&Yfy|5s>uNc&1i2>6g;U*($M4z6%>Q_R*B zD#~?w7$-mBN&kmc#>tph<dGB*Y4u5&(86e>d~8w8Je2T@8&7Gv^1M@sA>Nb zXZ^4vTdMa7rs$%1#$sQ}6WILrDVxK|4gEUZ?TCASoB6p`ATZfn-p`rv>sHK#0Lc36 z!rf|_57l2IhF+&!*MWOj!C(c+OwiPr!}o0po0NwWVr{fEA=tXvS#8E~f+kVHlE-t( z7&(<)U#G=1D8kd+*o5h_tGZZD;vVf?4+xma;6K&STGQ~nOR zSRa%x@_1^3`#QCbj{f>_qOAc9Vf8Mb!^#5bNgRQNDH&RZqS-|mr|%6MFcwjp;1)5 zGIu$kJ3lzY*`T}-qH{z3$NPUli{G^Mpum5MyDK05Z`Nu5KcY;(v5|kL31IB~*5!JX zyd*_mlFIN;!)m}}zJy%lf5ynlKqsMs^^&q;FkZepc|MAEZ+P$>bAe>zXxWQ{Do zFKI|XmdAGgl5^BTT%pEYY-{hez&gbX7Z1Vi|j07fBdN! z>4+?C95Ai?16=`ssN6#BuP~?vKaRn@myPvtJT2@Ru(ex}eX_g;N3C6hH#EQr>1Go7 z7zSrt=~c61N?~DPeSLivF>Ne(590gha({x|)dj~v=mcm6IF09TN??4E%;7h+-a;FqWb z&(cayuKcgeH#uq~%vy%T|21#;@^i5^+6kw0V6`Q>V@y3|(mM~I|Ml)KFw7PY-CI0i90)9>u0=k=H- zqBm}`Q2yl);I-;^-!#6rpxargisUh*Ow0UmA>|;5fEj_&arcbPTavT>-?uy-ZvH8i z#|0bOqcVoI?E55yxCGs{p%&xWH8Xiakb#%z;f#0sYbwUi#q@Yaso#%798a%c7nLsT zp4$KfcQfBLuk_rhEEG!_U6_k>T=9!U^me_Gci*dbI$QVXx8IzdPZ0tB+Tr9}>qk1N zFOEuPS{gp^-E8mCf_6w=9rsb|ts=j_bQN$htoeOO*oL*1$MHaa#E)DhE}HupVqkQB zVRKq0B{ip>UVb`JSIbDbFyyDH(%Z6TW=RP)Rr)=vV}vgk_AK93r_7`H3dQSF31gC% z!Cttd_4oJGEu*erA6^Dq`8%2R5Ne~swJMSiR)4=b$&_IZpDtg3LEVcHQnoE6qI9?n z@g~dh#xcKjRu`At_}ROE;|;$KhbkVPJ8I$e&g#lVU({M3Fm}!jn2b%0*Y?`*l@Yhp z-DW3Z2eP4amkpFMEv$0LQu-cyEOZ~VoD;6H7X>6v01|E}rqR9v<)cQa1{kwfHlyjd_gZT7bh1 zJVJUxdw^^Q`%XG+2C(~&Vt-_7)p-ip&`(WeEno=v*iRdi97z$4dmTkFpoQ6&(u)5u z`hK5{QH?%gSMwET%G2MYcef8i*W0e9?+pm~RJr5ofR#l`oa6%ryWKr&7XB~R-a4v^ zfbADW5k;gWrKBXJyTMJ064H%ygVGI3ODWxrba!sLySuwK%_cV7LGk^*b?>?7TW75^ z{sBv7&&-~9;up{CIsG{}Pb#mp_OU@hXTIw^qT#Gb4+m$rVO;JUalgC%OQyE6SE!+r zzHmj-(jWEODEDq{J3@y|Knt+-^@9T6en&kyahc~Bzha`|XfYuxYBy{y{~g_@5J+F} zc)J})a$czBYMOa#knC>NDERxAH}&R72THVg(v^OU47Wo*z>xX)`9V3=_%^5d6 ziN>c5c#U}FVO%jYlJz9pkSCGhj$}$hDERxpb9N>7O{^!6fb17EK3ht9sODM1oYsrS^)U%Q$(Y`>ElM6i z&Q7bN?DpF_zbc2q9N2b_e~a3b8PO+U>~&$tuV)8H6e?f|It~CGa|nwfx9Pp+dhdLP zv%ps~*gwcg!cySTw{HT{n1F(KU!z8l_0T74b~@_f@H&y#;hv}o^Qfw-nlxs^$H&K3 zHP6{gUg9mbLhg{w`o-e3BW7wGORq!9I2Ve`TKS_PwSd!dWV9}TC{Yew4 z8S>opou?6Pz=u(|OufKiMyZ{(*g7G2U$p)6(aylzz{P7LmBop07$6n)_a|q;1jniz z;yDGRJIHcEQd)|BC#n$H*9|oJxSdti80D^Izg13OvDw?@w7{PlBZ_YOTjL5v<{~`O z!aO|9`jEtMtGrOLx^~pu3Fd$4u``xv1dF#u@(Ghj|7508f;>28%1$mf6~V=!Eo4J( zzHd_4JDYy9E<$&`5mkf(vpM({0@S6}gCX()&PFlGrBDq3L5*c;Z#sw_8Um&uzYPbmrSe*-RZ6ma?BAprXICj3vipGLhS zqZ%ebVO^8D`SaZw%UAkrNU|XDr39TBi>R0V{9}#0bA>Clo4Q_ZHnU+*!IzO?Kj~!U za{}X8WlVB%Ovo;=OyupH)le}Yd%CBjwsa;06Ya9Jl%!ukrhPgyq&Sn!{!$)+1usJVHHZ|0MJd~ry3 zGS}ntfic2vnh(N!yj%FF5IZN&yn?IgW#*>&VAwedXcnoo!d(wSzfOs{8cul`NfH+Dm`G~%GWhtMaI~gl+TxfwbAzZ z?=*HTE62<9{{|yj{{MYxr+8b=l81Ik2OA3GbZ=;@xLV{SIk%x7(V8CSTzw)vQv9nb z!at(l>&AeJL0r2!K_J7bx!Ioi_Bd1In+>A*suA?YM6(Y7eiST>f>25U^eCqrt=!eC zDe5x0-Q{UzZm0GKNlEPPN@q)cM7TR6hyZ+C?G=+}vWxH=9W)Gd{oHEN0%`^fd{dQ)|yf(f;}ibgeU^=Hy^E>>UD(hw2ayBEl>7f#%Tyu30WVV+*PrFFkQ zM&pj;yZBQHcuF>R5uB4O7RT_`%z2>EB{tlQ<>Wmgg%Q&7p($N9b*o&BJB+RyX)RP9 zNxq@4Puy6TE5DQ@{xyMSsu)b_%~VPM{Z;(S@nC)zy>Xv{LI{s-vK|q<*A6bXaHL_v z^ZDzp%11`XN+Q_>FME)C0<^3k#`1vlbz7?b2?X-kC*PB`|IP) zq9HPLc8T?=xi^S>I)uFLGmy2QVCGI7>5s;^bGi0*A2{x|_ie-nMKVp+gMEXywhsB6 ztB6M+U)BU2S~vP9+{doHUajN8A^M4!EJS0cjbB1S!+c+uFR~>6KyPVtBITn z*t(uazj_YwNJ$jm^}#aAF8Nr9CeNkb?ca$~P1j;u*m$z!wt92(*MvDCQhSo*#|L+? z(3Qwu&cr~T((Qp?-wE))MLKML(i4QJq#x49jqkA<Qp(vFlzErVy9}>aTkLv zEVRdT5pK7agT+~QU0JegM3?7-YVJ1WeShDTFAo8aDTOz>*w&OeG-B{o^rhp&$SX&O zFFZs#6f86oP0`QwKGjV+^zyj9vUz)~rTaTf(n8bq729D+bp6$}p7|>`J_p6^Omh5ToxS9`c) z`Odb^9vRZomi6C5@GMPF5G%+#6kZJU#CSM+o^nGsez+G9o-Z3ciPahADB^gj^>NY3 z9-phwTo9bQ-U9dJzuq8)A`a?yVninxp7Pt93AvXEdC<(}Wjf=2V56(C%n3%BA!@dT z)^b*t4if+0JqkUK;8g0r+5ea}P^klc|AAwVyZUn0_wl>_Z-RFf`WyZ!acN^8@_yf0 zvv?`FlHz^)+egDuqxN$V*9v~?i4TEUxHBrVf{b~92Z?MpE;!DG-Tc6a^{S4@XmUTu z;=0BP!AdmrKVN6XINee#N3~O0xV8p|`SCy(f!kPBfWL?JnKPO4aOQtTreS$)7#W#S z@9rrnxKeqecZ`freQIY`@^g>A!fWB4F-K^c|^G`YDl_McehhSjC#-8dYt@k zoR1}QX$xU|>5H zVrDwg=ng-Lr1dBi&Wi5wHz8`QS9M6Qc@jf5ok0FVuh9`{Z|*ofONBR|gtvbs$gTb&WHt3GR4PIySV%t7S>)al6qqF{=6gU7O%vJkeSiih zkPbE&P zI##@_-0^;KoD}6na}?x_Hjc}_1tDHgn|E|Efe>}{iAlrnM(b8O7cu^guaj(xnZ!RG zy^BklcR5_#k^8;$X5yo%W`F-ASa>ninY}=XoAvE~ST#w~;!{gb{g1pLL{ic~R3F4A zkzgvme;2ih;B#>K3*?Yr54c?RZYqjkd zn4cYOmxG$x#5owWw^1@bV#Pr@d8yo0_kbcecJpE`v7@2UUz+s$JYgSSv|Vmnu;Aqh zjuoISj$4|R(g2NRs{g`2J?ngNOpr}YicIRgF*kj+TQo~1#~y*+g1Ap`Mj)so>-P3{ zaRq}rq$9k>PV6k(>-3?giCoo0JXH9)>Tz=CLK+Z#ZGY(~!Rg^(qbPUklSt6QAjow* z`&d28$E(4|8y$2aA7(?diw?c)O4(s7BKoY)fY1tITki_HOy!(X(-eR#YbUwXqSlQ_>T0S>EAwOk<#9ZF1r3jj%~9w1ysQF*LDt!!<1=@Mzq^!GtcQ=E%1N{8G^N?a9~+lB z2hkastJ%M>{hUK~HQ*M44)SN9$LBzc|8aC>fsti*r-K>&u-MyBCj;)P- zk36{<6($HVxI7~m%ZE}uTqSA;MZu=gy;EBvgEj`saVmV89RK;J#Xm03zM@W18=m) zE3GvQ>g+Mso~BDOY;U7DT62QFaW$2r9gN|gifvDtvVL>?PB%(PS2;bzW7eKztlTkb zTQz#g-2(N-)wFwUt}EnrzY-Rl@1gpPH9kpbd|o4pa`sjk#(D2%$ibXk-Gz5fGDnFH zB59Cf(_~r4`=|nb=P!CsD1%}vWHw9m{HjWI5T*D_j)Zjk8^hIb%RFz{0VO8*_G)*x zN?uZtRuK?gse_HE8Tsz^6y5=4uw2f+K$@eFJ8t?fkme`=+JpN*>MwBjJxeLijwkVJ z;3LOhjWz&_odG8g=mml`ThyJOJ-*p3&zWq!RcT}=3sz&h(%0YozI_0kD8Owd4Jl2~@wbS~(!W^>sGOpj}ZLmq~Qs+EzCp8!8EdC1Sq83Yrjf0fN?oB`y9)Kb3Tsj6 zpFkw!V>_%6f3ES}rE120*Z91^pR3RbmL@)FGEp<6d68I4k7%urILyyqyLFAdoWn-N&w`Wr(hBM*1P`WqMBfg=92C>9SiSbO ziG6VqpF~APi8fNVG9i1tb9wn)=E=w4M&UG@?>X;))Ox1tCj$KVK44;+dypl27W#|a z`-=?P4c~6^ZY(C`XE&HWcBCl#w6NRx>h*2&sx7wL8F8*s4h^YXO@7SsMMgRSYL?Rl zKGx@UbP#KyrvJ#4!;Q7Ri-S!RB3(VN2Wx$BJ6VQVno zHy=5ChziuPk3^l&u#O(J?j_HPkcU@%eZLL@tUpu5QqmfKnsoQaoK0(#KerZaZe5! zvZN;I)k|PEA>ctd-^iM$$OJGcf3w(ovC^N#QBP5ziPgRU9H)MLRzv}f&cT|l zcHhB6hJsuFz4OfRJv*d!d|{6n^g^8TMdwKJ%v_C<=B6$Lr)zJXe{s*}-`B-F|EEa$ zpG%Z~cL3b~(W-qxn)lyU0Y73SJ3G6pOG46QzSa%~gQ?T6&;0- z5eH8Y1eabul5hQ*aQnt7X5#L#V ze?Po!iy)X{6Nd;B#8pInT#q3uIc{oaZ(mvR;u{E1$xl(jVF9{N0R7Taek2sdpvmKK zFsE!86dFpxW4kq;rwq)(HHpZD`;qa7J1WV8qyB!ciDd(hNMamj?enLq1&UY#0s=kU z$0C_>*f)!2mX-{d0YRFmLZoi@$dPWz)YN*@HK4Eskp2F*2MdcHcl^}U)D#pMEFT^_ zCMxCi_xAoNg@Gt{UkY#A*3Qjofc#bHqPq2gY8TiO7Wz`Kmx1axq0f7?o=nK{hwuUz zOzc;7?OO?-k#OkML}=(!n+FB&<^YAhWZttBi7W0Bmr&{u(+Oc{9uZs6RxY z#Kj6qE_FHaz;DV1U05LTC)*ia#EtU~bQY0y%^(F8PfaS2!x#%%pe?y63$DCVeIY=F{ zfnw@k2AQwMwsC9^>U26Wyg0+U@1o?T>60o?UE`WYYJ7=ma2t77ZfexiH)Rf%G6oM_ zSAL<0vzVgcL%$)&@0mS3vk_SrG);NwRRuhUDsRuOjn6OOT@8>)*-mD9@J~?Z=S#0X z2Rb!qsDNx3Z_74Zf6I5$l!;Z?VqseJv@4dmbC=q~+Bd=Z;{n|b3XZ4`ZB z1AEv!UaL#QyGTydMAx1YOH%Tys~#lZ0epfxo#cwLH@RD_osh5I=nNbyN+HEx?{pr8 zfIx^CJV0z4Qb=$)ai&u~q^ost0S%hOqwXmy>*jO*1)$&{sTWjDW)Q7_uGGhqIQ0Bl z3HGlohH`T#TPd~bu42f&5uZE;vG26>Z)Z#ArFvsyQMkw1v)F2XR}T%HnfZMF&{!*^ zUA$=;Jrxf~6I%{5E+$r`9h zK9$j+2o}H}WwmTR0B5P1{8CYf{O%UUS@cDrFf;I2%Kg+?Mk zS4)^Qxozv)NW+9TD9HQ%=PH*g*=M=a#Fuv(o_~sg_OF9ce19tgHufv9f#SgR%Oyog zY)TQ@@cLJcswE07=$Vq4X#lQJI((SyZf1K7jm&;#GB(g4U6z7+JAer?KKGI0T_vm= zKr2+5m_A`(-Hy_oAaEm+#gYveZEHeILpm-xrgARGh(@2tpVD1rG>U z?6KiQhwgFz->3Gn&AA8BdBAynnJ6ue*eamifBX2GC#AP3`_%^>%&a$>er(%Ua^(GB zUQWMooYm_wO#V6_`3(N|ay^NfdlZm<5HJGqeqo9NHo~e~5`$hKotNodAF-UH1pLJK zAkxksg&t?^+M0@0bStl}#_T<`$E)K(w`kA(u3P?3ZhUoOS2R;Zf@B3KbAYflnM}{e zr|}xyzg@R=5eH8~&}N~p@8j#HiKm}IL)^{!sW#ucSDXNQ119Ln<8QIkE+_Q3xsPq7 z<>l=JeXO(96LH+O$BtI12E^M0q(4gDFTHy7gt>tHO0no*W0wSPELC`RL!c7Ws@($@ zG$%I@HkXi3c-6ySn_!MW8Z-yeniJwt)R-;ZujJeEvt)nb{Uy|%KBi3IM>WE!>8e>0 zw%yf~Pes{@cc7u2$Ju8`AUOd-{rX$ickkW-xuB&bjc)-DPb5_q%M}HnU8*x4zY)z* zz_FFS9}u@RC&@$#U^|SAq3C2dfxy)P(2y*ktN*73paxhOBo=^eVBzHCyhlcWde+{a zgH(hXt7qLBD1tL~+-sb#;Rb=%0YEQE?lh}**9yE{55^VpofE>ZxH!UolDlv)+5(!|)li6f+#V{2-9>Fy}oDPfu53VjwSmh)Xr+X?f{z zG>kyi;987!VV~irqH3|JRLnpiyUiqUC&${+2gdW4gjz zEYUJKn26Kf*yTd9QGYOIGIzB@s{z-6SBz+^{LuMDf1}GL^Ps;Qto}z5f&}ZJ_kq(E z;jxX2i!XXhR)=7hf!R6ovHLk;P+yU7%I>#fC=_{S=y_J_-!?|b$FU4f#s*@Z1lu25 zry|awCbu`8b6HwKL6ggkn$EU&xr-khGE-;$>Eb!~hNqw^ zw6E!S5pH*ctV!>r!Dac`pxfHw@U7~|1~G}CwYso=KYY9j4u0eKr`akEW;6DhEahfQ zII%aJMkcyyaZ%NL@m~u1B?gyO)lC!=|5aX!Lem#AS=^IRXopq;`bXP0Gj3V=QzVzC zM-_=+sjCdL6U+-GJ?Fa)GO93a$lIx4-{QM(m1G~F-%~@s;g_UjAe6#3x z7JD%_d1fA}4#4B|l40{UKoHj=cpS;h7a6gfffju7+>ev6KO453f|fK`Yh(_CzY)pl zR4NjH+PF@}%Az4$C;~$f?Z)|7 zC^@ZiD!nF}!%uYf$>8>$h6?%Wl*yPs0qY%Kt?}uRfn2Im&I&?EfmrCy*5R8=SVu6y z-OWj8v(N$ij)*Z#6a>P z!>D}o_~m`4MMh5k!oAw<+P--sTI^G?-VuQgwVT~tjqXr_-^KWV7V7#%v|6GgV|!1Y z7^QSNfa5Z}J-;cDSJr!JU5=^F3HC%KiktX}9ot}Xb*Ts@F{pFpC0iPY?N{$j-S^8u zA7ta|plqfDb3LOgC=9}EYMdIj_ zBR;D=Ohi(Bz}*Oxz;E?dOP~FF_Xbm3Ph8`}xrcj7oeWph54~6^Ir-{z)$AIFU9u4$ zVy8(&sNt%98Vd#I;lE+#e#qA;KUf@*8V-&cw}r*&`C$=OCDc1}6jo8_X*DpLYn)Mr zwcX^{U(jGA-XrlUQx(X=#|}A=3BM_8T&0TNiDo6`YF8sQ3952B-L|{-#Bg&QaA(V; z_#u2y{FEmqJ)0b`e*QPY!&$Ybz{!MH^{cyo5XCPM@A6RUJ6P?*1VP(i>N4wc9rf!s z;mdWL`&>4whYB9gTlwcJb;VgQAF_ATEE9Fp+W``R_z(D>TJQGen5{&_8SzQIf0N9+ zXo&*li?+LxxCo*So%+X^q5dDKm&;iUG`(4-fA{Wr5k&eZN#=(~N{YzRbcMPmZ2NF0 z&xkRP8so~r-00f8omX!@<&zXE=!B##>oemYuQgDTR(+Q6%z9Pfi`k;0s=mXSH7c5j zHn6cTG)uR7$BeDSV8*%<*T2q^xIaB%fTc#y4eyM`U2YQJJtqyypU$^2GUC7?&|v<1PMci1nrQZe4C0efNi^6p za>Bhyy>$$dNNwybZhF)Y<;Z{8#6v0%5`6zby@yf2jWfAAKAnS4o0FKHnf)c1hrxdF zN9#5*JS063W~H(`vOBfo3i1S$tA`BmVmu8haV-+6mNpWtfuPAp%ac zN)y#KFRH3ku!{j7Y_Qk~4&#e;00v9id`=1!AOOo-pY^wSsd`Hv ze@huVhl});%N^ZD>8t0#mVUMJI5m_)+*ISf{;!IB#yjobAv5Q* zrDcubUZ91Y#C)v=9xAHM0^)}na^2Hl`g=3qYH4L<;SHP&qT-u{M%zonzSg$#e7*vU z*-zTfxJ_6_x!>}hSl93r1@PLhB!xKQT+Y;RPV~jpL4GM2@6Ol1L-~5%Kv1QNS`>4{ zWwo%^n3Tq8(vg**#%Hm8$@VSN28Gv}(bb{t(xysmFnO-|ee&{U8#zIU{unXKA$(v} zk`j~kS{Gv@bj5-MWnB!Qk+6_J{ysiFDnX*epFt6l2JY?x_lVjDiZHoZc!gI$u^1gK zj>yc(87-Olhi(x|r@%G7Jc7?+|Cs43s!n%F~o4vHr*EJ z@kK}D*_=&H*wqo+h!#yNi(9GNEtM$kTnz6)o_2B$VEkZ&#E-JoC(0?19Xqo z)m*+hrsKKeV2`_IuV;ZGTwU;RDaJgBme=R|@tjs6KYpxm4YanVD;H@cyPe~QVd4=D zp42%&P*G4C>~^C7G}*xc?fG+jW?k>d@w~F=M?Eb|`7Kl=go_4Er36l*jG0+IokmK* zmL4}PcOiwauJoyj-S*|QiQxFU>rai98Ie~#1b)&2>nc@0G2Gme=_`!P0gdPcgX=)a zeaGd4e1;KGf{#n^^{x06wCulkU8ko-oJF11-=z8wNgo<7_t7GTr){~#fK%gafA(Uz z)$dAOS5MCYFlyLtj}#X(0T?U!#8XHOfXAQ--Y~s+Bj)izLt_r`E=SX;mg;q)$_$bj zkv@W}pTxw(ygJ_}ASNyZuLFMu3j6eaEd4M+wOXXq{C z3RRKcFf^=CV=LPv<78ZP;tVAB?hTo)qhkY^&F*=r8G56y7woH(XIS{2W(O3I=N03# z8fIi=;W}99>#rTJ_r5J7hHC)EC{R#PU^zfw*B7`98Y-~>-V@V^{Z>gtObo}t3b=l! z+oONhwA-D~1bp<>E;cqcR8nyYv1d(n%_%geJ14UaRA1WqtoCvdG!uSwmxu9a)5SNg zPsN1O_?>&_3G+l=2PSS&m)?&@kkLJr1ECEJaK)E+{_ zD`{B#>NOh z{?E(3dX!B!`K;F@bam6x)AfKg8CiV#Hy1X(>~grV#8&^+$dA9`SuBy=u9|8 z=G=6<^!F5HJFJ9WMV11kykX-nUvL#lOKk_f8{VUc2uQd3(KGo$cTJBYib`$yY^5FL zeV;f21m(LX^nR}xE|epN<<}|}u6x2nq#)jBv&W&eK`L}_MDj*WX-FA;YyfJGMyCz) z({wzx*KfkbFVyXMcVkgX0FHGnJw?)6r710Y9HC=S_{ginbw2NW*RKU|l{@TB9i17y zKtlr<7cRRw>bbc&Nee*F=jFwpatsha%*etYcx=27eL%RlxPqIIZ*mg zHKeDHI)6iWQHnl&L`y$=h8J_Y~+WMSw3o*fyNn$CJNZ=?CV|s^W!bF zw(0?CwMbA~R`#a~T?by=C%`6o35B+GbVLok;o`zeMpY1a@gmT$KTfmJwZ@wRFo>3w zl;o6$0hqO14v^{)Np(w9Xz7=)_=OA*QBm(B@N<;5pU2>vM)X1ybU5Kj$J4RDswTs1XSdcPL;sQ@#x#^BQvv_hV0(bwvy&;8T?IlnQ`Q z%k**B%bVZ%6nHllh%)*K1WtG|(IryF*`~0+zid?*nnq9KjSB@(vas8L(kT}3AAupDEV(_%@E>PyyVL&vswO#&`~m) zjlpSn>ts8;wc48jDv)b`L8u-~JU;yBvLW}E1O-sfo?l#0d8??Xyl(K?Mh_4vP^$n& z^kCfA{p1bsa1oNP(b+6!#wI7D(a9nue=RiI7SqjUe3Pf*V)wyge3l~zI|0Wc(Y^BCZ$_6F?~{_2ABf^Bk$y@yxo! zRSqI>JIo9pOD2vHJQujOYhB}>Cgz?whRkq4W6`?~^Dxuf?1Z>hIg z_#%=}a&4Xrx(MP~xqfQcmfw7VVXE0(-gD}KHn6RNvI==PKCAV9@g-FyStz+MqK=A^ z=gabkjnIaX%5NdKhfDJjWwM6PMnx@{EQHrxu=*Dm5h-(5mOP*Vj(Sg}OgUMr?D98^ z&9rtD;r7>)dDt!2?%JaiOuHV8IB@kvJq9m%Eu8xz&F~gDrDb8si$^VA2MYxNk8ZG--kdI+Rt6*i;C1uH zUsd6G;%@V1sTBY(bLpUex+Kash|T}DVu=^d!2N&zN^dAl=b~g-${*iW@PeT68ZVXDNRs-xdElV_U_VW z`7hbz&5bL_AK|36j(?#Dh~4 zRi=)Nbk@W8&dR{0xJ%Q>LZ;bD<6=-FxJj|bg zTWa`Z88}j*dW3-s$*BJ_!0T`~B^ZyY*~F37!rMN77gW1obwXmZQP2=}{$+P-IC19Swl8xIEiBzhCY_voG!X6ZG47&K z8MMYb`$H8O>knhvE~^x0`{&>P*BaJ5Tn3q~l?zM0SalYptm5YKn?oByU?Ar`ni z-2o5i*H$4><6UN5(sb8E*dgAy>dT%*Gg2_IM1Nm)h_lx3z8NAgFJMT%I4qKWS+tMF zTY8yA$??H`OKxA)J8L^j_=4Xa1%Gw`wR7cf#22Durf%9y;B`_IR=pOG?*vCJym-Is z!D`VIX5&FQFMyjQY8LIVV?Z}Yz+vYBvlrMN+S{c8C6nqsgW@1Zoa6-L@L_NPQ?e$; zes9rML-npLsdF9TVXM(+1^}a1UsGk=66AIr+Wrjf2n&!e?$}R7nYB!eIy)*Xq24}R z97a}sG2ZB~pBd>QOY==KO1i@D_iGmny?5lokg!vM95TKt)#-r&bax>Wj)PS#JdM5n~B9nbv;8X&dGlN%1p8U zmSp;ftm#Qc5n5HM=$*$P#Qx`t^{f+xy+1!wwLLs}*&lsB!W$8pjh7n9P<9YbIb2+K zoN3SI!MNwa*Jy13uzqs>mF|Ln?}yMyk=4znZiuE|iJP0<QSCl;k=->oeZ6sh8>l zA_5Y9Ba(ZjV@FoA2VQYFgFCvK->)mU4^MI^!)?fUuXxJ>CPRq>C8{h~>*Z0R&@JW) zrEsea^{zgj@cjM4+4@flR&k!4L)Nb*lS9dHf}sLm7VFPNQ7lIDU)X?E95sbi0Rzcn zO5b`7d=m&n%Y#g6WNY``?FM5|_~`#VIc+G3pcWwG*{wR^D;hQidP!vWqR-Tc(#nTk zl9?l>fio{Knj`>ULV?n*$5SQ*!cOe}q}3!34jP5@VOg~s3{kj9=|yf}Y^t0mjcnd{ z2Ltdb=t_+Ea+t@)IU+&jTL2MGBuK!&+UFFmtvV{&*QL=%ERwAASl#m%FRA+Xu>}7I>Jg>6R+fU7-)MS;GbtVh$h8LwIhRIP;l>p%T zM|;bYC=FBtr_!k!CKYN=9GRp5mZ9{XW%x?ym-pdw+z_{v$T;lUU`cu>f49@jL7;1k zeb-!V#D{&9AwO0OPwaIBVvbtDda^cDe+@;<&+?|Hu3qNn+wUCqkdR%Z)J1WAm_G>2{~;#PfrNe6?m2(n2aydNQ8`a ze*HxJoUoe9aKQUqx=3JZcGji+2X>L*Jb(eW>y|`Dj_*3Wqm<7ZN0284nj%@1Jpoj> zNSQ@$KTC!UHysk!bYjbGfy~jbhK2xM%h;9mJbPa z=SiXPy`|va*os?UrV1X0#jxHYQyrb*dZ!aIZH3q9uNmSV!`AQtO^@m6?!qv4^Vk-Z$Gw_o zxX8sw6$g-RzXaskfUic!-qD9T8ctitc#~kNm$O*;l7E#TEeBU-QR(F_Fsx7;yq4ST z9QQK^TZN})=Dk+Pv_gTLJSf9jmAMu2UYwIq`h=K`zv%}6ENaxDXO=*q>s{vum;j&S zuxaHhE;=CSu`DLt8KM*;f%Z3?V5hN!*yT~MosIb5(U%y{rCDN}T-rK-vVyPe_YySG zw5Qp=|DOM!6~A1mihC;kKh}?}v8VR1{%)JLr|#vF$-;j8S01v$`IIjCnM{;f_s_jw zj_Q=m8^c?7Ng@auU>8zV<3Yl+2lPK)2~7Tz`$8^L1a`MRjehs|az56u`3x;vS4P{l zM}6G3%EiGw_V4yToAwC!B-Pbrs8?yH+j`sGIO;)VzO}Td_gtyDzeFX()6V*?L|}Zd z@Iy)dU4gUvaoR=+>#PV#gh7bSig(&g`tA3K&0~p%wDDW{FpKQ zZ&uxBu1rk4FBg=aF*6{w;bvkxrhPEmd%L z%Ovp)V?_1JjYeD2?3zulE6ka=Xw3D`Iz^Nmot85i67l>`s$j>6o%3`0Ns&J$lA}qA zFA#oiu_O4KSkL>Pxv|!=i{s@reJ}3e zD!;K^$#_y&E-paz#d2HrYkWPo!z)2f&#nYlNyiWQiUoz*{kuG5oRrMm^6xV^q!~hv zb-g&(_h#I#<^9zQl>JWe^R8eK|igf^n`umjO(m)=@##ob2dIkw6J7Ym2=EPm=?!wQeA zik~*>&y2Ab>mMRyM#a%Z#L^8xd=fj)yG*fJLV? z)P;LiqIzx(x)qbf&6E4Sv3R9yV)Az0<%_f#fj-oTIx}KqWcHmwGu7kYVvD5rA>_V^ zbS}BK3%q}~$;J*C6`1IN4U@|YCfNvtQ|0?;-@kHjCQ&hio3jvC2)Sr zdp+_!RbEZ>rYet}jE^)zE>kWxDampVlp+g&bXr5dbaQy^zh8NAsN}Zh*)iy)+!w3RmIMk1Sd02#Ml%F-MB?GR-M)$u?IK)p3|Q ztw$cfTN@g7yJ6sKy_NGff*Q(xB;1K7{hbrO^((xseSOFt${!zfenk)jApd)zrDGB! z;vsbN@yB5I8B##^_{vuY>2bZ#YayUVPXmNR?n0Yv7>0&cSl;6@Makg!=IDu!dovk| z7PCKyN;oK*Y~IUYmPa6(+V7}UWek&+&1mwv^fwY0NIA##VVHe;#Q1lHBb=<0@j?ZP zAbr44Uvc%i*PXW6Y|fM*`%<|vZ9B19H_xIU}hbbHfX+xl_cJ7$Dbg@ z1F$%;6%7q;CUx7mGM~+~MviR$?h#H*v}}-`K)Hw6I%ukFa*2cEO~Z?ZCP#Ei9j7n9xio8%b;&j57)BX)%er_(SlO8?v2F_ty(G92=_s zP{B&m<`H<00-i-A&2X@OF1k@=b62ndc<#j8?2jH%!XX?biEIc9Z;Q)1$+5{ zcT1B6B&qBz8I-D@{LM~>?z(;a-#c%Ao@|{6Dtp_TFxP%UV8mn$g}w(Qj&c0kes{z* z(Y=D|Q&tm)ooHaBQt-7|UFF>rYEtdzg>1NMq@bglR{ED67WLX~sViv{~%XRWPu_RA5%cS==JH1v;;u|6j zF879mbV|V*6~!)shvZ57PPX8^9&YJ@a~pZ#FXi@wVaqR`o8f375SGu=!S!SpYvc-- z(7xQ^{ZMV_;&26tlfu|IA_F^31s&DF*Y{NJu_b#TBpL6G$_Y6G>(gnA;b=y2nZ%P! z@L3shQEin=a{`#0rTRN(n>n%tmu-i~6;$GM5M)zijv82cz*HC5%ga^uaG82SXh{SI zQbf{Z6`&vR@s=*HJy5P4FN67d5EIp0S%Q;Ff32mp#3~0m{bq+!r}xh3&Yb9kwSWpDVUG zD{o7UcN4ulZsr}gtNQ8jKy1VD9xz_eKKAkaBrJ~SSDP8H(rt~=d{O)k9W1{xd&$K1 z$VGV26b3%fmI3`e=%vJtypz%sc2P7{z#$Jwf*Q4V)AexV9~h6+4ZTd`YWy-VjGXj~ zoshuZ_$w|Iz%3SdrENN}DoKeK3e5lB_C#IWld7+F9A0Le^i_GKal8wSXv0dv;Hu{N zKpCd%VNe?bZ1$(Wwv%~rNtP(7S)^7L3gFo^>2?08pL z%%>-MnIyv(34q-k9*g(gbQG!$wdcgt-2Q*`0|6*b?YcV>?*}r14A{k0pU1}4IULC$ z4bD8XJ*@)F-CX4dQ&5Qda!GVy#`(^&L?jX=_SBT>w#QLWWMaY3j?MKGZ$yB8^tEVU zu;efBd&W`p7j<^LV3y;hqRL8x=W8tn1)4UJoZhabb5vr3+h%^^^-wJY^ehF$%7h9? zNybSH)KD zth8LW?$!T8*;~g&^+w^ohzb@WAstF1NJ^(t0@B?|H_|bbA|)l#-61K`(%psjCB7+(SZoulM&W@4Kp=N8|J)m|UJz@UUe zFkv40ibgXkHuhy;gv>;mkLPeJdLv1V1b7>Z%E&!6kWEnEGFAkv%d)-Py?3|&`2a)6|l$!gU?tA0MORtp9{;9aq8m^7x>Z9cQDDox2lC!P|S%kaa zixOwJMW=+KgUD73B+NndmDbr_tWu*96;&R$tn<66=AhuX$&v8^?UeXO9-cFGK{?Qe zr|TQ_<(VhWm;}|;1#kEIBD%6f38j3$QqSKPaO=PL?Q|CHgP&nNy=a-wwIyIKj2s#6 z3wUvau;l6BNE)w+j%(t+2gZhsul~-sFT@nn^Z1^al+@=EX3}n1fan7~V&5`QSf0>c z9UVgofZLjIby%#^p*)ZWfl=1L)jtAMD~OW7Z|40JDgqyeFn`%<{q95EH{<1YJ(IGO zlb>JrR}YC0BU($Sb{<;Flljh!W_~h^CTLGrE$$jb<8;DiZKY4qIfWc{Z*ir+iYeGX zY1qZcdXS@F5kOEcEYEUF{K+Gh)^95DE0_UOnwSY=Zt5S@!f{JZ-hNa|iF;o7&zQYa zFE7CFvARCN#T0W~cDJaf#`aj$;UVp=*+hAo$XJ*jvC8KVINy6^f8Q9)EYjF<`#h`B z`A1>RkJlqBkp%2KFOrRV*AF-F<;kLET#86(Jq81Mr`C^aW0|3^wc{ToQg_DgZOvsy zK%fuJe|5!U#orAqb4v=PbbEe##etVpBKX%PMw9Jl3qGgmFX(++^Zm3GoRoWw%cQhL zGeJu6HK%B08(g{4&wH9(>i{ds)|TGA0(5Tpy*oaq=)Qb;d!9SgNP}X2*X4l^gk3%m zP{Xr8@gxcmwG0XTIv9|eiYRS|WNK9r1 zE2Txy4fJ31Rt3keTz36{JL|-`aIdZJ>3W`xBZzSa#@C(jBqGm_rCKkw!X=QJyAsDf z+M%rZJjTBx-If%qlS&Tmn9eF3zcUk-zu&RbfU`Xn_}kgM^|cYUAMR%ou_mMA0tz)r zQNVfZ#8g1KAeV8fZ)BW}#9-UT#dswgw#;$Y7djX6hH7A1ljz!0vTbkpW}x0vt;ko( z=}#7b+)H7iqkByeuCApObKdrfQ0b}}=VJiSL0&6A)S^O(j0Oadgha+DskV78(CD1| z7Wjm0meW9M$m{9vA0MWso4tq!Fm?*;pfXoKHv9KZiwbs{ia(;wYk3d)jHq%< zxy2V(4YfqwwNK-I)s^(?;G>PPi=M^`Y&0(zh6r6;?!Q&TvdtS6nDHv#p8a`28uM_c zG4`*{*;w-_eB^XzL9@+j3iPzmEa3<3?sF0BHwI1=>rPZ!x_?ipi#;;O?%gr6?(=6H zwOt!%f5#D|byE1LYBIJI9kIpd{=DSZ*IS(IltGm3-R+vft?CfS#4fjE@6;#^o|tQp zPQFIy=}0d3t_g?V<~;;fA$^YL^UD2`Xe-TBEuqAA61tQ2L^?Tzo4i)Oywo($P9^O$E5uE$7giM}po2dH%Bl)jjH|g?F1aI( z_kUTRWQrF@$EwZnc?ilv)kiF5`)DcZofm{_>d>*FNuNoS^QVK>a1h7S?>EWap!8xTN{{%YgG#hnbR2VEl9VN`O*6D z6J{;^Q%AV&#%M}qSdk7nx0eGYv0(V}*zOExBJ^pGY8|gC`@yo^7=ESSRK@y^)%TwL zmS7PbFOz%sXNZ1Vi|3ddi>}l9SBkZ=GZkKZj{o>C>A;PF7<52|>dFJA@?8)DQ{%(s z&(GAmvkDSl=0PM{N>OXjqfcb*P>aK|!ypHJ#Yo+$7tXy-)ggtq<-!vav*uoF*9{J> z2~6rTzFAS(;4~ZR8_Q=V)!bP#vLQE77|B@eAmmNc+R6O=obiEg7UVngK``h0SLDc; z+nLAHPh0qs&*nP2zb%BFLb^-IM5m?S>C||RwBX#oj-!1Wb}V+U9Y(6n~>Fg)Lc7he}-baCMIYq2)~4 zULKy8UOKYTz6;XfeCgs*uzlE@ zsKh^cp^@V`A9VIJiF^dHpmW#FXK9ka58A&m;0 zhDo?DY0@(Ivqw`B+hr>%#l$Lj*tYc!2@wXtnPiN3@Nu{jyg_Vxz=I=2BKX^kW z-%BW~@k2)+-pW1a(x^S2y5xnPg3Yab*Ac6cpC$Ng*R3+SibI&q%bxY?it))V6vR-X zdg}_s0Yj=*(GDts8r+K2_Th>B=r?A@N(iHs`*bf+i)$*_88szZ?YFBDn-y}kx&?0V z5$D#;oS}}d@Bap}xu*x1&WxH6`VjV`4fd1LcjPW{4Fs{v z%6HYAiq#Pv4VXvFx{r@nHaVl064CLI4{yXcU1%1BzE-!HJWtat9xgE0^f8gMWHJnR zJ^FTUK(cfCI#*6g4SOi$uGg18V%6WgyQOCg2iU|UB8=FE_iqI5#&{xrRX^|v>grM4 z#c|<$&XGi|zjy<~0yq8+)c&sJYA@xy28ZdQkJ;Ye^W3qP7j%s*SgApGv@(64#5c3m z!h|83>9I$b{2VSX#p0`*G4fuNzq)!-cmya5nk=GJ;Q_3t*Hr`f9ls7rziv&a={|O)R_szA%3adj zF8v+uc?8_04yFjR8sF~xBtE^;`uY1k|Mo~FQ<7s@!2TV(e#JMGo&sjGyG@C(v?Hsnas^C;#}3r5(MBm2Y_w2>hNwiOd&S$~hB3nE$211k!m2Y}%E z)^fu|GUN9jsx=5bx(u@)qR&znswTw6NdkdRohc@Pmn1ds?6JbfTe{1d#Tx8^SkB|O z>x)zlt^TI_C@h8Zv^(zbaIYjnjnK5~JWyDn+tc8>%!xv0|N5UAl7~X|k5{gOg8dI8 z_YCFkKWy4#@Bi-XJz>=Aok#xmlYelxT-5(=r1A|^C3JZ|Zln1mq;(sO@uHpnhWWJP zO>^FWI2h65WFYtZ=W)-^m6K)aW@SC2iz|#(G^(k%07#=P&C6jCCKJBxLF+ff!Y~NC zKI^w%cJJ0Q@d+YcZVt8%1zY-RO27K1ZEd|YyRFsIqT9{vb0^5RCs45JG^U}SU&U^6 z-=Cnq+Sy-WP`un8r_QuZ@WkXSTHr1kwo5&Jv1`W)*nB(15OErcbXFC~F$Z5gzn0bcIS30}$M;EcC z8ZAw>ZE&sl#6!=KVvIfZ%PSh^l98M{3(BH>C9V z#U2w@Z2kB?j4x0-lXY_c@1>Fd7DaP=n16J?FGtB!@bt9`w+*zTq&u@SgP0=hvfSZi z{%#Kp`D;R2*}=l#Jr90`Wqq{RGKK_m;#A*r8+m?U%fRIH5O7|(DD~Pp{k}#g^c1V4 zD^SucRIje@%bWlSLCcC1FPYm;N$y-#o1TZdjIiiwSg1Slq}X4C@~79#6swK5;CAP7 z##*`GLsNb0W0JoohGJCQ3htS4OKR%x8ZrwW3h_<(9M=TZkc|_!DwWdGFH)_q2J#|5 zZw`IR2{TA9(&avb4%70iSFsBn**JO)f78iu?=k4qgscpdTZXO-y=(NhX3i;N|1s`? zUUmn$ExH%gQ_^zbej<9hcx)KWTIyW(jBS5J;3s+h~D~c)z5U zo;_3`R+Gpq<v$+zcrij1ZL&S(D!$qkKd@YP>AnNc6JnM-FENY5Z0oak(7g)8pknpfo4bD?n&S4C_e)N5xDe(4XB_nC2 zr1?9h2Rmv7AYLDASNBd<{fJF;KG(VMe6U*AY*&I3#+eL0O!g&*@$)}&5==NAj(^cd z9Ir-8ZTP;ml{}ftTI`MA<;kf>dvB5Tk!^(k3)@*7tlj*hcpi@3nx(k}FaCzWfPipi zIE&G;M6x@@(qR%`k+fj!V4WLl4i*(PHJGWFyR!iHgK|xPN{)x!H$&FmoVkg-_aWJP!w-|aRk~28xkEi@>H8DzN_=2`T6ny=YzqVuV zCgs;enWFK`O!eqAY7ZD08QLvs{y>oyOYyXaF$?)ltL&4EW5QKLgh9U2M!UfyOYSUG zEk5Dtagk_bIWpL}AbwZ-pBJFCqkpod#A1%?bFn80zG>4VteLc>PezV464SM$2~riq z7AI$Fm5Vsxmt9RVSUi`MXB?4u&c_#ESSqk~GoM#!Gr`QuL92uj;$xHP6$(N%f%h*D zdSKv+2hsjV?-(}E-!8UysRWFg^BYtwX?bT0dEoOvr^h}(zbto6L3os5KSrU0LIIws z3jrop7E0SuGY4x5{#=)2Zu~R}IJ}iaTqfEPwNq1pO1p&~7J<*#56IXvzqRbcrm7_> zqZv$?`SI{CSGEy1TFh(dbWG-Si?bkgYO*~>=fpqHJ{as^Ztbtb?DH}ql@#zc`X#1% z&&zrzbuc{g&#`oMItIf(MJP*sOa`628dkOaksOtDBl2a4rfeJ4sb60k8^qYp_O$u> zji-6atPitsFzo|PbV_@E*@}!qy7rkni-&Ed(>-sew3M2$6?n}G3F!s%d0o!U64i^n z3588g)g7V^o|o!!^agN?9VAoVNB;bpz5r4;c7GajX#Dh)miCUu_VzZI>B;0?0X*Pf z*X&x_8)#6aqdaMm0Ndvix-cDV6e-F4;pHh}yg<=ca|zwPsLCu3vbkjkZXVN%hM*pY z{m$j~Gc1Sx?(B8Xd(=JZF3CPQhZ=U)`=t!`eYD#9Pv-I!$VW<;++K3WoUQgo8|guH z9BFi+rCYp?hIMw*U#fbwCSsVz+yp`%^S)TU*xm6g%)__O<}=VO4P}7O*DW4X2@uN8 zR<~v(U{aqPqzq4feadvyog!4!kbbbqZT!|helNI_GNCf@e7^%@yCE3kOYJ3?;WzNAY1fmcPe}10?$`Tv=#uUNcyMJ=YziKupe7G;i7W(ZAR+^?qSrxY`qWX+LX7P5YN zq(r|IHSw6QBiht2*qDLr51WL@?;j=d8&6l{zEMrW_ShCasJ>}q3a!i=gp71QW*_H` zgq=DM_sR4)3n?eS$PcGBw}&w>)9vDD@nv-FlfLokkHwH$>h=shX8kR9_+aL0ut7pv z&=kI#=I~qX6gO-N3MK)1u^#)wqaLBV(541!4qat^eO~DSJH3VLrx~6zK$RG*6Z9<& z*>+xe7DMR6$C+zQZ+H(whuuA}(B$n7ZK>L3y%X{j5w{dmhZEyyJ+mt+y9)DiucY_^ z1_U#%4>`h?vh^lQP83fW&RA{7V3n42UqrtCeEFOdBe>Dt?2or_jraIwO>IVSxli2} z4?I^sB%}6v65ri|>oUW7lQv$QVUtDg4oqEml|CurATe9!a;;7f-1vwX&9uDZag7 zsOUoz96RY*3Omd(G9 z2$vx){zm$d@3#ImE*d2=;YaQ>+An-ALv}lKKI~DbF$hN4F02?r!g|U9`J44xDKt9k z6FXhKIq*r>^kIgI#E^S4*o4W>k#kz74rE-ehUi>3;kV8<|HKjgAda|Xt2QzGK-oYL zecF=UlD|w!L81TYVTgn~YwgRD%72_Tx{|yGo9NCO9pnd$QHrpA#tX|7%wG28@>ha- z(`ySgT-NQaISf33Awf2EZuL&{6nQYC>^P29u5V^0D=>5fp;^fEOo91h5B5FJ#8=Dn z8)}t)0^Qffc;V{2J+PAL9JO3JO7GnY6e-N!vLJLd2Z2m!iAW!|HvZoQuklW9&EMgB zDF0vaFtz5*V(yED+@+ZEtUA1zjG1)uv!4se*PFkvWE9;882{-CJn2k1t6tR`x#_#8 zJDzi$yVW*GMK)?7PPWMNXPo*l9q#yq1pb#=AZTte~snx$+@<~O6qS@@Nbl4}f@wSv5e8TKeP`d*!QqID!fZ=zdfH*!X)UfLR~R(7Z~f=o{?s-9in2ZMgp4J8xUP$`ejx+FkjkJfxSVwO!GOfNtU9c zjc~@p>q5BjWBK!2x?USAFK;gUiW%7&+3I+#m-p!*i0V0`p9hs*-{jhb&nb=U%gY)a zcn5{>3CrVw3x&0x1G)+k4pu7w;c{Tb@33~eJPVg&^ZFIjMaOsfi%&bhGe8>deVrU- z$*A$-IBNIEM$RN7v4gZk!j%N;zI*JpZr>=f{@~Ftjb{M6sG(H5TwnPwSDW3EQf`%1 zuG{$7LjGW4e{k}4kF(BQEhnDM95d`QB8&turmsL_5UqTQ)auVsNTqG<2F7!nn@hd2 zT#UNE0!?XnKi6<(W*$l@u3b3RF}U29GWw18^EC|;1y`vn|5uN@$g<_#xXZ=O5fx;G@z9uo5%@NQYjp}URvK$WKD_7&X1TM9~JsRWoi$h zy9QVt6cnK;^`~^wYTp}44UD&@a;-q8|3oGHSCQ*;KEv(nXEN*EqTr6c1QK!cVgfT! zDpSpfs;z^TiiU#=;ZazsTtC`xsVH%d)ppvh%$)tn^>I{aU?k|00{h$i9cTwq0_Psx%Cq1~LGGxN(kM85vpjLtyNh_1GoDc)GDoc6{dd zv0x%JZfF>mC?Wc4;?yS_;~VX=BJ`6Om7Qp2gY)txd$w`I8l@IQLppOrw@~5@xw=2T z7ou0dV_KssG9uwf$Z>-+w-94LJ0IMSTBR36eZhq6wY9Z0a>?gl0Bj+E7J`=e`uh45 z&(m+O(pVfFz^8}%9?^b)btKk&Y2Dk5cfRw&E6$A_QONh=QN;GNuXGm)R*o_qPkniz z(dAjQ`uO;r>d8dy=zsuau9gbD$K_;0Q~MtAHCJ!M6K#b4fZ)5snj!$6tS63Yt-w$;c{QkBQboqKqm>hM&8zH9(X+M$w3`&0clRR z;|3%Bur7|ad!!T!d8bh+^i#;j9T-bxQ!Yc62t3h2Yi&oiOv2^m89Y6djL&Hkrsq}I zc_q3`%fb7&*%EdbU&;k1W%#Tf-V6@O@cB}m@i|Oxp|k7f*7qA7CdCsc!#VgD7Cfwh z&Nc$wxWj~;1xe3Ew3n-&d(BjCoa;!?X2JpLP|(%F^Ahw{%Eqx}>3;;Dz{Ms?2Ng=c zV53vza?&<*r4=^aok5y0CYB z$pls|O=~=Qnt~L#Z0stA6A)w^sa_lF#-i~MNJZj8yzJIRdi<1@n3&jAFYe9)V-LXN z9~?Ls=;0ECEbKWErveeLVD6I?b%Xsg+=+B8$*xK6z(74UE_>Oni1dmF)E=&p|TdG0lGa#=e^uk)bnW z4Bd)r^uf8^oby8bL>ysZotoZ;e3c;3bT=q z(Tt8&1q&veih&`Y&GIVF$TBuWg$f>DL_B4LIRgdWbME+dwgK-hNO97M!N@OJne)lQ z{g?oDP`fD-aI_QSG7*fiw_cm4)DZ8`GUrD(ksvEElkB=ZlK7smi=HSHi-&g%Z#UmZ zS_tPS*GToo>=TIJYw}VP9hby{-JBlP=VwmaA5&DB3CW(g*}b2-l!(0#Rz7$xKtD88 zH{~j1{E-49xiYX_4u$xRTGu4yA$#$@O|tGSA+6;S%*ecuPwyfg!Uc*sm+i#g71Y&j zxg_!~TW@XzR`AlCh9}Rd9nqVAY&b;}S;ZVbNAo?t6Br=#)0I7Rk`Hrz^%;n6Tgj%- z(Rl2_Y$m1HYd?egoSH|>=VUoiVHLzX4$c4|M3fD$)L1yJ0k#XLAltJL>JQFGx)?Jr z>LON7U&>qEC^2)krj0F9MVYCwO(hMJ;dg$fc(|v)v_JnXzq<9ve4w*ww(&;Hqx!Yi z<{`Q_<3ZIy^npSe78+T~k(8Mv;Bh&J$kR=NX6wceH}8|*yL*?%{rI~m#kb?!tF?)% zooXi$Acz78Ecg<*%U6nPZTd7HPYA)?Gr|D|{t7$I(scEXdzpiPr zaE~sXylYP@YG4<5I75ULqgJkavA9za;L_eUQ=ChG4E~Htim$Jjsp)FD1q?tHh5^e3 zG{yHN^6a0VT!DQm%F8+aF8sF*K9BBhCT*TPFw=)m!@fX?w`(oPry-e?*yH09v3ATn zy5-gp;!}xrrvquXQ*;psTRbCy&z^~CI6rU85luBW%OeW^r&=caga1t}vowI?78;o7 zL_GQK;j>sE)qhhK1p%#^h<%s#FX%`#KgqROe!T`AZJZauR4ND9jvm? zd3Xz*nPy9~+Q>Qw%UXEzMA&MW8SpPom?wFU8d@8O<4f(2H( z=`najSBLrJ)!qvZ?|}($13*Xp8&a{mEaK0njLD0)@L0Cx(M+b_&$oYXdrC$zFcV8o zz|A~hGabb+Gqiw1Z-W%f+^=d#`QCgwqg{I-uZ5)Gs!v9*a|YwquRd`H!{*tL9b~y1n@*L?{#&$7r zo$$!tCQN9=u^m%W4J9}h8x;^!pw-XhKrbBMU-=%d{_bBO#oJE1BPyybd(c$gP;hL} z(KSK;j1&3kf!^|XCMQ_(t(CdktO30E_2VrN!gN~Wdx z3O%)C`cqky9GSY>wcO8?A6_%QqQ4dpycOx@h+LQ)b<|?8{D-otac3;L19MZff~59= zjCplA#X_5f}E6OZAaE{=Vz};m{QlsRND_+ zm0R=k^UoHo{3{p!^DXVm9>}<;{WC6sQ4PwPCrJ_S2*{whNwsrR23A(jqojhys@{Aa zHB*A?{=`m-&3K1lob@x9!RtJ!(XMuA@kKO=6yI2K4}Zz~f$`1hAS7(CO*qGd9Zoh? zz2A8c2Ng4iRKHd z4C`NVLRIR~^HC27*{-&5T)A6+uBJCz)57kabwTV1i2X>hu@!sU+i7;Rmrt@p&^$e! zs2+}B86{^`hS{&hblcM-$r|FnIlu^iei58I{>syXl+l-@D=qm-hXYR>vt5~QQJl55U({mx5QBiz#EDsNx99Z_W-E-KALdA_?`gz$+Xey17o=4& zh%tF%kY$T!Jf$d6XT-fHL;A2$nkYoPP!$JUJwcFZXDMQVOq&fDoE*7PnAaS5vlJyo zjg5^dusDT2N{JT37;x`VVBhoAFF7vUN~=v0^6~^j?nMup(8+Jzx)u8jOc}1UTk6IZ zXanD1=?ShZXv|s4+RQ*GmZob3$zIFHF9X49IKV4YPlk|Q#o#buF<8cK@zUA1VKbSq zJSdRVaqn{-A3|0zYt#zt{pvo%@QfR1O{8~X?@KD|8PQXMIG?$@r-;kqdKtmiRUk;s ziYx?ki-iC(Bpp)q$Ef12=S6u50}%3cKs8@W!8Drdil7DbxjHYTU|0tLw(EwG2^16- z%8=rsU@5q}p96UnK3Uzu5}cBv_xg1J&@I0Cpt^vb#HGc>N8xX1B3@Hnm1C<94h}rd z5IPN>Nk$qrHt;WB9)SjbCG^|ES6c&k_e;R`S#{u*lY?$g)j(Wb<9KtAbi~EQr^$gh zEt9~>p+ev)5=gDD4C1*7gxAEvL2Iem$yjkudqrM$UA?|2QVd_r1EzP{Agf5BquM(s zmLhw>-Ysaj7Om&T8|Bg%?}5EL(Z1jep9zyuAxch>6a86(ep#RPWW&jT#_^EDB;d8F zme;Scr_MA*^|myO`oaszvTs-u*5FAzXTjf2b`Qb}lUzj?huvBag3lI^+>qtz@0ymH6WL!G`raWB@FD2&ozFC(nQ3u zaJoQkNsd64(jUi8E2Kk}4i!AJ+?j0xVP<6+;haak6$+<-a@;ry)Hb*aC;M zq+`Hff`<4y90i~<#OQhQylcLtbQ>*oa82-ZH3Kj&A6Jp)D?h_$RA+Hml^<%jg+gc9 z{OPWq=gCt37#|j>+<@0R1NOg78@jio`{A$}4y~*^2ZWX)+K9_~k^mR?DzeCHOkHQA z`Cx)WCZJ&3dUQT}6OL~&v2;i@@@N&uT8yTybA-^J?0)^R@K>kzmti@P)ODXyw{!Kl ztjEjD2D4>0Oe!l`hd%v}e6^wCHJ}##m#+p*1;e-0l+vp!Xc?dRQ&P<1{`P{`7vK)( zEj)ihfmlnE>0sM9SeqfJnNxY!6aAbI5PB3Oz?a2i^;g>GVDu+5*=A%fY1Z*JdX)6h3ofMpv>A;-*c}xjTeXs8wNHq7cPQ` zMtk$DcuIZCP0KyG7U%sv7K4M5BLlsnpp|OR(ZUb%NW-I0ui^GqbArDqae0Qw2Sw^X zDe;{0?yi}(Z363vl|RM~&g2Zpi6a^=w`;1E+C8cdnTlnR@3t%n=h>iECuj9KD^E$2 zzSOH}kaX0Q%eR8c$uk-PZ`A4WqhEm5d(UM9f1x{yfk7n7%muShyLzurX$9zF5o+J@ zQjde^D^F9zYo?YUlq6mhXE+vlVyE7kn8FmqHtnb6Nth7&b0yL$3lJI|&L+j*MJ-(tro;4sZim#IBJmox8d$Ux1KuUbV z#I)UKC7ZC(CFW8X8w4HHLB(d_UbR?wTk`e+C}1l`+^(`BKvB00GCn5B0Vo zwHgPg#YDQO+fC*6YLOW={4*iRJ#SAF2nX#R6Yvzy#Py&;mQnMLo(PC-$k zwegN!3zOk}TxLKYI!^6rucevF=A5(dD_fxAHHU$;F52imW24{a@YvYn5l&F;92*;> zD}3;{fIBI|XbeQaCMGNQ0?=m*)uY=QX=0-_3z@zjOr^HAiUNxJ9wz!M2R%K#ja5W! zY(Qos)xl|{Ri>&2AqNSQIW~!5&%zLX^c!s#HYPY3zkyO^GI5H`*u_NX@yZx4vGZFv zH3lncjyVFD>JqNp)7*7MztDEClkvNsXA15l&X0#CaV@^|sj1oz4Co7(thAdpF+js4 z;I(eA%zrr9C93l+uerH-zR;={4A&5%Rj$>KP8Vy_MDcc~2+!q?`1*OjSV%aIpNU!l;o^G&1pTe=+)RYpfeMJ4^dluw! zSR#o`Tzoe9911~Xg}?h9#46dRlrn6zEYomn z1HCGE5f`H)fF#K>PM?eY)BkbeuHuN-ekMkT`;ZhEq^-n5NiV9scJ92=vS}IUe@l z_ieK=_WnQ1`CIr;^WII@y4vI#4t`V02@|@k0_$lf7vQyl)%3Arnd|;C*d_|DWWr|+ z0K4h9Ia-pN0r1LTZ7l}>y65}p6M)|3)a!IYF*aNnRe^q zU)a5s`c?4c+9h3w_SYp%ku6UO5e!@HOAr;N1`JyqEUe$@VnNORXqaRqczEp#Q7y_? z-uhp<4Cm(NfOUh;9n9NV1iAu&C@Kn_cgF$%C0hJSed&u?ZaD@pn$7L)+4C8wouxn~ zVl`-I@g4B@!yqOT@~j1zG@8JBc`p`58XK_WmbGib?YH)78t=bq*q*W=J=;h8F5kGh z5@v1KLOa0EcRAs=&to{2r`BO!i_HPSi1Dzs7`4iOID=NhFVxFDbm*M@eWws>%h(~$ zSUl6y&P7X$!N_>z%0S|qO00hw)P6?ETdJrC_K#R31x4~J1Lxz;77EVG+3W7y9 z06_$hIdKUI_6JI!hXqh7iEEcYWG_k;)5B#meLiNAVm0<-X}K@Kt!{c^q6Tbx;P;sl zxo;>_m8f5-*&s}!fAIpa>GOfU+|oh`h7o~Xlz{c)0DZ@;dX(Y1j5&=DQ^4?i35irI zoXAw*GAXWiFm1|-sq;t_(+Eo^H(C8;-4r2S`Tecb=;q^}=x48>rZ;I1cD*#dCc;VC zY8R@>LkkjlFLN7m_%~iI*5QtnZr;@Nfz#$=-%$6a(_9TvK5kCt9%{ZVQ z#0aR<0gHA#6=Fa!&TnjN{OP!1Y-EHL{?~^XdS{l$%L{q3zhYu+j2*7dSoCvgFqMr? z1{G>7a7@d8K9f))Ip!AV>ie~Bvne&(NCu?R>F)b1n(3t-wtC1;RX?F5ff0_B@oO@C)MQsh4 z$Q?a6@$B&H`hfRA zg~*qtd0!iqn0G(GMHPeSw-C)1tlhb$0s{q|8(PLhoUy3Tv>y_fO}8BS4+Sc4E$JE z?PaAq>-tRC5!J;PG0U_QhhyGAn zx0_bAXviN04G`Vb(}T{P228zx&oa~#>yAWTqwJ$lKcuF;<4|ZyOv*P__FS#F`fI|9 zcY354D%42A`oyW7tHHO!uoSv)`m(WhIt6WPN*D2R0)z8rgZA-M)w0u4WIH=HI;I}E z#fbb{ohjp>!@dj+Zl^75*+N-3iK`|=d_gVOS5A4 zAw?JVi1t>vNriJabZ+c;M`q5f9PP-)ygw3Zajp5$HG)qj!~f!W)yy8!?g8cp`)=AC z*c(gyU3iKF3}X+M2*PpmxSrgjbdgddoSNhkQL3^wO&+qX7$PCsnoUh*#+|GdHWqYN zMO2^EXp}Os(;>7bOIjZ*ROIXPrA%^Vq*o8S-g=>$hFxZ1dtFp>!}8oIh&_)vl_yk=nv{zJ09wG(ym9{>c-^H>;y*4Wect z4=ZnlQ}8O=k;Z2Jin1vc*sWTqaa z0^%_L2(Z9mZJQVZ`BnhYno^Z}z3?{Hd{i;Y%C1#0*DNZU6k)Z#rHfT!+VuNHbU50Z zhkUg!Mg!$C8*J$s3W$N=y*aLAMRaH(^iJ=*Ix8qFSOeDAy!s``N1Es4tlh#K>bdZW zU-!Y8eRW_U4~YSMamOzZ#a%FkJ4ADT?+tIEA0=6etLHRM+S$USNOpO7`{rMn)7zOW zhD^|=Q4Tg8owUCRVVYvg5=1`l3pNm(pNmoFNG#xR1j3+WWU)R90xtM|3m)R?=ta;^y#m{ns3P z)!y$w3#(OM_H$y0Ml^)3J-D%SH(bGt@veK#k@<1<$s#`t^P~BFS#6mbd%h1_%5{@! zq50`fmi!Uc)9!{dyv2T5GE%SK5!NF-M=K3lifMAWDfDmQk;&9xkWS}>i%j!9EfR82 z3_H7jRDj%DkPrN%EQLXf*hI8_7p4b3*R(z?jmwk+%*e+!#{ztOS4Ivr%(X+e!U~EQ zT<>>`XSL@~1+j81f#dR$925e+T^9m0qQ$`CEtUfjbP4L$kX-cA!M$%V_-^FqL%%TF z_{EJHHO7;L6kKzb(o&OZ%mX5zBRq=&Izqvg<#ggNHf&JloR0*J+BJ1ij}c>Y4WX^O z?tTwU-<_DNTbk>Cjb1oV=s-v(nRA^rYP1^()OjpeZwhH$;yc0z;U>;bb2wNA3SO<4 zLNFmLTOj}YmKN&P*vg+Zn(I?A?azQByYunRS!ie*pvfaIF981@|Eh2dYRDidWOJ#O z7Y8P!Zz3oxZ?1ZirNDap*86|#BUi>=wcb`a@2`J$k>#ze)WoQ%FoyQtde4Sp{9M{l zjY{K_`fO>TJk|P}BcKq(521w!io}oB9C0jC(*I0D%+*QoQlu5u$y8%DFckT1Rp*zO zj`59d9oCRr+PSm!uq<=D%I-g}D)E?p>2Z44H-miSbhMX}-$LwumL{TW&g5i%JU8n{ zUXO5)IhfIcDwUemnb*fg7OQ3Q%gA@lC2nZ2NTZ}WS|O5`AEg4Hdr^~%~? zD~oPr@da#(=KjoZ``1S_w~tKmZv~$j ztfUTXg=Mn9d&FdJ!p%glKR8DPuh*;iZWvJ2+xb0;!AUz< z=?6(m#7|Cg7BlzPh0O`g6JukCKu6sfYi?<|4ScV`yol_a!L>c(7k7PM36f`Fq+H$?C43=2r-?&rwaa-1Q63-&Ch_Fbov&1c$k|Nn?zsM@_nJcOsRRM?f*~^Q74|<=b zlvUE`M87e$b+Fzo{JTc9?78RzdRlOuJ9@Gag%CHj- z$eZ&{_+c8OhrTv8Xjm{0Ma#KicLf>~%p2M2aB_M8x?3;}N?pkWKDsBT_=b;|&2%5W z9c}S9)Aq)3y7ra2Fu%8)=4OHNzF+*P(Gi#*ZwAjBg;8yPY)xt!TCfJts{RrwO(Y-f>qJN&24RRVS}e$X2rj%-hIq+ z@0);$5_i4UVD>Nu_0D>3(gf38&Hl}~=kA^q)xjNYIhm$iODARbtu)Q?dP>9JerEU+ z{xtIWm28z9%VUa29ORYJhMP2xT-vVA9j->!doC~O&Zb(@l(YZuI;GRiU#zA~R>gJi zSzc@q7G!rEZ4MNR9e-u7qSi`ov+4U!dkH z_#p`;#%AiX%r_@{s()IH6-k-=*EZeOnl)X!8)?*AYvDrV*+Z}~5>em(i)sA*++KxI zOl6(B_9oejxgORK#(Oz4SM580gOT%YU6f#v9NUb`Gq>U2Z8((C(w{W5J}>c=C2|+} z2<-jMS-g)PQ07mR6mD!6<{(4h8{3-ogzjA?y8e}H? zmoH-0Vve+cg{=B?>2oti-fkPk~pYLtVzn+|ExC zbo_E?mcM5>`UJZw+>LU23@j61>=vwJCJK5OA~}mKKQXQUe70_$QpkYsUb(HO z>ttSIQgz=r{`$Kf)wE0_=y^W+7FABPbw?E8SmVxX@2z%%sZJc#SX!c`wk2&$=(_LP z5$Z`w$M8NjoO>9~=?EO393QnjYe-m*(XX?}(N+G4 zzRQrEU9E7}ePj$LhJyrJ1dAEGX9Q=KT$haTz&$;cXIQAM?-xalGV|7lg#LXw#Dqhtw26@yIB1Rf}h#ldZXE2*1=3cS`tBkF|_?o4N#4^C)}n6`YZ39G_Jya`D^4MKDmzXk?+ytikJ6!=6F)CnFGL!GW4OylfAHSior;1UA!sH z)wW}d*;eK;8+f=?V>mb!*&h)`MN6Xh;_^6sx0NMP`RQXvUY{KY>l+z)2#wZmxo2HZ4&gnZw)j;2`Bgk&bJ3O>Ib zcd-s(AfTX7SUe=!ir7q=Bk|!2)^R*NW`?`HJ-NRAkxcqd=6R}yJ!sL)sfwf@7-i%mfbyQHX3#sU-^vv}X&fII6`kElvXOmB9Ik zwG`f@e$yA4S3fQ}$=2zHF*XtbGCf6^F)ru5P%Oo-O1JF#ZM=1cvNQ9iz;lp#Y9Ar5hxNZWv&Qvr&BC?>g7{ew_VFaDx5pC+>UQ>s~9SLcOy|uQVPyq)$rCXkss#g&M@ zH9+-eWC?*cV{-fD(?@7M1}p%!@(Ey51-0sS<2nLnd9+xxE*jcsV9m0HqM!K1mrK9) z^2ojtzuJUZX@TKMlF2GvqJ%Rw=Ot2d>cyIq(vbe!$bpl`+LEDj=d;%4D8e1t+bN6a zjHz}{Ua8j^=psl=|3ygNo3?UR6WpAh)`;Wly}CL)ZS}V0;ZUJ^V#-QHm1F7@jx(5b zlY;?&+SgC)?~l|FcRp15n$NvC-`#Ni=0r20YC1dlzPta27qc|T=laNa;3eJU9xpmp zY&cHh2ch;a64e19^{-nO?uor26fE+V>JL+EjhBB3l;jKRsYW}972$nYurJ|!1I1qU zbGEu7dBbXlKC@0o1qa3VIn>bHB0>Zv{70{D4m-}gB(k_`tRw{ zf-fq`buhZR7CVC5B|!*xo5~U%unl4H@f!*WU>+pLQ$gM$-xw5)DKKddtdr5a;g zl7=Z3YxKpCT<2NxU_4jwzL$6W@pVZbRw$4lk&h?1er3Jta(YE*$kjHPRHP!d?V$j5 zo1I8xo13XyD(u`zi;HMswVJ04Db7egA;j=u2x12p{v9*~ices&OZK5U$mN?kfmAg6 ziP{@qtpZy;BgvCT3v+kx#GnzRvY6*=DzjUzlG&QvyxEQ5!``Pf-VAcRdHi7I-jq}} z!0&0YG%7BN&d@`%!3qZc6bYXbkqYQOCB?zjfVUf-{kgMDc2NKtYx;x??yv0Z=EUTW zv^}W}6R_9;FaXT|xjMYcP7gs#UNtT#FE2QLW9-i<1eTY(r)0kj!~SjxZ|M?F=5y&$ z)huh^KO9b51WzB!cEew6&u<;eB-RV4e3t2N0z2Q{C=8>>pFHQd&EK!I&>p3eL30xn z+1ZVA3B|>3e!k|mljSrR3o>fWt)0YoWvvGrIvr}?OS-C}75CQfKDt#acNh*R{r!N1 zangEnVVcEy7i4QbnwFL^{i!}NqNQTtbB0k~#F2O+vdR8UNVU_R*ny3Tj8;=b#Lx>j zhWZ3$a_38aGi&QaHc(Hk!AfMdame0uIpm^rpWix$2mfQgpE{56U>$Pk2`L)0{3PDI zNRVrDew!V!_u*hwG$IS#%UxBZ`!>0j#9+F`?Zj|c^CgX5e^{I}ANRXr<%KqBvwn91 zm*d`^Mh*L<5Mw$o@P&IR2^$3k^vR2yQioC+-OffDNkBnt^69n6JL)P~?bNvBs5qK( ztKg@vOqWiV=^u7>QaPM!De9;mwWZsIQY-}Xzd-yf8YjQJzV`3KzQ*O{n z(Ie0LlZkA?i>B;#yJQ)eZ{`H>&kAoo7mv>E&p7jPg`A(uwx1z35jL5ue=j=z`#2X& z^m-RI+ptCPDqn0a0}bGdc@U@Bc8jaMw&0G%O&M`~*#IOhWW4Q~LL92Kfn zB$qrR{e*J-<6N;4JW|MMt!XjnIc&+VyDwvZlEFt#$GQn(cv@z!!v--v=Yd!0D7rMl zuQOYfEdBhpCW_wRUksdq&*n|%b9hCCGK1$z*OMv_&qHpk+gTQd`oy@V00Htg)HA%v!04+q-9IV zA_bgelczo&iv?kx+&d6Ta=^7#2A@vFk5L7#cAOk{m6)i?5Hp7OwFgBOaCFpE?vBo2 zZ2mja$sIwt?LzTyGCMTw+hoh7<-HRiA~T6Cnk)4+x_l z?~-egVlo)gdb3+9mn6-PCTaVDary#-Dym_hgdcIq=*WcBM&jZpGAKJ7n$8#X zgv294Lk1c~XvY$R1)3cey}Jgbo_a-p_J6^_QNleqdC`CzF!}eCYA_6NZC(1o>aPd{ z0qX0;U@C0!cz1+VyqeVKq(E+^Wuw`QEX z`|GiZNU~+d{r<*_7S1uE@&nn(89wz_v#-I+In(usjGMaFw%v>vL=@ymW`-{&wo zLkgD-=?r)qY)#s>&ab2c|8W5I64p3f%p7q;go;E&0V%tY*a~YA+Gu7W?qCLJ zWLT`}Sx;g1qtO*&{?}Trjf4hVjIuha^?MPP$Nsz5&$T3&=;q3y4A zlTAj{_97W*0)nmFcL5mSbX3rB?7EjsiBJG|tWyMB!2(0)`9W9T6*hwuI+O`{-XWFM zA5BIDZ@v`R(Y_hZYlIJ`X!3sk&S`Ab9AK21Zz=zLp=;aRkLb;(LN>(Bl5bwXMCcDIpS@7Q9isV~I? zv7qR7aUkD(b1-CyoJt10CGl~)aYrJvJ7Cy9&`gB$kiIfd!XInQeu)+yW^=r;K!&Zq z{LY^L!+2t)nJN+6eX6z~G@TQ}$QL&U~IhTl9c#w@AKROnup>KdF=DD9EEu%s?`D|A|62%iiN43eRl6~6SjyK zY2-JFf_@&%0XepTVqw`jOz56{f^sxwKB#sR6Yz3&xe<&x&A_e+dB$`&K%3?YK_5~n>Dp6ws{f8ws z5ZQ)G^`bj&EfA%~7iKCKBVc~Cf>xH5h4;xhJ}UH+&H+pqgzgH10V>xB2-KgSdXmu5 z1^NN-F(7aFl@nnIC%7{y9+D3wYS~3cYjgw)T4EB_wPm2Hn5ZwikBMsE3Ew)YW31C~ucoYPu!387o5HkFVJsZ8PUl9xLtT6a1O1K}$#*X1;7VWZfs@`Fb-?4DO z&iZ%HuR3|dHng&0cSFUX+$I;ErH%;7tLs@xv;C&{ULJqKHFhJB7Nx_<3Z zc3mY~S{g5jM&^cMCD>K2%D>Jyc*je^Vn#|$hprOX{Wmy@4^0&s5E4j8j7 znqcy5&zLWcIuIX{q2v>^poJI8n<(Ln)HrM@&K;L+m85c~(w1AX2_7>_QyC?zHQwY4 z7Bua;ssUMLa?xuXDj4((DXiXwv(!~N8}DA1sRds|j=%H;-5mGH{;sRTQ*`-|9i9S8 zL+|%G6*47Erb0=`J8ZepWOSKV$A;&z8hh&+idujm(g8Nw;X^Xy&EbqX+&wYs0*)eR z|A7NLqbgx^cw$0e{N%)KIjwwG4V$HofH9Oaab~~jKsSh^ngPynXj)v-*m%0{CZ7H_ z*HTrsX&(XvVSX8Pzli4Z`PEjt>fo^z7<)uqj@z4SX8Ny^{jdmnFR(bD5dR%b37WwY zrTYBmH|qiZk)(r^4W*9=0lk&|h7vG;OsgYgPjzwvh{0pzDh%O6@lA(w#`O_{L)-F( z8lyywJRJ5xgQ))H`>C_dd@N@BM;C!-bAzvO)jC=$d{kJwf*y(}LEB{1yZUGBEO)&~ zL<bTv*>m?$-KhARCW-x4HeE{O0)xWHC6hUQa>knrn%%OK| zSusyhwCX*Hf0@tu;GE8Te##-%OgKIt=y){q)_ugMEN=>4O)mhke>W;^9;7fYlPyC> zT;@8OmfMX{rsC(fWYK8u4cl2P$V)w%R>Sq35zKa7If4JAH%75;B0*=C@A|gx2{<)Y ze{2%|3JKqyOp0V9S{%N;sP;cbCKhPrv~IoB$B!jK!=31?{%||w17flK!70rq^Q{{O zMPc=rNlO-P+xED;W>*I=+C+!cuM=r{^)lHI{HS=$)VO?-tjL_udfMio1iPZ~?ZhWq ztmrG~Bf-y&8HqV&A7WUCJD9Rx`Mxrw%hwJ_w#l8jPF!42pc0UjoprvfnR(;zH!O9G zv>6=#Dcb*>ij-8)Gp(Z6%z#~*sEHyG<@Hdey$IHrdwa>QX#XU|Vx99!Yetl0Vmd>(yc`vpQw!3H|$exa#xNmj40q5@yfc)E;uaK^pB+BbO#$ytHZ zwp#3<+u@-5_{nPm%<0xmE79f?qA% zvx!tGq?NLzs}EWE^S)SadgaY7KdDIl749Oere;^~JVeWVyY-s;nzrX$gyFpE(?bOC;X!|Yy%fGks z4Wu`p2q`M+=5MxKYxEqUcwtYzb=U- z7J$?g>~0d*H^NWDCG!O16Ly~t&Dgu1emg>EcIqQGCr-E}y5}K%tL=6x7|SQvF%jd4 zgG&PG_+=#c+MPB~@pGj6#gFd|-$cs4H3`U=6^v-!81C$I*jkB|?(I^!D(gsGR`^OA zB%=TQLlM3S+dPd&kAVuJ(o}D)W8@7El;;T-CcDFBSN6;}1y$|Ez};{?POD8}h3! zsbXBoqfL7`BbEQLWYkQUlm+jm0JU-0gq)P*)F$Em9swW}5-x#*e#%X#e9svBHjZAw z$UwmFK457nedc_5&`d>qb>19GODAS`G?~UKJ*%|M;yM@BslC2vGypup5a5=&mhz5JT}yCZFkm zH&o_vp@bE*ee#JaJ`e(^^WJ8wn z0Ak;>aK_40XZOmgSc3Ux#eRAok-rSLf8+yE9NSwOmSfz+o@6<^;R1OfPH z!FWu_!#4o+Ko+T8IUUzPr`$p8|1ISp@DJ3WHc>vN(1R&Ip@s)7l0< zXb>l#lN9zATX`bdo6rAH|5ja5)SfHtU+=Uw*7PzEyO<}@096h4Nl? zZ-E9bR_8{3Dr*&LOk1Z9Y!(mRwwSy#GB(Bl+Obur`?V#W|2VMa$&sW~ppA}sg_vuS z`rYK0nNvh83TELwt;@E@dnHai!u-=cdAsAC3Kg=I^DV=H#H9pB@VC?L zW<6u|l7kX^UP=CD=gW)n(PPv0?%oC-ef{=tM+D&vMz8=aIkRv%WxR9f)xv@`^szH( z$VwQ4`XkS!9_c%S)|u785}>o*Jdy_uGZ~{Wn3_MYsJ;9xFg2*YO4GI42#Z(XYM`L- ztf#9NVTgC4qOG9Hwt6yjb(rS`SXw>m75l`Y&@dF2^q$o|jjT$G3vyLZ%*MCMDC29a zI(t;aul40%t_lsZYnu`^<@^*7pl4nl>k;_5ICFBZ+aiG|9-SRLcb>aq%#yM|d2({9 z7mQ>hF0*HPtfqz+&gp;gX&Ac?7pDu)4XnE8AaOb0mouadXkbMYli-W0?6=^ZZu>aS z@~39^@i5kn*K!WtADz1H0SRu1Y~0*-OIMysO=f$oX!_Wy!?6>>B3b@73_)@BN56#W zZ59l92l$1wlB%DPzgjbcFsKyZgx=Rwbk&}E3hI^~4_}Y^EX*je{!$x!9(q!^H%FY& zT&MaW1;>e!R=6^6wkG7_Qzd|}nZF9Q9E0z3ibtv$tHA7BkPCYLL8u|2>byo*Gn^@Y zBYR2wU#MV6ZS$k>eRMQ3m=*iZah27&4w)5j9KlW%$*IWwm>n>l*t+2N{pbBs{S7YA z@Ro~80`AiI*tarE93uHg`LaEWwL`UW_H&_#R=|eedH==*<700I>HY9FsNpD0 z_bPc#bMw``Ln%M0frzVdV6*5gjTsGI>a&Wztt+N04!hqg3%xSUZc|_PJKkbV^V}Yc zRngZoQXsFt>S+ILmza=IO{jeo4o{Q88{6k|+Dtiih^TmJgo5|?%`Fle#g>V;ed(Sk z!gi_MQ8Xu4MZ(0Vw{)Fi?cCGZI_hkj8A_3UexJEQ21+uMGiR!EWMm4>HpTCER|Ibg zBOy!8%B}N8NzX8F07K?HYd}I=V|6%HRFXJ9*)9b zk#elmzmgZZKzD0kGJj4@qNbY}!tY)jfF0U$!7VU7BqPRsK6;)SN?~`lV5qq<(L;|N zLfCjUeNslBI{y)E8CEzIi9VNO%vXOEbNO$^w`I4MK4~_lZg@U8fOX2Be49AfI=SOg zfUWb8(QKJaCb{YBwYICRdQ5BW<}Voy$1Mfo<5vt-MorJ#P9ewDbHf$d^dizk6H^Kj zLgc{1VsV&>s8CV4ZmL)a`a&Gz=N08e-q;pv$1?}EES&p_w^CavN+KlnOV)oVHtqc)=*J@j?xzSI8`G|9C>ns zYaoHaZtrwF$W#!&CV_5#&6IsNw0{)Wj#5upI(n|b68pI<4IF6{WL$K@oG=jVb5EH zb!l<19v3h(nh_D)99OKL3i#FNGo314V6g5j%3$G`b|jFnGxD}_K2bR{5*G{V%d4O= zqM|hN530zif4BGu4Gpc!hI|WjY3$2^cS2gGIcVU-eDS{htfrjMg#+~S9~ZOw0dc+C z*7*}({g)wF8g5KmaJ9%fM4Keh$OoyX`>I`upLlB>e|kkvf+$e^fZE&_c*%_$RHq*u zH_zDf0nh})4>FxSkdW~`tvyN>*w>lV`dFudCU|-6Ixb1m@F}ZvAOTXWbF)TbL&8e0 zKrWId%OoW8^tZp=@fTxZH=~j3L=G!HG392}VQKj?cS4n$-mQzo^ifAr{_D#sCp7FA zgP<*?DAe%vqe}>Ol+@+dx4r}SDqn!Q-+w@^D`*J@|JJV>d%qHTL3a0-KLrT5%$FDi z1$QeaN!=IGbT$ZQv=dDgHpbz`Qqzf~Ozh*$Q>EKERVf5$U?*d%&-|TfXJOvH31fT1 zwo_H-7y<(ee2(IXE|G%nNGpofuvpMBKZ8-yv>mCM?oNRs-4GBXMW9DK@nM~tJGc(x zgJM(y^%Zou?I}0V-mYN@OLF)qm$3*sZy7wteTu8$d4Q!izHC`S($@7#Fr zZc;im;Hjtd0b0pkn<}=1Qw1DE=#4C_=(D?nU`dBimu$P-&BBf`&Vv*n+LdWo0Z({eF z4y@C(?VRwwYEAtFZXsL$eth&7`M58Gx`3Ce^iRIP7`yxh2S-?6TkN9W9Qa-FfXkEJ z7gUtHI%RY;Kj13+oJ>M`&SO40&=MZP95f*x$f4!I4K&sM)nGkMLvpg#eBDL=UT8>& zk(x5Cn=msUHA(HU09S~<+MNFby6*tehz!J+G1LJf8R34aF>9QPZyy4`Hnj{mr=jje zjQJe8cn6*lT|YWDviaB>?q)omS%|-3eugp$_?mlm-v+4ZpP4(HbehR2?s%rAv!spfy#wM2+R6UIPMvWf@-Bjjk;0J?`$S%5% zOh2Sw&Lx1t^ItDEYWR>X?^~xZuk9BoR(*g{eyB24LD}VLT7|#e-09iXk-nu<%q`Vz z{oc@JKAaAS7=HQk#c@8+TQ8D_Yr3;jgn=>qln}uim|M0nAnIrd?}u>Z?^wfT5TLzR zC=;-3>{YqCK%_PW5&<$YPoSBrwy3Bm3s6Dq)Md)d$-w|L3>5Md@1lP{a5HDK@@{_d z#pwuu#~EY@fJ>^N6%8IT%@_M4#D5?;Z-+wj8%CnLprY(Uq*(-EP9*zH-PUaB1HLhUoT7C59N%~}V5 zLRE7${c>1#=kLAoXrau;Qm0WRPjEG{(Fw9Vs$6Wy=`G&rt^j_Ow&}I_sp~QL@6kgI z;Y=OO@X^BIf}J0F>rmif-wY3>_<8vzc6yZzcC$Yt)yEh!@Bg+%@kM1e6U~O9bn=IC zNMYE~Q0$V9Rc)d*+f%8N^j00AcI2S9nWCXvMrs&u02rqE0&!V5G@Hz5=seS~X}Uhn z$!e{CQ{bwGgjVJQ20t@{_?wBgS+d28S9)7Zenqebo)go(Uexe|jlp@zD$R`7SSc_3 znAxK>*pCcH_U6Qyw=e(x!3d?#{(xvh7wHOw0=_xoMJC-6eJm_h?)xXnC-5QSN!fVg z%F8THNoO{Vs@tSZ{2{k0y3MNw2Bj*t-QjvW{jQA{;neV(+Ur@ej5CJRW|N^^i|}wI z0$y`Rwu;sLx=)E5B6({2gKYwq>sr51IrUeBP==Cfz%+EF3#)@oLl4o22xn~Hsjc>o z6+mNk0K<(s_W@k-wzA}hnKG|3%MCluR}CIr@Wyj~7IR7>DIJuMwD}8!II{jf(KY^J zb_zgkdn`&FkN4~DEBE&$Hu!pD4k!%C?q5s?S2(ayOe)UB@jJ8i_7|(J#%6}!I*^~o zjY8#xC-RbfZ!WVwIi(t75_x<`kr=$4C)(e+I22ng7ay&-Y?`E=>I3@za6n&=^QDYg zRYNDPd7i8uS0UTftL#W2?H#)}=1^?mKz}`rHU#fz)Rgwtcm(%Szk|KHs@>2{D4*eh z5J)VC+?R<}qyC|Gshj%CL0TKe=0m=_s^`a7!rQMzsLWG~D2zEZ7VEU0hT;iaHTF80 zS7M_u1C5;TJiNNPZm#xJ-Cx$Q%p2VveRvXLV|I1X{;sXiFlVqh-p+oY;vCCR;|1L8 zizU3_K|p+tvUSb+n6m`_zVWA*IIz+*PX_5L%l(}pp!zHE60fMGmgCC8gpOl%|L?re zW$$3kOO|^AAwD#RqaA~Ks*4vK52z^=7$J6N)GB!a&5`ji35=Jvp|nck)TIx$*YjK> z)ahuyUOBrrc4zp?Xg6V{W}`(*H+it(vsvS;M%`|~qWWS-uNQY0i+TeiS19;DudH7Z z&%3OS3~&Y)$>;`W7vaZLxAETtle!|=_4W>>y-b%}Qo3$!Ake?A&zW6Eh@+vb|OOz<$#9@#?ueDogwjBAgw(Pv)io#`Ql zJUe=2GGe`~1GmaN?s)zBeiiTzo%k=Nhk;<-0eQr}N=IkloXPV=2Q4Y)*MhqDuQ><; zXpGn_wxb??YN-x+yBPpoURHp!%(;_l_oM68mOU{m7#?jPSmu47WFOeGnm$f}99sh` zo)Ld&P9}&8lll1gU}ixg1`T3kJaK#O)V0-QXa>SPY#zWyKkdMx|NC7n8u^Z9ITiM6 z_%J+pUp6ZWy56heq(e2RtTlyTT|08(El2t%6SY(`LV;VG>v%`YvZQ-L)9+n{{>0R> zv$ez%A`a{0X*d!yqDnu`L~Ts$_mdu#sv+XhM(qnD)(ko5J=5zqTbwz}c98#vc&bfv z;asJAD>Rsw`~{l7NYU$p<@?qBOM3=#o2hpXZ^!#y(@P{8UfDWk`&;fwajy0Xla>k1Bh+6 z)JH8Iv5{ z?aVRdVMfnsp-ip=UVYrCtTb=Xfl6}!#Nwh_0z(_}!u@atn_M$O=3P#9hS>_U z5tZwj6Q~1#!-tNpZ^Q!u7bs$a0hbid2g0vg0|2B7=+T5*Bbc`w%Tz@TZ=crQJ}rIv z{mtqXfVTyvpKo;{P|#G{;k|fj@L=M7&Zru<*QCKRrt?IxW*hsw^U20^k!5am1WiiZ z0`RWi*tKlk*DPyJMBvT8;pJT|JbISoP!PHBC3g;_=UPK~3m|6MeQ3K4zMepHa*<$pR1DJ=&$x=e>?Cj6IfBxYb0?KVZp;31YJby~E z8>g2+K`%(oi9Zal zbZ(U0`SMq^3PD#I*f%mKH~qyFjg7IheePe5(7*r$2jXjW5DVOq)R`*Q2e|kBz~6A= z|8Fwg1g3`M{r4ZBM`r(uWcm7YB6tO#hsp|`>7Kl|$S`epXk{BS*bS9IWs?3IeIgvF zOXd*{Eoz>BXcn->39h20dy?dhueG~8Gc@O%(VF3}J*W^|hGvWr)#h|}2 z(w;wWt}Pg(+)9;vsvBjP_=@1#RJGO^bmBeMhmXxL?|x>VAF>*9lePYDE9hYkvWZ$^?(a*juFt*^RTkeaA$0!$qO1wq4fjx6SD>@UfTH3e|_$rL8p3h0mQcqm!MQTYyw zSgP77&zNZDAVgZ-g-1~$fGR0s>BOwo9RRfzY=h`HP}x4x@!1 zE)3K^b%H3$kQ5H@!gA!G$yzXCrX5o`XEx&;6;DztRR}HYhW^Wdcs|nNg?~N9`;RJA$$X^(K&6<12#|(vnau)p z8u>5AA~r|C+Erw<9p^u#ZC*f8$Fgs8{V|@7IaGW(bRs!)FO)U_3MqDOHtRgw8iTPJ z?NPoc<9el9?~h|1++H=>tXpjQEz7jza#nAnSCSfhN&ybhsppgACB&W4D+JlX?67$K-ZJMTJx&HB<}ukF*AS4(X1 zaV>#J=7B$VXR_Sr7qKBx}e4aQsxz(ro?TwjFR1*XdM zz{=e%L=cPTE@=M#{(#t3AG`+Wyu%9yTvwuyzb`8}-xB+9q_pZX;k*DUMhj-4!~!=% zsfoNpi*+D6CZWx%ru! z``1~4QieU4)Q&dOT@&$#lpyuyLhaOg!Hu|0_cu$R1hckzJ$DqR1+V^2j(V6 z46yUx^j{7*I4)mZGP=^Z40Ujh5mZ?$MdK0PG`N_rTvm+Yzl%z+?VRlDUVBd3sGWnY z)4L>`EKP;p6YbQwo;s|v*l>wCs%sGY1~)vnscfwaw{M67`u5z|U=JifAds*BZFx=Q z0uCRFVh8ZftcT}j%VNnQJAAZb;)RE}YdOoKViFQ4!xoWwS~f>U-w6Tk&>oX- zDEG8#S@S*)9lj*MK?nTfQ)*9=I-5^_YZuYxkEGj|Xo9{R_8S_4iR85|mbZbR;CLR- z0gxF-_kRplDg3af$CalT02DC;-aO+%;%#DR!TY+ua1Dqyx(Ry%T3kz=?^Fren1KK4 zvN>OWTo6%@_bPc*0@Kzu9*f10)sBRoP3w(bqidtK+eMOtnstbn*e7ue?de}Zst!8V zxMW8Bj$xO25I+7h_qUAY()|4SR~U}W@STL0;b-*l%HuV^_CnTr6sp-VqVs`Lz=QD- z-6RSz|B@%F>2fO{bezKG?hK}|2K40s*`FtTfF{xHHEc2Vk5aHT)!hNqI_yD#zD(TN zTqp62_?9VU2zQlz%{5z2G52BgH|GmRRt&EAu}5eXBsu!lLO4 z4K@n2@VJC13tczCd`MiBf1!|_nR#6A4?U{Cx3ABNN2QT!cM;gzUG{r+f^Oa&ECn%=XLCt;`Mc6g)X>bk4` z{f&cN#qasMlz_DW(CGo3eyAklPzeb7ZGdOu;o*@X8X6K5)c)$u+K!?vGw-&%ZhrAN zRRFqvP2nk!UuJnhHNJba3e{U*6L{)+HK~>B8($3->*fra(gI20CmeS^M zy~>r_2j~L(+p+A&jDT|YkM`QSI$+M6CR#W3^!H;vZe3W=&QrXB^#On0k2t3E^MJDB zTXM?6ZHyUn5CJ1g086m*%bm$Qr(Frdxt&a(eo>U-WS*MS{bte--d_PPpw~C~c69YF z>Q;~WsTRfp>A*?#BrMTj+wq_|c=C;YlOxCbco~C_J254qg$D3zUP#8IUB-6$WoNdU zKw-_nxj2?{o=ftLPVk;;+Ar@H8kHs_v3*)vT2l2m)Ie-Gn>U^70xJcs9WG8-n%e;jI_R?penucCmnKRDxkxe$t#P z-9)YeUklC9#ywvL0aMO~@}HBUE~SwwA-T~UuiK@%>ua4W-+P-F$4`w$4;l0K3DsGg zHeBd^t);;iTuV3kB*C;7oVAbH_c=ZvmN_=&G2iFd4W|JqEdbMh~z5*|>2LxHlp2zj9JhjhX zF<1D-hah;=5iAgldpdff>e(&!kbm0A-JQ;*Z@>J$oGSYqi^2n{?{1+9+Cd_uDG0rS zE=~_5BZmhcA8Po#+iW_|_OM8}CaAr&1G6d)+<&X%3KmJ~y0w+21Hiq-W-?fa`e+E$ zuberd`6@5~|ldI{5h;OR*|H3^F* zw~dI_D#CpV^n)6yAuK7(1l3#z@(M%0_WsJRO@36&XNaZGP!&%rtED!F}Fjw5bNuFs&Qd zgCNst#*|R^Ff-YjOW(2QH2Pifza)Pd8ASFVK5=4zz0-Y zc0FCSn9pCpNQHjXK-~ou;7<_Wt!2~l0)UtC3zt-92`#art~n>nfMfc|P& zaA25kAAY5oMg3R(bqIt`Cb(p}v=p?{KE@X~xwzslH)bhcLfx^iZPZY<66!sx=w{B= zL*;?ZT4HszIHzQcaVo=)Afi^a*)p8!6Rm&=i4SvjTKSnZJG3A@9zHiLP`=6H z{CM^UgoF1H>zI^q{y+}pS28gllaQRGmnyxi)7h((P-NHTO zwPwQc15Db*$c43(5p_krzX9m~KwiHcIm^Wr5-E`dY)Kn{ng}+dwt*V^n>Rl!rpsfV zx0$>n0wPO#`l16ynx888D>!HFF4Y4CkydZdZJl9{6t?o7ACfK&7awLQ^RD{am`qnR z!nYT5up>e(xZ~c)dS9ii5?iG*8(v^R=5nkY6YC7IaG6EzcY=8)w;WMrsFfuevs7G?0mWA$eD>yAKd|J)CrPc3JwAc{CZAr25a7IqewyG#E1 z+T~8G+(94R?47$p=JKO4)Sa7;&cZL(FU$B4ubkT6u$0pZHgE1KoJ^DY^m`!!>disi zcDgW+0#nj8j^`BF#2U4f@&nFQL_Bt?mM%Vi>JTeTCbyFkS}>c_v&JHN z$yBb+Qc${{g}!npek~8ZwvHugh!3yMP9`~>NfGG$ny*yR1#E7@dW87y)nE0lCviTo zXsfXswNH+&m&{>ATTttQzQ@^bE!xGCBO~qOCTNg{9$QQoSmH{H@jCwwC_Wenf>|xF zbr(R)mgt>kemJR0Wi3hf^XU)DvB(rHIL>nwTE}5e6#w1o-}lgLk$(}tK#AF3Npf+k z+t3r)%3@RCS7!x<6&ZJ!DP8d;5h$|o^IY5f+|BKq#3;W=a!r(b+`!>7kp4NXge};g zZuF#$VWS5-@nw0hfw_OZqnH1Cl+PuY&TyKH>epdJ2b^JQz0beqoF|>g1hay0AZoj_&ad##hrtd=Ga@^Zg}zV&e|p;u`Pln z)`a9?$2qdz2?Cck*qU5&oRzRwt}ebGF{9E#w;U6;bmSI({ijR3V?G)iWzY z+HuR7bVF6>Uh=kcqjvHcDe8#4!V|sZBtjjDeB7+aikD?22DeT$$W7tF+>YS#! zrg%XKWHYH1)rjAsyYr5ulq*`1-WNl8atz5HI3f)Rs7;x^!@ENVp9AP1O(XcHf6kx=_|Y|%ui1q7NZ#Y0qfQ6)M{tiEpHO=T9TN1CCx{EjxYg_gvl_&;u( zMoO+o4OvXKwco{njI1Lz!cHz{)L+OpD~aZ7#lwvfYi+jxQ>OAyVL@Vwi&IA{L2J;6 zU{ZiIYF7eZI+r7SbTrp=0E4A6>x(R80N}I~Wh8L=G_F$!9stA+MJ z0?^x6r3aqFxS78gLu{Xckq#}QtYd9tq#;YZX2=p&DJky|P!}f!H1(c)dyDWbab>|2 zsILL}g4N}2d37_hg!k@@n0=#JIi`N9qEw>UzXpyd{=d^V4TxDZ%yBeFCRU96yk766 z?Ial>ss}WyD?R?uE&lru^k9Orn)=Zl1=Z)XExKdK${j6XwRrPAv=SX%+*Hcz)2-2Z zVSPz;5E?5nub5j|F9xxTKn%7T6)`cE!iLcWYQ<3TG^*)hxeSj^h*&>&1^i37)UL;KU)r@%;(ei`?Zf{8a&#@m+!bRtsAQIRXjC<`1|MOe!jh8 z1|fFWuh}fcU2L|i;^x%9>#bi5)LR`Fx@5do!V)$;LuX$5(q0AU--W+(DR<*tS>2<= z(#uG&2(>Yr%bDGy>r3a5gfIN!Y0$=XQtxjC z#MlH#V;wF*15*9S(>qUQx?h?PO5g`?E8cA4cjRt)vDx!A?w6Q0Z(mmC=SqYwG%oKW zbHl1Wr?FGm)&AX957?bC&>cy|YZ+FW|wo)dC`}RI+J_Tqd}G%EjGlwH=jF-7b$niN%b=KKu?3RUn5aKQK;_+;{DAEoaM zYD?LH)NxfzJ`Ym|Ti4Q-o9ZUU19G>OaG~F=NuZW83WRuq(8{M-u&3`P{yB&IFL(Vv z2UB?G?iu`+LM6NgpY8Gd@UIAcbiTjWZGiNw+?SN}p%vzru|~7`dPtSM>h4()WYkOp z0x7bz9N(L$gDq&Jy+VD0=(nuPI6YnKa{K-P$?t~|EwJNQN{3jT?#Mp3eW$%zI%Z~O z01{kW6yW7eTH(}Jd5j?2hx+^Ofwwg5&ey{lVGH$+hkDawfRr}iQnyK0((sch2n{fk z%GJX8_$S#g+JUBmiFh5qbeu>FnVOmcu&v8lw94ZTOv;}T5FoD}{E;l^9|r_>7oE3@ zw&XAOn{S%{a|LGxB!nM(Z~lzPw_5c|lW^^llFuGbkYA5!0Q^wU%YX0mc!X%C{f2~O zX`)EuRN)1}do96lVBWg#W0{#_50lHb?}o_)&@u%9_63iyKS@ay`RXAcY54BhiJS1DPOz^&vaWQr%R5<#wLzW^ z+tQzt8;aLLy?p`q9KNX=buKe|Tgw*K;K z8^WIyTK}_J6=-{+tgMXfEm^pjtDD<)5jny)Zo``g3#12YYuh?Z?@n`B_CjG$Vp#`5 zo?n2!0s3>6il~QBHs>W3{l>k}g%@<_Ky;OG4q3b=%&Yo6qmSKJhiD~jRnm3fH*U-Rk$I8@Wvn^&)>c%Jvuyh} z%l$|XEG~O6{2dS%khw%vb|nuTo%Li~I!&zq$lmz*EjVw}1ER}YDV8IdNJeRh-+VG3 zJa+U{RGDC0j5fl!Sw_}MoI{@3D(vU=C^KCrFJX5^mhhJUy+dHp2T;z7JR8NRX&7h- zT}lFGD{qoA;q7Vs>jL(FChlC1K{?R1dS~~vr2x!ZK+oC?kn+A(d4eD-if8m~lG5_p z(m0@D#g5TyHrgA+&qJocSSmdjGTv-jES#v!BGq3d6&v`3l=OoQsBeNgt6kuS_Y<^3 z8W&}vx6g3+iYu_{7Vo}E(jzAa3MiC=0ff^1`Vs)cxVgAMniFDy&BFDU{HtK_9KWf^62A$s>qPi^Y zdWU0q3aKu8wLgD;B#r1Xh>3~mWO$D7>|M3Th z%Mx#6oU&-oUxtd

}l(fN-bBh-6Cl%5II@1*CPf52fM*{s|)xCj~hOw0dBZO-qpyE%-r1EnwqI8B?KH< z`MXPRhZdn12}T(LbQA;C=$Z}$9H3cPi8i{d0FWLA3ew|d`YI0$nh;f9GPG=4@2NYm z=!|W7FIgJG_*#t(4UXHH{tsPptdx=Vosw7;@JzQwyyWF||wku5ng9eZUns}XeR=DGW2Ra!bv z1RHfY6sly#3Z*zNjmZUaIbj|F(F>|vNM3)J57JL7sja6H!M~@mPc~86YB!& zE>xR$Mxg|Mk+y)wTe*j)y`KZs>=hFA&d<#$Efb8Mi**4>0h`wYEzLmr$W$z+T-8Tr zjyWqo8c`TxjDOFU&qGi}Gqoo=eF81}N^dvby9njcRt~Y$tXM6&tE&4))gBVhSU$vu zbzSY>+^3;F@L$wAm-2j`fNZfGZizGTpQ6j*J| zqN1S*cwBv*9tJv4YBDD9zJJfR()e9NiSvGPVL^i|3a^pK$L?{z{F;!HlLN>UJ8{Ep z0aLZhApy_=C}a4)y85_fk3+CIfIZ+1oRpZblG)dne2J7mK2kwJ2!K4Q9M~E060VQP z2$X!lHlzYy)jJy;jQ=b%=9&f&U+zB-dDUh$eV?(0~IK#jV;scD7e zsXKXD-Mb?!Q@qQ!woZ8%*KK@Qm_J@TT}kzwega1#rLVtM)_UInNC*Hgc{%+)7|dld zlnBTS3~bpMCOn$dNb#a+K&TDqC(Qv^RbY<;()xWr-acLx6BAP|gHSp31HS3Lr$46l zqS$_R3`g0oO2LkQeP@erg#VW}%mj*F3Pw3Yd1{b6w}^jlm|te8;~bCTZj)FR6m(Tw zSI2%v#9yA#_&q2n$_+hj%-#8*bS~7n$CWpEUlJ9L83R0$M#ycq2}sEdkN}G!RY>uiwn9`Z!h?8dk%oE&(Mg>x)$0#a%94Q z7a@Z?t0^ZZpqA~(bapmz&oy=Dt>Y6;jC-+oeugQc1H)@x(UYv zbqBl!fVV;|rgpnx?$K{>aJgh09FA>1Pp~K5le)UPfDZqldi%WlA|NH}(+k8nNa!S5 zEX@gk&d~@U7hq^$u-8hA_^>;~G)*Hqbcw?>gYJL1oM&kh}>d6zz z?CQ9jOd3_>{t4A2EI*rlEk64HWV7JX{xab_AynJwP18d|J?3wHtm{r);y}K)*}RXI z!)EPkMKpGY1}ATHv{cW{%{|SZ5*=Lu@I!BkAJeWRi$IF$lc<;~ZQKJ4^srY+VunXB z^vW&kYTbX|t-@@)A!ltL#E=}B%zZeMniB<9@wypsqjN_hREbK{&c~44n|?S<8g;=7 zx@-7FTqJMCJAy~7Msie-)2Xuq1^Iajw({QzWkxFNPvmqFt$6k|j$W<8yTG}?10uPA zga`){fW{|(2*sI}=La6=-AN}pK;#gC z^71{?l0k;k(*W+rY`s*sv$GS5%f$LeMI?})e4-Q#q)^bzCy9kj(8CpMk8 zhZe0kh%RPY(b3V?pcd_OR^Xj3NznLndfSDRXTwXM}(fqm`!D&Kxq`b4ZY* zBMW7L7#xe~nQ}R4VV}Sr3Oc%d-$5*FY#OW=^;$aUmw#^HBV|bcO?>2e;kJ~g?fMF- zpziA=bFRC^{(DESdGaF!^zhK|@L_}mjL`17Ld-vxkOWnla~sX%ZD?bvc-6iwu%q%k z6a~JZdv+G;i{By|h*lRv^yf@IER>Ajj3gePHw&hCHEf28n)ir;`7@GIP`GrUfXQ(Z~}I-A0kF4|^ z{k~GvMClGtBUO8ARn?!!@V~Ptq+$0w8qgCnDop!yqOd%(wy!X;M9-bpE`1 z8#iy8FV{V123!vi=t%xAJ`SnmD+i*p+rjLGc~#Wwjx<9$Ria^{TV?xS|62M!lXcv< zO1BTV2|cDdKV)GJ3I{08=H+kB54nGRQ#?d3w849#la}Nvp;vtC2C*`9)-~lVN4w`V zj@(gnV!ZYEkSel{!|EXvlvl-k+I5708_AGHr^e{T)r2f6_N(p+ns;79A=AWjdAvsU zOFQV+W*S_JBB;sEA?anqTPTw<^C7sQ`Qc+|!0GJYOuUcza|&C{gnOmQcTPYI!h>eR z@|_g<;)wkst+u&$mw}cFusht3 znV@1|WYpM9`|>@2dji!|VW0m_>=<+3?F~OnE3jQMu7={BpxeN)9pew+|O9LlM=uf#`*5tN)+~sb;hMJ%<>vp zOLeP-VXVVuu;VV5B72xDCSiP#q`T+h-g9m6X*T@V&cBtgKH?H%IG}#XdPh2r^N4lf z=CkV3w-;^a>iFt>o!mTJ%6YmCU$;!eTU5-C=L@a$>>JOdwy%a33(S4XAs5yfyR@#( zHqH^O$G?72BYN-SGy`3PE|0g%aDvlGSNi>j-`6^J2kcv(OEqiQiHShg;uU)`}V7Ji~U()+M{g{7NSXyjhXveB(_K?SG&B-?a7brK*ancc9y986SxnMpoNwJLQlyi~fb79$ia!YuMV)o@yu>w<_vD)@%1yt2adv*5 z{qtv`0ONDNKb@r_AJVhpcP^dflfP6iBqt!|oARWU9+1vy`$!3IAjT-qyuWng2ll4e z3PU+|Im}Nrjrr+6oNs!Q>}!@4LOZ^hyhgqrZ}n!kS&kEFZevaz8W|VwFb_FLIkQe( zK9!lePxD++e{bQ_m_I^i1NS6-RC;s<#ZTZ+qegeM+;E{z?aZGhL5_qEh>l#>h`J?o zW%tcGe`J0x43?yiQ##@^efJt!w3{|Z;|QAVA9oZ$1~delHdItr)9XqX}n zmRAO6{N9pH(6MfiJKw>;3v7WccT?k@yMX;`AZ-B>F2ddO)fWj1h+!UE^Ne!NpfMlQ zbhL--PJg6H$U}X#{9?Zsj@Y|b7@|tDw>Vd4up3*>^!By%cI=GdNUdg$g88!)&xoPI`3#~WQWqJzcf=(Z zyU>B}EhWWgOoy7)+dIH#8`qshK%S%wyt#RCdp@eVyaM>ks7hjQK^AAwY1y_=8>j! zh8wRk({CqhYn1?mCWY74){`Gn1Y(}m1066==RJGvH`8*x_%?Q5J;XoJ4s5*~-xT_^ zJ$?tqx;)MOYjiW^q?VS_)IN9|I5J>F{qh8*%ld!Eq?&lS!YM}`!h@qd^uA_)RlRok zcnS`_N#W~75tvFYwPJOZIBP;NWM5VB0u{Wr@kLl{Vy4}TOx&BFvLdl7JPpsU9?%5ow`Z`qXMvr+9N!%Xsh4-B@_LZ3#61|S-akF-%4}$L>OBY? zW8-`xoZiqk$401v{<0l0P>8*DC8 z?!A;I-kkrSufCs`b@=iR&|S!~uIwT-COa=s22*B4F4fCp*PZJAx8v#A*NQ0Sh4rOP zO&V9}<~B(_zk?V^#u+K@c8JBQMbG<7Pp^xpNL31=TQ`#h#k-C>ZOggnsZc6(qSn;x zYIClwoR$daPiLr9HnjWB-gq+Myb3eW`Siyo70GfPM1hn0J3rUJS1(oVr7A}0slz+d zWDrc0MI{a9Y~!}QdaTQ+D!;clP`EnK68F%PpY{xLOkGx1`>!m?G3d+0)-Z+OP05w% z`C9%j!c%o)`R9ALQ$^PWPzL(^7;>%>sVQdy+-;3oV<7EL^%@p&eZkg$mNAps)Z{&! zbC-k8np$$~^TkH)aR1Xi4nvbflc475@VYE+?LAR#wF67A#oXX=^L%qb);{QZ%CiV< zv)aqrL(<><33EJgUW*{NbNfK^punOiU^`NITCeU=4oPny|7dV}>uUQd-%N){`Lq=i zlaiw}6@E_b(dyQCl13?p@{qplO)+_9iGN^5I)T4ErVeA#`;ss2qiW zoiRTd{7bHchqv0T>-AsXAC5#3;l+QPzUq%@Zj9>#X%Y>)iCxX2b*-Lgba zGiJg0i@jU_F@%3{!AaspJyk8~;6zMyiDOzTj@ut}3%H>0ZLHQp{mu8awNpQxBv0=3 zt~jmqb7bVx)x8EFM*MqApS}E2QuWr-Tv!`hs`g22(uU@YtJnV;9yG&w>!l&+&Dk{H zV2G6#!ot%AjhWud-RCFb0NeC)!O-9!0Ei3uy@|cg{`E^NO98JI&nIMnAWtMqfi|v( z;aTR-iXH%G1ajcEp-=Mji){;)Xv4(~mki)&f!zlHF(2c2yf;53_|6jCobY$lRjc_3 z5M4G_-^;ywjoQy>3wlza|9+6{xQQ$<^# zJqLDERdA6_qjhuV-Qfpp+k zTW8XVJZ%ADLT8Em1_9OA#jk_#09cNxsyv?1o;==g@VfjMr#P(JDN%0zU6nD&Aqe0rLWQC zGu5kX#L(&EuW>W2fKA#6Y_+}mJC;JtW^e$oz+JJX_iVI*|E|3$0Up46qMEZ!_l0V0 zR_?dOCL$P21DbIesmKI)`BSJyIO;UPa!vu2wf8;kX<%%|Z(vV|^pDga>UUkLw@1Ca z&csdQHlOVyf+QVnCmYN@LD2(l*nL|6{hx`U&zOVLXzvYvs_3OaNwD6Oy&A-C{N27x zq%Mr%O(P{QCe~0qM3f%IWnWQR15O?7A6oms;HS}l%ujZSau#zopERGNI&xm%u1Ye5 zj>q3(e@XbynB_Zeldk&vEor5-NiOlFD0{KRtJnn~#(nO_#5k?TPa^PjbZ{!Zm$fe? zDJ|Z%>ze0VTbUXC&_knc%7q&d^u0sTqV^pRYU0{d$ywQc4GiH{N0r+_+Q%nBM`_XgSI9fr$ zhjO6L2~K*eyJ|a-lFRx0<)6&~L@IJVfXR18NoK)&ChSl_E!; zG{9()v4Y4SfVaG8!X#ty;#_{;&<=6mX|$Oq)1i(xBQ2^srmcuxgYSI%yMUDz{dGrd zongw%QBY&W)_z$E{f$8#l7!g1Qd=e2boXNa}*wX#au}4OP--&RUaZTvM)JHU@v!{AE7<2)*Aks z0IuHD|KuwY2o!u~!QECc$N7amENrP{M?8Mxy*Xy|FRsNV{0*{+OCHaQtuKM;XXSwZqyzCefwVBoXo58?Hl zGW;D>tyi~f#A)5=jiqy)i*ebv9Zm4WQSESL!JsAxDj}!6nhZ}K9dlPw#!Dut4lxI)ns zeAJM%jx5x@$2SgU=OGb>UTqxoop(GcfkPJ7a%RrNBoqJM0`!DU=Rm(GESyhQ&3N+P z)>8e*K6nf6Bc&U~wb$tsNQT~;UDwOG^L{?^Qxr3O>Nb>a z(^=~#%SGtC@DV*mR3`POldv?!$+Ow0_Z*eRm)##;{I_rA(>>a!p`dSIY$JRgFON*) zON6&ZR_;#arWaf_jN0x6zbyx}vZ%c_A}tsT@q5!n8x{!EG#_aiF?$h)5=hI<@Ntf3 zr#0I)>F?d{R>JL-f_v{}wyc_)iH05`|Md^Z9k?y?Ri)^m+I>1fWtq(79_R~J&(oO% zk*tnrIC|Yzdj;zrN9sO|7f~%D!s_RKcNeHKYRNHS7`0@tBu}igeWVgODxcrGU2Rw#`dOc zbF$z9zgp^BZy5GkMP z`AJXK>^>m83GgP?U0W_l9|;5i#8dGI@dVESAafw8S7KtK8YHN;7HXQ|$!WVTMy~Qy zA=h%W|C|q!s`uG}M|)QgU_6M;A9WnOX(RM~^W40;Av3dW&R>YzUqt`lH5@9?P!q8D z0DWT&NR>QnCFrWSod62(in$+yLqd|-Od~!O!v8N==nDxkP$m4A3!vxo*e8k|$YJ$) zpqIW{4foNVOgJA4fHB5*r))kA(Z+7GUA}JOrNtUNKNR^2lO) z(#8C7SMVg*r+}uh6c4Z{?LR++n(M6Ym$i}zc(~kQAtWv~(<{VVW0@*_S z7R+I}#UemeF$no4Zs5-T|R( z|39uShijijux&q(&3$C20nw9B^4#28k_4%kE7f{Fi;|CCll$9Olf6CVJx;6$< zjfSeg^Nmse@v+-Pqxy~50P+Nnqq(gu3x>LehQYQgaF?6!|1+#I$+Ugu;wd$AXrWq0 z0j9m~UG!0St(CME45nTeH;SC)&9wuwucW zGStlwEQ+*O4**X>%a*)*y(j7zXFgV!{p*gPkV2s7quOh8<>*|Yg2QaPl4p3gi5m3- zxA}I1396$GT}&_4MYA`QqP{uK5F|TGg|!GeYb3@5RjEJR@jz4VyzgMTeGd(BJ&svB zLkqH}PIm&Nt26J*1phueoS~T*zLKlgag-AO{`G}S_U&elHaJYXmxhUN`n%F83{4y0 zp9-*0)+kJQ4aF)@VPHh{(Krmu$wmJ}0l&V?(Hz$XwcPxwnXDVP`qkq`{%=U0bELzS zHQRCKDy%Lw23#u{b6r(Il?}g zh3USafAKH$n#vZ(T=idCtPOoCKGT!~yThCaPKLQ@zRK{TQx`QFFSG#X-qckKTGdCF zQBB~!J5a0O5w`%<%g~{aeeSW=A|>Bv`I66NdZqmDpWPGBu78$3Ix@5tTy5`=AKQ#i zCxL`aPHkUtbh1fl1zjQ>)zYjxJf0sdi|?1rnR@;?BM+3Z^hxfQ{7U7Xg8pi(EinUW zDNH*{VX47!@GCc8+2n5eg={#6-qz_|U2<&ZmIY@2#a%1-Pb>zoDsaTO+~ zV(;RU(t0pgm8Ii^-o?n!1=^^P#jyTTDo}#QVBtK}(K_*LBo7?XX-Oh~547T6YLu5y zrHN8_Rc|uLktci(<8YrF2V#C?rq)}#>E4p!V!$|gdEH~SQ00FB9AO|{Z+roVF!sLP z2&h;Bd}M&YhN|#LE?#(KwE@sMJHha`gl{}10B0?nvMoj~kKAC+UA0oR&6YkYb2>Kd zMGTKK9(zG&CJbjnYq+ZDMt8a_aS@Uli<%Qxl=7`=Tr23`VD?Oy!Seyx?Vu{2 z5|uX%Or0n|iPal%GHlga#;6~TW?L)8IEE&H^Q>Yb?K!f74^?X)lcRC*RE|h^?!$LG zZ;Mv2TBKqq0Si&mP_39JXG=sCC7FI})s9I^$Zeyq-)@_@0Qd3vqqF=7E%1$6RBMo} zzaQ!NIZm|Y$1SwM&an-n0nA`zSb0UcK|wxqt3v)oSm|#e;Vw0GP374rM(UrlR&{36 z?jw5W5t82S`vAwa`D1!kr`whKwyuB^^I4f|_s5aTDG9Nsd63j08u^8r zO;H$=x#UWXCKBN{;*-5W!r8Ry=NTs-QGiyVEOvA=g1h9ltsUaZ2Q;LYgv27E6M&Zb$MCHC9QKWF!qX|XXL;TAkaOkZSSU}@Ug+U7je0d(};%>?mJJcg&_`iNOk z+Vv%)@?q~?*lAfx6zHmI4WaHgpQ7g9wl$@NDy z_rina5%S)r`*bBt4`*k+jdq9z2RB|fgDJ#0&l0dhyBG+5$pk}e$;EZq zNRep*&etZ}Y4A^(%D=4d9FZ>3RLO%%b>csAY$1olvcaH4f`lnQQe$+Bjg2`YNLqy7 zHV2{}tiLU@rubHtZhzf%FB%0ddzZ6~L+g;bG&5e}x)0I!ym7QyR&oISyeaY?O|0hD zUV&|RuN=`Anz%dIw0fpPQ_jw3$+cy8^{X*-Wv^&OM7!$4Z}#%Qcc{lB{s-Emtq_Y; z@i(EFs@``7l8W0>vn5(#H+zLpL9L8maZg*WhQXanMV28zRp#-sF<4rxWdnyXb%(C> zjP-;Xa7B+gLmmm_*ajSB%7yIk>XMt$FB-{J7)$HyU!P^m^%U?pamKFtM<@23hwL3e z+P;2Hifx3(ygAXRE@UP@9V)f>gz-sH@ZoDdF2ojC4`Z3k#gep`3FwI1)wIUK8XZYh zMW&G%ZcZbqEMC|V`FOb#!g$DTB^pr)uOD=!J9ann^1M!Mb2=#mjyoDi<1It_C!Q<0 zxe0Y)f%>Tjh9w)mTJ{Fu-08SMbMltxNcEBZ6K0rh+^j6qb>0}@p_C_1b#wP8*he#E z2>fmw3?*&m?dtuiDw}giU3#cpb5*v{U)We?{np^|v{T^66wPvZ>bx78i+6qEDw~uWRi`zAN-6nq~rT~TxAvsMNP$F7`{dwWEBxs5-TpWmz-WMS0KW~ z$bJ)V>`!a@BUp0E_9lZ=kb=qEM-`A{SHC;Ykcc)^pi)!WZf6?TuLNyy^Ob9J>QQS|P}RFvwzFSlRrG31haxZf&^05xM6DB24)Wd`Z8 zfh!%6UqsgfR5FfVrc~a}nk-*J>?UJINEd6oBplAhb26ywK9E8jOREw10*vtm?plMo zOf|1nhn++q1wk$hEU2Q-{H|h(HKXR7F9;1sn_F4e)(ef@c}6@Jst?XL*wE>9F%-ql zI59zZM~a=V5M0!Pp6O8{rfHOTuOhcC=tAHAwtjM7qlD*p)-=&(f!mwPS7ww_ZGCw1 zmAp)?K(Q?o-{z(52BPEou06M-Di@rOuw&C`lSG+9etDq9LD=<%b^g;C#f`v1ddnse zpjkHeoIj>Rtz3H3X=W%sw?*WSV7827YE#6^IJR#Ym&ty5piz0x(X4P&=tX_WSHtdM}VcgrJ^1d+I5jE}paveSW5c5Ev&Ljx0+)=Tt z#*@nXY}qthL%Lw~K-Zg?DZe?Ohr?!4n${>iq^^)nK@qashe$m8Q&zh1IkE^9i`nbOK zM(vDK#Q@D?FZY?;jurcRuk3HWx%qrITlSz2wH~`L)e@)M+tH|^YYlOu`6;@((JS0u z2d8zmQET=3)01mL?iUr7Glhd=U`q6hN)vPaWeLhJm@7Qw_d*U`t{-LKf9}zaq1%kqNANx(C%Z=@i9aFDa!B5 z?r*LYoguf7#cf5dci(X7la^kqu5xjT-`JMbJT>1t_jzJYX@&jTvXYHKkc~l9q6K}4 zn^|@N(Y4~!=}ps;hp#0*8GsqxgyYIclJL1g=(Ljw=HLyN-45I*6+w45W&5L zI|`PQJrye;aNfLtrF{O0mO6CkdhB;b)ve~dGjrs2yQ_$S9D}g11nOllN{eVJtTe-) znf@}FbG4vd%ZgH_Unb87ThUE0P-O(o>W{oJRq5lYyUo}ktJq~X?s=nnaAbiD&o_mB z8E&QPbp!T?H`qcOL9^7K2v*L}ZCP4Yyv&@86gle85NJ$Rcw=wAGN2OYMb5xtcX+GG z&BPbiXe%2wF*CWi$N=6hnL>fZQ!QiMo!Ejx8at@ep;1AL-vj4Jm+X&>!pPX04E%6I zvdNi%c`el+u72R}c;?v8rFhY$=IX|45c{$7;yviO=fKJ}DMRE4rljT8&gM}3rU|8( zwQNw!4<2X{7{UD7oX0GT#65!sEA_$RsKaU&3%;|Tn5bmBp34B8qC5Tl!v3(l?(v!j zC@TQ69naLFX{&CvwXlr|1}zEeF?dGUp~f@iaj0eDZ8%X`u2J^6bXbU54J2S#0}L688^oc zVz;rJ!q|#e%cw`VE7Q9-xf;5)FGdV^_D99?#;LtCH-969I=@_x3HTl@=A)BwLSawlfJKg?XJvIcp_?8#gDtn4|aDc7LydT7M znP|wVOG`4KYkPwQNM5>D%RqT%B_-lw`LE&kx622l>-|HIlg0dVW=%W! zXqgz(w)9=G=Jiyr$R~%*rxg1*1lyg&?tNCs$xU4w4~+9k9hyiH`0B%55WQ^@FyULQ zhg$Ow4gwoT0|H9kksckk+vfN8ZrQ>7fvmQ&8C-EehF zn34F}`E=wxQ=p$rd9BO3$HFsZcVAw^t>uD1=i#lB0$W^r7jdkyYQq?U1th}_?4^~W zZCQTSo(j)eG?gffJo3_y$S)YZ&qcNi`igi^T7Q?y9bEFL)zXewo$SYCn=lIER<18z z2P$TV;sqX+eRO{TN?iDE|5gDV3tR&fk5!s+LzZzxrEOkz%-0Y=AcJe@iO+#*A+=tw zNze|M!WbwVf(fwwYX1IK{Pp|OwBU9p07vMn2cM)5tGzzj+&-Iqy%z(gX*; zSN-fYqDCDOQI~17EZW&r#Q(B5Tpe;&xkVXH%4R+A;KdiaJ!aU0OZxJ%OE3G`Zg~ra z$6htz3tvZ1bED(Nz_hTb!vfpv1|+1KiYXicsq?3G`{j_Nn_ta@@?bW(JJ7X@ood1a ziENZ-${HJpczwXU^mraKkH-6{fZ(Wg*MpAdi{0jNwuwlf$>r~RG`zM~!m82O-WSJA zt&0iWY-Gpj{N^NrOee`d!{`FNb+4SO20E%oQ95PHx-nj`RmQ>Fk_eEXzxOgoadp>V z^%J1u52X^|Wedc`s}Eg_Nkg8^$um0X;&F4;n0`QuD7OTaG{^mLr(Boz*Dn3x%ES7} zbey(0)KE4LD_u_;K@I$*j2|?f9I~EWip3_^oZI|5f-zvhpTs5Kec^llxj#JaSfzV6 z@-%*m96zr{2A`JF`9N5eBP`_Gw{9o28PqS$yftNC86&Hm9rS1v^VwJTq-H#oeyZqe zr#T%4apP+*m6+~sDFUy9QE`YP`pe68>hk1E#F7e0O~*4{s|9aDl{$7$Pc5V+l$JHu zCL2WZ+E}(OQ*q*Y=DwnDqPrt0mv|RP;BR{^MrK4L78(6{;u1M~+_fZk-;P_QDyAL^ z6lS#vCmD66XKHP1ZRC5do%U5OaoWdr@D6*MlX28CkWhuhJhmb86c=ll z4kd6|wqWCZABw_3hX4@Z<)C3B>{2l z=7Zx0_)z{PEpBO-wmq`os}^4})XrDsagYd9!>^!|cl!dgp(<;@2wOLPWo^jsRpdmC zQozH;C^F#h5NS4_;|&#E(Y6IyZ6eBknmG&&RK0JwUuw9vpR#Xt&3YLy&P5-29j>?R zj&@YHkyk0T2yP)5|7itV<$UtwiLaFgk0u$au$617XD;KV3az{!j~BFTkT%>^d4(1u zXmogBZCA&~^cUk1x~mMD*<9GghK?0V@hHNxd^b`T12vR_@JgG$%u_=*j54Uu2-d^|hqdMj!f->*FV$FTI-5Z3!D4!b!DrUbdEzS2j@sYa%$8&imuH)jF}6&#eESYm>nTPB^L*O> z{Ua9@Khdxyex^jEPvC67l||YBPrT0l0*bwSm*?0v6j_b@gk+CM_w&zJL2>x35F;i1(UqDo1w3N>JOh4F(dIgL=P+-)u(^gLo1O2_t!tZ+`IshC z`>sA8ftr^)qll4r;Xt;g1?V3gQB4RH*J|;CtOC79xiAg&_la4}CSqCMj?`7s;`wK) zP+!8Is@oz#yYRI{@>KSNk2n>?X=7H``I$+ogfKq=qr}1DK1mtYih=nAPn{rjfu z;c;?Pc_0#Xh>reY%X5~jgY{}<>n)>@ceLYiuPKQdq<0s-;^=DPVmlDm=Ho)`1rVsT zVEBETU-*`WH^nVZhJkrnSV+Y=UHHrqU9Z&GMHN>Cn-D{pdTHMq`r_reN3l&3eWv{n zY;rSe9r>OdrZktzM=Vc`~4S6>%$PN2YKBiK672W-y)r)y;*ntSlW$ zlBpa34S)%VHv3*}jPbPjliyt?&fS`O%fyvydINw?D`)Cp5u^(ayqsH&O`czt%CD<6 zX~UWk6@Pj{%o`~}uvAjXuqE-Kt03s=LJ?#30@&W)qH`SVO3joAWz>N4TC+p@8ag+> zh<5bKlCF`zu0UgY*l6V=%AF!Xbji0bZVs)C#dz^Kn(h#%nx=Mg&jiWjomS|H0c9E*>_1-lGr*KgqdF&sQQDyt3I>{M%e2nz1oaQBjJ&C@Oy2+Fy!DISMcr7hO;W6@#mKVgf)najo~ZXU8`*N`_5`34x01p$}kF<%F?)C z-y*%=C+ZhZz-9Ux;>AM-tEY3om*s@~$uEU#kxqrc07ZXMpq{QgP` zq&JuAjkq)3Zgf~mIU5e5dy6YK_xmU~=wBvJcEcNzF4QW@vJJ@~NQCX`$u9f~4~>^F z!f&wCO0l)vtw5M}NxgA@p-f1{ zl^np>A>{Q;Ob9w7^VW5Gk0L=^jW76@&Qq0>~F6H+T%zJce2Pl6yE-ikVtjE%0-+34DHd@`6!mwowV|JP`a}iY2BMy5qzM6Ugf!uO#nvWgH*C-RN(1 zv_8(TU%A-92xsHQr9|Kipj{@iw~O+ zz~W>~7Dtn(qy~o$RC(k=edwqo>eil-(U?E?<9<^_9eMy=I)`=A=M=qW86k4ze`rcQ zZ`M8Tv_FP=wt$Yov0Z^B;ZllLjW16gu;stP7{D(+IqIK-S_E|8hBZ?Tqn=&4-dcdq zH;Y#}fo3*dcM^^EbGRC$)CPgD_Ljvi*H(GW+Yg`TqQTkA{3*2E-1FUN%~TQ4Tb|pn zlEY;b|Gkh2r#2szlvZhZIb7-9CwCgvQe;^`{blT9Uw{?1R#hr5zB}l?^)*~oVa0x& zi7aMx)^#Ua-pkzW-f#Jgs^7EDucQ#ryWCy3qUa15%orwDkIaL;R0IB9dj8`4p>&mzWZXH&_%%}*r>5FG%?pI;NGFZ%I1(#aU0jR5b$NO z6yYxAgG3Hxf;7Yn;nsVaeaz;4?fUI(QO5+We>Uh>QYXLl!$R%x?NqTXl}>0~>{WN6 zB#6RL(@qu2Mb@@)|NGcX<02g#Kis$xJz~1K=n8SM+_sl@`lxmm_v3(&`!>&AH8Qpa z;aSx+Wa2~UgudQ`I~b}ffTReRDc!UIF%hg=0QGKP7ODmd1Wy_;=kY6dSEpU>P; zDYBLtnLO9b*6eNJM~Xzl-&aFO$Rq?lX6Q3DE=xbmI1*` z0kW`m-hfh-#@K0aGex~Oe1*sHHmL(8Px-p<}; ze`C?KFQ~1>im1esjm8=ft}Aif)Y~kb`(BA#Q5GLtP!ln{wBT^|5|4+cuk~`73gk`6 zbx3?OlPs~3osCW;&zF&;t5!(cvrXLG07FM6;GSy?CZivyqdI48PPIAo&sC1Y)?DVc zp)R@SkQudVM15ga<8InwGHay=(yb4xECt7BEMMq%b<-A|YCnH_(8zb!QFrxJlQo*N zcwxnJ_dDmiF_72L@On;WZP%tI|iU z)2XMOotq(On7vsQ@lvrjVs?bvBA|!Y|mpU@wL-sP_?B@&R68Ztj=MVcK`9vjBZFPbCl5}Qj|IVk+>h;Wudv~{SXbeiH`|N%H zKuN=h&VLHNZasG-E6vPW3jyxiRwm)o>uE?+??49W+r3zdHc#o1&!vgbGumQ&&Tm6pEXKd z&O2DG+!<^Ryga3dhW*%O3vgp5e*$Or+yxDI9G)z%2f0>~vh;i$k>~TCXYo}JgkYE1 z4Mh;)#n`Q(a`Gz?RsqbYR@)|RPe>l^9`|T@1}=3o==SK!LVN#mMaXJ}-y~jjtT$oH z8fCpjb0HKT8BR#F?WF}AoI{@-1+9no?lStdJ}f?F`qDN-M;2zcJ@+ZQ=ho|z#-5)Y zK&|!y08)<+iF&D+|1w9rCZkoPWPZXeW7Yc9+%nKkXu-Rt?KA1&6QD!WBi@zC7T~m^ zQxi?`P>Q?~UCvL|BCWW@>txnZ#t#RFj_5V<1a1t!e)_$lPnuDGBr5OMJVcxRr51`( z=@|Z=@2E`@1-t;o2y@Eg37n8k?92uwq6eP)8$sFwq4}Nh|54kO|3jg6aq4ENkS z8bS&!4B2bSGGv61k&%nS7*m$fSSy3#B132xOADdMI>vhK#=g$j24xV&GWKm|-qGiN z-rwIJ-hbeE&gY!xIp=$x=X=hH4n|{S;j@V!-{Hpo@@YM2OhfTi!J#kgKeGO`takpzo&TwMly+2W;-wknmvy(z59YJrIXfvn3C67p|lb{*L)S#9UPwkKjyd=!{{ z@+>p_$R=X$X>Al|NcxJy0mc8t(#3C~w-3QIo?pFkeV-$)d9Be3w5#j2%sU+Y+KB3q zW7LM4;*O#2ZhI<4Jx%c{yTlw;gczTFLqt|Fv*QpRVJARe=CsBgisbfM@XJjM2gPWK zderx|Pd%fDs;q+n9=$XPU2PoO(hloEUmYllhCkV?sJ5t)U0%uE@t!DOM1(oJk{8~G zT~6veM}s*bGGy7Uxk-o;LL<(+#o@>XJgz7E+Y0cCN#NWLC^XV9NX(vK4sx~ zcKlE|U7DgHs}a(rv@x5I<7Eg2b#^L+sy%|OMp#cSlhZEuW8QkClr7K)?}~krXFn~? zouo;ug}2~`OJ?Kwl|tH9pyT-xq30ZYEAESeuxD|l@MTBK)VScCd6I8`XrH`K-q)>e zajq9^clt|SuA31FWp!CKoP();oqo?Yy4DSCE?hW`qieRUe}6ZAyO;^(17I}f;t6U`@<3j6wgHVyX+cXOk3;FuW%$v zZmtXyugXW3NJ$d5SVggHKao<(=`YM{3?$C+^~=PNA&QZ^f!>Mw7ZISOu0=P8DNZ|K zZwDZe1iCy<0A_tb>6ll^F?Z&y2|@}NC=usY6T*6O^Kw0_EY7k!2|14e!y7>i-+04l z?$`7k8g@8X4SO})gSC67^hAB&`~tGe8V+*P+rXoB+{O9`jIOH^WkRl3^{r=v17aZt`*;HLZOr90|iLI^VoA#TVW8>CNQu}-xd9= zrT;!4fm|26Re(;q#pk%7?)Y#|Yp|S`Ym&RE^T1&ndrJNJb8EiWNkgl_sXc;MgQH`V zl4hX88@bBk>H^+HC@e~8IxzndM%rPFwZ!=_O>54|dYFhg;SNLPLsQ~j(R(YJs$(sR zRf?^0jT147VY1d0vs-- z*a2O$+M=pCWgn&$tQcUp{Ub8tW&oh1UPt&^6OJC3y|1hoMmeek;se5$=Vb$y$^K}! z9B2y62!%pz1z)N}RnscuFaxcHjI)XC(|tq zHMxy4C~e6wyvu&!S@L;}Gm<8t2W|OD${lCnc2INaS+}!9E>9v03C_uCYNPCy(;ae4+h6y>*(WXVy8p~XoB%Sz4~W7 zdBFFqXzjUu7e=qId>O#uZl|*tks~wk~N83AV4fO@m%CR?a?w-Bk<$3 zEimcvwVoM?2)iPS{x%M@co7~Z%_)`mmE89%&`2osUa=oWD(B)G{#fM_BS^#D2K;jQ z7wRvGp271Njh)uQLvZdPY$1H-XV@``>QW7&HIVKO_8cT2P4xNJ-s8O2~N{;_!|igsf_^}`%BvP}V>t&}uC zE??ehBci_VNTWr+&vkay+qohN*B-HVr4-ob6cpVDRL1zbp!6ywsdP=|puDzt=DSAaQ%?RWm#WtdVCoL0`(87xJp!je z=AQy2x({=%9Oytt6A2K}yHo@QmxA@`lm-T0aCR=PoQ0Bn>AaFKhF_vm(?pP()1#Cb zL=batZ_giHvZS*!6)%gU_l4Aj`-L<2#n3s0z2+jI%Yx}DK4bl&@-<~j1m*Y%uTcfK zZcu)4sIvlO`?-aXnOVB-VV-pcW1TL(3;yekoQ4m+Vkuq$fs8TzG?O*G@OVcJTQdLm zmoO(y;MV%w9Hlxm;>j`zE^1;Qu55pUoGDn%p&r*BVAfx$KUH)aw+!B^=P+cl%=5QW zlM+)S6sg{mYZ|xhMopl;y|rD9Qc-%|g&82x=n-8_WJ4x3isuHk?{(jg!kk-6k)Nzw z@I#CZy?R0WuU|&zeTmZ;Gr>Dogf`%hPe9Qy#9ZhP9QUtl3-mV{bN$C0~Z%)hF<8HY!JuuyWm}XhLUAY6D?`1~B z2e&GHEphAnaJ&2E?L(Rs4}$KP`Yw^|%?+`|kzs}U-En5TXf3{t`Z6Ik*+f!WY*(_{ z&^LqRGUt}Bwtg2Kyyqn~{d_8MqCDcnY@x*{ldg^{;sz8l{&2Y9rZ1s#O|NR-tA0FL zRHEo;XEb%hS4#7*w@KzF-Mxm2kX^~t@Vn{P210+s>Hj7S<&FZJY=G;9;BNvuhv~!p mZ=L7=l2iO!F#3OAgeAonXK(Z7R7%%@%wT=UoqW)vSN{QrJ#-xa literal 0 HcmV?d00001 diff --git a/_freeze/Comp/r-sas_mmrm/figure-html/review-treatment-bcva-1.png b/_freeze/Comp/r-sas_mmrm/figure-html/review-treatment-bcva-1.png new file mode 100644 index 0000000000000000000000000000000000000000..929d560525ad3d6b0fc25608157a0701e1165895 GIT binary patch literal 60506 zcmeFZXIPWV+b@c`kYxh_0R;g?0RaIK0qGVHrAn`XCA~*FBoOR0AxbgyNbjMS5Fkhq z>C#I`LT>>=XrYAU1lND>_rtl)hxeSb_otof%Joe0Jaa!YbI;7(e)mMY)YZ6jp8Grl z1H+{k&(#bV7#JThFq~=r`!xL@@Ly^%^vk)|&&_=p7iQzTy zufRX0i*FcMP6}^MU7kL5Qn<%(TIuArZ{@=r4x%hG$skT;#`vi6%pNB6Gj`W!={}^E>UjQI-R+UVPcw)`@4UnU-e3-O zeQid#rQ@)FulY{Y9RGIb=2%g(H_Vkvf$~UTh>eS%HLdR7YsFnHlw2hbO&AB3<<6j-Uf-gRiE;XO|k&0#9XQ@NFg7$;f z_h_zPqa5!x?iuAZyWeHwV~D_nf`Nk)Mfs1TtpjVHiWvICI(NbTE5i{YNh#`xGAFB*-PEP1Ew^^Q1gwU#EoLWMPBnSa*yfbY4UQo8*2W;$o1@8r9p3NpDW7yA-hesa(YL^PIIW*~cMlKj)Qhk8Ss2+NJOMOX}h^WhqNG}8SE8SdsL4;Cztz`PZ9TdpNa2RhMtQHcQ)mUp=`iWQ;79H?6CdI zSC=8_*&Fhmf|c~y z7y|Ekoof_cfi3D-dOhW0H7sYwavz9VLjzpDfj97De#eW2D>Gb&%y$#rxpLC1DA;WA zf#h(c5e5cUS%dXitZTHO(<_usxY|($d9Tt$bDRDd1MN2dtY!`CONCY}iWpTGt#}*r z6<&_T`B+!QMKz^V$yi8pcpN-IgMv6%BNUNd16`qeL@Z%@9$G4^8S)r*eAv0pBS%#6 zBDorU#&)ZgJu;gRm2ou{pA9JWc|0lx9xbLUO&4g9kWV5_^T&$4kdNZRAs*XA)w}Wz zrru+#GF5)(mTHY03ZJEanK1D#Z@%Vcx!zJ(J~w7PSPb67sHyENm$0kym0NJMqo4J| zF!2u}!u=g5plFZJ!^2;M`c;mehjW8%u~TR+4<$AN`_(H~u2^Low3_r5a%^gruV?YV zO7Y-BTcdY^`7e>&I?2O#R|BA(Z+dpCjMQW6Drt8w*_631F{Xr4-(@g9X%!uDnfmVi zyo<@+A-f{oK~>3VcsR{#?}8gI*=G_ed3ATDeM%s=rI)F?x}Oid`c4_LbQ2XG&W9wy zpc;4e)k)yG4l$23+|6Z>)2LlP1?6~*m|K$Q$bXxmu4(KxoNp21h8T8;L6wo=y_mhU z`YRlEqFE`lqr35~+>TI}z!z6Sw&FzO}Vp#q-6JF5SDq%V(#mSmPj} zuGVI?x2qFTk04AYbn>3Q!phoSd(fY^E~3J(i~_3|tEpYukXZtO+;pZWAoI%b@LAga z2Oo7ZEenH~u}FDf&K^-4!(Jk9i7A9dCabj>5W8Fb0Z~?D7s6~)%;~ec zNnG%?xXI|_N|oiqmt}&zxJUZOEy+sc?Zs|?VV>W9SiWmgp8sH966Y_7A|qu&Ll!eq zYO-wPHrZIKH@C&u1|?RXMBneP0ovI={;N&Nuby<~QvwnoINvk?F4uLI#}+HGYA#f0Wc=1Fs<;rU_P}ksI*` zhv_1~s)Eb*x+8&+$te3MQ|eR*YWSLn52ZNo_zJ83Qzmg8>r2o*pIVwIuhq)gYo0xy zLzTIHR{N$>q!J*#gfD=*d6f)#YNFjP-qurT0` zqJP~=r!#H9^LnC8Lr8n`1LYrzRsL}OZoQulwka>%6;<;W$EymKeYyvEwt!K})gJ4s zBSogA=4hIxf=tiaSLl-F>|b-35^kkmb3B{;kKT~49~JQP`DH8l(TZ5@cPd|EMOxw|`7KJQ;X+aNr zrGhJ*L3dn-!a;9q11^hX793B#gSnlZ+SSgFEgJkH=t9n_8LEVf5a$}KT>m)^fPibQ z6;9#WxgD%?Pg4LEo$aTcYCJqpDsBe1A3S()`}aU`q=+f*_sQppv9)EI?`;UU!)7Ik zT%Yh973N<&lDEo3kXWa#M7$HqeHsYG+& zwE1$+IgPv>(YdsZ%t#;-X9zX-(e#lY@0ScLSj7rKhMUjw1Cd{6?&G%^#@3Ig*wLC? zsK<`n(yjM-r?YZ>Uth869CDegHVXC4u?{%5neH6y1h~UP2by?M~S9=mMPNLf%>b%P|)iWNoS0oLb|lia-$ji`FLC|`g$1l z+baKh4ffDU+}ky;Is5kSz@75=BPh>IBX7q8*B_^%QqyI0#y4GE;=;p|(M$LM<}5@h zI7`z&(1i(-Ew*|vw=@&7%C)0;7-hQR#AT%`{{xVn=dU-7>b5Vrc z_FF&nt`*vY{9{|{1-u%bw4ZHP_~c6IlS_w9wCx>6u~CE#8bn&dH*t?6@Uw>MedDdi|(gz*lDzR$38$1Y;tiH2S6drC{lgQ!4HVr`NFFFV`<8u`=*L9WUcC^*ncIB%q`f2PQvwh3Lp zu)tNouC!dVhN`9T!RDqbaoI#6aw=rJa=dUTea>0Df_fFD;reVeVaZdyO0 z1*aX72W1FjC06&=!)g_c&uOm?;8;TIqz}|KuwB8WXRHFqKK&F?!0K6q%%ZZx{br0_ zbm$ao$#8I5Ue0O`!jI^Croo9D@4{72Q=9@LwK!~PL1;%rwoK7uV}*vjUieJ6Nh#fA z8PkIU_3y7p(iJqBz5L#Tt|9iu@3i#`qa21W7np%DfGUD5rIE|GE8!M@e|($B&sti` z{#gpsaq`ve@14?Et2gbor0;Hb1r?z9I9JifQIlH>b{R1@2e0f+4PF^LkBiJLwDAlb zv6@|pU>fusfy@^j#uO%M?ZUw-y%Lh1M7iGUK7)&eMB;PVp2@w?k>>vLf6T@r!VK3w zRWFyoRbt6|*QSk5sa%lhYWU2wzhl`savC*NbM3;WKxSf806h5ag70PqkRQ2SW+{`s z3kBO5KKL~Nx0tG*DqnA#U)rb~9NM_@;)>w_GdkO|wT|}|LAaN65Re$_8q=>FF>HnC zFhw7yvxeSPzPc1Fn8dPa?p(!F^eJ9e`1oBt1$0ApA@(Z)S2VZJH}ai1PEt_JxJGs? zO=aD3E-kvCG^wF{0UO4v5wv)osCJt$U@qV6=c7vgRXSMhIc;l)Gju#1JVgpB9q}7B z?ZvF2jyLj64^QEqKF!>g@Wltzkz+kMiWOB6UmqMK6>V+%=;kE@3Ds<3D8<6Tvk; zQZpjWJ*}gD*bmhlQX0$k?+J0MNA;FE4Z83D#4BGHHoZ%4tgjQZpqBD_UwL?e89JyB zvE3k*XIf@gyDUUGvNrBG zMxFvV-X?sTn6x7KfEAc|8u<=>boL_l*zvM^bP*Pe(bMJhtnhzB=!0xT*yI#po7;aj zlPZruV7 zwHwX20-Ptg4d&>Ciwf*jURd-#3Mz+Mpzkg9w-3~Za}QJ3 zU&;8$i^as2j<~%qv{X}r0?@A$hHCdM3ipDVT=<+q_4)af4v6wTi#;h|izC0(kNKGr#t1U%3b9C9C- zuK4YI_z@Wg1B|I(AwGL*2-`w8d0uIsag4#ciq_L?dSq^&(zI~AqLuvFg#S+^Kc6bWAke_XgD@4s*!}0 zzUtyfF#UAZV~G%Pz%$DkvdQ)&gGcEm@{c_HH6xr0L+|k~f}|WtUu~z2w{nTElWw1C zXXseNn+R=9Ip~o4b=|i(F6`e1+va*kE7Z#exBSY$CV!JOEB1uO z&S@ZireQcKepBYX5D+yrT#{!5y&X?z>!owHA}Tz`yLN*Ge@Lz?AqcSfv9{;wr!4lt zVd4V02GKkmQtH=h38W)`{)!hR!}tb~s>fO0eKu3JA{Kc%1E}FqkmHXjaZ&+(-Bq>0 zrQ1PAO}pK)>bX-dl3Qi;7ZG+%&Om~Y%N_J?+obEsTYwuz`%0Z!5?delPg}1B@K37> z14bvXWt8b;UJZ7ts9Wbq`tGkq^b?qkCF?%$6JgrC6`A@j@Sa~*oA54|YMKd`i70P5JvJ$bt-f{Dym6A_-+ zZ6XS@R^1l2EYE)7pZjoo6+!3ckcqd=Tj|M~Yk7&i1vb60Tf`=0u=G_U?C4PB70Fl~ zl)Yhjhc%zdgEbZFfcTHkD&e{oM!lvvu9KY!7^Q$WbGAN7T{|JQOZ~i|ik>5L&>l$T zZYnEdk!*0bSL@kJt8)1^dETs{Eb(K}$4Zyv=t>OnY&uZ8y4emGGA!R4m8!1h>*hX; ze4oNakpUhFx~)aD--3g7X4iV>Cg;&3@#*|USk>FP-*gnGw9C_!KrQ9cC7Uk=iN312 zoW%x$ZB>g^06itIohra-0P%Vx@Oe%yKvZUZYw+&@yMCet+FgI&CN57L2Oi0davosK zf^Qi8gYBv<%(D{wc|N5Ol6woadd?CZSTVpAa8%}j z??L;cO{)*nfGv5imi#|<(0sc*6=fxZ$AtY{A5!DXTpahW(RnshrU6q7vz)9h(oddK zt)cAAd;`ODp(w9bp;!(fS@4kVh(55}8@zLagYPCR_r|32^?S*98{u6^pPz+5lUp~tR)E3lB zs5wqAAh_pBbcVCK;a!f~mMraj3FQ&QW2Jf0jIp*kVhoGzzbOT#FCR93zh0@Eg^QP( zFlN(NMRhUiQ8AIDYEyBLcq_)D>NP-#15r+8Lh(5C*oNZe7B6%dtpjI);{tSvzZKsH zaYBQq%-Eo|!CKw>y(l-#8VrYY!Vgc(6>d|5}cRS7DgL z`mxfPKc70XM&Q5rKAgc9J9W&D4G9TZsH>@!&olQe!QLv@)S^4hSG03|ZF)UaYd1@D zLz*Un+aYd=rah1!>YDL)jSIDrkt41$p0>x;6W`GWR_97B+1aJb3}5?WOHr=#=X5Pa z#2Ec^JSrnlO)#1^h^tdGYsJKEFw@beUbQ#13pF1H7k87{fd?7p^`zFik1QU;=41f$ z`8xddlC%aeakups=zWLY!E9&Qb>s!_qtg3Y=(IIYZ9$tkR=kQTY5*4IZi__+FRYLY zujp~tC`*~($^QFl4HeJ35giU>z57vD=`w^t_3zp!{JoGp5mfk{Z{B*1 zlO=_E3h*J_yVsRn2!1;58jds$(yJ1C!yC9=1!eP{JfiNn2h-9=Vkj*`R44-fP>fPc z_Zz#9R07`GABo7^1AZ3}R@dhEBz{DG@47*IRsu$^Z0+Rh$oc9ID=lD{K?OuZac&by zjy0d9??SIO-~}*)GpElOv9cxL*MpS$3-J2rEvfF12 z3^zBtEer=#xp}7Qpx_M`Mpqvco8-O9?66U}TK?+zaF;NKu&Mzq9X!;j1>hdmV+^To zvJf+|M%Me7I-2RqC%o>SZ0=ZVN%F_f3ce=nM-jJ#*JH0tE8i2c@A#5f>0hV2|JT{W zJ8Y#9^{2`)i9l)505^(DhFQ`_Nk#LA^>7P&^p7w9F z$O~jHv$#{7533l;bz9PdS4JH{g=bAeRY60S=$vqsbp`BqCj?8^r0RXOWz_cWlBj7t zVxGvZ8G0vp9S2{WnBXh@ib7idV~UZryt)DIJV)1)Tr+0d?{+a4A=2C@kP`AMc*GKQDhY15TH`ba;>EqFQbPDs8e!e4xKavfubRTZ-D|B@KzU)PrynQ+%J)=jp29bYb| z-!d2>b-196>UR;yG>hbvj4yvs^y$h){e^czHFhX6KE6xWMlJ5$>=F?+Ea14E7Hxm< ztHVPONxX``sM>VuY+Hz+CugO%#Xe)lkdkYFyA8#2o#%Jv)4}eMHDSm8@zgIApmnuk zwWKE-D*=I=(OCB)IQA(zaY=D({H_e@g1Ovz?6+$AAS#>_>2IzUnQ%ML#_VCH4bi$0 zv>UY0|2_%%y&s}9i`RrkCKM)B~8(% zOG;*8DiFxDu~B-pS^!q9*M?G@h90MchjYIy-&%L7cce!J0fMCB>pOA2m*|GWn_sM9 z!7N`0-Fdy}wehHtm0{{eznt-D3mgubmM(2)7fDV6jjU3PUc`lu`+w-i)Pfcpwv)pP z0+-UG4O!}PE7sEsS98M0q0~b(oSX-W^N6j}$+A8UoaGOt$_L?s;J_w9W=0l@`+2^hzQvzIgizCDRIwVyQO*8h$${}f_b&gQ_g2J_E~&n523BtO^Yg($p1!od zSsLV-jNpj=?sj*=(B5Od1LOMfOWrT-EJ|Kz?10?0*VYSBVC`QuH>En(UZ-ul{LawH zpYz)tQc0Pn1PAsUmNr=j8FJ}2h}MSQPm`<7UBFahtiH260#x2z zgEJWt$!yHWqj^WQOdj>&1x9eL^uK>;r)huG`V=H4V5?edzxSz1O34wQeX7JJIdsIU zY04ArgR|(=H|{eKA5xbBFQ^;EK^JSQJ|yk&czE#mbX}5mp3D7J-llJ ze(}N1G{KHc%>z20WF>X&f{}nS(M;jB!-|gWA!uG|)^DI%u)Op9Vjw`RbEsJAB~RD? zzyfr{eH@IDn;5X7R`V#?bF!B69_+rCIvUjGS{;)OIar`L(*uq5b@uAA96}i7+SP0KWbaahyhB)uE{At*pvxl;pt>5m=`c;Al`~yS<}t_z zzu(l@r0P7PKK$L}DZC(&RFvqWeAuVqu~j6H2>?(Jv!It!D&`8ellsQ9ws9U95~1;2 zeRymh2}7*Wt?$9=&;{n5h$X3}2&Ff3JpU+7b*w)7pJB!Zf}g9+PP&sud~=6_Y`3(V zNtMK&k569dwULIAw4q<$!W31ELBmc=?h!Nz3kiY991HS>E>ObY2@1+5<7P*b=c@kj zo*g`9rdr^HMu@w8J4cE}1qPE>3+x~O&`z!dJ@mXs+iQH5gqnNNiK)=){(cau;abzG z>g&o!Mh7q{S!bzPYdslGI<|!M9xg1sDAM~WW{u-?duKeBDc55jyg1;mejQr8T}axK z1WH2e+f?OBG-%7#KEwuCI&r+r>XcSb!Fo59;h;5quoOKW_**CvRy&sVVVnY2wdc4f=_IdOb}Yjb;AMdGN5i)>_q+Uf6k!Tw&l=F6;U zlS7z7hPg^d(#L0p=Y0#s250whQyWuPS%&UjaBerjI33)Y=wCymdZIB$&q%+u@L81rI57bkvo=me%nSXRE82_Mc9Q( zl{xaF@((N;6*l`Uvnw)G6-&dNAPES+Gp0O4$0f7y4m?R`T=%+)XB=DYyAp862lwl2 zK0le|u%c##XsN}K&0P3K_!w|wzX>q1o5rL(l;O3|Lf!jiN0UgqzXuc4u#I!fLhsk) z#l;1!7)I^!k&nJTP5Ze99jY$kX<9M2qhMrsR0qHNj~Fn%_cpI;8z;c@cyWz>yIYr- zZk5!B-^qRA(Zs#|WFmXi3-eX)n^vsIXW(bxo3AUcxz-2U{9|Ar#8h-bAG-AazHgf6 z>zf+EDU*$-Bwh)vq8@FeZ7)X#v;t#t(YouCqoo!c5Ax8`FB>8QxY3fbyWSB+)}pR# ztR&BEu)PAIhL3wp=mD0?yx^^j^`RSKT(%asi@39BRjrEiKvi!ROq>yY7(%R#E!z5` zj)J&uh|@*FzV3$+ahWb9Ow(5xUY3^KKhZId)UPJqCC%o}m^PhK*bA?uZ0`H7Upz5* zE7xc;sXxtW*XQQa?TV$MMIBjLOF!YS2N>94sCsVZYHD|7G&U-8_T##h*ZI^3{jiW7 zw~p=4cdJ#fK&@)u!-Pf{CZYGb;Q-7;Fc%OJPzmqPk$}t-JsQ;2v~z4Q@%f5hKlT-? z)3!(KRs!fgw62|GkiY^^7?siu3ihs@{f6R8DhzUMs-J(dg$Ss<-_Z}6qzhje-`|5D zu5<@x!@ZJ9mANLM)F zF+-k0rKL{Xs*Muy!QJFnVPMICb=3%Ar%Q$2u(rI!C&?l8B3%c|AeNf#ZXD2hoPhH( z&_7eSbKnQ8NJbv~622hV`S<^DbiY2=h*{=p%_|8Xpc`Zb_yWM!i!vo_J7!^yLG<|O z`WEGyn1R)Ad&Fe<(}Qe4bZpU!(!PM8`=8lf5=J(|6YwnVFURFZiflP#^pd_k6>^`H z4SB=}TU6_HINN6??Ppe5W>sLaX8hf;w3L%*MhnSH27F?^vs4luzFgm%9Jo{=<};A@ z69L$LiVxcsol(AEj?2b z0XM79Xno0#^w^#jH-j5%)k{Cmd=Y(m!s^)Jd*Z(SGndLeSDJWZt*IYgRu|- zr@z5p?X@q{Hu%GUHb=(YU*^wPFds%L?-u!bgul*_!uF#)E6>KPIje4Me>YvUdG7C0 zV%hOwjfc&rg_(>nF1te5+&&Z^!h4vi-0;&hXn0E4IRcrty{lb4TW=I<=)BGHZS8$4s#Rq!Pq z?|#OT>cd5p70+~L3C&jB=|>dsM^$Tb!~Oarr)L3 z&-m^ZwWm@8u%ECU{WdYWRr8%TRg~6Uf{8dCBe9&9X0?OOE*VHp?5+u|wyZ`qISg_7 z1#Tx_QiKBPJa^BX7cnV^3>`YW_Y`JQUi@8g?Sep`@cyEtF`-=cQmX;;5Qu^dcj&&w z42PD_vlJPj9pbj$3b7J$$3$hh{n`-otM?8l#h!gcyKbxK`+DdKNJ}Ee(wabSrnt?T zspR?vU+E!D1P9&89j#pLH83k4CecyY^p;U`6nJF6`wv}7l zcx>(NHFj(KuCF6_XQ0mJHbeuPK@CrzKtbMsTXCuPMlVbw7%rgKoDLV=Wod6)2<8j zN5to{d3sWI>R)!<4+;)Txk=tojWFHwFyrysr$;EuUDGLybZiIEpeWEYKw0{M)Yh-P z@HD2F^m^ZQ$9Eg+T$jH{R;hh-9Vv=GzxM(okp6JXtK{~zmVN)ceU{Drvr7AaZ*C^G z6$$HE(N#^-ISIaR{S&t;cD2w|%sh1ut>s`~pVsvLRJ7P1fZ+|Lk9_G_Hb>x>Bj*>o zVWlsRb{%wW07W(y51s7Z(E(7v;i~&9yeAOEjdu(zdiq;3u7_XLSqBgwph^Csy@GC& zIqI|{d(tI9o6&M7BAPhlW_1E?Z2nGf$5q3LrdEXyg#wc!jPLmgd+iNrOLj7>1MFEb z9#ldc0_vOgx-B;rTYOjt6rRQR(PE*hrQ;j?*(QG~a%+DaAb`EpQ6@6h($3Fl@8zyej}HCSzCH zMtrX&8Z`JYHk z*zf;02+R{A|2-I|k3ZM+jS7`>hU zW{_u}lV)Ieec{jDb1zQrGM>T(Zg^8VGHx==Klt23AA7*xn=IThS5Dm!wu?W@z#ws} zndhPggyEkXN)>-GFx1h>J-;8=^QXdJpMRcAz8*R%*!~@P3GWcV%D~|HWpnZJCto^T z*u001^PPrhpUmjmTlDgmOV9V|_DSW*G_1QebDe=9@HF}48OJF+GXsOt+iU;ZLB)Rt z_Wvs~`Tr34Uj+I7KLh)J2KN6~Nsw5HVz~C*2E#)=BKpX!sqzy|t1O0eL%`K{*Ox{e zc6q|SX;bYsk^#a3B`S=XW6q=H0b@e#!C1>nf99f9-g139)}gK_aUHLM-H86^#0KOt z>@-G<)qP|K#aGoiP4LiSHgGe&jCpKvG9PA5GsY5;bKq~JuIJbCw37z2G+0=#N!hk- z&a_G!N&caE|CW=@lTsU-Lr{k)SGsYx?}0ZC9R}qce|U`LWMpSKmMSAGDY9eBiytQi``%W`@lEar`R5v@C5w~gu1M5HiA!`3Trs)F?;=IO$hX1B; zL)cV;%;YP><0VKW{6}_I8Qxvu7=K@|rz80vLAUNz)7B1cWd}LqAn;x)ly8rd^=s^c z>#%rm<(46PfNo+Lbse4@gwLBiNkd`y(v+%n2B=CQ+pv-}ak5X3U?`u=AEh|0{t0cb5fwdB8c>!QxLaXH#bWh@J_#tACnF74v@mW~>y5 zt?S4Q9C07LbK?MA^pW4i0!KH~o#vR`Okl?E-)$FW+Nk;{C>tf)TBYm9DICO}cC!el`@N$Bn=VydOM`|9f;fWh(TJd3%=(3>-(od=YM^bL{sjM zpzSl#|MYUol9_q)us6{RaD|_rI4oj8BGGXQ&}d{5j(0W5Vswq?%5>2(Not9O0>+VU zO41r0BL`hlwwL;$JaT>23V&KRoqRp`Tw|iRBP-UkHZOt>tI$z1iWBMX#*Obo=z~sa zlGH1j8IryJkc*c+$a}29OLFwNnPQ<^maZLhE*Pf#rzvDZ=JdCw6w2|H2xQQ$UL_Pn zdNEfhGSm~j!zTs%;jn%b8oUq{o@jMR#b%#O;Am7+yRDb7cm`brcvga)JyHjvWg1)8 z(isgrb3nVRtgNh^GzZdbJF$^$%~^Pox5Ut_JU@Lc&~RW@T_-b5)u%6giv=`{^pT^3 zIc^uf1Q~8#vot~1fe!lq!WO-Pt8g^LhH7>WmC=ScbU-_+IhJgOr5HRO{Y+C(Gya!> z&Y#ZL!LIX8ihI(#ouU@xZH*|M z3Nm7~w6z7X)c@BUu?}9X1>J_)Vx;9LueYx7q2+@~jJ;PzGmUe0fDFHWSGYb$dIuRR z-?YV!j>kt4zNU=QUx74gudx0z?H|>2cb>97ogSDG@Ir=!9{}noKnO10ask2H8>PYz z@RRS)j+Wm(PJtW`rvnrNHWY&8Z~Al}%B~DGR+%eGz1l28>*0_lDvrbtOE_k<5XA3L z2_SXMwD6~gukrKKVdD|c5kH9`a{rJF=USLX+^QO!2OJX7KpX815I=$- z(AUXN%(oOzICMk!*TgC7cuEP{Mn!I?9i}0fs^A(Wkr@~^ynKv>+Mr)bV~_CC)x~VT zH?uf+#6osPHRfRMHnKh`;ddhSp7(A%RXN-l>4@|?{v)4mIKA{$?oAo=xuHz$Du3D` zA$-Uo3aByrF1Tw`$~Rs|&eZ8NtZ>}TY)Z1+leOGgd8*N8U3u_VSb{ijvY0fPQ#W-{Ct3hQsiXYT(6E9ABkZA34M;lFzYjeTqEz5cSwHab%R zcKOe9dR+r7<$nws2Il{Vl(ObZcNi^A{;;y}W!9Kmb(g+JmMC9y z`_K`2%jw>)r)wuPo5QQR-835aTZAk3)ZIE`L9;n2V-jgom7aYc`c*-*p=jMe!q>|} zre1pgzH##+f#lL(v!P0QlqwM^5BT=_UE?oZrg#w z4GT68dW-A>g1)x0PM=JGW_DZIi^Sidvu_^)dJU`lP0_)-+Pw`Txf64`clWbxX#JyB z^+>y&HvyKEBg5`T0!ka7y6L(;z2@<3?dRYPLYi0otfk+tVh;30?F5{9TkSqOe^@`H zx}r7{6Qg%se&xeAamJ5c|K{m6V|TWPRH!b%Ohlk*kOSq{;*S-bk|hLfYg(u&a@mVV zxc{Mh0A|mFBvpQH@A_6SJ!j^lmRx1tUHL-ZikN?evS)5uB9%}Y~1q}$D@J` z&beSqJd)VjqqvZe*W_KJ7t+-ENVqkTUizLQ{ zT62}BTO7KOL4w-{jJ13)%GPOR5P65?NVdq}?zpPyBj=ULbsMV7dQgi0D|an^^>QQL zrdzuf()GLARZq{$vGmhG&%+KEqUN`^Nh}f;O^~pgHXr%JUkH<(%I@)H-0Bw6OJ8&D zVZHUtKEV8hJNAm}Q{$W}k@S$q(C{6uG1gFL1NeD}uHy!W&s(8ZQ=8G)kkR^uUT;TM z;;7wd@Q@Nb+3{vO+voyxP8KlchxMm-oDS`^Sf0o!cah^k`M!K*tVZ&@w;HY-q)!>* zq=TXhRpsWKZ7bT@$CxikN9oE1hiXXyH97c*tBI$N(WK_r)=gM+J7 zB-Mr^yJ$6*Ll4w`)}$x-*8ygf?NISseia}%8!u$iZ)l=6w!9_q^ncbBS#zq~0h}o@ zYh0k)tAr?igsy-6y2we0w{kU7kEC&QX_e}{aJhST&ayR>SK3EoK1YVXnvm!;S79m^ zlw;OaL+Dl!i%~x?Uf$nw>sO+_47XCKav7Do6?(i>xI5qTis&0;>)mmq-DtHySB6^a zXE@ug?jt8$<-D3^%i%?E>Qya`h%M5GN(j>Z%EyDLkHwGl6kf3#z8nz-2SBR)6Kt(b z_wK@#9A{Sz2QVFbt|f~T`>l2&{28Uf8`rQ$c`jU0HNm8bycAPO^M)t%NGd%OYNf2E zzxSadM`=s+(WMPcg#qMvFg=GobUy@}%mN0IG_OeIxc_Zhlb+#!neJbv*z{L{W=f^~ zrNV~SNl(zDEF2Hi`J&A3QI{r;GNCl(`G}h*`a#t=7X8xcaZ!%;MLJ9L4!S&^dAK1 zI9pn91i>xLxa^LA>uZ|&xtLjAYGD&hz*dyBHTl7a$9l{Crmd{(8>vfoN^*}^Cg|D-T{);O28Cf%8L%+4GWB!Cvp3>@J` z;ukGmDZK^%yMLAJxP7H#R`XB2J%vz1EUViI^VhCuB0jZ-2BCBxZJh&Xkr4P0Ol8@N zoy3GW>ccOqF`n%P3k&?X!ZhInHY9-@QXgxs7KH@KSgWZG$F3`mR@|{wt-cYQB6F}s z#}o9TzejM77doL%M}|8)<%P-}7X!2f;XGu+F}6-&NM<$-0Ik0kTa}# zqa)h5HJ4It5U*zY4Vo6%Tv)`Y5K2|Yyz9)J)M3OxsOGTK1(>dp5~niWDj#XcKA}M< zSD^7(D7NYKR`*b2mx}aOyC~;C30T&>Dc7ms-iF%0NzLTJ-E55-5rmaa+CDK-=4sVT z7o=T) z85#;li@s{HV(UZ4ny#=fN9rLpa|Uli^l$H449~7Bj_et<`P)PG(=lGsBIuO!$|IFQ zo~*3p8wtl7no2c4Zmu190dWrGb{?Y0@ zb3T3G*fzec4^{MD{w$QUAq6Z1;Fko_Jkvnv}?9Go)Qq|R2iNBxx zD4hD8d8;(%q3ZYj14XcVVc7XB7h#v~IXu z5Vg$JQN6<#P`f*kC6MLU=?9+EO#l97*kKG|-IN(|CwH21kXjYaZFOTjK63{Fq%+_< zpfT8(U4NV7;%?JY7(nu~rPP@}iSio@fRbkpQQ>A}_wnm#eV8`jkmT{PNr!Hlg5T<1 zXBQ9ouE;?aLW>ZO;mVWNs2t}ozP+h06eU7z6xaJ zo|>M)wi0nJqEj_LvByOy%CBg|zxHp=M|5xAzvluZh`4rs$+O zHy8QL^B>itaoARzugIRB8~g28oY>!-&h zl;yBD9C@M|O{2Sv%E82Xx?Un@8ob*+NeDQ|hR6pGQs~Lo7SrXjehu_&$PyzFGY&f3 zX{TSt$p<;AoN~+7m(>T^wP1x7u@E0boV2pm^McK5%_xAJ+Yd$^QsKby%J~39NNxOG z%cA@1>ib_hkm?+e0svY3m$FX8J8^FrE%kb@E9{dE{q-p+7gsF>we$o{FDSAFx8%AI zDG!P7a`$D{C(4~8KQCjKCr%kRURm$LKLrPu%Pg!@=*fmQ*Az%q0PT;8dzv8kV}J;w z!o>|%N4E8rDmk2Wu!yyqN>8tyn^8SQB-Z0(y;n8$-lyX6@p<-IQrCc6ZZAS6f<@B%^mN2~y zH=W8cWRjIktqcy$*5T3*7FCjS(S|+=$gBj$&ER*_Xv3u)w@lGP^E1_F;KV)WoXguM z3Xm}6;VxdxAJ93p{~pM6rGNHAu|+;0?`$c;imw*3cbZkO%mFI=)IgcqYvni2QcA^A zDtS#!ny__UE$L_KYlG_o`7jAttQR0Way}S`qQvt7l-c8N*DjPmppgCTH#zG+dF3D0 zJYB!l&%BOlFern#^bg{j(#jMHpOkckQ772*%o@aN&(GjW6bnL2OI4HVU;V1cZ&X!2 z*zChe?tvRXuez(*Fddt3;*mk(XCz^r{Oe0+oUkIdPg^+af8sIn zFJ?P~fGQSk!i^o7VomLs&sIJI>Ha_J&ai_MeD-7?v|CMoWf`^lFIDyrV&H+;U?0?F zMqEH;of@utEO(S7B_4|l%v~f(9VHKmj`#KOn(}qYDl1|&D zv6L3}x(|P1fJ#3NihG^Jn0d{lX5KF;$`)GDZN5-oRKP|9qn3%=dY43r9}=&zQ#AQJlS5`4cuZ`%*Ab9?2m0NA3Z~v=eCaPE8u$?V3RVW1|wYx zV!id}aF;ixHy}%ilg6T^kfP(YZ4v(k(KhwijJx_e;dWcZ@ohj$vpP#*V zY}ionzVVE%4NttkO_%kX1}EaxoczRDxRpMD4NCtG0fYC+2{|{S40wS~m+5nLTJR>F zR>iyT^a$8yVoKXAMM?mp_lCy>D8MbBwbvxr$@`)tmTAY z>3#LS+P#8q*kGfJ%Pj$V8*ZG5n0t0F@gn_+vHw`>kC^Ev^1Rtm;syVCFGDF8T~H0G zIn(>lobsM-(I=^=pOoF`!~bSDhkLAZvstSpRD$k@`NI(4U-#*@%nCh__egZ`9Gm^c z8yFJ)Jj0tF`f)zZQNP%l2`w}g28O}MbQL-7tJE9!w$MBDx+kIB&M$&*#{MtL-ZCu8 zF8UWneH0KSMHB=?K%`Z=21TVqxt1{9mA}32uq%H&ysS@{8X_#8!vDc~J(M63qlAJ63gZG^I znsW`Cf4^W;R4=bt;wL~nL5JrTb6qW*<9m+f!xy$u-1lVc^S9<=Rdz7+2lXYU3wA^=a?TbpNn|AT`T0>?J)eBMqgik9< zZP0BHTL0-NVtKn6A2YpBJnjHeiKxQ+BRl&OW$TPIaH~{3UIXp%5_F~Sq+J1B@lLAF z9RF$G_Q+Q7yEfk>iA$gbLjWnTIqs_b+!AE$b*wdv`Fl0nC%YV3y?iYqd(by!#CZ14 zilmBuq{L=fP#wLOs!Y4DeyDCUn)iLCl+0OwguHLC(#Ehtx%9qbA0(IkY#du6P{jX- zUSBw;XMJuv*qq-@+O}%u5ln9R@k60#2tQiL@0ZrMeZ#6Or4sjd3iv^aTiv06;4 z|2F&qEH`WxD{pvs$VVzpCGh0JKOC36-?^tIWq^$x7g8pM?tImRH_i}}fRzHI2-%NP zZIS)NE5Y~iQ)^D0BAxcNtXzvIx7593-u!_`WoSg<{TiX=oRZ_3KlgB&9BCQFln+#- zZr5NuDITXbj!Dt2D6p`)99oblIYqbR3ZHRIJZ4j!*0yk?7y9RBxd=+&bkPt#Ni(3b zB_$)g8hpGqISbZzZpzrgceX&a+h5|=Oe-!^;anENnHE-Wx%W` z_&{QBA`OJlbT|z9!)Md8^GVmLT^uARsYq~LcE`heZapZ74_II6yafSWl)4XUc42HT z+Br!e5%o5;&DSHa>`1ZQa2h3clhAs!$$y#2Cd_wvnyxCN#3d_CL?lCyt~8kIh)r)t zkwyh(w-n^RWqE-;7ucgq!!YtIb!ZSyDnXzSX^dqwgb18)x)i0Cmk=Dt^dR1wJ1222 zcFLj`sUDDH8Dg9`3<$ny#?_GYL1EriGlmZeC18k2$g^DDr(`t@>xFDHH!a^aJl~-~ z8Dp{vOZ zS-Wh9(qM?uznu|Ic5W)uh=w7MW>3=UBIa}XA@$+n=RT@Kkk@Jh5rcM3+$;V5vt{2D zfxG{01A>EyiuF}kQoOre{edT?(#&F*IzYElAJu_xSX@_z3cuRESVYq>B@L$38wlFV zi@_iD$?iV$VBj@tr8`Jd7P0?0RlCoL%dhA3SF77iGfC!uZWb=rUs^t=q~^CWex)-A zI2K!R<*PZeclI?b9ItdiwW&_6fTm@gip89=?RU8=OD2A=jt6e0lQ}|Tqjf_%%adHU zMk}BNJ1x17po4ib5;{ppv5fs!%MlhQ)$vYSiJ=1DDl6Sf3=jdNSOc0W=(w=Sh2v8M z(Vq|;>KJO6c5Yrpj$Y-j0z`laD4$o1g=t7$At zw`G9d-**3CJK@YcAPoC9AR<5~n)Oq6unAvMib@xGZ8Y%kzNBsu>;cPW5A0jL#q2G< zlw|;XZAN7D#jRGTU&Cb1u@K8y9LoturuXgG~?A80!r>bs&>`@9MWHi`rJ`NIT8s%({do1>sOgjFw z{<@2Q(r0BTo!R>AC!e!a%*63cd6MNP@`vp=S=PnT%>3*MTZE50A%6 zrg^cB-IDtXhY)TItReXwr_hM4+9&oH5TKruL>6RMd-i22>fzte{Pb2jxOpRP9g>&pmCbJD z19F3G>)p@BxF`kqcJ_U9gA{KmLrMQsDsE3~ZB|;ZCkKj@v_PXiRe$A?)L^&)n?*3M zNCNfIe+w$8@sFNWp2$u^F2Ztm4A0ljMn2uq;r5!Agt4M7ryrK;2)_a2w6b5}8?dTS z2hw^RI%L7$LpPXX?8}ZuM{rr$gbi{E6}_S=$u zG7pH3>YurYiwetF?CSllvS=YkyG#t;@X=G}gl_XCRvi>;lvov<&`wp_>Fjb2;wo25 zPmGABbY#|Hzy2Rc(Yp_y8E|Z7Hn;v7xs!N|#sD*Zb7=jsLO@&eIil&1)gBA|>sN8#?dz=$vwC#v z3ODy8YUE3F(m4$$`MZ{Dzrq+AHvQe?F)mwuG~P|NAbSG+$-f8tuKriB0Yu!#_8+M! zZI`D*RT^|#?{PV$ryem#Lq=QlrSsSiuAe`j(^WTthI~G$*4kQQW^{eRi}gT0Yq{C5 zgt?J^%FG}RIMf{~ZiEgk(FKXh#7Tq*R(dqUKg83X^&bG<|-Ij@m}$jTONgy zB)Oe?^#tOd4I2{Yw@P1?8IwhVxA!@sDjGLU`p0P#`*e8rg} zEnMf+6I-|f;Im0Wo#2NSpN-#DO_ke~AehMC@be@BuyWGej_;kJ{z)6VsX;UrxAjS9`&XmPDJ#^zXaDS`;orD4>(!P zLkV!tKF!}?s~;QcDz&cM5Kb2_bvkb0ckD3(FzyKeuO0jd{k4ZZ-d zakt5q-mY~rX_;0)QFavfX3bo2Vqoq=Z+~v!KC}=l?ERGLr;|(%dN6M4p6$cHq$~(s*vfHcL&T{Tl;#ma;S+P<$1wD>&hK{6214AU_?zIsuw zS6_}hOAXbFjJyV8?z^J%h6FGgfjj4A6|0HvUWlo5*Ta{tqC{F|dr4X9MWhna6@i_P z+?-O;k=?NWi$L!rV;RDClIL_AjdYua7_+gek9X6rM*;% zI8UZR>{Hf~F0Km!d~P7da;D75w&$QJq_IMwq&U2_d3iyqWEPxJ&6l<~y`Z%4nE2yT1f7Cj5euZi%FJ}+;)vZi73&rp-38~T7o~q66 zD(+5IgLdmKFeJ}}>6a?)V`uBsonLmiF;9I@5VJ6DBj*+sECL3SN@z|nX5KnaZb`=D zrh28>tB$01vf5nn3mIX{MF+Jwcy^7-Nix7=xKy8`YR|5D=4W#dB%ccYh;sr5#S?nw z;7%>th~Lm9t6lfKhUkLOQVW@$TUWZ98qX3(e`|026#fvUU{vCEftWm|X9-|fn9K*G zun97wmQdTeue$Z%T&w=hOu3pGu`3g^jLGSyLVK%;Umz9|2vnTsFf#JjQDynIy}MOJOTEN3G3{N*ya4V878->|0t9@Q{lh;LQmsZYBw&Bi)~QX+XWzPh z_?9U$Tf3j`9SYI(>5GYGs^98xa7Jg%_LDe-G zcQtq~@lSAG@o!LFRYF51CcOE`$|6nn4j-#>$^D7e79kR)Eq+kC-;&yH!Hk8w{H|)- z#lQ|86eu66f zf!|Rvd$#Ou9}^A?Bs4Elip|P^?XW(8K3HU#$+M|vnJ4}(wb?2^?S^iGs!y-^3c)#J zO=C|Rmn~qUt*Z?SWNXGcr}L47QIO~Ysfi6&8ooS(X8zzFe4HPZ#pM-_3i|xX33y>Q z6{9*67%?N_W`DH4LrYRwGER-$_KCMph-xfGVG$xr^X{MsXvHw>-jUq>qld<+eZs!k5kb>@8_b;Kb z6YD8=xC3&!wVIZB#d{oKZ*XxCp3`gnbT&vLQ(mlS;Sex#&0VM%n3*#Q0ocTjN;;|h zCBB1SAuVX~`rcVXMet zc{s3858#N!RKN~TPIg8L;z^AUmO3a{6uoDbubG19#K0ppHv-oU;@{j42J4w4Z(A-- zH7AbULO}O`h^$ph5X>Dmutd9QA9gWBOzq;ufuDUWzv{PlAG5_Ekj|MqYMZfq1 zcMMQoL4>UOIP5Rc?S;~!mpHQaZkOit6IqvV^PkCEu6er;VLNiTrJauTmI^c)lRs=7 zX1%|7gd1rZqMbCZuiszbi|KQhyv`rQ%qFtl(a~g&oJGp!Dx{bl+xOCx7CGTYACMmQ zekC~(@jOUofCF(<<5|7N`rbQ5fFvV#r4AR&?c?d5Bp8mwcXVc067@Dup6Uxs$^JLm z=;O2Nsr;Tb6!PtLwpq*=#WOr@Nu+x@vYJ%UW0{c=kv1NdtLft$UxV%%L4B0rk zG{TIm4gc`eT=XaIr5`O*G~C_va)#rAmFInLc^cTqF1N-&y%SH=2dhVDU%RgVFj?h? z9C^PjEMc9n@~GFjTlV5jKj?E>+C>P?5y3NsU?sz@?>j5Jk+LIa;~T`{QeQA#sX&J^ zY^BuW;LnnlN&mlhFso(US;{6qx)ZAzJGZ=W58oc$eSa?`c+`e~|9;WxH}ueCm6eLU z=`wo1#E=&j7Xz9Spi|(@#d+I!2OL=IOTMs5Urb<|Ppl8s%^nMuiZ)NxNe?i5K%m~e zhGVgxXjd{`$x9|CsDw0HFD2WBMs~DfG>R1Ht>?{#nZA*$?+Fxh%jb4zzNpv8Di=Q1 znV_>xO*`s<2>k;Q=Q!MU;AwwR{1unH!4J)#>=a9zjl<6t6EN`kZQBv_<&&kHr+U|y zJY4XBm(^KF=BV&9mA>2|Dt3=EQ#gFmfpLU(W8uzY;%D=;8Pbnqy^ivr3PsqIF!s~# z=NY2IHEDmMmZsWNF)MOm9Uq!Rjq<2h{gCVTO8wioZtjWA$?;UVOK)S!n6Y|C#&p9| z#)6318;+4XGnN;h&#CJ|?s)ys1u^fmUbd*BxP4tNHZpD%QHj|xPQ-w~J3-omEB3e_HtsRs; z2XYbw9fRdAd}D(X>NlM9XijE5ilejBFiKBn(6pOYz>wYqpkDyk9o4HHnTq9P{?Je? zCLnVoP_-XdQtlX99sO&cB*;RW2=JQER#Upk?%+-Q?bctjg+eA=_DU}A5__vu_F|%` zc13-}I5he!Z;}BM-N_1D-@g)JP?*Ee_XI*|woBE^(P!LgOaT$55J|=Dc6Zy@khXBJ zy0(dAJnpHN6wGNEsjz67fy~th+z`=8;)@i%X7qOZQQD#ojow=nNza!{oHzeK1(sY&>l@f$1P=Cix zeg}bnjGjm%_#dI6np-^SPJprNMcP1rjC-tLq1tC9aWS(YXevrLfwErGcHn_i`Saec0&Fu2J{$@!+*u$)}V(ZSniu^8FI`Jnf}e zqX3kU$Y2YI=^N3OtxU&yAX0CKNV*~pb=0JqgnnS=*DWjgLYGs zjm4~^m5Eri31{Cz5Az-hsLZ5u0lv&PkRcgDO_lSd!l9GHi8%WF9wfx)EoWSqD;*2* z*AzXod=7#n9a;qHEo}3!79Gwr7-@xvpaisfkyb(~@ z>?AQL9})buT2-#OV7Fx`>Uy;xHCDn!-S0K9$RDLpjas!!7#R#fAL3Rtg`krye@ebHRc1R%f5I&jY%SoWXsq?? zYWWvten3VTwEAz07<*iMZdcrmYHJ^YxvSThuItn!TCdyq+?dkWM)wiqyt(<=OjEV+ zlMcJX*2C!8L7AG$muUiZ$qP7=8W^$7FM-&`)NE=d?b{J$ecm`jkH`ICCAYsTFqHC7 zjAF~PvlGNEl!#_$ZgfmU6G}ZDDPHSxQYi7$gW@?=-dL`?zUD~=?spAp*y*BF;a|nV z>@E+@$NwvU=Tb^9D|u9X)KiAfFKUXZRjqMn>3JW|{1c&${B5PFc%RWapb{DFI1@AeLJkgDMnk#=j|ICAL$G87N&?u=suK;qL;zk?MoFnDRTsV8HgmFCJ zjwM`em`$E7m)|E}4z^S9@NV&NRLl)sW_L?;j&z>GNufu+JT?Ij0KBJ!&Yz7WE#ZO! zJcFWcrS!2!T&kTK!cQF!u*g@gisA$%mV6^Af6DZK?9EQi~jC9H51XSUv@n#=l8NvP)Sw^u`?0GHAG z=ce2xGQO#q6{%-6_h|-jM|kfbq5QeN_2SaT#)hfMq%MvR*h-l*tTLx@{DQ&ce=Eqi zWEgWTU?FF=!zB0i&c`I*pp+04RC(+N{W(e^54m*xZV_zJR2zAYXI~523pN-bqT*1_ zH&3Ca41L6*1MdHKh-{;)Mw3vRr^*r0%GqXfzfbQbJz9JFXPSilgVX)(N(&54fZi{Z zxKzFUFOVB^sgL@p%jW1nYL1P2ZqV*onP;xy1v$rzwTtD373k`y7$LPnVz)zI=RQ^i z)R^!sak8F>m}j@A3Ct$^gk##?;LqRp4Cq}^C)-o^7-$&DM@%)}PQ}Z`gbWQX5-ZW} z!@>Xh_;fJ@E3ZniW4h56{_s28!#L4d>Ro5dZ@Wd@(i(GY^)QBd&*kx zYw5NwLIn|9TRw_1xkG{#C~5Sj&)!7BoIJPeSk+h;_fHkQqhe1vZW><7;49wZxTZ`I zv&xDgizA)c{SwB*D`BxMOrlk)xcwPI=Ca;f&`z^C#m8=k+Rkx0}}KXKCR3 z?}1JJQ8q?NeIp2$^0T5E7h9dq-mI7y|0j2-NS>7~C2mX`wRS2Y1Zz2HOfRKkW~P2z z<=ev819g6+j=r6e^J3fITS*~#WZS|u1Jx0ApQ!UhRqDrc)YjNqH^d~W51!tBIh*8Q z-s%@}_0(EHTX0LS6c8^z%Z9TnsVZ~^fLbtE+UpR6SnW%AozojJdzFeTQaju9cYevY z(%Nwc)5mVN-1u`PSqwQ>`MlHCus%4{^hyJA&{0t4T_9;T(}>XI{--&N?PS8qek;=c z4D2p3^kt>-jryH4^DFo3a}^=b(W^dHcI4vW8#9ye#1*x%PT}jmNZEA3SYJ0QJbJMA zFY#Tc&3B?OCH{LJpVCZ+wbo#^hM>iw&G(Y&tZKuHfiIX>U(oSKQB$}uHow|Z)xT#y zhyX6taRZWF|JIpoQD^A>C-+n+sWHz$|D0bB8ZRjxr z_VLdqd(#A}zH5AYfbFs?ADc$_3KZpw#$NYu z*5NUHeBAd{03Zmk|Lni;3F>lcI#_aLBSuc@BK}sfBu_#;%9wFy_CUOLc{h!P7^pds z`LXHmp^~{P!ruVvPi2h&RpglmfP4~pA%rqkbR($~ppnmMk^YZo9JbFF=`Z9GoEmUM zB(%W}FuUqwgbd&RuJ#6+MbWA%jk$y4lc`4 zI=MOg9(cNtk=gj+Hn%=u2!Eagr2b;I_U|!2T3)~YtTgfDCD(dHp{)%QS!6=Q>4M>p z#E6Kz`yH*L!bMW%2L{ZR6a7YA{xeAZOI_~p?z%1qCnw-QM$4h5#D@<)K0Z92xY$DU z1=TCH&yOVt_};!yq5Om`qCe^FTj%kz&8xUv1MdM>)9{Ej0fBb|JqWMI^3 zX=!4OsB8_^$`hycOn3+j_9kTfe8aW0l{4Gw#&IQB)>YgC7ESx2qY1qnuX55!X9XO5 z86-+;F7#-VB!3J6SO0`rtL`X_pBUZJh7}yvk6=#3%Qus9;-dOTo|W3oUS8n;Fbq?9 zTX=+RVCH#bqL?Z=(ae2qDxPb~PK#AVa0K*^oC6vnG#Hy&+S+t=dC%8dqg|q198BLp zNHu7$etZ)`$>~-~4TZXnM!)Uu1aaz z5xaS^I1tQAuGC#Ej?r}X-CvK&23;5-sTK`vXg4vMS5QT#gV8A0{9bWdfZ(4{4T1^V z{@&&`Tg_S%=78Te00BX>UfDBuH9`n!;#-sO4IYeYf(EeHVvwNrFDm!F;STs0K}X%$ zUj3Nju9_$$o*GuJJ96>?T5;ttzCyTrbcQxn8%J{AyPV#by;6UjF0PR8lwCHsjD;tC z%72&!7BZ=Q>hYnZ6Wtl(DrY?`>Wrd(!I<%gZ+`9jj)Fz)X#j<|i`AKbe;kc_ySq-2 zfJcchmR)#4>RbDtx_YStNE2_z7Gy~a0LMp_^#L=N&keIHwiDIa)$`Xg|9Yj@pJGIM zf_H06@}hdnRZ1%$?3BZw^u_-;W9F0oRJ!?$WqY31T&%CKuO8*ZCW<7Go|sgC^|T&KSl&k&fCLsD*QH`Wo8qrB$RsR9@9li!TVMY7 z@0AF5N(9l#?6c^t5C)>jap@?1Obcp|>xC-nWoiPZLyCK2HX}=eJhAbim(S6F)SY@Q zXU-z!ON%GCa4R49kuEN@K1Zk<`ohK(Hf>UM zu4)33A&}9nT2x|bO7n?x@VWS1xy8i5q_F>fiZN}}afOB(i+k%xA06MWOzZjl0|7IB z=P7sn%|%A;;=IgiU4bnlA;W|weccXL?|diuaWP)^TZ3G}$r7--u^3@m`=(w2piRIq zp-!_xaa!mp-w=f~NwPy_>0_>Tt&7=&u!!a6>6 z<;f-6=bPLW8ZX>QIE%3R*ZEy2DhmotbU6qM#a2kR2JMVOzb80&sT@m@VLi6A$!kSa z{b|KLU32FQJ;QfDy^*@({-xm&*O{?onYZ;?uDyt<>W(b> zTE!GcqL1-_%}oUA1RWJ0J9y($a}~Kh!vg$kx#b}Q;h#SlQS8JdPhHg>BqizBA;l4@ z)jxG0ro2zoJ8F*L;KDQ5%Frk3#wT9DuCrsZNgdZZRRsht);6Go5lLnk4qIVBs-wMo=)!M2PFVO=d+sk1h(rh6+*KgcWrG56j-5Aao891rLNOM;u}6BY2P#gO=o|377bU348;_##~K32YzQ7y zWl&Kr^d;BYkVk*V0Bq;cCuy%$kxTfMsI;!1s( z$Qv@?@mRsFSTAQs;eD@Rx>58?*;tAEa|&FT7i}qv5ZFxHUsec@i287Aj7o|w_BX~T zlvF~s%yd94?Um3Y?s>Mb=geCM2`>U3QG08WVUz0y92EZZK~n<$lnfsizS>XnU$DlW z0FTPCo5S~J%QprpIYYz)l~|LFVT}Y9$LMv&u|MCv{}xOSj}Z^F%D!JP{q_z0;t8_IWQEbTWSAl@h5RycUjy=q-()_aX^qV)Jw#ddQPbtwxD?cq0W34xpCjNiU68n1zy~(x%#rqB zGAJnwl9?Gu+KB8hApZkYGj(ZwqXVlbk7UL|lJ0RNj8%RVrYJFM8@y|ZPLIyS{=8{d z{Br<^jpFw3c)}CaLHPTUK2tdgIllBDfoC@;D5rZ9MQSp->(WsSbKmVVq|J_N_gkxp z$U;eLjI%?50i-Q*ltbS>`()MIM-0}eg+{OJ?x2_3wksi%p}R^OnmvOshsfKos{Cf{ z$3?SMjzQL;`kir(vF@{P^Z<@$={gg9Ezc{RP>q3b*%hgJlig|0Jc7vlDqG|HYDq3H zDqPH5o#fhc&p;#UGo6j@AI2BUYZdbg8|t|aB`l{mMz{;eBvfmC{WGW;j&i+64dwnY z{`3NGZgHER?=^+9_qB9N5a^HGbKPQ2pPoUWJ-pdcXl}0@+YSCU%G5~P4Sg)Jh6HL& z8IS|dk<9FNIimL6PpSqn(kf8KJ!}s7Jd8e%3{yd?=*Y+UGA_|N;n`d)>F(tSTT+hK zY`uMwi!Uzc`TXt#ViVu?8FmrfV{|;cF=}<@pQ~BP9^p*QzzUmPk_}w~hJfNxd>{5r zCcgc(?!GuZIlu?8v-+Qn~yQ%XqpZUffz-aQ6Rf3*+lg(rAdqMQB1gOo%P_ZoS4`S~@XNY9AiC-+#w zz5$Qst)j<}ecm>_MZ?pKLoBT{y$VnuG600lG-A`9`|_Bv6`Vi!5DOJMud%*n_ufnC z|DQis_9W)Ug-bMMBzc*AGIuz%IVFQl`PMaphDK9rD1Syg0qri-iA(Ww&xM32egk_N%w!X)xI7Jw##NgvX@go%N>c4a!0kw;8 z5b|4)osJC5)$vbUz!txy#X{yuD@{HBpQr#*m@}eJ!_XrlFAqvh?|oEaXGk#3b`Iy`EN_Y!T-|H(cR;@?fo8KzqJVa{CId)fAY@rQBeiL0yUOj~hZY;4oujp&Fk9OtMj`ajzLb-l#teE}ys>9av)m$vPt z6>-^TT&Jye-m|lI;vhf0!rd&R!^z$Q2#L680cd$bSj0j~=%uKzvNE$mq~h))3M zY2;9|igWOF0?FeC(owAa*_iwH6W)uX57*Dqz%!Ylze)?JTL^Qnowi+KkQoAjd<@RBiiD@pM@s`3zjC2ru1_wdbVcs zvKC86qs;yS_1ms|wwh-Vax7DC6}g6Y#=zm8;hS374MWjLrp?B3OfS;eMY*z(^>*%u zMJsrP;_k9~zXZ9&NSi1@n5>#wM~=%_n!q{Q;7#=H%#iGrdr{CtFq@&l>h|_bfhIRY z^AZ;PuekKU^ZE&Z&9Nq{{uJsH>823iC;6uY8jN?s)8Hx33q_%$Y|9BdbUVQFi_Nir zcRoF})te5B=h%uEc(9rE?U%K{zw|G41gqmYRGZ)Uubh4R)sVz6?+EkxYiuUF)HnCa z)5N8{e6zBNoI^h>=9PB40b{^t#u7SlcNXPUgKEW}-W|Id#vh9HYRH~B9d3-k8=DF~ zZJsf2ebj1TxB`W`MgsX{C*@vVTsy$dWOJprZj|O8%<=n8CFK7KBKdznTH02K$v{Z< z#oW%Pvy&K%$liFZw)zM2I?H*THmt1~cj?|__PV2KZy7*NbT4}nN>;ZYeAgIm3ojDR z$G)sc?~b{ERc&+i>Z#T2!{SZZ?>BkSf^Nl;PAnFHdSv~g>7+nu2y-}m`pf??n6GGP z3{!)gF23eG9yay%i_zjKqZ=Be&TQ1YDpe^rJ8Lc$OzsR}^T0$civ zDlaN$3ZSN^O6vsN;4q&vVsV&4^}4tHZyyQHocS*$X+XV#wX$ ze%9x*lrK(3^Tm>UF5Ea%N*z|qwbWs+1RPr1*_Lidhc!=27qN?bjA$(Sr7n&dguwrR z_lGaS==JMsEZ=Q@3ZIhZUq+_EjWpy~R{C%piQrqYXJeTFJ=AqQ>&}_@FsasU&n!+q z$PBTKtU=@!0vs21>y|mXcW{YdKT&qj%W^q4vnj}B(af4qh|$-}c>kSXDVFuE6r#td zLai5RT)JMk?R&1ogRSTYc;|42LB|eLB(t~ZAxm;`#n}bbalNU7xG@bBEAr`@nOud5 z)AicgC>6+=_wnjZS7fUVI}vVxXx};C9+cd9an@|sce-;3@ez?qaXG$2Z`U5wcryhL zcDod^mBzLYQ{U9eK^jM4S!|B)r8ML*S+?1YEkIX#5sSmX*FF`oH@R%2`6xlHjbX<7 zzCZP+$yKhX&Re_dO(J6sm2u2`45!VB^g-@<8_HQ;@dM&T;?4rBa3~Hv{5&`OMoSe^cZz}#1XfScrS7ww_J6*&a>4EA#OOtfpXVa;Y4W1V-6e5d?;7q~!ED{O?*)rs}wancJEc~c49@yjD>Qo~&u55oJL2qx4Zp6&Q* zByX(B@KU>S1vUThJ`fw_0;Q$DsQ-7g_Em0st0c@8x6D3VS1YyrO8)TP`N6Ed|FyHZ zy~SZDi4zMy&5z(OAKw(2j;!>7HmnO6+*(9NRbFb9CsIM7f1Q(bWuurn;*KruvRR1@ zIqYn*3Cm!=sF~`lV$r1V@tW-5255|G;Vyoa^)<&*Wf&}%{OYNbVtpeJzCHficc$i8 zmqgz)4}8GM6m}k)=)R83Y=fuJ#Jdif8iKDk$*)#=a6(u}NUX9e5gHB&9U&?zN2Dw} zHw{0oh#NeFC>HM%KI6l_l&N}n4(_J^PDd(3j+$KrRFGTAdUiVgZORuDkEY$zc6S)r z5!${lYc>57(M_@^gYrSy>Qwq(oI&7yLCx$+&hXxB?;%AouiB|h=b`&t!lm$Q{`=|m zTJolfXX+hMMUn(+uvAkHsueI$avjukc^!_|mhvv8{O{{0p<_$Y!75A6d|CtyOfqo+ zm&4R)1!+Ini}75V%LnpEn0sjqz6AMgdu6C)2{t9OGd>_d4bC^|9Z_GE^WZCa36!BAgXGzUW5NYqol-guq8;G?Gp za3D9OHWWTSDd?$YD`GtGGw+*ON+EOCJ#hm$S9I0<(ki73b>-J7yB&0nD*F87Wb|gJ`%j54JaA2f2zs6{9i?wS~)x3`J$+Y#`YOPf}SB{6Pqo;cEZ(8jH zcNd*IEN2Trx|$R#jdEPE1s;}I{Y;Ux(s{OJw(`hpqR!)DbjIt*qf(cK+kI1$XA@WX zaiVI9I|l06=%lgLtvl;G1=QOXd^Izx)4M0S z7Z5$`s^fe5ugY(4PNlFfvxiiu^EX)|v2!xnKCRqUHtMxbfwCI$y?0Z!Ft#LD0f=1l z;mY0|kU~1^U^Qqa1pi8Rk>YioarSE49QFZ<;WlNv{fMG4)M>Z)XP>oIP}{3@wp#Uh z%AWAo7XKKix-B6^#x8XM&L88cvv6m#{XQSR2H~DUJ!F6BPrRYj8Z?sr%5z|D03E;R zE>V0>ua1KHx0gc)dzVI9@7g;D=7a9YU(m%$L&5_QuT%M7v!TcqzRo7G+3#oUf4UlZ z-hX8wp{;i@+dZWoeX%|$-bKt}HipO<@Nn5MnC+Z{n?np?GaC<9y+`%=h0E4PTbYLl zCrCX#H)D{C{8HL6>*9aF6PjiEaP78sZ6Tq^sq6F8@C?{an{N6VUl&1f1nYW@Z90Au z1j_haA`}+nwLFA9C`p66=VeP^du`Ka7^cmj=*!EBY`6vigUhFy?>qVi>;XCB!Op9w zf-md=q)v5SX%B79Sy|r&|C0J2EWrFr-1mmhZb0&c!Q%; zsE8X_dnQ*PflhOBttfTnn%&u)kvV3unLgZ{LK~`c^?LgW+&dqy`55`)Q|P0^#Jx1Y zM65j9z*(E$0}Gr(@eUak5n)8Z_cwVi@oDEa?Nl$bj2*34c_&JbZbaRC7Xf_AjTqIi z9I0I^^~h&knlC1UsV-J-P5}_FCa6vvq5_~#Vr}4KA7P-j9Y`VM7Qnh!Tw(8w2e4T} zO{?o>AI;tKteAS?MPsVx-fMmt<-pI4Bz;MH$5EvNB^w+b6r&BBnLEfjpQ-1jCLc2r zw`9#RP9dF^3pn$##OLPrls|H7%g~7|D6B62frM@`%;Mmpov(#W-610UQ!cpD)YjzC z`#jbZo0#k!V3?S8(w60Xk>p&G&s0e~Llcgy?y~5akQuSI3?x0{u^!Xs6Lwo4?UW&( zx0=r36u)$UnJfh`2%jE=D3prR!Yv1WX`r%9HpMyi@8=9y02<_q%aJs~h3U%DKOTzm zN)RwCm!SS)`xNttJJ*SdES-i1u<|UgOx*KN&oyAcoXX21!yYx^X5+pF!YrDIi=+oB7bJI2)njIzPePh zK`G#K$5;`FokfNqx;`gIYsyXC1@q&d6cNh>{xM6}O#I zW?Q<1$-PppsnU4Dg)xw?JJ7D)2ERF=q;d4+$XG)k~w=6gg#uh7o6eF)IR!Sjd|qc#&vYrDb0g}!1(EPHVzuk zIVcp>NGQe)FoE#5K|XwtM8x@hUNhZtj(l|q?EL~-^t5!@{6Z%si}RSddDP~Vn9ex+ z*jUl!?%w3Xv0-o98VkPq0u=FR_6vW=AjFXF})G1ZbU0DY{csk`Z z-%tc5!Bn|Lii<-Ro`(g{B)vSkxcsh+&5g@cn_pHeO;>=3f{$r5zyL*O1y|n^U1$6u z`V6LkyEa#jYvY`ejTuh6fO5mj+a5+v^0ilwk2t9|uiDO=IK3{)y&R5hqq|}pLWyd~W3+tN zG!gvZQl{okVN|mat6>ryC+%2~kehsaF#=xM{<@x~jwdh53wbSXZUcpO7 zhqb%hFWk>25&s(a_$CVbe3_;yP?xE2|J}uR%RpMO2vp{FppE`qRkn@fSfn^p`=Iwx zYx}s54I>~Ss*VOZ7H1wxcs|ptLkKd1K|4UH8`y2j<$$~ zj{fxEU9T?a+=c+!qD;qv6xgDXh9X0ZS11#*Jd!YV zYpgNQSw<;bugnG)_c)O#1S-p|8mtKpgWg}&5S@uvvrTxOpiR&o++AbZ)O2q)-`y@M za{elh2ppils#EW(Ceh)nsy=SI&4j(DjKzxF(T=KaCtkU|DrN&J>wxn)OyOlGojqZYC0nsU&;o6Q{o!if^I2Sj@WngmQ*<;v3^oI}j zhJd40DfKN=h*Yk27kEcGG4SRt9Ttsz3vfC9m4Xmzs_Z=7Xtg`ara#_vxJ{TjIbJ{} zr8MotrS<#E-$t_yA3^ZgEMms~;$onI`Ms>0I<8CXJ313ZKTQeyd(3$fsu?oTDoUdj zQ|kE!8rBbvkKrh$BDuT5iciOyg}pbI*6|m53UM0VPqgzIMcc}PMbY*a_bSn6vZ}zU z0!?cjk6TM!_hC{~CnGMczm5w?SQ1rlaE7>BiB1kq5!gI3y{I?8(B;+EjJvn$$W`0E zbF|Cu-Mcz#Umka|!35quB6aYSSAIlB8d1`z@(=zuapjNCz7c&U{haf!{QC)zy{sH7 zr|o(MTw!N7Tw=Z5a$VZaQ|#)il+zA^CG8~(D#h8!M0KbnQI^W}()--MYp|f!2Wriv zzlA=m)c%CSnKXQOrQoOOIa0KkPCoft4F2J8Ws2{X>m6RF(U3cBbcONp(L2r7W(>JG zSwGMJ+`v&ktL1YUTMWT8HA!NVJ=ccz>38d5;i3jkBz9W`qb)^-wJ$debjD`RtQR9W z7_4dwK(QM*(Zdsk%2MU&dnwV4HERsE*no4bi#G0n!{P&f15ea;b%6SfIs0P4eQ2xW z4SMls$V~{R49paxh5BKq^7NSIT6+DYWLYh+-ISDc(Z{8i{816kD!Vxw0i=yNGrFZ` zUinKgJoUXER{g(}cZ&rJTHjt26J<9|T-?yy2$$MHc~vmEF5Lo#!}(ibwKE^&TeT*&XCz171JlKPu;2%R$XDRC`7J0~-9HZS(twj5?p7ng_F zBVhOSi-)>-o3)AzWPh6j?*33mGu&(qoy7i~S|~8yv9u<^?q;6OXH++BHs9Uq+;N{< zKFTW%CQhq)L`NB)5Vei8>K-pfY7@GbanTWb%i<_|w;uD!FMc;w7k!zu%ZYT`E>^6o zoO)TB>RVK>8rUc7uCoE~*l44NSS`xjB*x~9VujT=?d*RJ|2h7xbXAuak35VXV!XVt z^Nt^7)X?kW+D^w!Z61fPkl=)vPHkeIh!K)5Lt}0kP@P@08-*$6v`59dC&jODOnI#h z=Y%}mQQ9oHbA|xG%qSu4wG|5^^M=VRjW--{kfoX%a`Qd-d*YE|I&0e4N^y1Z+SVgH z?xA&!_H4bblhyu4lo*`qI6o&&mALz=zNjO6huaSpCuH_-HT;?qhJc7b@8grvo?Mn9 z(WCA}#t)YWQ2_?N;pvor@3N}<^-ZHNLApBWRY}Rz#kl)BXI+!Fz4htxuaavE+y>u9V;4N|I`s@(aBW8tl%2emNUmIe$P22h zxuq1&fR=|IvPf!bDY+1~`6f?uH}^IDMULclP*C`aMkb%iiHmgPT&$)CEhHy#vu$hhOccns{sxF#>HmIz^Dr}$S2I1H=j~fF1||U|_YTVs9ld|Cr3OVMxm7lt zuzYSGb+%{8m7=aed7LUieD2|EtQ#O_`ajuv6-_lTQ@v#tQM%Hhvl?^f@G`ps!$Hup zpvuU_N~N9Yt1up`#?A@2xu1o`5rk(7oLpgj_?4cx(C2TsAtSq7c79H0*OVM5q3w8f zfwm+K7THUl${5hu)Rj_C_eJP_PYn+tRwI3R{B$B@^~Kl|duQG)Y$+%B8NzFUeb%sh zf4zRs!E|_pD4sEsyYMN6!*V=d+e5<$(bc0yfqLPj`DydX(f$NoO>Vt9GKxBDNg65r zqqJRa(T2e8E)A>OR&kdu`lqN}YQ1@NeseGS^ z8GVXTgxT>re2E?`9LN70chp;nW5MVB!BPtO%8pQOtd+O3n{i^3JJ=B^3)Zb`up8M! zuO>Yuc8Y?8m8LLR(4E_^uP@b>?Mz$mjOt(6V6rONRjbUPE^?FVd{Vyrxl)v|-l)^~ zJJh4Q2Mgct%2gUj#|26w9Bo$9*a2zd1p_9TW0IDEt}_E^@;36m_#FR>y|;{t>uL6d ziIb23ApwF13&DfCgy0(7A$agG$N+;SxCeI#?hs@kFu(+N*TI9kGdOcM%M2b>%8xW+n=Ck@0P0C)!kLqzq zj8-T8_Oq{!kp-Jt5%$OJajr-bUY9>(11g&3rCi3HerGdhdk;RMl?wol;0J{}c!t?k zY_o0snICvgM3Ryn)bZ#Z(X@-uDlRW=8B=xhIK{&tyWzG-+l5pomy-gq3c$(atV~ww z#Z<~DNZeogJ*I3(Uqv4DgaCei8Yq_*x;;A3trmmAqR%IBn|gZjsW*bMTVkYw^VcIZ z+%99$+b>Rq$`blvIFb34`Rm8|CciD0r!itu>Wi^!M_{zRhSkZ5hL?p~bSEp6DBGDp z?J@(2YK9A|^KSP<)lt9QYb3P&!oxMTa3UtZTyunpXKeebO}&0jBborbeYqSkaSGE^ zliyLjB4j@TSI|XK-Uyp;_Nc&G=FeXOWnkvD`f8(kb3vQpt{MzN)T9TY-NpeT7_l(B zyavSBt=Xi)B=v^i$Q;{zuRL#4VxO18B7a)4ISsy+|=RRsAaUzai>(OGb)Q{am9j3>cuZbyL#D&O>0hB0qn(6ol--EU_6KzA(t!I^)wa+J zJbQIZb?TpD{kagXLZ!8Siq$M?n(QnSW8?D zgQX@Si%Ro^^H|2pqua%bPA)?aXD_Ul!;HpvjMttp#tyL4x_jPk8J0gRA0B~GSHqyb z%b2MXQmoK?LRPoNWS7ig)1nCKn5c@(+0hXsqnXW|zUOsa>n1X}&gs}KI;H|#&*Yv) z$g>_wPEIO27h8<;z?sKsaUzxZk~miRT|&3{__)Fj%K&mONAQt^gR6mcrP}T|F_l#D zO8@P8fJe*5axP!cm$OfwR_b!xSPU9j7!XbA;u@IaJb73skI`&@;K`^AvSI&>Ja}J0 zUs(~J6M6r1%LpTf5vx1GH!FW=;Y?jZ5YaRz7lY&CbYdIqIZ06Ee!hl}sw)@NqL&#{ z{RTr?&2@}7>oHnF%O@L0y9;YQzieWLKlpyTXZwA3Sx0Mt%^1l8M_oa7hO1nJ?v=o5 z?6%VxV$A&e+)`#PrLpJjmx;+$M$efMFbm_~S8%(%t}H|BB6LD5vESY@%U4VHf)Ar*%x%^55jqJIkaFYsK)I3N3ut zOLNYdb)2jL_}}iO8^#N}SZy@ybq6w)mgv^Ipw2p8b~aY1Eh6h59X;MRJ+3o(+lX^s z)We{O9+MfBnQ9K{M6_z+h)f_5tQWU4d}5+r(LBJ#_ml(7^+RDlZeWf^>Kvz*Dqp(c z?^)y{f0oUS)T6!p2hqms?mu4D|G7=Qh13EDb4PlQ4OHX61$S?{%FPfuF6ovYe?tCr zItfxVzYakqjeSoGycL=sS<=MwCb~|p(p8PH(3wK1?xEPDFSp$YN5B2ecgk)vGsy(( zcncO6_x^oPtNtiV3ptku+Pck1fcDNOc;Z(6fBXwV=l^7s|G@6N{RgWD?ZH3DJpT_G zL67*Kn^eTK?pKc+fUg@O2^;O)(iLro$AfEm_q2s5A%T;QcA_yIo(s#3iK*dxAoCX- z3+*tttBcTae**{~yX2CuD{w7U>#epJ``$BJd;wxrf~sM1ZX5^%F;UZw_O<3GhiB+8 zq~B7HEx5aH$6>)L{fNm&a(E-h$FJUzvFE9K7Z0yMhxE39tITQN$;Idi2vlBCWmkfC zbZQwK{-#R=-_y9$hbHN?7BMwha{+a8TJam`X@U&2k(`-t{J=aegiZ4Ox_Ah6Irj_( z&vkT+zlO{$35QOaZIu*lZ4%`i+pH~XIJWqWJaG$=ZoUP;MvF%r4eZgculwy>F;ESojA*^8@Beu)eY_$c)+MGWd9Fa$@DRW`lcFSW$la{$mHCCJW#3LQZp?ph; z>(ejuXCb=ng63v%)ql=*9}2N}G>2NHSDh5C7YEjZKF|1mAc`$btVf;@Y|k7r(+-H) zzG8EoP%s#qz4PpHCc%U=Wb%giUOd^Kq20Fagyyd?_P2}Io@^?3)MmYSM3E?abs^nS z<9(r^!*??EP6*0muzC^Y(M7!192j6JaYEAIU@|^}=R~nNR0`kfuZ8QL|H1N?oyIoM z_3X&FQj}szAGe`yOVK`d4H!=GoF(GEKDivPiHoX|^>o*sLGU##*ysw@&Pz$y^UM4G zl_oD95_nQ#`Y?fQ1v5{LA+K`0@wma*0TxVs90*s6+4bV`8y!baFX7lo+7*Au1JrdD zZ=>2Ap%D)I)fqsR(eI06o(IPXvi|a9)*8Ge&^U2)@)4$I51CFEqjs1;N0ZFzenQT5 zpswB`QvGYz?80z4Um@P?8-%?ZQnZ?wC0V^lGO3lT8MuGErgr$=Mn(NFzXXqNTwbBd zz5}KRD)_TV_gFKx9=}@|HB-r^=(Kg+6)^dr0a|?EZlhraiEuRR|LT?|k;k7#Jk&k2 z6}PT@x8X0^@ye2o30+)gQ?qdrhysC_TJ0Z;1oRxp!H(%mwsTB0Je2eDvK839VO_az zOrV~~6K+zJc8K%^p$88=8Hd0=J{rMT$~_43{hM%LWu#|tP>b4R-PoIvYH$)ktp1Zt zmX*_>1eWBWPU9ezJLV3Sw!J5(WpVB;Q3(3t@X6Y_?Bnenx_31Ap|0QK;`TW+3nA*V zW?kR98aoxK+20R`n|8aJq+R@Ux$ZR1BzrV}=kKXH9zpXJYJniE^P7s_nAO$Mr7IW8 z&MZ-79#9(+_wzG(*+73dmXAVSLe{h*|13Xz7xH(ErYqLg^)T};h#g|xfI0#tHY2z&I76+Q=rs>G>7skf-hFQ7a5%VEy~m^hx2NkopZJ@g@sD0Tv@d!9m`bmNTf}JwT^mjSj>v)x)Ti_1Ggk+K(YuBJcN-VBopB34r2)Ns=RvSTdYB&m!lld}C zQ|AvJag^6PY)nQ72@*RC8jY7JOd?_it3>eY4SLoLgnTt2kJ$IX>M@;y^hE#klk9+4 zKq`8l01>B}(}F6BDN?+4uQNJF;H9F;e7{gFP~R+e(sL#)56l{BD$6%TyvpMLBK`T80{%k=%xkb`hu-k zcHC&yzw2VeaqiH+bg@ZptXxug`s0nk;uY^7c_k~#x{N9%DXCVg-@%znD$VVxIOXz% z^cB{ssS23u%Ssb!g3B`zN-Os_3E}#dV!zEgkC%A7g6bf+yi^O$`M#kca;g>)_YjzsMe*`Ruu3bAmSZgPKxcRKnK3-F4#`exU%2bkf z+rLz{wQWd<8JFv9nBCju^b}$jp0m9#*i)6w0~~?_aOu^alJKgavp|qr32bI~mPFl@ z+p{!a8?zlf4R&c`9BS4;oCMK8uYwkHUhU|=%GckLb4Ad!Wnv`3cV=4wp!cG!k7_nhDjmTz*zEPxz%9^>zi?oZ*<9-_< z72QE4eLvn;?2+^Ka%+VL<}gns>hof$Ez(?X3a*#*m8?WtAiv?w;)K~Qiqy<%TbQMq zv6q~V?4e=QKX2a*k*bt6x&!;~XuIM?1JT&Zmv(Z<=T^9cnUL4R=pcFCR&97gzbDxS{6d~MCI-qY5a zNc~%H`SJe{1(=V^(_)-lQ5VuioI9pwjcj|MgFjbh_EejAGrYf+;JT`r_HpMHJjWf^ zQHbp6Dt7;$&SL(bK5(t5AKkBA!?s!zZ>2R2d@|~TqnL5|Z(ZEd9BAH)LIX$&r&s@l zM_WxbU%S0{z9ao{9{y+Q^&OoC;zIhF8#bD-;f!F+s5(qAu9Uyc#%gJ0R#c1*z*(k~ z3k(XxS!PvwBpn@mOMh2(85N4?jx1>%ir*p{dpNGTO)2+NKDwwK(_PYfL(Op2rCM6& zq`r%V4q07%^hd|QxIj+X9?pTTFVCh?K7M{X6J^Z<_X?DA<%_kTPiN?TB&DQyocDf2 zJQN9NCx!I1wzf)oCvn;8)j98>V8)6EJY;ekQx^?DWjcjHLJ4+8VHm;?)ay23~&iHJ-QL)C+8C z7S++Yo&qmrtzOeqi(+#2y&u&k+}-$k?~#(SawvPAW`*fYtz#&AM_=D;g;_YzoXzwv z=@>e3!j`tSR9?rfQ+Xi2gt3Xq7T2qVOwn28`ip#fSFMT3pbN{5r}YMDa7S(qHlEgb zdhwAw#mx8K55;TKhu599Crz@xVrLjM`;8W;vJJf!6uds&Uu**pHMm2xwY4dq;n)PX zO!?YQ|A{UY9UG(gWp@oZWajH(9L!!@>|Ea$1E;D}*xK%jhss_uY?bK39C(e=_{K{0 z=i1xbCB3tV(lvn@-~UGW=yq%(p5BBdcaH0l z$lS&!xDvyNc_O2;P*ktu#4}k)=h@iU=4 zj8tAE4DX(u1zk`VEG%Vm{a$4r1ibEshK3=;S=LikZ3-{!K_FSY7C>-L%@=^_$nM@^4X^puwmi+ISy z!~_gLSxHG2FUxMR4fvaqlF}Y*_?clyV^c_u;WdBJg(qsoGaXl{J2SO&xaEUCW_ETq zKzuVZbpy(ZSr|x$Dqr@#Gr7+%7iAro)>_r2k)orzXVjhRwZTjvd!aaC-l)amod#~o z3OF3Tk48pBWC^0V;Tq}ui;ULd!Uy`FeVtu{&fm9%gLuwG-8QTvc74?(lQ^wI!h5ue z$Cixp6*K+zBX|S_ZyMQ0OM69WvjJ&cj{WSU;q%3gmT(@UhS2`psr(wia>eoW@rm{@ zMjjl{U3C!S;284%mJk(P+7{i_FgBvGSb=<7OB?;=p*tNk4LQ*>X5__tp!S%PVE*Pi z<1dZv>>ws4*6E4REDZ=2F)}eUj&<67@yCFIziv&}g<7d~jDv@jOweuOG};gNHh{6D zrrIo(O-xSW%qs?J0!lc&YzUp)5;{pizDmqPFJd9rdn;5p?}f!B*KT1C*+)zvAJRu7Jj*k^y(%+ykfhXhP< zPJQ$?bZ@5pwZCY4wm#w032|fY4UZcC?}!OrYK0@>S(x4 z!ZgZrnw22o>gq~vN7Q z6R&*oTtw%zT!XSgt&n5r_fp5!)|OS)>o)vdG-dcgCksdp`87iz3XwVL`Gtk%J7>eIZ*?#*8T;AyL1(PfZ^Y96SKV#K*?Q{+ag+ z-=vc-)cLaowD#zM-Ip(KSL6j>eflZtBv0}BweV!Uf=Zzc2ndNE(5GO|1$aNi|5_?i zS|6K_4>q*hGNw3z=Mwwi{`~+V#a0A=wP+0^V{iJFY5=Zm2Lx%#{XLyZKj-4S2 z$V376oObo)K)6)4QcS~ZBob-zE|KRkjN;GvY(<_kHO`5IX?+mleK$eE%=GjDtMy}` z`K1e1Nnv5(kid|lG~bXLDH0d_PG=K1{ZxGOH<2P})rrMVAK_9#*arWlLTtjydhdxz zBiYA%ccKS1`3{}A3VQUX1iHwY7pvvIyzjU>(xazhSFnSQ8_{Y+gO{JmRj4n_9Wr_N@ZLD;SBjDhE(*^Li38SLsGtqB2xc8`8X*~n! z>ntI;8iMV0c*?rV(Eq=}N647cC zPGwVrYdxGr=Zp#?m6xYm*R+nK)RRC))Saz|XC$FrxJ3nOA@<_H7Ib8g5H@+`wuC+kyzQf14?fN~vyjj^!WQ z>~1jAju4nn0)+W_uxGr)cq+CsLGWLOBQA`zuW*I(?J{MY@CxKb?e4^^6!mW$@Z1q z2Hl_u2LgSl1QGOg?g1~Oy_C4|^_g^<&WhOIB4wdJTg|3-U1@DY6^5EU0+xi;IKtv_7pWwSK{h~y=qlP2g+$Z7R? zM-R;;98LdWQ**ODsscDsY)@4uRkH!l{H82%J>CGsSNfk-SNKl+SLS=oe$OhZs_;Zk z&(F_KPvc>HZ+!gy;m2dzA^}y_Q}T)IozpXAqWH{u$$*;%J43tLrnKR_pay0XD4)#D z0^ZRf7X`W&4|!#uWhy^jt(XnY&CE=5+I}R)gDFj$D!twsMZ_ zkTD=xiRTCF!+2P`0+lJkfUdYMCK*o zO`-GoJdzhtU0r>{s}h|W98yxIn=^5KZf@>P6_d^{YytvWAN0qM?dF zmWkwB;C+X9`PA6`z2$HKLlbPw7c_>Zd*wL4t;Nsbzn;qetdUEuG*THhn`!g!( ziB?0`hbew07S3zaC3 z9symZR)89x^w{RN3QGQwTRMF~6w7?Iyo=E~DYg1R|zy;D>pN$J`fE!l!+e3gEl;yfnz>3BL- z^~o*myBA$w80obfufWMlfBOr%EjoCt4#iAwY3<}1C=qJ1Az%IQLfIv#1y+fFsm zWvq#%?X+As`;njG!=gkXp|_HhW@#E%7h+fbgJM=X;I}ekBM@YbshWxX$C;FEdx=mt z^K&u-YFA^#syfqLT6|<`@h&6n_R^}mh^y?i(wB)BMTo)Oy zR+)Kn$BW^~l$zC~ry-sM&*CQcplo`CJQjUb+)=r*1ZH@G#~P)b`Y0E6iebcfX`MpTxTLaWne4T1M)jRb|9B_@Hafrl-9)N}TIoPeP0Wyn+C{x-W2{V+5d~ z=D9(xZ2N%Fe49**zxFvHUY;g}LXJs^F!e;XxKI8&L^; z9k!Bu-J!2MnSvAgn8SOqR}7Ix9-i)}_%zB^8_%wgA6vHfy45prY{@ha41a!Ti!sU6 z*TYepRiFcMzE}z^A+#)buHdRf6vrgYt3Bt(aUiNGr+aB7`M|{r$9glfzN^#bKXOIE z?jimYRB^}+uEiXi@E8lsQ^VcR_@p>y-?kS0sY`h?1>K`q+7}*q-|uQV2R<3{^FVE6 z9S`Cv$GI`oiSqmzZ-Pd;qq3`&pVSZ?+6yJ&?Cu(9I0rgB?-myj+Wi7LrV%=$VPN9r zr(oI{stsf$X6<7pJ>22+#)$8Veh??Ax;WbV*f))z_NGYXu7I7?5FDtvk=!m+TqENJ z`|H(_OGP%Pwt}m)E5qI1EIuwu!^n(q27=k&yz2wgiey{NgOwd0D%Nm1tJT9vSa&}~ zDhEFOTrn8Z)8p2Vm|9NSjtr_KG zxn^*f8Gb$6Wp`gye9PA75 zWnNNT!76|y_!U{Xp&!RxHh-*tEh5$g1u8j(94liuZtwF&9h{tw6y>OifJiFebk4_` zbVbT!srNx`AvJ}>TAmK9%?^v@pBi)TAZTW-v!2t1H^OO_;5^H_>K1uElch~PJK_=u$R ztiZRRE8tn@iES;+(5v{P3%r)`@S2f_<&mfJW)1SVux3JQire);s|zNX@6Wmg_YJgd`*m!Pg!z~E>WZG}kF?6;B%Ui{;d>+v_<15))gp_u zJlyuL`Gk>W1n&tAX3PzoXSKyPV&ur+(}f9X{@&1-6q)0PDSkaVj!b9o%@3_JLN!LR zf4}t2nT`I?Hk-UtLZG*($HSjuO;vwK&~X?;uOOolkRmY&ODPcHUT|uFt}EUJL)XO7 zH`r`)Nqxf)Z|L`}heeWOa1(-V!81Lh(C=20y>P8lD$a|go=>6QiBnGnH$3)Ps~eP5 zsjbrV5ru8G+-DkU#9qqk+lasUcri(kw?w8}xqeqa=YN?&FgEE^7^N7sZxGK{h z-c(kuZeZBFK9hZW{07}Un$OG4hBDC#ZJj*U#HyIZri>|LBLz+O>Zs%1UUXER6B-YFz`|`|CHRh+qRp;_IqX?!y8-Ls(Cut#)^vB-Jq}_}Wf|Eo-$o-E9 zzaCz-_xrk4OT%8%Ll%T^aDng(>ov3OB41;nwG)RA+OaG$j1f!+n>L)0#;B3cPDgGYG<7q>Co-$b zvSSi+*pOnaZ8t^y>QL=tjpv0NdisI8t0WDCcD0MFF#Kg5e+(07xb|K`(?;3X{t=du zs1ZssN6(_Uk=^RlKvng9tUQaDFY9x-yI3~kX4eD)+riDShC#ZH>%%5VjV*{#=;o;A`dRcpn z529)>@jh4M_^I(RY)-JoeY3G54rNb^sB#r%#G2W@x$A>efvv`EX#XiW_1 zCf{cg)_a(#=aFywvm)Iwpy0cmQ0e7%+=;8p%g^LA65?+0h0Q5EwCHpvGWQRMS|K18zmcrb!U!t}KviS_PK(Zy|_sheviMN8$7I z(x7K8$xa}M&CGjVwE<&%qlV8GB~zSRi5 z_hLCw8$q00mi(TUYAdnh7!^@a9;zl0u#Ed8t&9OGJl(}vhYL(RH@ww4zsm|De)u@JZO>K9!J%UYtUKv+Y z7h~O{g&j_ja(4?C0w&{r31Dyhs4`vO$?iKH_`d*1p+Z znqN}^n(!3)ym?v|3`WJKgWN4DgDEp}w~d%yjM-Py6<0>&#k@kt3rExt8?|ZRnlA8^ zV&_!*c_DDMrZ}{4o+hF-_2HV0tuyU!EhU#qQq=IQWf^V$zJb45-3&sTF~3l#f}w7! z;p9`KNvtGMEUdERM8OxI#*{2oNrBSG5!PR#`Y8s@*XZGF#tba$^EtfmP=EA|J!(=e zLd!Aq+MAa!@zsDhF(2uT4iIt?u`Hd&^02l%PPy)rnc;<&W(1e+1@p%|>#XSio>~&g z={B=*g(a9jI&*Hn{Z-ar*lFHN%P+MbVHTd;^SxNBfz?qM(1$+Qbk*PY*=SDIhsyHZ zrEB!RDWH>!ii1G03@Rl%CBCk-Jg}MqDorrgx=oFPs8l}%&-&gpzS)jc%yil-lt_J0 z;wQB;A--S5#nJAry}S6NJrMK$QhiTB3D7pnniTkhy&hPB6c){BRofs~4^!A-h*ll# zv+XAyub1%`mNT?KFp_tin~7hnG&&Z*)GhZn3Vg(w-RfOBg+H*_EjNCXp=!5HIylG( zmKP8@v$2{9N?aalR+7nj2FEMGMdUWTI_om?Oi>v51>rFkOPQ&$8k<)u7Qe<*BklHh zTGoTJjr0vq74BuxDKK7xo0Ax!{`mD1f&<-yjWeaA-xZKq>>k#~o^Z@k&95|5 zv6>XBp-$WPI6!Tw+4LpLWR6R}di9Iz5-D#2Ale+aycQ^aYga+&H&~G+T4Z*fTRr?g zZIrXrw)sAN;?w>xm7lDuI}%A;eK-ysKk)EYSw6f}pF$yy$3vV&p;(fTZa(^Q8#c%063;46J^*}4Q!{dS znAMn^ir#XwiH4uf(vqHh>8SGPz~w}#qv9>4+tjp~mV*aaA;WP7AC5A04hDD zP1_TEQu&%%G+58?Z1Mh(K&JEH`lo{t!D|C!lP`rbC92o8akd23&f#E$3 z6^z%rf%g?I87>|n=?gNKd00f~78?-yw(KhwN598)!cZ;>aGc<HoKCNNy7Sh~Jh(#eq)e>I-kdKG%f(fxI4##WMxhIX}S z*HV5tm1B<24CxH|h=t!gg%A=nb)e+qu~^(B_jp#zk2>EfRomt#QN<){nczP=jtS6a zkpdgZUkNUk^<2%>kGaotsnZ`lwhE*k5-TAVLdJCz^fO+%#+M@IdI3f?rB4;fdD_mn zUk(J7F$#dx-)g8^A9qDMk~xB2(q1%NB0p6H3Ph6!Gek(ITohUfe(;<`UY*h;p4ITx zj#(YFWSJzbI5-Tg| zaWjLf_?$)+Jw5pPFub0isjOSrI}Qni^BRy`uEgFeoFkepuSK?fbqp4AHEM|pKU2*| zxVGfNY*Yto5{iR^p;Sv*Elq5>ASJ^{`B;@cCi& h1g>3f;?e=0{w%(-$ewxaHr1 zY-(gehvP8adU`jyPg{1HcVTI_x4cjohoh z;SSx$BpDo38=B>J{z*X-&-hdH1mykZOUJJUy)*w$a`OWbDOTCZmXMPfw@QGclkhn= zPr9j+#ct;N37bUpc+Q01FIU^swiMJJG$0rIjkH~7=|d)q*JWhX>TrwX`OaD;EDh-n zdb;n*Q2BgRg@76xhNl`lEHz4%P>KY9z@Y9vI}E7^bmf^;Q0It*BMH1r*CW8-qZ=)H_ZnQk{F5DxUAYP zTJa^_5SL+kDS7LL&Vq|Q$H|D}7OE5?P3K)Y|WYIrcr;@#>aTD2Pv3zlK^v0Qj z!sLV{9TVrz;7+lBxs)t|#G~9x_B;&JLP>7FJ7nfVGpEFiXQcD`d@QcaEc&(uvm6g> zzy9mhN=k=yfS-(ZrNeh@)clTnXgA`r*+(>$J1JH^F{)yWghU`n1zat^K4v*qT75rm zKmiG$Mp`1ioVdb*6rpC`n4@-6)hYB8*T}GdLb^WFy}a4B(V`AbhxjA0me#EHlRgO> z4M9(>j}S}2KOF@G*Nysmko1z?3IY+rjH0TwwXvC5mt;q0mw5;39`M?D z%WR+5^YI?N!xECQyo3~5N^(lZ#krIWrIwKT!fHY6kSm%H=RhMphaGld0Het;=Qqa= zTVAjM0=#7X@L}cmfExUMUE6^)%M_WxLqDFQl>M`|ju(S*OyVTx?yXzGG-osM&Z*o} zY9%>qUmxYszTB%VbrvPbXS%AoQkzBW>7KacUj#(xsp?&*nZ0RkZ=HigOHgrGA~>A8 zAZewj!;KUp3|`t+nv3!6&mBy0wQqL$O|ybkoNK|GUyqgq{?HtBvVR)!Dli-Bgt##a zGF=;q7W=L7kFKp1{|Yd9^EncJVq}{OH+AS9Y&@D zK)7`p8Xw={NM%Jn6r33>d&%N!V!sO7Vtg4A-_z5FKp<+ei=~>cJJ&UW0^s2J_T2{E z8V=q}ofG}pObG-<)Z`?`#3Fj|x>k_npFg8o(oM}5iF52MT+}==R3(9qF<8%wA-r)J zaIUB-u#9oy=V?9%;l(&jtMy59H08I;A(DA(Og7L&<_58y2ZC@s2Szc`2ee$wqWZsF zgdb3x~Nhw}gTz-JV)|?CZ z#SIJ9({^1VW3!%X&t*|(O-Y=MgpvsqKvg7zr^^k7k&9<}es>!dJ^(y@4`G zDBJQYsdp#g#yLFHUwf#cNwn((%Hf4umWt~(w(m|sbbFV!>xeGhB&wCD?9!mCrbT|* zkn2Sz>XP-AUM@o;qtYwx%CEM=qs1?uN3~eF+ajl;Op5~H=9l+iGqfrkeXfxISs7q&M{za7yf20CyxSoI&;i-z$xYCi%SUQP+xg>()?R@sw zE-#qywK-yN>f4JUOe>)xyKtS^8j=gry`Ep*;gEZCc0Y`0Dd@P%^T4@bBp=jg>D)tB zk~PkT5fr9;##VE2Y}t3cJVfr4Vo8gS=<2fAZVS9$;Q{@EjOxDWDUvyIN>~;()`KVbE6J8RPORaC&zu2!iBG8)kk(N&dZyj4>ib?`(u$}1 zduqEpke?6*MI_|Hk;QPsQmF6Zd?eFM-6TjU(*ynDjIcieygiE)!({05XsP@f%dk5C z%`O-@?T&OK5nS9@sTTK5q5GuvVUBp_d-xl=9Bo=4bzTfbpG`}M!C2A!NaS-{tGZ;p zOAfVMUBnS`08GJ_aCLg7oJ{1__IO76c|_J|WeufOiw{cfX)278Q~foC;MLINy7wGk zd~4#<1h$z){j?0TCw;`c5scDP$aTbkx_j4P*)ZA%R03OFbdxt!+0XM8fx4S|P&^n_ zTG|$NwZqA5$ziSbC0hYIkC``?f|clftd|vR`F# zKX7A$qJvY3wjEbsi%H*l5awEw zP_DLp)>A|zB{0jC)W?S`2BIy-rpTS`$y!df7&eMGwk#WQ&W^r?0d7rTf#nc8(-M!X zM-9i86#C}&r~Xs9zX;z^w9zt>=*;MB^(=)1aNDPyV5vB`aiEdNDDrF+lDT$DF1*{+ zXucvfA1HhISOK$t+p6n1i~MXyTjyv^LVn0|mVE2q_+}Lw+(^QqvX_Awdv8HG{s#LG zWs?xvL{hJw+nEaP{s~5MjV7FEOGz#4AqV4-%XDJn^7oz?jfL8&u~HZ+8ExV*T7_hX z1jJ8!?476|Y;d0@4nkk8wpQ0Cd~ zRl`^!`-7HjQQe&jx>VJa=XPU_ndd58z&G4EJ$p&H5w4JfJh3S0sDmt;iuer;f=@j? zPdA%(swdUPpa#6g`}jtDoKaNz+)i&+%xiMtqka)5bwd5(3?#djDUidF&ccM)6ba@@ zLPzI(+f2V-+@&8RBI2+23{rPU3|hT010^?j-Hh7_pMK;z(sY|2iEUkkRF@e5Fh9Ry z;^*>IclR;I(s<$KIwq7af31`Vt+OgIKL(yHX1S#1phN*%*U*yuR>R<+~d*OZ!ig6UE(9 z9uK<{SV}?ic-{3Huv%(F{oa)qv)1As>xZ&rXPeR|x;%5<4xSnGboXPPw0FCVfG>>7 z&i282yAwSd*me1yeTT_TJM2u4Wk=oPPwKLdjAS+8Bz4!*tlU5H-Q${6bH69*4lvP< zvr#CSzsl7qpcE!%U?HtXS?#4RbU}GOQhl#5nTQOn%P1ocLNj>lI|m|Z@Ld01Kt)44 zLVNNT7XS@S!0YZ!C=yVFv3~q@ucM(2+~ZcOTZ(UvSG?>T}^ zlaI#>A`92M)xWE__~6$#gGraLBBZyA-SA3 zKcs0-#s}JrnlP4T8tUvW;f(ZTN7RC6Ey(9GV{QekH!68 zp(lHrbKNsKbif^)^Y-S>5ySbZ7rY#XqM~PU)eLjbGg&YMJk3@QdhriqQjMnh=oQ zRH_uEw}20Z-lT>SKzc7yLJ8#^&im#3fHNQFnlF2PAs_l-O{IP%F}4`{);(E5*jp)F@y2{^>@#t4w;YwmkXOUHY-4_3pjFT-;g5niU@BtFuB(u?$!MNR~-<5M5y*74|dq^)ysZ_k^ zS5;ZKHdW2HPK5IV%~v1ns~`w#o@|X1{xbU>H}xO^q+(I!06W6X`lpfNO4S>L*}>WN zi!q64`+-v09xd4v^Ah~nK!G+}`8X18K6Cg>U4+1Ez10FG7uD^pPj(z{7D8DMzV_rG z*kynT!P8%$%9Q6s#(`q$7m+s7c87Ojb2Eu%98A96L32%4*A@l0i{!FNdysn9IUu9%5A;XC8IzwpQj3}vaCQu zzk)m8rQOs*u}XT~A&>TjB_vcPfZF|ws(Y2kw+1SL86oEp$-&_Ye0s9|6TA2hwobyO zkYm$^&>>wA(>lT_V&ixv|H(Bu6V*5f&n9`E!<wiLgH0dz~07Y;UiC;Gd>ym^(!NC*NzZxMyjNA?IlQtI9x1omjy6D{(>TqS_=F4 zkn>U3;>m0)=lVpTu&YYuQ_lLtQkfa6pf=-Mw}*dy^p3MoT->EUJMHYT4ksk`_xBC8 zkLh>mV#75xL^~p<_S?6D3W9BwHj90c_(V1)=11Ep)V>}&K)*MW$|`@dzC4%pzV;bN zc|w>DD-w zgSo1!f#_dqhsSCT>~REvr3qf?B+W3RA|N7vHOj|xmANg?ti!kFR%)JM#P1-2F2RBxe{C#&4K+W9BEl4r=6>}Gw7D*N%&fGf|_!a$F z%aOn}bXc_pupgvNi{Oh4mAcC^e-lPjV*sw+5RcoS`?>ZhnuGNl$j|=oq35<9sPn_3p z2R2pI{!gP+b_A*5l$jlblfHFroC9Zg9QD!Dc0V>y2Z&(>9_waby^de~DR%=A1)>BBobht>K z!)Ty05^lX-Vp`<)r?dFmQlq@#{k_toBMDb9$X3qRHA&hITqp0H-3V4ESkVMG9tiy% zHPVUqI|}<-bn7+t$g+TLIxgD4dRX+>6Lxua*S&Eq|fk_XLYx|d-1ZBd!!a5wR=m4R_cc3P;;|$ zHblGcwYyhf5;7#tLLKHg#<&F`NP$Z?9*^o=({#KT`D)ZdeVlHk?rn*f+-S@|vOA=! zdc2ioM+({*ewvBn;)dr06LYUsP#4r*OqW`zlPKP;U=}|sNosys$ot9O>8SGg zLIfr$BGM-?)$TvmSf}=}vFz(AcIYYD6v?xjU}fHdwV0Py2la*}hqIVbM%4ese0A{z zI+(FO20Sg5A8i>>*~2c(QCyZ_(h?G`FW+-AH>yKHUxH+em$&=hae zot$wxYvOXMa7j{Y~-76e6-L&>iF^PLb&M?Vvi*K`;iOX+^7Vd(c`|q0Y&O!npcjzyh2q!3CH=pyE zoF*1x_?pY2lbn#ZtXrrkuDt#lE|tDubS4+yFF{Vd)EL5loghV%X5`|YL^;3)@@ZuY z&o4yYO!H0V26CdQoBjiTGwfHxyi$=z1|8|^tvi=4rt3sJ$T8u&~)>FoDL`NL4&lA3w$ z8a5Mt$otCEiKZe02WznR`sfaRb;PodL8XQ%E-<*xa4VI{VepG9|6z{exH&CYladOB_sbY@R#`}OR0U%Ook#eeYIp;W>|}e zxgt1TfL?N&Z7}zFO=!)6D|=hOmtkiBE`DpCQ-ShkIhs4euTPir;?8BwOGjWj5O$gJ z>#T!WZs+aChIvP9=?zY^XnDlxO3_Gl^}#_Ma`ugD^P{%=1?4z>xHHip=PxT-R3wOh zMw{IPW+WD;HP%!U8W-z8TI`Ujtmt1kQ|Y8z(W{xmEQ?pY^u2c-i6rK7LJalmw#8|| zpuf?vFApWq!3ZWs{`~SuG_=hqi@Z+s<9E;-a~P@TofWK6+M6klG)r@n(YzZwQdeVO zumNn)<4aipsrj8uu-DKCyKMV878#ZyytG)tLoZ1KVeg@rD=SzXdVLoMoD&8H z(Zo<~8;2?PkB+B(!9)*17#4gR>@X~$3bpy9LI8sk{KGLAaMEr2p>@Id;oZ$lDBw&y z+CdWilK>_YcK3Fpr0vWS-W3>9!iNI*SMEP}!%`+u_D>83EF+O0X1oiS z)+;9cZ1fUUJtKWC*RL_rN1lAv`(y?S3Png@)spb9c1#kPZNb9+L_}$92Rb#aJSj+* z6l6yH^ZPMA;L(H`aG0MXB}u>_%%)t{Kr`#rWA~<;w{G3G#`xCwGVRfYU-muR>*FwA zrCW)%@klffw1)6FgC9|tHVf|9@*{-qtQDC&wx2ph(>!=%`~Lm=2n^xfv7g*|;dig! z%4@xxRGF_eU^L)`ilw#Dxru51M)IbK*ZmoloHo`&+ErVau|?)l|A!h!7bXmS-Lx_X zNh~}5Uux&6+Wd-;SVLpvrV?{r(!zZ|B`%Sm61QfS7r9)63_VS?*tqAQvfZ@GEfjIZ z)tvYvv)tuO88iH2d@R{Bu4}5%vkLR5GwnAfsVX|H9;fC}7ll$R60;WtCX@RQ?JcTI zX`UXn_r6+Ky-AM1g1Z#AgVUQqQY&oqmQZ8BR*FbIs16SzMb=gIP%x&qTj2L+mgh75 z&AH<0Uu)f#><$GghlsSCI>nTW4K8YVq5z~+eWK5IP5a;nd+r%>mC|5dYS%Xt9oU%7 z$8l4ulYI6C@CSt`BUfH-amrEy&Uc=fne#-+-%?-y_ebOLJA*#*>OdsK5e5+HOel|t zT9|JKMsKVFg|;G`dIcrz(p(oSIW{+aE-a!cD5(7bPAvM+HJGDs;Kix%IW&Y&h@U#1 zif<}q^Q+(Klp?Nvss08aJeGj4%S(8N9gX39FuDJ#?D5R3GP8EZ&U9(?Cr*?@7iigB zf2{H>1JXH?)cmtA$cFtdtqi(+C8s4W@(Gc68t~A9r>DT57I{}5`iL#C=<8}O&N$f~nIsPx$`DOtu3f%czz z8H5Q*?OIFTV`e?Tm)U(};%82^kcMyTvGjjXAT=K!n0bcjOl`k{9j}Q+efyPhSWAtaw+G`OFnz<=>1W9bENAP-DBPu+ZV0a&B%(`rOf2LPN zhfi7b_14iKSW8cGNLSi&z7?GO<1o0jL^i%a&>YrPUB9s>{_M*(gl|9b*g-T>ZUrIc z25>c2)pPdujGmaqwQfboZ6SQ6?#d&6SLrNRX!?Xm|6AO6Sv_E8^Q9uk`0nl7@(zSQ zO6w-FUMR$n@qz*Z!UhI$?$6F~4NM)%)qek=C)gsVec;~xP==SO#l;u@(z9+@X`nO*2^gXjs}iSa(Mz0Bk3e)uhmfpO~1;v8ITajVX*j+(|Db!_wQd0~Stp8}@gu z5{>A5oY{ctlUuPEK=On5 z_VNbN+#v6_f~x}O7afhS3;DHU6D6#k>9$v96mxvNd+*7gx63Kv@c7uRVZEgA`}CU9xpyZ7XG?ZfWz(>PY>eEpmmv$=cvWpAZTpt{CwaPY4)48jpp_|-bm?p+a*rQ6C*H4Kn>kpWKF4< z#KxEZKff_}*|ip*c6ol-aM!|Yf%#E-oj4%=JI zE*fv^tT*2wJK`CH?9uZ#>sv%_0NS=aQ(}LaVLqRdiP6fMv#GpSv1T-pIgI!E`*pL9 zZt0Tl2JU+=Nk@%Bf_$S z3wV0rvKDh1&AZ)MG_%|ub0}yFhtC(1+5!N_R4Ss=KGj}ec($=RgzLDKnoqJ!L@NgPyf%_TM%=q-CdXO;t{ObH&R zcb@Uun4LBf7uI{|60uW+=S3a2^sb!08WqX~h(mr_C`#Df#w4^7b7E;eQsr$l4xEtU zLD|jvLgZw@rq)dHhtbwA8t&Azo)1!K{ZAxf0`D2Ra|!LW)%_-~At) CNc*$^ literal 0 HcmV?d00001 diff --git a/_freeze/Comp/r-sas_mmrm/figure-html/review-treatment-bcva-2.png b/_freeze/Comp/r-sas_mmrm/figure-html/review-treatment-bcva-2.png new file mode 100644 index 0000000000000000000000000000000000000000..9df76fd2d811fa10d276a14984785e419c4051be GIT binary patch literal 68563 zcmce7hc{eX__o~iXhB3LA&4GCbV5dN(K`uYbkU7Akpv@J^fr2LGkPzHZZLXhL>Z%v z(R=wO_x`@}7kpXw@Z(I-TtqgL~v*Kf9|po+>-dO z`Q_nT0@DAQAO89O(3R~ucDjC_nK!K0ws)s<6ge8ANc$Sag8>dl=5`BDq4P`$h9fQ~ zPdr}CUN19GRrKV7#y-L|2Aozbx`6>5rG0SHCo@d>D8q@iA0CrH=0tvl14l=JLtWGJ zxaNtWm=zKA)i+YTI$IO<<@)xGMo-Q#XgMoCF*O6wQbhJ{T$r+tdw3XRMFd&7c&u~9 zTJkXEJy~uLlhNy}FnC&${@A?%#N@gZ=q$7_Je$4XqE}%^wD0Z}9Rc+%7h52~k7wxd zV}HJ&?VCpxAzt)hxVfR$CslWFw`_OY7M+?|SWsChEm_mJh?UzKtARg2dPf0q%+hQx6rcu@XWN06{ZUwI{e|Z0gPh;K z@6Yn|zmJRC%Y*m3YVY^ait)3BwSYf$#aYr#g`c`j{rKJbG$_Att5@Mhl>Z)1W+nF| z?+JlC!L8L0nc}Ok8@nEGekb;B@~dx)UOS9+0zyjlRWyK7!q*ye7ESU z;ZsQ8?0U!!w;qZU9cR-K=QPBgI61;`rnRO19c zsjy#%oaS;Ilrnp#8G#FI+u))PfKEeQo5?EkA=ccv zB9WnRskQ((;Nb1hSGR(>-k!&zBLr8e$EYYR{71aA+s(H$UHc7phk2Pn{G{057MJ{~ zoBV4n(+`GM^@>1}VJcJ$`d2j1gkNU;%{dMfs6h#6R*AjEqHy+SB<|!1Zge{~I;YwF z%s*(o7b74KSiZLsaNV5aoF1(*dr&y_h=#y(n>kMzM$NdjXCBZ$1EgdGR*wrAm+~i${j%vgE zW}MVyxM|n5U^O>iodGXn-_=7<7~|c5iUE^>mLM!Vt*q$o8@52NUD#sz$Btm)P45JmyNRz+SsSGhgi@Y=LLmmrGv9!!yR5u6 zva}rSZ{Si-xgJLswgYc=&U9I=^(u>(>(>9|N5=yIFPg;<`z!75O2Eqg{SN$hZJ-29 z^mlgUBU>cvu?GLJFh~1l$S|}>ZBA7XDFC?7RwHl2$$sA^_)@X5a6z>BVUUchV!Vem zV`IWkll4U|$}@xNRQ~jz3+aC+lM&VIZ0~wa{OXmkREENbrTf z{fW;?w;68HlR1Uc-}6Zc;;=I`jKM~lflj8;f+Y>s*MIMm-q zoo@7((YxXwJr_@%AF5k*CQogj$hFa)v2P<#lJ(xlKGO$@E&|40&KU?@`>B$h>d>_?_-kv2QMWST~+T^;Q3g*LCD>=Oh{*@4Eo)(iQ2=FRl`hZ9ZppMsJj zTB;JAvB7Gaado`gb}`%71wnSWWVlb76I!dn=OEItcs2}TaEU(Rij?Q57l;T^zbcG5 zS17LLjDBuotRiMzZ{xw$2|c$ToSFK%z2h%RMy68VxKz+k09tmSB}ty_Fa`kH4>IF# z%)%+X6KgGdYTcH!Gt%dpE_SmkdUAeQw|elmZrcei3^f8@5{>;AbeTi1<8&s zU9IC>;a896!ndpKZOm-^JI`>?f1uPoj>w zXMM!g$lUbKBnp=Me#;%0#XlzpxzkQ*lhv>3gfm}xCVhoKy38&y1q}+I8Bujv}r@N`L3Brg7{rdr9mz9=lpzq`e-oKHxwDv zrvw3!_9@SR`bk4Y&)A6iczZ7*Cg0>1Vu!1h^)wd0* zhJ@#K*dAx^!G!r8wNe`8>OFu zUd}gz(NqH6)GA<~^@|O566^$ou?AheSo_+R8nvpEr$XflEu$&ppt^> zjp+(Jia&L@f$FMDMz8uEPsrVBxvw@6eQwJOJXJnAUXL*)5|Ml>3!wA-pW7x>tV0{O zGK|~+olTMA2~E`Z95Fu9obm3q_vBy$**6mWU@uLxOETT`A_-ASRZmdDe)GF!qs6=Z zI&GPb4FZwjo@=2H?#sHO$u<<1xf%s$1b%nOIZDfpe*1#E7%$hieYgfw3+54HOwR9; z66|xnd9UAKuNkx_lhSzUP^2k5lrH6eewci4vS{}*tCoeS0}UWPbT(x5&bSaJ$O$1p zC2?fyxxbeOEvGD^2I%)5;A%gm^>EMfMGg3#E6xj!D~V=)W^9!<^e-M&5iEf~AR@kP zgAMA&l|Wzi$%+xbqWCJ<@l3t~r{$?hmaV~jUc(@^O5-s|R$K1yw{67sqh@l1;RMuG z=SYGR{8XekCFtC5@0h3c%*)e__*Qcy>u9>(Ex%hext75-WyWV|=5q;N9v&VQ`g&aK zm9FmiriTYy^1yvfQR1T0+-Hdj%4R(z4sc(TWiM{fpGQ|Nd=Acb#~=&3rIYN963 zDoUJFxQ$xSKdP=xmFE}H?GCF5W{{;7Hnpy5$f8JU1(5ie_3|F_u9C+;gDbHhT&T<)Lg^!NM(l9z&G)RpN#JBx)@)^(O?`d9K(iAvUt&)&%Ou)&?WY=d7s-S zauu-(*M5t+K6kOPUUtH`X1hdFNM~Bc%bpjVh7)Oie?Vm7k0Pp>&Zs%rH#5PO+gG?n zf^f!vyjk)g2BSuz+X>@0{DAZ$;B!xV>5Boy%)wDeGbSq6KZB#^b@SYMhaEh{+9XNS z5CG^DC!R?Y3|E#AF7dRE8!+^#VLDhzF;!4c69swq6yri_J`46=c9!l>e{qo-1b3Fg;w%IuL%x*s6-Wge{*1qY%cFoyiS#^isfy<8SQ9bZPLp;-5Cc zw%ro!FJLfVYZWg>5}m&Yoidmaa%=WayIw4v4>(=j$n>Y9Qr+Bk>?0#%Pk@MOQT*C6 zVaD2${i1r04-f_?jcTY(pbX%}KD3lP)hg{&G!i4S+{w$+ecigTJ5gTP>x-$q93B%F zWL)h;n;?FOs;^FeKINk$F#}Ok()c)Sk+w8fJ+eSqEU|#n<2;;glMR>C{QR~-%ib4a z-`1|YBu+=Fs5DRbux4VlwhaS&_koLUSN(p{b}~Uc;bu+^pV%!>F1qg5PpO((Lqwz3 z3=~hgxjXNw?GK*ICKpTQ^P#G5PLl;?e6Q1_v%cx;rRYUq76v3qbK^=>_AFvE+^kb; zVe^io&Rd>L;%?XTec$HvF5%{m7XwcnQ6Es)d0|&Vgxkm4CjND99rW>HLbn$sO{zw^qjgMOYPG;wZ}sZs$?GC8yZ(rXX6^8qxgDu`OBU4VF;M zf79kq;~9mRFEBv1*|M&!9lc30^~?MGL6Tv=+}U#1>TMR2xZ97STzbX_g8S=t zgZ6(Lp)3cYw{IzIHqcDGpeJ*QQNEg*AN1mqfizWSmXuV6Jpn~XCjem|^lT8ui4{v- z@lp%BZ$02YdI*hQhA|)0G@RriIQX!rm`H^0=(3ob6MEAceL_`Rt>?6U)EBG(Qtyy$ zpE$$PSHTG|gnz;`nX#0TU#3h{8m$d5jXXvK8USeha^@fz$oYQA)d(-fs;;?TfeV+n zR`hdMS{$h6wcYUiq~>DzuX`tBFaC3)aV@6WbfgkC;OqA$z<7UQWlRIW2@~^PYTF81 zZnGpE8)whq9Zmp zDE%+GLKU6G@5-0ojj1E#elA=WeOw#VC^4cq2B4cFlHGy28T$yD-eybq5nnZ-YEY8D zWyx9^drxSo(9Rk8~qz2eFeiV^;3iD zvA5|3hAh><;G~o9aeg+rm5CIffB-FYyIXg|2BIzq!l@=1dsMV*XUXIn2@yFsUo_Lb zsUDlVozon5Kv33H*2e(;6qJv*zGUjSOsB0w$#0JF8+FEvdB@eM`68Y* z$bBtMACq<`)9vqfqOh1esn(9HDpnQEb0YcJ9pSw7GGnNQuAD``ihE*Hc~Kl7>hRjW zPIj;Z_M-q?&Hp#Mxo|*I&OdSfQZnS}1j*5wvd*-$ddkWE1;bz*etmu4qc!7d;;VrmfzOUx2#1%Y4<{xEW*u6{cq=+qf z2|rF2N@pr|%;W-t&QER$VJDF7YcEKo27Mx~*P_Y_7`va6%WsJ7kAT%*zr8tre&|df zPwEVOt@q=i$#Xt|pBY&>@fHl1kbVMO%rmFLf0U%{wa72!5?uQRbqvEDLF~+3Vc&e= zYsL#YKCB-!a=$`4TQf+U($B7KpH>0KqVLFeiXWaWQbm-GTZM4VIj+^8lQUZsVMx=o znts4@k>wAujuZav!b#1blXOYn<89M(B~Xo{x&yD~wdrG5)V)-ffPR?bT{0!RsFWBG ztB7-B`nUW;H%+zo=BxsDYHD=CiM9;tTo|$ENz*r+~q+5P4!P?{e zsO&i#bAqe%i+07$@IEv`O3j;sAt!|$WsfL&-C)9-$a zBB}X^IcA(zq)=vZaSWE;#A^kB%55>GjkocoLI6^ZMu}9W;Oq|@SCDMNzNzljb;w;TO$`^KBv34J{m9l4_PRu=T7N!s}em^W8<4D77kpk5B}&? z2+yQ!Q2sFtfQsRPaf)urK#{5+}uRgk_%4Mm28#dfvWAz)f|2Qys)N4jpeX2 z!*^msgmoG27m;7Q=KNzSVf0cyFV9pQ{0;$894*HvZ1xzwfL*-%-0c-k*0a~#%D290 zU>ajYY^Pjn9xVV`Uk<#DO}KfJrGjpD)>s4i!+{0=*&oLGJ#Bo^y;wO$}&F`kI=jzXxQ57l~wJDQLE0R8kP6Ah*G#q{J;3gW`#mBQd zA$14X*O#g49c|AOjyVtaleG&p(xg1j2PQ&r0!fwKhTZ*KbZyt(I9QSKG0$l)4|t5L z`!6cS@z0?%t)KT5-P^u*1kTXa-{PK9KPkw7dO3OxJyd+(F*m^;@w996XRqZef2K9< z_?wofBQU_EL5m&~NwEob&&`G%68yYoza(>d^dwENvXy`jzVCz>ZrY!3mQ z$z4zrF=3OjXXz~sv03wfj{2t_P1LgN@9%MHDfHz&9MND5QIG)bA8w4XFOj~3rAgA! zrxm-V>7DPQSVYO9r|YbB)p#aG>w@bbPMC)MVmLX?*IA=c^WZR}I=PqhA#!s+3BFBqc=>W=6y`rfiMOwk-?v?Z zjb$&JL}bpQxNeY5THZSbMp`k@mp3z!T$aZ9nSx&~2M0~T@N2%^>7bv5cMO-PfQA+h zP$(H0aOcm5JPs>lR_VKIq}l&C$ZaBfOdPZa)8!372|Hk^XM`$~uQRiHOGi-#2$SnDP zfHrNf0^YIt24LIS`pP1VP0+A2*jZm~O ztSs@)fA;)2ugtSijl&MZh6(Jx;R0{Q;Q1Nq7$+alx^D-4%e^t`yy(o-iV>WGBAb&~ zm?YOaGl$#Iz=12eFVV6D&fx;QxCee(hxdn9NxiTS(zT_ho`{DfT=^#6*n@IquWc4l zO>T6P6cn20K?6(Q)q0*^H(U$=6VX0CulwB>TJ1;r(EWd5`s&~s+>ISN-_)0=Kk@$kH;sqTm(v0 z781z)uXveM57l*`%9a9#Yxx!x-S5T>l6_CEg$n1!?Jw$_W@@C@$I%zgqj%*2s!vZ& z+hKWB|L9$g`t7Iiw4EFaAYzkHQ3wkL@jTOHlbJVeH%X7DePoiLpH8AO{Ei*5Kav+} zpPz4%89KHjM!WCZ`b!&XnP&0%jg|^-sOi0_#ddccWM$N4uGB zC}GbMKD(ZTo@RZk0;*EEt>*MVsZDH|V!H6@S>||_C|%YeAwIdn$04QS<23&A$e>`{ zLk&`%1#cnEXJ;$F(xxy%NDso!W>6c7^4jEB=Q+3@ym%#gWEb}G%Hmfm)gxw9>)cN> zWCK#Wxg|$ZAsUdqX?|znGz)e;_k9}`b1Q?lF|HG`;#hJ-FKp|*h%DQ@h^y+Q8UM7s z5C`T$AyJX+%od}kr?1t7pBc!S_+7_-UsA~hcJ-D!c^HeS?s)Ov!^db8Y9yUq>*r&5 zPm?Wc-+Y-%8z1e-#aDy#)Q~AYs5JQcawb-$z`0G8BI$ClEARHjEVMxcUXjmOIlg{V zINnjA-575>nQ5$$DEE=?j+TT!eNpBB0gAw1*490=+oN5#qaB6%n&U1nyFF(Rh_Gp- zM}o#hO_5*5-+tq4~*(M6>> z?vih$?$AuuIH?5nUGa8jQpa~U=^w^@uKT3w4dOdwu+IzlTDW^|_v`mhKpwJFDTTEniy0eB#eKn+MYzX^1Bc z`kyIA2m`;L)}LD!(%{kaD=CtqA~Biu;}fag8FevrJ{r z$U*yIz#KiKKW^y@u(PD9tkOE*IveLs;v}kpXwgl?t+tM@wdI3LI5`DbteL3ycEtHi`1TPB7ZO2>&l8e1jN8MP{eGd+4Mafg-yc%2sb3kJ2Fh8{ zMV=iLpgigE;f0h*p3az#zf-fwUhq{m;)zOJjw_pr$`C3Hb@6?4lRN_NRTJv%s=6a3 z$R1Y6_zvaUohIz=|4HZ4!t*AZJSm07h1#a0y0k){@w@^nqhXU!@veSi#k74Q9>9fo zH__HqE^ZiIMn|=}S{TT?D{kJydsfX?KW@>j#(4Y91qvgo48{Wd9|g2=0H;_lB#s@1 zBEvwFfWVP-XODfr7f1?tps8ezC^UayKNwhP()nh<}S?;tEr?)Uv7vq>; ztrBK^S-sSzmih(5irK<;=7uW}T!2j>khWhzj2))AK!KIO|et1FA%=m=j7H6%Tc zt-zYfbkSe7$RkwVtzCC0JI|a{AMJWH7hqa)?c=?F78ItD{|XO6=w+a2^_+CAW&(b; zg+TCh&E8B-0!oSFbV~^XzQHOovoNDJSZ-#Z=VD0J*76j`fyA)|F7f~*zaA5Ob;TJLnzVgB34e-Lro=69X#!GCHMkt>hIid4omX`I#fD{+TmCQ@{`;;Nv-a*yY1oYLwN)(S}$&3tb2 z=kCNS2~qcz^(Zz5?}BDC*!xuzVGOyw;OfUbh-mTn}w1&!BJqV)1BC6wct@^|7z&r8)iwtWu=)vMk zZfwT%D}}ts!OZ&jB5BoLZW0yc0U>+kA%EEokI-({_KEl31=83fV2(&Uqc*lkbh`ZB zy_GJPBg;OR)b#p&H^e){Nn2s!e zwTW}j55{FIjDh!Bh~ztGYR?w8qXY^Xwrp2(LZH-)=fc@1<>1>dZM`2&6qZkS%tdzo zO^$_eQw6eFQ~Dwouxkd#+y9VLb7Y@5NmmtW*FpPxWA0u1bMc)Y1)BJ*?7H9`n8$r$ zt&7iO7%`iT>@Q&~C(Yynj83PeOJ@=9h zYdke1Tp~^Hxx0vzi|hTAHX4Cz&w5oUcsU#NX~vM9TwYIPegW$_N-!iNlap0G(Fm}; zFqw8+1%;TQjed3%n^-J5$tjdkGBJ2!f|X{>#kzhzSOu07T@jT)?Z z=;7vmFWHd;^V|h)Z)`$|Wa@~h3Y= zwXz;_0%u&P^gNg;d@bLZh<#(JisaPF8&02~>cnpoJZLqA7zdY4PtU#tKRg>#&{);s z;3}VR&UpXBG}{IGS?H;Z7I<=#=3;iJEgq@m#Dmkt_}#eVT+WW~muq&)&x_Qe;owLq z^}%Mks|%)g<0b=R;qcv=#Vz>vBV&K*h=({{Zko~0gEzD_wo{c#3{1lJxk8_*srgYH zyT4VhmNK+N`29$uD9Z%D!?RXC)YPO)N5N(YxpvX8V=iV& zx!JXar(krv&9bRHy(01;YrA_Jy0s1km9Q?Qj6GR~^KP%PP5KG^8Q?rNrbtNvi*FPu zQ33#fHOa{yJ4NHC$6loR5rl?&$HeT|=5zh1(i)Vgy2He3!DG;5w`O2YNr~x6YE`ii zdN)m4Jeh(l)ZWj{G(VBtR)g_eg#{hUYa`clSLbruxgCQ83cd>EFY3Z=Z^4Y?MpCi%#S|mBezr~ zYc)8NVo_Q-J3*h-PMS=&f1=V8&wdoEuB1!BkE`bt10P#XoxEaaCf7{|DIvyR%3D4q zhw*F|H-tV0YFe#N7J#DCMq#`&v4gj{lpoP2@7G3H(?|3z96mz0`TQ!Fq$k^htwdU_ z&U8P+&cDTb2Gj4n@rEy-ui)?5G~^xiBY29`JP);Y*e!3x$4@kO(;C-QBvyo)~4j@iE!C$RM(=Y}5)dt0G(% z1%r)AYP!Vpiz6MbvU3u3TfYxN>4;6eS0~TVSG2@QVvWV33V_Ze6Sl!*PgdCQ3B0O6 zkI4D`8~rxqUajsG6}@y-b(n`y%07sjEA_I7t{iYLOFKABvo3}$7EEg*hpi!d$CYe&NUr=s|1P@NMJ=6j+gKNzI&YL$56j5YITb$ImO#AUc!rH6{tc`b{L z5Sr&ZCL@g{lxivuQ`sRIgalZ;V~B6`AZyggzfKn3n%1Hb=VXM_3SYfdg%#EKU@4U0 zDzAenoC4*3dY_kxfvqmK%_g?@6_lV2_G4sW-(XCajma%|R;$7Dd})1wP8>`-IoT9` z{wK1^-FXzeRvdc*gEMKe6nr{oJX=qXjrn?wz1mtNP^RrkvXvw~mx`5qVA9O;B8@E6 zWG;$fE*}!LRsNleE`cdj=P{Y{qQmqmtRaI*TSD}-2Rs!^9bG7!7|N1Dh3=}9-LIm< z@P3y7ow3PqR(KS@FOvzg_K+wQIt=*VDuNfsSV7AV3;*cP#Pvuv`mMENAxVJbpl@No zh^JWNQ?XA9B;2ChHa)+$_yoDr{Aj(fBM2^M;la@y_vgNj&b|`sFkZ)!nJ~zw zMC(bLmi3CSRr_<70v|HZG1Jvt(Z|zs2lo0I5WUBtobG3S0Jn*|?lncM3j(F~3;FV< z5`L@7)2TYBQPfP6+z{%h-v#hOg4E~?*JTjcDJ7XtVr!EV(~&d=hKn93KZ^#|DRrb4 z6P-;r5#E%FUCYWapG2-G)M-igzwb}bQ%u0~{^n+F%QkUe0gP#E;dX{^Zp?(9)V`Vt znjmqWTCL((s50w`OM^@|^^9koRb~YEZe>e;bX~@S%BRh4fh^M&udU@|Ptk&jI||*+ zO3&#OP7p`N)urYmdIy8Kqopv_{ZsbeQ?QWlE#8Sq%oN6R@kfP5bxJFZW}YdoWzUCx z+`rX^tLL(Us&zac^YI2Q{v+^NeCj|mS=>_W_3bjQs<#}%!aNI1yam~>jjUW2p~e?e zN@(9qP{>*byN9Nru2GmPNBarijGtgobiI5V{CrA><9j-nxi$qNmHk`B=dj$}V-|lp zpQY_1g(3466|`g8e6wAx&0M_&jem=v4()}=={ffPk|umbTOg^SCe74vN_k>y5{}hI z%GSse;gc{PcDb+DJnR=Zmz#SZwjclcMi$!|=6C)kjSQ#>%hosafcIL%Z_aw{2cq5C zVCNG-O>M>IT#Ww;_x?zJ5oP_*Y;d{b;!&($qF3e<%lB=Z*RsTxw4F5AT7pIxT{+i-1H0=mzb*&@Q~n-4G=v;LLG6;QlL2^(_b@hUxWU#l+z{=KPhEKZ+n*queCO9U zmlbFex0Jt2M>P^<lT7yYF6iFnGTrlhs-s3`_jl zVb-h?lR6jgc3e7Hts9FVi5W#w{Wpca|3~5YD(Js8Blvaa*_ZqOz4q0u2Y>z>F@frr z`f5P}T;3t(_f&gqDd!u9b*r=^;e`Fe?ugzeSX9HW?EOP-6^hPYt*yAy*1RHI!u`5d}n!J-d?ywYk5LbOSx?JD5y)a zk)E7qVxxr_S_AwR9Pc0!)7I#_@mu`)0X%_nQ|P#sXPGbknETWRV6m3pY>MtalHjg9 zNO&A@bbR0*0b21EFRiTU$(m!~ zRSsg(fRrtGWy)*qAVJ|3$o``}q zW=E&+S`;q^uhJ*QIk|CqHCt-k6)rHmMLpKPSrnqN4DW*BKBiykBeSo5D9U}j&?UF1 zJD)>$qO5tSLNK#hKAy$;OD@7a$%!SCw%hIBqIAGtaw%f{@JV>8+zD#s%>(=>Kg_G* z7^$e6a>o`UOrI9-(j0898X2q-XNnl$0pW{uLb-Fh%Ze`)pcVR2KH9)MN{0p$_q;^596sjLlXV|{La^Jay0^(f|EZL;Pmm{WRx@y)|nv#zc z;Jv@?|Hh?K|6|B4l?RVR(%;9Ony!wtx3$z6qNE>H9ESOM4f^E217sz>C9xj+xaNAR>-R?URIlN?*QB^E z-4O1XS2e?sJC(^(OxIiO39iefTc^uOLT1ZGED6rkS!=huuFeJ{=Y8;iS_zR!wM($Z z2%U6){gd*O8I1NLakC%vZL#&JBwe}?@nZjB)8jRS^jHYE0xd&pNkghPWFT;PK7TGs zVLPy?0Uh!0g828WU4JfIl$P9HoT}shJ2Gmn&i9!L8;J=#V$pA7j5@k&c}9snU(zbo zm}vtClK5A`_R?GyR=zzi(yPlCRZ#U2{ot(gv1*+n^8P*6SIia8X(w`72Yn zo0f0*Y($l|(b2eE;sgUtz3~PMo9$W5!M7N>+)>1TU$}nnH^DlwFGIP_f~6=B<=6QN zYp#Ev0JMwtJ*zAK15tuMq5aKHPb!gsm*$n6Y#3K1-Os4s2d*@6Ee5kk5jD_doYc0e z^=N+c18`&a_pN|qKk8WiK6&_QJNGcvbJw0(mc4)QPkw8Dd!FqnUcl#$i-(f2bLy9Y z)PNp**KgET7%4o2pIHN!n@IuX-jFR$-x1$mo0=z41EU31q&31l#9W*@kC;?0( znn3M>oveD@J)`3={i{EPg?VFcI1Gd9>~)0Bo1~daa`}o=VFOAf&_`7Tn((nQ6A3Q8 z&xq2*L#jHMr21nKw-Yh7l65ij##4@xC)$c2my34v|IzU2A0Jf@-LwJ{j^A+Zca_u= zg??+&K=0Su&3+AxnP5V-05jGI)iYFM%^pruh=Pn*BSLhSL%-zg{S#epl+jwltVik9 zqbH?{2zzO!-Wfw4Lml6Lbvs#T6Q>Wts~K`?%Oo$geWjb0cm7qBSHGWy{|-3`k~mD1 zx2AV_icKgnKjs9|i2hZ1J$11vH6d45D`Q)NK&l*Xdfp{(QhiTixV?ZE!$o6uTb#c) zXRw7}&uk5R&F{M)xf%wbb{-5@bAh`$@wu-{fo#pDQwvatqRCMSm!0b`YRip_;90!L zE0MuP5o>4MNBYkiiS3Fd7XtBsz4iqA+Gk`BqZQHLn|I&OZZLXNz!k5h;tIq69*E?j zQF(af?C?&(W#>bfpjYu6W~a(Co0Y*WIp_owmt=j>8bQPkdfYrQndWB+r={o zft)0LmJNyTs8%-|6L4(^<4~)=8tQHI^n+=+S|g7xF^oP54K`Vapke2C#@;i#qLdDp zImhk<-@dbbVCJWZ=#cACMG()g(ysQNyTaGoR;vR|mkbPNKn0aZh@f-J3)utf(7}yQyQ-Q_Ym{8Y9D_8G88n9@?QMdNOn9D6S63wizo#yg`tnaVu2bDtVe~tqF`6Wz_jyklk;jqfCSy#1@jA~ zh4ZU52Qj+z!d}FK*q#subMY$smgB{Y$T1JfR|x^Yksh9qD}NTfbn%A?duzw(=J$mh zFVKBnn=wf5a;Fd9^5208n`vUB(=w_tiM4)HO#%J3_p{McRC!80m!x5MmA5$@rgU%RrtkhHE z0gtV}Zs9E*J~;S12QWx^tEj)=T+Da0!sh2u(E#UmdZc@pHfotwpMo#vIOwEig#|9u zPPmVEZ}J^|V9(TbTsjep(!8k@t^G{!tLlT!+x(B@JCmCxakDx8X^(2Td6#{LuPL%B zTod*`&vi!0Me4sfZ=yZSn{BTe%8F^r*Gh-xuPu3A5Ec)pMtu4C; z?3P>yaz+Q@ObOmQP0tgNoe!Zk8X+3@k8SiOW+X&6yGI7;P8P$oUnP(?yB|V#y!70w z`>J#Yc3hYu7WV+%5D~7SH74+UV`G?z=;q|!mCSQzawp!4+>DZVr@-2eT6*8%DZ6dC zFS_TZwkP2VO>9D@pQ5S$nRJq%OhzVc&mfhqEBcw$O+X4(#hx2eC6-mQuphl)P5*Z2 zR)k0joiv%=_9Avg+;GvoG;H~^;Co(|01vfs;-a$(6Pq`CFM1mo%FG4*~A( z^W!!Y{*Vn^Y2(rC@N&%`R!JEUW!kHL$!yMwP7~(tWXjNy8^Ft2=e;Z^x!s;$Pj@0L zA|ZH*q9hdPQ!e2Ex-XuN_dBJccG@;tu%Rav`cxiX+UizRx+#8~K+cw1`NLe<*H~S3 z8bX(+tr^!Ktm)3i;80}1+$4v7ZZ7(Xk5y&Dgo2}g90v3y9HN%t!%5gHPAvV?*GP;4 zwN0b<+%4T5>mtrnfIfe@)4eZ6>AJRTFB)Wqi6cyv)xg{ARBE%dy^*+OIENVny|(Do zH?p%zp|n&MiufV3$H0>f*o$xMC6O@gb;~>X-U&ist!U;NV>ay703IH6iI@@o z6rM*V7KAFj`f9AP&t#{;C{5Jh<2r>fY$@Ro#*}WUOYR$W5P`aK?hfMp6`%B8E(xjr z-`)Ks5`S!v`UJgfZpO=qkJf^Eh5Xprzwn72AWO{G2%mpH%^VVZ=RI}HIOlG+!E#ZN zE-O75J4|B9#vvWY&O`?%Vg-flfaMCi=iBEN-o+0M9ynP>Su3QRlm5w@YAwp*mc-3( zF?ooFii_qozTo;WF7*Y2s5bO7FHYGGYily%T4~S&rDrEB-qeKBC;&{7N%|R!*4WRM zOH&MIMtHwWNut|td&5+lq}%t|i@a+qzXb`%AWL_)OU@Q;pEaAz4Jxat+T*?GWa!Q< zKlMBCfU#Wv4yaofPVT`U!mdzgN~F23VCxsmoM2?6%TlP~sQ0zMn!>+@OzRBzvxekj zv*7Q|dXHVmnHH7rrM%4us@ggoLsUpvM1D{Dl5D;c5bwU!qg2xVDV#!|v2HUFDGIk? z&;H(V;{b*|!>@j*bYx+!&(#;Mi;?w!bY^C96smcz^8=xAKkrrDSr!x`(yv#QNTi=g zapAc*bi`M%MaEJ5YPHE_V!ur=4dP~tLi`na3ycc7rt2pxo;$Coaj=ks0^az#TTNUV z2&eLwq(+d8wG$@_;3c#}htazG?0rq+CAetO{_Lw;wf$ z@<7yk=IECs*;z)pqG;7cuW_K__FRE8gQO&R*Y{I)eGPgssw_~lNQH(F6F*rW(|l`U zRz}O;O0>62yF?P}fy)~!1^A*AQ%>(E&5X2UWP8WDDdPv_FJ;?f8h45}T#nNM%tX)E zd%!{c<_@W&(zC~Qg>f$Aiw{e{J4s$L0=BywVWHu8l>h2{LN~%G6f%&4d_=TK$@Lhx zvEfumX!N6pidUWsZ+V*{u9?9Tzv?TQh_M^ZmZsq(8ov)B(|l?xwNi`YYvS$7xh15* zI~7ofH^!;Oc#Tvoo=(gIH-=dLwe%V*tKE^Oif4t0Hw!sffmV4YTpBtJhAXbcdAjEJ z?`i0nd!M9V86ibL5^A23A2d3vlN-zhqFu%Q4ATWFn&f#cHp*=y&nhR33QaAe`?M`H zqkA8oT*kK@2-N35fVjov+F zJ!kI|#%rE75f}U#uzc3&JWLi{a@|b+)qyx|i861=K3!3YB87W0(sGlOD;-BLXf7#i)|5XIC-qBvqIyG?B60pR3F>6%iYY}8t z|7AQ27w6(ow^DJy^;8tr@c?H8@xQ>!iO3tg>uo7en{+CX5rgkreNi1_rf#JjGiN%U zlN(8^j)-ZSisvc;7jGi)(qvE95pS75FV_h%;b4^p2+{bpa=xK`d(+9Gv+ZxP*o6nv zvG%vR#;XED%n1R@2%}D1&|}f@&6};j+f{T%uK3LUcoZirr7g(*y-G)2`f*HbOhpX+ z-|ej=LqOWq^GU17m5A?Y;xv>5nGFqV5#R*M+R0|8g#0vAc73EBP?yWgZSyPv(MMe8K;1yAb48z3D{vBtwDM-Q_`!}iAuZ1FriJZQX(^s{bT=JsPjPNB zURvHI%S$7bc*24&uR1@Sn7W`~7A3WMRn+La&CSXYgqH%@!}quU*3uGCKl*Ef6^UMV z$;ck^^SOh98f&%qZeBYrVcq4Khd!2Q?4&Nv1hTV583>+TXoH-b-H;Kf-eIATJTp;{ zlJ=bp_df?>@m%^alRRWxM@r%#BJAPU%xm?A*Qc8d$p~}Snv^%?%Me8}|HEG%Mfsm7 zhUi;zuS#Y-p4$#LXW?Vkeh7xIRj*Cu?20fEWq0L9%ZnzZ$4z#$#otv>Wu&jiRD2!f zyTj-d=xbNTC64Uj#e))Dpk-puzf)qpi$CrM7Ftj` zKU8?pDear^VV&SLdbznG4EO03c|23Idw>@^PP2clx&jU9V=BA83~^IxXyt@Wl)0_N z2mO-vYJE>7Q0W!od50r0rNpSMb~b2w|2@j#2}-`Fkg&sWI}q&Erc;$6;3(x}9L&Oe zwnE(Ite;8|BW+UW?)Y%> zY(xVb-nm8M)5_DM`Zn3IgS>9kU4LnySLs`P_I|vjUG~#B6k3z>4bM4nFRcgvTEDEQ zU6%WE$mi9pNA=YW7O-+@0>a1U?Th6qIpLTC%!{4N`6X#6SB1jSnew}WbMb?}cmO!X>d@9buhE*B5dCZa>1jCk!75`Q@OK;tsR62`eH z=YR~`iy69k<+On(aGTw5i&V#MgMO!aTK2ODLt-9Li_u{#3UF&oN=snj`D)(SmH{4o zEqNC^SIf|}{a#Tu#H_P zefj3VHAv!(J1)w9j_3>*YVz)Z)5S=$xsS*mp8b(L(`{nA!Z^0k}Ml;d{eC(%h z%U=j+g(@bft6M3rWvv?Fl}v(YRyg;|6=uH-A+C~vu3=t8e-}GU!U}npS5LM zL2-q6a%wd*(NNZmdCG648Kl(Hkysb|SaWnnbR{7OkFZ)vr?k2MRDdZ?HD3=C6%DYqhN^M`lPlf;b%C4R zdH#r}jF4T%Z@{RVmF<`de6etSg`Gi%Q0zhr*YradzHBo#MH)QK#~{{_F7ByvBlyP^8 z91TbE(JeU5P@6HYptM&KbpKlmkP;yA zQcY1=6%UnoIb&D6V$#nm&GPsNZy4G4Xv_B(Lv?lF_7C|Jd4sti07@}WSYYg55cO@} zZwBvS(m4{zx!KR1SFD9Z_vmhjU$(4zch0u>-dkfdYajI#e*ervS_urv-@w<99Da| zrV3)&TOTx0or{PG`qVkU@$CuzSgb+B<-)s=Ku>BURoeRPr9Cj5Bgac!2Xx#|Zx1=rZs)=H+0&&x-XRoMC(A+tkN>qvyQQ^<>-RC!SFy0; z3(qb&Cdtq5CSeVm<>^&L7|0WiW5Uz^^Nrn4=C@0UIiGp{!;_+Jsc)0K> zcD9S~lirxrQ=o1i#rr531j(v87^1 zZx@Re2j`p2SZBc7-khciUU)HyQSHAzclP-qC8=a3NES(wlSq=Bvj~!dqyc6~N>Y#@ zQL_N={DukK#m{ps!v zKvBDN;M6^E(W1BdbGix(1~3Ph1?#u$h-Ag;mo+BluTinVz)8V9Y8PdD z)}2r|Q*R8@dc9zW9Wy(hY_SNs)M7oG150SGxMQ zvxvle?{MAy(%v<(zC+1x^yjUo0we@{>HzKcec{J{oVc8S3E0>Gw<_6+2IFKOVC69nS3x) zXY%*&-zl3XYgSCWxWhIFCT2Z;<6im07t?=53DtI41MZWgq4XdxO-Q`%ow?{SrASC5KsG4xwML$|J?p{kC<9zT9uhh1^!|te#N3tD&D`iDDCOD?QzPdC!xw6dG^TVloEWYr}Gi3BdFMa*|a^s|eC^c=i zO{eSmGM}Edri;lcxFtmP&Bm#AokFRH!z9APuuV`qct-*FZa|| zPo41VZKZ~)t+0WLk=d`>OK2P*m`(8u$&o0LTNc2@Q$4Q{tPulRD}3!hDOxOgu)5re z`mlBf81M%f0MDJ9mtfwk3LiY5MV62%!N$!~oR;)ley>e`Z0GtX1VDQk1#8%moBB{2 zSn;$7|L!x3=88e3M~Z1c|c=I z|EaDhzjIP=(LV_d5BJ$!!nKsvn`b(E3`K;IBkNIsYEqjs% zdkJnP|M2Pm5Bj*=Oa<%zTqBwP!UTTsbNv*e{M%6Jdb8)-W`O5-^ZTowHvyn>G<=cy zrXh}`f;1Mffn@&;TCQdQe7!}|chf;97BB(Tv}}L5|Nd7W9VYC5YojBp(Uc0B3cP`p zpZ(uZWvttt&x|rKW6q>E=yOv);4Lzcq}&Yq5c{gL%vYKX@BGS&e=VyP`#&a#|NlW# zp8oRzvcSlepb}vetSoq zjEv*QXZ+iWPJ$vfiWTsI- zj~h3+Zec~eU;SSa07;#9?AIfy;@zU{cxQ1fJgV5h=NGT8^ORJHfzOZehC~#u1Whvj zX%n{Uok!Q7N1ilc>-*PUogH&FSJw&hudh=8*Q=lWyi)g`cX79yG+$L;9THL6xZD29 zpI3p8SZRc}MWejd6wr*`SO3-jghedse)I5ZUQ3T#{t-4knD-vn*4!?>@c#~BcY8bs zWs!iIv1q>2T^y3M(}^G}zeK9j*v0L-IJjM=c=0yI0->7u)_mDu2sSTLE(+JyJ66rBY2l0y<)#Tcrmt0Ar$mqv@Nno!- zoFVK*qh#N*O;}po6d6NcbH$-yMV{MEl=>->OIZ2SwIML3l7374rAC zt*g!Sl9Yq(5xD+kJvbqMrs&eDOJzzsT{QT3`_Hg$(cSc3Yy*fWtt{(>#w7*UldUE? zyOMlqTr+s3cXOJxereZdT}7tVGo?tP+0Cek z+qG_mnk5Aayy>FePgHO-rY9@*KI^|c5`>~tj&2_Y6&@kfN>qd*Ig&`OY1lF3_Gpe% zm)p2XKIHbuFVh1j188mUR-Be2VuJC zDOjVcV#vGriait5PU|seDyXaQL<4gwQ>15_}-8YGFYpRA{$LFOOEILAUKEiCNK6*iDrxS zpJx}Cm)l()Bhzo)x~1w^v!!vLp(FP)GJwsmcV^k|UXM&7e~{tlJGD36gW+Qc zJlG6AKytM0ikaFi#9LFql;AVQe;yAo1v!u%c{OE!uJ7F3VE!M&t2rMr zT4o;(d@@d{(8@>alTLZw7xSg!oQ2?Vg$%LUhQk05R1S@sbw842&>Uw*&9|xOz=1C^ zlrW9^aQoP;wQ4GCF}|nH7>?>@NBC8^d53+RS*c*@(+{MmLZ8-{)TnxYJY_-Kw15}? zxEKEXHbEFgM0BX}J8`jF;ZkI9VPS!f0Y$T!P15pKEs$ietoO*>?<@4NMAQZWocUax zKJ)ctqLpu2`MYicA5l_2yt%nKwF0gDSLD+OI%?{ax@$C$UalU2%<9EnN&7Mo>+M_O z&A(Bon~stHD-~W>zEyF7-8e>0?Y^J<5iKaOFGJv=(=TnX|8OquldrPj5t9hVbW+1$ z&Jwq&v2ZD-)a&iW>Sy^1<|18>vy;ULGHy~z8V3q#M>>Ooo_$r!ZctXNo|&?FH((qft){bvxafZ$^CJ6<20nGWgk zF&Ja2mxdZtahWuH-|qNyhX9P(N^L&9+7)^vcpLMZK{K+|dT_vLY}Y|E*^8L~8PVt< zVGM6MlOEz${cVya;yiPiM97}sE;Q(A39Lf%{WF82&r(KTBO09U1J!He^f~4svrZ+O z%Sk5?jx_D$DmxFPv%kkrb-nx2552o$`92rc6dOV4ZDEX_0pQd11Dqr`HvRgaPXe3s z0wbmVfl;!@yUB^jd|QwuTmTv<)XIMTeQl*ma^+1M?5`1jbWWqy3cpEZC}w=>9amF|s3)!-n#G71 z9L+NgAHC|=`$i|HI|qh`^1P=HWyB5L_&C;rb>scow4N)h^*gG3r*L)LnS`== z!IoV?e2LW4__{x5QPcXT0d%cTHpUg)H_2HEXG)4yotullbIN$%(_jPj`GMG^;=v!> zCyCLA_F6fwn!Q!HOqdFfYJ3kWa^z%UirFnMTmwtg72Amra=2!V)yI1#7diMZA=Srg zil36IAS@JqgC#b@3u#;`jx+CVJ*annZ5dMVTU!#~uqyk~(I{6N=nKpBHTlRhuAfch z{}LW8oYQO?Z*Hy73j(Kx_*9*ApKHZ4$y@QL2IYznSHri=N$#fKvO^bqG5I{P2a?TF zYrrXjmDnU5&h_2Txi_eH*yAw&r8jJXKeI^cp!f}u*a1EZE+$uf_`=f#0s6f{CGtBDlYlh9qv)YXf%R8|a zPSLp!ru|^lbY>h@EW>3n($8e?Iz*_5+wZS5%aSu>pA%?u>5;}SL^qJ-*J{(1FTHDd zOI`!<#0wxHLirKE`4GCSup`050@ar`UT$T)72sGGc^4j=ZQG-rg)hpepyxX0+wz(A zaUTbGUFa@On^>$s+^Fv@%QT9hi+N8iMTxk5@;$!Q=S_BHa&4n-QG;r=!+D0qzpayU zZEW}BpfM&Kj?sfv7rGyb{U+k~hLUO(Y@SVdpS7_M6y(|@2_2zrQk5R&xb3D>TPwH+ zhM*+`s~TfsiL`@@&jl}{2dsQiCR6lr(%hV0XX#2)JF0d<@@rbf?UnGFMLuOs^J07I z+038niXpaiw4PCEk*}`pY?p618#tb9(zqg*+-@j@c3MqIQ_ZL1zlmn z#S7;Tiwi^=xD2Q+;%QH$vuG&s_967OpW3W807C4BDBRWYY*~UTJxUW4u)W(}_x1bq^C?%%7!3}!(+5ShNxPdql5pKA z81C{tw|q1D$1#*l`f(V%;e}q01&?X~Ep|h$*h}8RxKXbZK0|o1Y?p4nUdu13Z_-Jj?kC4?_Uj3ks#D{M+!O+e`L*$$jm|@N zweiQ57--mu^hsk^Lw4yu7!-cG%NnRf4G%qCs)^CSwLwlUCk(BHqOW}+dcm@pqsFpS z66yK_M{`puFx62pP!!ux;bF%yjRuH#WZu~B+^NL z_Kn?G=f~Hl;fw1qc4>L@I(5HR(E@JqCZ(9pUU+lf<&+Y&mH8#DW+!71U!0fwa8F<- zYNg=gd%CT;)aOlBZu?)Sag^b(dei%|xHh_=rcf%p$%`^8Kl_awU=wYXMIAbg0Kx)h zIKbpbJF<(XHJ-(Rd0}vPMV?c*KmapToGNj1l4xT(YfoZzxb&Twpj-@GDZ}NUzHzVX zVvdD>mM#od=Kkjgg|GNA4JuVkX=H2va5KP+)jQ{bhxw#nNexuJWoUecA>HHnM69h_ zh$Q+#{W?}6xn(9Klo70eYxeA7q__J`0g>rUm*#zG;XB?wT>1rSzee^W`@Zo@K=`0X z>L22m5yZqSC~bp?NWWgjzz?h2r|_evBLtczr7_YE|1?%`tNRe|!J-k9nMW$+2Pcu7 zDLEv;gH8^SZ!U!Tl=IQgg47yJ^!K)#7ej8D$ZH<#wD&0VMj8qJD18E)JfpkW;kF08 zyE8$tjJL#pWm#>C>p69QMR=F#js~$Pd##u+79<8nML75o_Dn%NAG^VtlH3X>7PH31 z#KD7Jp^z#Q(iKcn)VT8-+2n|%aksxQs&+5VLYRN?id^MZ_#3?%qU9QA=B+sLn#yp1 zwP-OIClhXJGLIMRzwitDIPHT}hFc5O`FL!PDF%_=Guz*|h&CM*D<$ibPE+*Oj*N)i z9LpJjKrp{eOkgKtNcVIZuK)5@JFXgzs-!$4qEA~!SpniDw~SRGc! zK(>;&E^D9~dK#YK)_laG8E_k0MvLJq`I#M3b&UW^MB*|@MZ%j6w!}#I zSlms65gxiQS!U-T!pcE8QY?NvyS@hOVupO4c+W!3f&kqmQoe|pEYiUJPi`e5l|wBf zv*t=!QM{!$iWv`DX>vXfz2i!OWrPl@t|k{S z$O&9{vA&M!eSh0PXHN?~Q&))3a4#s7Owmi%6M}w{7W!EW9TdcJ;wvc(%XTes&w16( z<=%EyC34t3QDr6WTO2AS+KbMqKP37PD#5u0Qv1#v2s+K9IF!h77u%RFg(Q7Aar%(R zjW8UX7#3m3C?FY}Thpcw63O(W*niVt2w9O+9raN|2TrL6so|3yR{Je<0yI||_;}^l z`WL-7g<6uwkhFNa^j$#mH<>T!I6arYa~Z=xsENG_)HhXOp|s3r(zqE0914%m)*`|c zcOs0}Clt)Y>%b^4Kr#J3tFw2AV*9!b-Y-6V-fzn@z++I)Hn{L^@j==%!ZwSZcRlvi z+3q~~j58R{y{x=FalLK|hCsD@YoA-I$zpGwgXS_3fwF=IFH-UmR-aML%j4B! zD_e#_Bov;7pFw6nUK{~v^qpd(N4S1w|B(wYu6e)31w8w4I~H8G5_3vCGaC3jzC=TO zk64yn90fa>RO<2eJgB#`-B0fty^}v!Y*~`PCZ2Zc$fn&pN*@S5Z&@DG;>s$}Db&^; z8gLaLD=`|-28LYk_(E1p0a^I4z;aRn+F$y*T&TmRkTEUcGUtYmHv_hO9a&#xJ52Pj zf`4U=GDhVl2G?P$n16xBaGH@M9ZwOL5E_Cw<7i7L8W)H9MrMWrpT}ghIRtNQau=o-JjhtTu zK43PXTy=jvD*r@+S&1mF!?AkyEH{#)I(KkP!~$6)1!ZkX%3Lv67oz;^W|B)F=qTZ zSf&lU1)zScF@L)$R}O4|=AzY93Y097;x$Lm7317=0XylP_2*lD2Q+kz!*JA`3DZ64 zun95Wxda?iF-oUKpm(LAYq5Tio6XBwQ|)9OCf(3BmZLErpx=qFaui!C6=qF?L;ZE7 z({t+|>8V;WxZDUE~p1S%Hdu%=))|d)Oeh%`AeK zCn#KsH^Tnx zc=9;pmzK@5k&T#|f=B5JJZmv4ix|(>D}Cv7mx~W2e7!-VSNYKN8^4W>dF#JK%Gh>g zD#^`)NIj;jnf=VW6YG4s@+zs40Pld$QTKDwoYCHFRu>g;VBwXlvx#x-yU!>P%5F=VjcQ_~O`kAe{$h`@vm?drZOT2&LoT z-zgV4$tA`MVG1EH;K+1?zrXbH&2*jqv z$DVdKxBn`n)ud_8HnSDxfb6wDeLuBf{8(#;b94TkQ@O{%Gou27MMoKy?n&4A?xeC? zcQkP=iIz>%JocAzZL&;k_3FOo6~a?BFIwK)R#EYp%{TK-iW6;jM~An?J>dq~%{>Fm zaE}mi&M6JiPPIx@teDkd&#|@SMp6!bok-GVidd*N-m1VZ?4Uh@i*^B-lEFa3)=+=6 zeNks#KgzDz-`bonxwDQ6?ip+JFzIV4Q`Z4_YbA|)-l}V2Am-cg2E1nNuLObkqp^C+ zd}&7_5YaJyJh<_okd)%W$8Gp?5|`?pjzy*crw?R&KHs49trU6&9z?R(9(lQ%6U&(x#YEMysZ-94H&k4Z!B(KAcauW%YJAor=8R(w|IK;e7JJDEXmEbKVQ3YRB#RT>ILSX&0!!E42ZnY zy6cdp4m z|Jl&L#hAbTyQZQ|uG^BdH)=xiE@L-p=Th`^c(dS+m`9XScKk4a4Y*vv zo)#5H`zpaWSd-rdZ!I}qkGq8T8-fLvyZS|J>5j@Z%5DSTpOV!KW!Oy>zYX@W_9Fm{ zN(lEGEtb91E1W18fI-niqK0qH=|7po?znS}9?{>gYc;iN>=)zxbgBvDN1McL z=M>Qk!wKVzP@$L$B9WF~Q>#_M)F8g|HDuJx z4@hks>z^n2KDOShU)maUWWD3qmH!By?Cm}PolP=CR9BG#hPn6jWfJk!!(<&X&_Sc{ z0dg{DuVcyupr)wZn>4_l6N(tf1D!_LPAy`uY@7J{vkNb z#I+aaVtZ#wum-VgXAYUt%eefy{8VZA8#5%S76)tl&44d3Wt*rk?>&SP&C!$ZcX6P` z*@ts;z31xV#>$RYoSnvcvJS8leVb>eFjbpby~B$Cc%f*rkuj7 z-f!Q&37xiDGpa2r3F%znaRV^BD+SVVkMm?=ej1kffWtX22_0fIhBync=_($JgS>D1 zyJb=>IM*Y`T(g^;vTFxAK#0Ta0-;??Yp+M{cyg|cbRsT=K!@wl{_RuC$Y7aZft?5# zq30M`4K|%50?rbvpsS@xQ$6D}C&a?`?H43N{;_3~#i3sCWCR z&x>%k4J_tbRA-JoS60Hlt^cy2X|r7vM$G))+yDET5DDI=-fOjzITQPGozxZ;#yE>( zqDL2KEatoYyC-Y*&^Nz-LK`iPw%>c!#wLSwGN%$^D5q~KtP%WbKkPFxaLOw2Eu+$J|?|vru zu8mrq44fwMd7Tn}f9W(d+dguye=u-)VGS@vFv8+oq$tc$RB&xx~r8(vFDV% zNo~|AC2c8wHOhm1{>GqN0z%X(Cj3kp1O$=HB9u5Ewzak4F`zb4Y07b63q$|N83C`B z_~+Wfz*|jDI?^lJ4jt5CE<0zJ(~fJ0XLIQusFa!FQYje1=}$uP6Z_(8&E=9$7EnJk z2qTdO0#grK0UMAAMs>OxrBNFgnMxK9pb^>M=Z2}|B4%`gGCVsU1jm|$l=3vSZ|PUB z=W0J>CpujV)0xSivIq$+QHDq~sTVX;>7pZ*kFSq3e-eWi zYu*u`t;Rs@;)k)sAt~Fi2OMHoRKluD8t7F~%$UKe%L>ihd8EcVc)CZ7+@gU)x;lj8 zbP=Na!x0C6Ah3?rxv-EL?nEKi)=pTGQjbTYZ`QlPv$PTMZE1N(U-Wx?A@S+R7|pQ< zb5%IR@Aw%kd{rYWb(x|fdLuM&IhloD=b6%71Hgkljn-w-p|xPgJ>cyg$J*YtDRuDK zNp`^`6Tjtv9IKkkBLqr<**n0EZD?mvpLaGN8NF#>gTm$x%-DoMRFsQj+(ojH7S-O0 ztzDB{^s-ZN6_g9F68Rc;o|}sM4L=FY_fNnjQOxS$s@&(XKD&Mt@vO{zW@MDPy7zLMV@=)aXd%W&e5}@fxEi5qM1&tXLOlgn z(t!fyg`*?GqA^>BmTa4p{i{+Fg&9=dipVbBjj728b2%_S|8(bbiDvdj(2o#Xrq9Ub|73tt*DI|7}f4^{!*0pwGyP zl6q1lyM*=b2Z`NlBoQm|on5wo_QP@uxVvQEKv$of>9~;9La)$HJ9js2zsun-*KEyW z6J8x;racy8xf`t5LT^MojteE$ zrG?>$$r-&VQqq4Obiv?ywEXn+cEE7ETv0-GuW{{sKVrZ@C;XZMk|uF?Zq0c63oD~?L4lsTj8$!N6g|@1{v$EcTWC9rCciskv<2gPGO_f_IZ2~bIq!t1`rh@;S5%Xm zD%*jiH|nVkJh@E#J6O`(#7(jJ()sBVUm4E>snwu!b_CT(gTavM;f!XE;Ue*j-EuS& zKXyM;U8@94>57JK-PV(nZ<>C3T7n=-sih_Zp+CC0VD?#c ze@XoC4S(0mAx6C9fXC7|be+lhf<@`Jx2WbvmjjX5!dZrm;;W(yEUX2d5}Kx=IB%4& zPQ}&mFPaaD8bIi!&_}%SoA_rKlOk=UjW_1wA6*$a{J6()z`<=2wsYb@5e$QljR{>XSm z?lDq_ltr~RcgpPEJ2AZoJZfvC_EH&Vl?m>_&6cL zIf@^FlB`&e;tqd|Y3HlG610lb-we9MK>N>q&2H66dV~93x8a{#anR|r ziAkN!R#>=_ty&XL%B zHeOT)X=mQk7gG1Jeh{2h-*$$l*VgQB?$jvqNvRP1p4+-XV+m z-?VsJ>B zw=0>-s)=a$XitpQ0Xlnedp9;E1vzZ!$1w&n9UEOg*SHgq;RcMYJm~$dGdd>zjdAas zc1IU|RjnA}9Q=gsxC<1{yd~zlu#GMu>!ONT*0co7C&h0RPO2b06%2+>u^BA8y9)=x zv!$u$+31dKNyj*ug?b7aH}U7sne{zm+Rj?L$eq3PvcNvAEz>+L8R{jDIrpSLYox+P zPTpRo4JlF5otIARJyK07|B&~QMtr=C(R%NvFnNOe+Vo;;yfW1=RIh&?55rhx3%G2; zlw2DdW`90#3dI9d1zvDMvzGoo2Ot_2R&FzdVAoT8$r<6Uh~ZlXyU+o14?H zR#4|<&HA)ZBTMlSXk7;hhr{|f4ZYuRZkr;6TJ2s7>FJg*TS2By(cd@*X}5x(6c=3`-aotx=Gp_75L&mMkw%G_Z(*li)}$N;_;V26G#@Rnak8t%`#VX^(wC0~?{)Ll!G_QKB#ZB0l<*Vza3@$Z-3CJlj{GlOCH>b#{F_8y`jD|Jl-e;ghnUQ7~MZKO|vm`l2&Pvrq^3$~8&* z+ccU`M6hkp4UO@;bluz>Hq4mE^w1K9+;do<@xVk3L;YR1jvE2BN`IOo5SF%2w5m&# zC`#9-_kQ)v!N#fEQmD>zK)vF_PmKZo513*(Iqk^MoO>`8W5YV@K1#x*?3Da;c|d)R zSNru;pUV-L;UU~s`)DSC_cK&l|LP3ekU$R}oh+QYia=Nz^w)k0U^+svN@VYHoh_wC zS~)g8JhtW$F|Tq?HFa9>0^4Q51i1L&huWCwLociSNwuwo3^0FkqC|A}BeRXjX`dhT)s3Ircl*coFz*nxzZmkd}{Vdhd8#2#C_iGC%8}q z><)jcr#Mw3L(9!+L0!dqIZAtFh+gk;cIPpjlD=A_i*N@9I||2e>q#Ed6K-2J14h67 z^RQEoE!b=g)DEA;+PTb9`I*Wc(HBC35`v`;CXoH=PSWbY{K5Q*(qXxa6#i=*n1wb5 zX#Eo4m>^60hGu%hU+>0c_t)|K;_#=0Bjc>XjEnnx0>ZSCjTUO+-OLqKgS;=`K4*^M zFmxYc`x+N~zAwv;j@8+xlzyq@?J#i~R?}GZ)N^tDybNVv@s3maN|GfbNa`G#3+T1$ zYc>ajkF`7)VxOR3cfL;4uWhx-Zv3i0WZOQ9t?gIJyKSub`A4W>8NT@OrfuZ$4vG`S zOqVtcUn!#vz^Qan8y@W%HVG$!gKTFf#%KE`sTf5KeS{kWAPee46F4TgHfa6)Hw=d2 zP7&{v{B@n~q~U7@m2HNl+Gg{Ak=n8Oo1N+0K3~e47x0&YNw535l*LVfV7BtD!_NZG z%B_JHexZ?DN@z-$Aa$LW%3($fU!*L=22p&|J|d%Huo!=^Z}s5A7Z2H>j^;O|ApOw;zk#ZJ2mc6|`iL zL5V|XdpP&G6})2+YrXKq;v@b66@REa|Gf!q;N7b$+V}#GVY{A}PYp+kChO6EtL8+$ zQHbdFWiFqs3pmV{Dcr&Qo)>F0a7{|@n9~&Q8Fg=q$13skf;`Ip40Ab@PI(ryuBwN- zx4fszDg7Pa)2sg~l7!`-(h2R!?r|6t{WTwi(Q)wOMit7JL}&_nc%NBmP7f4TN2++- zKb#q!gzLbk?No(T#p@=F+T$fTr_Zs^zDg6C?3}l3*cRm3cnEYGyK9oS4e}RCWN6>i z@Mu_Q%1x0GJj;s(gF^jOHkYS?4{*a?$G@t67^9c2SeE0#yqs(@8imadhM||Dn74Ty5lPX z3&q{!iy+@j39(!p1&w1H^!j>6xUtRJwH2`TOeabJ z^Yl;ovQI>htng76S~J+|VG4;XN))cte$S1=WG0_*7y0hOhnYj{!(2^5*W|e91K?5H zTK5-kEm{0e-M{2=iz|m@&PL(;DrSI|K@W>h&Yqfspo~~nFCyE-pwHn5GYXSW?)`|8;kF_|T z8;)=+FTZnok)*Mt=h}j27q5LKViq%`Ln1r0k}Bqa{j4q+fu8m2j$ZniYrx4W2<(+* z*9YJxAQ&C*g5^R6nBEK1OFgPz*GxZ}u#(L=EXMj=eGPIOy|wxcSQUl+^AGj`9PC%V zj;FwCA$I>$;VYs5Z4>>K`)CUEk{zO$PHA3FM(XPiCIdeitX_c*T| z0IcfH|L3t5C;b0@to^>mQ?Ooit!P1KW}ZL!;h0_`VDkBmO}H25%E?yxjJ*+N`A$Ml zHPL@K&*uKR+~AVR!;8p))yS7C)XGLba9g3zj zQT1_b09pN-(iYS<<1nr`AvHL-UIe}DeSdj==KUfN{q_py`f9lUt%Bg;> z6>1mylc@rTRX4P}lz(l(h50^6&}87zHD96F#f`pJS_jutA}#+P{7UH?zZU32CZonqdL%&5h=7( z;GMs98w_UOoSAhar5nlF6S|%LMBfLNxXl4b9DVzriM+uaxK-kM+WYeTq#LSXYpxD& z3{g!gwdS|)$`$q)MqcG$Dffj-IQj^;-uSwnYs5gT`&BkG-v*SbMq_sG`nbUoC0t9F zE+OMm?>3F_r-MmSEkQI&AL?T%1u3FM9=~b96 z^LS!)WDL^Hmu**PT_sLMnh692RgY&^+`_dTjD}1u(t^K^0FJXiwZFp> z7Cq$sTPmTonUmyj1Y)|_V8EM*IcC6`?=d-RP;4kq>OK5vAc*Oo!qxgi1q{GG31!-= zscid4F3OSIbQhgqZ||O@#oO$DZma$6zWc+#(p(4^gt3++AN{MC$$vBYBT)kdjf)tg z!Al4^#_a)MlKtFXDQHS~hZ6I#L123+{u{0az!4%ZN{P>g(}fcbIQ8el4~S`L%hD2O zGF*cG(B!6@(RtFN<)dd7BN-*G{C|48_iyQrzJJF_>gPz{HexEl_9BAX$t;6ht9DPU6H*cIL)SH6 z75;fF*nLe%yS!SBFSD4gfO;(dul7%o z`z^nFl0&_Bms_T-0?av*n7ii%or057d{_s-qg@GX}%xAUVF{rp{lCtk@wYy000>mZe8iSxqLwWso3DKJ)I6{ zDq1J=oM`EOwj1ybJ*<>b)c>nO+B}NIN3QhBMo1H|L|}J32X+JC)y*7D->b zfU!@u$?JECJW!5|%=Rc!%o-`7)BkpS-HsT`LhG*Z_R8l;LC2|5+}6jmg-pWHGiCU% zzVjN<0|NnMmG@6bt1<;I_T1HBm|j197)T-w9)>KVJRA*%Aa`WGflXeJ@?TRW3JE-w zUDhD-IM+S%X;5@xB4m-poNSO%^&7zv=+bJao;gM)U|uU9 zIiL|d1`E2CwLiC`fWLNmhsCF;$w;}QhV9)DQ&sAt1{}?UYdI{H3dU}r*ug9nrvj0= zuCwPbrK~WIMEo2A$Wj;5qWqQI>%i%H&%-mQd>C`(cPhUgY7Hl#D~TV64=kRY{W|9X zD@x(fy=~$MpXo9bL}cqHkY4M8x=Rd!dLS)_@kN+zN(*_USgN7lFZU|xMpM4<%}(*v zE3+&fx<4(Luri*%UiK2xqofH(q2r!8V$h6wkQ^Scik2tO_ zY0Ie=(;j<^%l^!moTb}Gu`0#KU3VvzoQeL1MOTG_t>L`mGkZ1x=?!)gJ`xt)p*d`w zRMwX$apDQ+&m?|x3WB(;5L};%^xR#N{(^LRFdljGL7umpRx|{;n2(^fgM~aaKPb8d zY_~Xpz4C_qhG|U_F0e{wi0(Bw00Fo{bCL9~d|Po(P{JcTcxHscj~;fvz6Ry6ETm5; z%Z6Ie!FJN}$mpB~-fGqqjZ&>n?>`Z0oT3~?q_VsQPk=y1tG~WNWVbKB2!MM6C%+__ zF0_C-a3QIWKP$Pu%iFP716R4qfcHoR3+rvn(wbOR0Kn%VUDgzR9EU)z_gvyrpg=Jy zS+o!}?HSx*<#fp+1gO~S2T`~ge#4geSnMm+iz9zGVERHx@v1~)- zrWd_S?mdsWVMBrw-P!q+GOLNK0LP!w4_^pVq^0qF0t1R^fBuS1=<_;1_>XpbaN*2& zJL?}|O#rAR6e2FR8y-nn- zuV?&3W-TGNn@+PiYQJ{5_oKek0d3=|chT{}lm~*k|EvpKS4I%Ykz?U;EpbtphXj8g zpKT0c>?V{LI4I*eh2HjVg+}dNSJ`6G8;IWFQCjoUM!s>veBPN?Gpm_8M8T6uLE z{GC=#yXFpO28#gHsLrM@->w-`q;e-THh-YjIN|U>>1ECfolR%}5b_*5>l@$5D2p%= zU$(7U{ad9BeUjfjSreExR_T2l49_bLdNoP8%c+^6GC2b%zigN4lKYHK#?a;mrIx!Vc3m`7=o=B^Xc)ud*r zbc6SnJ^VyKqe(x=#V+$Y(UeYyz1>2K2ILB79dHnO4kKgY<|kV)OKVkZJCBNm4@aevS;Qgl9pKaQUtQr zutv(_c63&wu0XXsGS@UM;vrrtxc^znm$wo^d5E6y(vO9bXNZZ%9zr|x@=t)I7GQ_` z?e8D-ZO%p-1d;@Q(mrp95uYszDyAE-`P(LPTv+KinGIfJ3tvn=Xz^YN;ZmNRpw6@% zT++``*a+6fPkh2fH`d^c9{y=(D8BKR&wkl-$WUy{N#f;+_G!F1C?tPyPDkK}as?8f zs)dVYqRi&9>eMO$KF$2M-h6S=_b{?ILhRT$&+=(a@K2z!Yd6IYZlE1D8I(3vW_G$J zB8c&xt=XyiY21Yq??c82rL@A>i(B__7BXj>4NcPc6NzVzyp*fjL~)E8s=DaA+5e6^@rC7L9< z8M&j6V~+NT_qAEBJ&6FWG!FhG#PisAx$F2;L;<(6ZjGI)*mA}p&1EZce%OJLff1Ru zS28Q~W@Ui*!ipP@L5QFFax@3RpZM<3XB?7^t?-j4t5Q^O8H5BL)-1HZ zS32`3?-jpuJ*D{63;CP-bu#s0CmTxAlDmi*b@kS)Ik~XX_+f(U6Q_GvenaQp|CRs2VVfR^3~ zn#u9ug^(d~IC#?B^=yx=)TL5;5!>K~eYO+MSfUr}KFB3qe<; z(8Q!WvMdD-_klM8qf$hCAq*Iwz-&V9qqU$Y;I&*z8s#-3aT@{{jja)vmX)%&18HEc z2^b5u&`b+j?}~US+zeMV587p`Bf-}b{LAqE%o$etX#O*U{G6ci!u~Gx4VEe!fujmS zWXzJNrQ9={c@boU-K_m;rcwj>l}?n@f68Jqg%Hb2g%PEFc@sy@z(_UQJEQ}>%nnF) zyG%P#R3${GZ0xu#mLWz%5(6#H5tJ;|?)6TQY`>T>2Y>OHyxJ$Ac+$Avs$GvKsLK1j z_S2p-KEuFPX(q))s#R->jSV0cue*R78s6^|Jkwu2$5ExxVD+%;tedXAPY4Zs9^^1i zZTq0bZ3SJG@tR@3`PU5fX1a*M#9um*9GENZ>_|ZE<@L_|>PFsL3X>2MJW9TdChzXg{! zJ+6x`%B}!?z!Z)bb&>iQ;w)A4Cs`PlD5M(QQv4$1aultOE3*tj-QIm1?ruMx@ukPf zFE58q%`B1~ihfy4U zo27h_57VCW`*j@Zyr_<5wJehQ-AH#ki&VMuEgG0WmC2*%ZcD8@%`K+p68#TU)jDw%++mne(04r~91lUw5B07S@;$ zjFcfgAs$erWp$~&_an8Cjxsw&l;6c>JC*<5roBfOZf+Tr*o9mv(-Q;a&)4!@lRd3Z zCxk?>R+Z0!22b%Z>)x_mbs8BVDnUI%!apq;vO2fb%gLa=*#XjbKGVQVnz)D<_&ge? zi_eS1r#KX-mMIy&M{FdAg+slbOl1+gqts*O88Bk2(=A@hCb}&Qp=~%-eKn*=S$#k0 zw%SfF;r_#h3nTYgraOWxz2~)so|4Vqss+<7v3<yVBNnx}gMxOe{=^6b!O}!z%4! z0Vn_P2xZ%Q0l>xXow=`RYE3g9i#3aa-dhIjsmiOnN|c{K+(D0=54DP(-O=_zIwVXK z1&g^g?TU`_*o!^#U31Ht5g7>kr(L3Q6G?P-Cv$G-CJNbaO7@5@iS?_vM$nPkHoUP+J2U z^94E=sUK|oyf`%b{*TY({V5<=R<1RxwEU|0p@B04O;>-53b`~8zOACy5QIB>89PJX z`1m67zI>6n(J7r|-r^EYgknD8Cij1{KmKdV9h67$D1+*o{_R{~-;6&Yd`J97<0efV zYi;>PeXOvq#P;@nXN_)4jn)+3yP~R2s&J+^@2m%`3xs;>-S*1W2~rz!4cY3^SNEmp zIO7-s5?=uNV?CRkng$H}m{Vem=Qt7lcCJS=KTC4JMQf&`fMa3`ve`OBkv znK1q4w`1=54?fWdvcAo`+Sjk##%(NVyIYL6G3~>`bA)02iJl{>V6n}kC;H($TC`{O zN_GrErMB3eK~Wtjvo3E?zGGfB;xuW9QCIVXpwp#0MT?~7em@howm@b^} z8WLagrwc0icE2NrWYiBHO%s>Yq4Rht7TN+5y%EHnvi&H^v*=P*O8~d-XR&ZfXzoy!wbV@-&txEhhm3B2ZV&R4z)zx*C1zt$ZD z$SKLsZAvdog5Oi+4vVdEs8%zvAS?KL^3w|1fjl#xq;X<|CpP|`tx7Ow&OA91@HtS(n?Xvjq zqoupee$T!C6e@8g1|6K8EgJg}C-M0%w^6(*hoO3-3ndXN-Tz=^JTWboVImFa%Y(I; zNcO4iz2HR`ZLz2{IC5|+o_gJ9y+HZ29HUgBxqXz_t$?>RQP_>@Y@aFX5E zyesqklp$msB3cCa*}})6%9vZKji~7+yk_E(6!Wg<#=&%^mlaM!qaE8mv*3|ri}j=C zAEx$`V$JjCt@Hg3QY8Aqu4scV4YO8m;{}f4<#TxRdzd-Q`f>=I`+ulaaSnV~?}5W*gmj z&*Xk~&B4vGAJeaYO4IL;AZ>o&A;Hx;Rb{3|DHVS`$+|7XwD!^6WraE*mF;z9iKm|N9&nEPMIC` zm3Q1IhH^bq>lvmog6|)>alE*X1uh<1yX9{Jnq|5|n?A4$oLSjxVsd=3PGktynt9=Kt)T zXTV$A_q7Hx^(g7Xyzk8`ejD&Ap*gFGktO8(==5~JvqYV@wC&df+GL&XrT#b} zaANM<5r>XqKOFT2fJ)oVTqUA2*D@~DS;4KvOp&b$#{3 z$X%Pm*%`-G6F3-gagKkC7-`CBh@G4T>H@Xn67Man+HsiD9d-{~8ifka855}TeJ>nd zJIKYN>G%JbyNefdsxg2FTAtzjaKC#48_B%3F%IePpa~{}&WYq{!*a2t8f&s)}Dwb(1H>uc5rTEhM#lZUee_qiR}bv% z9rXTOiQWKwwLic$ZU^HyT#I}-Sz~O7UJP;%{;P|a&QpJtR>!UBq8x8_joJ5y3Q<3rJ-%6~1%;(GJ>Z!6WP2hCUBzwWa9AygxR#Z&t?=_& z?^v{pE9*Yji}8zSQQ^G#4Ggm|r7IRVd!F(Pec4~v+Kd5Zp_kjU@=#-)-buU==}aj9 zs`ybI@5NwaGo$4zDp!7;`zAMuCje#RbuwTBSOa^B_Z5H74d9ddogzw)uXl{QbG5{B ze4T6g!wxEa%$8L*Y@_O|3)EC9_jr>xbFiJ);on&9pG6fdFQu_oOa%K=lK2-i-CX;^nu__q=!j8ZiAwB9+bWxBK)lsT)Tt~9z4>drnjS>0x_AYcs!CglnH6K>M z#FF6L1}(x!S)5~IcrD;A`~|gk(zngVDX^7bF_J6}sQXC%ktCg(GI!9vH&|l=fqzkj z{zX~C;7}aIYU=-Ih78^N^xU5YBRwt05rUBzB*tSFvQX@pGfjU}lmE9?_mrAMm^8$G zmfzE@L$l0KCP%1LaNV#}wz*9^?G|t{GhU!sP=nqDyS#|03u9AY|z0 z?2*zaLnDB=6szzbgyVzW$CJ~?3$AZJ+B`?$hufh0i!6!g^_g8}eqc0xgbP;o-XgXP zVbejF*W}9(gagZDCHC-V%77#w3YkYjJc^drt(?r47wFVrN1=(^TsczZNj)UM$%lT6 z`vF|Dy=!MBBIdaBpj8%xV)P!|bz6Q7Nc*>zgr;@&p6z&UF%*336Ka`F<0tx>$)fWH zj4#(`9oAR^8?(PlxI!dT+~?^M%igh&?1{NO73X@=JkQgEG(0{8foh8Z?5$q)8suJFR<-jgiF$; zXnj`OSBss5JikAgXcF@!T~3U%1QcA}I>k#R_i{>Ur@80g4CVv{)D~)d(K{k;PECza zvU_$KK~QLl!3ryw->=#YOb&VruTE1&OJBm z<4QMqE+SQ$gaGkW)LrB2KJ`~+*ASm(NwfQ;doIpuoY6x_xxwSZrOjd=I6d;Twu_s) zrttlfpcS3ml63!(wpiCV@ZjzwA(TwI=>ollmndD1EZNXsp}CdA5&v6R+O%a}ST`S< zoDPF$F@=P@N(hQEv7p6RtW?sLMJeY#GeN!cgaEHjwI!7QvBf}54VeFMjD*F1)0hhl zOMT>uo!=F6Fzq!rdBJcROJr!`ad2>ePfFL@wE8FPUd;s=6_w|~Qrnrz=j7z%!a_U$ zbV3F`etwth)1H6a0!!paa785tQJ9qq%txVWgVZ!64eY-BVuJF88d@Fu4xo~ZUXp4)V;rR8Pz zOGQOREv;*XKW^Q)v7tSo9~Ks-ATO_G^VZUGb$vZ|uBXawf{$;MuGiVcn+d8>1iRF{O;XvaXB6X zgF{0@Jv~b2kK?(FH7Kf@Z6w6SQ&Lm4sVyulj2b+Y;Z&xkrgnCAvA2NLEGsKxCw3Dx z&Q^>a^$m-Nc%!VWZS&U239-9d?Ekg2lt*7DS;A^9R)k;jC4qX&(9n1Ws`-tX+0UMC zkFD%*LZPj6Dlg0Gs;XkWvY*(OgpGU+_4W02b@>9Umbz<&x-cuNHD0n0x*v6|^y6B8 z{~q;a+wV?EN}_)FFyH^{u$v|7_}I(B1b~nxiX*Wi z3=9lUpUP#wd{@$|uCC6;#>P%e=xW*Ihbiy+j_}+2{au|rJDZ4MbUv>nEB`#(>h|GS zzdw*I=k*&nJTOc}mr^lMRtG#VfMS~oDQMRBL&Fk(YrM)wQ%#1XqcY#|% z0hw8QoWNH-clhwNV^HH5!@?K;=2XMO%4F@p4&NJon6Wm&<@!5yLYRYY#f<`PDgE>e zc@JgLsyc3IDtQ$J#c#nY8#@WUx~YLNnJP@#Oe`GIp`)DheIiNkX5Fx#5@*j5XS+&RPEN4l zg~1;(4W0-cn}y9y5sw``q{%qU6z1Pi;Jn-};fvC;dEDdjk!2-}B`wderbIEYv$OL$ zDWAgko>?_xEe&yirG30-aHD8L3^>`T6S0C}IIPuTf!P zZ2LitCre*ed*XF%K5UKV15B|%ptPzAyjwLf>O0v4KHVJ2)uz@eHb5p?r>RSI4reQA z?_H`r^ ziCk=x<5-<$!#EIK(sAUc%FPDG5GjhzA~XQ`o!v*a%_E5_zXh4S!JQw#{Itq3;_JQj zePx+w{*mO{U!gTrsqz|3h25?5)CQZ^>U+AoDWv%x9HY-qh;PBk6y4Dq7ti_6-m8T~gFk>Rq-F z9uQcFsOao#ArJFYCbk-UMpuvBm7Of|c3MM7<@r2@8=b!%M_mu@5+3Q0IM)N44rHLX z`zPsp7(MP=%@H(`{jQ%XW$u`=zqA`eX_5#!Faz#mm^&u>o-;(PIhWq-{aFMI`nvBi zOS;XyJ1baF&08ihUtfhk)>+mo{Vj?<=QFo15FhS5N=ur8HKcJ(lZ|uUguetq}8cbJLbPo%MrUS!AU0M;2MIkru(?>ZrA*i{z84w5JpU8d!{AcSZohaoy zYtZ|lpTXFOgX{GE&Tekc_Ji_B@dz3G{QO2qe@d47C;>St#Kpz6sPiB`J(m|~Ddb0P zq=@oQ;Ed@MKbp8mv@) zXsXsp2)!+JAUr>Q-TOJQO>|~5%CbnrFf%hlC@%l;azd*>OW~X<|JS7}z+tw>wbQk6 z&3<(4=X9||9UW;(g<=r{oInU8jb{4-QlI(ZM(?B7A1t}{olKOB6UQLwM6Q;OQr^j& z_Lj3aE_bKOj?z;Nh0oxaP87X~-;UnzyEU}lD%&Z<}thA zLW-5)--8r;^dzo>SsaN{$3#0`YQ!BMQP!(KnHJc(T>D->$h8BnYBKVSVV$nk!y1z-=i zh^d)^0R3@$rOgN-_$rOacfw7ZL@B@64qsfim5We)R7!{5~bP?unV>=x7Y->&e~+Nt9+~N9qauV zo+%B*7HW(cvM`;xlvG2ZbMLzkt6FD{n!TS**J89wQk+?SaqwC*nh(>$13Bi6ja3l{ zM-Iw*QwGYCD+OF$X;p>)JL&Q6?8CArjHizaQ{g}CRKXS3*VObrHQBDKyyTBcF>UZV z>Y(+low+UC&aZUwJRufgzt+2}Nh5V(r;+MOLYHKMaBQrEO*L5taIMH17*MNH7;1zy zQ!&;F+3EL`?sc}%`f8@{Iy+fKySSpIawDPO5govwOQut`HBpHLUnVZ+R>&4UQjNAq zGr3qTk7+LeoA}h16mxyph-N_RsPyRZk2avos}c4_D6KsOE#M{dwGU8#fL^4+;bZ&* zY^lDTKXPOLK6vL)d{mNGsG~`-8$S;{YF>4hWtTK z%e%?Uz{Cdop|CAioGfxyeeow{qIu`6C+oh6j&x{m9^5r&K7N-$HCSr1OXu2Q)feHSZIVRFa^%RGkWF}&YyIWy zB=D-*Nme16gnLabpsL{68%MeQM#`=w}};&sj_J*zRnF^90X*PfSWlDMEQH ze8MXB`30PD1+9!96*41C_D=gziSR{hDx}$L8W&MZq`_&Hv_@0<=1sOsX1m6O*qr|uL#9YH7xqbjr zXFm!rR{WFl<)FNu`}sb~wL{0j+EmEuQa5VF$oE&o{HwQBQ(&A(OJQJ zwQqPl=Ro&C+`z^0qCF=wHLmXxJp0jAn4gw?-1hvB>i{Zm5qk6iRb{H@vs1=N1Tk*f zZ7$t#gB!k>9gik`I8$7&Bzce6ZBNjk%1*O~zuBAMA7Ov@@yzr^apU5Lyto%(ZS8I& zuxybX?~!&^)HWO3wo+KAyVEo3UFq1dywiqYXp$od?yy@4(7>s@qkw=nAc0?QwpKmu zFvP?dEvApu36+m9Pj}wk-{O9%d}akPfZJT|CC-K3Zf`681CzdTsPVdSEXZ<>JwNE^ z;KDlV9npu(PO^)PKR9e%nvh2@e_REWEcczw<}Wg>xnIk6uvphs(Mse5dRiGpbhXX? zSmAaj1+A$+^lwJ81?~fC$CZkF><;4fRJ1{v+w7j^1e6&+I-tJT=XiIrGkNleIglA> z(9PwbfxDUCdL)<$gYxr}fp4ld{%u)4rrMq2`ap9s)j=QOxc4&R3amxm-`|YPh&Lo< z-{bpARiRn+sLEd>*(y+K`oo2z&wvWxiY4M{bZR9-W!%kI!+|?ypzde zxbD|pBBg6Ad|M{+wyV_MR`Ec?H;2geX8(%FRHpgAr9MudMP{$PP>7r75o^Wi+)D{u z@0Z0qQArDr4gR4BYa({gp4#-AcF%kF*W)>=cT>L374&8D8f&qmC9;8I_#RHD{q`l! zHXwn6N3gFiKFt^U4uBAt7`|7IR*H|;;ROr_&j8x#fRL7y#Sf{uHMu6_Ii(fr{Ms~D z#-g5+tU>*=Baw?&5kNq60U{p_TpZ?dN?vvf>-aT1H|noRB(LyqDcCOOxgl(;_Dk&w z*9hv2(SNdpWygChxmFNkewj9EJzP*KuA8SMCV>dP{e{t@tKQ@C0@<5bc12h(CM`{V zj$@?IZ6O>ak<%ZR_b0%1#WvHIiF3uqsx0i%Sw?aBVt$$W_JfjHG!2HI)nbab#grTDh6NqY*VfBU``K9!+jHV z@nQ1Q?Z_17Z`Ac>YhtF)=KMCK-_)nr)y}F7ocZ8v2q4y&aEgW4ov*k!J-KW5kI>$pO>J&Dx z4Sw7Rv#D6wd9;X>(`_Q}<=}UA1H};nY|tv*R1#x@F_de~1jo~upPnC($3+kHa`m^L zzGV96_(b>O?)Aol&h#FcNa+psW6Z%k^T5qn?=AScX6x(Y2V;$BRYR&0O6m16uHg*Dv@%suYDNq=Su z!@weA%}>NW^Sp-q6URIa!m9O}h@oe*E4MLVti#D6t@CR- zO9^xq{aJ=*`lem0L!pp^PEBL2vsDkNxhOhW()Z1Db>{kxGM8yZb-GMio4AO}*oMlvwPT~%s^OEt z$gfGwjOjF(Ez;SUl)dGtv(|e#ep2vk+|?$Du3h+Qvyx)pztMkGrvUGpDdeD6=B?|z z{TdZC;p@;^w;$u{NniKjFuEo7*kvZpljS^R$aC)7J`n&km-2Powu{`AU7HMUM)WsU zm`+`w(-UjiXX>&1^E*-Uway2>ktZ85O6|ff;gcwpOJ|6c>AH!6-evoVvbME7$(JTN zpAR~o9ZM&YfreY zWb9uvp%-78fXIpe(ZKbSamS^`g{?F)B-ZYu107*k|D2?6b(|ZN44V1HH7a(=T>~YQ z)05ku#R@%dlQN|_Qtj8DKdf>MSL17KWqk7+iDEc@IrgcTP6LyD2dAw71lrQ*V180$ zzascPm}o*Yx+v}U*U?+Ot64!z`oNZ7Ciw~s-1)E3)n}P^N&xfdBRxQS3|FW71@CEXcfw<_ z^fSrS4XF(GI&5?~-O&bK!OdyK96PGkVOPj^=kpCIF^|25uNf;zKNEy2^FV{Gs|^bgY2AO6++p&O?A)1t#vH9`n%XyD%< zO?(x!5%l8Vs0ukXv#~94n`Qhp;@7c9~?W$j!+? z!*Ef0AHmjQo%L_@F!9>um?*kGXnZC%!Y}v~&47xOm|8%d0C+b?*o@>*{_~j%AP>3w zPb>)hXU@D^|AB}Z1pONxi26Sh@8SImCO-+~b(d}*y$#r!0~q-!Jg*Kp-M5{(^SyKj ztA4H=7Z3&UlYLMCREf}rD_DSgj~y_0cRG!x4tc%c_}9kfYejrTT!_nzpn5S>|6@QH z1mLpx34FxelyMz?m&Lib1Q_A!J3Jl+?A3g@xR$KWom-%{&W#)-tT%z50-&#$;rDi~ z{d$D4^eUIgpuFPj8AYA3W57_J!ttohSoZ+fYHjW>UXLqVHb7ptK#I)dVj0E}xDG!9 z7FbY*z1a;qklLpNk$OqLv;?W&8dU&Uet!nE_QHn8&ty;P>&i9I3pQK=H_p0bm7CWD zlXr5NPvukqe@<3quUmAW%6BJ?GrY(PPRp2Xg@rR<=)*z4c&Fs#0E;~96JV?v*RCAy z{x6sJyFn4|#;xRqz2wNyxEb`d(WzIcIy?;BWOO(>WTB1>mC z6r*0TY>(?J66h<*SvZ3v{k7grP=;Ih8?p{q_+N{6mSmEvvc^*HthHwZfBeC>tqnuxt#0c1Zxm1* zMd>T@Uiq+t+U(mHoH@DXRVOZ3wob!d47}D*#I}(7aL<8_xJ= zC91H-r7C(E2ad6P{b|(B5D|V{^Zcb2c3dXCva({ZNjl0%5F6Yg6{7EpKStm-qQ8=m zjU!c{sHN~lF3qb|-3n){PV<*-`+p;(Do2|Xe`AH)oeD(FUG ztF_bAtLzlGKe%Qz`h@Kj0mEcf+Qsx~Dm!%$+3#)`Vx?t_Sb76b&QA(mmqY?+eK{tU zvdd`gnM6=NvPsz}Z79sj;{uD$r}cAqrmLFYO5p?g^#nMLzh|YUdsB=fVu5M=WXh(R zd3x}TSyQI3pY|LPdA!AOqBHtsB+8+o#;vwN{XZoLl(QPvpLxn}Jj?WDTyu*DSZ*(h z@?5l`>DttF(2K;?x8{jLaSW4_)7H^LPyqVu;^y36cF~|xFJ+rt1Cz`%_+aAH%fFtG zO#;kre{eM(7q=Efz;HGb)RbX?3e_p?Nf)_Rr*^spF#GVYsy%XV2XJKSRg0tVJlE2@ zk?fR%Im%T@0|gj18vhZ4w+(%8J}U@SxPL2k-<9f; zh9pHd1ES4vVB){=Zyx{bvATVjKqqO2)N+_fyb+n=>h$(S)c4k zis8muB>)0;h@;O83dUAEkRm-!II?Po?g6|v9JkeqEKoq2eaGET0)_*Dq{^&qciN0W zYI4Ns@yk7!C*k?PpI+^$A~fV(vq#d*04W6eVt#{;^;X^RX)!0-=>{kvn%5ijBL8Io zKAqqvQIPnZ8}7h=0k=3hHsrBNHGV({E=(9JEp8YujsUz!K>3x>t}N!+Obr5l0VLwD zCvCPkH3^6gcPmpQTao~YgG?y!0TJ}zBR*Z#nTF^l4jmalcwJoW&fdUQ)|H}R3m{zz zQgYnWEr9VIAPaHjf6aOWbNhGE_}`GI|34sPwI{&>@zP0Do|K?}>=X3<+Eq6%n)agL z`~;*FVmML-1xPcr;p;%K<6hIvF!*&8(um9EF|F1R` zIGDczzrmhSi!F8Vx1UP!YCXJICY52=-)j#dhqxfQn~SF0MHy#XU?!*j1H_v~lMFBq zh4q@VgXwVEjuNIxtPX6xU|{f~k3-L}=qelOi^-fJkdrtXu~j?`>p?@zVh4F&mFY=! zy>9Zs9_UUO_cek$tCS#Em+`N4+y2xP7O};NP_O-MYy3CH?!4z74P%gAz;Ho7r`O~o zXDHldHg>gq-CPd0yqFnwG$M-7h)c|b49B*sUzThB{kK0BJ(-V|_mZ?@FVg)b zt5i8dlJWh0tq*(qnkA$%6nBlC#v1q=xlAv>dI+{cM&qQ2i=|oh)PkutTEp|rQO}DR zyyA%$0xT4V!97pk%`x7se5_Sf+0>_AQwfG4b z*zub#<5lHiQ0es8rCF)hmzanUCIBKe-gJZoyl|=xAdX36bFat&bFA)toqH41Z)Pj( zQBbsR&ERlH z^{?4C>lX$dJjO1*K1=(y=zgfok#nY@pM;HdCltwx7vd!^U01i&u{vSd;RtJ->t8hy z)EWB>#unQcW+}hOS$%qZC04t+ly7;BRDF538Qx^@wYe!EPCs7v&6e0%$R+uT7_U!P zBTUi@b~#bF7F}cstImbiLeRG}EoN)Y?c#T)-6eOYbSDtc{okS?`Z9DC_u0gRci!9` zr}c5?OkcM8v@&FNV#gVN0l%8vS1N~FCp}bL$xIv&(?UF=Im5>DzrlHT@Y9nx>!tWt z4(c5z`M_IEo8Vhu$mqtH%hKt{=W3V<*Xr*%ZBzL9VnKGM7*THEms?t$5XXkiv(3S0 zuXM|7V_igEV~BAN@2pJTA3uawJN&+vj#!US7`GV+O1v5dkl+(L;?A7D9P@35V0b_tA%*%_4^@_>qU_7pt`i&-FLNyM5P4FynM4CD72Q z$S}>IXtpKWEQO63hXKeqct0X?@u8<2Haq^4Z*$Fui4ECS}Tb$}xX@Eim^^LepY{t6{Z?PmOx)+0#W1wyyl}ul5cOF8Ck1#>aKY6>VL@ z8H1p4Nd*MU`LK>A%ft8QzkO9}3fD8gJg>$~IQh@P4`WUAxlVPW33|aqj~N-w_Gaqh z#29en3m8`Ya?QlW{`r3FM1{CP#Di)6K{-In(+r9IygPl%%}&&&2QZhEK>S~F=N6gc zM_aLp3S*WEn3Y_DKCD`WXEK$fic#89U~YN6E9E2j_Kq)c&oHR}y#OJDn-82!I^j(@ znW9h5L`GW!mH4Xb_6Imt3T?5w*T>>C$pc=C`zYn60iL=N$pN2Y7XU89XSdK(_;Xdv zRH>TQs>r{gDt2&U{_+DAbQ%!o?cM*$4O!SX|Z~4_)FO~*|}cw-`!INP^M&|5uLvG>FZ`LqS4es(w&&&pXkDsT&lZw zbz#Y}fjia;VGaHguf7%e;jwQ48EP8gZj9}46^VJHd#etG6D9U!-_&(s*0;a${{rxQ zlBmDtlQlE7&NAl~z^3(dQ_dnBfk;ju#Uk=Y3&m6Z50my+&!r~mql zcRgXL2x_uT#l6sEJ`~$l59g1b1U+LYa;1n#O8H69HP=;kBi$Sb07wL+N>?*mO}AGn z>V?eNOlSDR)6(gM*Uny$%$H*-)yo0x?~hnA9rx)nhmgkp%C99Zn+@xX4)=ujY)si~5B@C9(}g#uE=JxpoO_76*mLVDMCAEX z?m?m-;f@sORaVR9tVUfLJy^=@GJqcKH~6Ewe4I8(2Q}D{I$O0me{>M8xvUFGoSeky z$Ta96dQ%(E4+c>EI~rii21kd&h8)wmX6NO_&S%L50}y_Arpz_k$r^7vR3|Zby@YP8 z5xLefYxj5ld9B%El4h4m-+Zb>4mMRE3AL+1H&d5`e|?I*XRwzzUEm%_sQ>QtauUon zZlUS^Cxtt0|M9;t+Py4n-C&r)$n}vvxKz_odoGcuiJ}0TG{NTO?yRQ~#cLY;eQOju zT=CxboS%7mWp7*Q8uUur9CxS%QAx6yVZ79WVRA=P4WHoeI_7*|zErz$7rA;OKOWr<0nO2mynoB46cRM%j0*`aSZYPG6|IJR8zq`Mn30y6RhBxu&V zDI<;`SrVM0C*EqZpE|RoR~5%!i4tB18Vu0Td2uHi1n@6`nENU}1{AGq5w^CMU474u z3%jd|*s0L0ZyfiuMrJl+rWJ_dsd1z6y`+iryc4|vyi^3@?sJ*pEI}D0b>{A}IKu9# z^t*bYEbu?Z{lfX_KUA2&KLc?6;K~c4$AHe@stjKLr@9i*-8#N4a`EKU+cGf=FcD_- z{(I;MXzgtP0fhlA;ThGxkCe4uWJEmxTDnLk`}ZUV-Xhm-0MD+gVgS1K?`{hUxc}nc zGaqQ}=2g{zxR&G<5NM(K_EnwtjIw~P10)jKBYJf&@K%amtY{J-Q3s3QCw2ZV0)B@1stj2gWT*3%(e(|>T21HG?ONfoWD zua9$Q2aS?mYm_6neOuwObpTgj_V@njRmSY??a6+%Qr*_vxY`X6s1hW%Aox4r8eWzL z+ucc^OijhPyAT47XceyODvZ>^jz$$$echrWNtT<#*+6Ks$3FoyDgc*nDvHU}my4vP zq@^v+%&Y)3>g6h5tiqzA=mc#k?vCZDn*$dJ%hX4GF3ata07RZ#y~c4Ci`>>7?;3h0 z=CQMnL^A(m)65ZgKbV1Q>OPl|*X0G65xWg*?(OZp>ap`H0|3AWvb;bp&pq{XUsVK! zgs7%G{}#G6Rw(Q+RaveQWp#i?n^xIU-t`9Zl>#{6q8Lrqy}dol)EfchA3-Kh1nmEo zzH9w#nHnyAn}9(6vUPSAt_w4&sp#l^*i8z~Ov%t-#FEl)_q&*(aU zgTEde6DZ3FtYTZ+&ES~U*#^%Myp#4w8eebkSa%nJo_hD~;meJ`Wtpj|#*+5T6y^dQt?}MdRA3pHh9V^lw`ThGR zZWkQqn`miim1UKa1itmGUTxprj7YWyTdd*%d*UM~6v}1V^hS2IJJ!|Jm1BG5eOzd0 zXb34i`L~#un7}|8eHQ!$DdPWwJN)_&mE!*`XZoL~e!{J>RJJI@*uLp`!ot`rNElnQ zUaR?dp>(G z)802XtoFLZ$7T0#ch&k|aEG~wbe^Q$f#kGzR38l{C=ScWxdJz6!XDmSaEXoquP7Mos@pDcc~`)CoBSXI*KnEI(Rq_wT zZa$dQav1kq%SnD8IZ_h^S*m#d-mr!+$HxVHMCiCHcjj_zsrZAG1QZ0iAj+Qj=~Q#b z$gp{~7l<(miA96a_^gSQ?+|n(wgoo|$kHce@JUZMR~i_NGof6Lvlvz(hHNG-RBxke zvqyUb1Y!|yYCo6EWO^(@-ooE1G)LQ)=o3}oS8KEMwdIM93Fq)lg;2@r$#rc%4O5j38Qtzls6 zrJMcrg`Pz2@Tc6!t%AFOA#eWf?~iz^!{;PwO#&}~U=im$P~n=2A!Q#hVnwQ=e!@v~ zb)^9cPXC-U!OXBXe5OO(XmyELfq#g#X$xOFz6iVQZwr*zBAvk~&Pqs#*k}gI{Ar#O z^4LL5FdMTb3cx>{lxBSW&gb(mGE0RXVQco%*&02U>zz(|fqXe{XY;96aloXCc}s-# za?a_Xv_-~xmx${PrFrYVYvLAg=Oil7vCl4+a*k03q^PbYE zTm9kB&0^kzcldIGE@+cd%uZ_!`xB*mD9MGTer6f*E#y48)-YHqX>K7;8vI(YVBWR= zvnJ^rp_Nd3_&z?%get8IV7WK7U` zZ1H|1AxFKnsc7Ath*0_5{)}Ly`vZF>3Pu}&6z1_M!*eDkB0CLWjN1Uwx6`MPiaqCo z#C3FTVCGy71bBOwPpC&hYw9BerVLvOu?~fqT_GhVf6zjHjpy*`&wr ztX4>9Xrdqu{C$^d%+wS`JsVp)*Ta*HDj7L@?{8LW);rbsi%x;pKQWri44kj6GKDo) zm<`INLx~p!8@RG(jrd#+kb3%63^YI}&0ujN0&jYx*FByM=*K z<7P2EX7?ggqd$LE`XG?}Zsd=p{uydB zvwDY#&ubT@wG;sB4p*3Rd$rHf90OTLo6n;n*^_(t_VN-I1Mt{vl@x?-zYfuN{S)D1 z4;fg;@?b{a8{T!cI!CqTAwu(6UK=(esJm@*&BM#XUDtp1h9 zSU7`kn}0Sf_C0s%I{emfs`r7(Bhgu69hDv&Ju}+&^{%e}N>|nly&Pi0rdNK2=b({# zK<76Jw}^Ax6P2v{jj2m}XGvyeK4~0*-}NukU0hX^Z<`*M{!>n6=nOq%d4QTA`%i( zB140KLpMqzodZ%5f-rP9igXG{4~#SnG14&%C=J5UIdpf!Py=WDo%(+`AI>^o&hznE z_pE2HJNDkQ_jO%&^N$Gzja$X>#kM7;$=alZ;*qh^$F=lGc3s{SpPuKDCL6P;EIhp) z7^SO1q8K3q=gCxkn&K0~Y2$axcGXS+oZV6A6T5fRHfu;Oz#n2$iep8UIc-~Kfd)a8 z;_kGQ2{Qok&o#ic{+nH4BhLL+=Y{{nq@2Z2J^o&A=$MtSYt`ZIDeqR>@azM&vl}@- zQWt&fxxz|gvtv$7ax#r4Wvc@(9diA@@?fCjOGfKuB%6?5i+Z4CdqTrZS*9KGXg$yCoX@<&kXSt9Teo6Yj2udb=VMA7 zr?dG#)1r}=R`|o^!>J9lOg&Wva4BiLd%Z7 z^Rv2`9SiW2Y0qYus6mhlY*NxSR^Hb$=hnX3k#PxX0Xx+~AkHd_`MNyE!fw z#XU|3+_u`mgMFk4>Z2`^Tu~txqll-9)=+S&>-2_mT2%Gs>RjJTXx8PBlbPjQjSowo>=43Em=$4Nl?h%h%6?5u`)i@WHn@Qm69c1uj&ZF zb{sCj+jZ|H8KKjXy>;NdMCwiXV8ZYl?^vH#7dCbdSr%)RQvwanBf(!~^dRQ)e*A)x z9LBnkEKuuulj^1a!anWvB>CzB-7zxpUE&qePq(kyfjPH>HhO04K+F;w`A`c$_)t-$ zH{6D*?rvx@Z?x*6Pnmw#Da?aTtXYEzOzOf~+eAbwRbzv7y~GOWGTu^|a*9@4bAy## z{Z&F|VkK!NJucj{u2pH!G#e~usH(`)Y#RUCPqrGRBgHDXmu~CoaSD(h4PJZd6N(Jq z2c70)Nz>xK=A8xA3F^xuW3@?b939e6D)A#72*Bz}KHeM3y>Q}3hacH@Uf{EfQoB84 z@t~4K%`BBR*Z)#e?JvHb@VfK#us#UZo6R!QX%op);MZtTXLzm=Eh3Iccw0xh3_;d) zW(;%Y6Gwhu{r9H((yD3i7A4g4-PPKjfYe230n#&~k&T2(U7n^r1}wjt;YE_A&+|j+ znuBUeAwXp$lut#_W-^epaL6WMJ~qn_-}LgL$d}B5_3P7(S3hb_o^eJU(=}YFZn-@Z zJ^80)y6Z-Ici*ASwO?XDYQ0_YVEbbmurdI8hLJADe@k+4QAB&KPtIS64m42+3aT*A z9fZ_y2GXd1o?j|>Y|mJhpho$LKiEP&l&a27`T`-J9mhQ`Kd zMy7<7`ddRZ-gv3>`#$Of9UJ8sW7O!(07KdcpgAW2oB6QdDPlp&HZMn)MUbeT|XU=Ub_p zr&m4M`5fEox04=h#oawiTiAX54w(W`>;h% zVF}sBg=QarL8#}sGld1zV>u&hAL}Ks6%iZyn*E+Wqc@{jk;*Vj4e^q)q#bwls1m3J z3k}56#H0P)4R>z3a>m|$rKMxMT0LDVh0@equN?rBRh%8lZ*1+Bie>J}>gG3Kt`|vL z>3!&!gt&`RDQ^!?*9ih>1)ewSAe|a}i1`o3;8{SO(2M<+n*-k+|0#1QR|zwC7U9=< z@=%J%Dm5H?d~;VDJKU}o^v^2^*XlKR}o&~Uzkq5x;43{!R&i$BVMncmtrsF zdOK6TLlx&}B9tRL;4AAzaVD74nNMF{dFXi83`Ma_KfQACKTFKgN=@#n>l(?>iGy5M zhYyru`Na{*!Xtt;o(`D3I2%^_K*=?XrKJZF_~T8+^kPJ*kgYxHXjoBrZ?FzOe|tPm z8l!k5C!?J^G56u#cOqOKQWet}85Qn#F45CGttmNuaIN$^y$<8(lxWt_Zlw?=rmBQ& zavQ}q#K)GHUw5*hAd;6V-MR2JqY1wCqyppF+Pjt9y%LZ=F;XjXw+VCpZKIyS=94|M zMH^xi?SC&x`gXK#kRi}!kS={si&L0W##OuOjVfpI>MKvy1ZC<+%F4alEsF(%YOB(8 z3qCh1k6SAPuWu}Tlw;?Sunk%~7lp1cx)ReD)2`bC-It}f``YWW#r%A`r)ajHLdfp> zTi3$TKAm$`*d`qz?!9kLq55nCa1~g`W{(%caw54z0^j~XHLcgzq%q7M!SL7vMQ^_k zBf>kus)(C9&os+Gw@Q@^Ys$DoMv;sA8mdK!7zR*#Ui;UtGJw1V<4ZqhkGb!kxtSi6 zKV`?=wfwQ492C005CC)=0Wt(mZMZ((5;)t>zLRrA4NXNbtziQ);Vsu`nznh=4^faa zqfzOFhE`3!_Y*bIV9>0<$@^WuJs(~F-Au)zyBCI&uh)9J%(1Bf=cN;mnb@>u4xT2yFFQ9Rl>$9)gQEC_V7jWh|CUdkvD5- zq;s+>NY?lZ|4ap;Bp=1Acr&=4EW)wqSn1b!? zraE=W8kc|qh{IrV=cY*7wYzU()tf?vP$bSdl|V}a20YC)Znq8ssIukOu-fxvZL4Cg z3c&!igz5_ddAp1^s;o&cN#|stO*A6WAVRY7Wb^v8y({Zjl$uG&?rh&s@a|tF-GYTF z_^Zb62re;Q*Rn~U&C`NAac`E_3fTHanYJWJesp8gnX&(NY)uvBpa`2%!=8MYCD6(P z`~&yG4-1p;>R4PEO;mT*sIw)UkHnL#ZF&!RZaFuKb%TlqR?O> z3_pF(E#iLD(TJX`i+2%LA>PhUAyOziF&Q(4lJS(kchftHbkZbj7s56fxWa@VdS+=H zI4rjlUmL$;oLiqaA@puf&;R`s7#qXB-gZ+Mhk9X}w_Y~oy1QQLedaFaOs6QpT^l@&Su0{Z|81pv=VstOAj{aM>yS#_9z_Fg7hmB_S;?23t4l7GHG!fk{)+_CTbj6^tbL@1q| zem;;T09AbMG>M2%YTa7KffO+&!os8ZpWP7pyw4|#Jh;)WaoKs3ro35^E88hj(3p{m zgY!qaq*^nA3tw|MD|O=M_X&{Q-;;N$&PTTNEYXdd0p8xOJIB*ZenMWgb?#fQKcd@1 zMz#h&geg3T37JZpGXBTkUKR4D|5uMDR@zc|PilU=L*{17L@^it%Dda3{_}W#M1;sS zBLe=yl&_}otH|>C5>cxv?L@5$i%RBu$qj}l+Mr_C6KUj6w*j$vl^fGP#@DYzuNL=F zM`tQrzHB2IO&wPHXAjF~MBzE#i7{BW!JUyto3VxmiV;(yM@CR3#6Lk5yL!i9MGB>p z$KerM;zD+#6%PDVNxL7JJh!EPrb9|(;HCk3oPvsj)n;Bsqzo>fCa2{9%S=xX5NR@3 z*~k+UyXc6Be8CoR&ySd!mVF@@2;@J>lOga^Ho!0XG10?YU)EiIS-ry2qCd}FtWCO~ z{2ZLw27=+g=~YFgwyW+|w5h*NE>5IHLG;X959+#L-9bUEf^r25 z*1Z)hCJ=Z2TQ2yJ<8RagvOnYyD_z#4l@VX~N%yO)99dCdrsKTmPkWhz2_o=h*<7*O z!ojVC8>UCe#`I;OrCk`_iyxXKzk-(d)>%34WA<7S6ks;j0SxRSr9AW5SJ0^h{&Ifr zl7pD$V)%i3V&N@Live-ilIP@GKu)27oEm zchPNz!dJH~3`hw@`u_AU7D~Pig*IQ|>-0nk2qIIDrL8^sQ8ao(T%#)gld%ceG8EUc z+;BT}tT;=Of2x)_6OI2->4TE3YUsqT2Z|I&0*m+IkEh255-rm^1lQ;EzHzQ%thKw+ z)Kj^EP?wOl(S|k-_VTd0I)#{vkOth=tx6W&Lg(d1#7i{qxJx`D4G-Wc`1ZC`DyuJG z*)w|!v{(7zdK`2hbC5k#X?L6UB9=3Pslq9JB4?mK8Y+wvJ2Lv6!G!8V32hJ?Y0sk) znWR|73f=n@WQ|tbVTzd(W~!8;e#G|LeFvT|0Yvt(97p@H-S6INBV-Y~PT5TQ6(-#e z$h03UG}9Z3@#hNE{x*N0C#G>5kh+kJ#yFZuH3n=ITMO-e{G~z>nPm)7aUvqXD-XlF zjZOs$1E*{z$MZj9TQC&4$0o?kEy|kcZzQBRfhY2WUr9YbWaiGzG~U7+c3}(ngbG&l;TdV>cq6ht_PGN zL3nTfF2t?m1E*yNb5|Ws?**kk@u0wiaqMRsqu$SjndDaZyayMilsyT+v_imIS6BP0 zz;Jkrzw;8K=ye(Juk^ATFg<3*H%QNUC54=Vj{Ny4tNvTn?k4L}sTInbN{s8TyaA*` z!&s4#kXhe~nrkRqh4vpk)J4RklWW(jzWOCC377Sh%a&mE0fC=_sQl)7Ka6Z>Pe~?d zugW$~utJdXj}-u*&EbYghkbtsu;4R%TYY~_d6b5ZjLcNJAz5xtX%D$g<7Bm=b zpDBd3f|KKsklwqv3qEa(giT)8J54S5nK4%Ftz#pzk89jUcl2B$5Jw13i{1;`ER8ZlQEYl;&l4kS-POLE}{S-s)$S0 z-r16e^fbyUf+9%*+|dab!?;xE!0s=ToCyI%JOm3ca-*)<4y(H?qq<$mqqpCLkM{Ks znL1$K&XC5CtkQTXl2kz~rJBUts;Xuo^M-FstI~H2Xkdz@m~W@%cyT*hy+CR|J2`Be z#oC}Wu3AZ{$@$RAJ&p>nAi7Njv+#BFFnPxvJX!WYQP*QRfo;&u+jkWWvsku&;1G0k zb9oXDMo?4U;mPM2hRUjv{{+wEKl;j65tNOsgAkuhyYQsvd(6~bjFB|VIB*(*37^;_ z`a?7HNGn{J!KXXAPZ(3C{l1km797jZ!9Xu@p~_K(y96_4*$ekZ+4K*xFoa`&ci|4u z9lt&Bw{gH+Vce=-QjT9{PW1HA@IzLC3bVn0o{6$nj-|OJz0CW%%kPla0_^22M0gJ2 zc(>AT=JU;n{Sr9c(PeA(6uD6ZJo_?oawAP@w78O4do6#eSzP~feKnBn<@JWMA{Ta5)p;`f=8?Kb46d6Pqzif*0gF$Y9sWLlkSP_2jy@I4pnwzmWA?&8&0jY1fkf}NTR1Eo zE&4*`tBK_Zn+A}G6y63T$+Vw9#Y?hfVs1za#z$d2^kUw5!gj)1@3B?#48#Y3P>i zkq_$mtse!th28icl?O+#vz(5_2PilYGnJZ3t8zN8-NWmFm6^#A7clN!#l-w0qE%1= z8abW?9w+9Ha;kO*NI|9a4W3|5tB-CR zmJpY z$@ovvo=jA|DXsfbD|DZFmD#ex-F`Ns-cbv!pGumfyICu$q!pvhE+FOVMkT8t*KsQD z5V3foWN;W&s7;1IHJ0NlA)3hf;coT*i04J%E#o|E!Eo0mfI^QHHuwgpb|B<2Y^ge7?cn`li$k7fg68C45OwtW84%Fbt7=cbtGY_Z z1V9y|>J{6ghqOz+h6#!U1Dth;#UC+si^b;p)J zsKyL%S<6Q5P~;XFPX=T!tkIu5@Yb)lI53q6q{*;1=ydzFtCs5IxC1rY5U|0VwJ|j5 z@)kX5*$(0osc@g@=gh_Bd~&MqTfO3CB*2}j=-d6EK&%&10+z?U*hlYgTni^bpuzI=sgewV)55zm68%POT5?gv z-o;FgoZoS}PtomOj3~f*s1KnNcsY=vj+V-VLv8}>Q?KG4D8r!=PyM@-;gdf7PKu7u z%lY=#P;3ErArHfJhpF&Y&Rw#ilg{yc#6`r=>sa5nWw=11yk_p_lw~H2bQ9@!UJodOBb+f)HCNF0)k*Zi)#WWw*@V zp_UG|Q@2%+9BzdoOv|st{pw%I)h0K@NIIYWS~lOao9?wRlL{mwWAZxO6kUT?*AvIX z6Bab{v?hI2uE)7}AGZc9z5jK6uPBlKtHO!K!l{Wser;(jGtcsUo241oMGOLNW;mHA z!y#Q-s$A|;xcaAvmyIwtsurA{x;L_f+XDQfl+xKPepm6mk~APbEhJ&AE{#}|7k&Nm zR95t#)+)|5v?CGNS!q|qd1E~3kD`bDVm_3Fw1m%VN5Z^dAxpX1Lz(?lhm`agXq&Imj5{k>ctcj> zwsPs6fQFo^e!CwGM4@GyU-jWpecerkchrOj=5P{FfU-x%&F@b$3Ql6-kG?`pcxv*k$5#z{&FOL=-(hL zr8daKy+*h`BrVEbJ8=#9y=R70L>46Ly+03n%mGXQa@+$pR)$teC>HYw4s(}wcj$h} z>{g#81*Cag*{&D}?;%Dg#j!rw5p>MfzSC{aQWrr<8;~a;GMmlW2A}f2v+_}m%;l~T zm)5zMwe)=BGwT=Q+YKpghlo0;2p-Z(EasaNvi z_Qj!TgF^nBIoQcjd=$*mD_-N3$HPO0w&P4pj?c+k+*ay)^7o95DOyc78hzEH8dRi5 zdr6hZhD+$b*=Ud3ZB68Ko}h0MHt$YZ!^rog{V+xv>R0Nq|A^}6fRG{==yUO2*ZLNnCcR~{dxMwzCkRaT z>5~|>a(b|!!(4uqi&T4F=pUJ04y&bng?VY*eSmL1xo5P*=2e3O;%C+fH$j(Hg}S`) zH!W+{_EVwk*-8+PXJ z{MOjoH-9g5+W|aUq?D4W)z&eAn@C>VOGbNhfPcgBVOVGJDL1Ds zcowEikrbpNxDZ3^_#uDYABVW~e~QEPNuf2)nC1S4adu~UerhI^-PCZ0MBU0lnQdNs zMD>fn-S*xDeL7P9%~ola9m>l3$#_TW)Dxh4*Oq7<)PtRwd#AqO;&j}mBb``01@>5+ zuOu7T9icQ~ttJ9(j9eZl3C*JSsPd;B_1M2XxfI3(mT@yC))}0=OLker(2Diph6Vf6tD6?HjWcfUoG0y(vU^EJ;% zd&L}LP?ytego+eCw&ROU*2`tRR@#5Aut8#!i5WYZq5ZnsC0Y}WSM2#}ApFuQ^7S`R zMRd6C=%8szQ}Q+T+i(&DcSF%Dg?U`Ah~JsXFOBgOT&;3)oU@uO^aI|o z@tJP!;x_*G2Y+=8ij#UORJ<@-x3>w37!#!{SU?=mn{m1Lpn&{K^##n2-(&{rhY8_2 zF{(H6W-@)*wQBt|nV#Dt^djQd=WoQek#D=HFuOlQJoZvqMWGc`WN@C%S-;Lb=90)H zuwcRtKA}02}6*}6Bb*L7HHl89D=7AN*@n+has*!Z!0m2KMLc9GL9#jB=_N)7Bp6(=( zoVc5>f_?kW82ok{uLb1ocNg{HFy*!rbiHXah?!vb`J4{MWP?p5#Uw5vP2bL-J`sg@ zD7+uWdUzBs6m;T|{65uM|IT}u$Kq;KC6|P)^7R&o%6e7Gdp~$R5>t1#ohfx<`39$| zrt_yD_qM_37ehzuxdy-2Ufi#8vI%yw(MoKiwB%t&)4G!U@3TvT%k$(FcQy zH=}RkabfVnIrv)!t~*Q7+^3UD3iPRE%-j#U5LqmaX{mQ{))Qs2U=29$2)t=IeQRB4 zHc+0}UyLeN^P13z$Z$jswQii^~iG1Oz!h-*XZ z?2CqLaPzKQOhoVdvd6u2Ke+8E-}&N89kgjZsIax^!zJfxXoB$j%OlcD&rIpE7rGSs zBhvaE&TCiu2ck1L-n#h3m8L~-Nng2794(%P5~wN4zyBW?{GyKqkC9rfQ3B-lgt zixBo}LBc9VB$HGW;C;InrhXOE;Ci39i_nMQ@Cf0%4FqU~4i~u&w8l>lw|^#8-sTNF z27XqJ*NTj&-frAei^DT;vB4pONR!6Xy|_ftZ2ldn`4nTVd)|QMK99Kzv1`VQpJ#h# z2ItO2#fI!#cv6FHka_w;GC zaCs#uah`w@4^-Kot|9wK>`ycr_Xp@}U<<>JGU?w5F}T6Y-bzcU=T18*Zs+jXGwQ)> ziSQYmHSWfSLo$mbZ+cC-B06~ZD{MSt|3g(!3)jOz*~en450>>nZ5I(t*F3B?m#v4l z4F{KUU-%t1#$}`rB*9NjkdD}^v!I}z1Ta>d-qKQ(2@ooi4wgfCQMrh8Hpf`>=QqAd4Tm+WZOC8n!+04Kuo2S{ zz~#lkqnGX0<=ZOt^vcSC?gx!kP63rO{473bCdi?Z>7OBgFr|Ky_{k3Fhadb3C*^p%gq_I{$f#9*>_{&6F;DXGV+`(?))4UE&b~v;F)g9)yosPCc9i_B75d~rCo^1s__A!pXvQFl-Ix?}{E=K^%?*_c3dIjA zKAt_QE*2zY3E{7O#b*7zt6w5)Fx~WWlo$>+D2#&K@+u{qI664AX*Fz*N2l7ZrtHKi zMg({+jYQ)o&P;&UtiR|5;N*~K7kvrvGy!rb69ghsnc{D2{@o2VGzlodZ@)G^A<}wb z;vKXyL{GOJkm#o`uXiQUy}^WSXW-H@PTu(cp%uh0*0^|E2TU!^76C+NEB{Ix+yO zGirC`I}|QiC>=%Y)V-ueiy;cnbQ?@?R;*k)|8RsY2P@newG0EU;?dUx@{p#m~PGS_p`w4$?Z?tKi5h^muKws5PIl^KDn~~2y06+9LyhJP z*%|wrN1cqU8))79XXk5}5KUiio9KC+S!37j4QKm)Or=}+KDdfe zia1GMoNm;>$-ok~=xwkE43C&W?vo2+OW4?SYp=gQu2{ID5SQSwoA=lroVfw}fk_Km zrucJ|3Al=8h-fD~>p0E4Hp9Y|@9{fBFEOW03}(hq#jfZ~O&t9(kr5{yV{=`S9TJg0 z017Lvx;46VHqSc6`0``bY3Gj(rB(h@aK)84Fhy3k@j+fksgCN?riWc3Op~V3apC!# z_w{Q9`Zhh1y%t{Z{wouuVHTjkT!J`E;Dd&6$Y}ICnQ3+1V#Tm_W_KsAd93sZIY^WI z_LO=g&aa8+KguzKTcx%aLJD8dTXd2B#bfKQUb(KdGk{ehL{7}RwQ<2Lgx=H@kSKI} z7Rot^3DTaOsBi@U42xoiMWwHGmkwPpLTZOKe};p8yK~h+YWB<}paJOIkMBsn^WN@% zVC^49RtsWYU4;EOiO)B- z?K)<-7&vsA=|d=1B6m(y1qbrN#6Ar3-qaf4;4emAJgHdzBK3uVdi9rvM5*y=K>m}o z?Q8eVB*G%>L92f%K(NxKVrRN4JL`p_oWTAfcWQv5W2^FT`A~l2Q_{%Nb4cxvceut$ znQ`DrNyi$RN95f8G)t$OS29j%yxPTKBFDpnzw#JW;Uz(8V+&>)kf>ANyLF=)14U)+ zSoyd#;g0my2^T9c+Ipgq__v_|gP_w}5x5chawT|CzCXAo3sj2qa9t@fr@a zxKWPjCpNg28<+kGfH|kV9r)4;-Kt-_Om<6sf!`Xdoc!ESeEj&+#II@Q$%4v=bO4!X z>{L+e{*h%WG^{h>i%0o6rSX^0bi)h4Lkz@;<3r(#_6_b&MP#5-o`Lhbw!fhn2tD1< zDAI3jB;2NOH6&p5DotdGq`t3S88PIN%s#^+@&kKcj$2lF&&{3 z@t#yF`?m%Kum+bgE+XUd1$@U@JiHdwD{kaWhI76z(qRdPcyy;;N4C6=6?I)-dY(-! zK)-6jGI?tCcl?m%Uj?qi0&S`5Qk_v3wp?%4`Vw?qIU*03`uu3OZ|fvJcLX1mzqQOC zIBG``xCi|ED|Vy5CD#^XnYOP3I<4uqtzf6os+u!ZN3{DCaqFExLrM;f8RLHpVtMjk nxPAQpH+T(q_AAL^lTfVfVlopCZ3&jf*FPaL{;uc(5+?_y?q=pwO?!`5@2M8@zT!JP* za1X&f zB@q#E>p#~CpZMO8jUYToUEb=u6A>{6UHx5)T;8=LB6>{pPWF|yck0%RXR0=kxoa1l zHtA&7&6cB-OMjDsB1i6};;VKdLuPsy{VR2f8#!!jx%4?O*p%gzrTu~or0+Vtke7Yn z6wGgyAK#khj!pGl=@UPetfK@1ZxV(iB3g33<*fVnjmRRLi2Um9)4lt1*REcMu3eY9 zdJ;*z{6uo~d`a@p&efA>jp!QF-xtQmt;AO^KZ$R6|NX$@(|@Ovme#jMQuVPK4Zrl3 zcHw0oVTxW^v21G8@I!&6?+wTl!;;xqqNNDWb68EEIIcZBT8&rr9h~WK0q!iqzheHD zcU}u$g{w{BXWwF&YBrk-pZMN!i?^q|+Nr*{VVdi*jPEiui?eep|+q7ro-nA zZ*WtM^c*O8?;s~%Bz?IC&rUZLao&v#%fDB;v57gWS0|!F`=&&dyZ5y+@vF=-W6kbs z@eZFqx)k=O>Oxl(A4EsqaLlFWY(3q>jyy&0ycY z-&|p4jYt3JHtFO`{wVaxaPAyc`qQNK9&kL!fgO}`dME(&KKLqzr0l==Mf2M(q2Jj( zZn?FI8J?2#v{SHF;Zk=fU-Q_Zq;SNX8IEjJ2eo*ILY6mdyWWfV{AxckNW6|s-A6rlA{uk6^39Y%1A^wA2LvG#K0sP113^7c`q>7w;Qu1Y!MCDa=A z#C>o|n?mEdqS)3D^i6Lgu$Pil)Nmntka?!+W5;w-7|2Lj2QvY}&6asKedupbx2(2L zfJrl}jke$HtV`vgF}&tYbFJllu(8y~ODb3}4GWWZ&dS*o&mvF+)Ex`SV#cIFlZdy? zIEQmqAF8A_y+?Vu9*_t{qGa!OYyi&MW2ox=az6=D+D|TqaAdpfpCSj-UpQ*WwR>Ra zA~PI;Tr=rP3+pk|jx1)JBZtuWv`rtOMwk{;a&P~2B)G<@*AS%9C&!)KY@Bpmp0sf4 zLu(9~Nb(_qUlcstj5EZW)P^NDd5z|4nWoN^CC;+*H%8eXX)T&~JI;!yaGM3&?Df=> zwxJUGUG&aFr~xNuLw$AU_b$M1zR!T{ko$vx(F-h?O{6o>2y^P4!(RHl<=RfL-}9o! zgQIIVW2ieP<=WM&7l)iPY^ZXAgMA#=M^6*>d)I4;DzlI!qCna9`wVoct}ZRR$qyLl zvg+Pv-wE8=J0306${%R1>A8VutxI{*^viw`?)PH)F-w+-p$H`Rp7X+MYPW8X%R)iN zFo;4Rep?bWZk;4p_2y*d%)?TB(UP zmQQ1-yNX!Mo(&i378ihK3~utUWOeeVNMKn=nTVp$w2B@Zo{*m`ELrRfGA=#gVpU%g z3KMkrP_`5lg#Ej<9(3FO>07oeOiCNwIX8=$^x5D=GoSNT@?XZ^AaW8@&71%nE=+gh zI|~WJAabpRARVijPS7vbcz+dO8GnZ0svrJU2SQon=;0j*4P0QH&#Y)@KA;Or=24-8 znioeaBjbODE*LhWlxB)7r9Mo!S>hXkhWFBCsLe^aG+KTe$Jx&s*CZC7H%+QNJNa;t zrw6O>NU^>p9I7t+bho|nX0Ip~Rn=9t7@Uw|DV5m^!nD_V(t{Y`E^O?xD-S7kgo`x$ zB@XT5Zuf0abM!9H@FY0Oxa^r}zO1MO(%0CTi1%@3nau2TsfDQ`W9NMQ18aCzFMbp; zI7V|URmZvM&u`U0aTj~0oh6Mnbz8I6R1_NKV&UgDC%3UIp`}*y%B4kth5Lfljh?M8?06^|?T`y$wK%_;Zws)IGy_TkUq;6(1`Mj1IE za18v>EJ0mPR(97}dM}pO_;fO<`8KVr?A?I#zQz?hC>Om6)Zzh!#uuiu!xCwX(k_X0 z^F)KsH*uUdG?Asq%g23hTMt%M!wV^`){yNBZ;`Ul!VDJ!9CgRh8ozeH?lB{=)m@mB zU2r7qPKr?ft_ggrC-6H&d4b+W@kEhZUQEN|>Ug*K3Y| zZ7Q^MY63SBAafn0FsE?|Nl(?}T|I*22>kOa4B-^0aNdA|$K7jKzcuC5w*`?}Rj6~l zI5BV9r0QDA>93p@>1txlqr*B>S@1<6Fp4HG$_rk}wwG-#^JiLMabB1d%@nI1l!6x-IF5-|8b)xZLpOE4w_`Gj1hPqI_3bT$Lk#j!S zdx5q%8}AKtgFl0-kmH{ZWM?-ojG0%IY4yP z==+Dh0C9#MWzb{&pj?}#g*KEF8owKrm4zJjRr|qtI#HV$263(2{u~TF)#MrM$xPq% z0Zkf{{iH0$!Zm4W_}4S){id(uK5e|AwoF%m=3B|=cMN`PzWX5@&2X;6N_|Zx!7$cc z{PqqVG`9ElYPBO&5@4^vEht#2mzb?jW&Tpu;;MusmzBJoc?cKZ)rIU~(w7AXhazgu z#(W$PJB$GUqv^HkPiEWzo69)m3>zn3WWRs3*CXX^rvE;Gz8PU9+%$_9;@8l;!r+#5=YiB#Uu8lG6r zA}Ag$5XPFNgcY~)v=qD%FO(P%SkBJu_`MgDRyF<0(-&~uUMH9^^Oz$%$x-~?jzCP~ z3+~cy9w~kzZD9>leBngckWR1hfTrgJCPFXGrKMLJU^TM{^_*oAdqMIk{d!Soka>n| z>?h>lpk!MA=ij-h*i-K!_JQXug?7J~(L`k|t(&QswYq)=*vyKr%(P|muC?!h>CA7< zn3wz5h`bEI!pec?7J?>}+-UAV7qL{skG70!Ekic$M5_*R!j3&Oceo9=EiJwfY>gh6TbSyKv%4hvDPg{Q5nK4ETrB-c!`ukMiPI>!rB;|p2Y@0RI< zvHecU7oleAlp4w%bb!N-Z#Pr!rAt*15ZRRH=;yllHJh=&0-5q&CJ({uPEQ>K~0!18az;$TGp!DhWKMIH4)C# zMttTs&yz)?KyskHdDTMv8$L1J1(e;=)OTsVv&f?smeMNqZOJNWt8Vu=C(3_Hr!PC4 z+53ze-Ijmon*$i#Afw@)w?@z3+avd1HccDV3O#J3RKXxVd)4@F@1yZ1O>X@gmirv| z1Gtk&o2G9ywwZbBt?y&q;sJfkTq7*Qcf~NN*eMFu*{6ArxCyhA7{oB#qh4^nbhla9 zYX)Wc=~#JD67#wa^1DB@ll`d!xXkJ0Ml#vF2&`!j^G#ey=b5SS{;^t+l)B6zLG7)^ zEeDUC_85nx;oJKkj#HU?hx3);E8njMY53C6 zT30_1_Q?6QSx=tj+TRUMsI8mc6dKBwzA945mo7WMu!ggZQ_xI37=`bH5s^R98O-&r zjpjkFwh-8$k{S%irtv)&b`f7e_OK2z^5T%jR6XGxZNzQ^M|3jx@ul4s^VII@0&&QI z?vH(h&6l=^pX|U^|h-<_yMwRAY;vv}+%i+nGon+c+nt~C~ zVj@#Z>!R~>eT@4?i649zp)sJhGG&2zz;ne`q6fAGtHE+T4Dt@KLL0{rY+ z`$o88`WmW9WZ%Xw z#Bk!J#?mi?CXvqtkUpNcCPN->1;8r=sds^OIx>6H(%vLqpiwmp6RUwT5+W*f2X^jk z{ov^e)EZ)sgM%7|jDVo06$t7OR#-r#z8iI~RPwxmqb05ZbF0Cj@7c`loxtEuVzfUx z$qVg+>&W-1F4Z69IP^BMC??mnK3Q04@rB*qVWN9r?2(C*p10~4V{hlD+e-Wr`_sx4 z>rQ7`iaYm{>nih?Ih2`6LHx9ru3HFGC=L>rzGEm5q-%5Xr&SD&w?F;7_RLJ%>DX`D z^dU73di&ch*$(%1Q-O7@KO%p-ynrPXf=D;c9`R39)Jt)4{7txZc@6UiUL(9gM&_c!n5&M` zy@fV^L?QB_AJCg7(=lOGjE*{go!V?xe|#DO8B7+jAK$|I#JVB*Ra&gN(3m4budga5 zLbhk5DZz>MGUCFA33CUVya;=nIl5G1cPIGA3) zl+2xVB1(Vcw*Qzz>1=dkydn9W?C9n>h_Ro;G0y-RAQT##FpheH z*t{k`sQsWxY<}hK9+I|da8kBi{h~sQMAwz> zT#}JkW2vLXRc1$hXn|?6q*YOTA}jsM#aVZG`Z18pB<}-!keW%&ts>b7ds^Sj&eV1i zuP?)dCgPtHaFei_5%=713Jd1^%*5#h$_N=UVSLh{cm1?VjNQ_&wiq*X7M`$0 zFhb8;iTTDEh7NvNeTJ)qE5S=rYX3Y;tzY6FDK3gsM&}Y?EWv#qitm`R? zO7<5-8_yU^LoIsGi(X*JL#Mhr{C%d|X%2_>ar73mB2TQCv~S?$$!V0RS1#PQ z^4Hw<>QqBjhVO!5(x%f~c&EqfnpC}u#V!Dl>2S79!`86xUu?YFBr?AejF}5m(IqS? z*alG;P`o-rt5}QG!*@V*^MXCGoxHA^;?MH2=KnxP8}TX7b2yjXLO?UU4xed-U&kI5 zj~HY9j}i$3BNZ8fDo#&6m@eZmlFE;w8k{`bD_>n)Nwy$fOu6GVdvdFu`kD#61eK?X za;Bo_$TtZ89W()z{!tIvZs|+(iOFvwXqcAVN(IxAm!#6CKaa1hv3GKq(FDS#M#`?A;>(@%Y1`fXtocF4+) z2zC`ZG27{%@`aDdM<>E@QV+%ac-+xsQs~6MEb^A!568;nZjT6Zi&Vgx+#wO*xLq3- z|84OOVBHTNkVG$E$M2TSB@NJlLEiTLQF`bUp{K?mzMZICJZn5NVJYxtH(4Z%RyVz} zkgF>)wep?bwq($PX~~&?qz)WAyc!sm?BcYV%ywr*IpxL2`E(u~U^^3R8P?TEW!$_K zj$F5k$(LrX8-FTEgO<%{jdcD*eSglA#}W~{9P~u3sf05`IjmWw;~Rvra@XxHaeu@A(*y0H_Nf8i9LnK3=2s zaB-o8h1Ob9)sQXTBjgeGdYg52XejpQCAsdM-U-9y0cT65#?M{!TL0$MOJgil67f4J z5<^K}oaT9txp_>ZSc7B^oK(!y>y%I!h6l|;1R1E=2kg40@kaIow4zl&JGC-IT9kd9 zZz264`vzs%OrA4F)Zg^ndob23-5b=NM*}Nx-2_0q6!H50tj=UBAE)Qnl7N(GT4p$%!cDNVE{cHT5%Qa0T9 z=oqehe$eu<(}Rhi5a>OO+&|J4j3g*@ky*tX+9OCVAvL6|tbp$Jh1;M3n6GCl62Cul zqB5~%CN1TTj3?j76~D27DlqIC0pMMxt)!oO4YV9@m)geWcnYngR)o;}M#@~i9Twh? zee#S&IrXAwW|DHey^#)`#Dqh-z}CpH(gHSRtEO56pPLRLwVXSFiR2}DROq1*grzSlkMQK za0P@D2YTVKv01werIf{PEKzQWxl%Do3~DVpBHTkjV?MSsUeb~zP^05l(z>iU6ZLq* z^+0Td7d>KiwiX)d@4xfi*rAI&qx_xSr{mVdx$q|f5S4n>lYoFzt{*v}(r0ef#$SvX z!>bicK@6n8-z=3$R=kESTSG?)`yc`I{``1Rhzq*UtZf`f+dh0=Y z<69E~opobS-*z##W(cCex5BqXjg*IDP{RBnLA3vx;|G8%ntFvGr8<(&VO_B_z!C*3 zRn&n-8@r5{>7L_cv-@bUb0qQ^AUGoQm$>Y{UNXK}cuWhmWpj!H?;5mU9Ie?FCxOf^5E zxk|c4Zae<`_G+zT^J}jRC2&-byihEsVP}a$UBQvN?~#c{iUi=*DA*P8{S;O@)uETw zolOpO^*J{!)4B-NwnY{yB2jIYIgO_|t+&kJbnb^M%N5(q51WLZuH=(W(3&UhS$=mL;~w`%9p}7_4Wr36K{jue z7|T++uUE2FCRmYmgi;Ir`3TdC+(vpC+r99=uOgxNUj4mYSfX;>?9)MC8K>fI$F7sS z^Ib9*p3;+{+!M=~)l{$P{%r7ss%4E##}G+lj*L-1$ZutotlQRfzYq=8($5ao0}Z6R zx5wA}oK07WQkZx1Mb@d}pOdjQy;`(mr>IaUGpJ;nQ5?}6#0&YoM9cV}9r(Kb{+yaR zWGz{{3=fMiwDt6HEf1@ZIXyd6=GaQY+9DsTQ&4OU=PHqlLwa+_!gMVqRC*DHOQ42Q zmr~R?kH>Mt63Is!VYcd5i*)Q18d{ml2eqmgZAYPL8kx&eO9%^b6vZpycxC2U~e!@(W})9&dHe#BCIrd!{i1eFpZ zZ`dE^kL-t_F-JI+xFJ!o)e5buyi%oODXk2j8Oa2(&|uH=)CON&i-0li3 ziII7(g2cq0hOC1E7-GHDa#yWVC9eK*GXX$%y!wsw;cEJRSYNktzJVr#*b9w|uh-M? z<#qsZ^SzBF)fe zHZhfx4L3aKk2HLc_RL07cs?@jT~Y9YZvJkwjH==^D0DEO296W78!dX^6x=dY8EStPIHN25X+dppPUw1-zDsR{jq{r+#8sirY}7*@u};*5)_ zvH=Krr0M>cSZ`@DOQ>47Dx9k|L-O8#r*K4i5ny9XE{0lG(yAiRSTeqQK}h>G%ELBIA~G#W+Eb-b z+QU6<+jv_rq#&Lb(tO#!wGX1uitF+b`_mNTRC3gQhg^Y1c65@;*#m6%`}6mGlf7RR zI3IsVYf&f-&w=mC{7|YxOohcMCxJW;*3|t*DdK@$SVlnkjQf7+3ow7(loQUWR90%Q z2O;7|MS-dq{+f~E!-6|ufa72sCTYIQR;R*Uccfr)Oi=_^YKtSHQT&Ook<5va6S}ie ziHWXX?_=woNJ`P*f^JFpmb67r#X%S6Xg#m#y-SNXKe)402Gd^tI~L$u^~Owv?8| zZmblzPK>0AEUyUs@_l0lPI57Jc)deBNLI3599%G=({#F~giq~v`E+nLD}Xz{@NUft zrTLAz=UZuTG-!6O9^X%nq811a#`TQoVj(-Z?@(ng03%5*7Tbo5jo>N{eJYLD$#^Uj zRG8lYvmNXr6WNvO6pOz8qUG9lJ5^p|wmg4g)ALR~mQd$xr!+L zFEW#q^zG>>GpuVq6skO(EltCoomJt+WATrk5j1-vm@a=lH8odsbIZ9;XCsG|aXAMw z%|zWm@@^#g+HjszhuLKabp*y>ccK7I&D>BL%9*#P=7$GWr5f+%$X0)>Ryk>cXJf|S z06gu}P6iB#nTxdeey1|H`=ImKguwLT;Es@|9bFR+?`jKw^k$g`@=wo$!Zflr# z@7px+1-~Ot7H4QyaKXkh5V4AZ&SoV(8GfW)Y4ek0^##_SZ+lAbyrUn{70Hv@91z>e z?bs+B+_%qPNRWHEYuRW&AxBSm(kiAqsPnkdx;Hle)^R;tC+YcOO$;ht^~83x>FaJ$ z@B{VZIC9_oe(bCR-<#8}Thb@pogW22=Wzl2!cFHHGQ^EFMOuZjiBk`*Rx1|+POIOr z$l^BR93?1Bb?Ge20he2TZ>092ea2#c9L*cyz%Q@AFl2l?JjpcI;qqdgZ1hszS%^vm zup1?$yl+{p(cm9!z#=g3HV!cW8BKeY<6mkZTa#>yDBiO*NEioh50%wMTBwi>Ke=?} z0IW1van5)hFw>KR=xg{jYE1RdF^is^7iCWS@hTe=Me0>qyAM>Ot#EnZ_wP&y0N^ zBME|+5c<6=))>@QbY!f?QE^I(o#FJ7kD$6=4~V`>?8%}VS8oQ7BPGSI zBVM68%+v5*gcI}n-YcEJa*06|)Pd4D{h*I7rPfPaBuL-+nV;KAYmsWkIrPRTP2>C3 zL8!l+tn%3Gw3%z~qvh1UzCE$uhkwIlj;U!^VY2<9T0&d~nz$|e?oGw3upe4HO@ufZ z0~PrFsJQP+qd{%67UC((0^$H=u39_Uh(-8VQBb(gs2v#~nqxQ9={&8%OaL_HG${jeAj02-% zta_)LR%3#aEl3toX#_ZeNe9<~M7Bwy&?(xRV3)x=XPCI+pycYLkaDHs`*O1FVB6VI z2qJCBqG+r}DQ>h-p44VIAJ&=nA$G`ze{iaDtLv4D;+eI#c8S?Pg_BlmWt(-Xw0a`C zU9u`vBr3+I6H3sm>w+bd`8G2$ro7WFD_u^W%#5Ptm1P73L6MG94-`?VoGXd1v^!P9 zVR>V%jCAiliGzU~JeIv2uRnaqIAMPChNTgVr0x%^8*pB#8rpEV&wN09Ne4i~6yERW69kEt=2&PWBE zK`c3lX`2%N;1C8Ebel@&h1@Lw+4oL2ug07wPxF0uzUk~oJpE1s=x@CFF~4dn~=C$VgaJ;)xD!19QnU$n#;g#;@ z`o0&1c%8Bt&#md)NlN#((IdcZ$LVsN?w8cr8MyW@%$?4mHDQIC`3b(`<9n$Ba9QcAFpc)>17f^r&s*{Bi{efyBf0WHjpdk$q!o2j%|kAJhe z&@Y!&B<6(3QcTonCq$s_`defiw5~TQONnLY?{7}J)yj$Io0#lx7C;7(vfp!2Qj$M4 zR!G)Xa^l={=cj8Oe|LM{dE_MJ&Gl=JS*O}>@g5_VtAuc|9sHm$pO8a!XlLTeyEboF z7R{ehWp|Y{y{9%$svSIZjG3cw=~@RoGXA?_>Z zcDnydcXCnkw5dnF7HvI;=c=i2592Q^iqoEIi|og-WUl_tyHon(dVCHiYx@mBj8g8) zNY5~?=hf=nJvEyX#f(%NMdN%M4ulwVSRqVHo2_7Ak-wTBvVT7+=)55X9%suFALbOb z*g5LqzxfTj!c?IMl7O1K^FuEW6D%XnU()}kKZKNq$7tocmmAlzy2I1z4;#7j>}E`` zCejG``i5O&&&&hQy_0V#y!yu(vAw<%rAW86g8&}Vz$liEYyv+t${fmJ+qB6K{Jl}} zjOUT5P2Pt-l^gUlbOxN2Mq|k1(GvAZe)lkMvq26ok?u#Gb%ta7t&8+?DmT2>jC$f8 zcJXr*W}Q&+(F6ZX@m7w7!E3_ks-M3T0fPN8+k>e9p1;7uJ@EnR&n<=@VTNfz3syzoHOeD2J6Q9s1ODDUfy8^(!ALL)zev$B=nnHwaWi8N_ z`PgP^aiUEJc$dNvjYs27SFJp*nnt}bA|}Ua@$-QT9Vf7L9g=62S&4prWaAcL82MG5 z3!X!qdAD<9yz2zc>+evti~mR+sBs}zQm8gNn|EHGzW+*)MB__2lnOq&%JXBA%_6pK zofSEMYoB#ecvC(;s()xW0Y!s$`r=*`JKM}>Sr!S~7~o4?s_HiXB04`R{5w|jMZo_O zjOKsZNn(@nH49x2kAn*HF4-K}kOv)M)W;yts}&N}Q7O!Mn$Q0Z@Z21S8rE$m3Ya$; zQ(t}4ijn{KiT*DJmHwv_{yzq9{qN7@{T}G%C&_?HGe;yIXehd?qjE*7H425f&!{JF ze7r8$_L;A+xvf;!rRH33{3Y@g$D2YNuCTk0Q~v*G*_z^=$Uvy|$$_Pnm`OqL3ey|q zb;ElO*Qa~3BEAo)+)v;(yBs$>MnJrBcY!qHxwwL~~AI|AwoV z3KohY3^~U~fYk3i)1^&!h`e}n{#=q|{u;i#Nm0Scxd5{w5u)eU(?|Xd|wF_#fGKHPHW8+CpPj%o5Qw>0kB?k!VVMl1%MJ zQC+pD?2VpB$OuPhlg|yjx;hV^2JR!TO(*@ouhP`l z>FoL69EzKy*O+{kxr<{6JR>5ae0#&2@}BW`$Z<2iNW0LYe3-S@Czqyj*-FPtN-SdSYG3{&LSQMpQ2{{H z!&^>4Tj_?p>S~6Q$D8)fAZuB66+>?nrvLt zu>E*pbCdr5{ht-LfWR`7<{}*5=g*&IWMt@WK6|9(*AzD1^h8*A^!?Bo*)T7e8wM$o z(x0dCQi1G8Vq#)!?9xdcqNRwYUDsYqR(4@wp+pB71NQUtyG~48qnjmgL;S{#8z*OH zc9i8)P*N^eWHBizsQ_I~i}$;C??OTh;1KT62ag`*>1%~VM%L*)qr2H><*u$Cb);BI zn2NtNl2K9+MyIVJvp0Ya?L6(aRj|_i^5sjP74*)~w4fk|{)70YWTRu1gqzz-pZ{=u z{dJ#?jQ)Ghi5Y8`@HmBVl=OU3#GIXGn;H<|ljR@Uj?oaT*_~XK z6wz>cG0u2H`wB>v$sQ@YUMB~=uk$TrfQAear$rCQFciCUX zrH&@efkoQGoGwd$qPt@_?8}yFCQJ~sAB?N4`Wrnrg>O@GX)aA!XN-Xp2>BuDGX4hH zFJ~G(QFYE`8m&%s)>Ds-8$Akv#qsi=5&3G7^kS;MZ2V0-Uyfw@JHzR9OZ4Wi*uFvk zm`uVVI_&cD9Gk5WY1gFGpDY3cVTz2$C|Kofu=X++J8g#>V-F`3zGan}CN?{Mmsg;Q zwod<;XEU5dceB=MrqO%XEc`lQ3zxw!8^?Wv`v}y=gQ?Me6cmL7(hjCe*XRoR9N1u% zE?SAng>6SUt(tSk2*Bi@Z35Yulfw_1*(!a z<<+lT`uXZ>HQI8aJ%pOiXoz#DKqH%k*-t(DrJI{uJlP0gu|>dQJIWV62WzGfTyFD^ zt?zw=<2%NwUhgu$n(nK5Zk(r@=Cd)%6@J}wV|0Be^Q-(dAh60Iim(kPN{tP(Tl=G@ zfCOTkk=w&V9FWzvr+dqEH($SgZQ`@ehKcJ-7TNq0%}RH(Qb=%ca&I4rt$B?lG>HqN z=8s^KHaIs9yi2QDk~q#cT#A}eF(KU!A;$qi075pgbY^9^9kj~_sd>(_C{X=DU!OS9&)=JN7#g2|YV>m>~6g`MSPnV+9u z?oIHj2>7vVYj7n(!X#}nHn>=>JSaK9wbKLWko)IV;O3h#DPpeAk;=zaIp7;p~G49^zsfh`( zUY&k7BQi2F&)z>J{$$3WTP#-`H(x4AnJpMz+HcDZIArWm#j`VZP)<9*!DYs_W|Bp=ppzI`WU(Q!^cFDSY5x(~3}1pgP-NEF ze64VE`>upkz^q=T+szm=xXZ~}%>i){ragM`(b#>R3eBA2{+^X9_2uH%m{b~i#o1Lr zP3R-5=^;>JiBUCMY=y9{WdArPKaO5!Mj$aGRsEZ_=P$TFkfmsg6?PlyIKB1ieSA35 zICfs5_!5_Q+$hDvb0v#u_QLO09R{V1`AI^=E}kz!2<>nQ?BVe2Pp(Lg{SOLMV5;US z@wCl-M}_`r-t~gkv^PN$tzi|K`Fu5i$wtrdf*M{l3a|*le1l=f=@SpHOWo-)5XyGP`Q( z+R2|la_01)T7o7{P}+aob2j5(VcDoT%m8g%%!c045lH+V+Ha>tg@^fkPX&~k=B6Ci zDyO5wDi6 zW9$RHET1Yq+_|)oh%n|fC-5~erPhJgxx|2d#vEB$Q<4z+SJ$75HHmCRe|~ZOuU70O z$T<5ir_?9O|B@Q~U$z(i=hBe>6WiY@VGkSiK)!?P>#$h8OLq?$3b4Cff4HjIWULig zvF#17)z|pmkzu0gBDQx#aIFx?){4nNYAuwdO4qwyPLh)dLua=-8uA@yOCKEky7I6P z%FADt-!~2hHXT_JOcFJ0rZ5*9#P1|`u?&^kE{C%gugC>^LV7M@t488A|GzBBIf1|| zG9M|C`cMJH9W12MTow!B?~}9-L+y;^SCOc0J<9aX1P?t_&Ad{sKcUn}OL1FkYRvb5 z1V3hcpJBop=Jpdj>)#1}(&g-6veBJTklmWN$K3dSg4-@q60+Oqld=&9pXpZlYyA)r zVbuw$78M1B$ga=taAiAv+-v=qtP`(P7nOE5x83|^x;TDHz51E8MO#+zNC*T1t~til z^vDO^JRzax12)XFszIba`c@ZbTq|WxKJFjl3onl4rd*@euJ?%RG!6udSVNk^9;bRN zpfgM{Qb_G<~zb2JWV8aC$q3Yo{mNh7P6lpF2e-|P?9Eq?KzO6 zST%8^-&BJVKAt^b2_CCl>DVKrodA6t7sHRcRkfB%iX^&3{+dv8jPd@kBYgr)gCc-c zklDC~Aw@Xn@Dn^DuYhHxz!0J)bFexW@um|C^JI|N?fb=Jea|WN7)72vn<|0BJqkb= zt7g<&d*CDZl_h_8XZ!tLC*D+J)&to04W_}9aH~p4W z1_R|Fejksu%=7R=&3;G}Mn6Io+}QTB!q+)oRjOeV4FK_;+fSp>5O4zOUaX{ZXezLV z=wEyJlbBwl+R0L6n+^d(>M;59ZJ^;~Ar}mOm%6HnU|IkDU4YH={iQpE$j14oITXuf zT5Cs0g4BTJbPs@5mv_|EVb zaW)g5*?kSa4S%=nz@x;EYO#W$oGY+1WtrsE+5v&hS!X({`x?q|CI&HyjiD;|Vk;d& zYuH&HKiK~`_(+T?%{rK&>BV8qW(@2sB-pt)+8W^;M@WC{*`V&BmHSR*I^&X^PNx#x zR`lrpmDhz(n9Qdx;6?0gj9fIjRn$ql#$rTXFrA{o2La5ZCt2!`CB> zuOf_hdfUx)fTT8O+s=6V`^iTl?o&(vKfL$5?1J51d3*z!oy80wd6;jzx@GokY!&7# z5vS~Xt5z+S1MsmnPhY!tyM~^Dx&$?-eEOk>Hif1aAbqyVwCpz$tMi+#8Fwxjj-wj$ zIT;jCDhmrX!XGC9jtsR0fDC)6;DZk0B63=sW!Gz`o)pz&mLMQ6=f4RhvQU9)gM`RS zeaRMGed`I)6g$wm0dLzKE<6qHv3j+~C=AQ>eyS0tDW)Ig(qn&Q2CjFh>^ZHwV#ky1{&DeU{1j_1d zW#})Ca`kn)L;5j3sx|IK?n|JZ_MfO>%zPT}!sDYb0HW(3g8PqRE`5?NU;?OeN@54*0Y zLe?&=9FytU8EXfY&dUDe1b3Zj_^Yt9OLMD+`G34u^8V-pH>ne+4d092xY`DL%x{Ko zNvG2pb?Enu=6(AXWM$ODLg;E+a@d|0*I+7^yJcL{I<{h0ql zc7H^-cC$ic@@Dmr>)7 z!?-y`b-mzs6ar&lNDy$}WUae6P>%NQ2a5Sy6crhg{sX6N3tPhpWlIDsHf4&rw)hRQXd~7yT2i^x-f0(HBZHxB?wre-YTgC(zy7d=X$&H>k zrW}jE+o{E|`U;ommVx)h;@Lxk8~k!BSLbDdoBlkE*P)h(XPF;n3vD^@)rL>lUE#MbYr0mwu{K00Bm; z+-czq5HR03bI8D?6Ql;sUvDST5GmW%{IBM$?^!0MB7SoTh6^KBy1m`&ZRz;M9F^Fg zkG`Hy2!{mI@ErO#ssyvtvSw%a`_Bhd7f!jIOeC}*%?L=U8m;^d0&om%rAAh=oIS~p zmo7P`PBBB=G~X;t;=f1l)x+NT!}z@V-XjW98}R~DU=_5-F#G(bV+(}y8d~MD& z!r~(27bO#g+5L*`B+(B_jejz#@d?%dqmR<(LA~yJV_&|&hOTs>PfjOw-p4_g`x1EY zGoOwW-fd$Mo414K_`z9!5(^;Cl>u-}>u^7cmoN3j%MR)L)Z*?ri9$lwPEdzr(Tk+K zcvhtDI4FMX_c8r!(S5LCRU0%u zLn>K)G%=+V*-1m$-RX4^iBp_ODITjoOg_SlG`H1t+{BNzbcW2nnRNsip0&Hd-&Q6vw}sEmZ=Sg< zicKkg8rRLPoVZi@Hy#mHRgL*-B=Y(ne<(h!`(s88M=oA6D@W%a$8HLIp7jjN^44z? zs%KS`bH*d$r{QRSEpSr<1MC>J!grRBQey8wDFc4&cyIREZz@dGA}l<;!YZ%1t3H+4 z^)fLU$zMR!yv<{iykMyQs7DGNs=qs(d^ES68AhyVr4c$bd8T?&lGgA)sC(BfF5I~6zynmpjZFZnk`b(KP^4j+R{aLFo7z4d(7TvMcXaez0uy+$ch&|^?X z&pv8BRmWdfA_iHFEFXgf<}7a?h&Q-n5K0N3gU8Z}-(Q)|{&wMp+$2qLR9X3WUT%#L zENEwyWGAaBM*`<|39;$XdCMWay7+w$?YG!dip5ZkRjs{G?jC^LSROUc*|(9)ch zd|xt@e)#@Mc_Qe8vvFIUrZkhXJQoz>%(tx_rH(ltB! zpvMfC2(Z4#6`!1UJ(MXKBDBG`9eNGbfp$flQQj_N;9u{N#^?Qnu4k?I6gor>l9aiV zT231V0~@Ak6TV4;#0;lbO;*N+br)-}>P@SJd5vp>a;}lselHN zWHim*QMEAHL@DV*?_??lliXa{Lth*mhpuFt)WloR2gO#At)9gXY4s8T4Qd;^H9Qrk zD$R_7987KFK=Bu2q0$B=qsye%#fQ2917By4)>G5-Xs3jBE8C$$2i!*tg5S-FvGp@v zr_1x-)<;<=J)CtI7~jcPfWALB5l*f1lj|6idV%L@?7+ERZkkU>CmOJ8Rm2eV>fHnM z6cl-KSfW$H9%d?Yb?JG1D*V1HOefj_@!JyF_@h!ShD%f>qRwk^=<^rW)+N43n9yJ! zRkf`hV<^}-;Kk)Sk;c`zuYb-byWyhM(J4tP9~c=i%ROOC=Zrrj#=@l+`T2It7~5QCJM@S;<0^?=UA)ZrNT@ZBS)9RrOhg*|YZL98e_gMJG+l)acB3wO_U9;f9Z| zaMHo0%@Sj++NjVV)4BE?&hV}z9d4=Eqh)J0U+t0zk^x5ue@$O6r&3xPNHLrhFV4&@ta^;x@XZaX|9e(e8N2ReG*-?Q0T%YAt++m+cW+V1Ek{!6s2P7ggS5Ya)STkdW> znNbLp(UhtzNzJy1Z}k3I?wcLaghBwpdX2yeaOQe8@VJS6LIyACSoB=Kud9_{bkJ$; ziCt_7)ye6y58RmM))X};%KN2ZM2ZyL3(FpqXtvI{GVN;nf+yH$;%szm%Sk1N8%JBe zoSuii08Bl1RW6X1*Hm18Q(V(;vzbRM9ZC19POdcOIM~TJVaV6lAD zD%p?<4}Q!-JD2Z2HL@|at7(pKX=xL5I=1?%^s>rXM~^kAar!IU+amor`^DLsN^o#a zPDgw0vr=cc?E4EyRh0(A1lKh>ZieX^V%AE~U1~&bvcGku`gRiR?5pGbqsWsJadbD| z0cwR3axt234nOt(8Ef5<`dLSvL4{7-eQ|cB>-$ODqw|h0aOVvWt!J-lP64wD-pw)c zTr7D|KZ`2}ROmreYt55}fS#wkF{sxPL(5ptFefFVHoiMdX0*Rqiy2AWFSNRRb!F#( zLh(zRW9Cw18?E=r#^hNRG9Hte?-7rzg(Q)weDquk7iZjjtHI*sSA2GHY%gJPn_3(L*i zd#CWE!hKqV?I(>;zR?Rx?ke=rZjSSVaku9;)dD!p8aWI#bvEY5X~kWSnx18_x##vK z_*{?(78ZntrhjN#oi!L)r$uxrJ(ZTtv&vI3;&(!bSK5*d)vt|6O>oXg%r)f(KW4Cn zg%b>eW_}D1^u4ROV{GvAJPoBn&Bz+x7W@9o43X|$XJBnwFHQPL^QN`MDqO$|# zx_%7p(SYHG(!OTDjfB{GT6?RGF%jNiGP5Pj) zux?^CJzhQ`|2JT-Cfo@nNR`WDAwdpA0#n=Ng>i{L^1gS5bW+JnD4)G6`2)74*`Ejx zOaHEnxWY#}u4`a_p8?T~0$HCVAAeS^Y1f^$`X?F3bP6^S(?pvZGjgmWTz?a89h&1l z!G@X{Im0%WDaf(rws$kXy4|>>?KRrupUQ8QPO?VZ@wLs>Uz!d4^5rQ>akt%M`U&Hn zNq+Heq^wB@9zTolD-O7Vo7a}<)mA8X?V4usa4(9flh1$iRJ`&rDB;laA~NxH7`#yN zIfa!Owru~wCsvP3xr;=>1&T|HwdH1|06mFfMc5L+Kv;B5R%Qp}gJ7oZ;T$|JoWy&8 zc}*Ppa{XOL^4rd*w2%x6*Av~J_2h#lJc$8D)4~zoqk6C8WM}KlT9pdD&IbWI=o z@v#NCa^|L%yA0ZF1rK*@eRWF0Ug4%Acj~_B*V|M&S*%V0SFIaYLDW!h)MtKy)*H08 zc~cH_Fb5xoPU%(Dcpp@BWS4Zu4yVX#Wdi@EJQk$cOo=#&aw9YG`CXXtv=%V1SUCu{ z0KYEZA+XtGaB|eI@#)`-;!zK~Kpu|~1IvUH*xWM!{^1Q@allr@Vj{f4Oxk4it|-xo zu64kjD2R!M%G|(@$x7>=7S|vLs~g-CM+L;>c-O=c8X~_8VCGU@s{+}V&u%$eb5O>3 zHc^VhfM)~c&PSwtQk#bTlBX9FnUd7i{KFIK^$A(S7JSwQIC}gKy+%>Bb@o!U#bI&s z9%w^Wq{eop=mTTixVXpOyK5ntECZ5>EToP?4Jx_hd#D1u9eF#u6ow!RZvTOSO$}ND6eZ;gAMtSD~lAVOE z=BDCS$*jPG+8(O}kcM?Zbbc``xJD5%P@S28Kxf>&w%5*-ofr@0Jj<;;k4P@iCY~3F z=8D#r%L&bdeCO&}{bd>OVKTTkS$Zu(beiw-rW6DR{Gw9+GKmfh@#sd(T;V?)bU*TT zoBu_tMbs#yAEy{W2K9VYZ(&eoY~G~H4TF^HR2&t|O%6gq=jNyIw((vVccsl#Ct0HX zRuAlWU!h>*XOrg}B}C`CQIX)0LOeUE(BLB5moS6sj(~9Zeq3dLKji`C^!8Ss*TRlf zOz5f62*gX^Na0CBI&ELRJvF<)-oApr>q~u5Ri5Ag?Re__1Jv;E6MT`o#u!Pw@4n9Y zy)f%}SRCNj^%3C#-xctWun&6N^2zm|n8B0zo~0Pt%Mk_kpvOI><7h8vwXn3t`siKO z#zUTt?)_l(j~_qA&`Jb7CT3B5Cge2R%XOm?&GzpbUA+3E_idG{;SR>|#RbdO({6&I zNP;5%g|%T9Et5>GgTKEOQMfqPIbEh;W(-XdFp5PQ5@)Os0@$oqiJs9uV^d(!`s=R- zIrNQ4cjV>aLQrQgtwe|8S8t0%q4oKT_BY9*F^Qb)Ah?MrAWUE5=U`PBdkoAr3O-({9wu7HBaBfCRHVFSBnuo+wcCWoKRwtNq~Q|%jW@1dOB3U z*1V3LqqilVxk{*2xW8D}s*TP!mn4!49j35WgIH)a_7_Y!e(0Bvs3UQ`+UTDQa)(w! z-Pi_ua#6b0*woJ7Ar@LcSE-oo8n<|3|6XcL;7S&aFWpw1C^*IlZgSwPJXev{>w ze}?y3*+Yg<_(@KpJ`yvYOV|gz=Axi)wl=T-SM&NqRYk%h|wA; zl}9?)8uw~SVe0HKzu?4a9Bvc6BjayEO@=NT+8-u}QQ-?7T?msZ$4T~`iZ?5|*y~92 zH~wmPdI$?> zy*^qMUFBJN?LC$N>F&tr`fdgvEcJVDlme6bI60Xxx-o+%f^8BV9y?VbaVUbtZWUK(Gqb~>e1;{;yx$=Hq^}grPod?2_JjgLe5#bdw0zh`Am%SX_=0Z z5T6__JxP0P2FQmSs-Qm;?kJ`2V}ppC7-+b{+QZiYg=wFg6iiB{LL*Y5(gLpLf!8#W zq-3PNVp43^tZe;dB(KZ~SUKZ7xrFjMSl%w2F~+0li6VHLa{q!FlxFVQB&99i z%LKq4nmDSmh4?#o22IcSg_!i7V33JY`6#=Uxe~YESv3}<21k`Q-WSeuI?caNmW@8M zGd_~8*rR~DUgnT-eVBcMT`3%K>3$Xu*S0!*Pm3(hanB8*+qMCm3`T)F@K1a%^~t2s zMxC8~Hh*KOHa6-(MLf8kgCaFGJ9^7rOoeZJh@Yp>squgl$4(7OkPjh)M0z_`E!oI@Tb?c)2L3K$RDk_s8#gnDoM&RZ> zuj%7KPmU46Y)#iNhqDP+K2J@(fZykJ1&84u>q(9aW_$)_k(Jf#8jyn%r@8t1>YAu$ zqfHv$C@qZ~`vH0xWUO+OQ6X;2H?NszGZpgW5}W1ZL3Mm#ANKr5#n?c9;xwwP`ig&a zrmC zb-{Go_F@Aq!E`{qz@HI$_3eA$lX^7OxPq8|>N-G=uXJ}@?#uAYn-vU%y&FRj1Gk)y zZG3?9i&2~kgzl;&r@2`_ zaCp7jJmWtx)FN@VCHgmf{!pEEw^OXAXM?D;ntNn6)QmyAVvt53_7Gzn8|NAs1*?}~D^T8X+{4=K_59lp}%F=f4Ci{V|PHu}EbpOlXPqAV>{tB^$ zzK%+gKh!G+2KbUkJ(lhoE@-ZdX7RKx3p!SS8*pfc{#t87EF&1+8Ii{-`xiB zCQdpZt)|Qur1QVln&OkUC5LATasg5&vHCpLZMu7Lm!i%lJ9^<_dk*$!dZcvPTS|0> ztFFpHr_-YK$bt5itah%vm|eoAL8CuVIZV!KYWXXvloTS=f4r{J0TBrL2$Q@@h+(?rb$Xoe>%iQbKr#s6O z-b;vY@iPdfI^{LeDF^u4BbxaX;LZ&PcaiedRDuqZ1uhYSrBsRK^mvr1JdJA1^ujx> z+AHaE&{X#|0h>0NOyKEwZPRv`_Pf<&+kmoMN}>x}U|+-(&yyReJ*0#&2=pC#h}5zz zja`W7(C0JdKcrVwRLl^@QRd|vTvrr!EOd(a5)D2#T{bwBPo8}UuUHTx&6JtjNH@X~X8hy~#LAjGnO278nlOX){72281m z@DJH4ZYEIdHWdG2Gh@bMKGNPI9tDT|HmG&Wh)XD8ucF}bZ-*=J)N(y=yj(h0L$PC` zYYv*J^;XU)s921j8pZs|kjeQ$8bDkv?tts?Oxdt%Wj+kRD2@jA!hEXeCh}{(Dd!kH zq@LXDe9V9quz0E3X}M)OqHB*)Zlnq<34>SkxpBQgu4~D$@(xgo9(k<|JQDHa;F(E2 z=o_NJUmZ00b!r&?SlXsELdSX7ysv8f{Mv8O$o}hOfDyGiq{2~NhYKn8Dx z(a!K)e>1+SW>5O%`)ld5X3lD}H^j5wZnH@1>9Ii5E-%F20$^Ex$vfwC0jXr{=d4Ims&zU%+`4+;J3wH3 zYMY;sDfyctInn0$5;VB^-D3Ff(>#`D_QFEj@W-I^Fa86)3`}kcZgRNH%p$=`xY*So zwbU-W-dh5|xK4d%fNvI(2PB;dsVA~G{v3Z`2iXq)obYtZ{I5V}!Ek9}0r@unuS70m z==n)*LC_;;ltQvy6k>~PsH8+-6B(s=;|0|V+cC${25g(vzp&5nZ;kwgpTrkDL=LF3 zKEswk8NbnjxwKPJNO*QT(T*ShGBBeGXp~Z3^6ecBHElF`|T^ zf*nA~^_{YESX32i4*j6+($U5uzU$*#F{G-P0QWbGG+8~bm4`USOq#K zxLC*;QE#=76VNZJNFXQ-cqY`4cV$VAtEI3OJQTmi=eXFD{Z>fkLI0Mnx6xAIHi~L0 zqb?ga&V6Mq`Y``XEoX`V(B1~bkgWOamG?cW$2Y6q%r5U5u!$satHJ~nh9`2Fe;=2v z)7T$fTEdCu=DcXmNT0KX_xP{;-vlPV?qpH^&oeAwX6D_VVd#UmWvudvgv`*5iCuSo z|0(7L+C#TRQ1#%D<(n%ikA19+qFQFxZ*FAZbttEe z2`JQluJ&@>)q6GY)1$k{FIBB_CNd>~%#CgNVm!4RZ{5e<7pb0+pfI!h^ET2bD#Y7i zgGkv;`nv=Tc<#kX`HIL(<93f_IRHW{I$mNuN1WMQE5^SxJjpA+7Zh~iF9=mgy}SKG zK|V{fz5HoyncYjvjH#L>Ud(`c%r6irJ14TE^`ZZ~V^Go`?U}ur*}F%A?E}I2_~_-} zUS+}6pL$G{KFs!b-Uk3t(fuXxf`f(a%hc^Pk%}0fE<|A;f!f~;R;1_mNUT-ipt;^2 z*_FnX%O1C@5am}!3TzK$aTnQXJ;LIPuNQJQp4H{pvl3AY$DOumy%)5wlE8EJ@_Kh0 zYINNdxMRFD;=ZO+KZ|}1b~ys5)mhMCiEY6I1xaa?hvzwQ!}2#f4hLt-^aC~aZ-2cW z7u$(Z`(NXmII5k`F!(L5@!9wTldFnD^_EMc4R`pG7=hEqC#w{brQoAE$&3Tk4#>*(&O>IcJCcs*aDgZHv-YVaJ=GP+ z9Vept#LovW`e25jY$32jY@U_yzqDbb8vTecEFri|5fUxX=StZ+2QJK4?kcw)eAoRF z3!mcr7*Z#mPZ%5He~;Te@W;5PJj=q#mJ2T{hbiBpd=P7FaJk!I5xQ2kG2RrsTfMwExd{> zyp057nYH*9r(NgH1Zu~-%BenhJgZ|7@1@yLALyfdCH_i1QL*UWJ+I9QyPJKD@#EhZ z{hauJYZZdXPjAhn5+G#OB{AdB|Wti0+9X!JzJh6 z=>ZccMQUtWZH*__ie6n6jRN=0h0n=HEJR+3TDdvQCXDfGM0L#c{Dgcnr=i+RG9ou7 z?eJ$e;sv}gn9C|rLt{4iG(wDz=;U-?`WNw%sUa=@y;qO89nyJ3Ip0GG_ZewUE zn|@1nPUZL#o@Slf`p)iVMX^r0YYixk#AS84TR1e2ptn3gjH>QB=5?cJaadta`P$j> z<=iR(E34+H!kHbhGG&H}%MC}W&kSYdYy%ihh7YcqK~zGJbdlplg{L8v`#wVUBQ8A? z>BI)w90rj0sfkKdCz`IF#UqzV#9^+pS;-PmBcooOVhwOvZK#_ZB1Z6oxXbU~^c#qE z|BZSv(qdhxvLm{p?5N$Z-oIM%Aa1FbSEYY@qDl_A{-!W4Rnl%*t;p^&T7%GPmu5a= z784HRc|Ik{dd2{L-}7?}A9#k1U%|yZ7#}hp0?C#{s)=>dk73ru;rmtv!ctskSR~cgLMOy_Bx2a~`fLEUS|}z(hw-p9!~* zLT_*6>osrnp`&B+ciIfc*YE(5JCe*V^A3J!%~(|dM)Bx7h`!I}Omv=03xVZr1pLtz z3tFW?kN2KkJHh9Y-a8}QpF<7AH0Xm=f9gciHq1*~h1FV>o&i1`LRdk$!JKrnMaX}m z=#I|+pqpaY5k%SYeVFgt=ukoTFHS< zQ&*FU$5K|S*jFiS#^b13w^3Yxi|aiuhC=Fki`5OAg=yc8F9%#fuQ=x{J7Q*PYm

9jnwSrUM#}RahW{^ zrQWQApui-uni%WX>woTV7K&VMpwm7J6LCx!1Cx zrX;5%rxj$O-qzjTzSvDjO!cNaIoyNGAM-H%|DL0FuT0Eb{x06Zi&dC3EVmD#n{s- zvXg}Lc&St5-cC=2rt7s97Ay8&2~U_A6zvtk);GsGncsQos>pr`SvzN-Oz3!7778RC4jbs)?R$jQlhDc$`D;+5Qhbz?frv&J3v z%f!3--*ZdUe4m{gTOO1r{ZJ2qmbC13mYC)XDy#QIqnoB^&t-#jwzn>wc7|L``bEE6 zv?uPL#+@@COq;Fr=*r71#`ulatV{{LoLOgPUVRh9@So@oA|DK3>5 z3*i3vR6b&djOX6BEL;>@bPCPQ)3>S8D&?RgXERfOOtfENd%DXSr4uzRlCMyigNKM- z)9p3#WVe`^$=4N77o>aT;<`!|wrP~~D_kw~L4@gUt;fBA^O&;zP*T=;-2(xLL>@?~!d!}$Gv zlcTs}g=zJs=H6be12O^@r@6(W@%Y^9j0oj!O_I1LbWfgSX4L}RFzJqtd!j-H0W3F9 z_F|T+S=kUcia2zIrp_?g$4p0E&a&5~O4Q$L{!c1}A;x%YG69T6CaI3E#Cga)bf5YzWNX^uHPL18M~(h%k7z;Te=jpu2` zh^bh9oE%$-%e*oTY0P5?&OHSbD4V1E^SldShDX^0I|?$(|;-*M-U=@bQ`-#FT0TYkP6?uh3iR; zHE&w?F~67pW9eOegGVgmT|{Z)D6ccRidDwwTky~-pd^jLiS`5q{I*&K_hsQ7ulVix zjSCGFtTE6^HPlM%ZLG7;o0U989@sXOP6K$iIvl4)a?@1y1L^4qfRzm^jGm&GN~M%0 z)V*HyB>sos7@Dhwinu0h=+tb_f>E6A-=t)Io8IvtR*EFDLz@Zb_7%nMv5E01QF=dw z3Gp#zI1aGAhN3hrGj73cO31VSkxtT~Hi{V@Fh0eP$XJZKU6PlWp z{STPIjH{2HLAV5wm8;knEf1N%QB5ai^?)dP1?Bf}TntF=e?DDio8GfV9DTVn@HK(w zn6hU$YXPsKtcUg7nN^ZL05K`A*o0NbjQ4DTBS+C(*`$=U#>GzEL|l|DKszU9xjceA zWBvkkD#VJSv*}<2ue+i*I3?A}O9+NJZOmmkij;x^&U8QJJ(m8uyz}dY-jZ&8JOD0un=F|x`|H_!LTYD> zk$~)KHRaeV|ATtX9OcU*ALb?Zp+ffrf47+}kML#vl6S#&q%;|QD-|60^xNPnAwxW9 zTA_`CPyEMntP(#rdH6AK@R0<03L8IHu0pcZ)#r}>?3}_JCZqaiadRYoT<5D5$sOi9 z^+Ux;Dt?Hz%AX-W&qv_$EmX12ZxoAc! zbbU(RV(QKE?&zz%n-zIdqnvjBHtkx&aE49Y+}K0TD`DB2{45DGIq};M+$bY844y7& z6EQg@yZE66bx$N!hPZC4*b3M5D_rzwj*XR%zulK;bBh5}lSP$fDi;oFTx}h&7g+UZ zH%!~&1DlBc_UXt1C?qNAR&I~_sL%y?Vw#+F?%HA_0fSpJWihelTGT%A$B4-HYtBBql;is7cIXnDwxl_kvaKU49(~Y)>zd`4 zSbjnRNqv8MmQ{J1wQ2(%Ygn}XHiy}f-xR8~v^KL*&&+IbxHC97cx7?s#saDChnd;msJ>Gbf@8BM9#IbMAAceyCmI|%qRgJ%2kbu_Y|g*lZ-i+3$eNEkTBbaf zt|+gL3vw0P9{7fFMTj$h;CAPIRq_C8W7f&#YhM=)qH4cr7%);(oR@&esM;Odl?s)R zMN8vGebeCsUau03oL`!S^ZO%9NA#HwcN%|0va&ml6$}$UxGygJ;l|T!-elr?-xeaC zt#lAat{(n)4`qH1_5n7Wwv5l;J&!hecvWf5R!UvFR2D_{Ii*Z}y%gI{8KtUfcv~b)q`;prsII0X{79|T?zdgg<7stE z1^Zxn-V6DSC7NZbByiqK(ED}X_KNdL*C`?Hw@jH=b#Cd}OM=FN``di`h{`t#oi!Ym ziM>*41|`2!wO@4Og(cT*-DuW4;2SkFNh{wplM#?U7vs?Ee^??1j(u+ zDIk_M*n^6=ooiN*U?gI6$@4ZPMc&onoke*0*~>84$fflaPfg+Z(tbwKhBvCAMPUtK z{Dr)P;^F`h<#xEyUM<%8g)psW%~0if=gV?-sIs-}nZ5v2DRVm^zp;*1I&4n7ercx~ zrU<=%b#;6&U|Au*E$=t9Cw260U1H{NpZoG88h@{!il<`eH<{bG_3(|?^V&2vIbtk1 zRj?zgco|XB*2<<%Ko_<1av13aPCEp7omv9Q$D? z}BN^4_Mn#r=G%YtK)+c;3LA0k#Bixe|hnLP@t5< z?JoM-Z(bQk$yTh?Fm^ERqxs^S7MHr)AS`+pQ>ysX+YXi@6_CucG6Qa;mfFmGq4#y{ zw}IT?0^V2K>;=XRy{?6Q8NVv}#qGuYcsKmefCf=zVF2a6@#(o#4!y(XAYZ_R%Y=!{ zQoQ(4zgA_R27BUzeEZq@M`(S;XBjNLm>kk~LF%EVlw?1D(`zT6d!q|l=Ibo{gDE4GT&L3V-5Du?X~m6{3hce&%cCt%CNnJ}nGfWp zcSgXHCwobyNl%x3-{>JF8b{~z2(9x@1&nPFs3kcBVG;6UTX%qCUDkd+`hcn;U?7!L zXskeQ;$n6E?H{3$e(?uaXK$@TA4plOu1iT$)Cn8w_=14ebYhMfXRAMxp900#KyRLv z_1l_?uom+IGBUTMM9T99a;7JlD^+T#8_f2RSC);{p6uG1zULbW-%0dzwNOnTySwDU zI}zB*iwwRfW8sO%LWEWb>8Aq#V?zmUg@wLl@1LSty+VRf2czpirZb~vM0ki+K z4k!Kf;HXI35&0k?@}#V`QMYoqiJvO{E&ixS#NzV%)6LWftLc{})C$)OLb(s=F`c|-68h^CnFnc*;LO_^)wKeB>C76>_7vPNL*9g$VuiU1g6x0#|7S3>7c^PyA?)m@ zroGzF;7>aY&?O$bzZ(9{)Fopwbrxb=IDHeBp^B)G#<`9we|ix=*gs(qvRP=rZf&FUg$Fpw zLM`h&O+dq7vLfI%g>#|yKf#}EWHE(t?VE=eKBo(<5xzRF)Fhr_=?hMDA*)uMMjd_*qRoTbEww(0G_50_SWr8Y3~yNF|eAp^-$ zD^J3t$@!|Y; zJYbL3MiUod-tiY-uw1B~JY#K^YDkX?bv`cV)1j+-3NVGyeG91qFW9k+m@AiN9-1bJ z(gq;XO*64ADL8w7y6#-vDOI9C(3)Fw=v6P~n-Zi0K#!%n6_r*>|II7C?HAdZl8?cI zLjw*m6LnxtMB%cD2@oA-4R-cXBo8(^!vRY$-uDvO-TT97uZ{V5lg|eQ3vuV|51=HgX7TdFvmj(&X)ftk){m0UJXuy)42np4Vbbv#ft%zg8@WBSUnc$pCcI3HLQea?NdWsH!pGP_vE56rS zfjUmNURzi9WdJj+!0m>^J09 z#?A;GIUMf&Pb~n>JRZg_)LeW8mc>*u-=ZZu4TUM6g(|BBv~TlF&mYtg?S$}+8q{Wx zf!O$!C6cI5Ox|9$ve(fRkE}&e$#}bhZYAWM5K3_pDJE2A1W9D$4>{pqDZjaBEj?14 z{N^0QG%!D%XAYfqCZd?`dY;2TFjtK!OBI;_TK7G85;6;av#E`UJ6^fnfT6PwyrK+> zV}Le?lcX@RY?kU*K62QH%*&-D8Dam2<0XhGS$lk8#3&=y`2q2_ls&2c2m)$|LK#7> z0S4{=RvQ{)^pv0wZ+CYWuW-;0-h%dY@}*k z`TCY8T-A*5@^ExZVoC~6mDh>p)ZCzm!5!1A;9z!ZwCc5Z98xZIdpWstGK z`WIjsQO}s(loUIZJq$Jw)bY@lNX3)ArX0)L^Fd8cJDA?In26Yvf#g2h>yE7JQyuW? z=ruSsJY^Rv$QLD%6=6mlrEZ}y8BKBDxvY~1`Ice^Oh}p8Qw)U6Ef?C%e6bhq$va|{ zkM8?4=0jFgEgG33wPkGTc_r*OLmV;Kx1TEce8MDvQ54v4Rlh)iVuCuZ-{HYRJ$-$X0Br^#f9NRiAVeFZr5`j>|n|jKuE#ppW)-t z`sIUFnE7R&xsdXSlV^C39abMP$H_e}W=>|MC(;$3zkr{p99?46SwQp`W+$!3ivruF z)7mhqvNGdFZY8Bstf)xzFI0o1)o>_tyPcIF_`bbYm(;yu1=@dxJTF_MPd;Mqn}d(|LN{P&B# z)>Z5+&7+!*X4d);%h8{d%&7U}x2&k}-w<)*3h;)OA$w=Zm8JJGw2Di{LDyHo2gQt(+|bOIF6Z2iis1le!6~`&FNrO0B#sgYd)2?QkMpC1U=3pkfcTosAV%{ z=6{h_F#Qp{+-Bpq8qLdOIceN1wh6a| zkaZ*W8C&!~tpZCKK`8eWe|0eDJ{rsIa>M}cacoHs|GGeZceaj$9l`bfvTq!6z_20C zOsyeYhe8bcPMNp;l^`K;o z`jbD^T#W@$tpi12oe)PUz-e zkE-8dIajs$e`r|BKdh!jg&r27a)@njoZ~Z|AYANW_WG&-f2UiF1qkKTPUL6^1xwUe zU$H_j>~*+It^W_F7M(Iap;gR;X@Ifpr5QiOE`;if!vEclARG*Bn4M|yNw$Q;-EtaG zVwyPmY&$?n0w5hN;mabgeak7PHa`WEk;=@uA@Gh_g{0^WjsryXc`=cOHK{m zek1A5X$kCRZ0P!rj!(;$f`x6SKD_<9hWrOo z3OOD0r2exw2u-Xix?&~;ru;gr(qS6(-wYePgg?es}YZANvvAO{#l6u|^)g3)i zdait#2|Xt)S-)5kzDjdL_bXelnh`Tf_fJ0YMN3ToB*|6qv@O)GRYrmJ!}Z^@d&E5} zKi0OpNB@+G!58j3fR4F01;TY%zC6Rs>2u3KOB}NZW%_39i|aVGCvzAT3vADVeec2? z7+18+%^Vc{%rK)8&HUWks@)#I2CwX#4u0RAqY~M>xb9QuUHB4e4e}8j3|lZ61xPGnK+L0MF2B-K=@3+V%qtF!eVmQ4c) zALdVclWLpXC5@))%zUtVlxS^{#-DEa{1nBafoXVqd$#@ik7lkRV~%K7D_#&o%NUQq<6bks1tV{K{C$xkIyzN_XOW3VQ8j` zxrFXDsNvQ*WQstNVoH3D%Z_R9nmn8HH{6H2)pF`_2W?HSr1rBL7d!VFVyD>4b1o%k zBCbvyieBpRr#0Ah>!J-GbZzz?pL$H-e>MuJGi&(u>1dTG<+#Fk79g$gkam8&fVi{l zS@0`yzJN0~449+GTIZg8;kqvcpG+vCq-;3rXTR-%$kiVi$OMX{l{_>W=ix)xrVXM{ zf~BDcMLRPwI*Ly|QxWDIY<$0@Y}g$*h0Yvj)cLS?Sgb5BObvkncZrI(J}cBUt|WQ; zGOSSg!=LF7@~%%e0@$xbNUW-=IG5)2>zs-S1K6PclIy*)xx*N_JZ+fdR7_= z_3;o_o3rV)Bz214kR~X7wo$?kF{Fp{obRgLyPtIimV^3H`uv3;#xph@q5iX_*Hwm< z2!pp9DfrB1ip|rUGRla;-I36{#TKH8Pl4Wpt0-13nY<~lF^7r5xBfbOPkvBxksSKEkpC)Mmu~TM9f|b>iz52tnzekf- z(nQS*$*y~RR!?UoX8AW7`U<&lwZKSp;p}d1?LW~fCP2+&k1kDPEtqc?e;ACr!zx*n zj~NHCu!1TSp;Ho*tS&RgHEP%S$JLvp{dO9M&AaJ1w(3J^T%!azNCa7Id;yI{N}?Rm z4^Wokd-R8>O&!l_MHMJ0%dE*8$xCzM^vRKLVVp$}6zg7-km|lMUa3*3Uq-err{S@6 zHnVadOo{n0P%4J?VJmIf=s+K#rp=d;M2dYcm|bUlm9`TZ4C*z(h#|;7Mjzk_H2VGx z+^YdBjpXw{#PwTW89mt$-h_>68!4yVgoCnpF-RVNd9N4QQNo6xoo2on1(YesqF}7i zNXb7M$D{y`51<_(0=kWE9y*R}4o~@b3E1f7gRfFiZIa`nxt4^QbBN;q;$XI!<6ZBRxz$CRvB>q;+7E;>!;u9&7D};TAbEX<5P2 z#;arFqaNHc5R)9}ebj-$*51va#}(%La=ps%fi*r6{P zuH`nV;@eXBXgGT6tRmm|DQEz?#QGV=o1r~29vCB5zsXx>YmK)PyaoM)e50jg+Iv4K zA^35HeJa}~S4Td&^j)T*_;7wCN6?U#58JE8uNcI+3`Z^3mwl%bl#Gm>r@hRyJ2=hwmqi4{C?b!3k`IkW!OJe(qHDo-+nQjTeu_gC8 z2A?g_G%ft?sfPiwH)nS^T=ty;V6Y57VTt8DTA!vGW8ChNqnsaW4#WJYi@|hi3wovq zGQL6U$k;*L9gO4S2oz32ZQmJ9&~xNqrgtA_xo0Ku{b{A8nSc>yIVa)x-n8mON@bH;c#z~ht>ZMrFk>9oMWMdORqPhOKxh%H`<=SPMBXH zjTW&wdp77$Stdt_lJd%jCNRZQ73;63B~poL^(w*s-A1V&V1;DWapqCcSW*eY0eeXN9q-8kUznaJ{w&{ zqFH&=#tWuEy=PcdL9{K1A|_NM z8W512vw&m;BnOc!NwVaeqXH_DBxexGSz-eX-6}a}8fbEc1{!Fffu_6PLGQgY^W)90 zxijy5-<)6Y!KqVqPSvixcdfk^rnB98eOipo4@o3OVvBjeD``U!uY(a;a7@_UcME@+ z*ZkbzZ7OpXMx_1dbq59OKb8s#&Yu2RRWh04h=W_(gy=!~yCMNl$=5@47az7C2q zneu<+e?ENE9RshZWQwan4Dn&z%wZc{9w*KFrJ&s36oyI-*$ZlNV<{Lc&+``)~R_f8t{7JD8#Ylbm3Lr z{rIKMNo1jE|Jq>{ncquF88%lDr_QoJ2PO;g#!YV1Rc|t6c#V~G!iV)v$}{^=qk5Gl z2E6K4x1q0)tuHoyQ@CxFD&3rh$PZ*jPOiOPHUCQaj~DH0lBHg)uIQX}ifRe#y)0Gg zy$smQ%A%!jgv1@7N%v1rVX-Nf3dxy=PUqg)H%EDd^@r}7m9i4@@;%uATT3@Ev zhh}mG-ZM58o_8fCL3MaG!{&4GxPBfG}tMqoRJTeD=fV;StA~pF` zE3BceroURz-^in!A&kk;z;OL-d~`jJd&Up6$xqbx_E%jcGTjYcM5m)G^k3Q4!|&}Y zZvvnQlzaAFW}zOYuD<3axZ@d^0S^w|YG8lnGS>5KLH>-z zXJz{P@n`icVP)Oym^BZX)N)5hCi<}S`#Ill!;RvcNl+-)(dzsc~!IQb0aO*x+1S#U@OF`|Vv|3>zu-Ad#x+r18A5Xd&DU zEuHBmOzqsBsbqtPXMNJEnxEy;4!w0$uW&U_vjzv+bXafY9VqWp<`t(%MyAW~gj?Q? z>X`9Yukjc4Hjqna2n|IR@6Nil$S9?tiLxTzcsL!yMR;s{+B+y-(_d}zJhMp`stx!8 zKRBM~Gknl0boo8rze=klSC}w5IW|~14tG|knLord_`W)wa+Dqtk1M?0Gg}m#!%+hq zRNIf&H3SVT)eR3&#}u3i_s~*J7-qoe6m`Na#~KZ?J~y(@{8$UgriEgWGZoSHCrslK zyTN=qNs{Lh@R2U1nbbZurnkY1U1=!thwSsXYB!~S$UziNJ?VuM=p>7Ejg zTDs1dqojoxNRrLNuC>=7wlM6vW1kqa>A^{M0z_Ak?#7vi-1cuZ!@SNreNuwP0l(@@ zuvD*Z!ybs3*cKG1p{>l7|V!8>`W{6g?dE<#_h=7&0&O!uIp^c;1;YK_KnQ+QkH zV_<^q99(R}y}kKyz-9#0KTTI7s(w#TzR~*ug1i}f3V2A8P!=%@i_Y_n8cbtICSs6t zsa8Cln4Zgz`)kj+;$cGy=QgmtrSls-z87J;a# ztb~?3@!#&;l@d+zqydV0VP1vS3&v{V$4at-N{Z)?tB5+R#kGeKLcp{&nEb%&fee=* zImqbJ9R|Drn18Toz+{%1-ny4}eE7GWbDkLOuiE{2zXbjw5MN&~ZDcR-_+TG#DzU}E zrQd56?e^79(%w~3Hf5Zp2;Tkml~=ybLCkHd;F->75*?qr{E)va%u~oDRiON%`kNoZ zt@DBHJA@zsCoSC--twTTjm4ZHAQn=q6`9|AyS}9S%y|Ur?VH7CYI{?en(8?4{`)+7qpx%GiypA&+qI z7rjLtA<@l=UT;((WL|;Q4eXQYvjdND+R}JAI_={lWH(utU1Zo9brA14mx6Ed4fc2m z0q)V!`QfeZlquDdVYz42YKF4DTPg}<;=JBtrdhF$%=CBEAkR}uqGXOuPEqe#T0Hb8 zj@Fi#1Jk~GW<0ak&ntI4*C*)tWB;eqFik<7qBK?E1Lxm}aR1BZ>`d?z7pDELgCNQZ zOQoASYLU=DPL7e{QW?dX3Y!hR;9FuX>&NoK(~p&79HoixqzmTi7ziqLg||K{vDEQ8pb>-^PY3WEa2)87+}M5o_z7(i z8@ySkWUS6p#rZ+)j=1sMGej)m$REho7rkcfe$4yMVnMjb-t$L~o%i0>8o&ml=LVAO zYiE2R&5Q$1o4wR3n00-2$h+1DXA*3;VE3F!#QgYo)W4>+Dn!dyU?1z%^weHlr9`pa zUla=c%dPgJZ?tl6&V_B3NgR-vXs|x%7)=m~tr&G2ynB{rs}K`#)Gym#iGQ|IcWJT- zXbF%1EJuU}-O{4@bU}G&6Ab5y?Qp`_TZA!qB*8t4U#SidqyJKm1OHmP9_XJmc1y@_ z?m^D(*Ef68-#lEmIIFmuo#Oiu66k=jE`BQGyi>cso7?_#{d)Yn^~K;%c;DmwlwEtr z_plNx?gzE2-HUB-5KU`#zVhTy%Xu61aioPCr073(waZZDYWW8@s z$-pBxb72V=2|z<7U;3*XHB(mUi#6($&((Pf z@^0*quQJ~F=X@?!kBUzmB24?L^81FLu>3oXc|rIeQG8Z=zkGJu#I7sVJUP{K+oR0i4F)%x#AnGnDh^EjkjKqWEMGZJ1zoV{`vMgJ zJ-NJF|Bkts)dck6aJWtHq}^K?2buI?!o92CcYe$|r=lt5A)%C+xVo~wQi7p#6 zTi>NMkf;7zg}&m+Qih_awpX0o{&&+J9R{r0%Ke4=hP)Xn#sAsIrHRa*<=ITfA(G_B zNRYl@KU#VtJ&VzXgdCwkGR}!aLss2DF|P}a&y_fACx#HrL$k=VHsAdHj+(S&gZrC_ zH;H4Y_P@LqA*?FroBQ#VKi)IB!_H+P#uC$As&5zwC--_Reo1`&_>Y26Bg|jK%LeD% zldLV*TpCot2?R?wNn&+Xnp2Bz{&Sa=O-u}oGai`$hWVzF4f0)4$6=24ziHv~!P|b=u@HKNfnJ=b)8Wf%fIFI^^ebqJ6+pGlf+X480`t_qrpnxR||sHbSgT^M>!Y^ zYks8y8%m#lv-$gbw@N}@RrZ5}9(kdI(Lo(V?1}U2{P7Zbi$S0BP)$_k{g3|tR>-To zGJadb_AD-Qa&krup5TI9sb?;o``(jXNgfdz5B{|$F`})bqEh|n3F6NAIhbF^Ls@RI zQ?>-%&{qQ-BYo{jRFF*ooNy)tEw1vci0Z%I54*A9mSkv}wpZ74~w;6ZLqbW)1&7SWA0p(DK2=UxdYX zAA`omnI4Aai`={sJGG*tv`yv>DZC-tLIw}I(BRGywmef1bGJ8CVHf3*n$8D;2EA?E z)?qlKTjf^vg4n*yW+|k_Y^YAYfN>@{#^!H6-zVTsbU?;iEQH5;K-{?kAyAbf+iC7L zdCYR**|M@u1jSYRAwFoOILomXQ^h3NO*+n0qTN^Iy~@%MH8u)=fK#D=yZ{Gh6@^ozht*3D2yH!M~zx-jTQhJY7b;obEE>d$4i)v{~nzzRxi{3gHHVKXsZbR-#5L1 zN|yw;uiw8J&hE$S!0Wh5lz|c;hsH~_lV>c3enXxBMUe*G<;ob2Z>bS`L`$$g{#Sov z;3*&xsVsQ;7a!&O$SVXIBLCJ8!N1KwWOgGE43LDS( zh*B+YQT*Hu_0mG_WZ>O@326-`H`nZTwr(OADD{3{b|_?2wf4%J)>cee+>pkCi>Q*? z!c({(3C3${hkr=_fgb$ouC)XTRzw%*&7MNw{Qf&ad z>7Ak$TfFH1e`Ld7{5>Nl&>-bjH!qsk2fM5|PK+W4NWk;pPRC-g)D*zh{nQG`EeM@{?g-DXeroTh*_L1S^9ady&| zhOtGG4)Bj&xwFk4%q9~$OM zgPzx`AFGU-1pH5S`V9v&MYX=8{(d8$$F_XG%5J&dD*eV)QBgBA$7Ar;KxtWBCcxa} zK^^f_!*}CUnUAug+7kdVV0~embm|Vmwn`2MvN&s@{M-;I^j)ieTN01h=Ax`v1oX}V z&(>0iq&Y8NT1ZjGb;6eWGL$lkGON|6>E^vwIwyY`xQCsTKhDn;HAUAV;nG2);2MK- z3tq^{;V)Om>w{oE^nkN`J-#JkT9)eX;D3?REUKm8&&)=08~tI0%(Y^n^IN7WtOf># zDh#6R{OlMjdm(myCCd2vXipPF>4Tq+FTs4za%yami91=F5`KX=GZ<99y!EHwM1)5A zDMiUUUfwE7PF>6S8h`X2UGN2eEbHGr!KYVpdGG5$>bl7YFkt625@lE=$)v5BNl%U<;Wxwfd|8$kJDY$!i z(LQ&1x6W;A+y9R=qyR?#Wbv*P(oy1>*fEaMXS%vH?Y@j&lkQ#Ot@hno8)r5e4r^sd zCkme}Qrdi2Zzs0uNdyUTq$`X(dh6*2oDrmjHyPbg`1{=!Z<)xe*Hq|(%dxIS5r0wi{4ghg%o0UVeo$ay1 zj>L5ttE=z^FFTLZ=AlBT#yIvrWgw6`-shNRqG13dID0MyHH4 ze7`1-Dqv<EXqJuT2PuAE1|E?p^$yCV_h$w*mP&qYTVq8J>i(LPMO*q3yOa8X zI$5=rhG6$!xx}O;leF^{&0fVD@y8}^V`meBDOrlieV#tH9xALQuN{f&GswKxel>j0 zd*Iw&(qLtOUT43dMGs5bxjkjOm&{`<7I5l9-+lxT_8~>&t~4(W{+oT#|CPVZc{~pRktg>2 z_$YX1z7^BQ3wUAXmsXcm7oaC@X>C<$zWx1_bWwD57qj7|m$Kh}xWE7O z*GftI{eLy%&q6r&(+_q4W$LuFTl5@&zAPl*ES~2EI{JyI06+i8@83%>(_rqcw;>V( zX@abQ`-641V~uPdSSJBV=pwZYmaxs?+_tG|3zg<4Pc?X3k4AUlAZ{)$Djp;K(fX$k z)Y6}^Z;^#4$h^_1&$lP28bMDW@@Uvqa(?Kn?RZL(_ig_|LJrBf+7MSeMYMb zC^C<{eW2KpKDChR{$B)+h}=*j>4B%CG(+XJYrExpS5w)3vqfW-qLJ6otg zO+Kf%wZ+H#GPiJr1x=R4s3m@j58^%Z?LN%!59o;Xc8IlZ>y-Q1YX*CF2`^-eP?kQ* zzHWPC0WF|dV=%@$w03;RsTZ_g`;9pihWR`icE2+eIU%69?r3eyy$|P7NC?9V^WrQ< z(k(QJ(#OUrs-5@1vRFGkmxk(}KRu5fMiFat4JK;r?~y(n#iYdkDRInMh%fc7zPkw& ze3hB%@wf7oXAT}4*L92> zxce(;`d|mtV!lgp@NkUhorkuigp^BCVb9gVe-)d2VK}6k{PY&Il+p+u^_E=qRg8jqPnH4TuwQ2^*g9I?!atQ zuOy_r3MqHsIWaxiNw9!j=Z9-SDObAz5z>NCquhm`!n*NW+ohELZkRER=zIE2Lb?-4 zNXc&48{k-?Xp%SZyu&6z%$MTybN)>=x5PLPq5TBSu?E(|E#tRah&!}z7M>PlT_BLb z?!f&`4U_m6rKMGp?-m6CfW$XW4_{O*uM`dZ59HUQlACG*`LvkjX(dFP>R?4txn!`Q z#A$${w#WXtElr;?$Xnh2B$Vb>xCCvTirc1GIk>i5&d~^RrtSp&8IL}#BGF^Q42FWm z0CGCBX_xjWUa6c>P&4FL7DcX7^LU$S6z1Yxl<%%4aZI>ESf)6goA`47VG8whz6lOB z;jJrwNZoKW+fE4}+W}GKz}#Gx#FH4#nufcTQ1-@xK23>#j+jGHwqq z^gOC5!&yx)r1ScEWjMxyw;Y+fLX`|nII75N#(pMsoE9qHdbq(u90^fbIb7iNgynOH z<)Sj{IN4QDgvg0e;vT1cQb$P1(?wgmzn_B9CH&7PEF~Pun9A(12?D zHjY8zb!**|H9OEFDaVd9)fKp)kfuzqlTBsW(RPfsnFPv~G1Rj7c>Z_G^}zH>T2Z$# zrs+EB**T#;!|}l?Qw1A?XmL-8#-D`pxn`4&pH8@ov@8fyoV#6>#C?Z)kC#}0^Zdma zD*@pS{4_6Ia=-8kA!`27Dq_vd+1AMn@phnSqdmtf%*Kzz$}YayQN}mR7fAlIG5AjK z87kH6)U%1j{u%$G#oDGC@0lr2jun2}O zOC5c--R+NDF(CB$O1h|NH%oKi>yG`pdtgATX>k=xXrbL|1AQrF?OUggLBb$XG^QBa z-6aG|m$FWuL5Jx)^e|&2XqN>UGi2p$oOF`c8RY=zw=ng+H8y=Hc-}O*hQini+GFOX zGXR-u;DTWll9+#`Si6M>wOy1mt%L)xrvfWqxh9Cy1cbtD|Is+KnJ0Gg)@P~R%`r@C zQ-;Q?O`|exPr+GHko#7btl`P&x;9U+obqHn7|Lg@h^i5z7avCo`t=hN1{fXs6_r-^ z_{APlQjeqR`hy<6(gcc6tu}2+%(uinUNC1p26>*H8yc9NueRcV6<%W>`+L_B^}aj| zCsbpL;5WtBZ@h_j3@@eJc~ePafQRadH#~;&Fr=X42J_a2`FTAmte9^ZuQf1sMm?J7 z5gf|2fUnQJ>PzPK%X>9xx~Y6x&zHbqka+B}9Gf$`8i6}{07S)MJXz0Om3%tlpaBa* zSt7u4X5XSG_+^nOwXEe87czF|uwp+a7;;ZL8&N zgL-o$);A&P=P?fRCd@bTWgzF;ahdq;QSM4lX>Ly>^0>m98_1~a3~#QLI3K--H$!wT z7!6)>3R!lvAGpg>2v+fvi*I<=Q)-k%S&_$kRqDHAabQlXDtt47#kff-)X;{m+%;uu zCN2+HyY)lVRl@E{_zru`l8sdollMqiuMT@B>7HFazJ2|;wX8{0_dQnq`}>2#Kf#c# zUzoaa-;dJRF&xJB$S}j)zExra6?5+nl<+m6LYK&~#VC~|o{{X>EaL2@x(K|1j@RD9 zzr=h8J-AY;K?&hoho4>AZ{My?s5jK|<|T}GSEyWsChR$K zNw04F+Jy_N+8U^GG64nL?DX5Zt65X+0IR0O&awOF7*i!KNS-Wb_P7QZKtj}VO>T<)X&W zvSTCU=_#Zm;zeh=#}3oZ7dIAoXM9fWj1FQ+HGjnP`!{*+vg8Gc-M7-icle4=w$2Tg zPWk^%(9m-q9%iv-uv$3O(HkWX=*8*S%c?c^0(;VszW`59{ZN=$(sALU^e11vphA?p zbHE9_H_L|6&9oR}rsy<;ee~op6)82f*E?L}<2XYI^?cCLSL*)B*V__tIk<=$(t%3} zgvxE`ApB2X*NT@Ui(3jI3hWl|`7B)=TQ7~)r<*#nm}}07L*?&)>PJU)j!RWX z3+vre^e<<|ma~=Gy36U1PiwS9=RhWtPpe-q48XA_gI_1KSQC*0th@!VD*uMT^r@!g zXjGyrtkxMcdp!P63-%aWifiA?4U{Bg~zPp)Ceq^i5Pw& z>(?mpF0sIJWdfg9fT@!IM7wafQEp3qE>cwtb#)mLpA&;X#W7^c;YIRDOr6RYPSE?6 zSe=g*?zcN;UDZLS;bE7!tp8|=+I`#4dgU$e$VzTpz)S?U%FuLEn=g^Xfd&f^(vD*8 zsLAkF`(PlOxgkE`r6>kHfN#L>Wzik-oxP0~2L(!fx=II5~d&=_MM^g=Z#`U8U! z-9gDL&c=5!YyU0|oFJE`4tTF?84Ki$l}5KA9+E6-ipf?vw$iI>u|YV0nFw^DIhp+9yEThK!X-{};dxhx7^lW{{zmbX?YisN9tcrudnDzM$C z|J^6r(hK0wXnwQHjx{B*h0R0WqOy5gP!n_+8pgAt9bGcfcl3XrZ?`d>)&(xyp?>~u zzXz6639i7^u_Pm??FXBvuk)VJ&dIMccs} z*j_|de=roTqPwRY|D^fO)^@6MkcRkyRoCNep>(zRB;WT%24}(*_FG};iHR33&g1Rh zd5g;LrEtUmJsyj(ig?let=d6mrU#nLrC%-FZdoko@(x$)U&4>$_0vO_3t0xeFU;pj zvK_9BsO1{ZZ~yFDrtEc0?YBMNo@Gz?qlQ}}cqixA>jX>VDoyTrNWlpC$(HlTZqe9- z8jQEDH?dM`eR?(WBAggc)M6Y9;t1Kfxz83ZARoYjyzuBvXPfACc0szG;h8o$Kn|Lk zMV%EH2YU+-$VR0rCmM;%q%Zs|KJuZQCZ6SJ7qGB0@m;<5DxQB?(68-tj)O}46Y$+V zeHQ%nPhyY(8Y``-c#@(UVb+#DV_6B2(y8^VYBf&laK4X>-6Zx{Nq+e%!;BOFZeCAdbV~h!|m##R?xtZ;DIbW8SN10M+L&c{g zi$6Gmdd6nNBU4**yh{K_P`B03B*|EEJJ!f$aC7FMHVrWA!RkCKJC-bIGV4-EcoU|0 zVQ|&e`zQPAX0_t0hZ?HR?<$9)3l>Q9QXiKmURukD*l4}G(bw-5AAARY^{nq4cxc!+ z>)|`?$JhWIp*?)6B==G~@_25pbh5-0)co0SLVc`Zu3?`RKW4cw+jX~ZkVm~naZb1( z5K}2BHI-=*cF~##OMSk9nP2lXqn@`l92u#w7=vp~%fzU^kOCYtWtna#$s3?rUl&5ziU|9Ys0Vp$YbLJ3WR!0RvKnZ%$mbU zven0OlRoY2&Nlv+!SAqdPsFi5?Ym|{*Bi>W-rteNBhU7qZPQd3 zo&Ug`JP9+~;Yw4abxs8r|5rrYj29 zF4=YzNfYcQXeaTl+ zO6+C^!5g`_9&2vv?CE3c`j)PUk~pY0r?j<}GCcs-$x+<$+KA1zQmEveDzjL}V9&n8 zIIR3C$YTRxRT%LrOY%B5dt6tW_s`}A!tQVHaw9z-@u6Iz7{Z2On9c4u+aZZ7rXE)a z_D@}iXbEWfnceb&U&{>9PM+et`O-9?f9aU^v>L^m3RGv^x z|7cDN)lr&zZhhP2`9Zz3tacG?4tJ`)hJQ}$t4e9u0|x5Q1kdD$l}2gk&Kp2a=;6K% z`(>tWMGohE$zGd?6sY8rJjLrNViksyPMg?=JWzXVI3ao{0)q;8*&*70>RW5nZy3lG!n~3m7 z8wuYwdw;%_J?E?!ERix_j{^`z*}?#3%_rHQwI=gaJ~Hq)&0`HfCi znax_F%5Q!R)o%bY7b4sT$}f9Sk`VvRQX42RIX>x?@C*ZrCII{s+}Bsnd{Khnd!jj& zAJF3Jp(I8~C?)Yruw*?)u$J};kNhy#ycka8P>UkrP;f=|PRCde3Q(#fC7=5@A zqM)RgYm-x0>xKL>)##>R4svSh52*}u)Nkc|mDJ~jfGf|Os+rsnkT(^pu~~>v5&5w^ zdL*pARtB@5EOoVlJD9em<4}0Jj*@46*W2cS(2TtEiv_;?t{ASWx+f9T{M_oRWT!ZO z_vTdKxm_Zg)(9NrPR)Cx^>MZRU={q&t2L)8DWnBZ5hphqAq_PSl!9{|6-L@ajs3WK ze#{SxV6;c?nt;A>ceNmlPlgXfMm-apZFTla!+abc-;1J0~kSOt9KXAhKseS zV@tE=rMRgR`!@WOZb-3@t=*O85_>}4ASgpB6D{82(s^Xu3`SMjh^F;C0C5?Tx@0ze0;$ zKT~1%ht|T#H`HWqYV9|t(V2phxez3@cC}W&_r7F+|4`D8=dY+(t4bFq%)z6m5gX|R zpZ>IZ*_m%HIlfc~ZTq1nTuEsUdcROw*oLi5-yo-(u2bcgb_S^T)rklsuI2~{11|GL zPz%LHpuM|N_g)@esoP3=?R+ODIsRK;%(M^dyG$@p{U{#b>9#Iu9y1tN;u*dJ%2lcH zoZv}-0hOatZ)@x^R^*FoG zl;?hQS$tGWO=UpN!xjb#6zztq4>tQx>+<3c@F#oXPrnKx2~Uw})A9+0&t4YX(oHuf zqGsUuNB%!)}L+RBlcC>(?WQ zI12aSDVVsiwuW{SjoR2rTtKFUPcKLa4!KL2`p7L`F9Pr(!$-qFz1q9CE%43F#U_QA z+3EUS(C8AWSm3_-No9N98p8&kgxGjy(C|Xb>FM%MGZl5pi1yUp`PRJ8c}@C>S=s4v zdIoNeLx+u|Iv9WUo{iE*|>X7HoMN09{>P*w0QpRjTpUu*Ur!CGTxNP_QVrf z$lhd9T1kn49{I*yjp+y44EGlu`a}7aYo8b{hf1F3as$u zI>DlAPVj1Eb1S(?KBuCRCdqu_WG3-JnhiLGPzXxV9e1}&-pKtjI3zChB^T>ZuNJRIJfQ_8$ zgm3RBV&8h2m~G4kOk*$KT&~}17EM%-DciHOPR_!%Bu#!06v=EYniQK^cO85}s@2$q zL!D-AOPS|x_BM$4!N-q;9?uXCJ+$cO*ylB@9A9OkFwv)6=@eureA38}R~V{mce`-! zLlx9@e5>r!w5y!zI$8El#`>S<{2R74DSBY*id_K ze3zusNtJ=YdfM}_Ir7!SF4t;~5t*H_yYh2@;0D=z_OAA86rGKy7C%G0XSB4ImF2}7 z4RhH8k^W*@pIyhKw#n6-o>XNCDc5=;o@3a-b>`nv!D5b9sdp9HS9cZgE5j8xg6(uf zyQFi5r=v*uU!hOO>Pf;C)dBXaU-_>AOm(VU)>V`7{NP!Qo-gicm0G`vMG*|#fLJUG zPMc{`@7DvznA&%80bm2naW-^lfx^Dy%c^9GX(jj5GJ9=?<;-Q_`APPWvO%707Dn^l zU$2V#>ZC|(p#Fn#32%Oplfqc+uxh)qH`DAHxl+BF7=?(m9_$|rtB>s2I*J0r_MV;= z$Qn@AfWpTzN%;{Y)HOsa@qXG@hTh&aNHam3MCyCJ_d0%!$i!|QgX%sGzzzHcy!gEFclCS8@ z(-Y##na}SWulhHmwYFXy)<@;2wXHc+q9p&|dd7wg% zPjWEOLpAoSE7l4*>y^(Xaq;`t)H2qTESQ^06l}&HTE4w7JnMaGxo|EuOmWfVyON8! z+fp}J?*I7|5p|tqR$VMeRxSVL9qM;QKnAoN^Xd&F z0z_jY&Y7j|3h`g%mS^tJ%z?6fuNLcUe#`zEE#B7`KU7p=y2);TnD1i~d$0LMGUuwC zoIur}lycU!)!VzDQfTpn*K_1ndIBsqq7a1+9!We|(Rh9ae(!_jk^HN1sjs@be^(so zuFsmx>Ft+}R0A6t-$UufVagx?%a%))Pmp!t- zn}uWYI7g%PTfBT|w*qm$s5K^nKNt{Wqd=$+P9 zat#PQ#)Zm2NIScD&zO{1Q4!hD(~j$V8j?~aO=W_WHn-&h#7&Xsy@7yfTfavnLGUZ# zm%m)cHHbR{yM5P5-kNNiwwyT*(jFSlU^C`}tsIm%>d6$5o^r!M&drRwS0{JeR(^H+b%A_mym^OmcuPn8@Bnbvuzs3>kTkk&d_S$)(z0-(jJmEopMR|wwT zo@e9`d$$xpJ}usW#VP5he!Y!m-Q&HVc09cK^E$%&U~mw6F9O3*U}R}s87SWo^W@duA_hS8u97;>coa_zhTB3p2JGlGIX%eMAuGr403b1 zTT54G=@{WnWLJivXk}rPl+{9hiY-vm;v*d#td5S-WUfVuI4;e`#0L4crU3-sxS}ts zvsqb=qP?RR8{5aqhhvz#OjJf^yPFf{ZyHY?E^mam23;Zee)fCf7Qj=u)cWHG)-u0m zrW&j7JMw$iA4N22r>NSfCVLD{_1~6DPgh(Qx5dd}IW)E281dvjh(qtu%%(GziUAVl zxcl(!)zHti>5vfT84A@7j-PrqkK)AE^xrrIF5MJC2h=>53K|)$V7IgTq@)yuF9DW@ z?~9zLl#JDz``LK~ew>ls4G?|?0Re&SV{VbVw1WUM=c1C#6%vB|{QY^0AK2CWo(o_2 z{9J@T>YB9>8_R6NlDF%t>Sm^rD!vj`L#UniheR38>EMg0dlC`hU@0s*{Pr&49CjYK?qIjoBGYk9IJrqDQj+Wa0zX6rR9&Kkl{BYfF9ST!Ze6 z14f$X)fchM7DeS!?$G8oV=nLss%3L4W3RV#1y{# zNnl|W64H1_MVQC9LrqO=IB!2`Xy*0A*2yt&dZ^WY^UKsmR}Afw;oQKSM`h|YLALVS*ZzdeGpCU~$%=koi`^mdu&!YR29r+LpS%QF=pnOS2jw+3nGH0mY5)>|Y27 z-d*kU>yA+5U(9)w-W)lRl6*~f2r=tC6dzTe?6tk%^ir$d_+n{Hm0dClZ#o~vTybNlJnjQ? zMycldXhnVfLJ=t4Tsi{BGg1bT7celqk*|bSVW#sQE3ipGm+D_SiJ^H@eKEhC9a^-R z-dAyQvukRb8cY89P-S3Bdbo90Eic+&?DXqPPUr2qP?lY~z*V{Yuh`1=?o;NR_Up^e zQ`qA;WdVv>->o9I)=O9JJ@!NKe|d5o9we-5vk^bok_(szV_ihX-e&dC_^F;<#0Kov zACo&j1A1=y4$R`uPwtrMcd_J(`;4^BeNov zrUyVLIf8&eW97HnUR${Lq{#(q()4GtgY*6HBCQkE6Vj2L9CeHn+;`op$rp?x?Ej)T z$?o*>qXzQ^*-T+BYq_Ok8@ZQhnx_{&J1^K;PruRQ77gj-gh`QHyJ&MWn_Y|} z)jp8o)~N62{5jaZ)q+lKiiD>Ke12_%m36sCXw9SCP1aA?Vt3vC`y-sk29ug3!I%ZK;FW&=vC07`)%na&+d!?WX$LqG0H-iAjUs z1C|S361zZCdsiLNDU8?F|Rh??>!2v+Ex>|1gI0>15cx`6YQ7YAH>;@PBk_*K9E)) z4P4UiKlAOYVrJ-R4!`U6C+V`P=exn5p=SgkHsNcFsV;^w0F;o&z_s#pX+_U`=`0v3 z5Uc?CRW8FKiOfU$2wXP(jMxW22rf?NLJoZ`yLU|YZmX`5QP9%vn6)KYnu-Q3x*X0a z$JO372y$6~<37bo{w?azAiMOPY$Egn4$4FY*KrFGnZ06oY+A3->KaZ&*?f=1MadbVj0yFnLRoE{ZS9+x`S1I%E `PROC GENMOD` | R
`MASS::glm.nb` | Note |\n|:----------------:|:----------------:|:----------------:|:-----------------|\n| Negative binomial parameterization | Variance of the negative binomial is given by $\\mu + k\\mu^2$ and the dispersion parameter `k` is estimated. Overdispersion increases as `k` goes to infinity. | Variance of the negative binomial is given by $\\mu + \\frac{\\mu^2}{\\theta}$ and the scale parameter `theta` is estimated. Overdispersion increases as `theta` goes to zero. | $k=\\frac{1}{\\theta}$ |\n| Likelihood optimization algorithm | Ridge-stabilized Newton-Raphson algorithm | Iteratively reweighted least squares (IWLS) | It seems SAS performs simultaneous optimization on all parameters (i.e. including dispersion). R uses an alternating process, where glm coefficients are fitted given a fixed theta and then theta is estimated given fixed coefficients until convergence. |\n| Estimation of variance-covariance matrix | Observed (rather than expected) fisher information is used for calculation of standard errors of coefficients, which allows for non-zero covariance between coefficients and dispersion parameter. | Expected fisher information is used for calculation of standard errors of coefficients, so covariance between coefficients and dispersion parameter is zero (which is asymptotically correct). However identical vcov matrix as in SAS can be obtained \"post-hoc\". | As shown in the numerical example below in R the variance-covariance matrix corresponding to the `PROC GENMOD` estimation can be obtained based on the outputs from `MASS::glm.nb` with the `glm.nb.cov` function. The \"correct\" standard errors, confidence intervals and p-values can then be manually calculated based on the new covariance matrix. |\n| Convergence criteria | The iterations are considered to have converged when the maximum change in the parameter estimates between iteration steps is less than the value specified in `CONVERGE` option (default: `CONVERGENCE = 1E-4`) | Based on relative difference between deviance, specified through `epsilon` argument in `glm.control` (default: `epsilon = 1e-8`). | `PROC GENMOD` also checks for relative Hessian convergence and throws a warning if this is larger than the value provided in the `CONVH` option (default: `CONVH = 1E-4` ). |\n| Confidence interval (CI) estimation method | By default asymptotic Wald CIs are estimated. Profile likelihood CI is estimated if option `LRCI` is provided. | `confint` function will estimate profile likelihood CIs, Wald CIs can be obtained by using `confint.default` | Note that by default confidence intervals can differ even if same method is used if vcov matrix in R is not adjusted as explained above. |\n| Hypothesis tests for regression coefficients | Asymptotic Wald test | Asymptotic Wald test | `PROC GENMOD` reports Wald Chi-square statistic, while `MASS::glm.nb` reports the Z statistic, however the p-values are equivalent. Note that by default test results will differ if vcov matrix in R is not adjusted as explained above. |\n| Estimation of least-square/marginal means | Calculation through lsmeans statement assumes that for classification effects the groups are balanced. `OM` option can be provided to obtain lsmeans that are using the observed proportions in the data | Not implemented as part of `MASS::glm.nb` but can be obtained using `emmeans` package. | In R marginal means can be obtained using the `emmeans::emmeans` function, setting argument `weights = \"equal\"` corresponds to the default option in SAS, while `weights = \"proportional\"` gives the means proportional to observed data |\n\n: Negative binomial regression in SAS vs R {#tbl-1}\n\n# Numerical Comparison {#sec-num-comp}\n\n- SAS `PROC GENMOD` procedure\n\n- R `MASS::glm.nb`\n\nA dummy dataset is first generated, and negative binomial regression is applied to the dummy dataset for demonstration purposes. Every effort is made to ensure that the R code employs estimation methods/ optimization algorithms/ other components that closely match (as much as possible) those used in the SAS code. This is done to facilitate a comparison of estimates, 95% confidence intervals (CIs), and p-values between the two implementations.\n\n## Prerequisites: R packages\n\nIn order to run these analyses we need to load a few packages.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(MASS)\nlibrary(dplyr)\n```\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\n\nAttaching package: 'dplyr'\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\nThe following object is masked from 'package:MASS':\n\n select\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\nThe following objects are masked from 'package:stats':\n\n filter, lag\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\nThe following objects are masked from 'package:base':\n\n intersect, setdiff, setequal, union\n```\n\n\n:::\n\n```{.r .cell-code}\nlibrary(emmeans)\n```\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\nWelcome to emmeans.\nCaution: You lose important information if you filter this package's results.\nSee '? untidy'\n```\n\n\n:::\n:::\n\n\nWe also define the `glm_nb_cov` function to obtain the SAS variance-covariance matrix in R from [here](https://stats.stackexchange.com/questions/221648/negative-binomial-regression-in-r-allowing-for-correlation-between-dispersion).\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## Helper function to compute the variance from negative binomial regression\n## This matches with variance estimated from SAS\nglm_nb_cov <- function(mod) {\n # given a model fitted by glm.nb in MASS, this function returns a variance covariance matrix for the\n # regression coefficients and dispersion parameter, without assuming independence between these\n # note that the model must have been fitted with x=TRUE argument so that design matrix is available\n\n # formulae based on p23-p24 of\n # http://pointer.esalq.usp.br/departamentos/lce/arquivos/aulas/2011/LCE5868/OverdispersionBook.pdf\n # and http://www.math.mcgill.ca/~dstephens/523/Papers/Lawless-1987-CJS.pdf\n\n # lintr: off\n # please rm -- variable not used!\n # k <- mod$theta\n # lintr: on\n # p is number of regression coefficients\n p <- dim(vcov(mod))[1]\n\n # construct observed information matrix\n obsInfo <- array(0, dim = c(p + 1, p + 1))\n\n # first calculate top left part for regression coefficients\n for (i in 1:p) {\n for (j in 1:p) {\n obsInfo[i, j] <- sum(\n (1 + mod$y / mod$theta) *\n mod$fitted.values *\n mod$x[, i] *\n mod$x[, j] /\n (1 + mod$fitted.values / mod$theta)^2\n )\n }\n }\n\n # information for dispersion parameter\n obsInfo[(p + 1), (p + 1)] <- -sum(\n trigamma(mod$theta + mod$y) -\n trigamma(mod$theta) -\n 1 / (mod$fitted.values + mod$theta) +\n (mod$theta + mod$y) / (mod$theta + mod$fitted.values)^2 -\n 1 / (mod$fitted.values + mod$theta) +\n 1 / mod$theta\n )\n\n # covariance between regression coefficients and dispersion\n for (i in 1:p) {\n obsInfo[(p + 1), i] <- -sum(\n ((mod$y - mod$fitted.values) *\n mod$fitted.values /\n ((mod$theta + mod$fitted.values)^2)) *\n mod$x[, i]\n )\n obsInfo[i, (p + 1)] <- obsInfo[(p + 1), i]\n }\n\n # return variance covariance matrix\n solve(obsInfo, tol = 1e-20)\n}\n```\n:::\n\n\n## Dummy data\n\nA dummy dataset is simulated, including\n\n- 100 subjects;\n- $grp$: a dummy variable with 1:1 subject assignment to treatment ($grp = 1$) vs placebo ($grp = 0$); note, variable $grpc$ is a character version of $grp$, which takes the value of \"Trt\" or \"Plb\".\n- $x1$: a continuous variable which follows a normal distribution of mean of 0 and sd of 1;\n- $x2$: a categorical variable which take the value of \"A\" or \"B\" or \"C\" with a probability of 0.3, 0.2, 0.5, respectively.\n- $logtime$: An offset for the calculation of rates (e.g time in years) on the log-scale\n- $y$: a negative binomial outcome giving the event counts;\n\nThe dummy dataset is saved as a csv file, and then the csv file is read into SAS.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nN = 100\n\n# set seed for replication\nset.seed(123)\n\n# Treatment Group; 1:1 ratio\ngrp <- rep(c(0, 1), each = N / 2)\n\n# Covariates (one continuous; one categorical)\nx1 <- rnorm(N)\nx2 <- factor(sample(LETTERS[1:3], N, replace = TRUE, prob = c(0.3, 0.2, 0.5)))\n\n# Offset\nlogtime <- log(runif(N, 1, 2))\n\n# Model parameter assumption\nbeta0 = 0.6\nbetaTrt = -0.5\nbeta1 = 0.25\nbeta2 = c(-0.1, 0.2)\ntheta = 1 / 2\n\n\n# Dummy dataset\ndf <- data.frame(grp, x1, x2, logtime) %>%\n mutate(\n log_rate = case_when(\n x2 == \"A\" ~ beta0 + betaTrt * grp + beta1 * x1 + logtime,\n x2 == \"B\" ~ beta0 + betaTrt * grp + beta1 * x1 + beta2[1] + logtime,\n x2 == \"C\" ~ beta0 + betaTrt * grp + beta1 * x1 + beta2[2] + logtime\n ),\n y = rnegbin(N, mu = exp(log_rate), theta = theta),\n grpc = factor(case_when(grp == 0 ~ \"Plb\", grp == 1 ~ \"Trt\"))\n )\n\n# save the dummy dataset to be imported in SAS\n# write.csv(df, file = \"df_dummy_negbin.csv\")\n```\n:::\n\n\n## Negative binomial regression\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Conclusion for negative binomial regression\n\nExact match (at 0.001 level) can be obtained using `glm.nb` in R vs `PROC GENMOD` procedure in SAS, for parameters and lsmeans estimates, confidence intervals and p-values after manually adjusting the estimated variance-covariance matrix in R. For the dispersion parameter the MLEs also match, however discrepancies in the confidence intervals are observed.\n:::\n\n### Negative binomial regression in SAS\n\nAfter importing the dummy dataset we can run the negative binomial regression in SAS using \\`PROC GENMOD. We estimate the model parameters and lsmeans for the treatment arms using both the default and OM weights.\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc genmod data=df;\n\tclass GRPC (ref='Plb') X2 (ref='A');\n\tmodel y = GRPC x1 x2 / dist=negbin link=log offset=logtime;\n\tlsmeans GRPC /cl;\n\tlsmeans GRPC /cl OM;\nrun;\n```\n:::\n\n\nBelow is a screenshot of output tables summarizing coefficient estimates and lsmeans.\n\n![](../images/negbin/sas_negbin_estimates.jpg){fig-align=\"left\"}\n\n## Negative binomial regression in R\n\nLets now try to reproduce the results in R using `MASS::glm.nb`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit <- glm.nb(y ~ grpc + x1 + x2 + offset(logtime), data = df, x = TRUE)\n\n# model coefficients summary\nsummary(fit)$coefficients\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error z value Pr(>|z|)\n(Intercept) 0.72652157 0.3507054 2.0716007 0.03830269\ngrpcTrt -0.61401736 0.3414815 -1.7980982 0.07216145\nx1 0.25663164 0.1890455 1.3575129 0.17461831\nx2B -0.37406342 0.5069487 -0.7378723 0.46059203\nx2C -0.04999267 0.3916689 -0.1276401 0.89843376\n```\n\n\n:::\n:::\n\n\nWe can see that while the estimates are exactly matching those in SAS, the standard errors are slightly smaller. This is a result of the difference in covariance estimation mentioned above. To obtain exactly the same results as in SAS we need to re-estimate the covariance matrix using the `glm_nb_cov` function we defined earlier. Note that to use this function with the fitted results we needed to specify `x = TRUE` in the `glm.nb` function so that the design matrix is available.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsigma_hat <- glm_nb_cov(fit)\n\n## recalculate confidence intervals, and p-values\ncoef_est <- coef(fit)\ncoef_se <- sqrt(diag(sigma_hat)[1:5])\n\ncoef_lower <- coef_est - qnorm(0.975) * coef_se\ncoef_upper <- coef_est + qnorm(0.975) * coef_se\n\nzstat <- coef_est / coef_se\npval <- 2 * (1 - pnorm(abs(zstat)))\n\nnew_summary <- cbind(coef_est, coef_se, coef_lower, coef_upper, zstat, pval)\n\ncolnames(new_summary) <- c(\n \"Estimate\",\n \"Std. Error\",\n \"CI_lower\",\n \"CI_upper\",\n \"z value\",\n \"Pr(>|z|)\"\n)\nrownames(new_summary) <- rownames(summary(fit)$coefficients)\nnew_summary\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error CI_lower CI_upper z value Pr(>|z|)\n(Intercept) 0.72652157 0.3517882 0.03702942 1.41601371 2.0652246 0.03890176\ngrpcTrt -0.61401736 0.3479606 -1.29600763 0.06797291 -1.7646174 0.07762809\nx1 0.25663164 0.2066499 -0.14839474 0.66165803 1.2418667 0.21428575\nx2B -0.37406342 0.5073695 -1.36848936 0.62036253 -0.7372604 0.46096404\nx2C -0.04999267 0.4013463 -0.83661692 0.73663158 -0.1245624 0.90086997\n```\n\n\n:::\n:::\n\n\nNow the estimates, standard errors, 95% confidence interval limits and p-values are exactly matching those in SAS up to the 4th digit. We can also provide an estimate and CI for the dispersion parameter:\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# estimate and 95%-CI for k = 1/theta\ntheta_est <- fit$theta\ntheta_se <- sqrt(sigma_hat[6, 6])\n\ntheta_est_ci <- c(\n theta_est,\n theta_est - qnorm(0.975) * theta_se,\n theta_est + qnorm(0.975) * theta_se\n)\n1 / theta_est_ci[c(1, 3, 2)]\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 2.370525 1.672211 4.070264\n```\n\n\n:::\n:::\n\n\nWe see that while the point estimate is the same as in SAS, the CI for the dispersion does not match, most likely due to the different parameterizations used by SAS and R.\n\nFinally we can replicate the estimation of lsmeans in SAS via the emmeans package. Note that we need to supply the re-estimated covariance matrix, but only provide the rows and columns for the model coefficients without the dispersion parameter as emmeans does not need the latter.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# lsmeans with weights = equal, equivalent to SAS default\nlsmean1 <- emmeans(\n fit,\n ~grpc,\n data = df,\n vcov. = sigma_hat[1:5, 1:5],\n weights = \"equal\",\n offset = 0\n)\nlsmean1\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n grpc emmean SE df asymp.LCL asymp.UCL\n Plb 0.60837 0.245 Inf 0.128 1.088\n Trt -0.00565 0.268 Inf -0.531 0.519\n\nResults are averaged over the levels of: x2 \nResults are given on the log (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n\n```{.r .cell-code}\n# lsmeans with weights = proportional, equivalent to SAS OM option\nlsmean2 <- emmeans(\n fit,\n ~grpc,\n data = df,\n vcov. = sigma_hat[1:5, 1:5],\n weights = \"proportional\",\n offset = 0\n)\nlsmean2\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n grpc emmean SE df asymp.LCL asymp.UCL\n Plb 0.6527 0.237 Inf 0.188 1.117\n Trt 0.0386 0.250 Inf -0.451 0.528\n\nResults are averaged over the levels of: x2 \nResults are given on the log (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n:::\n\n\nEstimates and CIs are exactly matching those in SAS for both of the options. Finally we can also obtain the z statistic and corresponding p-values:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ntest(lsmean1)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n grpc emmean SE df z.ratio p.value\n Plb 0.60837 0.245 Inf 2.484 0.0130\n Trt -0.00565 0.268 Inf -0.021 0.9832\n\nResults are averaged over the levels of: x2 \nResults are given on the log (not the response) scale. \n```\n\n\n:::\n\n```{.r .cell-code}\ntest(lsmean2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n grpc emmean SE df z.ratio p.value\n Plb 0.6527 0.237 Inf 2.753 0.0059\n Trt 0.0386 0.250 Inf 0.155 0.8770\n\nResults are averaged over the levels of: x2 \nResults are given on the log (not the response) scale. \n```\n\n\n:::\n:::\n\n\nAnd we see that these are also identical to the SAS results.\n\n## Discussion\n\nAs shown above it is generally possible to obtain exactly matching results in SAS and R for negative binomial regression. Most important to ensure matching is the manual estimation of the covariance matrix in R, as otherwise standard errors will only asymptotically match those in SAS.\n\nAs shown above lsmeans-type estimates can also be exactly reproduced using the emmeans package in R if options are correctly specified.\n\nFor the dispersion parameter an exact match in the MLE is possible, however CIs were not matching in our example. Most likely this is due to the different parameterizations used in SAS and R, since the variance for the dispersion parameters can not be transformed exactly between the two parameterizations. As generally the dispersion parameter should be of lesser interest and the other parameter estimates are not affected by this, this may however not be an issue in most applications.\n\nEven though results matched in the numerical example we have also highlighted that there are differences in the implementations, in particular when it comes to maximum likelihood optimization methods and convergence criteria. It is possible that this may lead to different estimates for data where the MLE is not easy to find and the methods may disagree on convergence or the optima of the likelihood. In addition, the different parameterizations may lead to different results in scenarios, where there is only very little overdispersion, since in those cases the dispersion parameter will go towards zero in SAS and towards infinity in R.\n\nAs a final point it should be kept in mind when comparing SAS and R results, that the two apply different rules for rounding. R rounds to the even digit (i.e. both 1.5 and 2.5 round to 2), while SAS uses \"conventional\" rounding rules (i.e 1.5 is rounded to 2 and 2.5 to 3). This can also occasionally lead to differences in results and may need to be addressed by using a custom rounding function in R, that uses SAS rounding rules. An example of such a function is provided in one of the references given below.\n\n## References\n\n- [SAS PROC GENMOD documentation](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_genmod_toc.htm).\n- [R glm.nb documentation](https://www.rdocumentation.org/packages/MASS/versions/7.3-60.0.1/topics/glm.nb).\n- [CrossValidated discussion on covariance estimation](https://stats.stackexchange.com/questions/221648/negative-binomial-regression-in-r-allowing-for-correlation-between-dispersion) (`glm.nb.cov` function is provided in the answer by Jonathan Bartlett).\n- [Discussion of general differences in SAS and R including rounding](https://www.lexjansen.com/phuse-us/2020/ct/CT05.pdf)", - "supporting": [], + "markdown": "---\ntitle: \"R vs SAS: Negative Binomial Regression\"\nformat: html\ntoc: true\nexecute:\n echo: true\nkeep-hidden: true\n---\n\n# Summary\n\n## Goal\n\nComparison of implementations and results between SAS vs R for negative binomial regression for count data.\n\n## Scope\n\n::::::: columns\n:::: {.column width=\"45%\"}\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n## Methodologies\n\n- Negative binomial regression\\\n:::\n::::\n\n:::: {.column width=\"55%\"}\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n## Technical implementations\n\n- SAS: `PROC GENMOD` with option: `DIST = NB` or `DIST = NEGBIN`\n- R: `MASS::glm.nb`\\\n:::\n::::\n:::::::\n\n## Findings\n\nBelow are summary of findings from a numerical comparison using dummy data, where possible we specify the same algorithm in R and SAS (see section [Numerical Comparisons](#sec-num-comp) for details).\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Negative binomial regression\n\nExact match (at 0.001 level) can be obtained using `glm.nb` in R vs `PROC GENMOD` procedure in SAS, for parameter and lsmeans estimates, confidence intervals and p-values after manually adjusting the estimated variance-covariance matrix in R. For the dispersion parameter the MLEs also match, however discrepancies in the confidence intervals are observed.\n:::\n\nIn the following sections the implementations will be compared in tabular fashion followed by a numerical comparison using dummy data.\n\n# Comparison of key features of SAS and R implementations\n\nThe following set of tables compare the two implementations and which options need to be adjusted in R to obtain matching results if necessary. The following aspects are compared:\n\n1. Parameterization of the negative binomial distribution\n2. Likelihood optimization algorithm\n3. Estimation of the variance-covariance matrix\n4. Convergence criteria\n5. Confidence interval (CI) estimation method\n6. Hypothesis tests for regression coefficients\n7. Estimation of marginal means\n\n## How to read the tables:\n\nLet's walk through the conclusions for [Table 1](#tbl-1):\n\n- SAS and R use different parameterizations of the negative binomial\n- SAS and R use different likelihood optimization algorithms\n- There are differences in the estimation of the variance-covariance matrix, in particular the covariance between dispersion/scale parameter and model coefficients. It is however possible to obtain the SAS variance-covariance matrix in R.\n- Convergence criteria are not generally identical in SAS and R.\n- CI estimation methods are by default not identical but by using alternative confint function in R SAS method can be reproduced\\\n- Methods for hypothesis testing for model coefficients are equivalent in SAS and R.\n- Least-square or marginal means are not directly available in R but equivalent estimation as in SAS is possible with additional packages\n\n| Attribute | SAS
`PROC GENMOD` | R
`MASS::glm.nb` | Note |\n|:----------------:|:----------------:|:----------------:|:-----------------|\n| Negative binomial parameterization | Variance of the negative binomial is given by $\\mu + k\\mu^2$ and the dispersion parameter `k` is estimated. Overdispersion increases as `k` goes to infinity. | Variance of the negative binomial is given by $\\mu + \\frac{\\mu^2}{\\theta}$ and the scale parameter `theta` is estimated. Overdispersion increases as `theta` goes to zero. | $k=\\frac{1}{\\theta}$ |\n| Likelihood optimization algorithm | Ridge-stabilized Newton-Raphson algorithm | Iteratively reweighted least squares (IWLS) | It seems SAS performs simultaneous optimization on all parameters (i.e. including dispersion). R uses an alternating process, where glm coefficients are fitted given a fixed theta and then theta is estimated given fixed coefficients until convergence. |\n| Estimation of variance-covariance matrix | Observed (rather than expected) fisher information is used for calculation of standard errors of coefficients, which allows for non-zero covariance between coefficients and dispersion parameter. | Expected fisher information is used for calculation of standard errors of coefficients, so covariance between coefficients and dispersion parameter is zero (which is asymptotically correct). However identical vcov matrix as in SAS can be obtained \"post-hoc\". | As shown in the numerical example below in R the variance-covariance matrix corresponding to the `PROC GENMOD` estimation can be obtained based on the outputs from `MASS::glm.nb` with the `glm.nb.cov` function. The \"correct\" standard errors, confidence intervals and p-values can then be manually calculated based on the new covariance matrix. |\n| Convergence criteria | The iterations are considered to have converged when the maximum change in the parameter estimates between iteration steps is less than the value specified in `CONVERGE` option (default: `CONVERGENCE = 1E-4`) | Based on relative difference between deviance, specified through `epsilon` argument in `glm.control` (default: `epsilon = 1e-8`). | `PROC GENMOD` also checks for relative Hessian convergence and throws a warning if this is larger than the value provided in the `CONVH` option (default: `CONVH = 1E-4` ). |\n| Confidence interval (CI) estimation method | By default asymptotic Wald CIs are estimated. Profile likelihood CI is estimated if option `LRCI` is provided. | `confint` function will estimate profile likelihood CIs, Wald CIs can be obtained by using `confint.default` | Note that by default confidence intervals can differ even if same method is used if vcov matrix in R is not adjusted as explained above. |\n| Hypothesis tests for regression coefficients | Asymptotic Wald test | Asymptotic Wald test | `PROC GENMOD` reports Wald Chi-square statistic, while `MASS::glm.nb` reports the Z statistic, however the p-values are equivalent. Note that by default test results will differ if vcov matrix in R is not adjusted as explained above. |\n| Estimation of least-square/marginal means | Calculation through lsmeans statement assumes that for classification effects the groups are balanced. `OM` option can be provided to obtain lsmeans that are using the observed proportions in the data | Not implemented as part of `MASS::glm.nb` but can be obtained using `emmeans` package. | In R marginal means can be obtained using the `emmeans::emmeans` function, setting argument `weights = \"equal\"` corresponds to the default option in SAS, while `weights = \"proportional\"` gives the means proportional to observed data |\n\n: Negative binomial regression in SAS vs R {#tbl-1}\n\n# Numerical Comparison {#sec-num-comp}\n\n- SAS `PROC GENMOD` procedure\n\n- R `MASS::glm.nb`\n\nA dummy dataset is first generated, and negative binomial regression is applied to the dummy dataset for demonstration purposes. Every effort is made to ensure that the R code employs estimation methods/ optimization algorithms/ other components that closely match (as much as possible) those used in the SAS code. This is done to facilitate a comparison of estimates, 95% confidence intervals (CIs), and p-values between the two implementations.\n\n## Prerequisites: R packages\n\nIn order to run these analyses we need to load a few packages.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(MASS)\nlibrary(dplyr)\n```\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\n\nAttaching package: 'dplyr'\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\nThe following object is masked from 'package:MASS':\n\n select\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\nThe following objects are masked from 'package:stats':\n\n filter, lag\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\nThe following objects are masked from 'package:base':\n\n intersect, setdiff, setequal, union\n```\n\n\n:::\n\n```{.r .cell-code}\nlibrary(emmeans)\n```\n\n::: {.cell-output .cell-output-stderr .hidden}\n\n```\nWelcome to emmeans.\nCaution: You lose important information if you filter this package's results.\nSee '? untidy'\n```\n\n\n:::\n:::\n\n\nWe also define the `glm_nb_cov` function to obtain the SAS variance-covariance matrix in R from [here](https://stats.stackexchange.com/questions/221648/negative-binomial-regression-in-r-allowing-for-correlation-between-dispersion).\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## Helper function to compute the variance from negative binomial regression\n## This matches with variance estimated from SAS\nglm_nb_cov <- function(mod) {\n # given a model fitted by glm.nb in MASS, this function returns a variance covariance matrix for the\n # regression coefficients and dispersion parameter, without assuming independence between these\n # note that the model must have been fitted with x=TRUE argument so that design matrix is available\n\n # formulae based on p23-p24 of\n # http://pointer.esalq.usp.br/departamentos/lce/arquivos/aulas/2011/LCE5868/OverdispersionBook.pdf\n # and http://www.math.mcgill.ca/~dstephens/523/Papers/Lawless-1987-CJS.pdf\n\n # lintr: off\n # please rm -- variable not used!\n # k <- mod$theta\n # lintr: on\n # p is number of regression coefficients\n p <- dim(vcov(mod))[1]\n\n # construct observed information matrix\n obsInfo <- array(0, dim = c(p + 1, p + 1))\n\n # first calculate top left part for regression coefficients\n for (i in 1:p) {\n for (j in 1:p) {\n obsInfo[i, j] <- sum(\n (1 + mod$y / mod$theta) *\n mod$fitted.values *\n mod$x[, i] *\n mod$x[, j] /\n (1 + mod$fitted.values / mod$theta)^2\n )\n }\n }\n\n # information for dispersion parameter\n obsInfo[(p + 1), (p + 1)] <- -sum(\n trigamma(mod$theta + mod$y) -\n trigamma(mod$theta) -\n 1 / (mod$fitted.values + mod$theta) +\n (mod$theta + mod$y) / (mod$theta + mod$fitted.values)^2 -\n 1 / (mod$fitted.values + mod$theta) +\n 1 / mod$theta\n )\n\n # covariance between regression coefficients and dispersion\n for (i in 1:p) {\n obsInfo[(p + 1), i] <- -sum(\n ((mod$y - mod$fitted.values) *\n mod$fitted.values /\n ((mod$theta + mod$fitted.values)^2)) *\n mod$x[, i]\n )\n obsInfo[i, (p + 1)] <- obsInfo[(p + 1), i]\n }\n\n # return variance covariance matrix\n solve(obsInfo, tol = 1e-20)\n}\n```\n:::\n\n\n## Dummy data\n\nA dummy dataset is simulated, including\n\n- 100 subjects;\n- $grp$: a dummy variable with 1:1 subject assignment to treatment ($grp = 1$) vs placebo ($grp = 0$); note, variable $grpc$ is a character version of $grp$, which takes the value of \"Trt\" or \"Plb\".\n- $x1$: a continuous variable which follows a normal distribution of mean of 0 and sd of 1;\n- $x2$: a categorical variable which take the value of \"A\" or \"B\" or \"C\" with a probability of 0.3, 0.2, 0.5, respectively.\n- $logtime$: An offset for the calculation of rates (e.g time in years) on the log-scale\n- $y$: a negative binomial outcome giving the event counts;\n\nThe dummy dataset is saved as a csv file, and then the csv file is read into SAS.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nN = 100\n\n# set seed for replication\nset.seed(123)\n\n# Treatment Group; 1:1 ratio\ngrp <- rep(c(0, 1), each = N / 2)\n\n# Covariates (one continuous; one categorical)\nx1 <- rnorm(N)\nx2 <- factor(sample(LETTERS[1:3], N, replace = TRUE, prob = c(0.3, 0.2, 0.5)))\n\n# Offset\nlogtime <- log(runif(N, 1, 2))\n\n# Model parameter assumption\nbeta0 = 0.6\nbetaTrt = -0.5\nbeta1 = 0.25\nbeta2 = c(-0.1, 0.2)\ntheta = 1 / 2\n\n\n# Dummy dataset\ndf <- data.frame(grp, x1, x2, logtime) %>%\n mutate(\n log_rate = case_when(\n x2 == \"A\" ~ beta0 + betaTrt * grp + beta1 * x1 + logtime,\n x2 == \"B\" ~ beta0 + betaTrt * grp + beta1 * x1 + beta2[1] + logtime,\n x2 == \"C\" ~ beta0 + betaTrt * grp + beta1 * x1 + beta2[2] + logtime\n ),\n y = rnegbin(N, mu = exp(log_rate), theta = theta),\n grpc = factor(case_when(grp == 0 ~ \"Plb\", grp == 1 ~ \"Trt\"))\n )\n\n# save the dummy dataset to be imported in SAS\n# write.csv(df, file = \"df_dummy_negbin.csv\")\n```\n:::\n\n\n## Negative binomial regression\n\n::: {.callout-note appearance=\"minimal\" collapse=\"false\"}\n### Conclusion for negative binomial regression\n\nExact match (at 0.001 level) can be obtained using `glm.nb` in R vs `PROC GENMOD` procedure in SAS, for parameters and lsmeans estimates, confidence intervals and p-values after manually adjusting the estimated variance-covariance matrix in R. For the dispersion parameter the MLEs also match, however discrepancies in the confidence intervals are observed.\n:::\n\n### Negative binomial regression in SAS\n\nAfter importing the dummy dataset we can run the negative binomial regression in SAS using \\`PROC GENMOD. We estimate the model parameters and lsmeans for the treatment arms using both the default and OM weights.\n\n```sas\nproc genmod data=df;\n\tclass GRPC (ref='Plb') X2 (ref='A');\n\tmodel y = GRPC x1 x2 / dist=negbin link=log offset=logtime;\n\tlsmeans GRPC /cl;\n\tlsmeans GRPC /cl OM;\nrun;\n```\n\nBelow is a screenshot of output tables summarizing coefficient estimates and lsmeans.\n\n![](../images/negbin/sas_negbin_estimates.jpg){fig-align=\"left\"}\n\n## Negative binomial regression in R\n\nLets now try to reproduce the results in R using `MASS::glm.nb`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit <- glm.nb(y ~ grpc + x1 + x2 + offset(logtime), data = df, x = TRUE)\n\n# model coefficients summary\nsummary(fit)$coefficients\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error z value Pr(>|z|)\n(Intercept) 0.72652157 0.3507054 2.0716007 0.03830269\ngrpcTrt -0.61401736 0.3414815 -1.7980982 0.07216145\nx1 0.25663164 0.1890455 1.3575129 0.17461831\nx2B -0.37406342 0.5069487 -0.7378723 0.46059203\nx2C -0.04999267 0.3916689 -0.1276401 0.89843376\n```\n\n\n:::\n:::\n\n\nWe can see that while the estimates are exactly matching those in SAS, the standard errors are slightly smaller. This is a result of the difference in covariance estimation mentioned above. To obtain exactly the same results as in SAS we need to re-estimate the covariance matrix using the `glm_nb_cov` function we defined earlier. Note that to use this function with the fitted results we needed to specify `x = TRUE` in the `glm.nb` function so that the design matrix is available.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsigma_hat <- glm_nb_cov(fit)\n\n## recalculate confidence intervals, and p-values\ncoef_est <- coef(fit)\ncoef_se <- sqrt(diag(sigma_hat)[1:5])\n\ncoef_lower <- coef_est - qnorm(0.975) * coef_se\ncoef_upper <- coef_est + qnorm(0.975) * coef_se\n\nzstat <- coef_est / coef_se\npval <- 2 * (1 - pnorm(abs(zstat)))\n\nnew_summary <- cbind(coef_est, coef_se, coef_lower, coef_upper, zstat, pval)\n\ncolnames(new_summary) <- c(\n \"Estimate\",\n \"Std. Error\",\n \"CI_lower\",\n \"CI_upper\",\n \"z value\",\n \"Pr(>|z|)\"\n)\nrownames(new_summary) <- rownames(summary(fit)$coefficients)\nnew_summary\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error CI_lower CI_upper z value Pr(>|z|)\n(Intercept) 0.72652157 0.3517882 0.03702942 1.41601371 2.0652246 0.03890176\ngrpcTrt -0.61401736 0.3479606 -1.29600763 0.06797291 -1.7646174 0.07762809\nx1 0.25663164 0.2066499 -0.14839474 0.66165803 1.2418667 0.21428575\nx2B -0.37406342 0.5073695 -1.36848936 0.62036253 -0.7372604 0.46096404\nx2C -0.04999267 0.4013463 -0.83661692 0.73663158 -0.1245624 0.90086997\n```\n\n\n:::\n:::\n\n\nNow the estimates, standard errors, 95% confidence interval limits and p-values are exactly matching those in SAS up to the 4th digit. We can also provide an estimate and CI for the dispersion parameter:\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# estimate and 95%-CI for k = 1/theta\ntheta_est <- fit$theta\ntheta_se <- sqrt(sigma_hat[6, 6])\n\ntheta_est_ci <- c(\n theta_est,\n theta_est - qnorm(0.975) * theta_se,\n theta_est + qnorm(0.975) * theta_se\n)\n1 / theta_est_ci[c(1, 3, 2)]\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 2.370525 1.672211 4.070264\n```\n\n\n:::\n:::\n\n\nWe see that while the point estimate is the same as in SAS, the CI for the dispersion does not match, most likely due to the different parameterizations used by SAS and R.\n\nFinally we can replicate the estimation of lsmeans in SAS via the emmeans package. Note that we need to supply the re-estimated covariance matrix, but only provide the rows and columns for the model coefficients without the dispersion parameter as emmeans does not need the latter.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# lsmeans with weights = equal, equivalent to SAS default\nlsmean1 <- emmeans(\n fit,\n ~grpc,\n data = df,\n vcov. = sigma_hat[1:5, 1:5],\n weights = \"equal\",\n offset = 0\n)\nlsmean1\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n grpc emmean SE df asymp.LCL asymp.UCL\n Plb 0.60837 0.245 Inf 0.128 1.088\n Trt -0.00565 0.268 Inf -0.531 0.519\n\nResults are averaged over the levels of: x2 \nResults are given on the log (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n\n```{.r .cell-code}\n# lsmeans with weights = proportional, equivalent to SAS OM option\nlsmean2 <- emmeans(\n fit,\n ~grpc,\n data = df,\n vcov. = sigma_hat[1:5, 1:5],\n weights = \"proportional\",\n offset = 0\n)\nlsmean2\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n grpc emmean SE df asymp.LCL asymp.UCL\n Plb 0.6527 0.237 Inf 0.188 1.117\n Trt 0.0386 0.250 Inf -0.451 0.528\n\nResults are averaged over the levels of: x2 \nResults are given on the log (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n:::\n\n\nEstimates and CIs are exactly matching those in SAS for both of the options. Finally we can also obtain the z statistic and corresponding p-values:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ntest(lsmean1)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n grpc emmean SE df z.ratio p.value\n Plb 0.60837 0.245 Inf 2.484 0.0130\n Trt -0.00565 0.268 Inf -0.021 0.9832\n\nResults are averaged over the levels of: x2 \nResults are given on the log (not the response) scale. \n```\n\n\n:::\n\n```{.r .cell-code}\ntest(lsmean2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n grpc emmean SE df z.ratio p.value\n Plb 0.6527 0.237 Inf 2.753 0.0059\n Trt 0.0386 0.250 Inf 0.155 0.8770\n\nResults are averaged over the levels of: x2 \nResults are given on the log (not the response) scale. \n```\n\n\n:::\n:::\n\n\nAnd we see that these are also identical to the SAS results.\n\n## Discussion\n\nAs shown above it is generally possible to obtain exactly matching results in SAS and R for negative binomial regression. Most important to ensure matching is the manual estimation of the covariance matrix in R, as otherwise standard errors will only asymptotically match those in SAS.\n\nAs shown above lsmeans-type estimates can also be exactly reproduced using the emmeans package in R if options are correctly specified.\n\nFor the dispersion parameter an exact match in the MLE is possible, however CIs were not matching in our example. Most likely this is due to the different parameterizations used in SAS and R, since the variance for the dispersion parameters can not be transformed exactly between the two parameterizations. As generally the dispersion parameter should be of lesser interest and the other parameter estimates are not affected by this, this may however not be an issue in most applications.\n\nEven though results matched in the numerical example we have also highlighted that there are differences in the implementations, in particular when it comes to maximum likelihood optimization methods and convergence criteria. It is possible that this may lead to different estimates for data where the MLE is not easy to find and the methods may disagree on convergence or the optima of the likelihood. In addition, the different parameterizations may lead to different results in scenarios, where there is only very little overdispersion, since in those cases the dispersion parameter will go towards zero in SAS and towards infinity in R.\n\nAs a final point it should be kept in mind when comparing SAS and R results, that the two apply different rules for rounding. R rounds to the even digit (i.e. both 1.5 and 2.5 round to 2), while SAS uses \"conventional\" rounding rules (i.e 1.5 is rounded to 2 and 2.5 to 3). This can also occasionally lead to differences in results and may need to be addressed by using a custom rounding function in R, that uses SAS rounding rules. An example of such a function is provided in one of the references given below.\n\n## References\n\n- [SAS PROC GENMOD documentation](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_genmod_toc.htm).\n- [R glm.nb documentation](https://www.rdocumentation.org/packages/MASS/versions/7.3-60.0.1/topics/glm.nb).\n- [CrossValidated discussion on covariance estimation](https://stats.stackexchange.com/questions/221648/negative-binomial-regression-in-r-allowing-for-correlation-between-dispersion) (`glm.nb.cov` function is provided in the answer by Jonathan Bartlett).\n- [Discussion of general differences in SAS and R including rounding](https://www.lexjansen.com/phuse-us/2020/ct/CT05.pdf)", + "supporting": [ + "r-sas_negbin_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_psmatch/execute-results/html.json b/_freeze/Comp/r-sas_psmatch/execute-results/html.json index ffc857b7b..9632e9c46 100644 --- a/_freeze/Comp/r-sas_psmatch/execute-results/html.json +++ b/_freeze/Comp/r-sas_psmatch/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "5227b7241bb74da31eabeeb35caa1685", + "hash": "dd4a107f646924baf465853ee611e57e", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Propensity Score Matching\"\nexecute: \n eval: false\n---\n\n# Introduction\n\nPropensity score (PS) matching is a statistical technique widely employed in Real World Evidence (RWE) studies to address confounding bias and facilitate the estimation of causal treatment effects. By estimating the probability of treatment assignment based on observed covariates, PS matching aims to create comparable treatment and control groups. While both SAS and R provide methods to do PS matching, the syntax and available options differ considerably, potentially leading to variations in analytical outcomes. The PS matching process generally involves several key stages: first, the estimation of propensity scores using a regression model; second, the specification of a region of common support to ensure overlap between treatment and control groups; and third, the matching of treated and control subjects based on their calculated propensity scores.\n\n# Differences\n\nGiven the extensive number of parameters and arguments available in both `PROC PSMATCH` and `matchit()`, we will focus only on the observed differences.\n\n## Options\n\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Option | PROC PSMATCH | matchit |\n+=======================================================+===========================================================================================================================+===========================================================================================================================================================================================================================+\n| Distance | *PS, LPS*; only for matching: *mahalanobis*, *euclidean* | *PS*, *euclidean*, *scaled_euclidean*, *mahalanobis*, *robust_mahalanobis* |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| PS methods | logistic regression | glm, gam, gbm, lasso, ridge, elasticnet with different links, partitioning tree, RF, single-hidden-layer neural network, covariate balancing propensity score (CBPS) algorithm, Bayesian additive regression trees (BART) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Region | Always used; *allobs*, *treated* or *cs* (common support), with allowed extension | *none*, *treated*, *control*, *both* (common support) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Caliper | Applies only to the distance metric, such that distance\\<=caliper | Can be applied to both covariates and distance metric. If a positive value is supplied, it functions as in SAS; If a negative value is supplied, it imposes the condition distance \\> abs(caliper). |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| PS std.dev formula when caliper applied as multiplier | sqrt((sd(PS~trt~)^2^+sd(P~control~)^2^)/2) | sd(PS~all~) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Matching methods | *greedy* (greedy nn), *full*, *optimal*, *replace*, *varratio* | *nearest* (greedy nn), *full*, *optimal*, *quick*, *genetic*, *cem*, *exact*, *cardinality*, *subclass* |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Method=optimal options | Only K should be supplied | In addition to k (number of matches), the tol option can also be specified, and its value can significantly impact the matching results. |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Method=full options | n max controls, n max treated, mean n treated, n controls, pct controls | n min controls, n max controls, mean n controls, fraction to omit, tol, solver |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Mahalanobis distance | PS are always calculated to determine the region; only numeric covariates are allowed in MAHVARS option | mahvars accepts any var type; region calculation is ommited when distance=*mahalanobis*, but can be performed when distance=*glm* and mahvars is supplied with a formula |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Covariance matrix for mahalanobis distance | Computed from trt obs and control obs | Is pooled within-group, computed from trt mean-centered covariates of the full sample |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Replace | Could be specified only as method=*replace*; in the output, match IDs are shared among all subjects within a matched set. | Could be specified as an argument (e.g. replace=*TRUE*). In the output, each treated subject receives up to K matched controls |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Estimands | ATT, ATE | ATT, ATE, ATC |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Reestimate | Not available | Could be specified as an argument (e.g. reestimate=*TRUE*) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Exact, anti-exact | Only exact matching is available and can only be performed on variables listed in the CLASS statement. | Both exact and anti-exact matching are available, with no restrictions on the variable types. |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Normalization | Not available | Could be specified as an argument (e.g. normalize=*TRUE*) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n\n## Output\n\nThe way SAS and R show the results of matching is different. The output from SAS matching is presented as a dataset where each treated subject (or all subjects) is given a row, and a match ID column containing a unique pair (or group) identifier is included:\n\n| | TRTP | \\_MatchID |\n|-----|---------|-----------|\n| 1 | trt | 1 |\n| 2 | trt | 2 |\n| 3 | control | 1 |\n| 4 | control | 2 |\n\nIn contrast, the `matchit()` function in R returns a matrix, where each treated unit's identifier is associated with the identifiers of its k matched control units:\n\n| | \\[,1\\] |\n|-----|--------|\n| 1 | \"3\" |\n| 2 | \"4\" |\n\n### Statistics\n\nIn SAS, descriptive statistics for assessing balance are primarily generated through the `ASSESS` statement within `PROC PSMATCH`. Conversely, in R, balance diagnostics could be obtained by applying the `summary()` function to the output object returned by the `matchit()`. The following table summarizes the balance statistics available in SAS and R:\n\n| Stat | All | Region | Matched |\n|:-----------------------|:-------:|:------:|:-------:|\n| N | SAS & R | SAS | SAS & R |\n| PS mean | SAS & R | SAS | SAS & R |\n| PS std | SAS | SAS | SAS |\n| PS min | SAS | SAS | SAS |\n| PS max | SAS | SAS | SAS |\n| Vars mean | SAS & R | SAS | SAS & R |\n| Vars std | SAS | SAS | SAS |\n| Vars min | SAS | SAS | SAS |\n| Vars max | SAS | SAS | SAS |\n| PS mean diff | SAS | SAS | SAS |\n| PS SMD | SAS & R | SAS | SAS & R |\n| PS perc. red. | SAS | SAS | SAS |\n| PS var ratio | SAS & R | SAS | SAS & R |\n| Vars mean diff | SAS | SAS | SAS |\n| Vars SMD | SAS & R | SAS | SAS & R |\n| Vars perc. red. | SAS | SAS | SAS |\n| Vars var ratio | SAS & R | SAS | SAS & R |\n| PS eCDF min | R | \\- | R |\n| PS eCDF max | R | \\- | R |\n| Vars eCDF min | R | \\- | R |\n| Vars eCDF max | R | \\- | R |\n| PS std pair distance | R | \\- | R |\n| Vars std pair distance | R | \\- | R |\n\n### Figures\n\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| Plot type | R | SAS |\n+============================+========================================================================================================================================================+======================================================================+\n| Love plot | - [plot()]{.underline}: - | **Displayed for:** all/region/matched/weighted matched, trt/control. |\n| | | |\n| | - [cobalt:]{.underline} many different settings. | **Includes** PS, all numeric and binary variables. |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| General distribution plots | - [plot():]{.underline} **Displayed for:** all/matched. **Includes** all variables; numeric - distribution plots, character and factor - histograms. | **Displayed for:** all/region/matched/weighted matched, trt/control. |\n| | | |\n| | - [cobalt]{.underline} : Highly customizable plots. | **Includes** PS and all variables. |\n| | | |\n| | | For PS and numeric - boxplots, character - barplots. |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| eCDF plots | [plot():]{.underline} | **Displayed for:** all/region/matched/weighted matched, trt/control. |\n| | | |\n| | **Displayed for:** all/matched. | **Includes** PS and all numeric variables. |\n| | | |\n| | **Includes** all variables. | |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| eQQ plots | [plot():]{.underline} | \\- |\n| | | |\n| | **Displayed for:** all/matched. | |\n| | | |\n| | **Includes** all variables. | |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| Cloud plots | [plot():]{.underline} | **Displayed for:** all/region/matched/weighted matched, trt/control. |\n| | | |\n| | **Displayed for:** all/matched, trt/control. | **Includes** PS and all numeric variables. |\n| | | |\n| | **Includes** PS. | Presented as 2 separate clouds per variable. |\n| | | |\n| | Presented as 4 separate clouds. | |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| PS histogram | - [plot()]{.underline}: **Displayed for:** all/matched, trt/control. **Includes** PS. | \\- |\n| | | |\n| | - [cobalt]{.underline}: Highly customizable plots. | |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n\n# Dataset\n\nThe dataset used in the example below can be found here: \\[ps_data.csv\\]()\n\n``` default\n trt control Standardized\nCharacteristic (N = 120) (N = 180) Mean Diff.\n\n sex 0.1690\n F 70 (58.3 %) 90 (50.0 %) \n M 50 (41.7 %) 90 (50.0 %)\n\n age 61.5 (17.12) 49.4 (10.55) 0.7057\n\n weight 67.3 ( 7.33) 63.8 ( 9.64) 0.4741\n\n bmi_cat \n underweight 34 (28.3 %) 63 (35.0 %) -0.1479 \n normal 57 (47.5 %) 61 (33.9 %) 0.2726\n overweight 29 (24.2 %) 56 (31.1 %) -0.1622\n```\n\n# Matching Examples\n\n## Greedy Nearest Neighbor 1 to 1 matching with common support region\n\n### SAS\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc psmatch data=data region=cs(extend=0);\n class trtp sex bmi_cat;\n psmodel trtp(Treated=\"trt\")= sex weight age bmi_cat;\n match distance=PS \n method=greedy(k=1 order=descending) \n caliper(MULT=ONE)=0.25;\n output out(obs=match)=ps_res matchid=_MatchID ps=_PScore;\nrun;\n```\n:::\n\n\n### R\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(MatchIt)\n\nps_res <- MatchIt::matchit(\n trtp ~ sex + weight + age + bmi_cat,\n data = data,\n method = \"nearest\",\n distance = \"glm\",\n link = \"logit\",\n discard = \"both\",\n m.order = \"largest\",\n replace = FALSE,\n caliper = 0.25,\n std.caliper = FALSE,\n ratio = 1,\n normalize = FALSE\n)\n```\n:::\n\n\nThe following arguments, when altered in the previous example, can still produce matching results comparable between SAS and R:\n\n- `region` (`discard`)\n\n- `caliper` value\n\n- `order`\n\n- `k` (`ratio`)\n\n- `exact`", - "supporting": [], + "markdown": "---\ntitle: \"Propensity Score Matching\"\nexecute:\n eval: false\n---\n\n# Introduction\n\nPropensity score (PS) matching is a statistical technique widely employed in Real World Evidence (RWE) studies to address confounding bias and facilitate the estimation of causal treatment effects. By estimating the probability of treatment assignment based on observed covariates, PS matching aims to create comparable treatment and control groups. While both SAS and R provide methods to do PS matching, the syntax and available options differ considerably, potentially leading to variations in analytical outcomes. The PS matching process generally involves several key stages: first, the estimation of propensity scores using a regression model; second, the specification of a region of common support to ensure overlap between treatment and control groups; and third, the matching of treated and control subjects based on their calculated propensity scores.\n\n# Differences\n\nGiven the extensive number of parameters and arguments available in both `PROC PSMATCH` and `matchit()`, we will focus only on the observed differences.\n\n## Options\n\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Option | PROC PSMATCH | matchit |\n+=======================================================+===========================================================================================================================+===========================================================================================================================================================================================================================+\n| Distance | *PS, LPS*; only for matching: *mahalanobis*, *euclidean* | *PS*, *euclidean*, *scaled_euclidean*, *mahalanobis*, *robust_mahalanobis* |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| PS methods | logistic regression | glm, gam, gbm, lasso, ridge, elasticnet with different links, partitioning tree, RF, single-hidden-layer neural network, covariate balancing propensity score (CBPS) algorithm, Bayesian additive regression trees (BART) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Region | Always used; *allobs*, *treated* or *cs* (common support), with allowed extension | *none*, *treated*, *control*, *both* (common support) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Caliper | Applies only to the distance metric, such that distance\\<=caliper | Can be applied to both covariates and distance metric. If a positive value is supplied, it functions as in SAS; If a negative value is supplied, it imposes the condition distance \\> abs(caliper). |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| PS std.dev formula when caliper applied as multiplier | sqrt((sd(PS~trt~)^2^+sd(P~control~)^2^)/2) | sd(PS~all~) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Matching methods | *greedy* (greedy nn), *full*, *optimal*, *replace*, *varratio* | *nearest* (greedy nn), *full*, *optimal*, *quick*, *genetic*, *cem*, *exact*, *cardinality*, *subclass* |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Method=optimal options | Only K should be supplied | In addition to k (number of matches), the tol option can also be specified, and its value can significantly impact the matching results. |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Method=full options | n max controls, n max treated, mean n treated, n controls, pct controls | n min controls, n max controls, mean n controls, fraction to omit, tol, solver |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Mahalanobis distance | PS are always calculated to determine the region; only numeric covariates are allowed in MAHVARS option | mahvars accepts any var type; region calculation is ommited when distance=*mahalanobis*, but can be performed when distance=*glm* and mahvars is supplied with a formula |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Covariance matrix for mahalanobis distance | Computed from trt obs and control obs | Is pooled within-group, computed from trt mean-centered covariates of the full sample |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Replace | Could be specified only as method=*replace*; in the output, match IDs are shared among all subjects within a matched set. | Could be specified as an argument (e.g. replace=*TRUE*). In the output, each treated subject receives up to K matched controls |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Estimands | ATT, ATE | ATT, ATE, ATC |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Reestimate | Not available | Could be specified as an argument (e.g. reestimate=*TRUE*) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Exact, anti-exact | Only exact matching is available and can only be performed on variables listed in the CLASS statement. | Both exact and anti-exact matching are available, with no restrictions on the variable types. |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Normalization | Not available | Could be specified as an argument (e.g. normalize=*TRUE*) |\n+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n\n## Output\n\nThe way SAS and R show the results of matching is different. The output from SAS matching is presented as a dataset where each treated subject (or all subjects) is given a row, and a match ID column containing a unique pair (or group) identifier is included:\n\n| | TRTP | \\_MatchID |\n|-----|---------|-----------|\n| 1 | trt | 1 |\n| 2 | trt | 2 |\n| 3 | control | 1 |\n| 4 | control | 2 |\n\nIn contrast, the `matchit()` function in R returns a matrix, where each treated unit's identifier is associated with the identifiers of its k matched control units:\n\n| | \\[,1\\] |\n|-----|--------|\n| 1 | \"3\" |\n| 2 | \"4\" |\n\n### Statistics\n\nIn SAS, descriptive statistics for assessing balance are primarily generated through the `ASSESS` statement within `PROC PSMATCH`. Conversely, in R, balance diagnostics could be obtained by applying the `summary()` function to the output object returned by the `matchit()`. The following table summarizes the balance statistics available in SAS and R:\n\n| Stat | All | Region | Matched |\n|:-----------------------|:-------:|:------:|:-------:|\n| N | SAS & R | SAS | SAS & R |\n| PS mean | SAS & R | SAS | SAS & R |\n| PS std | SAS | SAS | SAS |\n| PS min | SAS | SAS | SAS |\n| PS max | SAS | SAS | SAS |\n| Vars mean | SAS & R | SAS | SAS & R |\n| Vars std | SAS | SAS | SAS |\n| Vars min | SAS | SAS | SAS |\n| Vars max | SAS | SAS | SAS |\n| PS mean diff | SAS | SAS | SAS |\n| PS SMD | SAS & R | SAS | SAS & R |\n| PS perc. red. | SAS | SAS | SAS |\n| PS var ratio | SAS & R | SAS | SAS & R |\n| Vars mean diff | SAS | SAS | SAS |\n| Vars SMD | SAS & R | SAS | SAS & R |\n| Vars perc. red. | SAS | SAS | SAS |\n| Vars var ratio | SAS & R | SAS | SAS & R |\n| PS eCDF min | R | \\- | R |\n| PS eCDF max | R | \\- | R |\n| Vars eCDF min | R | \\- | R |\n| Vars eCDF max | R | \\- | R |\n| PS std pair distance | R | \\- | R |\n| Vars std pair distance | R | \\- | R |\n\n### Figures\n\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| Plot type | R | SAS |\n+============================+========================================================================================================================================================+======================================================================+\n| Love plot | - [plot()]{.underline}: - | **Displayed for:** all/region/matched/weighted matched, trt/control. |\n| | | |\n| | - [cobalt:]{.underline} many different settings. | **Includes** PS, all numeric and binary variables. |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| General distribution plots | - [plot():]{.underline} **Displayed for:** all/matched. **Includes** all variables; numeric - distribution plots, character and factor - histograms. | **Displayed for:** all/region/matched/weighted matched, trt/control. |\n| | | |\n| | - [cobalt]{.underline} : Highly customizable plots. | **Includes** PS and all variables. |\n| | | |\n| | | For PS and numeric - boxplots, character - barplots. |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| eCDF plots | [plot():]{.underline} | **Displayed for:** all/region/matched/weighted matched, trt/control. |\n| | | |\n| | **Displayed for:** all/matched. | **Includes** PS and all numeric variables. |\n| | | |\n| | **Includes** all variables. | |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| eQQ plots | [plot():]{.underline} | \\- |\n| | | |\n| | **Displayed for:** all/matched. | |\n| | | |\n| | **Includes** all variables. | |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| Cloud plots | [plot():]{.underline} | **Displayed for:** all/region/matched/weighted matched, trt/control. |\n| | | |\n| | **Displayed for:** all/matched, trt/control. | **Includes** PS and all numeric variables. |\n| | | |\n| | **Includes** PS. | Presented as 2 separate clouds per variable. |\n| | | |\n| | Presented as 4 separate clouds. | |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n| PS histogram | - [plot()]{.underline}: **Displayed for:** all/matched, trt/control. **Includes** PS. | \\- |\n| | | |\n| | - [cobalt]{.underline}: Highly customizable plots. | |\n+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+\n\n# Dataset\n\nThe dataset used in the example below can be found here: \\[ps_data.csv\\]()\n\n``` default\n trt control Standardized\nCharacteristic (N = 120) (N = 180) Mean Diff.\n\n sex 0.1690\n F 70 (58.3 %) 90 (50.0 %) \n M 50 (41.7 %) 90 (50.0 %)\n\n age 61.5 (17.12) 49.4 (10.55) 0.7057\n\n weight 67.3 ( 7.33) 63.8 ( 9.64) 0.4741\n\n bmi_cat \n underweight 34 (28.3 %) 63 (35.0 %) -0.1479 \n normal 57 (47.5 %) 61 (33.9 %) 0.2726\n overweight 29 (24.2 %) 56 (31.1 %) -0.1622\n```\n\n# Matching Examples\n\n## Greedy Nearest Neighbor 1 to 1 matching with common support region\n\n### SAS\n\n```sas\nproc psmatch data=data region=cs(extend=0);\n class trtp sex bmi_cat;\n psmodel trtp(Treated=\"trt\")= sex weight age bmi_cat;\n match distance=PS \n method=greedy(k=1 order=descending) \n caliper(MULT=ONE)=0.25;\n output out(obs=match)=ps_res matchid=_MatchID ps=_PScore;\nrun;\n```\n\n### R\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(MatchIt)\n\nps_res <- MatchIt::matchit(\n trtp ~ sex + weight + age + bmi_cat,\n data = data,\n method = \"nearest\",\n distance = \"glm\",\n link = \"logit\",\n discard = \"both\",\n m.order = \"largest\",\n replace = FALSE,\n caliper = 0.25,\n std.caliper = FALSE,\n ratio = 1,\n normalize = FALSE\n)\n```\n:::\n\n\nThe following arguments, when altered in the previous example, can still produce matching results comparable between SAS and R:\n\n- `region` (`discard`)\n\n- `caliper` value\n\n- `order`\n\n- `k` (`ratio`)\n\n- `exact`", + "supporting": [ + "r-sas_psmatch_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_survival/execute-results/html.json b/_freeze/Comp/r-sas_survival/execute-results/html.json index 36505787e..81a33d30e 100644 --- a/_freeze/Comp/r-sas_survival/execute-results/html.json +++ b/_freeze/Comp/r-sas_survival/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "9cc45a3506267e1ee4f690fb5d40a8ea", + "hash": "bb8cc0e7c77f547f9b1623131c93e749", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS - Kaplan Meier and Cox-proportion hazards modelling\"\nexecute: \n eval: false\n---\n\n# Comparison of SAS vs R\n\nThe following table shows the options available in SAS and R for Kaplan Meier and Cox Proportional Hazards modelling, the capabilities of each language, and whether or not the results from each language match.\n\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Analysis | Supported in R using {survival} | Supported in SAS | Results Match | Notes |\n+===============================================================================+==============================================+========================================+===============+======================================================================================================================================+\n| Kaplan Meier with confidence intervals using log-log method | Yes (using the option conf.type = \"log-log\") | Yes (Default) | Mostly | 1\\) Survival estimates can disagree when last event is censored and survival estimate does not cross the percentile being estimated. |\n| | | | | |\n| | | | | 2\\) Survival estimates at time X can disagree when the time X is after the last observed censored time |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Kaplan Meier with confidence intervals using log method | Yes (Default) | Yes (using the option conftype=log) | Mostly | As above. |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Cox Proportional Hazards Model using breslow method for ties | Yes (using the option ties=\"breslow\") | Yes (Default) | Yes | |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Cox Proportional Hazards Model using efron method for ties | Yes (Default) | Yes (using the option ties=efron) | Yes | |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Cox Proportional Hazards Model using exact partial likelihood method for ties | Yes (using the option ties=\"exact\") | Yes (using the option ties=\"discrete\") | Yes | The option ties=\"exact\" in SAS uses the exact marginal likelihood which is not available in R |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n\nResults from the examples shown for R [here](https://psiaims.github.io/CAMIS/R/survival.html) and SAS [here](https://psiaims.github.io/CAMIS/SAS/survival.html) were compared below.\n\nComparing the non-stratified model results side-by-side, the CIs for the quartile estimates and landmark estimates are different between R and SAS. HR and CI also have slight differences.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_default.png){fig-align='center' width=75%}\n:::\n:::\n\n\n## Reason 1: Cox Regression Handling of Tied Survival Times\n\nThe default methods for handling ties in a Cox regression model are different which can lead to a different result for the Hazard ratio and associated confidence interval.\n\nR uses \"efron\" by default. SAS uses \"breslow\" by default. Both R and SAS are able to change these default options. By making the changes to the code below, we can force R to use \"breslow\" to match SAS, or SAS to use \"efron\" to match R. When the software use the same methods, then we obtain an identical HR and CI.\n\n- R: change method for ties to use \"breslow\"\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit.cox <- survival::coxph(\n survival::Surv(LENFOLY, FSTAT) ~ AFB,\n ties = \"breslow\",\n data = dat\n)\n```\n:::\n\n\n- SAS: change method for ties to use \"efron\"\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc phreg data=dat;\n class afb;\n model lenfol*fstat(0) = afb/rl ties = efron;\nrun;\n```\n:::\n\n\nIf there are no tied event times, then the methods are equivalent.\n\nThe Breslow approximation is the easiest to program and hence it historically became the first option coded for almost all software. It then ended up as the default option when other options were added in order to maintain \"backwards compatibility\". The Efron option is more accurate if there are a large number of ties, and it was therefore selected as the default option in R. In practice the number of ties is usually small, in which case all the methods are statistically indistinguishable.\n\nFrom the arguments of `coxph` in R, there are three possible choices for handling tied event times 'ties=breslow', 'ties=efron', or 'ties=exact'. This last option is an exact partial likelihood approach, and corresponds to the \"discrete\" method in SAS. See [here](https://www.rdocumentation.org/packages/survival/versions/3.5-8/topics/coxph) for more detail. (For {survival} versions prior to 3.2-14, the options are 'ties=breslow', 'ties=efron', or 'ties=logit'.)\n\n## Reason 2: Kaplan Meier Median Survival Confidence Intervals\n\nThe default methods for calculation of the confidence interval of a KM estimator are different in the two languages (for example, for calculation of the CI associated with the Median Survival estimate, the 25th percentile and the 75th percentile).\n\nR uses \"log\" by default, and SAS uses \"log-log\" by default. As shown below, using 'conf.type' option, R can be forced to use the \"log-log\" method to match SAS. Alternatively, using the 'conftype=' option, SAS can be forced to use the \"log\" method to match R.\n\n- R: change to \"log-log\"\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit.km <- survival::survfit(\n survival::Surv(LENFOLY, FSTAT) ~ AFB,\n conf.type = \"log-log\",\n data = dat\n)\n```\n:::\n\n\n- SAS: change to \"log\"\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc lifetest data=dat conftype=log;\n time lenfoly*fstat(0);\n strata afb;\nrun;\n```\n:::\n\n\n\"log-log\" prevents the problem of having confidence intervals of \\>1 or \\<0, which might happen if using \"log\" transformation. However, both R and SAS will clip the interval at \\[0, 1\\] and report a bound \\>1 as 1 and \\<0 as 0.\n\nFrom a [reference](https://myweb.uiowa.edu/pbreheny/7210/f15/notes/9-10.pdf): The appeal of the log-log interval is clear, but the log-scale interval has the advantage of variance stabilization. As a result, simulation studies have generally found it to have better (closer to nominal) coverage; for this reason, it is the default in the `survival` package.\n\nNow if we change the confidence interval type in SAS to \"log\" and tie handling to \"efron\", the results will be identical to the results in R.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_chg_default.png){fig-align='center' width=75%}\n:::\n:::\n\n\nBelow is the side-by-side comparison for stratified analysis with default methods in SAS matched to R's, the results are also identical.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_stratified.png){fig-align='center' width=75%}\n:::\n:::\n\n\n## Reason 3: Convergence Criteria in Cox Proportional Hazards Model\n\nAnother source of discrepancy between R and SAS in Cox models can arise from the default convergence criteria used by the two software packages.\n\nIn R, the `survival::coxph()` function has a default convergence criterion for the relative change in log partial likelihood set at `1e-9`. On the other hand, SAS's `PHREG` procedure uses a default convergence criterion for the relative gradient convergence set at `1e-8`. This discrepancy in the convergence criteria can lead to slight differences in the hazard ratios (HR) obtained from the two software packages.\n\nTo achieve comparable results, it is possible to adjust the convergence criteria in SAS to match the more stringent criteria used by R. This can be done by specifying the `fconv` option in the model statement within `PHREG` to change the criteria to relative function convergence with a value of `1e-9`.\n\n- R: default convergence criterion\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit.cox <- survival::coxph(survival::Surv(LENFOLY, FSTAT) ~ AFB, data = dat)\n```\n:::\n\n\n- SAS: adjust convergence criterion\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc phreg data=dat;\n class afb;\n model lenfol*fstat(0) = afb / rl fconv = 1e-9;\nrun;\n```\n:::\n\n\nBy making this adjustment, the hazard ratios obtained from SAS will align more closely with those from R or even achieve bitwise reproducibility.\n\nThe convergence criterion details are described in their documentation:\n\n- [SAS PHREG documentation](https://support.sas.com/documentation/onlinedoc/stat/131/phreg.pdf).\n- [R `survival::coxph()` documentation](https://stat.ethz.ch/R-manual/R-devel/library/survival/html/coxph.html).\n- [R `survival::coxph.control()` ancillary arguments documentation](https://stat.ethz.ch/R-manual/R-devel/library/survival/html/coxph.control.html).\n\n# Other Cases Where Discrepancies Are Found\n\nNow we look at other cases when the data has some special type which causes a mismatch between SAS and R. Suppose a dataset has 10 observations, and the first 5 are all events, and the last 5 are all censored.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ntest <- tibble(\n time = c(54, 75, 77, 84, 87, 92, 103, 105, 112, 118),\n status = c(1, 1, 1, 1, 1, 0, 0, 0, 0, 0)\n)\n\ntest\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 10 × 2\n time status\n \n 1 54 1\n 2 75 1\n 3 77 1\n 4 84 1\n 5 87 1\n 6 92 0\n 7 103 0\n 8 105 0\n 9 112 0\n10 118 0\n```\n\n\n:::\n:::\n\n\n## Differences Observed in the KM Estimators\n\nSuppose we are interested to know the 25%, 50% and 75% quartile estimates, and the day 80, 100, and 120 estimates.\n\nBelow is the R code:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit.km <- survival::survfit(\n survival::Surv(time, status) ~ 1,\n conf.type = \"log-log\",\n data = test\n)\n\n## quantile estimates\nquantile(fit.km, probs = c(0.25, 0.5, 0.75))\n\n## landmark estimates at 80, 100, 120-day\nsummary(fit.km, times = c(80, 100, 120), extend = T)\n```\n:::\n\n\nBelow is the SAS code:\n\n\n::: {.cell}\n\n```{.sas .cell-code}\nproc lifetest data=dat outsurv=_SurvEst timelist= 80 100 120 reduceout stderr; \n time lenfoly*fstat(0);\nrun;\n```\n:::\n\n\nBelow is the side-by-side comparison:\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_special.png){fig-align='center' width=75%}\n:::\n:::\n\n\n## Reasons\n\nThe reasons for the differences are because:\n\n**Reason 1: Survival estimate does not cross the 50% percentile.**\n\nThe kth quantile for a survival curve S(t) is the location at which a horizontal line at height p= 1-k intersects the plot of S(t) as shown in the KM curve below. Since S(t) is a step function, it is possible for the curve to have a horizontal segment at exactly 1-k, in which case the midpoint of the horizontal segment is returned.\n\nFor example, using the data above, the survival probability is exactly 0.5 at time=87 and remains at 0.5 until the last censored observation at 118.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/other_diff_km.png){fig-align='center' width=75%}\n:::\n:::\n\n\nWhen using R, the median is the smallest time which survival estimate is \\<= 0.5 --\\> `(87+118) / 2 = 102.5.` However, SAS searches the smallest time which survival estimate is \\< 0.5, which does not exist in this dataset, so it gives \"NE\" (Not evaluable).\n\n\n::: {.cell}\n\n```{.r .cell-code}\npl <- survminer::ggsurvplot(fit.km, conf.int = TRUE, ggtheme = theme_light())\n\npl$plot + geom_hline(yintercept = 0.5, color = \"black\", linetype = \"solid\")\n\nsummary(fit.km)\n```\n:::\n\n\n**Reason 2: Last event censored and prior to the required landmark estimate.**\n\nFor the 120-day event-free estimate, SAS considers that 120 days is beyond the maximum observed day in the data (which was a censored event at time =118). Therefore, SAS considers this as Unknown and returns a result of \"NE\" (Not-evaluable). However, R uses the rate at last observed censored date to estimate the 120-day event free rate. As the event-free estimate at time of the last censored event at 118 was 0.5 (0.184, 0.753), R makes the assumption that this is the best estimate for the event-free rate at Time =120.\n\nIf we change the last observation in the dataset to be an event (instead of censored), R and SAS will both give 0 for the event-free survival estimate, because it is for sure that all subjects did not survive beyond 120 days.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ntest <- tibble(\n time = c(54, 75, 77, 84, 87, 92, 103, 105, 112, 118),\n status = c(1, 1, 1, 1, 1, 0, 0, 0, 0, 1)\n)\n\ntest\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 10 × 2\n time status\n \n 1 54 1\n 2 75 1\n 3 77 1\n 4 84 1\n 5 87 1\n 6 92 0\n 7 103 0\n 8 105 0\n 9 112 0\n10 118 1\n```\n\n\n:::\n:::\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_special_lst.png){fig-align='center' width=75%}\n:::\n:::\n\n\n# References\n\nBreheny P. \"Inference for the Kaplan-Meier Estimator.\" https://myweb.uiowa.edu/pbreheny/7210/f15/notes/9-10.pdf\n\nBreslow, N. E. (1974) \"Covariance Analysis of Censored Survival Data.\" Biometrics 30:89--99.\n\nEfron, B. (1977. \"The Efficiency of Cox's Likelihood Function for Censored Data.\" Journal of the American Statistical Association 72:557--565.\n\nEmmerson J. and Brown J. M. \"Understanding Survival Analysis in Clinical Trials.\" Clinical Onclogy 33:12-14.\n\nFranklin D. \"Our Survival Confidence Intervals are not the Same!\" PharmaSUG 2014 - Paper SP10. https://www.pharmasug.org/proceedings/2014/SP/PharmaSUG-2014-SP10.pdf\n\nHertz-Picciotto I. and Rockhill B. (1997) \"Validity and efficiency of approximation methods for tied survival times in Cox regression.\" Biometrics 53:1151-1156.\n\nHosmer, D.W. and Lemeshow, S. and May, S. (2008) \"Applied Survival Analysis: Regression Modeling of Time to Event Data: Second Edition.\" John Wiley and Sons Inc., New York, NY\n\n[SAS PROC LIFETEST Documentation](https://documentation.sas.com/doc/en/statug/15.2/statug_lifetest_details03.htm)\n\n[SAS PROC PHREG Documentation](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_phreg_toc.htm)", - "supporting": [], + "markdown": "---\ntitle: \"R vs SAS - Kaplan Meier and Cox-proportion hazards modelling\"\n---\n\n# Comparison of SAS vs R\n\nThe following table shows the options available in SAS and R for Kaplan Meier and Cox Proportional Hazards modelling, the capabilities of each language, and whether or not the results from each language match.\n\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Analysis | Supported in R using {survival} | Supported in SAS | Results Match | Notes |\n+===============================================================================+==============================================+========================================+===============+======================================================================================================================================+\n| Kaplan Meier with confidence intervals using log-log method | Yes (using the option conf.type = \"log-log\") | Yes (Default) | Mostly | 1\\) Survival estimates can disagree when last event is censored and survival estimate does not cross the percentile being estimated. |\n| | | | | |\n| | | | | 2\\) Survival estimates at time X can disagree when the time X is after the last observed censored time |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Kaplan Meier with confidence intervals using log method | Yes (Default) | Yes (using the option conftype=log) | Mostly | As above. |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Cox Proportional Hazards Model using breslow method for ties | Yes (using the option ties=\"breslow\") | Yes (Default) | Yes | |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Cox Proportional Hazards Model using efron method for ties | Yes (Default) | Yes (using the option ties=efron) | Yes | |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n| Cox Proportional Hazards Model using exact partial likelihood method for ties | Yes (using the option ties=\"exact\") | Yes (using the option ties=\"discrete\") | Yes | The option ties=\"exact\" in SAS uses the exact marginal likelihood which is not available in R |\n+-------------------------------------------------------------------------------+----------------------------------------------+----------------------------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n\nResults from the examples shown for R [here](https://psiaims.github.io/CAMIS/R/survival.html) and SAS [here](https://psiaims.github.io/CAMIS/SAS/survival.html) were compared below.\n\nComparing the non-stratified model results side-by-side, the CIs for the quartile estimates and landmark estimates are different between R and SAS. HR and CI also have slight differences.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_default.png){fig-align='center' width=75%}\n:::\n:::\n\n\n## Reason 1: Cox Regression Handling of Tied Survival Times\n\nThe default methods for handling ties in a Cox regression model are different which can lead to a different result for the Hazard ratio and associated confidence interval.\n\nR uses \"efron\" by default. SAS uses \"breslow\" by default. Both R and SAS are able to change these default options. By making the changes to the code below, we can force R to use \"breslow\" to match SAS, or SAS to use \"efron\" to match R. When the software use the same methods, then we obtain an identical HR and CI.\n\n- R: change method for ties to use \"breslow\"\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit.cox <- survival::coxph(\n survival::Surv(LENFOLY, FSTAT) ~ AFB,\n ties = \"breslow\",\n data = dat\n)\n```\n:::\n\n\n- SAS: change method for ties to use \"efron\"\n\n```sas\nproc phreg data=dat;\n class afb;\n model lenfol*fstat(0) = afb/rl ties = efron;\nrun;\n```\n\nIf there are no tied event times, then the methods are equivalent.\n\nThe Breslow approximation is the easiest to program and hence it historically became the first option coded for almost all software. It then ended up as the default option when other options were added in order to maintain \"backwards compatibility\". The Efron option is more accurate if there are a large number of ties, and it was therefore selected as the default option in R. In practice the number of ties is usually small, in which case all the methods are statistically indistinguishable.\n\nFrom the arguments of `coxph` in R, there are three possible choices for handling tied event times 'ties=breslow', 'ties=efron', or 'ties=exact'. This last option is an exact partial likelihood approach, and corresponds to the \"discrete\" method in SAS. See [here](https://www.rdocumentation.org/packages/survival/versions/3.5-8/topics/coxph) for more detail. (For {survival} versions prior to 3.2-14, the options are 'ties=breslow', 'ties=efron', or 'ties=logit'.)\n\n## Reason 2: Kaplan Meier Median Survival Confidence Intervals\n\nThe default methods for calculation of the confidence interval of a KM estimator are different in the two languages (for example, for calculation of the CI associated with the Median Survival estimate, the 25th percentile and the 75th percentile).\n\nR uses \"log\" by default, and SAS uses \"log-log\" by default. As shown below, using 'conf.type' option, R can be forced to use the \"log-log\" method to match SAS. Alternatively, using the 'conftype=' option, SAS can be forced to use the \"log\" method to match R.\n\n- R: change to \"log-log\"\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit.km <- survival::survfit(\n survival::Surv(LENFOLY, FSTAT) ~ AFB,\n conf.type = \"log-log\",\n data = dat\n)\n```\n:::\n\n\n- SAS: change to \"log\"\n\n```sas\nproc lifetest data=dat conftype=log;\n time lenfoly*fstat(0);\n strata afb;\nrun;\n```\n\n\"log-log\" prevents the problem of having confidence intervals of \\>1 or \\<0, which might happen if using \"log\" transformation. However, both R and SAS will clip the interval at \\[0, 1\\] and report a bound \\>1 as 1 and \\<0 as 0.\n\nFrom a [reference](https://myweb.uiowa.edu/pbreheny/7210/f15/notes/9-10.pdf): The appeal of the log-log interval is clear, but the log-scale interval has the advantage of variance stabilization. As a result, simulation studies have generally found it to have better (closer to nominal) coverage; for this reason, it is the default in the `survival` package.\n\nNow if we change the confidence interval type in SAS to \"log\" and tie handling to \"efron\", the results will be identical to the results in R.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_chg_default.png){fig-align='center' width=75%}\n:::\n:::\n\n\nBelow is the side-by-side comparison for stratified analysis with default methods in SAS matched to R's, the results are also identical.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_stratified.png){fig-align='center' width=75%}\n:::\n:::\n\n\n## Reason 3: Convergence Criteria in Cox Proportional Hazards Model\n\nAnother source of discrepancy between R and SAS in Cox models can arise from the default convergence criteria used by the two software packages.\n\nIn R, the `survival::coxph()` function has a default convergence criterion for the relative change in log partial likelihood set at `1e-9`. On the other hand, SAS's `PHREG` procedure uses a default convergence criterion for the relative gradient convergence set at `1e-8`. This discrepancy in the convergence criteria can lead to slight differences in the hazard ratios (HR) obtained from the two software packages.\n\nTo achieve comparable results, it is possible to adjust the convergence criteria in SAS to match the more stringent criteria used by R. This can be done by specifying the `fconv` option in the model statement within `PHREG` to change the criteria to relative function convergence with a value of `1e-9`.\n\n- R: default convergence criterion\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit.cox <- survival::coxph(survival::Surv(LENFOLY, FSTAT) ~ AFB, data = dat)\n```\n:::\n\n\n- SAS: adjust convergence criterion\n\n```sas\nproc phreg data=dat;\n class afb;\n model lenfol*fstat(0) = afb / rl fconv = 1e-9;\nrun;\n```\n\nBy making this adjustment, the hazard ratios obtained from SAS will align more closely with those from R or even achieve bitwise reproducibility.\n\nThe convergence criterion details are described in their documentation:\n\n- [SAS PHREG documentation](https://support.sas.com/documentation/onlinedoc/stat/131/phreg.pdf).\n- [R `survival::coxph()` documentation](https://stat.ethz.ch/R-manual/R-devel/library/survival/html/coxph.html).\n- [R `survival::coxph.control()` ancillary arguments documentation](https://stat.ethz.ch/R-manual/R-devel/library/survival/html/coxph.control.html).\n\n# Other Cases Where Discrepancies Are Found\n\nNow we look at other cases when the data has some special type which causes a mismatch between SAS and R. Suppose a dataset has 10 observations, and the first 5 are all events, and the last 5 are all censored.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ntest <- tibble(\n time = c(54, 75, 77, 84, 87, 92, 103, 105, 112, 118),\n status = c(1, 1, 1, 1, 1, 0, 0, 0, 0, 0)\n)\n\ntest\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 10 × 2\n time status\n \n 1 54 1\n 2 75 1\n 3 77 1\n 4 84 1\n 5 87 1\n 6 92 0\n 7 103 0\n 8 105 0\n 9 112 0\n10 118 0\n```\n\n\n:::\n:::\n\n\n## Differences Observed in the KM Estimators\n\nSuppose we are interested to know the 25%, 50% and 75% quartile estimates, and the day 80, 100, and 120 estimates.\n\nBelow is the R code:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nfit.km <- survival::survfit(\n survival::Surv(time, status) ~ 1,\n conf.type = \"log-log\",\n data = test\n)\n\n## quantile estimates\nquantile(fit.km, probs = c(0.25, 0.5, 0.75))\n\n## landmark estimates at 80, 100, 120-day\nsummary(fit.km, times = c(80, 100, 120), extend = T)\n```\n:::\n\n\nBelow is the SAS code:\n\n```sas\nproc lifetest data=dat outsurv=_SurvEst timelist= 80 100 120 reduceout stderr; \n time lenfoly*fstat(0);\nrun;\n```\n\nBelow is the side-by-side comparison:\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_special.png){fig-align='center' width=75%}\n:::\n:::\n\n\n## Reasons\n\nThe reasons for the differences are because:\n\n**Reason 1: Survival estimate does not cross the 50% percentile.**\n\nThe kth quantile for a survival curve S(t) is the location at which a horizontal line at height p= 1-k intersects the plot of S(t) as shown in the KM curve below. Since S(t) is a step function, it is possible for the curve to have a horizontal segment at exactly 1-k, in which case the midpoint of the horizontal segment is returned.\n\nFor example, using the data above, the survival probability is exactly 0.5 at time=87 and remains at 0.5 until the last censored observation at 118.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/other_diff_km.png){fig-align='center' width=75%}\n:::\n:::\n\n\nWhen using R, the median is the smallest time which survival estimate is \\<= 0.5 --\\> `(87+118) / 2 = 102.5.` However, SAS searches the smallest time which survival estimate is \\< 0.5, which does not exist in this dataset, so it gives \"NE\" (Not evaluable).\n\n\n::: {.cell}\n\n```{.r .cell-code}\npl <- survminer::ggsurvplot(fit.km, conf.int = TRUE, ggtheme = theme_light())\n\npl$plot + geom_hline(yintercept = 0.5, color = \"black\", linetype = \"solid\")\n\nsummary(fit.km)\n```\n:::\n\n\n**Reason 2: Last event censored and prior to the required landmark estimate.**\n\nFor the 120-day event-free estimate, SAS considers that 120 days is beyond the maximum observed day in the data (which was a censored event at time =118). Therefore, SAS considers this as Unknown and returns a result of \"NE\" (Not-evaluable). However, R uses the rate at last observed censored date to estimate the 120-day event free rate. As the event-free estimate at time of the last censored event at 118 was 0.5 (0.184, 0.753), R makes the assumption that this is the best estimate for the event-free rate at Time =120.\n\nIf we change the last observation in the dataset to be an event (instead of censored), R and SAS will both give 0 for the event-free survival estimate, because it is for sure that all subjects did not survive beyond 120 days.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ntest <- tibble(\n time = c(54, 75, 77, 84, 87, 92, 103, 105, 112, 118),\n status = c(1, 1, 1, 1, 1, 0, 0, 0, 0, 1)\n)\n\ntest\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 10 × 2\n time status\n \n 1 54 1\n 2 75 1\n 3 77 1\n 4 84 1\n 5 87 1\n 6 92 0\n 7 103 0\n 8 105 0\n 9 112 0\n10 118 1\n```\n\n\n:::\n:::\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](../images/survival/r_sas_special_lst.png){fig-align='center' width=75%}\n:::\n:::\n\n\n# References\n\nBreheny P. \"Inference for the Kaplan-Meier Estimator.\" https://myweb.uiowa.edu/pbreheny/7210/f15/notes/9-10.pdf\n\nBreslow, N. E. (1974) \"Covariance Analysis of Censored Survival Data.\" Biometrics 30:89--99.\n\nEfron, B. (1977. \"The Efficiency of Cox's Likelihood Function for Censored Data.\" Journal of the American Statistical Association 72:557--565.\n\nEmmerson J. and Brown J. M. \"Understanding Survival Analysis in Clinical Trials.\" Clinical Onclogy 33:12-14.\n\nFranklin D. \"Our Survival Confidence Intervals are not the Same!\" PharmaSUG 2014 - Paper SP10. https://www.pharmasug.org/proceedings/2014/SP/PharmaSUG-2014-SP10.pdf\n\nHertz-Picciotto I. and Rockhill B. (1997) \"Validity and efficiency of approximation methods for tied survival times in Cox regression.\" Biometrics 53:1151-1156.\n\nHosmer, D.W. and Lemeshow, S. and May, S. (2008) \"Applied Survival Analysis: Regression Modeling of Time to Event Data: Second Edition.\" John Wiley and Sons Inc., New York, NY\n\n[SAS PROC LIFETEST Documentation](https://documentation.sas.com/doc/en/statug/15.2/statug_lifetest_details03.htm)\n\n[SAS PROC PHREG Documentation](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_phreg_toc.htm)", + "supporting": [ + "r-sas_survival_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/Comp/r-sas_survival_cif/execute-results/html.json b/_freeze/Comp/r-sas_survival_cif/execute-results/html.json index b7bf2940f..7d9f8a6ac 100644 --- a/_freeze/Comp/r-sas_survival_cif/execute-results/html.json +++ b/_freeze/Comp/r-sas_survival_cif/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "077fddef664329230c4523bba2e7ecad", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS - Estimating Cumulative Incidence Functions\"\n---\n\n# Comparison of R and SAS\n\nThe following table shows the options available in R and SAS for estimating cumulative incidence functions (CIFs) in a competing risk analysis, especially the capabilities and whether the results match.\n\n| Analysis | Supported in R package `tidycmprsk` | Supported in SAS `PROC LIFETEST` | Results Match |\n|--------------------|------------------|------------------|------------------|\n| CIF estimates | Yes: with function `cuminc()` | Yes: with `eventcode` option in `TIME` statement | Yes |\n| Gray's test for equality across groups | Yes: default when the group variable (a factor) is on the right-hand side of the input formula | Yes: default with `strata` statement | Yes |\n| Variance estimates for the CIF estimates using Aalen (1978) | Yes: default | Yes (default) | Yes |\n| Variance estimates for the CIF estimates using the delta method | No | Yes: with option `error=delta` in `PROC TEST` statement | N/A |\n| Confidence intervals for CIF estimates using log-log transformation | Yes: default | Yes: default | Yes |\n| Confidence intervals for CIF estimates using other transformations | No | Yes: with `conftype` option in `LIFETEST` statement | N/A |\n| CIF estimates for specified time points | Yes: with `times` option when summarizing results, e.g., using `tidy()` | Yes: with `timelist` option in `LIFETEST` statement | Yes |\n| CIF plot by groups | Yes: with `ggsurvfit::ggcumin()` | Yes: with `plots=cif` option in `LIFETEST` statement | N/A |\n\nAdditional details for using `tidycmprsk` are given [here](https://psiaims.github.io/CAMIS/R/survival_cif.html \"cif in r\") and for SAS `PROC LIFETEST` [here](https://psiaims.github.io/CAMIS/SAS/survival_cif.html \"cif in sas\") .\n\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-24\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n package * version date (UTC) lib source\n cmprsk 2.2-12 2024-05-19 [1] RSPM\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n tidycmprsk 1.1.1 2025-11-14 [1] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n\n# References\n\n[SAS PROC LIFETEST Documentation on CIF estimates](https://documentation.sas.com/doc/en/statug/15.2/statug_lifetest_details25.htm#statug.lifetest.lftcifest \"cif in sas\")\n\n[R package 'tidycmprsk' Documentation](https://github.com/MSKCC-Epi-Bio/tidycmprsk \"cif in r\")\n", + "markdown": "---\ntitle: \"R vs SAS - Estimating Cumulative Incidence Functions\"\n---\n\n# Comparison of R and SAS\n\nThe following table shows the options available in R and SAS for estimating cumulative incidence functions (CIFs) in a competing risk analysis, especially the capabilities and whether the results match.\n\n| Analysis | Supported in R package `tidycmprsk` | Supported in SAS `PROC LIFETEST` | Results Match |\n|--------------------|------------------|------------------|------------------|\n| CIF estimates | Yes: with function `cuminc()` | Yes: with `eventcode` option in `TIME` statement | Yes |\n| Gray's test for equality across groups | Yes: default when the group variable (a factor) is on the right-hand side of the input formula | Yes: default with `strata` statement | Yes |\n| Variance estimates for the CIF estimates using Aalen (1978) | Yes: default | Yes (default) | Yes |\n| Variance estimates for the CIF estimates using the delta method | No | Yes: with option `error=delta` in `PROC TEST` statement | N/A |\n| Confidence intervals for CIF estimates using log-log transformation | Yes: default | Yes: default | Yes |\n| Confidence intervals for CIF estimates using other transformations | No | Yes: with `conftype` option in `LIFETEST` statement | N/A |\n| CIF estimates for specified time points | Yes: with `times` option when summarizing results, e.g., using `tidy()` | Yes: with `timelist` option in `LIFETEST` statement | Yes |\n| CIF plot by groups | Yes: with `ggsurvfit::ggcumin()` | Yes: with `plots=cif` option in `LIFETEST` statement | N/A |\n\nAdditional details for using `tidycmprsk` are given [here](https://psiaims.github.io/CAMIS/R/survival_cif.html \"cif in r\") and for SAS `PROC LIFETEST` [here](https://psiaims.github.io/CAMIS/SAS/survival_cif.html \"cif in sas\") .\n\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n package * version date (UTC) lib source\n cmprsk 2.2-12 2024-05-19 [1] RSPM (R 4.5.0)\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n tidycmprsk 1.1.1 2025-11-14 [1] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n\n# References\n\n[SAS PROC LIFETEST Documentation on CIF estimates](https://documentation.sas.com/doc/en/statug/15.2/statug_lifetest_details25.htm#statug.lifetest.lftcifest \"cif in sas\")\n\n[R package 'tidycmprsk' Documentation](https://github.com/MSKCC-Epi-Bio/tidycmprsk \"cif in r\")\n", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/Comp/r-sas_survival_csh/execute-results/html.json b/_freeze/Comp/r-sas_survival_csh/execute-results/html.json index 1c0c1622a..f03b9bde6 100644 --- a/_freeze/Comp/r-sas_survival_csh/execute-results/html.json +++ b/_freeze/Comp/r-sas_survival_csh/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "b309f1c14f7f5fd8c8f488305a56c0d0", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R vs SAS - Estimating and Testing Cause-Specific Hazard\"\n---\n\n# Comparison of R and SAS\n\nThe following table shows the options available in R and SAS for estimating and testing cause-specific hazard in a competing risk analysis, especially the capabilities and whether the results match.\n\n| Analysis | Supported in R package `survival` | Supported in SAS `PROC PHREG` | Results Match |\n|-------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|--------------------------|\n| Cause-specific hazard ratio estimates | Yes: with `coxph()` | Yes | Yes |\n| Stratified cause-specific hazard ratio estimates | Yes: with the stratification variable `x` specified as `strata(factor(x))` on the right-hand side of the input formula | Yes: with `strata` statement | Yes |\n| Variance estimates for the parameter estimates with robust sandwich estimator | Yes: default (`robust = TRUE)` | Yes: with `covsandwich` or `covs` option in `proc phreg` statement | Yes |\n| Confidence intervals for hazard ratio estimates | Yes: Wald's method by default | Yes: Wald's method by default | Yes |\n| Estimating cause specific hazard for multiple events | Yes | Yes | Depends (see note below) |\n\nAdditional details for using `survival` in R are given [here](https://psiaims.github.io/CAMIS/R/survival_csh.html \"csh in r\") and for SAS `PROC PHREG` [here](https://psiaims.github.io/CAMIS/SAS/survival_csh.html \"csh in sas\") .\n\n### Estimating cause specific hazard ratios for multiple events\n\nR and SAS have different approach when it comes to estimating the hazard ratios for multiple events. Results for the hazard ratio estimates are the same between the two; what is different is the global hypothesis:\n\n- The global hypothesis per `coxph()` in this case is \"There is no difference in the hazards of experiencing any of the events.\"\n\n- In `PROC PHREG`, one syntax allows the hazard ratio estimates to be generated for all events. However, there is no corresponding global hypothesis as in `coxph()` in R. In SAS, there are only individual global hypotheses, one for each event. In addition, currently, when this syntax is used in SAS, stratified analysis cannot be implemented.\n\n## Summary\n\n- Most of the functionality of `survival::coxph()` and `proc phreg` also apply to estimating cause-specific hazards in competing risks settings.\n\n- Due to the different internal numerical estimation methods of R and SAS, results only match up to the 4th decimal places. However, overall consistency can be established between the two for estimating and testing cause-specific hazard ratio using Cox's PH model.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-24\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n package * version date (UTC) lib source\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n\n# References\n\n[SAS PROC LIFETEST Documentation on CIF estimates](https://documentation.sas.com/doc/en/statug/15.2/statug_lifetest_details25.htm#statug.lifetest.lftcifest \"cif in sas\")\n\n[R package 'tidycmprsk' Documentation](https://github.com/MSKCC-Epi-Bio/tidycmprsk \"cif in r\")\n", + "markdown": "---\ntitle: \"R vs SAS - Estimating and Testing Cause-Specific Hazard\"\n---\n\n# Comparison of R and SAS\n\nThe following table shows the options available in R and SAS for estimating and testing cause-specific hazard in a competing risk analysis, especially the capabilities and whether the results match.\n\n| Analysis | Supported in R package `survival` | Supported in SAS `PROC PHREG` | Results Match |\n|-------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|--------------------------|\n| Cause-specific hazard ratio estimates | Yes: with `coxph()` | Yes | Yes |\n| Stratified cause-specific hazard ratio estimates | Yes: with the stratification variable `x` specified as `strata(factor(x))` on the right-hand side of the input formula | Yes: with `strata` statement | Yes |\n| Variance estimates for the parameter estimates with robust sandwich estimator | Yes: default (`robust = TRUE)` | Yes: with `covsandwich` or `covs` option in `proc phreg` statement | Yes |\n| Confidence intervals for hazard ratio estimates | Yes: Wald's method by default | Yes: Wald's method by default | Yes |\n| Estimating cause specific hazard for multiple events | Yes | Yes | Depends (see note below) |\n\nAdditional details for using `survival` in R are given [here](https://psiaims.github.io/CAMIS/R/survival_csh.html \"csh in r\") and for SAS `PROC PHREG` [here](https://psiaims.github.io/CAMIS/SAS/survival_csh.html \"csh in sas\") .\n\n### Estimating cause specific hazard ratios for multiple events\n\nR and SAS have different approach when it comes to estimating the hazard ratios for multiple events. Results for the hazard ratio estimates are the same between the two; what is different is the global hypothesis:\n\n- The global hypothesis per `coxph()` in this case is \"There is no difference in the hazards of experiencing any of the events.\"\n\n- In `PROC PHREG`, one syntax allows the hazard ratio estimates to be generated for all events. However, there is no corresponding global hypothesis as in `coxph()` in R. In SAS, there are only individual global hypotheses, one for each event. In addition, currently, when this syntax is used in SAS, stratified analysis cannot be implemented.\n\n## Summary\n\n- Most of the functionality of `survival::coxph()` and `proc phreg` also apply to estimating cause-specific hazards in competing risks settings.\n\n- Due to the different internal numerical estimation methods of R and SAS, results only match up to the 4th decimal places. However, overall consistency can be established between the two for estimating and testing cause-specific hazard ratio using Cox's PH model.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n package * version date (UTC) lib source\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n─ External software ──────────────────────────────────────────────────────────\n setting value\n SAS 9.04.01M7P080520\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n\n# References\n\n[SAS PROC LIFETEST Documentation on CIF estimates](https://documentation.sas.com/doc/en/statug/15.2/statug_lifetest_details25.htm#statug.lifetest.lftcifest \"cif in sas\")\n\n[R package 'tidycmprsk' Documentation](https://github.com/MSKCC-Epi-Bio/tidycmprsk \"cif in r\")\n", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/R/Weighted-log-rank/execute-results/html.json b/_freeze/R/Weighted-log-rank/execute-results/html.json index 38da2ba41..d5beaa97b 100644 --- a/_freeze/R/Weighted-log-rank/execute-results/html.json +++ b/_freeze/R/Weighted-log-rank/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "f8f3ef2a4511507643f3f8f10a1edba9", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Testing approaches under non-proportional hazards\"\n---\n\n# Introduction\n\nIn clinical studies with time-to-event outcomes, it is commonly assumed that the hazard functions of two groups are proportional. The standard log-rank test is widely used to test the equivalence of survival functions. However, several scenarios can lead to non-proportional hazards (NPH). For example, a delayed treatment effect may be observed in the treatment arm which can lead to departure from proportionality of the survival curves. Thus there are many tests available in the literature that can handle such scenarios. Most commonly used tests are as follows:\n\n- Weighted log-rank test\n- Restricted Mean Survival Time (RMST)\n- Milestone survival\n- Max-Combo test.\n\nWhile these tests may be explored in a separate document, this particular document focuses solely on the weighted log-rank test.\\\n\n# Weighted log-rank test\n\nSuppose we have two groups (e.g. treatment and control, male and female etc.) with survival functions $S_1$ & $S_2$ respectively. The null and alternative hypotheses are given as: $$H_0 : S_1(t)=S_2(t) \\mbox{ }\\forall t \\mbox{ v/s } H_1 : S_1(t) \\neq S_2(t) \\mbox{ for some t. }$$ Since alternative hypothesis is composite, it includes multiple scenarios. Hence the power calculation is difficult to implement. One way to tackle this situation is to consider the Lehman alternative given by $H_L : S_1(t)=(S_2(t))^\\psi$ for all $t$ where $0<\\psi<1$. Alternatively, $$ H_0 : \\psi=1 \\ v/s \\ H_1: \\psi<1$$\n\nwhich implies subjects in group 1 will have longer survival times than subjects in group 2. For more details, refer to Page 44 of the [Moore (2016)](https://xsliulab.github.io/Workshop/2021/week3/survival-analysis-book.pdf).\\\nThe test statistic for weighted log-rank test is given by, $$ Z = \\frac{\\sum_{j=1}^{D}w_j(o_{j} -e_j)}{\\sqrt{\\sum_{j=1}^{D}w_j^2 v_j}} \\to N(0,1), \\text{under} \\ H_0$$ Equivalently, $$ Z^2 = \\frac{\\big[\\sum_{j=1}^{D}w_j(o_j -e_j)\\big]^2}{\\sum_{j=1}^{D}w_j^2 v_j} \\to \\chi^2_1, \\text{under} \\ H_0.$$\\\nHere $t_1 \n1 1 83 0 89 152 78 25.5 1 1 0 0 0\n2 2 49 0 84 120 60 24.0 1 0 0 0 0\n3 3 70 1 83 147 88 22.1 0 0 0 0 0\n4 4 70 0 65 123 76 26.6 1 0 0 1 0\n5 5 70 0 63 135 85 24.4 1 0 0 0 0\n6 6 70 0 76 83 54 23.2 1 0 0 0 1\n# ℹ 7 more variables: MIORD , MITYPE , YEAR , LOS ,\n# DSTAT , LENFOL , FSTAT \n```\n\n\n:::\n:::\n\n\n## *survdiff()*\n\nThis function uses $G(\\rho)=\\hat{S}(t)^\\rho, \\rho \\geq 0$ , where $\\hat{S}(t)$ is the Kaplan-Meier estimate of the survival function at time $t$. If $\\rho = 0$, then this is the standard log-rank test.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(survival)\nWLRtest <- survival::survdiff(\n survival::Surv(LENFOL, FSTAT) ~ AFB,\n rho = 3,\n data = dat\n)\nWLRtest\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nCall:\nsurvival::survdiff(formula = survival::Surv(LENFOL, FSTAT) ~ \n AFB, data = dat, rho = 3)\n\n N Observed Expected (O-E)^2/E (O-E)^2/V\nAFB=0 422 86.3 94.5 0.718 7.68\nAFB=1 78 24.2 16.0 4.245 7.68\n\n Chisq= 7.7 on 1 degrees of freedom, p= 0.006 \n```\n\n\n:::\n:::\n\n\nFor the illustration, $\\rho$ is taken as 3 while calculating weights and the weighted log rank test reject the null hypothesis at 2.5% level of significance.\n\n## *wlrt*()\n\nThis function uses $G(\\rho,\\gamma)=\\hat{S}(t)^\\rho (1-\\hat{S}(t))^\\gamma; \\rho,\\gamma \\geq 0,$ , where $\\hat{S}(t)$ is the Kaplan-Meier estimate of the survival function at time $t$. If $\\rho = \\gamma = 0$, then this is the standard log-rank test. When $\\rho=0, \\gamma=1$ this test can be used to detect early difference in the survival curves, when $\\rho=1, \\gamma = 0$, this test can be used to detect late differences in the survival curves and when $\\rho=1, \\gamma = 1$ this test can be used to test middle differences in the survival curves. Also it is to be noted that this test gives the Z-score as the test statistic which can be squared to obtain the chi-square statistic.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(nphRCT)\nWL <- nphRCT::wlrt(\n survival::Surv(LENFOL, FSTAT) ~ AFB,\n data = dat,\n method = \"fh\",\n rho = 0,\n gamma = 0\n)\nWL\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n u v_u z trt_group\n1 16.77487 25.81609 3.301521 1\n```\n\n\n:::\n:::\n\n\nTo obtain the corresponding $p$-value we can either use *2(1-pnorm(abs(WL\\$z),0,1))* or we can square the test statistic *WL\\$z* by using *(WL\\$z)\\^2* and obtain the corresponding $p$-values as *1 - pchisq((WL\\$z)\\^2,1)* , both the $p$-values will be the same.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n2 * (1 - pnorm(abs(WL$z), 0, 1))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.0009616214\n```\n\n\n:::\n\n```{.r .cell-code}\n(WL$z)^2\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 10.90004\n```\n\n\n:::\n\n```{.r .cell-code}\n1 - pchisq((WL$z)^2, 1)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.0009616214\n```\n\n\n:::\n:::\n\n\nFor the illustration purpose we used $\\rho=0,\\ \\gamma=0$ and in this scenario weighted log-rank test becomes standard log-rank test. Therefore, the result obtained in this illustration is consistent with the result obtained in [standard log-rank test](Survival%20Analysis%20Using%20R%20(psiaims.github.io)).\n\n# References\n\n1. Knezevic, A., & Patil, S. (2020). Combination weighted log-rank tests for survival analysis with non-proportional hazards. *SAS Global Forum*.\n2. Magirr, D., & Barrott, I. (2022). nphRCT: Non-Proportional Hazards in Randomized Controlled Trials.\n3. Moore, D. F. (2016). *Applied survival analysis using R* (Vol. 473, pp. 1-10). Cham: Springer.\n4. Therneau T (2024). A Package for Survival Analysis in R.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-23\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P nphRCT * 0.1.1 2024-06-27 [?] RSPM\n survival * 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n", + "markdown": "---\ntitle: \"Testing approaches under non-proportional hazards\"\n---\n\n# Introduction\n\nIn clinical studies with time-to-event outcomes, it is commonly assumed that the hazard functions of two groups are proportional. The standard log-rank test is widely used to test the equivalence of survival functions. However, several scenarios can lead to non-proportional hazards (NPH). For example, a delayed treatment effect may be observed in the treatment arm which can lead to departure from proportionality of the survival curves. Thus there are many tests available in the literature that can handle such scenarios. Most commonly used tests are as follows:\n\n- Weighted log-rank test\n- Restricted Mean Survival Time (RMST)\n- Milestone survival\n- Max-Combo test.\n\nWhile these tests may be explored in a separate document, this particular document focuses solely on the weighted log-rank test.\\\n\n# Weighted log-rank test\n\nSuppose we have two groups (e.g. treatment and control, male and female etc.) with survival functions $S_1$ & $S_2$ respectively. The null and alternative hypotheses are given as: $$H_0 : S_1(t)=S_2(t) \\mbox{ }\\forall t \\mbox{ v/s } H_1 : S_1(t) \\neq S_2(t) \\mbox{ for some t. }$$ Since alternative hypothesis is composite, it includes multiple scenarios. Hence the power calculation is difficult to implement. One way to tackle this situation is to consider the Lehman alternative given by $H_L : S_1(t)=(S_2(t))^\\psi$ for all $t$ where $0<\\psi<1$. Alternatively, $$ H_0 : \\psi=1 \\ v/s \\ H_1: \\psi<1$$\n\nwhich implies subjects in group 1 will have longer survival times than subjects in group 2. For more details, refer to Page 44 of the [Moore (2016)](https://xsliulab.github.io/Workshop/2021/week3/survival-analysis-book.pdf).\\\nThe test statistic for weighted log-rank test is given by, $$ Z = \\frac{\\sum_{j=1}^{D}w_j(o_{j} -e_j)}{\\sqrt{\\sum_{j=1}^{D}w_j^2 v_j}} \\to N(0,1), \\text{under} \\ H_0$$ Equivalently, $$ Z^2 = \\frac{\\big[\\sum_{j=1}^{D}w_j(o_j -e_j)\\big]^2}{\\sum_{j=1}^{D}w_j^2 v_j} \\to \\chi^2_1, \\text{under} \\ H_0.$$\\\nHere $t_1 \n1 1 83 0 89 152 78 25.5 1 1 0 0 0\n2 2 49 0 84 120 60 24.0 1 0 0 0 0\n3 3 70 1 83 147 88 22.1 0 0 0 0 0\n4 4 70 0 65 123 76 26.6 1 0 0 1 0\n5 5 70 0 63 135 85 24.4 1 0 0 0 0\n6 6 70 0 76 83 54 23.2 1 0 0 0 1\n# ℹ 7 more variables: MIORD , MITYPE , YEAR , LOS ,\n# DSTAT , LENFOL , FSTAT \n```\n\n\n:::\n:::\n\n\n## *survdiff()*\n\nThis function uses $G(\\rho)=\\hat{S}(t)^\\rho, \\rho \\geq 0$ , where $\\hat{S}(t)$ is the Kaplan-Meier estimate of the survival function at time $t$. If $\\rho = 0$, then this is the standard log-rank test.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(survival)\nWLRtest <- survival::survdiff(\n survival::Surv(LENFOL, FSTAT) ~ AFB,\n rho = 3,\n data = dat\n)\nWLRtest\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nCall:\nsurvival::survdiff(formula = survival::Surv(LENFOL, FSTAT) ~ \n AFB, data = dat, rho = 3)\n\n N Observed Expected (O-E)^2/E (O-E)^2/V\nAFB=0 422 86.3 94.5 0.718 7.68\nAFB=1 78 24.2 16.0 4.245 7.68\n\n Chisq= 7.7 on 1 degrees of freedom, p= 0.006 \n```\n\n\n:::\n:::\n\n\nFor the illustration, $\\rho$ is taken as 3 while calculating weights and the weighted log rank test reject the null hypothesis at 2.5% level of significance.\n\n## *wlrt*()\n\nThis function uses $G(\\rho,\\gamma)=\\hat{S}(t)^\\rho (1-\\hat{S}(t))^\\gamma; \\rho,\\gamma \\geq 0,$ , where $\\hat{S}(t)$ is the Kaplan-Meier estimate of the survival function at time $t$. If $\\rho = \\gamma = 0$, then this is the standard log-rank test. When $\\rho=0, \\gamma=1$ this test can be used to detect early difference in the survival curves, when $\\rho=1, \\gamma = 0$, this test can be used to detect late differences in the survival curves and when $\\rho=1, \\gamma = 1$ this test can be used to test middle differences in the survival curves. Also it is to be noted that this test gives the Z-score as the test statistic which can be squared to obtain the chi-square statistic.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(nphRCT)\nWL <- nphRCT::wlrt(\n survival::Surv(LENFOL, FSTAT) ~ AFB,\n data = dat,\n method = \"fh\",\n rho = 0,\n gamma = 0\n)\nWL\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n u v_u z trt_group\n1 16.77487 25.81609 3.301521 1\n```\n\n\n:::\n:::\n\n\nTo obtain the corresponding $p$-value we can either use *2(1-pnorm(abs(WL\\$z),0,1))* or we can square the test statistic *WL\\$z* by using *(WL\\$z)\\^2* and obtain the corresponding $p$-values as *1 - pchisq((WL\\$z)\\^2,1)* , both the $p$-values will be the same.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n2 * (1 - pnorm(abs(WL$z), 0, 1))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.0009616214\n```\n\n\n:::\n\n```{.r .cell-code}\n(WL$z)^2\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 10.90004\n```\n\n\n:::\n\n```{.r .cell-code}\n1 - pchisq((WL$z)^2, 1)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.0009616214\n```\n\n\n:::\n:::\n\n\nFor the illustration purpose we used $\\rho=0,\\ \\gamma=0$ and in this scenario weighted log-rank test becomes standard log-rank test. Therefore, the result obtained in this illustration is consistent with the result obtained in [standard log-rank test](Survival%20Analysis%20Using%20R%20(psiaims.github.io)).\n\n# References\n\n1. Knezevic, A., & Patil, S. (2020). Combination weighted log-rank tests for survival analysis with non-proportional hazards. *SAS Global Forum*.\n2. Magirr, D., & Barrott, I. (2022). nphRCT: Non-Proportional Hazards in Randomized Controlled Trials.\n3. Moore, D. F. (2016). *Applied survival analysis using R* (Vol. 473, pp. 1-10). Cham: Springer.\n4. Therneau T (2024). A Package for Survival Analysis in R.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P nphRCT * 0.1.1 2024-06-27 [?] RSPM (R 4.5.0)\n survival * 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/R/ancova/execute-results/html.json b/_freeze/R/ancova/execute-results/html.json index 86d4367b5..6a9fb9a05 100644 --- a/_freeze/R/ancova/execute-results/html.json +++ b/_freeze/R/ancova/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "ad6ead4389c7b9923a31c04bd64c4aa7", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Ancova\"\noutput: html_document\ndate: \"2023-06-01\"\n---\n\n\n\n## Introduction\n\nANOVA is a statistical method used to compare the means of three or more groups to determine if at least one group mean is significantly different from the others. Please see the [ANOVA document](anova.qmd) for more information. ANCOVA is an extension to ANOVA.\n\nANCOVA (Analysis of Covariance) is a statistical method that compares the means of two or more groups while controlling for one or more continuous covariates. By adjusting for these covariates, ANCOVA helps to reduce potential confounding effects, allowing for a clearer assessment of the main treatment effects. It assumes linear relationships between covariates and the dependent variable, along with normality and homogeneity of variances.\n\nWe follow the example from link [Analysis of Covariance](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_glm_examples04.htm)\n\n## Data Summary\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndf_sas |> glimpse()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nRows: 30\nColumns: 3\n$ drug A, A, A, A, A, A, A, A, A, A, D, D, D, D, D, D, D, D, D, D, F, F,…\n$ pre 11, 8, 5, 14, 19, 6, 10, 6, 11, 3, 6, 6, 7, 8, 18, 8, 19, 8, 5, 1…\n$ post 6, 0, 2, 8, 11, 4, 13, 1, 8, 0, 0, 2, 3, 1, 18, 4, 14, 9, 1, 9, 1…\n```\n\n\n:::\n\n```{.r .cell-code}\ndf_sas |> summary()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n drug pre post \n A:10 Min. : 3.00 Min. : 0.00 \n D:10 1st Qu.: 7.00 1st Qu.: 2.00 \n F:10 Median :10.50 Median : 7.00 \n Mean :10.73 Mean : 7.90 \n 3rd Qu.:13.75 3rd Qu.:12.75 \n Max. :21.00 Max. :23.00 \n```\n\n\n:::\n:::\n\n\n## The Model\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel_ancova <- lm(post ~ drug + pre, data = df_sas)\n\nmodel_glance <- model_ancova |>\n glance()\nmodel_tidy <- model_ancova |>\n tidy()\nmodel_glance |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n \n \n
r.squaredadj.r.squaredsigmastatisticp.valuedflogLikAICBICdeviancedf.residualnobs
0.67626090.63890644.00577818.103861.501369e-063-82.05377174.1075181.1135417.20262630
\n
\n```\n\n:::\n\n```{.r .cell-code}\nmodel_tidy |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n\n\n\n\n \n\n\n\n\n \n\n\n\n\n \n \n
termestimatestd.errorstatisticp.value
(Intercept)-3.88080941.9862017-1.95388496.155192e-02
drugD0.10897131.79513510.06070379.520594e-01
drugF3.44613831.88678061.82646477.928458e-02
pre0.98718380.16449766.00120612.454330e-06
\n
\n```\n\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel_table <- model_ancova |>\n anova() |>\n tidy()\n\ntotal_df <- sum(model_table$df)\ntotal_sumsq <- sum(model_table$sumsq)\n\nmodel_table |>\n add_row(term = \"Total\", df = total_df, sumsq = total_sumsq) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
termdfsumsqmeansqstatisticp.value
drug2293.6000146.800009.1485539.812371e-04
pre1577.8974577.8974036.0144752.454330e-06
Residuals26417.202616.04625NANA
Total291288.7000NANANA
\n
\n```\n\n:::\n:::\n\n\n### Sums of Squares Tables {.unnumbered}\n\n#### Type I\nThis can be calculated using, the base R {stats} package or the {rstatix} package. Both give the same result.\n\n##### stats\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::anova(model_ancova)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnalysis of Variance Table\n\nResponse: post\n Df Sum Sq Mean Sq F value Pr(>F) \ndrug 2 293.6 146.80 9.1486 0.0009812 ***\npre 1 577.9 577.90 36.0145 2.454e-06 ***\nResiduals 26 417.2 16.05 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\n##### rstatix\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndf_sas |>\n anova_test(post ~ drug + pre, type = 1, detailed = TRUE) |>\n get_anova_table() |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n
EffectDFnDFdSSnSSdFpp<.05ges
drug226293.600417.2039.1499.81e-04*0.413
pre126577.897417.20336.0142.45e-06*0.581
\n
\n```\n\n:::\n:::\n\n\n#### Type II\n\nThis can be calculated using the {car} package or the {rstatix} package. Both give the same result.\n\n##### car\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncar::Anova(model_ancova, type = \"II\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnova Table (Type II tests)\n\nResponse: post\n Sum Sq Df F value Pr(>F) \ndrug 68.55 2 2.1361 0.1384 \npre 577.90 1 36.0145 2.454e-06 ***\nResiduals 417.20 26 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\n##### rstatix\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndf_sas |>\n anova_test(post ~ drug + pre, type = 2, detailed = TRUE) |>\n get_anova_table() |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n
EffectSSnSSdDFnDFdFpp<.05ges
drug68.554417.2032262.1361.38e-010.141
pre577.897417.20312636.0142.45e-06*0.581
\n
\n```\n\n:::\n:::\n\n\n#### Type III\n\nThis can be calculated using the base R {stats} package, the {car} package or the {rstatix} package. All give the same result.\n\nNote: Calculating type III sums of squares in R is a bit tricky, because the multi-way ANOVA model is over-paramerterised. So when running the linear model we need to select a design matrix that sums to zero. In R those options will be either `\"contr.sum\"` or `\"contr.poly\"`\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Drug design matrix\ncontr.sum(4) # Using 4 here as we have 4 levels of drug\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [,1] [,2] [,3]\n1 1 0 0\n2 0 1 0\n3 0 0 1\n4 -1 -1 -1\n```\n\n\n:::\n\n```{.r .cell-code}\n# Disease design matrix\ncontr.sum(3)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [,1] [,2]\n1 1 0\n2 0 1\n3 -1 -1\n```\n\n\n:::\n:::\n\n\nWhile not relevant for this example as the disease variable isn't ordinal the polynomial design matrix would look like\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncontr.poly(3)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n .L .Q\n[1,] -7.071068e-01 0.4082483\n[2,] -9.681035e-17 -0.8164966\n[3,] 7.071068e-01 0.4082483\n```\n\n\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel_ancova <- lm(\n post ~ drug + pre, data = df_sas,\n contrasts = list(drug = \"contr.sum\")\n)\n```\n:::\n\n\n##### stats\n\nUsing the base stats package, you can use the `drop1()` function which drops all possible single terms in a model. The scope term specifies how things can be dropped.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::drop1(model_ancova, scope = . ~ ., test = \"F\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nSingle term deletions\n\nModel:\npost ~ drug + pre\n Df Sum of Sq RSS AIC F value Pr(>F) \n 417.20 86.971 \ndrug 2 68.55 485.76 87.535 2.1361 0.1384 \npre 1 577.90 995.10 111.049 36.0145 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\n##### car\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncar::Anova(model_ancova, type = \"III\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnova Table (Type III tests)\n\nResponse: post\n Sum Sq Df F value Pr(>F) \n(Intercept) 31.93 1 1.9898 0.1702 \ndrug 68.55 2 2.1361 0.1384 \npre 577.90 1 36.0145 2.454e-06 ***\nResiduals 417.20 26 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndf_sas |>\n anova_test(post ~ drug + pre, type = 3, detailed = TRUE) |>\n get_anova_table() |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n
EffectSSnSSdDFnDFdFpp<.05ges
(Intercept)31.929417.2031261.9901.70e-010.071
drug68.554417.2032262.1361.38e-010.141
pre577.897417.20312636.0142.45e-06*0.581
\n
\n```\n\n:::\n:::\n\n\n### Least Squares Means\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel_ancova |>\n emmeans::lsmeans(\"drug\") |>\n emmeans::pwpm(pvals = TRUE, means = TRUE)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n A D F\nA [ 6.71] 0.9980 0.1809\nD -0.109 [ 6.82] 0.1893\nF -3.446 -3.337 [10.16]\n\nRow and column labels: drug\nUpper triangle: P values adjust = \"tukey\"\nDiagonal: [Estimates] (lsmean) \nLower triangle: Comparisons (estimate) earlier vs. later\n```\n\n\n:::\n\n```{.r .cell-code}\nmodel_ancova |>\n emmeans::lsmeans(\"drug\") |>\n plot(comparisons = TRUE)\n```\n\n::: {.cell-output-display}\n![](ancova_files/figure-html/unnamed-chunk-14-1.png){width=672}\n:::\n:::\n\n\n## sasLM Package\n\nThe following code performs an ANCOVA analysis using the **sasLM** package. This package was written specifically to replicate SAS statistics. The console output is also organized in a manner that is similar to SAS.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(sasLM)\n\nsasLM::GLM(post ~ drug + pre, df_sas, BETA = TRUE, EMEAN = TRUE)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n$ANOVA\nResponse : post\n Df Sum Sq Mean Sq F value Pr(>F) \nMODEL 3 871.5 290.499 18.104 1.501e-06 ***\nRESIDUALS 26 417.2 16.046 \nCORRECTED TOTAL 29 1288.7 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$Fitness\n Root MSE post Mean Coef Var R-square Adj R-sq\n 4.005778 7.9 50.70604 0.6762609 0.6389064\n\n$`Type I`\n Df Sum Sq Mean Sq F value Pr(>F) \ndrug 2 293.6 146.8 9.1486 0.0009812 ***\npre 1 577.9 577.9 36.0145 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$`Type II`\n Df Sum Sq Mean Sq F value Pr(>F) \ndrug 2 68.55 34.28 2.1361 0.1384 \npre 1 577.90 577.90 36.0145 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$`Type III`\n Df Sum Sq Mean Sq F value Pr(>F) \ndrug 2 68.55 34.28 2.1361 0.1384 \npre 1 577.90 577.90 36.0145 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$Parameter\n Estimate Estimable Std. Error Df t value Pr(>|t|) \n(Intercept) -0.4347 0 2.4714 26 -0.1759 0.86175 \ndrugA -3.4461 0 1.8868 26 -1.8265 0.07928 . \ndrugD -3.3372 0 1.8539 26 -1.8001 0.08346 . \ndrugF 0.0000 0 0.0000 26 \npre 0.9872 1 0.1645 26 6.0012 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$`Expected Mean`\n LSmean LowerCL UpperCL SE Df\n(Intercept) 7.900000 6.396685 9.403315 0.7313516 26\ndrugA 6.714963 4.066426 9.363501 1.2884943 26\ndrugD 6.823935 4.208337 9.439532 1.2724690 26\ndrugF 10.161102 7.456182 12.866021 1.3159234 26\npre 7.900000 6.396685 9.403315 0.7313516 26\n```\n\n\n:::\n:::\n\n\nNote that the LSMEANS statistics are produced using the `EMEAN = TRUE` option. The `BETA = TRUE` option is equivalent to the `SOLUTION` option in SAS. See the **sasLM** documentation for additional information.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-23\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P broom * 1.0.12 2026-01-27 [?] RSPM\n P car 3.1-5 2026-02-03 [?] RSPM\n P dplyr * 1.2.0 2026-02-03 [?] RSPM\n P emmeans * 2.0.1 2025-12-16 [?] RSPM\n P gt * 1.3.0 2026-01-22 [?] RSPM\n P rstatix * 0.7.3 2025-10-18 [?] RSPM\n P sasLM * 0.10.7 2025-09-28 [?] RSPM\n P tibble * 3.3.1 2026-01-11 [?] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::", + "markdown": "---\ntitle: \"Ancova\"\noutput: html_document\ndate: \"2023-06-01\"\n---\n\n\n\n## Introduction\n\nANOVA is a statistical method used to compare the means of three or more groups to determine if at least one group mean is significantly different from the others. Please see the [ANOVA document](anova.qmd) for more information. ANCOVA is an extension to ANOVA.\n\nANCOVA (Analysis of Covariance) is a statistical method that compares the means of two or more groups while controlling for one or more continuous covariates. By adjusting for these covariates, ANCOVA helps to reduce potential confounding effects, allowing for a clearer assessment of the main treatment effects. It assumes linear relationships between covariates and the dependent variable, along with normality and homogeneity of variances.\n\nWe follow the example from link [Analysis of Covariance](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_glm_examples04.htm)\n\n## Data Summary\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndf_sas |> glimpse()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nRows: 30\nColumns: 3\n$ drug A, A, A, A, A, A, A, A, A, A, D, D, D, D, D, D, D, D, D, D, F, F,…\n$ pre 11, 8, 5, 14, 19, 6, 10, 6, 11, 3, 6, 6, 7, 8, 18, 8, 19, 8, 5, 1…\n$ post 6, 0, 2, 8, 11, 4, 13, 1, 8, 0, 0, 2, 3, 1, 18, 4, 14, 9, 1, 9, 1…\n```\n\n\n:::\n\n```{.r .cell-code}\ndf_sas |> summary()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n drug pre post \n A:10 Min. : 3.00 Min. : 0.00 \n D:10 1st Qu.: 7.00 1st Qu.: 2.00 \n F:10 Median :10.50 Median : 7.00 \n Mean :10.73 Mean : 7.90 \n 3rd Qu.:13.75 3rd Qu.:12.75 \n Max. :21.00 Max. :23.00 \n```\n\n\n:::\n:::\n\n\n## The Model\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel_ancova <- lm(post ~ drug + pre, data = df_sas)\n\nmodel_glance <- model_ancova |>\n glance()\nmodel_tidy <- model_ancova |>\n tidy()\nmodel_glance |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n \n \n
r.squaredadj.r.squaredsigmastatisticp.valuedflogLikAICBICdeviancedf.residualnobs
0.67626090.63890644.00577818.103861.501369e-063-82.05377174.1075181.1135417.20262630
\n
\n```\n\n:::\n\n```{.r .cell-code}\nmodel_tidy |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n\n\n\n\n \n\n\n\n\n \n\n\n\n\n \n \n
termestimatestd.errorstatisticp.value
(Intercept)-3.88080941.9862017-1.95388496.155192e-02
drugD0.10897131.79513510.06070379.520594e-01
drugF3.44613831.88678061.82646477.928458e-02
pre0.98718380.16449766.00120612.454330e-06
\n
\n```\n\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel_table <- model_ancova |>\n anova() |>\n tidy()\n\ntotal_df <- sum(model_table$df)\ntotal_sumsq <- sum(model_table$sumsq)\n\nmodel_table |>\n add_row(term = \"Total\", df = total_df, sumsq = total_sumsq) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
termdfsumsqmeansqstatisticp.value
drug2293.6000146.800009.1485539.812371e-04
pre1577.8974577.8974036.0144752.454330e-06
Residuals26417.202616.04625NANA
Total291288.7000NANANA
\n
\n```\n\n:::\n:::\n\n\n### Sums of Squares Tables {.unnumbered}\n\n#### Type I\nThis can be calculated using, the base R {stats} package or the {rstatix} package. Both give the same result.\n\n##### stats\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::anova(model_ancova)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnalysis of Variance Table\n\nResponse: post\n Df Sum Sq Mean Sq F value Pr(>F) \ndrug 2 293.6 146.80 9.1486 0.0009812 ***\npre 1 577.9 577.90 36.0145 2.454e-06 ***\nResiduals 26 417.2 16.05 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\n##### rstatix\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndf_sas |>\n anova_test(post ~ drug + pre, type = 1, detailed = TRUE) |>\n get_anova_table() |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n
EffectDFnDFdSSnSSdFpp<.05ges
drug226293.600417.2039.1499.81e-04*0.413
pre126577.897417.20336.0142.45e-06*0.581
\n
\n```\n\n:::\n:::\n\n\n#### Type II\n\nThis can be calculated using the {car} package or the {rstatix} package. Both give the same result.\n\n##### car\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncar::Anova(model_ancova, type = \"II\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnova Table (Type II tests)\n\nResponse: post\n Sum Sq Df F value Pr(>F) \ndrug 68.55 2 2.1361 0.1384 \npre 577.90 1 36.0145 2.454e-06 ***\nResiduals 417.20 26 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\n##### rstatix\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndf_sas |>\n anova_test(post ~ drug + pre, type = 2, detailed = TRUE) |>\n get_anova_table() |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n
EffectSSnSSdDFnDFdFpp<.05ges
drug68.554417.2032262.1361.38e-010.141
pre577.897417.20312636.0142.45e-06*0.581
\n
\n```\n\n:::\n:::\n\n\n#### Type III\n\nThis can be calculated using the base R {stats} package, the {car} package or the {rstatix} package. All give the same result.\n\nNote: Calculating type III sums of squares in R is a bit tricky, because the multi-way ANOVA model is over-paramerterised. So when running the linear model we need to select a design matrix that sums to zero. In R those options will be either `\"contr.sum\"` or `\"contr.poly\"`\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Drug design matrix\ncontr.sum(4) # Using 4 here as we have 4 levels of drug\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [,1] [,2] [,3]\n1 1 0 0\n2 0 1 0\n3 0 0 1\n4 -1 -1 -1\n```\n\n\n:::\n\n```{.r .cell-code}\n# Disease design matrix\ncontr.sum(3)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [,1] [,2]\n1 1 0\n2 0 1\n3 -1 -1\n```\n\n\n:::\n:::\n\n\nWhile not relevant for this example as the disease variable isn't ordinal the polynomial design matrix would look like\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncontr.poly(3)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n .L .Q\n[1,] -7.071068e-01 0.4082483\n[2,] -7.850462e-17 -0.8164966\n[3,] 7.071068e-01 0.4082483\n```\n\n\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel_ancova <- lm(\n post ~ drug + pre, data = df_sas,\n contrasts = list(drug = \"contr.sum\")\n)\n```\n:::\n\n\n##### stats\n\nUsing the base stats package, you can use the `drop1()` function which drops all possible single terms in a model. The scope term specifies how things can be dropped.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nstats::drop1(model_ancova, scope = . ~ ., test = \"F\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nSingle term deletions\n\nModel:\npost ~ drug + pre\n Df Sum of Sq RSS AIC F value Pr(>F) \n 417.20 86.971 \ndrug 2 68.55 485.76 87.535 2.1361 0.1384 \npre 1 577.90 995.10 111.049 36.0145 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\n##### car\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncar::Anova(model_ancova, type = \"III\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnova Table (Type III tests)\n\nResponse: post\n Sum Sq Df F value Pr(>F) \n(Intercept) 31.93 1 1.9898 0.1702 \ndrug 68.55 2 2.1361 0.1384 \npre 577.90 1 36.0145 2.454e-06 ***\nResiduals 417.20 26 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n```\n\n\n:::\n:::\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndf_sas |>\n anova_test(post ~ drug + pre, type = 3, detailed = TRUE) |>\n get_anova_table() |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n
EffectSSnSSdDFnDFdFpp<.05ges
(Intercept)31.929417.2031261.9901.70e-010.071
drug68.554417.2032262.1361.38e-010.141
pre577.897417.20312636.0142.45e-06*0.581
\n
\n```\n\n:::\n:::\n\n\n### Least Squares Means\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmodel_ancova |>\n emmeans::lsmeans(\"drug\") |>\n emmeans::pwpm(pvals = TRUE, means = TRUE)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n A D F\nA [ 6.71] 0.9980 0.1809\nD -0.109 [ 6.82] 0.1893\nF -3.446 -3.337 [10.16]\n\nRow and column labels: drug\nUpper triangle: P values adjust = \"tukey\"\nDiagonal: [Estimates] (lsmean) \nLower triangle: Comparisons (estimate) earlier vs. later\n```\n\n\n:::\n\n```{.r .cell-code}\nmodel_ancova |>\n emmeans::lsmeans(\"drug\") |>\n plot(comparisons = TRUE)\n```\n\n::: {.cell-output-display}\n![](ancova_files/figure-html/unnamed-chunk-14-1.png){width=672}\n:::\n:::\n\n\n## sasLM Package\n\nThe following code performs an ANCOVA analysis using the **sasLM** package. This package was written specifically to replicate SAS statistics. The console output is also organized in a manner that is similar to SAS.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(sasLM)\n\nsasLM::GLM(post ~ drug + pre, df_sas, BETA = TRUE, EMEAN = TRUE)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n$ANOVA\nResponse : post\n Df Sum Sq Mean Sq F value Pr(>F) \nMODEL 3 871.5 290.499 18.104 1.501e-06 ***\nRESIDUALS 26 417.2 16.046 \nCORRECTED TOTAL 29 1288.7 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$Fitness\n Root MSE post Mean Coef Var R-square Adj R-sq\n 4.005778 7.9 50.70604 0.6762609 0.6389064\n\n$`Type I`\n Df Sum Sq Mean Sq F value Pr(>F) \ndrug 2 293.6 146.8 9.1486 0.0009812 ***\npre 1 577.9 577.9 36.0145 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$`Type II`\n Df Sum Sq Mean Sq F value Pr(>F) \ndrug 2 68.55 34.28 2.1361 0.1384 \npre 1 577.90 577.90 36.0145 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$`Type III`\n Df Sum Sq Mean Sq F value Pr(>F) \ndrug 2 68.55 34.28 2.1361 0.1384 \npre 1 577.90 577.90 36.0145 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$Parameter\n Estimate Estimable Std. Error Df t value Pr(>|t|) \n(Intercept) -0.4347 0 2.4714 26 -0.1759 0.86175 \ndrugA -3.4461 0 1.8868 26 -1.8265 0.07928 . \ndrugD -3.3372 0 1.8539 26 -1.8001 0.08346 . \ndrugF 0.0000 0 0.0000 26 \npre 0.9872 1 0.1645 26 6.0012 2.454e-06 ***\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n$`Expected Mean`\n LSmean LowerCL UpperCL SE Df\n(Intercept) 7.900000 6.396685 9.403315 0.7313516 26\ndrugA 6.714963 4.066426 9.363501 1.2884943 26\ndrugD 6.823935 4.208337 9.439532 1.2724690 26\ndrugF 10.161102 7.456182 12.866021 1.3159234 26\npre 7.900000 6.396685 9.403315 0.7313516 26\n```\n\n\n:::\n:::\n\n\nNote that the LSMEANS statistics are produced using the `EMEAN = TRUE` option. The `BETA = TRUE` option is equivalent to the `SOLUTION` option in SAS. See the **sasLM** documentation for additional information.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P broom * 1.0.12 2026-01-27 [?] RSPM (R 4.5.0)\n P car 3.1-5 2026-02-03 [?] RSPM (R 4.5.0)\n P dplyr * 1.2.0 2026-02-03 [?] RSPM (R 4.5.0)\n P emmeans * 2.0.1 2025-12-16 [?] RSPM (R 4.5.0)\n P gt * 1.3.0 2026-01-22 [?] RSPM (R 4.5.0)\n P rstatix * 0.7.3 2025-10-18 [?] RSPM (R 4.5.0)\n P sasLM * 0.10.7 2025-09-28 [?] RSPM (R 4.5.0)\n P tibble * 3.3.1 2026-01-11 [?] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::", "supporting": [ "ancova_files" ], diff --git a/_freeze/R/ancova/figure-html/unnamed-chunk-14-1.png b/_freeze/R/ancova/figure-html/unnamed-chunk-14-1.png index b124310ca0227bbdc157b2ef0eac57ad841b1605..04f56e9d3a38049efe942842374fc7a3e0842e52 100644 GIT binary patch literal 20930 zcmeIaXH-+$`z{(pMP*a=R$&VQvi&L2RRlyzR8$m1I!G7UfYeAYA&Cu85U^0BM#0c~ zCm{)ffYO4r(2~$0BoH7#NJ5eu_W#~-KHWR+Ipdx)#yw+yAZx6UHRn6aTb}2A=XzuQ zz)0kn)G+`6Aad{SElU94&~*Snu=U6R{x{%*x03jmqfhVJ2Lk{yar=J)$qO4E0Kgf* zy<0b|BeJQ4$n2?5$nFL%+>_>~6@K@o+TAA5ktNRWY{|DSq`2MZm_r9F07>AAr}(P_?ogpp(yDe`K|PE z^%#L*<>=^W^|5kGB|v6|TErS~*FqKe#7RZjTO_dEuq3e~2&Ph}T8ZFX3H*;Su~=EloBx_b>mk zfm}-|ouFF?q33Qk7fxseP@g<03TGWfW%l=1ipbR^q9dZ_A(@mZZ1SjU{V zx7D|H{LAvNJxU09xZI?<9a%9p{+ZBcW0i-EUk1;nSfpiUN;EXsMg8_?SfZ)4l|wBD z+h1s=8#*M$*2J#GBD7P%Dt|_qny%*P7uOaJ2EWp7`0}8YRu_+R3TGB#YD+Fozo=)! z)>r6B(itM{_W|pdO?%)aH;UYV$NF};(K>x{S(%y=_Bk#NliU-NXfZrx|NweZ`kmTU*vNq`que4lE9w^ z^{W&w?zFNM6cTE?Jh^2|AM7e1Y;6~k0s}pSUiu7z&>*NuyP!>h!vO=l-u5&^^whau z0!1xl1KJCtzMfECPrWoAO)W1-6;{6dhc%*nn7w%Dik`VCzdwOTOBY$yx}L7!Kz_3< z7|L3T82H&oK08Wa}pEtcoSx1S{>tx@)33}0XwJ!c6(m^B9F zz9@zXnpa4+pBc8)zB#!5{`F?_#o9s)y`2EWK2Eg8EZ zhYfzJd`(o$`nk{1-N)=rBX}k-kC!#(p^92X86^nO4w#35p$u9fDPo~PowPeOGz8nk z8ET7JH(FqI9vj)>*RO|+{IMQ`0CHxvp~x6*-{goTyt^^;rkegG(rgQ53j%?R ziyIq3Gi~P92zxBWjCv*SA+8+-RNI*#}#`N7R=pHUU){im72c&RwRcCd7@^o7N_+_%$|{7fm`k$veHf z_};WD_vznQ5Fw=9UlB1eF#*{b4>szWbQ6|Pd7!w8X2|aQEE^-ErM%m?2cIs6wY{oN z&5Z%rs)3=O8XtD!)qvy{Zy+1*MpUC_IxG>fb3!vR(6|!q$G3(IGzzSv;>(*E{Y*tf zY%J?WnWHLyKc-eKzcmZLpQ}%S8CVTfn@UE7B*GhZJ~tccov<$NN;YFdSk*TwD~QjR z`5_7QnTliN9tsZIcHG~0WNnuWiu80;L*kIw!>4Nt?HW?n2mR1wLnw%0y5h>0FhKY6Leq9!3J4T*hm8Hb)7Sym=<{?6UHLZ+gxq7p0E% z?KtX{9>kQyj$HD#MHIYP(Rh?y8vePP?-2g{1AVqj?dp;;1IA6KFo2#z%Bv_^?bfiE zO_);AWygCV+oINT;a`NF$=3z41jJmnUjA+(S4o~4(4wqc5=kxQp4uvCaw-F1N4stl zkCA?#6}hNTp<6p>1!%RcH=C>+JbPUeY~RlB&*8RM1Yq|iy6f$}xvGUi9KqnF`Jcyw zbR<#LzV_9#%`6rxaPAI>7M~2-TZQ&jPHwe}_Z~7V&Cd@eARGAq(!Mfm7nQaywaI)g zdCa6$(EMD%+b8?2Svim#me+EphTm1Lpv5B)^#sCMijNS3GUb4*q|#g0AL56G#tHhE zh5|%9SIzyxu8SN$ius=rTmI{2|2K29|6R!9-_`$j_5a;4{|&qsDGn{27*&r9Tj(5Y zP|5VLuP7+MRC~{VvT)Dzt+{N^DU_>~f`%i7ZLFF^Dd~vv8>7~Bha5DJ z4%~HZVdS|BuZrNF z5YFNe>E9T=kP)J4J+JXrvG1xomem1^Ca?IeJsfh#+-~ME=ixM)bYp#WJ?=EFl1GBI zY&IN{j*@hSFw=>F^LA!FN<-HCF3it$hgYGNieWkoIJCaxlWFt_qR|!1@hC*E-QLc% z?3U`3T`vfC37o9Q!=B=1=d_{g^Elp<$V(ofa}3~0uL(A~N1X&7{-JWx7P8ds=rUFS zp5f12tJ+s50aqXN#tIBmSev3L10$0l%T=m!H*twTrZS98LbR9Q36l zx4;Hq;&8N5C^3@Zyz21}kyLWK`2dE2$)Y*C-nZv`0Y&zF7ci(Jw>D-sfDKOCk=8$E z;r$FIT*7zn#U=hk%*LKGQ&mp$R^pGW=ns!$yMvt`CfX8K-I)sa15T~~TpRydJV#ed zMM?lMRaMN9%r|qLiod?)ZgD&F?9lu9=J5!hS0=Ssu!fxL>utB zCe%Yi!_zIheSCZl@9ooEfG|~lAAa94LJx}X6%TW!6M^=+t)|OS5}Yjs+|K?=Pl6X0 z1>$ji;Kw$X5L2&;t^VPYT5dHbwJwQ4*+;=6U#I^2O zCheoywa7g=2XqsQ;mTZ!i!v_uq!}jdoY|+JB8qDeL49ss(JOD?FjJ281aN-%_OnZ2 zO%U!1FJ-&w3fEnwHacm++!a0ggkaRck6m8H;morn-U_f~b=hnWZxv%XdT}XR zTh_cHZfKtYtH0o??B!0}N=}Dv71Jx5eBIsp=J3$HJ&J(^FaUQ7{NR*kUd7l5a&7z& zuHwn+WJnlf^dh~?7qdRY@wHT{Cq$13;jp#3h?JVe{C?)3P~lRK)^WbT)^CDjsr{X@ z&vJ7+{o*T`ow#uAB@2f0J<3xHEel2WRQBpRvI|h+kp8$8RDGB9t-sBWn z94|1P3^pE8HcOU^Y#X=BzIf1z|IGE1r@Gl49-;1xZ2d5t{M0^wNc~H%N`!wsZ~_1O zE=JQ#)nUSR3X@thaTMG)0r@IS;_Ja9!2UKw%0Z^}mq>%U>mSt9vUV}9oZZbi{T-Rl zdv`p79ZVQDQT+YXeMgU$HxU(u4+kwts$Hb=X+T|RDVGZ;bhSgQqtYIU`sipOt|(>E z?@q4r;l{ToCiY&2?SqLq&DjZru{?{=`ur8&eeSQlkgs~!u0Vxsg9cCnO%my7rzhS>)-poi-W{S!=QKL=MXxQMX$raF*=)c4tm%I!kMzKb#bE>R z;4xhky?7ELZz+A>sjpZdo%1L3XfM(BGXKg+DVzMfbIb}&KCIhG;N`XhC z0%nUrx>afjYadt;H(u3M8>=ceqPNN?9(1BfJT?hxgjxFmj9h{{ubaX5sLeue`wC}S zhw!dUN+G!Wiq7?zx8P|>Bk1OsJ&!Zqw~e{y&t{oc`k{&1<7pm`(k&-3n1b5Xj>_SX zx_%7z!&&uA-JLJ4c0Njgu@CW6hHBCp{*47r#){f`2=BY+e)|*bBK{M z`mQ{H=2-p2V&#j{8mC-1GYzOY{o!RP!}I|>Y^6;&TyAT{TqX!jn5=oo zUxsnKiyl)OuXdHOIf_m?y;2N#WYRrchY-P{_uyQ1wE!kVEvyW}`qCxqR8@gWvFtY0 z>sIHy{YZ4MgFXW(7mu5@56hGdR4bvRN%P!w=29GTdUI7 z_~tH6Q)^U=X3kk&UcRrG8u%f1@108v;5W-J6@iWtzByo+y_5^OeMf0j+a~H2u!NuY zDBJ6|&Zut;pQks+#&**n-=40lu2R<4ic3q&i>uRJIGarVMo*XSP1oFa|69^n);45R zBs#yTtiY{z>lFLG^d&vmMUxz7{-(7Yy^{BZ1k*P?E0r8Qa)VO=9ujFpr1<#y?u@xl zh$l7qOQ<}|wp5z?kBL@YCCFbLUnsCpL!D=swXnN1S1CeHL0!`O06OH_mfA7 zvV_aE9BA~Mhxa+D+7k|?4!6ph9uDJV1oO3u?=%T)$I|8`%YV}naZjXqAaYt z%)bA&z~g1vM)n?-YuhtK_k^*nRE34E-Zf~7#q!+=PBcuf=se#FH=di;?$wK!u}^PU zgKS-Vdl2AgQ6OB1lynZ~E*XUMdjW&SoqfY{g(?_&!Cf-yCoI!Q!Eiw%9e~)a!Gt>;u6V@3X-9DHV_DF54e(No9^$mkF$ANSB9MvN)^MYrUzT~Lg zMM?!`gC(cefP9#nJmm(#ad!(F_3ZB369g1wFBPu@Y#gx0It46DY0xmUxn9K0jmLeX z6(1&c&fC$d5{;P?9tv1mZsk<<%9w^_f51V2Zp>v)OiE&Avr(`b0#hide!91By{CM* zdAGn98%_AIW@)SKyQZ5Iz9;j##GJI%>!{vD6UN#HkKN;_zwbf#D+h0js#{I8B@GB0 zwanzI^8)xYvZMLi_HmHFBi$5=)GpGU2%a#&@8Rdl!+Fb#4ceG`k54a}hcVP&F?8PU z0OB#oh&{1jB8zb+Ld-xaDK%5eXwvq3JWc1*BwK)b>)QFgsQEqLE`uNn)t6V{-C>pG zN|OTCS5|ur;r1D|E$_kr^6_Ea7E2KQ)7r0(EBMNZJ!AGTL^7Aj3>A`f{PvI82|JF8 z%_o6JB;~;LE*j_8Ay<9Qasn@e;nwW!MEW4D&c;i#A<8{>s5(moImqOOMhW{4q*d+j z)f-(T@OU|jx3Y2y$cmdtq(o(P>1|+?U0Yhy_!?-)3u;#2Zgz8c3miVX4mL`HES5`_ zUFA|OiMfIR-QvsKlOsU-jUarE>7#;mu%mgQclPxoX@laz;N~zh`7rJNcBR zck+0goQBsL#q(_#E%GxmWwXAAI}cG#0+zaM4^|u#>wt3x{lP}&$#OC6l_J1 zsHnpwM$+*WyIDFfUsI>bYJri2aa}Eq>km7Su}K8R#2`vNAYxloP-$2*U_tU2AWU}Q z+5ytWkMK&=W*SivYpt|hr{+3}jP6lhINv{BMb`@nz}Gms8=5%#BMP4hgoKAzKHS(F ztIjt{43aBerQgTrXh^&4Us7dX-H4irsh8HO!+oj-_j7RLHL)#LA_8Hm6`K>cHHYV= zefG+6{T2D<4+)dnfAw?@qa+u?d{qf{=*<3Pp-Z7nNr~KT@~QtspZAY zlpyyzlCVzle!Kg%08y7u@Spa3xhl5n*X!-dPBwH%;Gpqrsu*v<-8J%w|0NIR@8Hwg zjJl6vIuSg@<3@Nd844V-_1+P~sFkKU5K@gr*ty(38tl>9tuEqz7@qap?2+icR}cn# z07vOKNkY%-+v&A5eA1K)nvle9m#tl3%@TN-Sj3a<-fqG3Sqpo-x@o+gaG z&;{}wf-A_6IB77gjK-PA{b8|!7(Y8 zwvz)FAFc8S2k_91!2~ce=7JRwM1_FeZ-Z^NlB)GYdV3aL(yDu-w} zAY|f6*FP}KF-3ja%IDfj3y*mNXHR2Ql7OxkQse!F<#vCy2|>5I7iv1uq|k5mJm`mZ z8o%eN#O}#~@m)M@xr>hBc3`tLAh%#b_*&#Wqnw~$3=CwlzHoKOd{lKGH`^#L{l07J zQ+RsgIq38TQN1!^H=Mqf%b=Ew_)8zFKil;-_lhq~r|Wd|GBs_Y2J{ENgA!+^AXf}N zd3j;os_-;)MI5p}V5);t>^?^QAE`3=mka)v3;vf2{%>6H|Aa!We^u=N=PLHssnovW zts{Iw@{TD&=#f$R8r20qVntSbba!ieyIxzfoT^;m$gfJNrJhJS4URVUw$@rA);HQl zNyOTM6)6Ee4r3F!9`*>YFCSJmn=@3;D)AXN-Gq{3yKSsgGgZn3K3iNsVC3j2xssp9 z-gY#j#qabRn+X^eU@)cT_qp#1Ls)fI+Pvubu{qGgD62zpZp$9U7^sJl9ir=VTc)os zpCZn(v1(-)gmG+Kn<626>L>BqMlGK*H{{cMe_W4j`9ImQffWb19E@ZdQ;;@dlGP5$ri>6Gdsrk8!V6{3nFvLowO+2+JkIMj*!Go1mLNIKww$EYQCADszXD%pLos0b5UXs-&y~#!lH7wWg1f zwp;i3sggwSNFaDvlSW@@m#Vd1iC%v$`S_nmpV|$LVX4YoDwX6eG^W$^hdH=&lovOGMW+ zw`XM!vsnqHD~@pRR=XX)niNp<;+c9GivvoRG`1%|japdAUtD0Vm)P}FQ6OJZZ8(wH zrU=TZLYN$yaL^_)CnGtSk;%z)%M~&iGXGv7?$O$}Tv4%>;|2VCtghaF>JYD39#QxA&rRdx)~8Q=CqG%PP^nVi zku!prswc!8lLW<5AirIA*O1TXna_q|au8BED?Z?lQQ_@JtzCrWctEkx~uFMbE;~qpyov5Id?(GVtcHf?GY^HS@{4G5a-2;~6YJ||1QP}PS(VwiC86qoEudUXA(TB` za?_ZZ*14U$$6AzxE0MA~=W9bDG3Ug%+zdRHPZ|hgs*7g^&hAhc7Bn!RC&Rjy@pyH^%!R zEf;aHn(AsduQB$EnYsXHN{vS;zixJNaL|ct|7TEQq6i-WASZ5=N$X96b20S((2f*K zx9uE!G_4<+uBgXglT^W!+#w-4=WuJ)JsFV(+*<0XXfa1mz8Ecn^=7E_HJ_$+6UtbJyq|l0_MkHuXTU4? z&<|ey^jx6T&Lpfg-3$hrUGaNuX?iUzfD}Q2fI3!^CCXKL)Z`f4uEG#%E|qa?Se|F! zp6syIJGgpDswgxQ%^UO-aCJtf(*`|>Z?F#y=5^(3PN?zi>@B0!0&n+1$qxB$T2x!5 z`qh@qxDJ~KY_=SC-N_=XBW0uZP6#b)SzCBml{XnHSHGOGs{Ln?RZZl!L}Js^84F2| zP3hsShk!>Kn!$#6?Z4D=^aD0yuij!p*dl%|tK%C}xojFy<>bwVC$rMQ%3)go0j?Pa z`mr>8x1fw1zWd#m^eKD~H=+yB{W5_69o3A#7^H@W8Xmv(*A}=?`Re!B(g9 zUdS3Amhhvm)lEHkd05*tM!XV!2^#-G=)+Pe+X>wc`~0zNBaSnkU#7E~Py|TwX4vO#{9~kmkxe*9h-#!=O+!)cmr%9HLK^wZz+QGxT z?(_oGwj72u`{s6_HO42YQG1>UO8f>0dMhY&PP+8Lh{U^ zXHxG6o+V3(US-7#xYFv$JMFNYmot;$dVonn`aZGAb3RWiE#xEg!jA*bmV{-bw4T^t zT}+b?G$fd2bxW$d5L+UL3QVQ(TMxsICG;qS{E(|_ioR(o_G&d?pfAqg9{ssO?*Z$|f&H^0#C!T!wU#A5A_^&|w--rI4 zh<|s)v;T#v@uQe5K&8fyN|K&#%V_Gyo7>x;y!oSVdpj@S&0JGWjZ)5o!l8aG%SLM( zkD75RJrXp!b^Vc1X;IOAL|*pkF1=yZaleKVrH>y!K0utN4?_{>q`{!j4K|zC>`VwPbl$1+r zp=sKWlefze@>zghW?y@UsCHHO4OD}!3B-gp;BJ(193V7daElzilcFlJ? z(aZ^5`BxkfbK9WwBvZEDfw4v@KT%TKka(n=REz@wh$*OokAH_4eh0yVaIDi zYq^3^c>Z|e9BQ6))F}qD#HfUlPhN9-xS~E@-s_H8rk?wF$*dpRvm7Zl1s)50$!-WaX!4=M<9NU>$Mt-j!8T1& zW~71T71QaVd>#no_BAd>x6~Cqao4JJ``AE0t_6=hk6WEYJ}*od^@noKbrl>+<}Ab{ zYju3Pak}2wY46u@?J34sN*)e(ztUBYR^CBZz-Vo6xx#f?YT*Jtg-uWTU)qs0qIP1b zULDYq8q#DSCXjsOj&;U4|B3xsjntv;sPLAjaTp``|yO)L5L zKDx;2Z7Kj|r+$4gmpMOu{4e4aI#H;rmt8_QJ^aT>lTIfv{_IfCO|-3G@~LZeu9N4# zglr7o%QrH0ZrOPWu8^*!!0+q?Ze~mmUYmE6rX zUv*Pw>^hUp8*b?UnfcLHlzkF-7U2=V&DMEXre}A;2Rk!HdflGkf5Ydz5@fUljQF zcYzpx>6A)`i?B@%_b%u|0NLWgwZHPZ{RoPG*OU6~V) zA}KHGM>10Ku6EP3sPUzk5roST`Bwos4C>?_vM2YG?KSox^m`qI64fu;_8g*;Jq=?SO0RFWxP7C>WMoRo zMIWCHRp+t?6L?XazI=G3uebM3DKs_REb9t1QtN@{IIr#td=$(dWcJQWGudC)WM;RI zpU2qwcwM)&HpMK60nyY3ERnm}q~mHqlm9ynOW%pVe|lj@Dr#lGGfRfKRChY6e;`1| z!m0l8=J%5lu(I0TR@&C=Hi$b{t&M*r5hn|~Q&^47tS%`KIyh;;+)|`fBpV|8poY5s zkzl)F>N~h=*xkh6gd4DaTGBU<6lYJB@AQ)Q%(yC9zi9t)3Mfqi@8F7A^Dn!%gA3Kw zi!py+mNR92z94EbM0^=WdJK9;^vCK?=r=@LjK|ULtCC=M(LRlFIrr;K(q+ME5SEFa z3pRw+GP@$&^0=eTBk)LZ7&sJEJ#y&0MX8#;qfv0$-*&2wSx5c*zK@z7fGVUrGuzIF zn_V!iKNZa#he%{;(SL**m)tIF5XtYjy%0>wbwbj(uE@qq z#4Emg*|S$~ZBML?5^;y#NN7q$40{LyIcHq7xhr(p92a|n@Z9WeVPifRIk`3(NwIQ} z2~$>iEpgILB@=sI<10Gh_`Me)I9fKiBiw-VIcSftTqAP|IJ^o90SVjLWb9$ zqZ-D~O;km1q%6U~)6MncMW1m$&&FDVuL{+hd2WC37#Gj$EYBJtu}}tH>9(qtU#_Bj zHdac`Oc&eaExv??;XeCA8aC%nS=o9SrVLeACDQixrTwmy_LkdITiZQ|g0jCKXOAYl zkVY4+&Tf8~y?^><3YLrp^2nI#QXnk48`%*|a?lHJGx?ar9M@4;^`5cu$q%8wR6`Zb zK*XlF^HX*p^!A7&kasJ^r;buZH#69t9!>jPSn;|?%kMY&TZx(0jmFNyRe^ICUR_1G z%@tgopFC%-{oOg~Pwcn$R}0r@4b z| zcc}61Nkz{#wS+7T)*~`TndhB~&MGMxFmL?m!oLiy+`jy;k_giV%vo&NKm%f3pVak~8{c?+8YU-aJyEJQtUyj!p&kl%l z*+89*f@*CKBEl;nJEu95q_@rwH)rO=aHoTQuF1;WQaxUo6UJ6wi}b&D-e)Rs`Dom4 zG6U77`NN$j6AD^bE64ejue>k1F!b_wS5C-vWf3!H@h)C_Xk6eC`S0U?TB6_9o=x}< zmE22CL^6KbD|gRbAVlDXaLo}9gHbP#d>P-)ei&zP7ZT59?B#l8M_5a`o?Ur+f${NP z>brozU1ob!!058Yx3zXmEomNzwspAT-Cedx5NcA3nfQ-d&eN{5m8!+Z5|MCdU!)P0 zMFw#@O<8qhVy){80n4J)jm*hm=MZf@SdWRAYiHEefhS|cm6S_L6t|j6YA(MLx7^-z z2AY-IR)_2*@G!yDx>TFM*PD~HOPlPo$;8?>QI&H}vB={kpP;m}m1@b-)JwBqqsI5N z;d7DOl-u&JitaT+og2AC9If7~EBnHCX*(5{1#;oAQs71}ULoMMVaaOM?{PF_hc*m3 zB5#zmoF*E`d}~)wQFHoO=#7#a?JvY{IyuyR$pkSY+yn3NH?Yt8&j*4&Uz#F_pMZNC zn-rH^3*tA<-hmnt6L^<2FpmDaY-0%vAQKH$;PdmntU%{2$tdMF23G7#DGu{UrW z`62%5D;PWUZvfzTr~O)hqR;o-gVl!Bnwujnjs#}+Sp(Z;y=nrFShxg)Do5QYKdw3M z-x&L=H?O2QLBexqt(m!Y1{J-r_kqvopuz&YwLVNJs4=H=OJ$|7deWNYs424t`cU=S zHNQKpwCKDS5VfYS^v?3+nHq{KY`#+={qG!P{!R%=MQa z#zAJ|!!URsC^*@KlvVdx>6LXU0jVWOZk)B&==QpMm`R_$gjh;iQJLXP?p= z!<0#ksblcP;iKdVL8Nw6faS(Y#z&wYa>?@$+}Xgpv(4}ICTfe+?sH+-AT?Mva&rNZ zG4Q$P4OX8vrdDy;%|!HI*v;ZQs}tem3Aa*S*y`A2;4%4I+m4c9%Y*fXON7i(yGz4r z7NzLeol?1ub8~A^D}+#2gTz#4jkRJUqGED>NZ}Vjuv7Y=#pDQsx;n6(O@k3%KPA`oezy5q~GcnBeAJkqn^VE zWni?|tOgF&qIEndTz7pd-#O}L`SFv#>21H(FVtf}f3;a)G2Sz;1!K2X2Qar?m?gKZ zTp)AJfjwHC7B}GSX0j!uVvSAIX^1hYk$c61qif~@wptih*rrYBmGPdeEbHFEHeZw0 zq-t$`HxS|InNzGGSHO4fozC0qOf?}zdoZEIw68AZ^gZzNvFYnKt#r$YDAeRP-eT+?`~$U>51J&T}_8^L&^4pB2+ zty!u{fJHqu()C|?S2X^dgpLUOWtyz=@bb##xKM%&4b zz$0!$y$W)f&3);JiJYf_ytV`MQ=uQ-1s-`F*OU%vqk-1wvqc+oBARiOj$8NRsQY3)A+4F0nd0A9m^lZ z^f}*Zu8!-nMqy=sCSIp zL0O~nlM{fMm;CSD&h~lOWX5P<%3G)Q=InCoJObN2QFVSpHIPEBJ_=~#6Eb2bb3s_L z(iy8*c#ka_BXbNhDsXG0Gd00Z#kKkpA+HmqK2 zD27*XV;s@%#nV-DH3Qo6W<&CA{j(ZZ1~4XZu}PtTB2DTr;0$y216fReJsk_=P+uWB zQsD+?w2T+(mvxtQfmbPm!Hg&MWk-`Mb6xB$R@L;48+B&8l7T^w;fLWoi;*lz8E~(v^5sZQWlKJzNO&!) zA&{bKmB?UomnF^Oxq;(H)6Jah>qmqwO~ud)NpY}&z#g?Elk=qYgrUl4U+?<-H-qJ( zdh}5t4D9ZtDrk1{>gtBD5pjLKYXtj7{ALN$B|RC*^fHu!k|&hDdVQp0?MuB59w?+nlrF&-dn)DqA&%O*uc{;Pv17QAC)q|FC> zI}277&Fe+Vg(tiWUt1<3cjN;9#u8ClzuoRBOoq&jYp11n&WY!-AZC#fRG7>zLS$~F zlJ~ULgQ5Q8yZkMY$<^>-E}y^f{dLjCqK-8eu7F*iD)Y#i70=5KD14g{MV^j=gBat| zKga!lOeY>EI;Hl3H0D$Cll7Y%d`A{rJ}w&9ou9q>gmLGx@=^J?(OmTjPS%LiX>XBqOqgX zLt1MwdOLsAb1uFago|GZ)uDXD5}#7N>^>*!%?`hZ z0cWRC1%C0hzC`HEYDs<^EKv3O_$wG^y3s13R=(R`V=uE5)ciM2!u9kMrGM~+ojn@_;& z1@`4@^vsEmOtS5?iQ6C5@q92{IC5iYioRZq-pmvCk|fY+LK4vix$(sy*u&rB#5Moi zDK$x=)Ga&?-v-Nt{|a|R7#Ug#S4?s|FVl}Dh$iHd|`Zb8i z@Zpc>`$a)9z)=Z1kIQ^1y==?vbB>xej#X=4olZl{aZh*28!yZUKt>G+jNbQb2|s3G zGvQk3&!cn=`EC-WmyTQjZIFvTgK1i1g(ce~aP;wEz5=UMlGecUM)I$^-!wv(b|+=w zbtQhaZHYmfL3BEOlO8nam>!t2_He8s^W02uFfikwS^dgj?e=KgF^d2f&Eo*XoC4PM z_qcaHQDZk=HeOR)A1^SCJw13<`<1vIWS!fBJ3mAZlZl>w-z6I||4R^Tji+TvyhXu! z?abhXU^%VNneX`=ageDkz}$DNSm62#gs>5Jac~J6yjnKc^q6SXn4`~f-F6IL0X|^} zu-Evf8X=7F#w!1pl~?WM5T!p^>OXOnj8KpiPUTbcQ!BnDANS5z=Y6A0PE>5RpUe=y z`9nh05v}-2HCI@&W}(y|R~NjK7fDZ7b!dA0v(1AJ{6y!zO+`4WzrNCjT}2vc#ME`=*m1ZUOI{Lo9pg0%!Cm?~;Zq;YV)*;F04=7oH7{(Yv-BhB z&}2BLfr`}Z80Ys{k+V1G{1MxE7$I$3`(?wcP_lb52|LFUT8s4Cq>iLIeve8SscEK1*0epa9UBKO@q8pVTp&`6`B*zWn_a!*|%Gx%A1ZB4~DN3_fzb{Zc z_1Md)xI_&7I~=%0b4P+Ab=lf0LqbCZnLEpqg{!>iXYWy9IKYX!CoEibOC1d_DXy{bj9<(BBOrU6vKk8x8$a3~7><*TLYtnnw z^msd4>^w$?923{2w>O(763Cp2tKhhh4Agnpw$-qlNS}0xL@PHNt*>CGrz6WIlMs$g z*sKSRvsGqPnEqY9gfxdDu{oe5-8!%QzOIZ6mZ_K|;KPrubPbGD3C*DN!xl&6HNrdhHY+3Cvim%1 zGVbrx#{K9}zQccW$^;)my);yd+Tsqy`P=i2#ybLw|K=8R20uZ0w~ShXO6J%uNyLQ( z1kOi_owN&2h=Ahq4~O?1m4D@Daqp*Hf+zi&sY{m#z*quQJx)ITr-2aIORm0pr>0j0WBZ8RpB)qfK3tGr1MridXR0kt!8;S@PTqX2 zjgBPsB$Q?p^s7nfMh4jM=btk5rG3Y#NPlE!Js3;i8@sRtWgu|wxwmvf)a{m;97-kH z`*GxHzSoJ%siG<1N)n{B{r`*+zZo_jm(WPO@G4Ow0KH_xXV8uBOgj?@I>|K4v-pPCJaw zX?NJqwnxn@&aj;}y5b5{)%s$^Zx;A5!Ii}5PUcw{bB*QqdQcV1VOt4BBlBEJ=DvZ$ z`;W^1jn~^Ml;a;qjgQB1+nOGC=&HMf@12Ts-kliiRp8`0>w%gPR6tR!%3~qGQCr^U zVD8hEEH6@R-DTp|#>d7S_WWw`)ZQJ_l|rpl3IC#A=m)fgOZt!+cIWHjB;!Vq{QQZe zy-9t#+0~72iXG#F8R}f_SINqxpcQRzajldT8z}NFXr$#rA3jn2O=$sl?S-HV!JRau zhTNb&klho#?(EJth#yo&W|t?qfq1i>fON*a-@C@UO>dXt%au7_iTPV<&Sp*Aw52t@ zGS*}d*3F`HCfz`Pr<*SNBtwqBNT==mom^WshN6;JLjR2TB^YkNrRZlxGo5ahbTsm| zm$Dv^EyZKRXS?%G)}A-Z(qraFmUhpD3~`s7w4!F^V4Fj8$pP?V^50V_o|w(fv5a%J z^@5*8FY_JDn|D4$6p1U8-c@|wIm@3o%Un(7&Xu=AFH;Z}Ak&DYxyT-4W37&DDNUd8 z;-1`>Lb8GLvNBSBm$qd!$qilaUgACTTXI$T_VPkwB<b0DSPS?ILh{Er!ZheaF4;xS}S zB#moM`l`q2Eij0+Sn_RZuSAS4eZks z*EpKQyGP`|X6{^jN*kGkssrAgu{iB`V*ijtSeV+E{g(&+FU*Sm_mw8J@`PC58CFAi T*GKu{0`A>@aI50xqZj`NJ$6~n literal 43729 zcmeFac{r5s|2ND;NS2a4TSzG+A^Vg{B~-F!iDWm~*D>bPLfMKe5u;?^_pGCAA^Vy! zri^_X`(PILMfrT|`}o~|+{bet$MYPAKODz(jO#kD^L(H0_v^i%*NfYlY7B?C4pC50 zFxJs5AUBCG{U7dpcJN}WQ zwZ#;A90^Pd1?_KNpAR&>diGRCFu4vDNHZb0W=KWjvVaOzlupY&eWK-RlT2o;reJr~ z69TN{D%!3N^+YSI@Er6Q=D?@5Xi?{=p8?DWozWrJz+7>g!g}T{%Q73fML`PVippos zhZdFVEOdM-k59U$XUKNeoJs9@YsHLt;_3ETvAJRX3b9zy^xYvhzq{`ak2CR7{1_1x z%k#zy^5$vShz7QV=vm7&E(NZC8fSY1g3gOs*~s^I$R$5|dFkwY5O(Q^ZmM#fUV1|G zWs<2_>9Gc3rROH^4!eFm=SE!Jbk~c2OFI%S$4<``EG(g0ym z`kM$@=7IKscDv@ten%5|sH0v`#Wh_-MW4%*tB@6!t(fcA!Jc^M_N_0b6a9In6S9Vf z$uxckU^6J)Qt6os-*gW_%OOT}YRgKR50#U}bnG7#CVeoF*4+g0wNOCIJS%9e4jK_h z@l7h>c1`$jzLPNQ@Tl_7HW!35!U^O5)%b_Fv#GPG6Q2u0&9=u2cfoExH!1(nxJ~q( zA;e|F0Wrl>51%Q0sr~X$U2Br{`BSb)~^vc!SHLy&|=_uc_|^n5DrjLG3ATwZDvq6gS=%&+HSqC}@)-5vrR@f>WMV4}k=h|IQkvek> z=wk1a+Ru7gT5wf1HM%zTLh9fbooFJhT>D=)`2ZyPx^X`z7_x=E?Vl} zc%-30Aqrg6QcwoiP*4F^l)xVs@JB%bN(AlxLJdwluzP)=X8WHdM{4g-P=G0JT)A}5 zlX4bK=WA^FR%Id8q01FAb=j!=Hk*2jVZG&*`(QSnW5?L*s98TVTzP$mL+vWnJ&w`` z$4G*Vj7e$1UjttTekl0FAIGn5dHPL`tEo-IqYaFY`epUF;n52aP0b}GMAMK@yBy9d zVHwJD3nqt8@DocGK1)T*%n$abpalJSG5HKc^7$zzC;Zp1yZ7CrqDrhjvgdQfAW(;i zLa}D>e+{_vC}kS%@7Meu&^z0oqNrapn?>;dF%f<+m}1)Jzo)nT9Ngp#KgC?J_DtPg z6Jhq>U2TGN>$maH7YdqN1WJq=x5d>FF=AnoMaf&f)p?gpH&YwCU!-dv4N;IpVxG2LCzX8+~pHgW0W9W9q7eCE&Aex z;eoxG{kj~@DukW8z}ng_S*J;}{R|mF0;&gBLn7RxsA%gNx*{>$3B$~_&zZ!ku-f-V zjZVPejePN#wF+xJS8PT3nPVO!91^hJppk_nlX4cd$>H~7Fxu7xlKyNp;bY9X2lw%8 zdLb0>A*%5aH6|#H3D(ZW< zLu89*K^f=OJs2KL2}m7vjF8yTT@wDm*RM3d6tS(Xr}kp%L;U;{Vl_fGulEp1xIG}R z_Kw$%>@ntT3Q9_A%9A&O_S(LOEX@9oT%=y^H71Cb7EFKchM~|N(##*{PjMkGZcmmt z2FSXh|BbE_dx6AvVD;aJf=}!*W+DZ!2+*x?Ho8A3vU}gxll&B|!LA!5}?pvgL80-5^#lBzo|At=(KS7^mR9gLYZYF-)=RG8T zTYG_Ato7S-oIDC2kBYLkww({RSdDseVReQ*c`GO+M5Ai#%{v-OkPg*!TNLz8Oh@8E?Wk0o+`1As^!cqPXda8!t z7pxj$i?`W{vs@Ufjg;hst`(Z^;zza3yxhg z>lb)3QqR)BmV&9|Vm&e;oGbpP@9dpl6ZO+-;{tozTlNVg37c;__0q(*PcvkjD&8N! zh-#2)lXeR>iSHt^dwEGqB94`hROchF)JE~!$OC)O9=>0txJE{71)}|6#upekxtI_u zi~5g8AZN5OM=hIT zwZ?{DGB@|^QcXOl!{Om}v@DZ4;~64*^*@6u0P%|UQ#e?M@*=r=xQ}Ptc6X-bx6CFh z8_mZ(#DkqYh6`#f)*!Y+Ki#>3x#Sgsco}RtwtlZ5RDGAf z&EcJRlJFXG6k|U5r7T|Ac_Vwt0ynsb8dZSaK0wPnxeU@XAq9LI7m>vksoK#_&o3xw zOi^r4BaerYa?<^F)^6x#gzEjooY4(B6zf}ku~hmA;DV)${CrA z?3X*llU(_?S;jNepJIjc6%eyy?qP?Paqo zfes!Oi;O&@n-f~*!fs!o%4Do`$*=0N^W`r0WEMNm5gFl z{_V26jcuYMQ(az>(G42LE`nN=lTu6g;pN+JkQri#vL7oMl#?S@)v+(+bS}4Akwc10 zj)|T%sVp-5&RYtd8*~FijhVlrlqyQhsIyda;QQKX`18TRz&nCIqjF4!sq-t=9%R1QxvX>}14AjX;CKHT2JF zB^&Hu@eDSy5(kJ^S5rl${{%o^>WT4K?!&KH=^ zSz-*WEH^5~(SPu(n@lsphB14!!po^>70;YYNs;rDv~;R;9Db_bb&rI(-5xqSP$&yq zL4Rovz4JW?01HX;(S?}ei8m%9PK^$Dowjtx+xHpS+*(yudNFgM@zB-~^;B*yvT zZ9ToY+oYQcMfZkp8h$+Y04qi3yCIf$b-H5Een8W|CX~7vCL}81=2q;t=5fo?4o-bA zYCdOJJGw!j78@#I!b#)1I?J&!aNzA=7N6Wn83x>2y-$zi1)k6{f7LN7y}OzzQdDO& z?--gKry`uGzIi$T0g=ak6qk#7vJDRsv~MJOWjnkqFkp%LC2+yjxShbcwtyGBp-lxstme!q=5)0GD?Lv zGdeq~#IASlOMAu@_w9k;!JuADd;DU}^M9kUuKD083F2~?pWTr22E?K*PV>chY}Z7T z;kI-q)qJcWR*gZ1eZp`L!|MAtV z_yP~T<6V;5Z@Q5XQo|i`%oj4aBbvHNLuU%lkI$QBoNm?5P{F!0^bL z`e}bt8t3V}yII-MXGy9sA6ApqfO~~lCar;4+>)VYytVya)RD*R>vM_vyYcf= z88sVa z0Cp4EX%&5Jz!DcZQ+E{rf1ZIEsiz9t=>a{Pa0)Hv*S&B@ADC-LHYi(Q(gnvTL7eTg zxxJJ2N;opvt>>>fvIwS zg{@8%8yjokHf06t_)*Sr{^0~U?&4w%bR^sk$!Wz4^Ul&QKQT;rp*@1OiXT${`0gNl zSUAtg0;*W$r+S!A@~o{L!S&v~PcL|MWb+;?O?2krrWJd7e+S{c-_-=di=7tMC6#h= z6}ihj7Q-j?Zi1PPBh?Zd(lq?RvWMQNzVew63w%*~u(3yc9b&$L%p{B=zy%2}w_V=)&79lZ!+c$wpF%@3Yukj_x%tlvFCd=29IhAr=WoC7f{y}$|IxZb zf4=s&!s34w4Y2b5>vnnXH;ML%`CkOt=A?aA*_XRr*x1&?`>3)9pzcHTz5@D#E%q^U zA2atcbKiK_H`@OI#eL&p-{JZ%|6^ofhF|j|Lt9{Je$IL$< zVn6cmr*QmV4Sn}~6t8T5qgVGlRrZm%lo@zyz;Y#xL&4GYR?J0l6X?o&r*BTVq4^eS zR>~>x-!u8k4_;w_k5W!Hvu1jTQiy5Q79^j+zJ@89ynSuAg@1k#djJt4)g~{e?DM(t zLrF;qm#v0!E!xQ~?I@=RL7eWGaeX)hfi3MdJlpKjM>tFQRLKS!`mDzGDL=x_k`#m#O zJW=TW23e%fgG$^Jh#mbE{D9p}BD4*HT}+1MDEe*1ZUt*AyP97?wz7E4;2>c^ZOERp zdo^Kz%_=}m2Re+l9{W@-w_07nrZlWiSTKbVCWemjZLBv*T8x`Wx=y~jseOQkG5ARW z67tSk(bCJw6aP*niXGMw^$x=fLs(ic$kB^doAW@BMc6 z+N$yxqcSoZX8fr)rnH*0>NNk%O1CfDxKP=V*ehn_fd6L?mZJn&!aq_@lJKpe6}WYu ztL4WHLyyZ<)=pPw#T}PWTKUmTRza?nc6M~+^$w`23)sv!Z&eiy02NTQ!M4P~l&Q zoGb+0IM4yK0u_I5gE`+pEr%+*>ifS#COS5imv#hdh)soB7n-@k7%l{kt zkZI;wo*eAg`3R}l$X#4eNr0;RN|Te-qZ)hlNRBrY@KWZ8~wOg&a6+Jok@OgOq^q- z7u=5JIFq$p$<1ACW>${~ktvVPC zFXFWRS|VtP^+S?7=aFp&s*H#Fon9bf#XRb?{8n15ZS6CPWSqKkwwb4-*lS6l?d4Y0 zG`mACW??+ZqHC1SAy-M@-~8zx7vN0r5=GH*N0ZfVNZlO?n9o^*%B;KiJ`9F;EZ0_D z!$E^MVaZr=E-MHxS4b-xYk7np{wWlu1PHQg~;Ll42ca_$ZvJKkd(AZ%$$6%Ne zeud+_>*JYe9_8xkz~~0k`3mt+@`Rpap(&C~8W{_y67q9FFA zpQuGIgaj4ocnsvdUrcX@3Jv2wmqsentV}Ep2x`^6YfK1k(2^8<74iZb^B~Q zSKC@CV6U??_0XT`$mVT<1R#_x+BuZGdvs^2_IihZjSF?T7=?qFqTiQK7qM9pMyBqY ziwH)qOR0?AhJZZCtJUk@rIDe_5}Q{bib@k#{&)ra+qR^IaL-`1N{1uM zqyLjvQAr6p|q1mRmp!v5S<-F*m@n5?P&pC3P%imoi zD6}h@TG!D&FhoNG`nINwWl|1wfTp4OxrS0IU;I5P-U^8Oea|it3)R!avl9JKTxx!P z{^?<#Va~{YG9NttXxevTra5#fY;ZeQo%sGcE~d&sGoEQ@)z(^o=$Fr|=R38oMi=W< z6jUb*o0G~{^PR_HJXQEY%l`V81`_C$ z9+%qoJ|0IRxm{H2Sz`7Hpd8IwEAbI&FlMa~SDz?MZ_yOQ6eJvQymvUKZ@0AK(3w{k zD0mb-Zy#SB)N|*Wh~?Wluc24_u63aVQ3D~l|L zBZ5YlkR)|WMD%gz+wYdYGc|NPP#xs4IPh!pMd4(I)%-^3=*9#+n1b!J>^(+>jpuxg z#uFxN>ZZ<~MsL@?0JHsF;byIuSSDxU+ooVKY7r-)9OI1LYIsR9;b4OP=+1wHBcXU6 z{Tdf21hC#_n=B59{Lh#HTn(pFoq$N4McB6x3Ez1YtX;I9_{`4|g&cP?hsv#55&Pn7 zS4rEm_w$5!VXh{Mh8Y!p=x!$hVm_o>i*)}TzCNtJ$EL!e7|Fh4e&Ozh;fCn zETz*>T$nxHbEY2>7^J*4x8B0H1>mp}D?)yzWwDCS^tg1pFPm<~0f%2gAAgKPdE$M{ z#@w>SHxkUj<6&#Lbfm*}u^*Gr?E;eFBD1(6*B@Ii(d zpgs|KOk%6U&SGc|In%+Ruv?S_)Odz5f>Y-t%UWkU1Nb05!^T=b7co0dd2F5IoEbQI zlqr|!Wk1+6U_92ljsI{H(T-dv?OPRV$PreYkK3)PpK5oi2Qvnpam^Y==((yvaywd_ z3YrtxPepr$hK$nID2GQQ-$(}&$>kfdjs3&glpBvNd7j|__9LpV z?s@TyLJ`bIf$kD2B0mAl$9jsCkCbZpQ~?Q5Rq?$wC8(gxCvB(a@hAz~NNWb%h+>gHQdQYQL>zHD>n?OX7{-ZLXLxqj!} z#i$eG4Vr&WaF2(D6}R^09-aH@)FLoB+GRz6+DwlteDio13?;#6)_2bW0BS1ZL1USn z3Kc65$Coz!#+(%-a=GV1Eh6Y@uCK8b(*xKpH(Mnm|nB~C2`+WRzicZ4B;#NZlH(IyEH+tQ+rMwG`ospHf_uXMx<>3rB z*M~~johSzG=`-1V{gCOXY9qtgpvy^Gz^i6&|M1U^kB}qY?;a|K46NrSyLG~}p}t%iX-{w0mCFHy3LQ$w}wOq`n_bWc0{X~a!HH4K_oZTrB;bY~6 z3$E=rLi}7i8K^b|>KHM^Len;n^t9e61eDE*3(=>**OlSgQ*G z3YK0h=}lX9Pa1z%5y(n!xu^_GHk>}VYNAB5SATdwWmvy_sW1}&GbXoEsQ;r?C!ecWB)bd)wt+(=$ z*Md3vhR2&XG3fQi=83j2E?^qB-hQfu`g8%CHDfW=#d>Gn$8)NAsHY_(B<_oJVtOPG zf&xCztZ!O=)3y#H`H84QZ;!i%>vw528?}s*5}8M{6?zIrrO5J}l_$E@KRZ`AeHx53 z6;Cqyb;41YeD3}nee!HTWf-qihV2;oB#vt=mq%g@_N`>wz_TbtH{d+`d5hdur;8&g ziiYq{>9FA*Yd?mr=BL{%Tedd^o!*9!ryx9Rh2CjlLNlaXCr)KJSuk1iyQK^Gd#`}C z9I?`UNn6s{AMfY>kat!U0DI~L-#L`|lsY4Z+yKhrQSyFx>$v(pJ$Bwh3i8v87iLc z*XYc7DJUY{sV!PFh!DV-Q=ZEM2sav(#9OYgR+dlT*<>zS5-`CcSHP zy$S%FqzhpzH}5Q$BFPr02ME)0!OCYlw&24f*2m=)c`uTPoV!dlW1sHJc@l3kj7@y5~NvFHT zr6uf37o>OWQ1q#Vl~JXUoOYP({3&Q8E^+Afn6VFIC_cgCnwmR$UU_U`SsUTau2B@x$YkY~|Z*2`8Oc{XCB-K>M-f9-`DRaDhNq!TI`85}UY1<6l$b+axW zOY`83K262yryEfm4`BVI9{JWLkAymt-_@bD>>2EPCf}1>0c8Shp~d#XzpQTWjBTGH z46#0|;mj7Te7)kjI&i`SaiJN}=+;jx=~~-%uJarD?tIklIJ0d+9RnvOr^qCD*7{g{ ze$ZIIat6t@t;SdE9EVeu$Ki3f1_Q=REiJ4QCIzj^C?lX;H{qol73~}9SUZ{$`h35x zBG1VlSWvQrauynDzw0X;y9$`6-e2F-sJ+?ByI=0V^qz+IMMFbFTKWycg4+2p|K;{l z^zu8Bk@?t5>3q-5f^}@GM?<9uyG{uckBks`OgpvCU%Y&|$K|E>uZV$p_t(zkxR>{t zwW{aiC6`eXjWI|}b!2nF6Vu_f@+5(@g~E_;EI>a0`i*qu^agR{Q9`H|cr z4h9v5Kz;Af-4+*MJ|Z0#JgtYE$gGmf3hWI(k^%Er-GnDB!PEOfFjrv}I(zr{m}9yC z%?x!nzC^c&ZfJ=Io05olr_PP&h5@G}>>H*84iZ6xwBGrR<*q`W%=XzMCpCEa)Q&^1 zR}%BsS7S1Dr4Vm+!dtg5!c!Tc8gG#9MMdoY98&T4_=uYlWOOL&tsi)Zj8hqOj-1#L zmu)LY1hC7+Of9{(pj&}b?aIXvwhu!=nLJqhhFcSXI2r(XrgN zv&O9c?kOqIWp>Zi9=Z+L_Xj{!=JP|;`RnPHk~4{%$tw39syUgW5(FQAvr;}z2;AWOFYs!` z^d{DwheeOH5*KgXnDd+1f8GQ@sXRQv>&#k?f{NO1kIA1&`9vmqS}80@iefY#U0y-2}q%?hwY6pqo9@mvc8irGp8>fN_I{CaxsAt#HDKR0>2 zD1sfphGng6ypR$PT+06+j;jjZ4koUtU|3xE5T;WT*$^1?Fjo<_oeo{F=uEqrpp7v! zC@xspc+aG?@HHw{ZC{?XFI;h;xb+=eGUi(7AxT1Yxp}4&m)z+_=ENyRrtX~8i*71gelq6 zBee-vj>S1xtR*OC&y|VnNvO#{V2Q5de(gaIY6EmP*U5N4F4?X5cAly9f_9+OXe86L zJj-E=9l5gM(EdNf-PE?F5r}Ze*3g0B6~$$y@k=IVypVQ(@-bQ zt^(dQ?{h2kNrH+?fQ9|8XZ#oc0bwiKgo+QDrZDUhJr;=Lny)9dlY&*9xqskG|&}OR7n7BBvrvijjUKLcMIY%cbKNTY3>X z`Lla%ZjB7+1_p#NbZG8Dk7(hEroO~?_{N1JaVjiDZbm*~0g=P@&L_ z-(OpirwSj{^DoimE1ikChzqi+e`|%{(!0Ng{WQ@=uxPW;YQQGvO563-kDg|+$eqv# znuwfVBM{GY?qA7tQ#Fz1E)w9Q*X2R@uK9;u$M&?34TS zg*{PT8d&}RK+{uMz#{UrQ=Rq_=?tJvC6n&^o?||srb>JkFCVfex&Q8b+U5G4T|0CJ zIJs-K$L8H%zy6~8{ub_U;a+U9FNOP3`2PhUt=WR$lxt>Bwa8E)Z9d8#y?*`rsrPu3 zWcJr7fr0+~{JKdE4P(vohKuV{zQH2ED{N_l7L}yvGlI zHrn`8Wn48|`sbNssw% z=_PR85O9Bvq;CfE$K>g%#AALY-b-bR>)H5Jh|{KQd$6q4x6pXT!OZ00E3xjb9RhS`sH z;yd6LT3sRichp_!b3}x^ZhXhJ?2UKGDa`{8RojWrgX`%$X}bc1%Lr>~ z)5okDRty$LEo!IF{_Y&OkP5c=Y1K;Td78+C_Vj2yfdN*M*8`d{al6FjcT(Z{JZ^BS zI?iGucaHT*PSsgMq%0&IZ>PFDx-gK^R`0z*{!~d=3+R@2s6wCY1is1QN={Iie*Piy z{&nc~*1&z~E(jB|IA1dtUHjqzqKeWiNA=iuJ%}guCLV(itW~$ zj2Z*f&iGN-emKnIWMy36Qli8>kC8*g9r>-gr1pkyUESX_c9xhZM;#?Z%ZR#`d-3PdfzDd38?-RJw$t1ef3m$D$K#B*n&iKU*4 zb*6P6#oRN{_6ffO{IBTY8BAbYIka;OXLp#3*)B&q?|zXnen^G^96{Y=kf?T?^xZj* zb7!h~LZDVozrIR{GdEJok?%-Vfbvn(>3P3iY4L3iJ`ZXlT`Jvdl9Zf%6Eu>IR_Q+v zG0*+Qv=nf*qiNF=u0sQhiz9h@cL%H1zUMP8%Ag=zKyh54oXdEFl817^70OAM_TWI- z#^BsSWEk>#+whGPCu~DwVSDw;;`mEwmD%TiiXbl+XY)?LhAtm=xs+`+{P@V@nF_^50}FJ#lMo+eTIq*jO(#+K-@7zU)l0GrVa`gQf>4>KhJ9-fAx;EL3-iEtWLprp_5jFX>N2d1@1cz=^ih zY~w!<%)cY~8Qaikx6(2BVRDO_Cz;?;tk!7nOTM?8!A8Zl@tMINEOlm~8;367Zn)`i z`=3-9z?ETN7e(3SR$Tk`R>>p3P^KSk)_Q?LL$^HDmD5)nXtaqvq8ZEbK(ZG;#{bIq>uRehnIAe4 z;8O(siJacm1j^xftJ+j`{jCX=I=lHjp+(!YlG(6o2Mfqe^U>xK=xNKDfg` zUv!0bFy+`4pcf&;qI;FlkoFv7NDoNOH0|L^(xA11RTGX|F%vVr_~ph#-DpT@cA?6@ zEnr^EX@eJX$A7A6h;^;lNMX|(5n*#G#a`O7`MleL$9#(Zf)(vxxyH;za(K5zG08pT z^_a)GZ3@}Lb$Bb?UVu9AShRX|u$iR+&Z_C^q3elsyrU2CI$iJ*>y=@+(bJ@9j|%hSXg>l^WR*UO-%i5CEU zoS00-R^-TH!MN^R>Ae0z5^0Bz^5!P|E-<9^Jpt0;1WI?-UwGoGu4=u$nLr+XuSz(W zPm`^5uJ67bIN00GUU6-qYSPU04%oU4K1H})1X|_HY^b@s5~s8LGV5PS!Ok3F};;3G_n^#od_{5wI(L)_ch_o}XT>h>%6WkKpkfhsx| zZCxJW+lwG6J)RUO8 znH+>EiH0Y7jG>)0eC+h75~D9w2D?QJubAK&=*o8vI0H9Li!jqa{>AoOB zZn;W%)tdP?gEH8+8FVS=Dm;?lzSVkpZyzG)uc6Kh{*XypJj-4d$7PvNtl4!}Xj}v&jijjCJAi~RvvM)7gh>fKys;sG4o=Byf%rI6cCB}Kt z>@e@nLd1k_yD?p|6;P}AB@3-5Ufl;yP_<{fsqkmVAXt>2!i|5Iop-qOw?_O`q@r$T z_FHlNW|&_@q6PCD*z5OSg(h8G9Dm*Ydw9_8#@}QACiU;%cZx+yiVj};HTGYV@u%pK z{QK@-uLbWEe28Vk|Fp?>p9hw7_^(m+m-N?u?C;VqQ+%KCeyOE>S=zzPeZ>2%0e7ke z_d$Fg9`@nk4>MvP9`@nkPy2Nr9`@m3A0Bpt+W%7G|IoRxHX3+6_Rvfs%u`83$+)L= zJ4W`DE@EU(R^(b~jN7%)>}5Il=fg7Tw7qtwhQik!|3SO=$cPXEFW|- z18-nc$1}0s>i#u%>AxXVRKj5dNGLB~Zm6e4I^F^Z?k@?SgEoANIqzjysjxxo?Lk3t z^2CiRm+n2Vb)D=`HjVE`T|RWn*UGstw8b>PwYh9m!Dl?3UGR7O;4e1{=*uL6#NiM> zZJ)jRcg#4c4X;i)e|!{UD0Wt(HIJ_%?I@fMD^d@H>Nt(;t1e|#%}?u`x2+Wdn@x7t zV#j{wHj)u2X?c$+s;3li(IcPcU`%g-@mUdaW)q^E`45irpL1!*`m{PuV~%F%3v4aX z=L3$F(K)fO6xb;V#02wECa8vHsJBj28be_$)jLW^+;r{Eh zW51k#xGhk-tE*ic68R_A{dT$|w>vPhemqy-^|b%vnkU9?w~^d)u-FZ~{mT$QNZJyp z6&5}7>E|BZIcX=iyFqkN`g_z7-+K#2iE`)u6i)&-hVq!NN)yyd9gU-$v}8oU{92$h zYtsp26a>{57NXdzVM=#46TC$U1-L-BrJNhZ!v9JTUNSeBqOz=aKVL1>J3AA|pk#te z=i|F!1RnV!(>~+x3E-=TiVXE*MGLItcfV-!7e~V_k3FU%o(B!?*&xykw7KXyI;`w` zljaxw{^bzhNoId>!Nk@*7RH=*A6WHR-?jGtdT8fq{cAw3@Bi?X))m1hROLZAEJ_GW*GhVq(k+5 zl}C=I-Gu|^vf&`1oP16Agfs48Iy5U9TXUs0*urH|C85I=W+0bG?IG{AkjLT`Jc2b! z^^rG{jPy6FU@K{|7DI6?A1oRf^FLp+yV|J96I+xW34CqtFg#kVk~iX{<_oueQz`G& z`Oc4;(1p+ca7cOI7ebZTpiUml>C0VFxP+PPNwURr`@LH!|GW-5G^xe zQaShWxSy`G%4!G&JefU4&O-N{`_eNf{9M?_pH|K(KmyN!IWxOc#jw~m($AUrBtkq; zgwMxu3R4#aPXD2!0Y{!FVU91A=ZVZzNr<{Bn;XB`n4MNQof3PLJ zBKd2f&HDE@CT+2D6)UAR64=)?2=M>}X*2iunN*(E>;kg;bnfgGo!2hZ1H1Gv$BjYR2d1kHoDR5&#R7G zktSs&p@ZM9w;jYd&xABPh;_1WCw|7L+0N=Sv%e!XUUzhgc*>m`zmYxgAXTgxLR=uJ zj*Kkl29Gj*2x_l5FEivhw-UuM#`+ET)C@g*^hs!DZl%CAM!w#UWKD#!RNKZBPMxxm zX37$mxOlgBgU{HJ2`2?8b!^(By8K>n{ zkH!fml4}Is;VP&_y@oTfMO`@nXSCed8Qk^u0f}a%=_jr0TMBpzZKa8POXm3D+}coi z{%}+8a00347Pmuzrcg`7RBJvKTdddEb72EW&T3MJ-xiXKIP=zh7>bhrX;m<*UE#=z zQCwVjDD$X2mZe=uN--V}q{c-}`ROFXrOHCriA_#J(XeAo!~H9TpK_6Fp)pccj`ClH zh(nJE8r=(02Cs(L1P!8|o>*45iZsFFjYRoWndt7luHo||gst~UTGg?yDOJ0stc^)l zxZd#5P8sF>(H=E_Tm@#qAQcmQ?0 zl(;kIx?q}YYWtjl=QX*&rLGf^{VsG0Xg_ZiedYTy?pBk5SrfEsI&N*iC*T7AjW*^u z1rHNwrv5d5a43~6>{Whasz+Ms1Q6`L;kCh?T+;*aq%u`_z*}Mb!mXT9!>*|u6z@%0 z1H1U%u7zbR3O|jOS|cuOakd}!l1$NvUtH9wJP9_Fqf=ZR$XmIT(!`mleqL#HU9I%f zpdEr|27SScb}3W=^0q`+?^%@;8SUL)_zEmc_lc@HJmb!1=nZ8#GAdH`S_SK1eOTM3 zZt!Y=7PX#wKkeDL8W&ae`0vFt38@?@V$Dsm1K5R2tJhmt`>oa{> z-r#jD8Xr8seIJ$_=@W?>j=vzRJ5T7Gael%HVif|^eWLk2TfIk-;%@4WjtRuO%OP0g zF}pY79!LCjZHrfeY6|FXy*;ng5EB1tUZ||m^t`ZE@U5t-^GmBM4^A}+L!g^$nkZrb z1>BrAtMqfC{`(by2_IY)U%#=7`qT~F%vDqgA1O9Q@yCx`*d|1NK*nLPE(&FTKN!wU z4HspeEHlSnueQ~SuQ==d$zaZs51}kLkx~f4KvtghXZ;zJ>cHNyVS~?ttEGT(o9>7 ze|WH)@?bHsG0KypE&;PuLxdzCOHuR=y!2)wHXSE=rrv^yDI}&Q=n}Q!0^1 z1ubq1A`bPNU=%4qkzyU09%1VZbOH4^KQ|2_D>xsy4A-!hY;@ngto53tTXXOyY!uB% zZ!{)QH`}mY54l>epNzQ~sl;C;2sV35J)G^<=Yi)%a}AjSmP)e&kY_*B9wz|iM}SZ; z+o0>!TyZAQv5-%&1(qq%Cd>f_W9hyD<)S0&AA$Z=M3HCPFfKuX@5N^iB`)I%w@_`( zEyK+KE#L!!6@cvmulZ(pX+k@UDQG6QLJ7b|j(8wD#G72G7ETBhwt6CB_3ZdoFQVZ) zl4xRYJX7tjzM!is3{Lz)#dy&=jN_9ba(xjm@iIM`u#P6fm_C{oS9o=$t2I{B%p3Z= z_5!}RD|(wy+de2T(HMYAaeZr_+Q!pO^HC#So`h{RVd1}h4t_+UiVcA4(%0|q43@iP zuFo@to*tlONKFF-{)27_XT$2+Cy$Y?pd3xS>k?B~$mlFH3ni$YEse*YOhA-g#dH~% zh$JjU4oug!7)P{NY-i{{6}dk0TvBhZpCHMR*GpZd6*&tAE?-ki`~*@}*Yd4eVArc7 z;M%ol-oBfAdtWAwic1<#T{TiL)1{8Z5977#pPf4oqPhz=kg|#v4v!GviVLR^X%k1w z*v%hjx@Q0F1~*)h{@ZC1flSbmZGdf6O(r0llrAH+%TL&&SNp#$=dDz27b@x8n-%7q zrdd|Gjw8rfwG9neZ5fYfSJ)Q|#Ti9Yz~3GEmhG0bGV13)Vd8o-OFj?cb)l(XbT)U? zCLFfjs&bP_ErzX$#hl6Jht1*`Ln~l=bb^|a?Nvyhg{P2nyh`N_{p!)RxJs1F6!g+W zJAzL&!BapSxq?{KoqB(YJ)V%-6~Jq3j^dHYoRpk8P6=u`*^%$Ms@D33=)jzeq2!8v z)TVzd%6*31Ikv!%Z3h&;Y#hCwK=T-f4y#ANws6;Z-QV2WR?QfNxy)@vHfDFu+o%`V znhbGR?Gt4AN>f{4^tG!xrhGfvU@P5DVaNBwyNF94e>Vwl>DL zi(tAKMVPK`jvOI0yf+ z&o|hw#FaR7x#f`YSA(+cfNb{05&bf)TxTmi(wq_#kVs z6!7D8LRRiQ#39UMtlRWcL~%gAGsYy}XzIypBgb-C0Q#5#&+;bjh=~u+GEi5fmN@Z} z4^0;J;xNQT>SlR7?oMqzE1}iKFAJoO55s{XsH5x%nC0eYtMea}@ZTjvNe%2=MA#+C zYYxY*vH=b7W#z5sN=BWdV$}_~2GMmMCuMsc0=p*1qF)BVIfq5u{+OShQl;MCfTQ0* z0scH9bvR;K+6uDDx6mmjF+KC$zgPFPM8duD!{UHEk3|Ap^-2M*oXn(ToASC=tscnJ z>n<0x;RpGi<^ZTH?*N#*ue9J+2(Qy^%2CTuJWyWdq)e04Po)faCGy9n zS+DbYpZ8fC@K@S-lB5NwslpO7-xy zDw&ec$|sp_aO#z_o1M=FENO{>ZO=QIq`Zcb#%>)4`sAJ8gk%pV!5i;rNK2<(1k~3GV@g=U6qP4;-K7NA)O@ z*V=v9AUgG@72wW|^usRJ9?)9WR#tMj0(=9LR#RHmQmJW<#DrUp;Bd(6G<@yWa9!Gn zO98G(C!_^co_tk>9n9R4;N>7*Iqo$(aZ^AIQUCmLqZrd0#7cRw))V&Se1GsT)xpTN zS?`jSW9tvvL(duyD_0LAW}0aDv=le*%NLEZaWp||ZUIc7$6r>je~N9UpYyTv20=Bs}3Gjl(0SO<~^Af(`i5niqw=gf5bCDODqrgW-`na0VfKB z$y94wLm#9n!kI3@*07jsOZV0XyWgDd#(puqeJ;-8w!EpTQpGV#ei#M4UhLgon%yA>0QSm=Yu zrn8Y}HtM1!gA_pJg)e#0-IPLn-Y2)#;?9!*w=GcmAcJ*tMxQ|55C9x4A;a9jQ4opF z)ZzYuUjM7TYYj^>-P+ddw!50NT~}IaS-VlEJZGL$tfoxOG9AlGprVl!mCOT*0yRyU zsX3#Wr@_N^)Fx9h(mYdnO38Bpih_xzq<|s)TAynyv@})aL zF*cym6+b*X0XoM!>SWssnquQxrw!8F=xg_z^*s`OOKLY#LCNg&;R2uGN%~RwSKmHi0hKqOU!?TOnhoV$K8zqJGx}sp&H}=#83aIQ9=>sxU%LAhu7R zLYJC$=Q@_pXIB3)egQqJdo-XuYv#l&Z~RXO&wd8N;K@*4&i|(MfS}O~jLiS@%&}d^ zsC62wWg)H4=OOG9|TnK&AKU}3eI@9WDTEQ6W>!2u3ydgSU zh6x(Fy81z|6onVfAOO3N*s5R5PGObs=K(K z$74bskCdGe&9F`ESfPpl%mahgrXyi$Vt(ka~_Zkr961%NLek?f4~ez#|(!tYk92lnOTH@sAXBOY8aWBdFc>W_=#n@n1ara z{Wi32u^muhDYc(&H&6u`p@C%BKEl|P z|&<zqSmX(wm+BRazg*gEou2-N~VolQd=#KO}OL24S9(}gGT>|Cr|lAn4r?IQexiM@mf_Hg+x3zlgH z{!ZkDhx_k)>wtZ26FW#2&g32}W>o3Mef*A8Mx`;YR>WR;ya zu;$PNo_7iPB%}e!^B?Ev09wK@OSQPv#>eGm^XpoQO5_z5PQ@`Q->dPw)U0&gJX7Q;5U!;9p56P&RSaqyG{>rs8)X7MDA^3`CPs(wA z@LBfZ3@v9NIoi%yn0-?h{BBFSrWViF>HO0mY+UQZ+##XF5T{xa8i`HtDh9Tt_wwXK z(Ghy)D@HCCLTCBnFZgvoop^cDcCg6ZAOJ1Rd4}PS>=6$Up6B`xCYoo$-`e}Fi&e$7 z2z0=o`)@8F8DNgZl}{DupHlNPzq^o!K>$KL`tVWzvt@8A%vtl{O}-xsvFU{o(#uYC ze}S!>pNinOm6gnZ|DVgCW{@2-%9`l*MlXYSzs6_;#K-M}T&Pw~9?9@TdeQRp^<}xmH1q}@0ZJpWr4)F%t5-Liw4Fsfj zCyGu0g!~(dbE{V0^l+L6XIZYizZevOLB~Y}#H|pBB0Y+v&asg3qH^h3ZuBNIe(eIS z3S7Fqr$p1xXw?#c5%Ar zAX|uNp`g>4N_&SLje?|X8+kFcXQ~@#2}J!bMKgtm#Yq$x#^YHylyJSy>}AmA`rOKW zb+nvpqNilHH~)+Jul$5o#lwT^TaLEk8wf_m6z6L3`1Ei^rh6420lhNjXKjM4jxeVMZh~0PRC1&E$FX+_hbaYyd8{aXa5b z%U_xQS2HlnV=S?I-p^Eo6lj|{p`jJdHeu z0JVViv$w@7iS)Um7pIw_zBR`VwNBu(H+Xjr_Uk4**%`)4C{NWp$J>0CReq*d3V_C5 z3cU^q)l+*eJTyM2>wa*=xIJEPfn88w!IIlJ(;Ww+i@jZ)I5u%ozo&TIV6nD#5)R}1 zsM+|B%jcc~luAl=%yNkFFy@A(;lihBNFln*D(dz6s=)`EZ8*sRJwbEBv}=~S_Cbv% z@pGe*0~@WORsO9>Y)DMqAWk9OeW^veg&KkA=m=--*aY_Ml*Ed2bP=@2L)J@eXT}W) z4c0Ym(F@RBfl7-fHC|fcpQJn-ln}incHVqN&T$SKYvM|t`H72f2f@Kt$a}DbELFRG znxnZEV_{l!(4$-guWM9r(zH^o(J3ga{pO?7+ z`Pwvw^9kwRsxEqxvmF~=;-h#p$ELJkw!iB-tBg#|FK2ZrwR4*D3JxJZTt5B;-1J8# z9S{ZErr<&j<7(=tA1P5l*k>1rlc+iJWU{40^-BqDWt97(!0fjh?aEzQ|K1xJlIx(e zQipPTUU=0(Vs+Vx9GJ<{?DLQU=>pp_CDKxMk+}F^D!JP0uk7&fXy zou+TwASp(GBhM1Xj*ZaR>1Y7ciI<%DvmDd}=26P(1D@OZI z+KinPpuOou^F)8C%=i+xpnv`t^x17LKkRR|M+mn3?Ed?_WyiQ;ivE5t|JQt(Q@tM{ WRF}jUAJqUKk0U3(sXTPy`~LxWb92l9 diff --git a/_freeze/R/ci_for_2indep_prop/execute-results/html.json b/_freeze/R/ci_for_2indep_prop/execute-results/html.json new file mode 100644 index 000000000..83bfb0873 --- /dev/null +++ b/_freeze/R/ci_for_2indep_prop/execute-results/html.json @@ -0,0 +1,15 @@ +{ + "hash": "d8297685631397381618f06b4e089199", + "result": { + "engine": "knitr", + "markdown": "---\ntitle: \"Confidence Intervals for Independent Proportions in R\"\n---\n\n## Introduction\n\n\\[See separate page for general introductory information on confidence intervals for proportions.\\]\n\n\\[Note: information about cicalc package will be added to this page soon.\\]\n\n## Data used\n\nThe adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `ACT` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4.\n\n\n\nThe below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.234 (23.4% responders).\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 4 × 3\n# Groups: trt [2]\n trt resp n\n \n1 ACT No 118\n2 ACT Yes 36\n3 PBO No 65\n4 PBO Yes 12\n```\n\n\n:::\n:::\n\n\n## Packages\n\n**The {cardx} package** is an extension of the {cards} package, providing additional functions to create Analysis Results Data Objects (ARDs)^1^. It was developed as part of {NEST} and pharmaverse. This package requires the binary endpoint to be a logical (TRUE/FALSE) vector or a numeric/integer coded as (0, 1) with 1 (TRUE) being the success you want to calculate the confidence interval for.\n\nSee [here](R:%20Functions%20for%20Calculating%20Proportion%20Confidence%20Intervals) for full description of the {cardx} proportions equations.\n\nIf calculating the CI for a difference in proportions, the package requires both the response and the treatment variable to be numeric/integer coded as (0, 1) (or logical vector).\n\nInstead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output.\n\nMethods included are \\[TBC\\] methods for 2 independent samples.\n\n**The {ratesci} package** is ... \\[TBC\\]\n\n**The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson and Haldane. See [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html) for more detail.\n\n**The {presize} package** has a function prec_prop() which also calculates CIs for 2 independent samples using the Wilson, Agresti-Coull, Exact or Wald approaches. The package is not described in further detail here since in most cases **{DescTools}** will be able to compute what is needed. However, it's mentioned due to other functionality it has available such as sample size and precision calculations for AUC, correlations, cronbach's alpha, intraclass correlation, Cohen's kappa, likelihood ratios, means, mean differences, odds ratios, rates, rate ratios, risk differences and risk ratios.\n\n## Methods for Calculating Confidence Intervals for Proportion Difference from 2 independent samples\n\nThis [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^4^ describes many methods for the calculation of confidence intervals for 2 independent proportions.\n\n### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) using {cardx}\n\nFor more technical information regarding the Wald method see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html).\n\n#### Example code\n\n`cardx::ard_stats_prop_test function` uses `stats::prop.test` which also allows a continuity correction to be applied.\n\nAlthough this website [here](https://rdrr.io/r/stats/prop.test.html) and this one [here](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/prop.test) both reference Newcombe for the CI that this function uses, replication of the results by hand and compared to SAS show that the results below match the Normal Approximation (Wald method).\n\nBoth the Treatment variable (ACT,PBO) and the Response variable (Yes,No) have to be numeric (0,1) or Logit (TRUE,FALSE) variables.\n\nThe prop.test default with 2 groups, is the null hypothesis that the proportions in each group are the same and a 2-sided CI.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nindat1 <- adcibc2 |>\n select(AVAL, TRTP) |>\n mutate(\n resp = if_else(AVAL > 4, \"Yes\", \"No\"),\n respn = if_else(AVAL > 4, 1, 0),\n trt = if_else(TRTP == \"Placebo\", \"PBO\", \"ACT\"),\n trtn = if_else(TRTP == \"Placebo\", 1, 0)\n ) |>\n select(trt, trtn, resp, respn)\n\n# cardx package required a vector with 0 and 1s for a single proportion CI\n# To get the comparison the correct way around Placebo must be 1, and Active 0\n\nindat <- select(indat1, trtn, respn)\n\ncardx::ard_stats_prop_test(\n data = indat,\n by = trtn,\n variables = respn,\n conf.level = 0.95,\n correct = FALSE\n)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\n{cards} data frame: 13 x 9\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n group1 variable context stat_name stat_label stat\n1 trtn respn stats_pr… estimate Rate Dif… 0.078\n2 trtn respn stats_pr… estimate1 Group 1 … 0.234\n3 trtn respn stats_pr… estimate2 Group 2 … 0.156\n4 trtn respn stats_pr… statistic X-square… 1.893\n5 trtn respn stats_pr… p.value p-value 0.169\n6 trtn respn stats_pr… parameter Degrees … 1\n7 trtn respn stats_pr… conf.low CI Lower… -0.027\n8 trtn respn stats_pr… conf.high CI Upper… 0.183\n9 trtn respn stats_pr… method method 2-sample…\n10 trtn respn stats_pr… alternative alternat… two.sided\n11 trtn respn stats_pr… p p \n12 trtn respn stats_pr… conf.level CI Confi… 0.95\n13 trtn respn stats_pr… correct Yates' c… FALSE\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr}\n\n```\nℹ 3 more variables: fmt_fun, warning, error\n```\n\n\n:::\n\n```{.r .cell-code}\ncardx::ard_stats_prop_test(\n data = indat,\n by = trtn,\n variables = respn,\n conf.level = 0.95,\n correct = TRUE\n)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\n{cards} data frame: 13 x 9\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n group1 variable context stat_name stat_label stat\n1 trtn respn stats_pr… estimate Rate Dif… 0.078\n2 trtn respn stats_pr… estimate1 Group 1 … 0.234\n3 trtn respn stats_pr… estimate2 Group 2 … 0.156\n4 trtn respn stats_pr… statistic X-square… 1.45\n5 trtn respn stats_pr… p.value p-value 0.229\n6 trtn respn stats_pr… parameter Degrees … 1\n7 trtn respn stats_pr… conf.low CI Lower… -0.037\n8 trtn respn stats_pr… conf.high CI Upper… 0.193\n9 trtn respn stats_pr… method method 2-sample…\n10 trtn respn stats_pr… alternative alternat… two.sided\n11 trtn respn stats_pr… p p \n12 trtn respn stats_pr… conf.level CI Confi… 0.95\n13 trtn respn stats_pr… correct Yates' c… TRUE\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr}\n\n```\nℹ 3 more variables: fmt_fun, warning, error\n```\n\n\n:::\n:::\n\n\n### Normal Approximation (Wald) and Other Methods for 2 independent samples using {DescTools}\n\nFor more technical information regarding the derivations of these methods see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html) or {DescTools} package documentation [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html). **The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive (less commonly used) methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson, Haldane and Jeffreys-Perks.\n\n#### Example code\n\nWith 2 groups, the null hypothesis that the proportions in each group are the same and a 2-sided CI.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncount_dat <- indat |>\n count(trtn, respn)\ncount_dat\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 4 × 3\n trtn respn n\n \n1 0 0 118\n2 0 1 36\n3 1 0 65\n4 1 1 12\n```\n\n\n:::\n\n```{.r .cell-code}\n# BinomDiffCI requires\n# x1 = successes in active, n1 = total subjects in active,\n# x2 = successes in placebo, n2 = total subjects in placebo\n\nDescTools::BinomDiffCI(\n x1 = 36,\n n1 = 154,\n x2 = 12,\n n2 = 77,\n conf.level = 0.95,\n sides = c(\"two.sided\"),\n method = c(\n \"wald\",\n \"waldcc\",\n \"score\",\n \"scorecc\",\n \"ac\",\n \"mn\",\n \"mee\",\n \"blj\",\n \"ha\",\n \"hal\",\n \"jp\"\n )\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n est lwr.ci upr.ci\nwald 0.07792208 -0.02710792 0.1829521\nwaldcc 0.07792208 -0.03684818 0.1926923\nscore 0.07792208 -0.03614191 0.1751254\nscorecc 0.07792208 -0.04396244 0.1809901\nac 0.07792208 -0.03292487 0.1781699\nmn 0.07792208 -0.03606525 0.1774952\nmee 0.07792208 -0.03580439 0.1772849\nblj 0.07792208 -0.03062424 0.1810792\nha 0.07792208 -0.03415013 0.1899943\nhal 0.07792208 -0.03143484 0.1769438\njp 0.07792208 -0.03207751 0.1776615\n```\n\n\n:::\n:::\n\n\n## Methods for Calculating Confidence Intervals for Relative Risk from 2 independent samples\n\n\\[TBC\\]\n\n### \n\n## Methods for Calculating Confidence Intervals for Odds Ratio from 2 independent samples\n\n\\[TBC\\]\n\n## Continuity Adjusted Methods\n\n\\[TBC\\]\n\n## Consistency with hypothesis tests\n\n\\[TBC\\] - cf. chi-squared tests\n\n## References\n\n1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.)\n2. [PropCIs package](https://cran.r-project.org/web//packages/PropCIs/PropCIs.pdf)\n3. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000.\n4. \n", + "supporting": [], + "filters": [ + "rmarkdown/pagebreak.lua" + ], + "includes": {}, + "engineDependencies": {}, + "preserve": {}, + "postProcess": true + } +} \ No newline at end of file diff --git a/_freeze/R/ci_for_paired_prop/execute-results/html.json b/_freeze/R/ci_for_paired_prop/execute-results/html.json new file mode 100644 index 000000000..dcf053a2c --- /dev/null +++ b/_freeze/R/ci_for_paired_prop/execute-results/html.json @@ -0,0 +1,15 @@ +{ + "hash": "ca6cc539eac61a522b4e96ffafd6389c", + "result": { + "engine": "knitr", + "markdown": "---\ntitle: \"Confidence Intervals for Paired Proportions in R\"\n---\n\n## Introduction\n\n\\[See separate page for general introductory information on confidence intervals for proportions.\\]\n\n\\[Note: information about cicalc package will be added to this page soon.\\]\n\n## Data used\n\nThe adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `ACT` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4.\n\n\n\nThe below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.234 (23.4% responders).\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 4 × 3\n# Groups: trt [2]\n trt resp n\n \n1 ACT No 118\n2 ACT Yes 36\n3 PBO No 65\n4 PBO Yes 12\n```\n\n\n:::\n:::\n\n\n## Packages\n\n**The {ratesci} package** is ... \\[TBC\\]\n\n**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs).\n\n## Methods for Calculating Confidence Intervals for Proportion Difference from matched pairs using {ExactCIdiff} and {ratesci}\n\nFor more information about the detailed methods for calculating confidence intervals for a matched pair proportion see [here](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html#methods-for-calculating-confidence-intervals-for-a-matched-pair-proportion). When you have 2 measurements on the same subject, the 2 sets of measures are not independent and you have matched pair of responses.\n\nTo date we have not found an R package which calculates a CI for matched pair proportions using the normal approximation or Wilson methods although they can be done by hand using the equations provided on the SAS page link above.\n\n**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs), claiming to be the first package in R to do this method. However, it should only be used when the sample size is not too large as it can be computationally intensive.\\\nNOTE that the {ExactNumCI} package should not be used for this task. More detail on these two packages can be found [here](RJ-2013-026.pdf).\n\nUsing a cross over study as our example, a 2 x 2 table can be formed as follows:\n\n+-----------------------+---------------+---------------+---------------+\n| | Placebo\\ | Placebo\\ | Total |\n| | Response= Yes | Response = No | |\n+=======================+===============+===============+===============+\n| Active Response = Yes | r | s | r+s |\n+-----------------------+---------------+---------------+---------------+\n| Active Response = No | t | u | t+u |\n+-----------------------+---------------+---------------+---------------+\n| Total | r+t | s+u | N = r+s+t+u |\n+-----------------------+---------------+---------------+---------------+\n\n: The proportions of subjects responding on each treatment are:\n\nActive: $\\hat p_1 = (r+s)/n$ and Placebo: $\\hat p_2= (r+t)/n$\n\nDifference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$\n\nSuppose :\n\n+-----------------------+---------------+---------------+------------------+\n| | Placebo\\ | Placebo\\ | Total |\n| | Response= Yes | Response = No | |\n+=======================+===============+===============+==================+\n| Active Response = Yes | r = 20 | s = 15 | r+s = 35 |\n+-----------------------+---------------+---------------+------------------+\n| Active Response = No | t = 6 | u = 5 | t+u = 11 |\n+-----------------------+---------------+---------------+------------------+\n| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 |\n+-----------------------+---------------+---------------+------------------+\n\nActive: $\\hat p_1 = (r+s)/n$ =35/46 =0.761 and Placebo: $\\hat p_2= (r+t)/n$ = 26/46 =0.565\n\nDifference = 0.761-0.565 = 0.196, then PairedCI() function can provide an exact confidence interval as shown below\n\n-0.00339 to 0.38065\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# ExactCIdiff::PairedCI(s, r+u, t, conf.level = 0.95)\n\nCI <- ExactCIdiff::PairedCI(15, 25, 6, conf.level = 0.95)$ExactCI\nCI\n```\n:::\n\n\n## Methods for Calculating Confidence Intervals for Relative Risk from matched pairs using {ratesci}\n\n\\[TBC\\]\n\n## Methods for Calculating Confidence Intervals for Conditional Odds Ratio from matched pairs using {ratesci}\n\n\\[TBC\\]\n\n## Continuity Adjusted Methods\n\n\\[TBC\\]\n\n## Consistency with Hypothesis Tests\n\n\\[TBC\\] - cf. McNemar test\n\n## \n\n## \n\n## References\n\n1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.)\n2. [PropCIs package](https://cran.r-project.org/web//packages/PropCIs/PropCIs.pdf)\n3. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000.\n4. \n", + "supporting": [], + "filters": [ + "rmarkdown/pagebreak.lua" + ], + "includes": {}, + "engineDependencies": {}, + "preserve": {}, + "postProcess": true + } +} \ No newline at end of file diff --git a/_freeze/R/ci_for_prop/execute-results/html.json b/_freeze/R/ci_for_prop/execute-results/html.json index 3261ad57f..26e4168e7 100644 --- a/_freeze/R/ci_for_prop/execute-results/html.json +++ b/_freeze/R/ci_for_prop/execute-results/html.json @@ -1,9 +1,11 @@ { - "hash": "7544235e7d33c69b56f9d8e5ecaca819", + "hash": "351ac751d726bcae0e5f27b4fd5adca4", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Confidence Intervals for Proportions in R\"\n---\n\n## Introduction\n\nThe methods to use for calculating a confidence interval (CI) for a proportion depend on the type of proportion you have.\n\n- 1 sample proportion (1 proportion calculated from 1 group of subjects)\n\n- 2 sample proportions and you want a CI for the difference in the 2 proportions.\n\n - If the 2 samples come from 2 independent samples (different subjects in each of the 2 groups)\n\n - If the 2 samples are matched (i.e. the same subject has 2 results, one on each group \\[paired data\\]).\n\nThe method selected is also dependent on whether your proportion is close to 0 or 1 (or near to the 0.5 midpoint), and your sample size.\n\nFor more information about these methods, including which performs better in different scenarios see [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f)^1^.\n\nNote: information about cicalc package will be added to this page soon.\n\n## Data used\n\nThe adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `ACT` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4.\n\n\n\nThe below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.234 (23.4% responders).\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 4 × 3\n# Groups: trt [2]\n trt resp n\n \n1 ACT No 118\n2 ACT Yes 36\n3 PBO No 65\n4 PBO Yes 12\n```\n\n\n:::\n:::\n\n\n## Packages\n\n**The {cardx} package** is an extension of the {cards} package, providing additional functions to create Analysis Results Data Objects (ARDs)^1^. It was developed as part of {NEST} and pharmaverse. This package requires the binary endpoint to be a logical (TRUE/FALSE) vector or a numeric/integer coded as (0, 1) with 1 (TRUE) being the success you want to calculate the confidence interval for.\n\nSee [here](R:%20Functions%20for%20Calculating%20Proportion%20Confidence%20Intervals) for full description of the {cardx} proportions equations.\n\nIf calculating the CI for a difference in proportions, the package requires both the response and the treatment variable to be numeric/integer coded as (0, 1) (or logical vector).\n\nInstead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions.\n\n**The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here.\n\n**The {Hmisc} package** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. Documentation reports that the method uses F distribution to compute exact intervals based on the binomial cdf. However, if the percentage of responders is 100% then the upper limit is set to 1. Similarly if the percentage of responders is 0%, then the lower limit is set to 0. Hence, in extreme cases there may be differences between this package and the standard implementation of Clopper-Pearson method.\n\n**The {RBesT} package (Prior to Version 1.8-0)** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. However, as described below, there are 2 cases where the results using RBesT package do not match cardx or Hmisc.\n\n1) x = 0 (0% responders), in which case the lower limit does not match.\n2) x = n (100% responders), in which case the upper limit does not match.\n\nBecause of the relationship between the binomial distribution and the beta distribution. This package uses quantiles of the beta distribution to derive exact confidence intervals.\n\n$$ B(\\alpha/2;x, n-x+1) < p < B(1-\\alpha/2; x+1, n-x)$$\n\nRBesT equations are:\\\npLow \\<- qbeta(Low, r + (r == 0), n - r + 1)\\\npHigh \\<- qbeta(High, r + 1, n - r + ((n - r) == 0))\n\nIn Version 1.8-0 onwards the equations were updated as follows, which then match the Hmisc intervals:\\\npLow \\<- qbeta(Low, r, n - r + 1)\\\npHigh \\<- qbeta(High, r + 1, n - r)\n\n**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs).\n\n**The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson and Haldane. See [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html) for more detail.\n\n**The {presize} package** has a function prec_prop() which also calculates CIs for 2 independent samples using the Wilson, Agresti-Coull, Exact or Wald approaches. The package is not described in further detail here since in most cases **{DescTools}** will be able to compute what is needed. However, it's mentioned due to other functionality it has available such as sample size and precision calculations for AUC, correlations, cronbach's alpha, intraclass correlation, Cohen's kappa, likelihood ratios, means, mean differences, odds ratios, rates, rate ratios, risk differences and risk ratios.\n\n## Methods for Calculating Confidence Intervals for a single proportion using cardx\n\nFor more technical derivation and reasons for use of each of the methods listed below, see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html).\n\nLet's start by calculating a Confidence interval for the proportion of successes observed in the Active Treatment group (a single sample).\n\n### Clopper-Pearson (Exact or binomial CI) Method\n\nClopper-Pearson Exact CI is one of the most popular methods, it is often good for small sample sizes when the proportion is not close to the tails (0,1), but it can be too conservative (too wide an interval compared to the interval containing the true population proportion 95% of the time).\n\nThe cardx package calculates the Clopper-Pearson score by calling stats::binom.test() function.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncardx::proportion_ci_clopper_pearson(act2, conf.level = 0.95) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 11\n N n conf.level estimate statistic p.value parameter conf.low\n \n1 154 36 0.95 0.234 36 2.21e-11 154 0.169\n# ℹ 3 more variables: conf.high , method , alternative \n```\n\n\n:::\n:::\n\n\n### Normal Approximation Method (Also known as the Wald or asymptotic CI Method)\n\nIn large random samples from independent trials, the sampling distribution of proportions approximately follows the normal distribution. The expectation of a sample proportion is the corresponding population proportion. Therefore, based on a sample of size $n$, a $(1-\\alpha)\\%$ confidence interval for population proportion can be calculated using normal approximation as follows:\n\n$p\\approx \\hat p \\pm z_\\alpha \\sqrt{\\hat p(1-\\hat p)}/{n}$, where $\\hat p$ is the sample proportion, $z_\\alpha$ is the $1-\\alpha/2$ quantile of a standard normal distribution corresponding to level $\\alpha$, and $\\sqrt{\\hat p(1-\\hat p)}/{n}$ is the standard error.\n\nFor more technical information see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html).\n\n#### Example code\n\nThe following code calculates a confidence interval for a binomial proportion using normal approximation equation manually. This is replicated exactly using the `cardx::proportion_ci_wald function` which also allows the continuity correction to be applied.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# sample proportion by trt\nsummary <- adcibc |>\n filter(trt == \"ACT\") |>\n group_by(resp) |>\n tally() |>\n ungroup() |>\n mutate(\n total = sum(n),\n p = n / total\n )\n\n# Calculate standard error and 95% wald confidence intervals for population proportion\nwaldci <- summary |>\n filter(resp == \"Yes\") |>\n mutate(\n se = sqrt(p * (1 - p) / total),\n lower_ci = (p - qnorm(1 - 0.05 / 2) * se),\n upper_ci = (p + qnorm(1 - 0.05 / 2) * se)\n )\nwaldci\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n resp n total p se lower_ci upper_ci\n \n1 Yes 36 154 0.234 0.0341 0.167 0.301\n```\n\n\n:::\n\n```{.r .cell-code}\n# cardx package Wald method without continuity correction\ncardx::proportion_ci_wald(act2, conf.level = 0.95, correct = FALSE) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n N n estimate conf.low conf.high conf.level method \n \n1 154 36 0.234 0.167 0.301 0.95 Wald Confidence Interval w…\n```\n\n\n:::\n\n```{.r .cell-code}\n# cardx package Wald method with continuity correction\ncardx::proportion_ci_wald(act2, conf.level = 0.95, correct = TRUE) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n N n estimate conf.low conf.high conf.level method \n \n1 154 36 0.234 0.164 0.304 0.95 Wald Confidence Interval w…\n```\n\n\n:::\n:::\n\n\n### Wilson Method (Also known as the Score method or the Altman, Newcombe method^3^ )\n\nThe cardx package calculates the Wilson (score) method by calling stats::prop.test() function. This method is often used as a compromise between the Clopper-Pearson and the Wald given it was found to be accurate for most parameter values (even those close to 0 and 1), and it does not suffer from being over-conservative. For more technical information see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html).\n\nThe package also contains a function for proportion_ci_strat_wilson() which calculates the stratified Wilson CIs for unequal proportions as described on page 47 [here](https://cran.r-universe.dev/cardx/cardx.pdf).\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# cardx package Wilson method without continuity correction\ncardx::proportion_ci_wilson(act2, conf.level = 0.95, correct = FALSE) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 11\n N n conf.level estimate statistic p.value parameter conf.low\n \n1 154 36 0.95 0.234 43.7 3.90e-11 1 0.174\n# ℹ 3 more variables: conf.high , method , alternative \n```\n\n\n:::\n\n```{.r .cell-code}\n# cardx package Wilson method with continuity correction\ncardx::proportion_ci_wilson(act2, conf.level = 0.95, correct = TRUE) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 11\n N n conf.level estimate statistic p.value parameter conf.low\n \n1 154 36 0.95 0.234 42.6 6.70e-11 1 0.171\n# ℹ 3 more variables: conf.high , method , alternative \n```\n\n\n:::\n:::\n\n\n### Agresti-Coull Method\n\nThe cardx package calculates the Agresti-Coull method using the equation from the published method by Alan Agresti & Brent Coull based on adding 2 successes and 2 failures before computing the wald CI. The CI is truncated, when it overshoots the boundary (\\<0 or \\>1).\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# cardx package agresti_coull method\ncardx::proportion_ci_agresti_coull(act2, conf.level = 0.95) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n N n estimate conf.low conf.high conf.level method \n \n1 154 36 0.234 0.174 0.307 0.95 Agresti-Coull Confidence I…\n```\n\n\n:::\n:::\n\n\n### Jeffreys Method\n\nJeffreys method is a particular type of Bayesian Highest Probability Density (HPD) Method. For proportions, the beta distribution is generally used for the prior, which consists of two parameters alpha and beta. Setting alpha=beta=0.5 is called Jeffrey's prior. NOTE: if you want to use any other priors, you can use `binom.bayes` which estimates a credible interval for proportions.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# cardx package jeffreys method\ncardx::proportion_ci_jeffreys(act2, conf.level = 0.95) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n N n estimate conf.low conf.high conf.level method \n \n1 154 36 0.234 0.172 0.305 0.95 Jeffreys Interval\n```\n\n\n:::\n:::\n\n\n## Methods for Calculating Confidence Intervals for a matched pair proportion using {ExactCIdiff}\n\nFor more information about the detailed methods for calculating confidence intervals for a matched pair proportion see [here](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html#methods-for-calculating-confidence-intervals-for-a-matched-pair-proportion). When you have 2 measurements on the same subject, the 2 sets of measures are not independent and you have matched pair of responses.\n\nTo date we have not found an R package which calculates a CI for matched pair proportions using the normal approximation or Wilson methods although they can be done by hand using the equations provided on the SAS page link above.\n\n**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs), claiming to be the first package in R to do this method. However, it should only be used when the sample size is not too large as it can be computationally intensive.\\\nNOTE that the {ExactNumCI} package should not be used for this task. More detail on these two packages can be found [here](RJ-2013-026.pdf).\n\nUsing a cross over study as our example, a 2 x 2 table can be formed as follows:\n\n+-----------------------+---------------+---------------+---------------+\n| | Placebo\\ | Placebo\\ | Total |\n| | Response= Yes | Response = No | |\n+=======================+===============+===============+===============+\n| Active Response = Yes | r | s | r+s |\n+-----------------------+---------------+---------------+---------------+\n| Active Response = No | t | u | t+u |\n+-----------------------+---------------+---------------+---------------+\n| Total | r+t | s+u | N = r+s+t+u |\n+-----------------------+---------------+---------------+---------------+\n\n: The proportions of subjects responding on each treatment are:\n\nActive: $\\hat p_1 = (r+s)/n$ and Placebo: $\\hat p_2= (r+t)/n$\n\nDifference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$\n\nSuppose :\n\n+-----------------------+---------------+---------------+------------------+\n| | Placebo\\ | Placebo\\ | Total |\n| | Response= Yes | Response = No | |\n+=======================+===============+===============+==================+\n| Active Response = Yes | r = 20 | s = 15 | r+s = 35 |\n+-----------------------+---------------+---------------+------------------+\n| Active Response = No | t = 6 | u = 5 | t+u = 11 |\n+-----------------------+---------------+---------------+------------------+\n| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 |\n+-----------------------+---------------+---------------+------------------+\n\nActive: $\\hat p_1 = (r+s)/n$ =35/46 =0.761 and Placebo: $\\hat p_2= (r+t)/n$ = 26/46 =0.565\n\nDifference = 0.761-0.565 = 0.196, then PairedCI() function can provide an exact confidence interval as shown below\n\n-0.00339 to 0.38065\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# ExactCIdiff::PairedCI(s, r+u, t, conf.level = 0.95)\n\nCI <- ExactCIdiff::PairedCI(15, 25, 6, conf.level = 0.95)$ExactCI\nCI\n```\n:::\n\n\n## Methods for Calculating Confidence Intervals for 2 independent samples proportion\n\nThis [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^4^ describes many methods for the calculation of confidence intervals for 2 independent proportions.\n\n### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) using {cardx}\n\nFor more technical information regarding the Wald method see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html).\n\n#### Example code\n\n`cardx::ard_stats_prop_test function` uses `stats::prop.test` which also allows a continuity correction to be applied.\n\nAlthough this website [here](https://rdrr.io/r/stats/prop.test.html) and this one [here](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/prop.test) both reference Newcombe for the CI that this function uses, replication of the results by hand and compared to SAS show that the results below match the Normal Approximation (Wald method).\n\nBoth the Treatment variable (ACT,PBO) and the Response variable (Yes,No) have to be numeric (0,1) or Logit (TRUE,FALSE) variables.\n\nThe prop.test default with 2 groups, is the null hypothesis that the proportions in each group are the same and a 2-sided CI.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nindat1 <- adcibc2 |>\n select(AVAL, TRTP) |>\n mutate(\n resp = if_else(AVAL > 4, \"Yes\", \"No\"),\n respn = if_else(AVAL > 4, 1, 0),\n trt = if_else(TRTP == \"Placebo\", \"PBO\", \"ACT\"),\n trtn = if_else(TRTP == \"Placebo\", 1, 0)\n ) |>\n select(trt, trtn, resp, respn)\n\n# cardx package required a vector with 0 and 1s for a single proportion CI\n# To get the comparison the correct way around Placebo must be 1, and Active 0\n\nindat <- select(indat1, trtn, respn)\n\ncardx::ard_stats_prop_test(\n data = indat,\n by = trtn,\n variables = respn,\n conf.level = 0.95,\n correct = FALSE\n)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\n{cards} data frame: 13 x 9\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n group1 variable context stat_name stat_label stat\n1 trtn respn stats_pr… estimate Rate Dif… 0.078\n2 trtn respn stats_pr… estimate1 Group 1 … 0.234\n3 trtn respn stats_pr… estimate2 Group 2 … 0.156\n4 trtn respn stats_pr… statistic X-square… 1.893\n5 trtn respn stats_pr… p.value p-value 0.169\n6 trtn respn stats_pr… parameter Degrees … 1\n7 trtn respn stats_pr… conf.low CI Lower… -0.027\n8 trtn respn stats_pr… conf.high CI Upper… 0.183\n9 trtn respn stats_pr… method method 2-sample…\n10 trtn respn stats_pr… alternative alternat… two.sided\n11 trtn respn stats_pr… p p \n12 trtn respn stats_pr… conf.level CI Confi… 0.95\n13 trtn respn stats_pr… correct Yates' c… FALSE\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr}\n\n```\nℹ 3 more variables: fmt_fun, warning, error\n```\n\n\n:::\n\n```{.r .cell-code}\ncardx::ard_stats_prop_test(\n data = indat,\n by = trtn,\n variables = respn,\n conf.level = 0.95,\n correct = TRUE\n)\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\n{cards} data frame: 13 x 9\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n group1 variable context stat_name stat_label stat\n1 trtn respn stats_pr… estimate Rate Dif… 0.078\n2 trtn respn stats_pr… estimate1 Group 1 … 0.234\n3 trtn respn stats_pr… estimate2 Group 2 … 0.156\n4 trtn respn stats_pr… statistic X-square… 1.45\n5 trtn respn stats_pr… p.value p-value 0.229\n6 trtn respn stats_pr… parameter Degrees … 1\n7 trtn respn stats_pr… conf.low CI Lower… -0.037\n8 trtn respn stats_pr… conf.high CI Upper… 0.193\n9 trtn respn stats_pr… method method 2-sample…\n10 trtn respn stats_pr… alternative alternat… two.sided\n11 trtn respn stats_pr… p p \n12 trtn respn stats_pr… conf.level CI Confi… 0.95\n13 trtn respn stats_pr… correct Yates' c… TRUE\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stderr}\n\n```\nℹ 3 more variables: fmt_fun, warning, error\n```\n\n\n:::\n:::\n\n\n### Normal Approximation (Wald) and Other Methods for 2 independent samples using {DescTools}\n\nFor more technical information regarding the derivations of these methods see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html) or {DescTools} package documentation [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html). **The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive (less commonly used) methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson, Haldane and Jeffreys-Perks.\n\n#### Example code\n\nWith 2 groups, the null hypothesis that the proportions in each group are the same and a 2-sided CI.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncount_dat <- indat |>\n count(trtn, respn)\ncount_dat\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 4 × 3\n trtn respn n\n \n1 0 0 118\n2 0 1 36\n3 1 0 65\n4 1 1 12\n```\n\n\n:::\n\n```{.r .cell-code}\n# BinomDiffCI requires\n# x1 = successes in active, n1 = total subjects in active,\n# x2 = successes in placebo, n2 = total subjects in placebo\n\nDescTools::BinomDiffCI(\n x1 = 36,\n n1 = 154,\n x2 = 12,\n n2 = 77,\n conf.level = 0.95,\n sides = c(\"two.sided\"),\n method = c(\n \"wald\",\n \"waldcc\",\n \"score\",\n \"scorecc\",\n \"ac\",\n \"mn\",\n \"mee\",\n \"blj\",\n \"ha\",\n \"hal\",\n \"jp\"\n )\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n est lwr.ci upr.ci\nwald 0.07792208 -0.02710792 0.1829521\nwaldcc 0.07792208 -0.03684818 0.1926923\nscore 0.07792208 -0.03614191 0.1751254\nscorecc 0.07792208 -0.04396244 0.1809901\nac 0.07792208 -0.03292487 0.1781699\nmn 0.07792208 -0.03606525 0.1774952\nmee 0.07792208 -0.03580439 0.1772849\nblj 0.07792208 -0.03062424 0.1810792\nha 0.07792208 -0.03415013 0.1899943\nhal 0.07792208 -0.03143484 0.1769438\njp 0.07792208 -0.03207751 0.1776615\n```\n\n\n:::\n:::\n\n\n## References\n\n1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.)\n2. [PropCIs package](https://cran.r-project.org/web//packages/PropCIs/PropCIs.pdf)\n3. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000.\n4. \n", - "supporting": [], + "markdown": "---\ntitle: \"Confidence Intervals for a Proportion in R\"\n---\n\n## Introduction\n\n\\[See separate page for general introductory information on confidence intervals for proportions.\\]\n\n\\[Note: information about cicalc package will be added to this page soon.\\]\n\n## Data used\n\nThe adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `ACT` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4.\n\n\n\nThe below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.234 (23.4% responders).\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 4 × 3\n# Groups: trt [2]\n trt resp n\n \n1 ACT No 118\n2 ACT Yes 36\n3 PBO No 65\n4 PBO Yes 12\n```\n\n\n:::\n:::\n\n\n## Packages\n\n**The {cardx} package** is an extension of the {cards} package, providing additional functions to create Analysis Results Data Objects (ARDs)^1^. It was developed as part of {NEST} and pharmaverse. This package requires the binary endpoint to be a logical (TRUE/FALSE) vector or a numeric/integer coded as (0, 1) with 1 (TRUE) being the success you want to calculate the confidence interval for.\n\nSee [here](R:%20Functions%20for%20Calculating%20Proportion%20Confidence%20Intervals) for full description of the {cardx} proportions equations.\n\nIf calculating the CI for a difference in proportions, the package requires both the response and the treatment variable to be numeric/integer coded as (0, 1) (or logical vector).\n\nInstead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions.\n\n**The {ratesci} package** is ... \\[TBC\\] - note, current development version at has new features in the rateci() function (including more of the CI methods described below) compared to the CRAN release.\n\n**The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here.\n\n**The {Hmisc} package** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. Documentation reports that the method uses F distribution to compute exact intervals based on the binomial cdf. However, if the percentage of responders is 100% then the upper limit is set to 1. Similarly if the percentage of responders is 0%, then the lower limit is set to 0. Hence, in extreme cases there may be differences between this package and the standard implementation of Clopper-Pearson method.\n\n**The {RBesT} package (Prior to Version 1.8-0)** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. However, as described below, there are 2 cases where the results using RBesT package do not match cardx or Hmisc.\n\n1) x = 0 (0% responders), in which case the lower limit does not match.\n2) x = n (100% responders), in which case the upper limit does not match.\n\nBecause of the relationship between the binomial distribution and the beta distribution. This package uses quantiles of the beta distribution to derive exact confidence intervals.\n\n$$ B(\\alpha/2;x, n-x+1) < p < B(1-\\alpha/2; x+1, n-x)$$\n\nRBesT equations are:\\\npLow \\<- qbeta(Low, r + (r == 0), n - r + 1)\\\npHigh \\<- qbeta(High, r + 1, n - r + ((n - r) == 0))\n\nIn Version 1.8-0 onwards the equations were updated as follows, which then match the Hmisc intervals:\\\npLow \\<- qbeta(Low, r, n - r + 1)\\\npHigh \\<- qbeta(High, r + 1, n - r)\n\n**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs).\n\n**The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson and Haldane. See [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html) for more detail.\n\n**The {presize} package** has a function prec_prop() which also calculates CIs for 2 independent samples using the Wilson, Agresti-Coull, Exact or Wald approaches. The package is not described in further detail here since in most cases **{DescTools}** will be able to compute what is needed. However, it's mentioned due to other functionality it has available such as sample size and precision calculations for AUC, correlations, cronbach's alpha, intraclass correlation, Cohen's kappa, likelihood ratios, means, mean differences, odds ratios, rates, rate ratios, risk differences and risk ratios.\n\n## Methods for Calculating Confidence Intervals for a single proportion using cardx\n\nFor more technical derivation and reasons for use of each of the methods listed below, see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html).\n\nLet's start by calculating a Confidence interval for the proportion of successes observed in the Active Treatment group (a single sample).\n\n### Clopper-Pearson (Exact or binomial CI) Method\n\nClopper-Pearson Exact CI is one of the most popular methods, it is often good for small sample sizes when the proportion is not close to the tails (0,1), but it can be too conservative (too wide an interval compared to the interval containing the true population proportion 95% of the time).\n\nThe cardx package calculates the Clopper-Pearson score by calling stats::binom.test() function.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncardx::proportion_ci_clopper_pearson(act2, conf.level = 0.95) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 11\n N n conf.level estimate statistic p.value parameter conf.low\n \n1 154 36 0.95 0.234 36 2.21e-11 154 0.169\n# ℹ 3 more variables: conf.high , method , alternative \n```\n\n\n:::\n:::\n\n\n### Normal Approximation Method (Also known as the Wald Method)\n\nIn large random samples from independent trials, the sampling distribution of proportions approximately follows the normal distribution. The expectation of a sample proportion is the corresponding population proportion. Therefore, based on a sample of size $n$, a $(1-\\alpha)\\%$ confidence interval for population proportion can be calculated using normal approximation as follows:\n\n$p\\approx \\hat p \\pm z_\\alpha \\sqrt{\\hat p(1-\\hat p)}/{n}$, where $\\hat p$ is the sample proportion, $z_\\alpha$ is the $1-\\alpha/2$ quantile of a standard normal distribution corresponding to level $\\alpha$, and $\\sqrt{\\hat p(1-\\hat p)}/{n}$ is the standard error.\n\nFor more technical information see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html).\n\n#### Example code\n\nThe following code calculates a confidence interval for a binomial proportion using normal approximation equation manually. This is replicated exactly using the `cardx::proportion_ci_wald function` which also allows the continuity correction to be applied.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# sample proportion by trt\nsummary <- adcibc |>\n filter(trt == \"ACT\") |>\n group_by(resp) |>\n tally() |>\n ungroup() |>\n mutate(\n total = sum(n),\n p = n / total\n )\n\n# Calculate standard error and 95% wald confidence intervals for population proportion\nwaldci <- summary |>\n filter(resp == \"Yes\") |>\n mutate(\n se = sqrt(p * (1 - p) / total),\n lower_ci = (p - qnorm(1 - 0.05 / 2) * se),\n upper_ci = (p + qnorm(1 - 0.05 / 2) * se)\n )\nwaldci\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n resp n total p se lower_ci upper_ci\n \n1 Yes 36 154 0.234 0.0341 0.167 0.301\n```\n\n\n:::\n\n```{.r .cell-code}\n# cardx package Wald method without continuity correction\ncardx::proportion_ci_wald(act2, conf.level = 0.95, correct = FALSE) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n N n estimate conf.low conf.high conf.level method \n \n1 154 36 0.234 0.167 0.301 0.95 Wald Confidence Interval w…\n```\n\n\n:::\n\n```{.r .cell-code}\n# cardx package Wald method with continuity correction\ncardx::proportion_ci_wald(act2, conf.level = 0.95, correct = TRUE) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n N n estimate conf.low conf.high conf.level method \n \n1 154 36 0.234 0.164 0.304 0.95 Wald Confidence Interval w…\n```\n\n\n:::\n:::\n\n\n### Wilson Method (Also known as the Score method or the Altman, Newcombe method^3^ )\n\nThe cardx package calculates the Wilson (score) method by calling stats::prop.test() function. This method is often used as a compromise between the Clopper-Pearson and the Wald given it was found to be accurate for most parameter values (even those close to 0 and 1), and it does not suffer from being over-conservative. For more technical information see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html).\n\nThe package also contains a function for proportion_ci_strat_wilson() which calculates the stratified Wilson CIs for unequal proportions as described on page 47 [here](https://cran.r-universe.dev/cardx/cardx.pdf).\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# cardx package Wilson method without continuity correction\ncardx::proportion_ci_wilson(act2, conf.level = 0.95, correct = FALSE) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 11\n N n conf.level estimate statistic p.value parameter conf.low\n \n1 154 36 0.95 0.234 43.7 3.90e-11 1 0.174\n# ℹ 3 more variables: conf.high , method , alternative \n```\n\n\n:::\n\n```{.r .cell-code}\n# cardx package Wilson method with continuity correction\ncardx::proportion_ci_wilson(act2, conf.level = 0.95, correct = TRUE) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 11\n N n conf.level estimate statistic p.value parameter conf.low\n \n1 154 36 0.95 0.234 42.6 6.70e-11 1 0.171\n# ℹ 3 more variables: conf.high , method , alternative \n```\n\n\n:::\n:::\n\n\n### Agresti-Coull Method\n\nThe cardx package calculates the Agresti-Coull method using the equation from the published method by Alan Agresti & Brent Coull based on adding 2 successes and 2 failures before computing the wald CI. The CI is truncated, when it overshoots the boundary (\\<0 or \\>1).\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# cardx package agresti_coull method\ncardx::proportion_ci_agresti_coull(act2, conf.level = 0.95) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n N n estimate conf.low conf.high conf.level method \n \n1 154 36 0.234 0.174 0.307 0.95 Agresti-Coull Confidence I…\n```\n\n\n:::\n:::\n\n\n### Jeffreys Method\n\nJeffreys method is a particular type of Bayesian Highest Probability Density (HPD) Method. For proportions, the beta distribution is generally used for the prior, which consists of two parameters alpha and beta. Setting alpha=beta=0.5 is called Jeffrey's prior. NOTE: if you want to use any other priors, you can use `binom.bayes` which estimates a credible interval for proportions.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# cardx package jeffreys method\ncardx::proportion_ci_jeffreys(act2, conf.level = 0.95) |>\n as_tibble()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 7\n N n estimate conf.low conf.high conf.level method \n \n1 154 36 0.234 0.172 0.305 0.95 Jeffreys Interval\n```\n\n\n:::\n:::\n\n\n## Continuity Adjusted Methods\n\n\\[TBC\\]\n\n## Consistency with hypothesis tests\n\n\\[TBC\\]\n\n## References\n\n1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.)\n2. [PropCIs package](https://cran.r-project.org/web//packages/PropCIs/PropCIs.pdf)\n3. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000.\n4. \n", + "supporting": [ + "ci_for_prop_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/R/friedman_test/execute-results/html.json b/_freeze/R/friedman_test/execute-results/html.json index a1629e8cf..9a59dabdc 100644 --- a/_freeze/R/friedman_test/execute-results/html.json +++ b/_freeze/R/friedman_test/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "d288eb607ad4bec254cf11e0bd79d26e", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Friedman Test Analysis\"\ndate: \"2025-04-09\"\n---\n\n## Friedman Test Analysis\n\nFriedman's test is a non-parametric statistical test used to detect differences in treatments across multiple test attempts. It is often used when the assumptions of ANOVA are not met, particularly the assumption of normality. The test is applicable for repeated measures, or matched groups, making it useful for situations where the same subjects are subjected to different treatments.\n\nFriedman’s test ranks the data points within each block (or subject) separately, and then analyzes these ranks to see if the mean ranks differ between the groups and conditions. If the test shows significant differences, this suggests that at least one of the treatments differs from the others. Because it is non-parametric, it does not assume the normal distribution of data, which makes it robust for skewed or ordinal data.\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Load required packages\nlibrary(tidyverse)\nlibrary(broom)\nlibrary(rstatix)\n```\n:::\n\n\n\n### Hypotheses\n\nH₀ (Null Hypothesis): There are no significant differences in weight outcomes between the three diets\n\nH₁ (Alternative Hypothesis): There are significant differences in weight outcomes between at least two diets \n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Create the dataset\nDiet_A = c(75, 68, 80, 72, 85, 70, 82, 78, 75, 83)\n\nDiet_B = c(82, 70, 85, 78, 88, 75, 85, 80, 79, 87)\n\nDiet_C = c(78, 65, 82, 75, 84, 72, 80, 76, 77, 84)\n\ndata <- tibble(\n subjid = rep(1:10, 3),\n diet = rep(c(\"A\", \"B\", \"C\"), each = 10),\n weight = c(Diet_A, Diet_B, Diet_C)\n)\n```\n:::\n\n\n### Base R {stats}\n\nTo run a Friedman's test in R you can use the {stats} package. This will return the chi-squared test statistic and p-value. \n\n::: {.cell}\n\n```{.r .cell-code}\n# Perform Friedman test\nfriedman_test <- stats::friedman.test(weight ~ diet | subjid, data = data)\nfriedman_test |>\n broom::tidy()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 4\n statistic p.value parameter method \n \n1 15.2 0.000500 2 Friedman rank sum test\n```\n\n\n:::\n:::\n\n\n### {rstatix}\n\nAlternatively, you can use the {rstatix} package. While these packages give the same results, the {rstatix} results come as a tibble we can easily use. \n\n::: {.cell}\n\n```{.r .cell-code}\ntest <- data |>\n rstatix::friedman_test(weight ~ diet | subjid)\ntest\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 6\n .y. n statistic df p method \n* \n1 weight 10 15.2 2 0.000500 Friedman test\n```\n\n\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Create boxplot\nggplot(data, aes(x = diet, y = weight, fill = diet)) +\n geom_boxplot() +\n theme_minimal() +\n labs(\n title = \"Weight Distribution Across Different Diets\",\n x = \"Diet Type\",\n y = \"Weight\"\n )\n```\n\n::: {.cell-output-display}\n![](friedman_test_files/figure-html/visualization-1.png){width=768}\n:::\n:::\n\n\n### Conclusion\n\nBased on the analysis:\n\n1. **Statistical Test Results:**\n\n- The Friedman test yielded a p-value of 5.0045143\\times 10^{-4}\n\n\\[If p \\< 0.05, we reject the null hypothesis\n\nIf p \\> 0.05, we fail to reject the null hypothesis\\]\n\n2. **Visual Analysis:**\n\n- From the boxplot, Diet B shows the highest median weight\n\n- Diet B also appears to have the highest overall weight distribution\n\n- Diet A and Diet C show similar distributions but lower than Diet B\n\n\n3. **Interpretation:**\n\n- If the goal is weight gain: Diet B appears most effective\n\n- If the goal is weight maintenance: Diet A or C might be more suitable\n\n- However, individual responses vary, as shown by the overlapping distributions\n\n## Reference\n\n*Cite all sources and references used in the analysis.*\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-23\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P abind 1.4-8 2024-09-12 [?] RSPM\n askpass 1.2.1 2024-10-04 [1] RSPM\n P backports 1.5.0 2024-05-23 [?] RSPM\n base64enc 0.1-6 2026-02-02 [1] RSPM\n bit 4.6.0 2025-03-06 [1] RSPM\n bit64 4.6.0-1 2025-01-16 [1] RSPM\n blob 1.3.0 2026-01-14 [1] RSPM\n boot 1.3-32 2025-08-29 [2] CRAN (R 4.5.2)\n P broom * 1.0.12 2026-01-27 [?] RSPM\n bslib 0.10.0 2026-01-26 [1] RSPM\n cachem 1.1.0 2024-05-16 [1] RSPM\n callr 3.7.6 2024-03-25 [1] RSPM\n P car 3.1-5 2026-02-03 [?] RSPM\n P carData 3.0-6 2026-01-30 [?] RSPM\n cellranger 1.1.0 2016-07-27 [1] RSPM\n P cli 3.6.5 2025-04-23 [?] RSPM\n clipr 0.8.0 2022-02-22 [1] RSPM\n colorspace 2.1-2 2025-09-22 [1] RSPM\n conflicted 1.2.0 2023-02-01 [1] RSPM\n corrplot 0.95 2024-10-14 [1] RSPM\n cowplot 1.2.0 2025-07-07 [1] RSPM\n cpp11 0.5.3 2026-01-20 [1] RSPM\n crayon 1.5.3 2024-06-20 [1] RSPM\n curl 7.0.0 2025-08-19 [1] RSPM\n data.table 1.18.2.1 2026-01-27 [1] RSPM\n DBI 1.2.3 2024-06-02 [1] RSPM\n dbplyr 2.5.2 2026-02-13 [1] RSPM\n Deriv 4.2.0 2025-06-20 [1] RSPM\n P digest 0.6.39 2025-11-19 [?] RSPM\n doBy 4.7.1 2025-12-02 [1] RSPM\n P dplyr * 1.2.0 2026-02-03 [?] RSPM\n dtplyr 1.3.3 2026-02-11 [1] RSPM\n P evaluate 1.0.5 2025-08-27 [?] RSPM\n P farver 2.1.2 2024-05-13 [?] RSPM\n P fastmap 1.2.0 2024-05-15 [?] RSPM\n fontawesome 0.5.3 2024-11-16 [1] RSPM\n P forcats * 1.0.1 2025-09-25 [?] RSPM\n forecast 9.0.1 2026-02-14 [1] RSPM\n P Formula 1.2-5 2023-02-24 [?] RSPM\n fracdiff 1.5-3 2024-02-01 [1] RSPM\n fs 1.6.6 2025-04-12 [1] RSPM\n gargle 1.6.1 2026-01-29 [1] RSPM\n P generics 0.1.4 2025-05-09 [?] RSPM\n P ggplot2 * 4.0.2 2026-02-03 [?] RSPM\n P glue 1.8.0 2024-09-30 [?] RSPM\n googledrive 2.1.2 2025-09-10 [1] RSPM\n googlesheets4 1.1.2 2025-09-03 [1] RSPM\n P gtable 0.3.6 2024-10-25 [?] RSPM\n haven 2.5.5 2025-05-30 [1] RSPM\n highr 0.11 2024-05-26 [1] RSPM\n P hms 1.1.4 2025-10-17 [?] RSPM\n P htmltools 0.5.9 2025-12-04 [?] RSPM\n httr 1.4.8 2026-02-13 [1] RSPM\n ids 1.0.1 2017-05-31 [1] RSPM\n isoband 0.3.0 2025-12-07 [1] RSPM\n jquerylib 0.1.4 2021-04-26 [1] RSPM\n P jsonlite 2.0.0 2025-03-27 [?] RSPM\n P knitr 1.51 2025-12-20 [?] RSPM\n P labeling 0.4.3 2023-08-29 [?] RSPM\n lattice 0.22-7 2025-04-02 [2] CRAN (R 4.5.2)\n P lifecycle 1.0.5 2026-01-08 [?] RSPM\n lme4 1.1-38 2025-12-02 [1] RSPM\n lmtest 0.9-40 2022-03-21 [1] RSPM\n P lubridate * 1.9.5 2026-02-04 [?] RSPM\n P magrittr 2.0.4 2025-09-12 [?] RSPM\n MASS 7.3-65 2025-02-28 [2] CRAN (R 4.5.2)\n Matrix 1.7-4 2025-08-28 [2] CRAN (R 4.5.2)\n MatrixModels 0.5-4 2025-03-26 [1] RSPM\n memoise 2.0.1 2021-11-26 [1] RSPM\n mgcv 1.9-3 2025-04-04 [2] CRAN (R 4.5.2)\n microbenchmark 1.5.0 2024-09-04 [1] RSPM\n mime 0.13 2025-03-17 [1] RSPM\n minqa 1.2.8 2024-08-17 [1] RSPM\n modelr 0.1.11 2023-03-22 [1] RSPM\n nlme 3.1-168 2025-03-31 [2] CRAN (R 4.5.2)\n nloptr 2.2.1 2025-03-17 [1] RSPM\n nnet 7.3-20 2025-01-01 [2] CRAN (R 4.5.2)\n numDeriv 2016.8-1.1 2019-06-06 [1] RSPM\n openssl 2.3.4 2025-09-30 [1] RSPM\n pbkrtest 0.5.5 2025-07-18 [1] RSPM\n P pillar 1.11.1 2025-09-17 [?] RSPM\n P pkgconfig 2.0.3 2019-09-22 [?] RSPM\n prettyunits 1.2.0 2023-09-24 [1] RSPM\n processx 3.8.6 2025-02-21 [1] RSPM\n progress 1.2.3 2023-12-06 [1] RSPM\n ps 1.9.1 2025-04-12 [1] RSPM\n P purrr * 1.2.1 2026-01-09 [?] RSPM\n quantreg 6.1 2025-03-10 [1] RSPM\n P R6 2.6.1 2025-02-15 [?] RSPM\n ragg 1.5.0 2025-09-02 [1] RSPM\n rappdirs 0.3.4 2026-01-17 [1] RSPM\n rbibutils 2.4.1 2026-01-21 [1] RSPM\n P RColorBrewer 1.1-3 2022-04-03 [?] RSPM\n Rcpp 1.1.1 2026-01-10 [1] RSPM\n RcppArmadillo 15.2.3-1 2025-12-17 [1] RSPM\n RcppEigen 0.3.4.0.2 2024-08-24 [1] RSPM\n Rdpack 2.6.6 2026-02-08 [1] RSPM\n P readr * 2.1.6 2025-11-14 [?] RSPM\n readxl 1.4.5 2025-03-07 [1] RSPM\n reformulas 0.4.4 2026-02-02 [1] RSPM\n rematch 2.0.0 2023-08-30 [1] RSPM\n rematch2 2.1.2 2020-05-01 [1] RSPM\n reprex 2.1.1 2024-07-06 [1] RSPM\n P rlang 1.1.7 2026-01-09 [?] RSPM\n P rmarkdown 2.30 2025-09-28 [?] RSPM\n P rstatix * 0.7.3 2025-10-18 [?] RSPM\n rstudioapi 0.18.0 2026-01-16 [1] RSPM\n rvest 1.0.5 2025-08-29 [1] RSPM\n P S7 0.2.1 2025-11-14 [?] RSPM\n sass 0.4.10 2025-04-11 [1] RSPM\n P scales 1.4.0 2025-04-24 [?] RSPM\n selectr 0.5-1 2025-12-17 [1] RSPM\n SparseM 1.84-2 2024-07-17 [1] RSPM\n P stringi 1.8.7 2025-03-27 [?] RSPM\n P stringr * 1.6.0 2025-11-04 [?] RSPM\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n sys 3.4.3 2024-10-04 [1] RSPM\n systemfonts 1.3.1 2025-10-01 [1] RSPM\n textshaping 1.0.4 2025-10-10 [1] RSPM\n P tibble * 3.3.1 2026-01-11 [?] RSPM\n P tidyr * 1.3.2 2025-12-19 [?] RSPM\n P tidyselect 1.2.1 2024-03-11 [?] RSPM\n P tidyverse * 2.0.0 2023-02-22 [?] RSPM\n P timechange 0.4.0 2026-01-29 [?] RSPM\n timeDate 4052.112 2026-01-28 [1] RSPM\n tinytex 0.58 2025-11-19 [1] RSPM\n P tzdb 0.5.0 2025-03-15 [?] RSPM\n urca 1.3-4 2024-05-27 [1] RSPM\n P utf8 1.2.6 2025-06-08 [?] RSPM\n uuid 1.2-2 2026-01-23 [1] RSPM\n P vctrs 0.7.1 2026-01-23 [?] RSPM\n viridisLite 0.4.3 2026-02-04 [1] RSPM\n vroom 1.7.0 2026-01-27 [1] RSPM\n P withr 3.0.2 2024-10-28 [?] RSPM\n P xfun 0.56 2026-01-18 [?] RSPM\n xml2 1.5.2 2026-01-17 [1] RSPM\n P yaml 2.3.12 2025-12-10 [?] RSPM\n zoo 1.8-15 2025-12-15 [1] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n\n", + "markdown": "---\ntitle: \"Friedman Test Analysis\"\ndate: \"2025-04-09\"\n---\n\n## Friedman Test Analysis\n\nFriedman's test is a non-parametric statistical test used to detect differences in treatments across multiple test attempts. It is often used when the assumptions of ANOVA are not met, particularly the assumption of normality. The test is applicable for repeated measures, or matched groups, making it useful for situations where the same subjects are subjected to different treatments.\n\nFriedman’s test ranks the data points within each block (or subject) separately, and then analyzes these ranks to see if the mean ranks differ between the groups and conditions. If the test shows significant differences, this suggests that at least one of the treatments differs from the others. Because it is non-parametric, it does not assume the normal distribution of data, which makes it robust for skewed or ordinal data.\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Load required packages\nlibrary(tidyverse)\nlibrary(broom)\nlibrary(rstatix)\n```\n:::\n\n\n\n### Hypotheses\n\nH₀ (Null Hypothesis): There are no significant differences in weight outcomes between the three diets\n\nH₁ (Alternative Hypothesis): There are significant differences in weight outcomes between at least two diets \n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Create the dataset\nDiet_A = c(75, 68, 80, 72, 85, 70, 82, 78, 75, 83)\n\nDiet_B = c(82, 70, 85, 78, 88, 75, 85, 80, 79, 87)\n\nDiet_C = c(78, 65, 82, 75, 84, 72, 80, 76, 77, 84)\n\ndata <- tibble(\n subjid = rep(1:10, 3),\n diet = rep(c(\"A\", \"B\", \"C\"), each = 10),\n weight = c(Diet_A, Diet_B, Diet_C)\n)\n```\n:::\n\n\n### Base R {stats}\n\nTo run a Friedman's test in R you can use the {stats} package. This will return the chi-squared test statistic and p-value. \n\n::: {.cell}\n\n```{.r .cell-code}\n# Perform Friedman test\nfriedman_test <- stats::friedman.test(weight ~ diet | subjid, data = data)\nfriedman_test |>\n broom::tidy()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 4\n statistic p.value parameter method \n \n1 15.2 0.000500 2 Friedman rank sum test\n```\n\n\n:::\n:::\n\n\n### {rstatix}\n\nAlternatively, you can use the {rstatix} package. While these packages give the same results, the {rstatix} results come as a tibble we can easily use. \n\n::: {.cell}\n\n```{.r .cell-code}\ntest <- data |>\n rstatix::friedman_test(weight ~ diet | subjid)\ntest\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 1 × 6\n .y. n statistic df p method \n* \n1 weight 10 15.2 2 0.000500 Friedman test\n```\n\n\n:::\n:::\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Create boxplot\nggplot(data, aes(x = diet, y = weight, fill = diet)) +\n geom_boxplot() +\n theme_minimal() +\n labs(\n title = \"Weight Distribution Across Different Diets\",\n x = \"Diet Type\",\n y = \"Weight\"\n )\n```\n\n::: {.cell-output-display}\n![](friedman_test_files/figure-html/visualization-1.png){width=768}\n:::\n:::\n\n\n### Conclusion\n\nBased on the analysis:\n\n1. **Statistical Test Results:**\n\n- The Friedman test yielded a p-value of 5.0045143\\times 10^{-4}\n\n\\[If p \\< 0.05, we reject the null hypothesis\n\nIf p \\> 0.05, we fail to reject the null hypothesis\\]\n\n2. **Visual Analysis:**\n\n- From the boxplot, Diet B shows the highest median weight\n\n- Diet B also appears to have the highest overall weight distribution\n\n- Diet A and Diet C show similar distributions but lower than Diet B\n\n\n3. **Interpretation:**\n\n- If the goal is weight gain: Diet B appears most effective\n\n- If the goal is weight maintenance: Diet A or C might be more suitable\n\n- However, individual responses vary, as shown by the overlapping distributions\n\n## Reference\n\n*Cite all sources and references used in the analysis.*\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P abind 1.4-8 2024-09-12 [?] RSPM (R 4.5.0)\n askpass 1.2.1 2024-10-04 [1] RSPM (R 4.5.0)\n P backports 1.5.0 2024-05-23 [?] RSPM (R 4.5.0)\n base64enc 0.1-6 2026-02-02 [1] RSPM (R 4.5.0)\n bit 4.6.0 2025-03-06 [1] RSPM (R 4.5.0)\n bit64 4.6.0-1 2025-01-16 [1] RSPM (R 4.5.0)\n blob 1.3.0 2026-01-14 [1] RSPM (R 4.5.0)\n boot 1.3-32 2025-08-29 [2] CRAN (R 4.5.2)\n P broom * 1.0.12 2026-01-27 [?] RSPM (R 4.5.0)\n bslib 0.10.0 2026-01-26 [1] RSPM (R 4.5.0)\n cachem 1.1.0 2024-05-16 [1] RSPM (R 4.5.0)\n callr 3.7.6 2024-03-25 [1] RSPM (R 4.5.0)\n P car 3.1-5 2026-02-03 [?] RSPM (R 4.5.0)\n P carData 3.0-6 2026-01-30 [?] RSPM (R 4.5.0)\n cellranger 1.1.0 2016-07-27 [1] RSPM (R 4.5.0)\n P cli 3.6.5 2025-04-23 [?] RSPM (R 4.5.0)\n clipr 0.8.0 2022-02-22 [1] RSPM (R 4.5.0)\n colorspace 2.1-2 2025-09-22 [1] RSPM (R 4.5.0)\n conflicted 1.2.0 2023-02-01 [1] RSPM (R 4.5.0)\n corrplot 0.95 2024-10-14 [1] RSPM (R 4.5.0)\n cowplot 1.2.0 2025-07-07 [1] RSPM (R 4.5.0)\n cpp11 0.5.3 2026-01-20 [1] RSPM (R 4.5.0)\n crayon 1.5.3 2024-06-20 [1] RSPM (R 4.5.0)\n curl 7.0.0 2025-08-19 [1] RSPM (R 4.5.0)\n data.table 1.18.2.1 2026-01-27 [1] RSPM (R 4.5.0)\n DBI 1.2.3 2024-06-02 [1] RSPM (R 4.5.0)\n dbplyr 2.5.2 2026-02-13 [1] RSPM (R 4.5.0)\n Deriv 4.2.0 2025-06-20 [1] RSPM (R 4.5.0)\n P digest 0.6.39 2025-11-19 [?] RSPM (R 4.5.0)\n doBy 4.7.1 2025-12-02 [1] RSPM (R 4.5.0)\n P dplyr * 1.2.0 2026-02-03 [?] RSPM (R 4.5.0)\n dtplyr 1.3.3 2026-02-11 [1] RSPM (R 4.5.0)\n P evaluate 1.0.5 2025-08-27 [?] RSPM (R 4.5.0)\n P farver 2.1.2 2024-05-13 [?] RSPM (R 4.5.0)\n P fastmap 1.2.0 2024-05-15 [?] RSPM (R 4.5.0)\n fontawesome 0.5.3 2024-11-16 [1] RSPM (R 4.5.0)\n P forcats * 1.0.1 2025-09-25 [?] RSPM (R 4.5.0)\n forecast 9.0.1 2026-02-14 [1] RSPM (R 4.5.0)\n P Formula 1.2-5 2023-02-24 [?] RSPM (R 4.5.0)\n fracdiff 1.5-3 2024-02-01 [1] RSPM (R 4.5.0)\n fs 1.6.6 2025-04-12 [1] RSPM (R 4.5.0)\n gargle 1.6.1 2026-01-29 [1] RSPM (R 4.5.0)\n P generics 0.1.4 2025-05-09 [?] RSPM (R 4.5.0)\n P ggplot2 * 4.0.2 2026-02-03 [?] RSPM (R 4.5.0)\n P glue 1.8.0 2024-09-30 [?] RSPM (R 4.5.0)\n googledrive 2.1.2 2025-09-10 [1] RSPM (R 4.5.0)\n googlesheets4 1.1.2 2025-09-03 [1] RSPM (R 4.5.0)\n P gtable 0.3.6 2024-10-25 [?] RSPM (R 4.5.0)\n haven 2.5.5 2025-05-30 [1] RSPM (R 4.5.0)\n highr 0.11 2024-05-26 [1] RSPM (R 4.5.0)\n P hms 1.1.4 2025-10-17 [?] RSPM (R 4.5.0)\n P htmltools 0.5.9 2025-12-04 [?] RSPM (R 4.5.0)\n httr 1.4.8 2026-02-13 [1] RSPM (R 4.5.0)\n ids 1.0.1 2017-05-31 [1] RSPM (R 4.5.0)\n isoband 0.3.0 2025-12-07 [1] RSPM (R 4.5.0)\n jquerylib 0.1.4 2021-04-26 [1] RSPM (R 4.5.0)\n P jsonlite 2.0.0 2025-03-27 [?] RSPM (R 4.5.0)\n P knitr 1.51 2025-12-20 [?] RSPM (R 4.5.0)\n P labeling 0.4.3 2023-08-29 [?] RSPM (R 4.5.0)\n lattice 0.22-7 2025-04-02 [2] CRAN (R 4.5.2)\n P lifecycle 1.0.5 2026-01-08 [?] RSPM (R 4.5.0)\n lme4 1.1-38 2025-12-02 [1] RSPM (R 4.5.0)\n lmtest 0.9-40 2022-03-21 [1] RSPM (R 4.5.0)\n P lubridate * 1.9.5 2026-02-04 [?] RSPM (R 4.5.0)\n P magrittr 2.0.4 2025-09-12 [?] RSPM (R 4.5.0)\n MASS 7.3-65 2025-02-28 [2] CRAN (R 4.5.2)\n Matrix 1.7-4 2025-08-28 [2] CRAN (R 4.5.2)\n MatrixModels 0.5-4 2025-03-26 [1] RSPM (R 4.5.0)\n memoise 2.0.1 2021-11-26 [1] RSPM (R 4.5.0)\n mgcv 1.9-3 2025-04-04 [2] CRAN (R 4.5.2)\n microbenchmark 1.5.0 2024-09-04 [1] RSPM (R 4.5.0)\n mime 0.13 2025-03-17 [1] RSPM (R 4.5.0)\n minqa 1.2.8 2024-08-17 [1] RSPM (R 4.5.0)\n modelr 0.1.11 2023-03-22 [1] RSPM (R 4.5.0)\n nlme 3.1-168 2025-03-31 [2] CRAN (R 4.5.2)\n nloptr 2.2.1 2025-03-17 [1] RSPM (R 4.5.0)\n nnet 7.3-20 2025-01-01 [2] CRAN (R 4.5.2)\n numDeriv 2016.8-1.1 2019-06-06 [1] RSPM (R 4.5.0)\n openssl 2.3.4 2025-09-30 [1] RSPM (R 4.5.0)\n pbkrtest 0.5.5 2025-07-18 [1] RSPM (R 4.5.0)\n P pillar 1.11.1 2025-09-17 [?] RSPM (R 4.5.0)\n P pkgconfig 2.0.3 2019-09-22 [?] RSPM (R 4.5.0)\n prettyunits 1.2.0 2023-09-24 [1] RSPM (R 4.5.0)\n processx 3.8.6 2025-02-21 [1] RSPM (R 4.5.0)\n progress 1.2.3 2023-12-06 [1] RSPM (R 4.5.0)\n ps 1.9.1 2025-04-12 [1] RSPM (R 4.5.0)\n P purrr * 1.2.1 2026-01-09 [?] RSPM (R 4.5.0)\n quantreg 6.1 2025-03-10 [1] RSPM (R 4.5.0)\n P R6 2.6.1 2025-02-15 [?] RSPM (R 4.5.0)\n ragg 1.5.0 2025-09-02 [1] RSPM (R 4.5.0)\n rappdirs 0.3.4 2026-01-17 [1] RSPM (R 4.5.0)\n rbibutils 2.4.1 2026-01-21 [1] RSPM (R 4.5.0)\n P RColorBrewer 1.1-3 2022-04-03 [?] RSPM (R 4.5.0)\n Rcpp 1.1.1 2026-01-10 [1] RSPM (R 4.5.0)\n RcppArmadillo 15.2.3-1 2025-12-17 [1] RSPM (R 4.5.0)\n RcppEigen 0.3.4.0.2 2024-08-24 [1] RSPM (R 4.5.0)\n Rdpack 2.6.6 2026-02-08 [1] RSPM (R 4.5.0)\n P readr * 2.1.6 2025-11-14 [?] RSPM (R 4.5.0)\n readxl 1.4.5 2025-03-07 [1] RSPM (R 4.5.0)\n reformulas 0.4.4 2026-02-02 [1] RSPM (R 4.5.0)\n rematch 2.0.0 2023-08-30 [1] RSPM (R 4.5.0)\n rematch2 2.1.2 2020-05-01 [1] RSPM (R 4.5.0)\n reprex 2.1.1 2024-07-06 [1] RSPM (R 4.5.0)\n P rlang 1.1.7 2026-01-09 [?] RSPM (R 4.5.0)\n P rmarkdown 2.30 2025-09-28 [?] RSPM (R 4.5.0)\n P rstatix * 0.7.3 2025-10-18 [?] RSPM (R 4.5.0)\n rstudioapi 0.18.0 2026-01-16 [1] RSPM (R 4.5.0)\n rvest 1.0.5 2025-08-29 [1] RSPM (R 4.5.0)\n P S7 0.2.1 2025-11-14 [?] RSPM (R 4.5.0)\n sass 0.4.10 2025-04-11 [1] RSPM (R 4.5.0)\n P scales 1.4.0 2025-04-24 [?] RSPM (R 4.5.0)\n selectr 0.5-1 2025-12-17 [1] RSPM (R 4.5.0)\n SparseM 1.84-2 2024-07-17 [1] RSPM (R 4.5.0)\n P stringi 1.8.7 2025-03-27 [?] RSPM (R 4.5.0)\n P stringr * 1.6.0 2025-11-04 [?] RSPM (R 4.5.0)\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n sys 3.4.3 2024-10-04 [1] RSPM (R 4.5.0)\n systemfonts 1.3.1 2025-10-01 [1] RSPM (R 4.5.0)\n textshaping 1.0.4 2025-10-10 [1] RSPM (R 4.5.0)\n P tibble * 3.3.1 2026-01-11 [?] RSPM (R 4.5.0)\n P tidyr * 1.3.2 2025-12-19 [?] RSPM (R 4.5.0)\n P tidyselect 1.2.1 2024-03-11 [?] RSPM (R 4.5.0)\n P tidyverse * 2.0.0 2023-02-22 [?] RSPM (R 4.5.0)\n P timechange 0.4.0 2026-01-29 [?] RSPM (R 4.5.0)\n timeDate 4052.112 2026-01-28 [1] RSPM (R 4.5.0)\n tinytex 0.58 2025-11-19 [1] RSPM (R 4.5.0)\n P tzdb 0.5.0 2025-03-15 [?] RSPM (R 4.5.0)\n urca 1.3-4 2024-05-27 [1] RSPM (R 4.5.0)\n P utf8 1.2.6 2025-06-08 [?] RSPM (R 4.5.0)\n uuid 1.2-2 2026-01-23 [1] RSPM (R 4.5.0)\n P vctrs 0.7.1 2026-01-23 [?] RSPM (R 4.5.0)\n viridisLite 0.4.3 2026-02-04 [1] RSPM (R 4.5.0)\n vroom 1.7.0 2026-01-27 [1] RSPM (R 4.5.0)\n P withr 3.0.2 2024-10-28 [?] RSPM (R 4.5.0)\n P xfun 0.56 2026-01-18 [?] RSPM (R 4.5.0)\n xml2 1.5.2 2026-01-17 [1] RSPM (R 4.5.0)\n P yaml 2.3.12 2025-12-10 [?] RSPM (R 4.5.0)\n zoo 1.8-15 2025-12-15 [1] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n\n", "supporting": [ "friedman_test_files" ], diff --git a/_freeze/R/friedman_test/figure-html/visualization-1.png b/_freeze/R/friedman_test/figure-html/visualization-1.png index 7fe7a5006aa170649ecc522d4098136909efe064..007d4db4b05a765b1634b8900a226c6c7c119040 100644 GIT binary patch literal 39334 zcmeFZcT`hb+bBy@ zSOjOUPX|SwC;Z^rRK^5}v__pjYjHk|E2Sn<7H0TL$o_A7M!|LWxDugr=Mp+Ebv}9U zD)Z&NFJq_wI3;apQJeqwah*`ETN|(-I6~wPt{^UAlmC7=`}ySfxH6?RBZt1x7iKEb zbp#+Ga{fLc@bCku%<HchuC9xsFX?c{M7q^^E^4Bt@B2@dHvYeIM7+6DQn09rM^Z{P)= znb!e)XHr`=J*&7nV(uB3Ny&L2qBXJ{sZgWhiw+>1w%J{LDXiC)bJBD;3b)0YgOzt? zj&?_=9w0fHwU-39Vc?ZG1*(q?nctfwiND}K=_oHdkjipTaUX2(iZv3&ti$i+9Sibw zfG9+poDMV-br!q=QKE0y=jG;1n;|DgS1k#arr+#2t#0$L3RI%`3a}epQ26Mxgte9j zQ1?miu2gv(V(W{xk&a?n-iSr^sw6~cyKdfYO0|=vcolGmbjbh>CQrjA)MTjK96y{VX&iVk2;?w#Ho<}R(2(-Z(j#O4uX5#ajpA@bu5~x+NX~e)cZS#%& zi5Xeo#Z@+WhsgCCT?2_=#EU-K+nqF8Ym=zDhr)edk$0m^3A_#Y<2WeaYTpvoa3z*! zmGIXoiRj>kl#@Q^)6r8CS08mEUmCDQkvZzC^abBr28)Y0b$Yuew?s5{aXIGPir>akCS#nqNl;^u9C`>LnAJHrBYv{_8%S@^Z*Dk3OZb0PPan0W{E+h4GI$G<#92s-J*6F5x_%OT%b6&xdGj@zIF7@GKa4jsmV7*V z%~6Gt9CRfCeh?ERAlM-xm%X=(PXve3wT)Tur(T+OEEVi6@-F$rE7ke+ZFUKGGB*vp zSB81j>vuur7RUR|%!)dNZA;{7DFP1V1B$&V|K&GFg!#@4q)ug*fZ<4(`)ZK^*BzS zM?Cx_dVJ9Ka7Ch}Pmvl;W5=GfYC zc>-MkXGF=d`-4tUAkx3v?5wOG9|xLEmlK(*^V+|Ou||q6i5?yr{(nOMDA%1A|Et!g z*MeK(OC>Z{rL`*d<7^Bbtfp~J>t2xhX8ynmpBAo zI}2LHO@e#+F<#C6?@_!HKBX>`UHsm6(j~y551HB|aIM>Ut*^4d^Lo~a+0fM^FG4LP z_bUFf-(8R*ImAxi*GFKie0@I2Q0;D zAFtA>H^fDa;sg{uzA<}=SJ$fL$Df@CMcPS#CLYYSpV*Uosl_d!<~mel>-@^0c01;J z@~JsKGg&UqW|v;0VGQ35g}exA;SB-VaOc^Ewnz$} zLQv}?^^>zY#=nZA)Xzv1spSMLey{1uRcf+5ul3S8L3C^X=*dB)hh0p+Zx9e@QY|Uc z>x%od&v$r;v60*viJisht1{ZOv|>T2?3;@*slME%g__!2yjPb=yBl2AUozYZy9RK6 zXrp4pt$wWH)~eJQx2(C&Hs;3q@tB05<;EqaD6ZWn8SKAj56W!u_3~=4rR~^9FNu8_2kZ6nI5k%{ zQ+Pd0jZE;kjbo{%$eDQ0_y z%}a=fU}(A4HB3k3p7B-BW}{ifote;(p-Kf8d7BpL$yyzZXb=fUe9r}&-k*u89xs@x zDrJXLQWly6(l6p7*iO}je4jII&*cg+wkLgB37cPVkZ)jwjIdNR-z**+60&|8iC~Tw zDDq4MtheY(-}z`x~5;!P}54MH5uk;$=WXrT*#Ss>zJ~F?@QiZ0ngT`+M}p zO7&DhwG8<{Ox0&=DMWT6Xt{;x#Nu6c8-JfaRWYu*^}Oo^7;^*I>mb#ISL}p>6J6O#P2(3i_Gh~)^TNJ= z3t4g;^NmMe|B{C|=xGv!a+d33D$Qio)|=MYECvPY@~N?8#h}um*vq0wmPQZ?vo!BD%UVt;frHdJh7MBpW*#Yg>u)8#;iI z-l1KTSJMGnQ?vTuDGm8(KE-83uip676i%EVbxPuHzp-dzK=MggecqI6*<9NOgZBXIW78=}fCUb_bQvYtCFX z?lr16Q*+qwvl{q=*|FYnT-ztlerDv0FwSX}5SAMLWJU~}YlgK9d~3h8SSv1ucCEx3 z)A>>QvV$8L0;= zZr>IB^Dyib4W;UZ=c1uGw52Woc%{jWsRB&m)k7sI2F`TAj`v;`XDpA)76O8>pKe?; zeIPrb71UE5?mCY*o~T@KL0J#JCpIf5WS~l_3aTikAq>{vc3Z1K8UcZa3N|3Bm-gU%E}CI^F8;psYMc3evAS16JoHJ)@wk~>dsXrbg?GH!ts>y! zJZg;muJQ#A_4n{wFT-K$(xKU=-y%>aW-Q-KOkF=Mu7+O!(YzA|;VgO(#Y`3l6`YNZ zL1UtpMwS1zw1O0SbBb@hwyLvhgkjvRMRZf6*avyAehjMxOuj9p!~ z#aCaKZhKx1_0>!(9z$Uj^qf~?tZ?L3hbgmalKu9WEulHQYf|%;N9zlAICIvanZhFx z-BmzeAuH zRzF^*xu?z&olxg5hH2nPlsYmtUpZ$HW+p<+H=4J4|Lg&g*0U&exE z(XrxGUpeR-7xK)qDn~pthdAoMfnKQgFW6{{+oZJtRWfR zoCTHFFHFYW$1BJc;;n+9hic};lNPNh_5rl5%Rz#76Ou4t-#T$xF5p(Lr|p0?+klPl z7pNr8%@W~_J`iq+B8;PI%?NL4M>1mM(X@PZnBz`QC(XMm_(v=wE&D=jnJKYdp1GI9 ztI3*b`e2-}KZ(bahd+F3u@(xzE&mf!j5sRG(VO^ejUahjVWV@rDA zP)Y(xw8fxX_I?OG#8irQCVw{(LmrZfGpjzM%2004&0@3H#}%NWYU?cuyiZy;j8ZMu zhBrt>H@7Mn#%{lBaj)dG(U^= zFh?TqjjWNO7O_-!^jeLqxx{_c6s6CDqp^shX=W%e$p3IK3k5S?#J#ezg{>eT%XS|* z#t(5u`iKqt84mtP`=p=22Do~m<%Q_e-dN|EV=4Bk2N(>J)RDQTvay}T*%^E1JEsd~ zBG=?^6|5b!*7@Q+pwr8_udCe-+%4<~2kwL!ln$!(QKzt|!s!zna=jtgse8JMKblXg zg^_x$SGCN7{=Hc-!DcVjx^j}Zbj&TVqWC$vqIl*s=yuF6x1D?1II)_S@!rL%4!eE2 zg3p^Tw;Q<$!~kw1KEgAwn-tdx9hDW;;tpXj*HH;ZR`vCqx{V{X7F}jCbY_eU^=pzA zw<&u7+Pc$kSqxS~2W%(k1xnpgFbZPfOl6edAF%$reUpgg$)Zb(E1?G(lvMZ?XL`yS z7i#7y()!jB6K;u~$(06-Yg(v}2!ee;VWjt+JcnX-w&$h8eJ3I4FFN>ihfgtsn@wS9 zH5`vy8NCd|^6(#kwo5qE7W%ow*sCf#y#LN+!J8pr5RJe>-N{Ul6rsp=*kWb?id)L* z7hf9J0$B_lCC9Siek)za{d*zw;jNsZwT zdw5UTW-*QxIJBw~y=$~Yf_IHi8)#fVt4N`@hrtmRy|SplmflL{yok7(!5}hZp~cQ+Oi*>kxta+rnlm8MYNI~ePxaL&;AA{ z!2NLbxoJgbEbeGCL58><{*6RYDb)+P_}S{x^g=db3So&%E9S~`^BpX17u#p&(oMO2 z7uyjeYOoI8iqJ@Sq zqlX*^n#lt@%GJd$7zj$NBw9a)Zx)r;Z)+11^0H1rJp7@%uZWh#)8z}$>8I18S}*Oy z_6QdZLOr&6FN%dChEuzC^k%~GTp22NL#A;NG%%j>t!$*%v=R=+kn19Vy=ivFYi=`p z48hIMOOU$7met77zj={WJ?rlXW*52e+d>~o68lgDvWHReM5R(&?W!f^IA|=FJF+JY zP5H?7GLwjY9pXBh(y@kr5)X}lP0c!Rn?`BB_;u{*Nm3L3&9O{fiRf9k!cOmZ2-Tg% zqVmdFKBWsFsa)D7A{QbS``SHl$Qb_g-dvrghP(f6s-XBgajlnN#8&r`m1lW_2ULXm zy$Uc%5hNc+)dro0@uTHA1K8aLQEBgWY(!hgxM_9o??DAlXh+pjZElGHm-;|qkfM9t z#;wezrn0ixf-e~lx)rUwf4J@_iQ_L5_J)>5AK%D$uvPKKRjJbR&NfYr_O?CCn(7E> zQvThx^-N^LRgd7Lk_ZHI_?#U;{M4+g*ac{iwT1^&i2gMjpoOrtW(~#aWkfYA2?r2p zTuQIBUdESb?0$8*V*r?T%!sx?`gP~MGTpHCR&TL<&l2o=nDpxi#;3Ex@c{GdvIbO8 z9_8MmDD~}zy_18L7j*}&Io1ZxT%;vH&c*^%;MU`g81~wPM6rz2LXcR%VmcrzYffPU z78D}_rK{Z|P*!2V3l=V2Rf(CZ!D-_!6&tthR;h|*W?Vyd`=3wkRRCwu)A*O$3sTF9 z%e9?=3o94u{CNa*8PV6K23wZ|9Pb&3-meo+R5t{U#fnV;u^qYWo95#iJ0HuZxQ>{5UFZ7cB;c2b4kEgMz?59}R%f5AgY08cmP&{d;52V2 z1w2WrbESAqQOekc(n6LuTj%mGQm^gQDfal(u3VB0yad^yx7@KWWS-!WmNRE-a*=by z4?SUT&AC^?xp2fn2{@qCY$ZT7$H)>celKWw5|PxHDvBXu}7?bpAt zeyj87)m5SEYW_nh|B64nk>v3-8!#&$=*qF#7LIV~oG;IsmOhLZhz0D6Iggs9kp94} zYDn)Kygch85X)C^C2ByWdHT38mXV^ zhCJMg_mH3dk$#3X6R_0Q)iyseW1&)~nv0Vq2fm7gnLxhwhir@=`o00_?MY+%NB^Ao z@$x;_d_ty;2r|@ZBwDdld{`=45m}XW)yNOtmkIfidPxP(bOMOp8E+J1WpB! zM2F@yyE}En5u5r-4O0!eLu#R}l2+4C=FNM}o#*;Kj?Cz%$gPY@C$mh!) zioR0tIUH}>-dqIL9-1(2JW38PS7L0SZT)s*EGnk_(uuo-iiV0k!o>QIRErGJdLj4K z@3t-%N5^_RN+*eB_ZzIe!kuDFkfHLRkJP&$i`29=D(HkGXZH)9{f4nJ;O|^;8+t4J|uI$ zJqFmYd#H`6=04Z6O247h*p1gYUO`*M2sn(IsWK439&a#hFG&!hF7XvGtuR!f@F}+? zhu|4&>cs5WJ?E+x+Z`9n%1Vn#n3wcE)`D?{JS1^Gw&}_Ekfm=KDkERUtSP`2P%GT! zqh2!CJhNb4Wy)TGSP!MBMgu&C4 zkAbnx!K?vmv)zhA&bI!^*<ExC{A=K_1AT?Ih=4)SM$*Ir36h$x=yT;cgA9eM8-JJ~D zNa?PxlkmI&GU0w-RtALJwNgk`URhV{w-j9O35=A!RiVN(`+8YDk+?Utd9)vQf#p z>bu`YdGoVEXrIm^S+gGl(PkYT9c!>QEVe@FVY4B$|9+(%v>GJ_xtcn2y|SX@Os=!i z--gxLTyY)k%afIhxY{_!}Y$Wbnd#BctGmp6;}++i{2I(~pXUNbzZ0kR4Oo+(;?FRZ4q%Dq*f#SU$81)Bv&&s> zj6MjX_rINbN^1>5ei2v{=+A*ZN|rAUns6=+ut__afi7*eJl~S$zrrsczLS9WlZckW z8K8|OUA`pZyLb@6z|4T~S;STX5C_BlXwbH4Bw~`X*&l~g9%}|Kj?!`FMmUO}J>h^$_)Ul4e3f&N8U#!q6poI*?-f3$X>!()BujZN=vx>MfSS1X;# zopf2)(_fhIn6xyAP|-XFx|F-LCi8lA1o5!j2H!CZ`JKLBPjLZU;hLy3r0Q2P=(3Ld z_pPT3_+}N)<+X9fyD!tOrig(6BO0|j?Rsdbcz5rATXPkhbWo>sLvs)VeFy3~-HBP}<$qCCW_C@j&Eza(QicHi8N=Lu8zj5$pD=)p?!h7Q!ud!Sl z5KalBu2_y%5fyz1V?UAxWG=jJB$i7CRxW=h{Jnft+rWFc7tS1fsejq=5BF~vyk-a> zW}a-HD-=j;Rr_*Uj81PRwS0emS`HgN)>o~mX;nB>F>1YLz>Azuw3S8qZcZu5Ai>g; z5R|#>Uh$!I7WB(){_aVShd5EdgbrTtM_|3&0n!Pu@2`*bsJkWp43-_5etvx|G0MWg z8o!=lMJLld_+nnOzSrcj$mD_>;Nbo*Sv5@?xd_e@n5LnGc8Zmqas1}_m6+(2bxPvA zF=GjuC;#aF##A;JoB1(Yo~$2$-KXc}?(O)>%mIuuS8g5QAManQ$Fqpr#jAPaSXA6J zZ1|;{v+Kq3-4;|)=1n(jH7q)qTm)uO%P%`S^Hxf-*%Pq>OMttL$%E@6d2+7Vfv;M6 z&-BrHLSM=6Y=Al2U$!$1XhKwtg{Cv*d|I%9O%khc{jM(|9?=xC4mfCKpEM_3DmJ?u0@774>`kCB0uZ7*t$l}}xG`!%` z*G9|3bn$me+~wij@HdINktStNkU2DpA-kJBly?MkVJ^H^{r16oX|kmZ!A$ezx))r1}eHyy-NgIYgEis?gpM$<~CW{wjQ8xx5G1=BYB=coD zm@}6we}KnSD%(CJ_aSn^7V~EMzW^4n25l1R_Mzi%-D=~eZ4UM7ORcZ)U0OU&?Qg3U zg;mR8Z~CHTQp9!YgML!=SwTU5;t*MVTmsBs|=pZL0Z3VSpI5gw>h zj_Q&rnGRiCQN~#Ub~;LBcIY(kjMQV#p%0JZfdm>md^eA+5<<9g+;Nv_jBkd0Ub4UD zdAYG{e2@aE^*LPJS$ZXo8YKyyoFD&0P}oqXtmCkN(+xOhyUwHu4|{dFbMj#>`{oAs zxAU}(HGFr3i|)tUiG=jL4m>%St*48%`Hfl>md#0u4%vZr;6khd%`Q^kUOXQ*5wo2Vxt(Z-;)j&PnoYIvcsqRs3+bIz0#$klOLnjk* z8I~U$mu@;vO~W;<2vIv{PiR$ zJQ(gx#?Z3;H{)sWIp%PZ`eU@HP<009n=qb)U^o_P+K9&TEs(eS!K?X- zlu5(-jgehpGntj!BXUsOW*Nx{9_}WG-d~}Jhki_U%|lFlj|!rU+`t8|l=>KlxQu}} zgu!8kB0z@Vo3%sm_ZLNI7`U;d!;4OgJY8++y_nadqlUukofr|CJ+ z(oe%?f`x^5**&IYHl0Fm4kkB{5YB_lJofJFTqtGe26}gnsLRV3ruZOWFsGLVQeS*+ ziHVlu1OZ{xmKS z@a2)tL|emui`D()HS)8Y7L|5A(=}d$VMjnx1`lFxAUW=Zl5N{S3IkU1yx1BHgJ5`E z38urz6#KU_VV|bzt|C-g9zp0!G}9>W(&<}sJBB1{mF+dd0zfRbcNC6#Z`*CH$I6Y> zo7z?axhDS=b0kTr=#OZ=ZE8!X`_>C(enV_sm})fNsYFbe{w>SRPSj<~!H0uT7gvK; z3m}R-G$&CEYud~(?6pjs)wfPnOfw7F`&PU5ji`Yut6IH=G>QYaP+ob{`5{`xnc{vO z3OhD22&us+AF^G!ozKM{=`T5O$g%ch6fE*K3D|CuoJ z3_%=dd4JN7ycvH#^z#q&K#wPX5U{sd4 zpbxeJi0i7#w{&seH8;F$hN`er-EI-q8{jPi!1 z?7&XF4jMdR8lP*et{S)wm0O=|Hnj-NkRpVmFUqsJzF3PlJ;dKTC||BC=LJ1kx^e1v#nll;-ntBTrM6Mye3I|z!FY6rYp-F9dKAk?s4Yg758tYHOTHo;Mfs{h% zgjShM<;#%WS+F;PS`QfV?lU{ty@}OND}b}{wfbN_RS*(7SJ-?yU2s9!hv=LXbs*9L zUAnS>m1PWk71qVtp4T(vWp~n&CX4(2l9pq7t}b6UCuL!zagM}RhRCg_Nogy5agvZ` zYfjb-UN^zG*=OUcLMD>4Z>-1JgPu#9*P7XoboA|HJNH9G<$veUaA0M5;i2io*4^yf ztm8a+_}tBr;bd1iraGa_nc;XKf|z~ggp@N0eO#Y_Wd_`a+Tgyi{ppIBPr$(xTjnH20DBg1-ai?*-V&7eOar4RBhSq$&?ARVIg?bH+%Cj_>Af-?v8GU8 zFEJhR^)hJ{!*5;KK~WFbd8rI&*&)$0xaEQ)_P4Iw!-j8bAVNZ|0Qbl}WVxB7=AQf} zs;(FO?xy$bWc{M7gJ@>s#t7@9W(4p0DE5{22Sr1u-Ga1N)O{Mk9a||f`LL_6Ka5QY zO3cbT9fA=F8XwQvee-#RM%vx%djP~T{MkV-tlsTeZhW`x)~%ZFsU(rtSkD^f=cFxV z?du&CCjenw_8KI0HckM45NzrsQt-kH6X<0=I-;}n93eKj_qp`+9FPeQUFj$7`FZmI zc*05EAZy7x*mZ;;1-&E|w*J8tS*h!z^!`48N;sVx9{hO%z(!%%v6PD}0ICQ$*C45r z1xhSl@E_n)66ob`ld<@jb*IRw=cRCJR;hL8Az&u>AhI*iR2S$ye}FskH;Gn82w0_9 zTN()MO11@!Tu@e5ir82J!ZQvHyY2cMR{u(r5vPlhp;BZg&g#t*F>RkqKSV(8z{2qa>S5g`xQfW`Yspnbfew|=zjK8k1f*Z-Tp-1 zt$1sfQ+5qrxM8tY#@^~oMM3x4;gs+G2S`)zBhcu-5r_Fd2;l^|n>zjc<9@H7e1%S@ z2Re#GR-@br6%VeQ(|Yo)K3p$m?%#oV{P*EX{|Dytzgg3eKu3Tn3?sI?C1tE@ zew7gM`zleMLaGFTDhvRG?ueT5bilB%5B989@M4DRM1z|-vEvA6e`qIJ=s$7Q7Xak( zYPw3|S_`*{%&f|2>ofU_vGU&ydOpNQ|5H#_6MsAKWaNF@y2t-qRQ#s_;4{lgD?;E^ zP~+$e)bd06BqHn2Z;T@8Q1?;LFfM; zoNE)dMebu-hsPq8%G^mNF`uw2Jfj3egQju#vn-k$F@NOG(_Pc-CDXh`C4_&)@M}s0 zW0U%=QTo~;|F)Wwj^!&-zz4Bwl-!oT&LKIyMN4_V(+7W$+7>ZUs$Npz)cOm_P{bpB zES)i2Nt1b{&G)lZU=kTB3hJnYHm?N*89Ju82Mt=|RZ%`ku;Y37kchrbCu4m0#;7oC z*>COwW)W%c{fU|CTj7ZFaSXrPHN=b3Nl?SE4wc_n8(|3~1-V5p_dV@jUIS%yE4o(q zhN=e4p!BjnQ`j!+dWE|ED&)7U9U#q%e!B75;a<=L5{lQh`zH)&54vBDkGj1i09Qh* zB-*Vv`w*G(Y$3&ISI!1q0_)Di0! zq+6j&1}`?IE}fFV>{bI9%K0dVis?uRE`gM5>Gmo@>y`U0A*DobqgWoP0~ z$snNJT4aCq6bAIvHF{Uo@Q@7by4t0Y;t21)yT12IhufmY;MW{mZbkzctSn_82+{dj z=HuJ{{8ENHVuD@oUlHf;3So#{5AHH61|k)Tk8h_-Qu6jXJt!_G_M{OuktZBm0`JE< z%cRNSqiX|?n5ed*c;i2%8=Ti^QJ(xq%1-d$yz&{H)ZdSvDboSQno-$1y88Y=7!(6o zcd=5_W%})g#w7CuVdX@vuP!%yxpjIM!y6k^rm^jbUP`uqLF}89&b&7r?o4b2mg*y{ ziaqm%AGJQ!Y1^k8_Q4O^aYF(7-Gc^W zKhsYeVPxr_rACd9_LM=Vc&NAjepniSeKbG1^tT}*G6^7hpG zZ|q}X=)^j4jqu4$wAK8mJU=pg=@Xd5S)3@F{)iNr_>2 ztU?}pI;do$&NZpVYsadeu{j;2>6n{fCmPNUN+f3kxGlr5+_3IJn)mN}11~+~;Jy?V zj54N&gop(C0$gL#bt!NCP+{%kjlD1Ly10ddqh}7|8le7Y*N$%z@8bIEGR3V;st+ z+K5WxA@JREyoR7I#cMJ=w^khh0DX=cN_y9U*5dQj(~ASpmMb^i z3Ge+GCA#cGS;Y-~4l29b;yWofJ?ZWHf$2gHavy~YryNS#@4ba$yhjiHt=oqG66F6A zYrg(d)$gGqJe<~V^M1`F=AyDhGyrO4F6)6S)@DH|e;%?|*^y?4jTk`yfVz&|88~!G z3`2hT6=?K7qu}j7b@KmrD)j!>A$WVW1+>>YRBYhi?BJMw)cwJga0Zl`#Cjy{JYR22X6V`STNv+%&o01aoQ@Ovc}*}U{3gUH_^f0@Q@>3i6ly(^f>Vs9|A-D4;9D$H*IWh<=Rou z?dm6B3F3D3p|3lsjQImJA`EyyZ_DY750%CskkpT}e*v`oU-6zR-xJW=mBQg+85m8C zYowS}Ba`@R#~ILG6o@{UXp5?Fo!-nt9 z$LEo|Ew!E*JMfPup3B*$jcYg%OZ(SvDR3^Yt#8k1(hfHhAm-o2$bbB6ijlzOFzWU1 zn&kf%f0UD)JQj`#CQo%g>T25_;5uz(y{PCp-E56JMqT@+PFd?p4QH;_+!WrAHzm4w)&=m(0x$K`R#!Du5dI}~|n>2BpZZn+U2jefE@nkG4K#ouI zss%4)o7`CF(62oI2SDJ%FsrO%CKXM0SO6l@Kp;I_@yFw|YqoY53bGIOP6HJ%SeMQ) zap!`mnpT+X!Rol_@HEULJ=S4bJSfx9AsohXYw@A(Mr= za^9)^uT>8!=xpt{Sxf(=n%Z9uS#B(mu*!N< z=%OtrWV%fRPi@VT||0a+gO)@s*OQ zM(LfxP>CuB;Fd_HH?!N&zTU-9q*q5G+QiN-lYBNZp&C$RMMak;&7Czv7XJeaFaldx zSQr;OGIsk)^45$cKY&G@ya*C5D=XWi?!`QR{ybnpqj9a#P85JTNu*jk$jg6mF7DOL zXufs`tuwd5sc_{NoVaY|N}U`(wyxkXws2jxLLb$cx}HPt!nmxqsv zi;F#oHT~4{C`TO~okl&75?|K`T1^FCWPW4$zXrr&)sK|bR1%InA%_J)ug}h8Ec7$N zmpD0p)#}BTG+#dL(z-S87^KeL3``%m#h8GK4JnCgsY;4!gOZlVd(<`tWtjyu|!GQ@AyC_^4g7jAl`8e^h$2okK;UOVKoXe9F@wP zEKE%Jv{tT+(!Z?TCw54jKfhp2K0xwmMV;VjTue7f=jZ2-VBGoGNXWkv^@2y(B`j`2 z4h#k#uGqa|vl&dqXK=N)S8{s#5_#?Pty`!jF%v};jq^E}uI2+b5nEb^eS8fQesF=G zd?SEEO-f=ele>pyPLNZ*M+0BR&27ENWCE zFMH>Oos^_x*W?KLzAfz}>cM#(V9Y<`Dws)WAC=zRFUCf^xrEwB&&S2ZsoX999C}=s zbL{~@w2p*@l?4DW$qe*_igb8rX0ujaq#Ihp)fsbfuiXhBS`nuYUnbG$ic$vYBuq3` zi+3z2ln&USf4I(CQd$|l&i7E>Bqd1z5$B&b9aqq&hs55XR30z$Jdp9Z7H{4kMq7EJ zC<}B^t5Ml{L~4Fd<>AAu0PN-pV)~}jk92gvRo4Crj>LA?ml|7%WK96|eDBLB@3|UY zNxLMrk(-aF=7}qY?TS7U=nR=YBQ|QtIXvHu9;lGKbZic8CwXqBb#>-~F?&M~88i`6 zn%M7Wg-{JU2+OM~dnH7>#nOvyWL<+UP(nZ3w$C&zmRobE^#(H{BWOoa_S79#v5j7>Ek zURXD6Z*%1=Zg+B;w%!oCG}JGik?8Rn3E8w=%JzMd{(L>*(6KU+8CeCL*Wx}dN>w9U zGa`KYXG3<*8Y?XMg<$5=E>8Eod_Ir-+XgZpj4i*JYF77!T9O_2C)!QID zdB>5eh;$cHs7(qnpKyl9&MWHKA*HLo3eO*20cYyaD+Jz+@$WZg+&;Vuyhr13ss8^T z|IbAjiQrG>_%UC9ii%}_U!(!ic%b%JmlH7d9k}DTy&tF2+wKfCYCQB=L1m47br;Tc zCgx<>-F&#`+Fa^6lj8fRtEN|8rC1$PNH}5jbA3IHI#Qsta;S|Ba)PY<^Hr1_1ncP@2aIn${8TZBI*`#5J_zlMNdmbE& zf2@U$1j9&C#yHiv_ncHb=IX>3V zWOmU9+$wwn+x3;a>ex2pG&?max>5Q!@|6VA3axRP$wzMTIZqHts zl=nQ4d$?71lH)(htE;{_vx@Xy+aY?$k(|bk6G_1fPGo;zZRes!gJ-m4>)Ui)Hx(S$-cS_~2CzlGi z@;C#T7nE6>*=m|Evqbm0Oa(XIB)g9kTRKh|>;1IkRwLukobb_9dD-4*i(wWb1Yaie zq5&eVIJ=O{2F|w}8;=>DGl}U>v`!NC?W92LGxQaO_bbD69pSet@k94IO7X14FXxbA z=-^Q@hf<(@kfi=zXQXURlMz&11}*_g&j);lJ#f#C?YM8tROts^w>vzIlUZ+xj;JrmII4gv85-8g{$VM!AEd|`SUa*~|(;zg;%7SmpbPy+y^iqD+m|LQhaKw5h zjO7c|Q`T?3=QQuJdW80c>hArVjC*UHwH{a#cjInI1R`L?sb6X4dFiL`@rWsT&ZP3< zKZ5F^`1;DZ*ONQzdE(>Yb904hUD>p0(P0U>JoQC-%+{5>_cgyo0^~(b$SEu(bf1>d zzDEfd{FI~oVN{av<6{(zMUC(tH53AuUpun`i#kDWDCgG>!BD2vz^8mW7{e}B7Kl~5 zFJiZWK5l!Z+hU*kUWtA^T)vi=<#}PnXd1JqhZIY~$N^jwZl?|A?89EN3413@iLHj@8|i(3QCohePhgoC*^ zhBzmDzD0e`PNjlR^+nK;*AI>xXwxOTCz^PHn&PyLT65y_CqNbNB+>{HW|UQc6g9a% z*I({|BNK5J!!WP-0uzCBB1Qe-LnQfXoTo8V1~i!rQ@{mW~d`+ zMk(~$d(!y+8OYkq?;)H9A5vrUQ{<6|ZNQG*Xzdb5aJC_8i0ZhioPA8Pe2Ay-6TdIw z8yq?$t9h)-$WQQ4Amoo41gf(~j>Q$ew%z_-EI6a=8iiPow{8t3haf7oI48@Mx^iJp z3oRU!R$A&gn+K~r;0wr5GOr+u`o>p2A@AFh(j28kZC{7d^mO|pzuYbepv@HPU5Vje|EiHBGCd%A+^kCxRD-oJrJcvPrnrA%tewia{?F8~y|rB-c8NJ$HgK z+W3_G{K7dgEW3gg&gK(q}?-seY6F3nepW$tcsL}wN&jZ@tHhmi77&fJ#tSl$z%Y@j7pFb(3LHwg0;eH-V4ZhK7FkBbYL|-IpD)(p-eOy; z-=bzeFzu>YlOK)~64}PMcCxP}yVsc)Civ`&5h(D!mv6cpOk3s)$HnDs+T_qqYoZvK zXEwUr8h9H+)8FfQVD=`HGV#@quGh5?zLO zzG3|XwAUv*tt@}#4Cdza`%}nt(&Jb$!wSELUY3G|`~bxu``bUy>3^!P>{sFIjonXu zb5T;xtp%?yNuIuTP;!xIC>$|2EMC$g8cLOL93lkk9Maop6KDl;UepMg))90Fm`>ig zL4qxZ70R`4GEXBoU#5ICegmF*xMqj;p2mjk00BI--$IWk{;ep%_K1aZu>vW_Q6RQ9 z)KVWS3jvI})ryrM@pCag4iMgvq6a1FlNU`cLL&AEr{%GKML+GiTI%rh)A4aJ$9rhG&lZvH>P^Bv0(`?-om5e6b$%0{5fepBCxkg*g4l+ zGx?drDfkSLbcBWE#Q|WltbOzX2Y0~Zchb|gtvAV$HnT7Oa zP<8r!+v?*eBk9y_(iI@1bFdzCfRN0kMFB zfKsIfWh0_OK$>(LAYD2HNZ5+t76k7T+oV^TT8x^t5_|UCtk)eY zk(E#OG4n}pQi279gq(uI3C&Vm`7Z&qW`RR6-C+C1K6nOKi&1>#5t$LwZQb3Q3a`!= z^c*eCcP3BawVbr6jRx`Bz#Eo2n&QX*UYGlwQqr0dH39i?>C(1fjONyeG`gW&H}A4d z%`+zVRBMcVoi9Z&ZLHj(Q;a#UXYG4i#xxB?V1q~G&``Icw*ZCkRok_vjAwD$Mh~u& zJ~G=ZN7J8sUiDiT02}lkPCa02aTG)iNm(Xp0m}_(+!=E5GHIIb-x~y|QaCS0YJOm( zR;b$0m=v00_Gt^4sY6N->)%m(Kg-o6e(zN*SZ4mJdi(iE%v6%?A(Q?L%$;Yj2{+xB(OOZjpYTl z>&)?M&3qB?x}~(!qgioNbogls*4J;pQN#9(P@efWV>|R6G11UL(XlJm96M#;>T2_k z04Nq5x8Ysol6so8zS{xGZbiaH7ITN~1r=5h3+j59;kdcP2hDbdv`#;v; zBJgLCz73N}?_*?2|0(KHE>YWe(WN*4;fWiC&g;2@K^z`XSaxT~UQ?UUaKs)K#_pGk z9upTq!#xH*@l;#xD>SUtS0eij4^fO(S(S9@YJxT>0{O>8mC7Dg*%)kt35{0I zHBpm2slM>u-QT{%MTXc;MGlRI9+f6zXYety{OJU8+IYo#*Ds)WnENA3%bn+a zeo7nD)^z0+0ou3^-g>6srXG_1YS#F0E$ByeCkm+&+N{Zs2wZ-dR(;)|01kOA)EL-r zFeMnGd3RGSpSC?_$DAv`(i$J3#9!?Cdk$A#L&gOJ!>Y#FZxps%KbLG_IDU5QVn=}E zli+YEDdgj6&3>tS-O<-(-f+-`jTRP)hRLI}0+<38 zjkP;O#H5+n8|3zjRq~1zwp3$rgK0IeC&ck1ja)xqQnuV#2jm_Jobf4F2-^4)S2h@=Q1olwv;Mln1`s6V? z`?^O(8fKI`=ft-2S4d^^|yiTQvuK_aNhb%A9=A5x>P~2jSw52T2PThZZzd5 zqEB9_$vDJaU<<<`nL)hSpBw)MK6UI}lD}ncUb7M`rH8*cg)?_{T5$X9hc-!70 z{c7NFx*pqWZ_U$Io#X8rSvbsUX?VNor+Oea{$Z+M8zhPD+pAcgPhERI=oz*=%FS4mv>>u@n{kc9}L+O-S_t#i# z=HYWT`TS@?Qh|I5I$JwDTb#brc0mtC*rfGu4l+wKEJ0J*6_vXh4p8Zyr$xZekoN z_9Xw_0>g>5`H!`j*xI)jiA#1TZsgy;#zH%#KBGML5yLIOrVMw<*Oh`^{zC2i0gsu< zjEh8cUZ?cr5fx`EPoZ+G_Fc5Qwy@?ZQEXVnAa%#EWgk1|p@Z~Gr#eCv9;by;>qy2f!ym$_t@ zgBb#kMdAKtsms^)y1pt~B8Q@qMc_lL>f<|e6bxy55zP<-U4E7Pn+b zFShwIFK0Qs6|6=jIs}KSeNt0jF_rzOfWFLEV7_FPhg)s(rTUlji&uDm-`Gh4tI_jE zxtG+F1R2|eo$n@`zU1dF6hm$+xeZ(qxUthp!^`s6Hcps3IEZ!a<;#crA`qlGezVg1 zjZ%EZ4~;WtprDBMV482AJ`kBAW|@%S)DkKhrxkw1ZKP}}THPb6N9JK5W>8DqK^XH2eAx1dA?jQq6kSO98>YH_+nIvqJ(JCkI(zO0V% z|JNZ|AYeSnjiC$~wVdqfir`|*(;tt((VPEwrc1rcOrmd$n9Z|W6{e1(A9*50RVOE$ zrKMEV&Q!R3|AbS8&%@I1$LcGou4Glv+l~kn%{Hhl2(_|)Ccvcn-J?&FkUp5}94hkz zYW|Q!ulOKq0LP~0&~d@*!9M8?3nO{#UYB9))_93eio~Y3)X?QLmGz}zyOdo@^4-%} z&(jP6ci)iq@hDYPt%9*i2g+5qs;(_8*o)rxlnxaE6+_%^ZtgcaT;Sij=d&0`#H%xg z=~xVWX%7RWI=6U)MSmAqe760{+H-HTDZzg6gh!1J94d9{Il zLMIam$6M#wvc3T}hvIC`FezCRNqJqBnQvU7Y#U5d-+o17uBdYn1VUEfny`ArfGU%p zfVTeX+33SzAU6&D0HI*)7C}0kqYTEeHm$`WAS?o@<-_cFccl)#DccpUPwA7;0N9Q24qs~weIN_NT9|aF zY4WZ^ZU@f`PNn(!)@A$WQ$BQ~0%-;qO-0AalzwP3E^vp?>+>yJ|C~Q_`)^lIYXYs( zgl{BsWeoL#PymaI;7U;e(StJE;Gc0jV)4b`-(Z7`bdXQ#>+1ta!r-&K6{ys2epuU; zgk91jtJ1bG7KjX3Ms0Mqpgymo7c1-KJ-O~Dx36e#rP?GS-D_-2nq!EK2a$Cm#0W+0 zbFfna%1_@3wd9CJvvwYSWvf*c*L~$Yc-c&$P}t;7fPLWVxpiqB!<2^>ET>&bg;pp6 zU)5U4-qY_VBDcCy=&0d5#0N_?|8KgOkPiR@;S6WSAya9HSx*BsqBYSmt63AdIj?h$ zkNA0>8aY7qUSwMj*c4yIw*#M=H`gA`?Vf?!%-tyNg?azQnS(wYP_( zSqbFCYyO=U2L|-n#E32|UFF%zKoNR}($2uJ7v?%=V{n@m&|ONmm}x|6DYPaq(8uDJ zsbJlj>|YRX>msG}SG@7z_9NPZi%%$}(8j51F<-gE@lg_c%tbCW6<5;Cyf6v>DQDV6CQgy&60ydSaLFACK%R97Q*3P?#+sNaQh2WvDc)44@fG zWqZF^ucErDFL_Ue79g%)wB7yjAmy zdvlqRb|0?=?WXG9v521;^6c}D6bW@hKttT$J5x%-l4Z+8>iONP&R&FnQ{&F zWtFqTlAOtN-mLC1UKEJgC4ML_ptO}c_+6@SWLe{h>q9Qg7OS@2-S^P2|LxxYH4hExi!Rrr4y5L!m< zHZxViFLrnmN@!vDM@BfirPoV4oC*et^Ur2?xGuK4ZVRqBM3Ld!$vy~$%wjtliZ!`y zu7oYlUD$V#xU}aaly;R)9F{cvc+G1Pz;5;LZ%LOy?>xfZwfXEg^SWwAoVZ*@ zU9R29LbNDpH6@f>xh;ety7#US5Ai~!!;f_&;t}idc%R-P&(N~s%Y5SBW;z|QYbZoBT@2@}7q za(Wx2N2TK81UTzXd$Yv#l-ca?NJVSrQ`Z&?nD`CIB$PiH zUGuZ1&VXjZmd0E4@+|xd3S9m$i_nxse-??j@nJY$2dyL5_-V6)PG@ze1lR{t9U|Gq z*1V-j1%DglP64Uk5U;L&7?jzifN#|B`(hOk^=tKn?-BWgtEZGqNT}WK>NAPi~dw*~B_hSly9yHDMsj33PW#PnFn3*)h;r6WINqX1r zf&z=ED{PL+8q%bzKN~4~_%(u;f^+SBp4^zm;azBpnCyI6f9zxyk?NsYH@x_^qFMvu zV2)4HSWe1AAwXtLx(L+$M;+Q`qN;KCHxXsGk-xu2@EL`IvJmAG@9w=z{C1&Ro)X18 zn}^cjk9f(7MPMG<$+N^P*fcCmQ+)#4l9I2*%_pq9*p$fy&)4FV6uyYue64A1w?1Rg zDTrSzUkW}OLW%>s@S!37ULXQG> zhx7d@|AZs1%Kv9?g**0?-Di$_1iF(+$-c`RRjQVnk;c^B4lCr&Fikm!4&G~{4#~g+ z_Iw_He{)j7v8zT!%gWf;Qa)hK1Or1XY(Z}tU$0nZ{3fdQWd2>I&Y{hvfoHD7N1~w8 zpRUPw+ktzo_BiUC0b>?2|rcUzW91bYH~weG%iLA}RR`K6x5`tsnB1w>xbu43ibJoVrQFoJYS^=|QaXqYk-88Mb$K+p@2NN8tlu zgHwMNR*&s6HvT#&6a1TMZ7>)WAokPKxu%&lmL6DPgjBT@!G@Y z@YShS*vAz>!@eV>vZ!Tjh`7FN9<0jwVPUS-zI^Y^@dC8LcJa>LR1F_W-CNK1f-}nF zne;HQf%BeHeD*Vpq^|hv-vy>zowF}LWM}cxaj2&jx>3`~p=DT>#u@S6;HHf0zY3Hg zY0+7nVEj7+fO}FwlGfo&0P~KfUr^@&C}QJV%ulkR&la95C80ej$7hrYUNQS#3|XL9 z^#S>EyPSSieFNE{SG}j(++gnw%=ooe3;D_ETa@H3A^yK=->Pq(yz&g>aDmnq$#*t> zz1jHcl|RZZqP_Aow}chJW)A+iMH_6q|Ls5GYj#KJ^bP>Rg zPWdm56&?C@v}N49-_1E_{wuieJ|WQyUblhO=8w*T)V*K8WnWGO0F4I@a{{X_?_*H4 zmjUtQown>gPRTz!_O@49`yiy0K0}zf}p=F8m^^!@o4B zcvReuNveo3u=_Cj?WzG>RKsmJ!kD+`2r$q5wuoq8rIG- zx0p$p<8iQ!sWWDrNLDt;R>SUL`{rtoFVAzlOvl}S_Sy=2j==93)TT&POZvUYthSkj*xZx)QiSG{liHgC?kmj2Ew{{*qMg_Ly@eF@2sFq2|t5D-W8b<5i z-wFrNuyj_yoM93$q(58&IQ3jl+QAI>;HNa&Zx@m>>M0-T z2j?f@WY_|RoL57%13;};?hQ!C^T3I9x>o%e7oV=rOcla{s;q&^G69?MMYc8FuY)9` z#dyxAwzk4TGFoWgyMlLTBO^D|s;tv?)WAXv`%5Z?y=JR2zSa~sC4o=IJ@Iq6KBb%% zyrsVVt)hoX3aeIXj5M%4a!x=&RE*hEms-j#GT=9bJa{(%I?+Kj9N7j!W>`bb&-hMc zkRR#WKVl&m>)}FMZgW%9N~K%_K%jf>76WlB28230EPOP5fAvccM?nRH2u|o2mtTvJ zbXeBKHIi;~cE3lRscjCB$)S(AB#YJpaHr+%P_CYOvWlQ+Ez5fBt)4J%08wM!SN z%pAIO@#2bd%a#NbO2%j!S%moUpP|_{S(x?IG6uK|>ydU(>*n~uwJNQWzm1moJ(Q|& z{8G8fuAQl~w};Md;zH)OhmNN+EMrdXr3AV+gwtFn2jkLFH5w-tl^@`5ni0Dt!;j_k zQ7(q3HjK+$1KSC3{0lc0w(P7+=%hV+=SA8YZi|vCVz@)m7wJoyfsr+P(!;-UG0gDL0qu;pqemi@Ka3ICFREX&|s zB6Q@vr2fr;^=P+7ipF~k?Ii{TFGN(*N2LcIKnl5BA4@r5RjbY$R?$AY_8Tzub!8$D z?3y__oa^pKs%*N-d=rakt#T{AD|9Z$={vT*rXi5tkGSTmfNTO!08uIo>06g8d1E4T zh?jz0H7IF(*YAZpu0FJJrV0vRfFNqxg{1s^lVSgq(Kc0d=c8<8RO`Xv&W>9ZAJRN3# zi`!LNuI+a;v3#`<09|lmt1L45$?)KzJqOpRW(oIt5EQGj{2R0nP9{??2TOzHlm>w( z=9jJ0kBCco7^TkqWz66*BJnZO%5=>5-W;{|B?sT4pAzNR(fJ~M2jAYKoNDlc2emi5 zXz7^Z9s-7K;AUa$wQx7(41!n~_!UGsbL#Km6PU3g0LJ#1HZbv_g*)esy;7>*-h23i zyWc53QUr(KFWc@O@61d7mv)0Y0 z)$}fwDkNmT*Wc-ZEXtZa^eYYyak)7#-*lYMBf0Qn5CbK)4HDP|&F+S7zo*2D=XGZM ztyHxq3784sqa@*?wO5W>v^ypj__Mj6P4A9Hhc>$HLXik2a0tq1bQs0#h4AKo|22kw zAz%;jw7=|ULDZq;;I%3`ZcASuNc|_urL2y>o#Q_l_(kE)WHYgcF;z?KSGxAnAcJED zxEQfyp|!O3XM|NKa(d}K?iylR?$dbl?oRxL3sfnNl-h(Sn>Nc`x@>RrN#Xv!UlD;% zSrMW5;FYuhQAX!MNj_3ZXhm}uq1=}#n8py@V|IFqs`%}ZqluZh7W4nD$W#{GmQR(9 zg8&qR7$yD& zGF*J-daQjrx{?%V^|Z!g&7@#jXwyD!w;#12{bd{&o|3;fU-@_Z$)R*&068?uFler)V7{aOsm0hws6p9KL<1aFGJ@Stp~Kie+GPeW@2r99poa4-nb|$+p~3fRQy+; zj$nrEf;Csa9Uo0UoAQ5N{y!|p|1VNO6XTZlgM)s|&0#dZcI#=MuFOc$WMEc8AZ>h!VVo)de~lyS!G-%f zHa7NJG>2iK1i0%|=tVg>do0|~KXTu;96&q40SX;6ftc#+>b3}pF$SZ;FI~QTH?N;# zH2)r^Y-DUK4tamxJ_PC{pM+-J+;7wwy1Hd%Y&@~VlP}%4ihCNQ~w;-QP#bw}P{a1*o?Fn5fl!FanV74{y*%5|4Z&IXpf?y_3QH?c_1$>T{dRKr4+aDtyGVm6a=1wq*|mciQGPF-uZ zHEdjXeY_6jQ2?1N=F}QlOLZw-7&S77(W~;i%88|vi4+AppRf3tgdD$G+dw@sCo=57 zB&1iFq=z6p916cqKu?oOo-$)I^1V>~hGGL3KQz(@Dk`KZRAoVF?driQ;L3?IXzCK^ zvDJB>Xo7eEQ)i~%E~P=<5HprXj<&bOn~qc<$%l+*_Vm$s<1kPxY{`;%RD1RH5e%7S zrJwQwf0{J2o)$WJK@8cT7JXFR&pna!^XIsst$*8*q#d#DOJiGcoE>D~R8=5<23k6l z-l%S8V~!*%M3ZxVMsX_uZRVbE)2Uell|?8p@JNc#*Tb@QHHyazYj};oZZ~xL&vKb) zXnl+w$QD}S9nb~-KdCWHO)~m1&6bnd8hut}v~+C22-49Ny0(=!n^Tv%mWJTPt+^*& z0$j03qx+_1tMo@wxUw@xUGLnvqk@#)d0Tw{^JEn6iq8I!)f7ET%h9!&&Iq1auRG(kaq|CE&wW)7NaTZjt=!x^hgma zB2%&hkLS;9;E<9ZzJRrW#3-*(ddYqUsMP@Hb^5Jsy; zv5>(5qE3Fca)`Zk&5Em+zc!xpIQ%-C=S;_>hs&kfjHkBi2{Kl=zS7!+r%#e=vj+#q z%&0DoN0NDxojoq^KYRG>jmxqZC(GKE6@F9j>7moPb1!o}y5uuEwZ0>i?&CPi-25`N zk~X{(0wvEnbSpe05(Zr{zp41S4x7Gwr|9`3B$Az_>^iha786`+{Z+*!dqFUgi~1g8^vCrI9wU+W1ytIV-u>bONn z&7q5ot$tWf=@A3Eu=j(cgWA6uvpjcLdlch@EtgMp9?PT2Zo*P1{^KVjs|}>`n5>BU zJXuQ=&uA4I>NB?aNoYWD$-kMRV@2V3HBC-dEYJOEr6CSOlQ#DnobW`9-zAH_M@=en ztd4TmUF+jcIkEm=V`bgHN1&NbB($6f!7Vbdi_`BS&+=J>~?5 z^!u?e;msl@3b|10%#%=D;@+1VOw);0D`d+Wl>CxV&{_w@1n8fg+uW2*lq zwR-=jAm{b`FEKb4BfVS@R2H~l5g3cdZp4Uco`Q2irKjFN4qT;0AJ?S+#eFVu&|)a%FNA`NbFQvyaG{Qet{_l-<{V#<^N|` zjC5k>PWSV+4<9}}kvD&*=%JsS%d4p)YFX=@!xjh6R#uQ8V=X8kptNyf&-%cbSC(eK zK8G`b+!`P`^J?-nMO%UFRd;vy!XyS=otdmR}h_9sj^)}kRmKN+r>})2AZtuic*ut4~ z?&{mhjfEaTJFji-4HQyKmy+FBn+J)66O!skH&9L3-h1-Ci?&M3)%KdT=Z9Czc=SSP zy@nuBs?vouZ+woHw!TOcy*AtA0~Z3Uv}Np?gOamj%)>-*%XUf&h!;2YR0pn3etz8A zld9?iA_wkc)gsrP{REC{M@PpgxNI1G#!t3g_Ov>Dfsjr7F_(_{%nCmio2v;^rG7Nj zZGSD4*RaJ>vafZR>Gak&W<)Qo?LGtjx<~}kl)QBKLRtVxHFS4lXMGNquCX(l;wcK^ z+#oJb?ESQ_K@JtAzg49lNA7l1bsp>J0PDjjmY-Lu~WO=7-%9V#VUfexc_ELjSo02>%c)44`0KQzo z_}&{=R6sJaz^65OcXoi=Q?4_jMdY2k>l{2~QrUzt>B5E4-vOuhXI$lz_E;ct=3|YY z^{%Cs)|XaS`#VJ+FhXmPxGq|-direIDrqB*OGPo`jMi2OybQYIWmW=_!KXR#rXNbK zn1QMUl-im%o=rS4ARF>v0D}rCXi(!7KmXF9o>ya{7|lx~PfVSDlst6&L+~v57W~Q8v;>HPx*ciw{IR!ru zwv+^{MZHEx5M;-~Hwj0+Hs$339Wq^Ae_QAAHC4!dt z?OZoS$k&Si2CC%+qMYouVN-MS<#2sSwOkCB(Z$PBzlonOiAW9p@rv0&f?d;HVxU65 zIQ#VAg1>A?N?ulfbiXzG8P12+z#bDfF}lCKX;-vbEwho z{-*cuKRtNtYWs|Ltqr(;3 zJnpq>gm;!X1R|Oqmz!^PF5n8-@B00oh%_f?c$25 zzc1Ne8PtD1+@>cMMEZaBKkK^M3UU<@>T>gB91@?d89>m0jK#L@yXag$Y_}x;keK3G zI+FEsps~J2Ls;Hjy;ANlrH+DZSWb5J4E#B`AcMQ^Pl}IMO9`ljIVj|!))FQ`hxw&f@9il`~LRlm7iNrBp~TCO(3_kL6?wpGe$t*~OQdV4p1-;A*HZ*OColE;#ow+ty_@X|Fug ztJwxeO{@u;(_%|qJIueYM8PRS3B{3Ublket;ZMytn_@JIpB^ z$bOdD{$stjj%aJ21JHj^=l^dFDrvX*ePJ-;*z@|aZhJa9`28zaLsqs9_SOK!!Zq{Q zTxT6TI8-1_d2RnvqW=f`U^WkR_#q)KeSTAD#?ae1seYrfRtt|>=f474gxV}9g*e-t zx7JQW9;~OaClVAr`*T_mI*!UIpQ%=zL4%^#Yuhy2I?aZl`%kk>PEt%B8xw zMlL=mYg;@xL6w_p?Qh6|AGe1auOV)low{*&RQI@~>?zg^k>?a#B5djQ(bsoUPF!H` z;VV6K{94L~o%WcUNpW_6UR^6xpgSkK(K`2?`m2sr)#}08VH;Rl#YMb3e^9q2iBLF5 z>q2ey>``Fs&3_bR8y7c1Q-T@$PbEVRe7 z=NfZF2Oc0TecO9bnw+Y+^Un{urYbEy7>%u6GJvYpy*O|{@p94c;`bi`XaRB*ll<`h3?*BSNE%5(^hR(yZlcX#NrH2-rJ+Y;>6&{*0z>dx3PWM{nXO!Fq z;kq30nxnrx)$aLp=H@6yA{u#4I9W7syD35cR>v+c%O>tT=21uZsFo)8sl>V}Ps2T; z0iF4&^hwNdm{=cyW~i+-dL>|}fuPv$fNLBh{mjdzB6Rqnd%Xza(j$JwE7DcECUs-SVfvj zAiM9pt%BcPS7C$*-vjkNx6QgLF)NE&@swQg?$XrNUfahKG6l}_I}qjgqT>9!I`@nS z+4-4z`?jB&L$3R}%tohFUQTp|&X;M4h1^e9gg$HA*DV3XdF+OtjOXH3sW@=5?aoU^ zstXZ-g9&QWRf9#v% z*TvPi#JL-Fi=V}Kk#0p!(z!7x^Q-e^f(RryF0#uk8AJwZ!`{7+gk5O_Dz-uJ?J$!% zE8K6>k#~!#-J@cStXYOZ%8Q@h7B%&4qM>ZMrF36Hu$b32$-(eTSj5zcqh~8vGpi=% zIr?6mhQ#WDOkLyHQk3dMOO>+@?8wUJ6H5dn+b~IyG34qwsI+kzBKEWnhSKDT9amRy zlI?bI#=Yif2qZpx?aeNi;zcmbvhfPvu{FvDRw8f~x)Imnfp?x+R|#1cIxgk( z7FDhbt(?x&{m2LVJeS;EsMsn2*3vA7{OE&}K1t*{d8z|rmvtZeQw|yE7zDDw*~TxbL-PW|(r!SW zzsh#ajYidKbytd9dqSRsr+e11e$?$4(zhAXd%f?PL232LYYw7*A zG&^vxP1;}=vDrMKdE2{Fg5>=phpp*PNGWm!StM5Vjz20t9y`kt;yyYZLVrC?JhjxXu}TtlWr_^z#g#zJzT~}a`&R6=;4hN` zu7wLnQ!vXTReC5yq)5PchNiX0)S-tEb+g!9*#jK^^Hmc5mk87UD5&w*F$Y;Zzb?AB z6I!91ZLOJpLh+I-xq#|{5)PT)XL~yij#E~rv#8%2tWqfOji9;TJgnKSTE?S6{C4)V z$}!Wjen&3I(~ww2=ATaH5bdeWHcJ^;ozqe^djfXUYSDjW)MEU~!XlTIXYI|vuUt-* zs!NT9&HILXl0zt@gQ(>U5Yw4>U!mp}Kqxt}R5=P@HZkci$WH}{OU|Q>eiVt9d8I(X z+E2LLIoX!}T>+;$_fzjXYj-smH(1{;$^O(o6Jz7u#v9S2CyU>YAXo$7+%#x^d}@ z{7W&+?!61R(aA5#l8v>ULd*9P1h?uR^?eiDDvK9~G(~&Q9)N^DFDLSyM6A;C8aDDO zf)ma=6UXSxVO^wUUHe)}GgqTmX9jS0;Ei;fD5GP60> z$a9}q7<3$!t~ggRV|GC#fciNAcb$8Q^e8TIrUS(wkF~~w`-hE{!;Qd|I(US9-j#mD zyh@K|`8tm6WxDmY)nn&trC;V#l!>ec7S=xWRVHvd5t(c$@H!X;pKxXfM%Jl5E)`ag z9A3*nBiP%vRKN?`YWid%XqNb)%9$KA9Lg=C>5Ed<9b-Kd^U8)mg7g7ErflLrkR<89 z`*rC~c~m+Aj_@}^JwJ9MKZV|lN7uQUO@zfx*b9&QS1cv=E6I^VTj;XvrEeFM#B3B2 zPE?Q5zk%zBXZ60)J%I5fZo%p*`GnZLP5homoqbcgTX;3Ij2_dtT>?7`aj+^$@$H9+ z-jAbHZ~T;F(%)MPW+WHjm=g{vvRz7=yF+OEKW-fMLq|967BXA|c91V328nw!XeNB2 zV=*$Z{=x-f^fpP(Q&_3Ni6TgLii|!1IU`pKl!QlYC}7|Ga9(_%8>Kw;Y58kbfrvV~Be=xz)%8}EZ^w7qJBsggz)d^98UAQ;$(eAY z5k{q$z2BhOIUGZO0Cns}-@?bZ7`OM~i}f*@iVj~B@Iwt(X1}PEi#x+dPD;9$qy26T z<9vw}``Ve8@uTl9x#RR`TG+Y-feYfqvh5^c{I!s6cOPYjp*sA(20XEHz6v&c_po!f zFb|oqtgKDx!=HLj@t zn0y)^l>S;;&+ge?DUCZr%r69)M4aZ%7MeDER=Q0 zSv9416!I<+#8@8iB$>NgIe=?P2})zOgZI>#3GeALJrXTgJiBvBC3<@t8qjlz0w3XL z?QI*hR`z83d5$YB^t?o|9Q*XgrVOz})5sywR zJT{9YG|rx{%G;0{3%qSQ?)6i}139^jUbZ^c*HNHCUcpoe+b8OQ?ySfG!djTj`Cqe^9JBZy z3N`!vM!Lu)NE7Y`STfb`O3_xRh-0xpCdHk>G9TSJE*bBnb5G58H6Mf|;1k|(HIcty23ptSpXR6eR z8@Zi{5&W0*ZveyK!)R4d0GKrF{Q-c*Uj5FRNm{TuEPlz=ta=Gzidhlkq29Rlx&y_U zfe8++C*AW{eGc^Mtt}cW&+=}DC&0}6R<}(eKCqVlDGYinmgFzRt2I ze4#sCUl(figZmnagP0}`9af1BK-zw|q~KQn-A2yHwsQ2~Qi{^GoWDmXhG*Gv@r`U9 z&B!h*yK+SQgs^(LTg~Xc4!?Nk!}|9Km5Tc1zlKyS5pn?H=18nSBBqHphY!{R3P0Xm z7IHMRQYTU{$w=fjoxK)QdfK#d{m^*z-8s4RkX!Bjc(;!`DHEk^{oxqoc00Y=j_ZSL=*Hix)c$uLw$uZ0&3hX6AYm~aoj2Ac zM$cEW@M^!%aGrSoOr;pI9`8TI4u4k(WmTTDI`7;)kRn?jVC>(SH*C-_J4P1v+r*a! zZc2k)bjZDxg=IYlheK~X#a-QX3vJ62W#scR|HuLqh@e4;N)s26-L?qeQfkeUAor} zCOGl@=mq`|>PH2S5V-LmBg)y^byh2I^q;(Od5m(m@3oy+&*gfDTemH6m_k-}0LwjT zSu$NC+J4gZb<~E(xV`#eDo9B{ zkg?O*{`!a6J@5BJrjGM%#!?btaOcsw4s^|~IlK?O3u@I;aA z5Ba0-P~$#JT~o4f7s_)6GO+7~h2QdMQLrWPwTe*V=VJMki0sk1#&w20iuLf;?PO)o z1pOQNLCdYJp%Y1=)&XiPNtLv!VM$>F`TF!gZ`_@9lqo<7PKYT!+OEbSC(`vqY&9tV zI3d)Ehvn2aM=DHy5OjG+A)Y#?Zrtmg$ah5|E!q5n8zJN84no=#0PPxd+zRW`K52lh z&&rZW1a(!8Fx%JxbJIz5WDz~uuEVsB%`ci1$ab(_{Bc>oLg3DBA zhWW?(qi~dVJ+x@GQ=Cmp{kmQHRmNMoa<=%TW$%$4x(2=XAS0ed`UK1$_Fz33a(1DtEZFuRijc5|X%KooBQ|pg4#c#irXcTKKsW6juNxB#5IEc-E+lk%6ZN2MsMq5VxRP)I=NatuX&FOO}cE z^dSCj!j-MHYb{5OYV{QqoFl)?6#Pi|@_z5$~Ef#~ZR{#AP8 H!SnwOY&ki8 literal 66768 zcmeFaXH-*L_ct1vqF6vs5Tyu86Db1Hn~GSdB2A=7?~z^t1mq|R2rAM`5QIoqdIyzW zrS~E&)Px=&K<TI?*CfYgkUPXWSN3)N{7|d~o51+? z2MbOq8d=IOA4fw*@j74FFIc861YAkTRY@TA<=FVXCI7yi_Q+h3Ku$?Wp;cMq;O0~e z#tSvMgIbZ zxn{U$y#R|ho6A&@CwejTtT$bxfo04DvU|EZTi2UZ7FTzq$8h6FmAG{b^Vq3xc@Ryy zxRZ<7t&wl8zR;KB=12|Ptr~c77{$8&ko2&_;y{pu(8ml;mPWbyYiUtRtQ}>(IC#F? zkX6-yuX1n>pC`=-N#S}pmwng|KT3r9#GvDgEFO!TYD&CGk;RcER)}s%=`;JIOHx%v z>af!D(~c>r!fln8k)4rdlq0^bPG2M%YS1^g^Z4~+jyieWk2yAVk`XdG#&zYBJC>F4 zlcELJi^f-)=GSVMUu+akoiQc$L|-wp5bOCanq=xP%Z&~gU#8MPN>yp5#D!nq(Z5na zQ*%l3xn9gM$2vZz?X@ju&DcotvA3eM6!d|Ycs25tuGhge^VmO7>gPMcgU1Hz6bC{Y zh!7PB`bm`#rjUh4QdOrTUe(Kd`O9toG@k+W&DYE)Ej)A#5|Kygr_J9_Cz-$B=o+0L z`O3Mu3n_SDxm!0k-Ntu48PB@-J@qTbT%EP_n3tP*n$!0qrcYQq`S0-FdF#7zo+V_0 z@9c#{=C|ziYQ}-aZDMy%2-SwMI2f-KwRO4?vED}r2b2a(yU5n;ni)ro=C)SfMF>;& zw)VDKHH7xq>WO*UY6g_t(m<4SJIpv`61T{|cO!uZuahSJF&*}al{oq#E zA$E`Caf|?olI{3ktaqf(vOKdXasO*!E!o-Yo@9TPPmRNpwFH@=hrWlP6pi%PC%$J# z0?JkAmpbidvKvo~1zN=B=0(l=f2*GuH$0kkmFvv0$;x<+x#H=P^KHo|^1{t0A)jDZ zMZQAs?C#JGxFwl5AP$Lavx{RN=8K*jO(*$|GjF`+Z8lzpfuu_%yRP4NWv?Lu&)%Mabq}QWN50Be%_go?A9Dx+Z<4N$BTq&xJx*LIl5HZoZW2&eK z;R2t@Aw+%_5K{1o2>eG6{)0e>}-vQuA!Kx80yZpuDzC7K^P zQg~6_jc-(dZ$5yfj_c!#r!2QG9*z1~7|-F)QTV=)S~C#m+h+bKvGCNZqvqc^UOXkO zA|q2xyq>`EwDG2ie7@=#)hEw0@LODP_1SetY(|_;*@KaSIFwS4{cL9n_Q6^#tUT2r z%rToYlAMxR#v4LJO!_}Q_(Y!4GD>c#sXyR%BH|)%h&P4LK7UHMSyQ|>r1F?Xsv66_ zcPAnyhGhQtL;wDU&lmhEpp~NZ@_?I@LkbW5kLCVzugYUb$stV!Nn)4&Ye#=y!VDRu z-+#wM!~z`55I*0~*>?xr9BkNXKUxv){J(U=+;V&meJ*DfvgkTA3vJ4F>%kTr@wkzS zZlbJ;-7UwZYT9rM;bGVD+;1nu-!=N0rq(`!@2qtwi7yCpY+6DCmgPi^`ke=CI&?Aa zJzXtUjtfuJ=MUJHUQdc4laNk>a7y$9Y;k*T-AqO!x>C99uFXX;VCtOd-~N(V5ZFD}WG~etA}SpWKdJl&`hUN z;fl@XNw3fFZu0w96}NLoyl|m7MvuiWdDwH;esmyqSC>WP&)ZdfDmfXto1*M#>6RW0 z-yXlQAg(`_{@J4ZB1r*!{M-mfqekaieCwrqVUB3$f)Xda#iz86O42*4K5m-AK^@{j zB`V>?Ch+!O`KY}88ggdc7;hCgcUJFmQkfbVWPbgwo2iNS?hldOXTUq^0qjt&O&Bh9l+%*D|;$ zhptMF5}m9oiFje_78S#|J$e{9&)F@ieJimWf!{{0BBggYb=@XI9q5Z%M8*SiYZ+l5 zn7~z=c~{M9`FR7|O`4GHbw6C2S~s5YdR)lR`plwSSbaukLN~5jXg4&=_#3=m{sQda)we=%9oh0PXm{6)^B^1)H)Ma zEm8xQFYHlxm9ymJko2Jb*SIPjh#Z8O-VGI9zS#^7d$oJ`{1cgBCi{76C|T*!E9m(` z9JU-=W)CB!NRYLC&~aIRv*Df7$Xhdf{9#G+2YzSol(!W)ZU-7}trt|pUK@P!tz284 znVDD?>fRCEAfrs`$%rkiP0DEzyKGZ5877Hk@WXz5RHM2cXHM}*yC*RNRUi}RflZIj z*>;#|Pnwa$xGC?_xTJ4Xu}fWmyl%`zA?|p&?96IbaJ?TBb6*P3f5by8;M7iI;hD*$ zg2h>PNuu-x>aEOn+q>@Z-$l4zdhNegJ10_oC;M*x2sQC=;n4JaPAktjEb3w^cU8!4 zHO=u|e#jGrHuLk!uTMtpu1Mo^1|??EaK`mdev>2uHKT#z=t3j6M*_ZNEFzSzcG?&^ zt@LsISM82n#61@-eg6m@XOZI)-8ig!vMkJOXWKp2Aw9sh-fKAoF%gX&bZA{|GLy|y-S=oLPxe0OWEgBP56xqR7pka&AR`C5htjxE~yZNi0%>%p9HO5!U`0xO$d z_&@>3_pD>xJ=8C3pGqzIoO}mWti zxmmB=(j(VnBG2e%Tr|`)PaZf{k|t(a6^nL6#%VdVuMV~~q)^dS=Lrb0BboQgP{e&A zUC68VhW^jD4I!hhZ!73JKj9Xa1Q(cg7yUfe5}9_UC7w_!2O#VlI5&%@4}CN@m>b5z zcTw>Bj+dy_R;`$nFP)4Uvw6DGBF>9{$?k^nK#3!VoCmIee{2QB%T-7uye}5f3v~F@ zd~IO$UX+UsHOjM1jyB1uTbo^S;=;*#xMAR55ef5}`kqcw_{zglpFHL+=(nK|Pn@NBnE>C#cwtLSwkmc8Diqsd>qV03kQeAxzwsP; z8RW3F=qDXxNZEx>$#nT)_v0Du)VS;*N(y@iSd^V*1Z-$dZ?o{!klV?*Q$3ANJ-Xp$ zYOXu;X43cA5gzu#hdlzbo_v=*9J;1eFfUh>z`H0C>}oUx>{lI*IN0vg*jC`2*>{@B z7d{q7{wu>CE6q5zlNihI+~`R2c=~Rm-j$Ns)U(zV-U{r>VE03-3>xfxQYzNgWMrWpK|h7}-GVSDyRuGi zj0d-(q*o$aKe&|5=~@64)*SA)i2{bk$V;+XhcsN@%We3i$D@k2o(+Bcnm)G4vUe_9 zJmgvTy*DGXgTB?}U9HK>DN}u&9@HbcRLAf@og-*d`U~-dtAWGI)7Vp6cD^Foa5mgD zr}^3(PTc8z1?r5w;+552#Ua!;FT`Xo>o)SG6;g9aqX%|Ds>36A{c3j5FIn}G<9aQb zWI^vM(>($UoQTGs2R70(cq7QAG*JNy3?r9vxX?t{92%S&b_g4S*fx&ju?n1v=DB9P z+^3-_bI2YWe}9ynNjA&N{_u@b!Thd6+z$BlPe!W@OsYN6W6|Bh$?FJ zw2fr5Dt&Y*%bKsv9#<>9yQH*!7{(40NO+{#ENt~?T4IbUc-r{Y`?QG=p?vHhvKW5G zTic{U_HuT7Q;bB;trxQOFx$o&caD|1r!0~kO(+IiZKEfV%4;$Ho7{_?^~DhnFX12S zGU7LC!)xn_iDIoQTkMNOG`h`u>W5wHgMq8kCvs}CT%N({0sFbSVux1PaLyym30j0p zQKPOHP&GXooN%XV;8dxjqmQfT4COX>Tw}Zud2N|@)>bQ67Nsd^ejd{d^>SdhqC_|h z&Wf788i-AQoLfA=wbF(A;eR=E+SpgqIonQTYicvlOd>JF36XtQib8uSiPtklVt5F6 z5=v(j>T?>0AIuudhT9LAN#pw7pP-)HE~&tk_>3sq?U;ONR9|LTFXp~#o~n1o{^S{x zmw$7Dl~?Lx4c?2&_Z2RnBHXc%HxVwcSGll znAN-v^%%0Qwb_lpImQJk<;J~gM2i8a5t9t^@rW^tT|;_7$|reve8NfmLcxesbx0O} zHFfz?CoC&v>!QPFs7tYHzx{|OR*@Ih*Vx`HJo3mk$nH?8a+w)?Fxk}~H-g_8X%zg{ z663Ws!)X=Rj2Potf5nc!)@$Hv$ZsMWKx0z2-fz|Dw9;H9Qv_ergol=|#9~$zqK3#h z58VhVIOdP+gt5gWgla=Ra4Sk|idm1$EM2sb@L{-CO{4s**86Lh`z!XliMz^TGap#` zo=R8ju4N$$fd7y-kpuKtxH393`*W*59Yj3NJa~!H&_7!QLr7{}6=*z^daeTXU;_bcrrO&986V{{4 zH>!E3pu{xLTdVhko3_&1(iIqcpmrxz&04+Ia~f^zf_q}Wa>OmR63eD}NS}s^-qc%S zdl2JrhI=;JX=pAx#oc0-(V7cU&UYNDHSMDLt-3zvTo^XX)o6uPSzb*GZ3Ci4=+pv&cS_r_}vqpq~EBvY={BI#E%G zljwuw`eLG45bpRvfP=~U{D=zm#oY%Jx+hQwkeNuvkL5P=I?>$&DMK7=?C0eRf!>@d-7 zn3yy4+&0!FZtgQHf5sN~R}=k(#^swu$|67hQa>SHCq6fY9F&e*Zz=x3iKuHSZk{$a zl6TBLv7|Vet1O!TJ5pQ_ryprp@TFj`SkYjWF)7uG8>`rtEKmN*o4*x#H2_Mp<*3E}N zK)b6|wdL)Cw0ra&S~9dg4xd~~miur#$0%6xTC2eD>w5Yf)L`QqHVQ5pZSHA-xwlB_ zk>hX7wzs!>D&j))C1Ulh16tE{l0^q#SE`~(>~zoIOmyj-b~h1tKxU%QbI%^1c`!c+ zT)HbARRdS6b=cmRtM@ z-EiD==m+ZK5!FlCOQ+nEQ%bUD>D|r@)pZU@w^vKaxs3@F2T#Xs30*ZC=X!@T1;TXP zz<9?pd}fTU?c=ayms83$e>uHAwSn0AS8uPDOy9pme7CwYYyNYYIAl>6z9Q^E?-U?0 zRvei5_$bnRG3eGfI^`uq=8c&y(8;uJayOh+(bJ5`aL*4EIcAb`+9v(e%a@-r52@un z=38zaPMAr5RFo~Q%?|5`r^Z6<<{6xOABPI2<*>~4R5S??pDRnAzw-QQ^F6)gw|aM` z(qS0bh)#0w=nps^3vcby<2K$|bzzCuicwqBJG)+NovIgAd^oxR`6&wHKv2ocCPszt zocgnHAP7oDbyi`D9%rT`2M?`!>S1x|4|#W{(opo+u|haTt|-;SJRx&#BM`^9fPzZ? zldK>Z<>$b0R#^|9D@^8ws4%8#3d?k%;r>KI0Swu@=oMK>Bhlim6z37@xr9bw^t*PH z*FRV_JUuG+*2AEM?tE?!jW50&cnQko#y5uknF{&6h5?3l zEoSSv?hW*!Q|Zyr3&npON%Qh8DWALo<;%_SVFW=JaRgSODgC2q+tP>r|EHC&N-AMlWTDu$E>;2A+J}q#@YOg z?#>J}J(EBDnDlG|GIgY|MXjg(V)|R>Z4mEXy-^9{E#~#-Nq1L#942Irv`+W+aB$oV z6}peT`!F*g(@tvrb8sjLQTVy+$dA!<4~ioUWhUn>dJOIIq;mB6W+IG&II*6)D+v7- zO|#Uw9AWDF-7;E3r48F}QHl+ON3t_+qy z2p9M{GmXfb;Ap3v7u}K$=7h0GN^bu+iOBIxe08k>cPluwFQnXoe%olsV+U`hYyTnr z-m}|L*fhrD*1pNpwKx!JK8~D7)8DZal^$tILB>5?Z!Jf~!KOU7Pxhl?fh!*0oMs(h zaC`mY3fEk~Y4L?zwLT`d^u}Ux&TEQmW`fUY)UNT_^NVB50=rZcv5Y(lAB$obQKzb! z6Jj8wHdI((tQSpmudUq;%L@3+>2JrxLcUB0(xCmlugfmI2^OdFsUfFZi<&CoiB1ns zUAm6`lDs@K$=~3Vh8yZ~jJ&l2v;8h4G;0)$lY$o8ZcP>x_q2R0!-W=PV9hI5KF95G zRI4w5bX<*l5Yj7v9hSJ1vPHVSi9#U8UUo(#ELHS$bab)fN=V47&q0<*orJ!t#OO&} z(N)lv9(RcA=IM5UYm88SM32N~!bIovD;}LDac>be8E5!lzM3wsj)!3KlqHgiK;kPj z`pj09LuA_vv((AbT1m7j&RiQ_m2#Wu%X_56Bz_b8}h%kry>(|LD9 zu`sH@>2R;|wXQF4>L&2Nn{OIl&!xk9ar>EpG(SZ z6KoLc*5TB_ThUeK zd~0DStJJ1U9v2(Dw#WD5KH-xTFfNE3=T4*uG4B;$s-jjaEstBXi{+J{Ngcu)`!23? zKS_s!Kqv;K!M%y$K?cap)Qhk>78bX4nnn zFPsp9w$EZxRJY^KIV!%|>K!j_MLo0S`0`U%oD{3hA4>FXslK)+m2k!nWD+9gMB`d` z>AxrNr@OWn&_kxt_zDg~LA1Q@00xf7t%5k616mgle|>2c{voaWlFB?SqDwl=Xxkec zb$YSe@TOH(*)@#VM2k!LN~uLSufAULOpm!X+Xh~s&Bb#FBI782)k;lJLml~;1Y*Zf zy+)%uO`|J$dQ;b~`t*z%V(Ybz-ck|BH<7&NV&C;4=yL)$DiwDlINK0n73L9Zc#-}JU=GY}br1zAeRLA%;X63ZH< z);4y#&(Mo#?7GZa%1E8``Z?!R+(l2vR#gS3!#Othq<5;_gm&E4`^>_%Z-rl>qCD%| z?}mz9u~qb!jZVaDpn2@aES7^kCRjOxu)up3g^L^Xjyoc=%5B=pYj=`j zp-Hc{UJ8qAxcGnZ3?T9{GUy^F^#J+r+en0$`{t^%qx4#haJk)wo-ecSDwHZY(6NU8 zXhRl*cBp@5{vky`+clb@=P_`)65JbgB zgl9cXDoK3AyTFO~gj{(VPP>e0_7Eg>hJ}ts7p~~;{~Tx7Z&%BpX;HyZ*1d$G)!L~Il-JLXl)O#~@#h>SejyA2 zdBeupd=|wufi1BSkV#unr-HmdjHz!}K8Y_@ZqmeSfzTQri{%`SO@@imFEFmgA5KC_ zE?CSUc4rX9!dP9e-fxT=wm6U?7Q?ubAhsvi)g0Zj>5#50Th+W?c5bt6ld5Ic8RX}0@$u2GA6D!# zxJjxg+;5&p$j@2*M~0E8Bf^E*DVWSd0q@L&T1Fg(Ob`>FOB&muMK3WqUT!_QBb|8y zM=@ejHbwGt-F}Y0`6om~-bXqeR#CbV)lAA;-3!2US^_qG-kk8BkN&&6Y*1N z^F~aUNlqeTogaeY$FLw0vBY9Cf!jt&SDcr*M!&7Mma2I=+RC$hP8Y+~A?G6hM&D_% zyVrGh6NNl1z465&V)T0C46#|NZ_us84MO68Y)S-=rZO*_in3}aWS3}$!UsWSNcaq6#XpWEZ)2&_`uohsSx%eoR?1i`Lv3I%)#K>Mo zA0I*yv)a(TuTigiC2GVB;n^U_FI7WGO4%1hyd-kQg7O&+^xLthsKpnzk}TtY0H?s* z-}Z^SVw;HbsL$artVw6h30U0SmZAuRt7de5VixjiL+YsBZy(NLF-4Yr zY;NIV}W!EeUGP9ZoEp&)CE7-ZGr8g$_lrF~1gqxMbgrSP5~G zzn^wdd)$N_cf+Zr-0!^o?F`SHnru_Y1-s0?j;Yvri=h?@P-0bDU)5uWJry4*2fl|Rp}u|9 zhSx7W81r|=SUsq#j0JYIQ~|LjG*~98pPBqJd!YnV9-F>Y7_xR_M8LBmRNEKkmM+pN z5{^cUJt~zph60D>dUlrvX8hE1zS>SbK0WV^)79c8YSegog*%bO)b3%0SK z)Sezfc2BHtgFF_vYO$MZYVY8L!iTZ$%!8AvEoMY{EugTiI~95SR15<4D6ubOyhzb5 zXepAcr6ISEddozrTH(#LpdpOdSw}B9DQ$y938jg(33lnC5VBe^?ZuRpye!X*5Ftw! zP#Y2twMbwbAA~7>8$Q|Pk;AS9GwE~D9XzRxH8v><5}W;4ZgJRVFRAtMn|I(Y>Wvy@w{i*$ zQllF-#VlCtLMl+t%aWrc);Z8CVk2gkHoPM(ikzcXgF0ec%?7d4-dvx4gJkrO3dx7L*hJDlHbJtVhRagqh_BZM68?E+oL&pT1dQ8@~ zr=nXA9iJ}+i4R1=N_yBs{p%N z8nV_bwJ4FDnJ(5h08#BMSG%@e72_>iAj=J{*jbOYemyJtB}9+NTa3}SC5vO?mGE$H zeGQDx)mBb4M;5+v2xB;cZhegLo81wzhAJ4 ze{_mazf@Z9R&{F{>m5Q+DFm)|S@h0bUAOrshcw#`^`b8?@y}MzYI}AF616QUJTKHW z@mbyu(X%OadDi$lD)*7NqI>OT_tIIEq}az3*1rj+?$0Ft-H?ccin)2trRDE!Oirck zdM>H!SSH`Y@&%ku$s(q!S)RZZM*2xiT<|lS3G|%jQ?^;#)6m9~76YakiVvz9~JCAO#8lS%zy| zh;iqXGt9+d^>H+-gr2scb7M1K~! zbtXMUNa;?+@`IAXVvsdg!^XGr_bw6iera?%w@7b?`G8G&Q=dxAP)N|Pa;IZIq&?=R5So-gu}60s`}!Ju%4yJ8d&bY(A7N- z7ilW-J(l|_x{tl=ju=B3CVVq@LFup##}Is|&HQz1vDVphH&=0SWtUXe74g8O5jD~M z61cEo7oV0{%!PkhQ$0U3_nEO|#MLh#g!=d&A^5$4`*qa;ibg!uOTNufL8(@5#3 zr?qJXIpF)MkfL#}4f{p>?i=zLn$yk@+qVp6+y~mH)sNFHbFo!z+TiO;CCM7E07 zDe>;YeBl}0+FD7m%qXYxdxDfREXHww3RP57zYGy!Z){tR-tYyMY%5m z;-;r#{aMa`M{$pZ--9Ll#OISlF)%bts&V=Sk7l-y*UIrF;}tnt-FMeO453bKYBfgo zPP1aGYcfAwA7ej!Y%jfkRZ7y!WpenGBEX`PBO^ z58`v?=rr-iXQ9b;`~SI&ffVrMHvh8TlSlmxH+O}G5@P;1iSsuh`gN$35rhy}>cYU0 zJq+{95BHX#AR~o5)Uxnb`0cL07BdAx!wBU%=RFL&=kCnkuY={C(THL>Xn&ieq>w4< z?ATw)$luO6S49RA|G#znVfmk$2)pk_7&j-X`6sn}9+Rb3s2S>4Xg)vv z?q@pnC{cyl(IW@~dH8Mv!oCi&0iWwg!+7=M&;XiN`o^^JN`dk9pTyMA8Lc@7RS+KW za45&?qV2OpDVbBAIp|cC!KuV?Yn(cO>6|IZA-N^HfA4AUasG!hRsmB@|9SlW zJpYIIe^~F2;O(&?|EhMG|I^iuR=-0==r7wxF37V8P}w3^bn>}dUGO_tWdekEy!Q2l zy?Xg5(6!gU{z|9%P*D&-JcO!ZOYkzl_NhP&s)npwKjbKzbj~@g&g1}R`URd_s{kZKK)OQY^|35 zO0&SbR4xFpQku})s^pY(8u1-76D3ushS1A?L%Qxi4_DJz`diKXS zpUhtTea~fp&?S|3;R(l9I9IR6Q^G7J@!Ye@< zTR8KE=n2A!HpR;bey8aG7`2#fJx)Onf)`PFoit(Lf|mj*d9J3xO0%%CRc4VXOj)KL zsR7)eD)pRU_qn7Oqo4=IJgX8sFna7|^NT7dXk=KBq`LIer+G=it-A#>ufrhm({37V zb8YD==TFs@W6Q}T5}5C|QD(>@6JJJbmHr;NzTQ3T7`~0)C5}Yf=jS`xtrYv{ z=y)DgVy1oZ9G}qmJg@sC#HP!DQa_<2rRCg^Y_ZN@<;$|Qi@nOK-y2AGIw=aK*u1pE=ar@Qv@80|M z-ER*7kB4;rdHnv`;t%ovu-+fR+b>Z2qjvr%-9IL0ADjBe;Qnzp{y1Cvx>Ns;uBL%S ziN849g5a4uZZFj$O&Y_JCL6<%8TtKO3&nGpGnFJK73R`&TDZ;*0szv?bYifciKkOF zs7gnJe7@f-*UAB2vOy}DDHr4k=%7aSluilFfMQj18=;k&W{!~ebOaEJWKgzdvH4tm z#(`@Bw3gjwk8>T1aT=m1S;&uu$O!wJ>>sd3So%aBfeZrp;>-dR4jbO6Ahe>=R%}#* z?pEi!CaR`ass{(B#(#(MZZC@+|2_YFcJPs}m@m#dcIOT^?8T25o}eBVN^~H75|& z(rsia3WPq+&kYBzuZ*z1u{(IzEj*!-4Y!lk-IN92nOh|^if4fK=owIeffjKV9|z6p z0qyNc_kNZtvMK-`9je@2JS(bBTrpXlX;zKh^ZcMu)yXyrVhe1Q;DyZ}e%T?^6 zVjPqREzirveV_xP1C)xjx;v7>^`dM+Z+k9?Rg*xUC(lkTd|O`Xc_qD7`kO5AjZxnY z!l7K_Ga~2tce6kjam8R7d1l}WGxxG(O zA>Q4l8ZSNWE1tg<6+Jy(4=kg?is;*NLa1nJoO8bkm5!|*g3dZ{D1YRvuIiHhwv z<1KtHY5R8L^R0zqS4fuw)I$ikE$_5RjUSMJX5+F|` z@jQJ%NRa7{{x`g3I*19d_6|;DOuJ+LGAJ@vYLQ88wFM^=I``9S9r^y|Zi6+lgt80n zy!=$=4-HI*A>UXjvrfL9z?r!K0G|xjoHp1BasUDMMlbQY-6=*F&$VyOWg*G|9*s9M z#3lc~xiaH)L?on=fCbmXZ}p&duobiFS$7E?)w{X2r9@BWKi4KJd(NtHwtGz9VB*5z z-~a&@&bTuzkDQh=2lBvknZM?g>#Q0gs^SQxk9RA4cO{JhqGcBA)DHT5?@!r0hxj;C z6e0$6;dhnegJibRJjthV>2GXDEbn}t4gqnp+Lcf1C(*a9cGltgi$Q)d+Z86MY4?7R zb89g(ItS_}lmaT7j-;;zh&MPq1f8oP_=0!mP05?mWq`0d zAI5GsY2;874*Dsn=@f52(4~`*r|5HAcZ(Et^hgYG9rn8 z3HeGe0un=jF_t9|_Ur=tk_hB<$cP1~AzInScicyC!{O3-_{dfdHfSX;{gN;k3E(*W z-fv3wsb%cV9$5nLcKG;rLiurOkAtDLhpHtGI)|TD zUBi7!01=rmfNOGqsB zbMdJWc_*Ao_zVZt#wNJG(~9S-%M; z2&zGi7wCaC^$*^AglF%w3|xUA{%D?;JwT5n>j6D8n|7P%0HaX32n79Gbf539HUB2k z3T0q@otkfm4+1#+0E*%9-%bMY|(flDJL4^ot!V0{Xnr#YDq*dQH& zz<(Iv4+H$-`QK#uj|TXo0sjAP0(_Fx^uLDDE)quO6%5+nYizx~bcEX$WF*vx36n4; zOV5IkP>tEkB^s#V@6JA_FKb-HLoHxexTJl)5O;yuTOcGrZ+>}{e=5S=hcJ^0RN0MZ z8&kX&2>o<+v2x>hfP5DjORS57_}CpOMATkd5Fuq_SJ^Rgw@-Rv46F;c0y8~+r^4j% z^K~}91>(z^ncM-cpfR!pb?>+UX+P&D5W^=6^*qXneWp2^a)xl*wVSIve;4B8>3v2( zK4k#La#zV)=86vp8M8FR2xV%%X5whuVvA*bm__CV6V^{pe2xnberhrXp$K~O{q>F6j^C#a;g}{hcrO6N-VB}<0 zz{;AV8{QmXBNZM28#ksN57~RuJz6f_4H)?;#mR$gG#h}S@GWtvKn^hS-+>b(vyyCo zkdHu3aH3pKa~!ZdN^@X?2y#K-1AH${Ie(CkR0YV(rbl-4 z_dxv@F`~fw%!eNY{rcuNLK}<$@=|9JlkFkdZ%b(g>l1`>$sa%-K083(Kga*)`1j`@ zpa1vLe=oZAeh_I5?~|IdW*>m^4a8Rl6>^YjWIl>Splk|;!pgV0vUvO*oZpVqYFTCE zwU2|rjY~|uRDVm1&zmFM`sETJRVhucy?a$}keE;?I6^2;;B=BY`8skdL2~!CbH4>L zIVY5fh}Z~3T6EO8-vTv888EK$#h``p7L2lp*WHIODYOjjl%=zJ0&6G);{Kmwm@ zWd0N_14H7*uq!M0+LtrDnIw2HhEHM1h=uQ{&y5qJHz5WzpO!E3u@MFi{@*4(N{RTC zY-;I?D%&sJYo{pavz(~4BWNbUON$syy;zMwJ9)LfdcVJwO~ielHFU8~|C8v7*!Ff# zLcU|_X!`cJ+)PbCg_KJYnArOeUxc5u1$_j_m_lsj5lvoNNJQ~N zRpy_f{k>F%6oM`uc;9TsO%V1#nbDei%CrzVpET|x3fE<5WkOV36nQbs5NH7^ zlEyv5Mt%!Pz~4g*PR$10`AxO_Qc1VCff6<}=-VI32=TcCM#d;yAL2hq3BCYI@YXSI z*g?xX0+x5@qd41di~41-=sp7*L@;~@ERT&aH|2;^?Ljtxm0*$iKLHHgzEV(m5g^0Z z-lHRaZTNQ`TtzUxhr)gvW3=arh`+${E_N~y(&YQyB2WuZx2kdvw)FQC3Zc{zd+96o z&+g-o6VJhS*WZPiN`q`Vd9I8>z5YX$+j9TzjxG&6PpgO_@bBlDhg8xF=cVK}-2I(A zQ=n(XJk#%>xzK#nU{?aFuaur1u(w z?RCv)L>mc|Xr|yUl1dV?wUyHUWn6PLfwe7jLQedBqskX#yY`h)GwTece4>V?fWN-K zu5{NgzLK#su5_cQ{@*N&!5PwRIdO`->GON&zV|R?!hst=`$BHt0|1^D1=wtkNL&hK@}rVP-X$C)7eM5T5II z2yo=cmIFE)e^$iEAUx$8W2;d;rEI#OlvjiYy;(7aLxgELN|I}xj#X6hAFhBIZ%RX; z{BwMUFcGqNd%1CCjWF7@oiIilU*!db4GgpcV1uL?6U82lwO9Ka^#3fNgHC}fWnegW z>p48O*C^O=6>o%58eNS-*ZAPsaL<|R{BItSZR@qMK|DKYVrG5lQx`)Gd7*w zsYw*4bf~U;exUO*|F2ZfkyFdhZ zS`r0&t28)Lh6<#kavAY-+1|xgw3b6$L5bF|_tWEhf$|OaPU6*KD?p>SU5~Ddi9V%u zZriP;QQwm_li~X1HyO?zupwp)@o8#EgoK-YDQG}eRaZZajB~fN1^|*s&!}A#KpZ^I z$8WE+)`Eh#t~lFxZE!Hab+GgoK$$>Q+BW@j4Sin`qojKk6wC#--a_yj)l{Wx=UC45 z9#lq}WLDuP^0w$mHW@B+U-L%~mGtB&fAQwgEjD(Wx#u0V3NR|hC7|4;lWWzJ$tI4A zOeXI4!Yvr>Zg0MM3OTjEv83c+_whdh`5?U0F@T4bF5S9@fe*<>@IAiX6v595ON4_V zz-f{~wu6P%(u#OBjwFc8UEb2g>+d_357JJyWL&0Hqk^8T-b5s004)%p4~pn^VAi`l zes^aa%%<}v%)2?~E$e$o4yhDtc{p*(%DW|2yl1LwD5XA2o4F8Obb1IO?qb^%Ckc}a zxUBi1GC?lDKRtyW>|yzWimon$fLTkEyit_f<#voM!D3GGEH2eD;mY3n%5H-e6Z`K+ z6@f{Su^@JsYFZLc68w5PH8$=>zPTzt#xhF4E>#XE? zAHGWFX1ss+EE7b=HAUeqs0??Xla@?ZhT@pO?B%{_{0xQn8&>(6E=~Q|=1F|?$Tr*7 z>P$%NJ=v$LXFJ*%7keg<%-2724kg+QU;?u;g1heE>y{cUXjb< z7VRINI$16YgSSmQMVah>N)VsYql9~S4o6L`DbIjX{bh4WAYj~ZFzYc@Qu@hgr8ORm zCCg||>$?*`8q6SmO?(jaRa<569ON&VTu`AR^1i4mP~E!tz_~SXh-B=Z!fT=>6n8lCLT-(Y+=d39KnCC{B!-<1-JPfFp_r>?5lnA5e zYMblWOZKrUYUL)ze(b`4@GzKv2HqC2`{oK|DVS}CexZS28 zL2jLcmfDqWj0N<0MJY|^b|?irP};u_3W)MWf}UG_z5A7dH&8Y4RyD{WGJ$c1@cg7N z)14`5+aQ(!ubshzp*taDC)(dht09$|8(bFpg&yX9j5zz& z9)DPsWdE2Z{xKOB7ewVo2;(~P@P>s>WG*lY^8FVNaF12c#CmqTSgKC1gD#K(emlgU z`=FoEz`dT7dfI94niz6=z`l1%B$e^-;K8n5eN{8IkFGph$4^#dxs;p zJfPK$k^{K!DiFDY5Creh`Gm4W6WVhyx-q8cIqChc`>W0YAS3oqjS!!SU+{F&Qy99WoW; zgYX{qAO*?s=kWg={$2<9AMRB7Bl!nPzE3-uEihppQ&SuAf#a6H{xrw5Ikj0D;Oneb z&bgz&pk|BddtqH`K)fvr1;h52-!S$?f@M^dXE;rA_{1Zn!}}9t9)zBZ0Ihb-QU4}7b^ptT*HR#Y2Vi6Kk@B5WPQxHLa0Okf z<7L?4#+IZlWr?Mm`6GE&JzC)1Ug00r>dz8GLM4i~ZvDfXd$qOCEgC|Q-~nd&2GT%2 zMnB9or=r#OSi0{EA-K!HuAa~iB@B)xfWeEqa(2K=U4#f@DO~}+K3t>#RPbs*1&No`IPg7+m@whH+%fO-{Q!`8k9E1yGWTmcr{u+p1o zX%~kv(?kyxK&?Rx#Bb8@&J~Pc8-J&!^wMi*>$Fo3@PJr@q_(DS`#jAxSOl^mBI?smZ4M7e@y`)C7Bjoe*1QcW|m&wWK?4iM8@H) z#r}Sn49SrXDNg0MA&lusyiT6w>2N)Fix}W9&T1%?lfR&ytZd5W4h%qlgDda zamj7vlM5i|s_6xZd&@MW_ct&qB1&7J|R3=PB- zDQLngt$zMY7>P$G>F)S}_r!gAF-P+cLS!E#geck4Sw?FD|ZjD4UfjLh)dr zJ>KO@95lP0%?@Bh&qx&HHa(-Dts(gS9XFmIdl~E8pJLGFII=n_on#j0#-;~e%l33K z3YHA$iKmXo?<;#+cYHu`LMrX^mqX`n1~W>AJ%qJNVjs;GU1tXk6t5zLp?c2Xg(dm^ zf(+l=^Jxq|-l1k!O7C0C3Gx<#%o_Aw+RL#2N~g~xMJFl5eSN;D>Fec^MKpf972l}` zaLFwJxf!=lK{(PUD7?S&*zfNf8E_J3m90N?Hq8JsclRY8F$4t0Bu;;O0&r82YxhwG zI`)-XxmHA2)4iwp|L4KgD>2eu<7uP6Ju~A>FPeaJG*nvT( z+03q0Am%;zND4(RG51NBMZWX$xpkfJeZYHTzqq>#gfAY5L|!$Cl++#GtiDWOvOH~SHiZ{m8af&4y)6yEg-b7ur2l($G6Gc7uH#=#+tr5l zh8UF%M~*Q{h%ka-zpkiQm$CPLjQ#+&pEOFg3+5NnH0eK_0=*7&3;lTkMGmMMfHh_S z?I*k-SpO=}7fdh2MVPjB+sSKtoTHZcJ6CFE=rnd+}{E4G39--@xxDk zAm2v754=;@Qq zc9u}%FSF--@w$C`bA>h3CbOOv((E*`7X7!*<>2@P_^Apg0L{f11>&Tm3Q>ZbmQykd zKy{>-Z_Os@-S;F00Wg>TYq|f_2)=t0M_Xs#c?2?K;8kiKth?Z?NwpxeagLz3(hG*& z-wgj>?Y((8)NA-ZJ_apHDO*uU$xg}=g=w)wmSo>5QpgfQD$ATs8(GR$BvbY^vhO;D zB1>Z@OGL<8$Ufiu8RxW|V?Mt>e#`a!UEi*&KU^j=?|HwU=YH<{b-(V{E58Jkq%)ju zV5zGvDAMsEszfW(5&kU!O8sdJTD5q(6K% zRDB+u0_N|_C4Ma0Rkty%T?)XrT4@&knHC?Fa3%prIw+R967Pfv5nl5=(p zSE-3>N7m6+m6|=Ynmqog#VaPggiW+HaB2>JSb%yku*0I~)t%OtBtWpvZ=CR#?tk!f z^l=l@<-wTH9wd-F-Dfj*pNp2=nq%qPXvYCX!0r$eg~d{TVK-7G>p>2K(Jl?qaPfKP za#|NAX>l1_t+%hZ-ElAJqH~7pn^TaX*IiC|x$x}sKEm0z>E%y7$l1+ocw0)%K%cK9 zR(E%sOT`-H?m7zo<9fF5|CWo{@f13!e?=C6?_Em#COQc$p2 z!=QfxuPbPHQK1Uj3?^b%CI_~NTWU^zZnnRXC<|Up6EOBzuE=hidAN^x<>!O1-1K0- zgV{pARLL6|zAi^iZaAjWSUuBT+f z!30t0fE-+Cl!A{%?Vg}Y5ddzom!KPBRAB!%N#LJlr|pfqG9!x7h+9Ng?;XI}@Pr2N9y zbt9Q}9WpX5!&@NB_j~3(av>c~4i*93CbxAgLJSyv+MS>ZP8ZPe2L`Xm%CN`ol9?1& z-AAl5srtU@gNbj&Z*Co>UL&4ZT3NiMIo{X70ip~Oe?aW`w0#Kogzi&AI*c+6+Yv%E z24_v4?Xi{cRjtRyU`p@sKHg$gk{6emiqE2FBWb+&5*u#4z>-T1Pe_FkYBFzFGdY<) z@oHYUQCdSF;kTDha=I;bHaoWL(7XrQHR*~9)2eX*;+kRy^Fll`#Gi?Y#^%5Ps$JnX zQ>zO~a1JL-HvzJL7Yd2=>D&hqlV9Af;EVF&?o(rYs5!t4Hyt2Gg|1?ER{6dMD@G3* zf#coadE))y!ykz&iomks!xA-tq0-R9;A0nfvsmf5iatpi;xd;;kk* z@=;GjKFPqUu++x;$1MnU7Lr!`UfIaXivtq_*k1&@3^*D%AlbY-`9{%jy{nrQbV+7MdK9r1;H4D43fCQkw%PQiMbP&ky71UTW{2FNfi3ALMI^9sD-rXp_uxA=z_x%gwFEgr; z*-4CP_KVL_5Km#Wt(t9i{N+Ze%jPDOrW3@o$P!NyXU4lukn{XhJE7h_^SW zCApGT(PN?o$1-Z=TIrD{rheM#Lq*}z-!I}{KlBOhar^Qy7d8ii8OZw|hDXi1OYsvh z4p;`2b6`@c#0+=oky{&lua4&Nwr|+q?Uf66A_{(!H52(&ApOGbT-E=3XtI zr}zB8T-bBIu~AU}`HUG-nb!97h=i_;yo~GUBa;E>5}jLsspGq_mnt;C4Hxq87hOf7 zi7ymZV@69zODLCp5k3tmT2%=YmQm+xAHREcs$>B2=u2arMW<)O-cnJodIi>ND@CUqu7|3Dn--&`vT^YO5gy_e6fmHI;ZgMG*p5(w>V(~k2A441FIIEj zb*hzh&GRKeo0dCebsF~tY&7kkw3rap#Fc}>8gV=>Yz{nT5z0RdLR?wGmXHo_gVBA5 zMjbr`!}Ynt)R-*Uj3Ynp)GzNO$2#+FRgqamO_J2ZvJbWXKDBHUqY^$O6q8#2Dsi?C zE?%Qduyu3N6^6xY8@OZAZtZN5YdQ zQ0HWpX3^J?B;H;Pg3Aw0uD-gx$pLiV8w)fHS-E1w)ie#)bwjn9h{~KRAleQhJx~{2A8Xi_L1vpE1`ieN-Z1FgVPyqldJiLF)j$uUC+$ zGup~+ye)1~S)uhxjla8d<*^EVF;HMG--;JZRch>;iv zm1);PSmCM%O!kz(RB<{?z$(ohG=l+gALRXhTKXZSE7e&S&F`+ow&yf%a^Ri$!yHmm zPBAkm1Dx6^E66GDaSi1-5Bu1^p1r;rBLd<82nji6I|{k}HgGa?08@CQ>lG8TbZfk}0(?>(0sPx!{2I5{Jjkhv6M0nBCn%E@;7=UCeW7 z;em{^wK0t2PXLLd?9a7~JiZ31H79CtCtX>3UVEHY}ognRn{#@OcYqPVcV8x zg~AyGsy@SMTS;y~194a@-zw|~VVwH_TJs=7Z%xWi>8q{FQT;HVypsoDY z>6Oo4$YT4}h+AT~I+8p|ORd7SsMQ;3DZ+6s0&-V^*U)~ zD3J;)C{zbI1kyY&K>lPb)T`Ft1}Tl4)8?^|X(7ufg;%(!r{!Y@^5c^%=-{SmBRs*0YK8q1?xPZt@QB zP0HUs+-E-2kkD#KHAv;DVs-B}a+GWJn2QvzI9c6HZ6)?WTW{s+2cEUAF#)AvO-N-w zo23*K0Rq(ezQPt%YNLyJaeejq7k|Lnz3)Io@%nhT8Bd^cp%%SPeAvMi*9k*B10LvC z+)Kb)I$va9?kgb>t*6%ItGE`Ut)o*0slaxWf!*D@Z#R*1J;78s2&2YndtOTrYGrUH5d_xb5or{2nuGxd|nC$Ja^69Xtd zdariPn^vC@GS1x-USA>^RjLl?OfP8vd*Js`>J($}%*>YK3SObky_tk(lpu*U;>Eak zr!aSc=zuczQ?LM}}5&pG1@G%1?+Zt&-LoPWrq1YHpv@kd?S_@rqN)ZqR(OO((pT zO8aaY*MG%>0ucn(y{P6T3_&ei@?Lh?$@%FK)>=o3;V0P)|CAb2miZO7sqfIx5F$f} zH8$=@GsY*@s@k+B^g34YBMhQ&&E5Hr_4*G&%Ps#k>ubr4heS^nAhmlP?l4g0PrYG3 z;f|q}L5^($rgU6hAWmJyqmR{&p=um&$1c|}By&vB@b+b(Wdi1d4M`#Po+)@l!grP{Wp z=N76|UaQz*U@=GMEQrE7otdrm*5&%^VN?_{2mqKANSAO|>X=URbRwU%*rgRmJTDggtZ z7U4x4CLelPW}Ss@twy9MMPKjfk#^B6E5zah+da;A8Ru&s048hfR4w7~PJvZ8tB?Y^ z6(EZDmtWdX)KyXW{H7@y$|vzo_(EYXpL)bTw1UjZP`g7Vu>m3}a12Gh9y>)P^!bA) zy_rB@@rGQ$5i;N&d|A19g~2bZySqh<`{qW@&eQZos8M7a6K&U#Uq-$0pk^hHz*D>H z>QtE*Km4^t%FcLIeTgTDc@^8i%NKR0VVY0>gA5(THLl8PStp7CL|`kFmzA+!X8}eE z@n|lqX}H$;4q%M3n*BdF9eyOniCu2X@H0O4-$6a6=tH zh-F5+p!dM+Y?Z+X5_>Z@;8Z{=sLY#5?>m+K*T?M0x5u?MB>i;Ns));`_UF90D9cg`WmM-n<-1I^D)MYDcH%vqjFaUDfSuAo%ghSrcdG{){`ZYTv3WTY*m z`PFkXaq^4&s9pX5as=DR1D6IEfF;Tz%(ig-&$g=+S*B%lAhsen0d)&&@)CSdV}B$Z zyyOg6N2fmJhGzW(MYo>E+}jdfB1&1cJ9$|zWm?q_$u4m2zj*rKRKv*$up@qwoxGhw z#0*fvbHV`91SEiscBtzNKgfz*8mJq7LV^T^wFH!zoZd~YB4G)!dQf|tK-+ky1N>ki z&kJG6Dk(M9PffCQ*ZeMSS~`^Cgb7fJuz2HjP>aeUj~kx0TqsQ{AWdNeiJw}thWHZ` zpksdd&fX-MsCQUQ!p=d)TgXAOUDvtdJtZ-iWEK1~Fpo08aH%l|ohwUt$mXe&{baPr z`!RL^APrymh-af&6w*`-{+!@A#Ip15UBKrHU-;l@t&9%7ZPI%^AV)eMDHE;1j{J;J zp62u};oN+jYQ>7_SKXgjTMiSZ6kZ|gd=?>_e4`R5Fa_iZ>5{c${rs zYzw~(Mzg*pF)^_P;&8^uOs{|K^OKW1-M4BoGCbI8p~5nOSO`09kwrBCf$OD@AuB2B=DRe(kr zCkhfT9FqhxPAGEAzSZ#Ah8oCX03X??&htOB?J8R2 zy#ws!3Pkco3`{d4= zW{_Bz>SPX(;j`9JZ{#?5`Oeq9l)mj_+WzWrR>aa@Cz8u`m8uwuXjEQExCOF{V*)z+ zHP(^*U^QPImp}8@pFA=xv1;Kh7NI83%RK9`rNB95-6_0Fh%#)t0P~jKP}8PlKeS4W zgeFK6M_dG2pXtaasp^9Fb-=h0d%?Y%1SiHTsA@t6lniI2WQR43hAsj45X!fclJLt% z6E3x{f!Sd}rWa4y7fn6A`fW5F+Wcj8PmY*}+h!+7cSOtkY{0w+zK=nWIWhW}>LmiS zFvJki*gA5uPF@&G;nHyy>$KgAKVE3$RV$gXHRp@78m$s$-i|@N+Nb>=JYmGs99qIj z?%gGTsUNKjT>1VIX4hQG&05mlUmf>pcG&72l~;bpv5h3;Q4SAW)(k@X)Od-ApF4#E zqmBj(F{TNoQiczC8ouvNui&W4ZdgK<)GB&5?E{}YfgKV?FtVdCP?>a?fWS$}S?=3S zL1*O^U>wgcfNyEcN2|ViiRjx9R*?*9Vx z%fInvusm@zf;o|xa58QN%6c{wZqWj2T~G@(aWPoGb?+`>UQq1uhnrvC+USoc;>1VC zfpNsCJJ3c)VK-1=t5_^EF{M2@9cKg8%m(V7piZMrgjWTp=fRr&y?4>n^7E(A&bkeU zsr`gCg$4i=I@~mDm*@x<9r7uw5f!Oor~kptR)=n7#)wi=pk7;cVy%i=ApD9Q(Rb}| z4wq7-o-T&NALb+uO{d8VuLAIYie6$1x)>-B&Hwx20nypx% zAsN~L!z;Ms&F%q{>$DhR`+~&A`Fd&+P`;F$ou@XNs5mWd+_H!3>OdsN0+h~I@9d=% zjxcy44IsOPOLL#eQ%UyU;#QM@p``vIB-6f24)r?PRxbez;tVJN^Z!s3S#CikBS`lh zfu@qQY9<*ef$^|&C+DK27F=P-+p;}-WOLuyl@9?(#eI2Efoc~xH$7rBe-q6Va6nW} zz1k+XGP1?MmF7*0P-)t&senlizwm_9U(X?^O2$=yQGz#WkbG zSNC-7*hte#H9ER@+|IR3i?q0%Srrz)CQP`pwbv%+y3A{?)@mD+iKiNGk=} z4dYJ5yRX$fMlPIU#PGe%B<96p46XJuuN(-ya7*L~9j5JJ5*L~r%Zz|EiHIq-c>pyx zD~4F_V{M0O(;d~Uf0^ZN_~in;N{O^CFmHiumqScTw_&TLcvuZTTRrE0r}NYT7lh|J?Gv&cp&FK=lFtkBc>2eZInz4XLJIRCX7itzJvFd z&pGXY*G(}jL(IC%+2&Y!?mCD=ZJ(o8XD%f-LNgaZ3^HhZY6eg#)C*6k&my&(^{{v0VRmimGBa~FdzcBv_Q;7p|9g));vmjMv5sE1vB#a*{`1}{-9mABpP+QD8JUF zK8yHPI3^l$t9W)H;XcDY;~X~Lu1*kxIc{cP)cCAs(o zG46uBO<1V?`@M;*HTJ{pLDf@&l8{j?74?M}J;(2d(y_x;Ox)X@QaM6v{hfyDzx(>{ZfSTgeX^73%aKxIb z8%DtuiHG3kq^#9cNQ#r+iWOgi+$k%3AvR+W4fj19 zqv=o@l|ukC9cQbmI^LW(&v~UvaEz=|Dy~6QA{zE7_p|d?MrF_D3-?RDNo*$9lsGZ0 zPsy53tw+SXhZtV_<2pQf`{Q&Bp9|rk%6UbGC(k_u4CobL)NI=ixZH(K6dxKbLSD%X zsGr`%qx&Wgu66;s=2e*08#x})Qqfs)IK34^379@$x+r1GQhV2g7SuXh^q}@4u{-V=Lv%Sa-#Ht$ z9EDVM*zLiL$|U(F=mrEQMT2vY1#ewqBWdyLfkmT+uKDNehv^}Mz}e9Z73+j~l*~lj z4}bEG0jsb!n)QCn7Y2m5_5vN;XR@&zINF>@EPLeb-h|cAaKHHI+Lt#Ir~~r3N)C$r zRy``U#sbp3Xh5v>p`Gtha^ISH0PN>O&7i9)G`9g#F;iVvm}qU~-Bwof;iKrqcSi`^ zupz^Oy}DSdt#h42aJ#$5isaXzEn3Hb9sG8ui=TeSAR@ZzXO{v=cPl8{etUk4gs6;2 zt4g&{yco7j1wtKWfKIs$b#GX>5MCDQ{z37e(LF&`1WoUR7SHRwgNnXv%&7X6Y}9!2 z2O_V86hT1!iPz_<^{UR(~FRzG(dv5E80a2?#f7@0DvUS*AuG7V*j=UIw)F?H5owhNi8`@=+Z-IKf6=JGsY!~o}*{n9*gac2qc(&SJ5z6+A z5rm@5kCn(LUHxkCeJplp~fUVG~Me0kH61Ui9$G!H^zqn&&x1Ve{@pbO)EYZ zgS!$aZz~wqmUSUiZ36ah#Zm7b$aA#D3BaEfyt20MpXdGS@zCvr&&ru`$vk!b^X`nx zU%Pmf#QzeVul&0hU5@w7v4ASq%hSs0KfP5s>KT z{3C#IpfXlC%f>tsM7wOxG*hUzp0lFz2#tEYnR5Vb_Ht3ca1mlZG<5-r6~v`~g)jwR zc>Al07G-wM>bRfY)U6{t>hbH7`e$8hhLBZ}t|3YU@Iz3aEy5Ofk0-7Ts-E(G6UbZf zwN{=J!2ijZ(%1KPngecZN%WY*5{}JwJX$%=tyCZ&NC8GT$^nF9K|JQ?=i;7WP&fhp z`dJ~vO!Bco<}d*QVDrTea)~#$aMUXTU;yoM~9@KR%>Gf~D;&sa_1Sbch`#G^H&LNW}?=O6Wy zg0VJ^aFps4CTLB(%!;cjETa?}*vxP&^WQlQQ%_>HZJEPtgDG=JvLgC3l9lIad=^v* zvqgLF&#j1m<99Rc2MQQ!KxBH2!c^Bo6SLO=I}m`bB@#DEqI-aS%}KlRSK`vdBYyP$ z_jGEGUAnaX=S}d>ih=VlkPb;lIDmD%h3x=D zn856q`*tE2&0|psqAR`~-w`~_kO;2Y$YFd_xYjjLjVfqt;++ho?-c~h;}bRl6e=YA zHK2Weym~E@Qk})b17D3PTxG>e;lgos!EKx6;7kGG{uNiS0x0${mBmuvFfqJF=F7zu zN}pF;u5f!!!22ttCs2ASUxc|Z0l&Zg@29`w`qBKo_*Jhw-zGK4Igdc544R61AVhVaMZdw{N)peB^43$Mn$W z>a8RrA_(Ta{>OFHf7P9o9`m!*U?g!?6R*5j{vX24L5Nt8spwnWrO14SiKRE?ng#iuh-F+&z zayEm(m%*SE3$??FW7lvE0*7!I!e+%2oGZp_oL8wCocBI>3d{~FY%r1tSsN9VT<3+)lcg2;%Kr5m+bqy&!&R`SAsG z9ZjOuxN(GQv`{5kvzI_yzj?HMng1v#SO=Y3aSBgto^sz-^-2s@(CfME3h#b%xb3WZklIx!ZPY)lj%S_z(-)Aq*j)(T7 zLa2zz^CcI=Wt}uqLJQVCix^UpB2oc)VZJ*W7XQAYziW^H4Y-5X{CspN zl|A?28>#f1e7IuGLbK*0csFrH_CD45by|Vmp9ZbcsITN{q=u8 z{T0{G@5TRn@Ba>cE1%u}he!TW(uFbg(&6~ZKdC%B432TF4M^kJ#lX0=>CJ;PCmy8j zdcMWt4%eEaN-T#s&axjhIP0*>n~(3c*I!<$r*VJnHlo?Bd{ZX`v-4hc8YBKR7H@NQ zR(gu`Z7JtKPN2wUnVeDg+ONiTjR_}pP8G#v9hlOII2n%JjLB*heM)IIMQ|4vRodZe zE|iYCEbz_J3}56Hpl}qY9fl7Oe0Bajh1WHIUqCoE;=;b_X7+)(sf^Bx|ZYcNyc&JpSiYXT8JF?(T6LJ1uGfh>ij zdz7o=s{XyJtb&eq^P9QS9BB~45~hmkE2SSq!RVR(TijqiUJIrn0T9p&2^whv8c<=ug`evv zyfM}}9Fq&A?R~eANv$e2Q`gGQ&JT6`P)%>!(Jk5t;;UIxqa&vF*4FonF64yVq_IAQ zlPsQ|SN;=jS-b@v&8El6O(h7un*3%tXLMq@zUEZmDoS|7VRxKoQSMlY9ut6eZKpwO)w20W^q!Ry2=qJADY6%sYL@B27zDbycQ7r6V~z&%}9<{?Ex1YZ&-a zaebm6CRH9ECKnBIiDc3j$V$I)(Rir|{nn8vj!JSLv923yrG5S3zVqkk28)kaZRuTg z-eDF!**X=1Nc~H6aJR)5(Bp!>K4Eg#7%<-xvjrQGalKAX62(6i&~#nvHL2n}SiT5cnb1Kw z$B(aV=8<%hc7eE`DgV7a&|8biPNEnvgwPuR#YKjAbidWp8a(xCk$MkcP*mMqa|Q9G zU<$+j-86_=-Gw-H0LYR|D0KMP>QbtVQ`R+f>?%c4YUkummwEx8zH0Ac{j0~wR``*} z+@Kgz9PjC#Y$_!!jgih(S&?9jo?n0$W71D$OX+(`aBI5SW1{1xPi}?Dq=MT@ARUG5 zZIDEl0|eR%Q5dNn?)MBkh$M?0Y=DqIBIBn>Y4?TDgvo6aIe$!;HdDa-*qiG>``{sd zEal%(_uklb_MXn0&+lpI1jms@GW+IaC4Ua2I{26Atx|HW-M9{h^bg$YPgb!2O1w9W zQl-ohj*r_x6$WKA$2)Oa^_Z!=|?g=+C-)nd1v~cu73Is^FilQ^qhAtuFgHIQ$GYB(71$XdgO32|(}`fiL?1=#BXO z<^M@Uen0u&%YMc9L-@Vte+Rr3bMoKO{{MiZZajV6rP24&L{w;T&gVV)4yXV4p|o=I zK;iQZANQj?T@Menqa>;s)YR0buG~reue3g418pId=BBut+QhNI z$*T%GaD-ee$em1;^!B)%Qd8>8i;YUU-c79j3-a~G7s0tSSYN?_&2BM0p0zw{yU<&V zpP&H{fdP8TZ(Tr$QdL>VlSyG79h11t${Pi$VPyCvdxJr?ovuHsR{AiG&iX#@yD=w)gM-@Bs!)Fdy;wQ&~ zw6I(quBhHuy48IRU5O@!YU~y>A@EO1P5Li%%9xML^^|NVnQ%BP0TZ$7CKo5uI zdiG%X#{Jld-?Kvu>^jx3Wx1y=Pj601O|FcP%9=K<_vu4^DV?YrU|S?lVxjJX3hR(5#J8QH6z=#`OEquXN{u%=-J$ z|K9a0<0ij%{qKPPPl*243i%!If1maL;|DDr3+^50=OivbJ!k2ZU?4;&F zu-kao(7<$k^+rFCx8njq&Rp#oR58Hp*QJEHN^j8GtO(d~L?B6sQg#)_5}!m<3MM@a zNXQh8rER?elXB)D;fAJF%g_iHh$tE&jZ}qO2f^r73}j(goKy5}(zEZTs$(l#c{plK zLnh9I2O~mmhkBNHEcPJTK;fMCdqac_HY52NXqgEiuZ(0nXq*bWR)R8CsPINkxq~3j z8tUlY_ZReU1oedfbK$xVOF@2P3o;}@@?$I_#t(ci2Plfp{KZ;K9z(WpkQPI!YQtmH zmPG@S&x+Mg1Q|N zT5)9mk6kfdNBAP>B8q32**b4-a%k+0Dgrif8Lywb!`RMfO^KT!$F zPdOT`9?ieomO6wlV}HSzphd5wXM=ewS%ZvCwRV^KS( zut~aH)m^aGGp(;*oX%&moEv9;&NFq<0*yAm4VMxXDIEW6rpC3H5eU+LFsy%8YNme5 z8T8PSIDsEN>xk6UkOQt;5fG2XEb9o$=AUAY;smTVM_elOHzcMV9ZbRgr@udbf)>er z4Hqk2{YC<>gfqFl(`HsmgjJ}2ygl%8Gfuw2e2w$Os!~uf;&#`H74p3mdUIYHCW)Vi z&kias#e|xMsRev}$+@(rJsYXox*M>OJbfjaZaegrmjNl_`CJfV9w;35fQFqFvOY}* zHN7&>me=-Q91Htb&W~qfRkrtyxyA7{Nc>FNa+h|9n=0Vu#4(=5!^4PD01LRao|GUW z+vtgGG0+gW)nhNREiv&JZ;V>&tNnw%UWnrWcBlc3F{J2bTcrC}v^HQf73@mV>$P4% zLF1Js51X*~!*tp;>i`b8F?{$CY(v618e&&U%o#6Yd!p6AvE{p!sYHNLO4KvYinD4u z2OD8jDbq;x2sX*IbX-2Ec;k@7ANk6dX!aRkYWUKwV>!#X80&dFOJwk-hYIE(xB&F2 z4Zb0%Np)lvc3wFHQgyB27-YLVy)n!G%f?LcNzFMv)adU9@x|9m@?491O+hus`|1vE zRBhvRM8-kV{;Yj!ERNXqbaPcwp(+jf{akY9!m83>gvZ zIHXlH=Hj$wcX?Vg$4F6YJ(P35VllNhRP_jL%&Ws}Ri~C$>Q5DKG94ZXQMA7FACdkB zBB33R&$W@k5b8Cs<+;>??x}>8<`_N%#)@vl^X?c&=xdhqgc=k3 z37*0ifM{b6G@{@3)5d4lgDOP04zf6CluY{1WaFg=xcqcd(y2uOutLgjkZuMA&i!C@ z+ye%+M%LkUC>B{7c5U$0*2Y5eB=Jqm1&Kz|!l2N~Mgew@d1v8E9@&8US(h_MvjW0YGR5p4**ve-z z%{=V12by;^dv)7o^ul9a7cwG`Li-iZy#@u9b2X11-e`k}Vl&gj{YqGod};dl zd=<>D{7YZueD8cig6{k|_x64u>-d7;pXy-jsRveD2uZ2=0$M!;ZDPxnifvBEZGaLH zBLJY9csxKg*If#$fUHtl0l><*7(oVEzp!ZoLUrgsrv+>@t~gu=4@RX~9_Bb6NZJ}s zj&;cs_g}6O1SIY*lCL11ePQ*q8{`Ar-3nUNy>Qn7A`w?sDnOdhmk?OAn(F`za?r3H zMQ>U0X1djY5e4mSyWN)QVE6~t!*BJ-wzd=E842b6d zZoS6G@c(;N|D9O!p`QAQD=LqE+DT~F{IfW?Zo%stgw!aZLni>d@+O|_p#)w@!|RTx ztyI3h;jceKV8*7Z=%Tr;MCK}p#79CZv(u(O+Snh zF!&UKS21y~DBV3rc>QB3G3cMW@z3Y}eSeq~-~sZ<@~Bg|ynrKsTdod?UHRg11U|qe z6$beoC|%w*cwIV8d__MoxFm&5fLp3)+@T~z{QJ@We)KD^%>V0MkE_4GQPppn($1m6 z(lU&%`i*-3f86lEvu%2?@4UM2QSSBV!Nx1#YqGWkJ%HG#DTqauq)1)QU_Yd78h1Hl z^K}n1l6BV`a!Wx3)vPVkN>!m1x>GGsN18|ZN_vCxk`RyuC#N=GBA{czXUiVM%CiML1;7{eQ68X-p* zA9lOk9T(&wkKY2GCI#Ob$yP;dM&017avMHhg!RmDOS%Jc*F1T$-m%@998Z0PFCyCT zH_&!On!T&;*)4qg+$;(d+)XeLSv>@@L5zq*2V#%W>FLJgcFb99CuaOQ{5f8droT2a zQdTDe$cyJRtiiW85V7E9;ymZ#B&Ri(Cuc)gz?_I0QAY*sAuHP01B6dssIK?yKK_8A zWO2ma94Mszz{pH{U%p&sS_eb0xsyR_`sApJ353)s@;8O(jF1Xl$Hqz{X7 z8;3VMp=a%BgV^=aIfcK;v}ZN$h)aRbZX7`ti+n?%O(=@;t6`nCOpQTXuj%1u^yGW4 zjQW1nkN-NZVHR5po?&c=vA7MWl}Pv1LK)H|DIxMUu8SqhMuD!ZUA8-zXE=Z$d`|9c zq$J`Gwg!@rlBW3aI(TTE{S!9i%P5B^C+8W%+L(+F2~W}#o#jbjIW(6_i@kb`m`MZMjvfst1hPK!hjBwLai(i`=M-pt1 z>kCW!$4ZmGe!K`-iJG0rumNh2lul(~^lR}iDZi|Z4xf+t;0eiiZLq7l261w@bssVVz6@3xPsMGH?abw8C^78ybStgu1R2IFpxFnh+#P7K1H{UbrK4sL*Y79s}Qtd!~^oLQs(g zVJ?H4GIx=80npF&?6Q{!%tJP#V1Np_zBy#F4CU;@MUFf~GzSk?H)Lmv5TpU%f9FLk z-;Dqo;;WK`=$vw5`)hWav_sjW^280^o^4dky zlNGKiG>C`)on6d>zZ7*2S;#c9bRqZXj4mGNyZeI5X$7kO%nF#7H%*Rx*IqD_D1=0_ zDUBRxMmCK(G9EB4+%txftDZ=i$+55I_(L|ne@meWv80x`<)^u$%!xS z0$zNY1b-Y*eI#UeFfzh5l(I@+LSnOsK;Ae4F2usw2I#Wz<&1DN_fF*rLhPZ(BL@Z| z?;?J&wbmosS9jB(Vl{YzgxZ#X+0Z}06@&g`L{q4evO;)OpF~as%6QqdPXoMC-KpsT zV1+q&<=P5F=REYBtB*X_j$FLAfYngAt?hE4`LXpgk8sfs23A6y7X{cu)qCV}(mIVq zb&2~QE&Fi1-U-fMe_V$6)%$i|?OpLDPZc?T5TCEzk%^LB{Ho2!@N>8X{Ct@G*`-7f zl81D(4A+pQXc2Y-7AMRIvoQCQjxBS6VFTEo53p*S#n!_nz2(DK^}zn|1}=Y72E<*Z zFq#=aOMM~Q7nyw~`k+D$oQPk6utCNLFF)`Ph+$74-@2PD%8X^9cB4{KB+@54Vq_P9 zAC>ppI6!_mSSN&BW#HD~CkuK(Uh>!i1W(cCxI6t)AZ3oion)uyoT~~%jvi!D1L5lF z`x^2wBA)6cjbY0fGl7uDj3s@QCk1vaAP=~LCCyRL-UAvv2NL+Qc-GC~_VFGvqwwDO zHvyld$JH^e!C0vmf@G${>%{1*18zFceB_lk$ZOq84x#!gTs(m#Qz`aHZE`y@*9OZi zZsJMOV4zC=L}Wj&>eq);V_pA@Mc~ZoE>jc4<~wxI3I20!L(78F3$oKEEJ~pSbhnkR zXdYW$?ZL1|gV~nv!q0z2-oakDh&2nO7s{VM{$WgMjGT*M2UU06FQ2&lx8!$gcmk#7 zc9%p|N`qsYF&?YPyJ3Y7-wq6uhBI;b6y%nnl&kFHah@ zsyBhspbwfnLH0`MSlHVe0h!aRK0x7>d)rfAmm~ec=25ARODCmMvr}E&reJZrc2@WO zsb03UfzV$*>Z$h+!6e=a&!Zah3YMigV2=|Zj9!M+{Uc{y@%F?=`FVrL{M9 zP||dmC}XVvr@^hrL^nC3My7AtYKN>s7i@;)RHskDq)($@e6tlbCx8=`wSVl3E^?_w z*=~~LRY)d#0oH$;Jp!XN6Sbs;S{-mcDr3?4tn+huO$aNkB(a{z%837g!qsiSqnQr& zFL5cc{8Sg=q$`Jss;#y1q-#jc)nGGM$E0X5;5o8aFAy?K1-l{_KJrGT?6?EwcF!HL z-QZ1m4cYT$ZO-2R)B5B~^V;D>_ulO=LPF845D&v$*}GIinlCX2edpE~c{m5o7%wQ! z^It6bX&+?j!I?1sK^Tf73!2*AqGx9@vdgnTBW}TJ8GQ(%G6jL;%g~a@n93v}Z+jaz zWL}7-X|if7PMq-Gx&11VaRnqg$1M!n3GxNm4P4a`X&@d_Q6*ZkjqKr2#GelNtU|sY zGL>izXBwPC<}fqE=jGWK=fBy1%?)NpB(k@Aq+|O){#D?e%N=96tn^`$Z&H zAKcjL73xm9|#1$mA1)QdEF+`#JAI$@mS?S2<9cu-r0)zDvd%xB02hp^b6w z^#Ml*GoUq*Z8`6dIt*Vo{hcDaMO-fOs&q^20_W@2X~;0-08wsxI)V~mB<17sRlZu| zBFITrfSTb})R_!=KjTlD@mJmehoVjTDZ_^cT1InlCTafX6rzI0`RvccJ9v}l{6Qyua!MO$|m$jXnm8C{b z_HglFQRfkrnC4KFwvI~;o|d0`=l}eSgUV`k#;PPEye*kDHp5{_rQGO10ed zSkPsp{vQ^>TSRrQ|4_Vcta=0+Xb)Rv51s<6&QR<7H|151aK;=9hI~-I;s&ef7UW5i zT}WD*KR<#dFq2cKYEb=za?_-xa;f4?{C*&D%&+y=j_qGdmOhs}1-@hJ#Utqc8I9SI z)1c)ojo+|205ia48Mk~E!UNUM!wx2V)p`iFMph0FyR2S6LL+BTSM@@l%x}Bv`*ekQ zaYYX{7GB?!$Vza_WVBMp8w#S%3QhE0kA<1U4~x?|CDlXq*H$5A;l%7?M|e!2y^1QP zFVhWjBStD|nH+})k+TWZ^}H8Kz+>+$Z1?cR;6AVczpW~gw<}5IPH@mR&7b1Vz-0*N zep0SvK%5+XgSi5Brwa#5JM{;#v@e)K1!fE_!`A6by>t5_XOA~#1JGP!w-D!x+!m*g zUTeIWl(v_iD(EIwr7!G4>K!mr<}wml$)nJHi9A$LnQ78Z^Bvoxo8 zD#b>&`dPs4)gr;DpLOEPQs=i~y+qTYdC-#f*1ErYWVrxYJ`?IDVB>GMAHDlyV;mwl zwnU-`^zAlG={mYSZxfevXWI~;qQS;kqD|D4&eA6q(k4Vpu|apCh5MCw=83^r_pA?| zTD_AjuR#6OXjm^#D@<9sQSIU0n8JjQD2o~v#%HfT52iEV#-)&b$|~~9A)%iZZru)S z#DR}O>dVbKbPrGStc!N|q-xRYD91N) zch5Te=Khy7g(@4Ho!?JLaNtveurKIK#D9O3GFpVaRC`c4`Z#~`D0GRltpmynpOq1z z{q&_}3H)1~Le?ld6vOIvSe$A;zr!E;x>t!Mh6^zt+!Ae7ok5&kz+S~(BdAiz;SucX zOxMRZiEh_^%KFlJ06@`qIHL4D5ga&;N3|TZm%<{lprAr_Sbd5Ck8g?wPyV?8D$pi| zCjG7;DixaNu48xnB_`(BwR-yLOZD>f-<&`j`xW$BLT@xSjmum0g3x~Y+qzRAmH%bh zV{tAxG|*fHYXc*q6Lm*627eCC(Ud;_@=IGANPGWtBT6d)vCxhM@rlS?Op z`Y|mqbZiFBz{aDkf`K#$cu9t-?5d8qCidt%cTp0q-0TEx?O;X>|0oP~*PoCMJ@okk zY8^%&RSBESHB7iRB(El8Ebs;yNZl0|*9sHbUrkH+X+?l|z)ehIwa2FK1B*CztQFrw za5B#*oI;jIZf*wa26wv*o|^&@X*M5LQM-&;Xf3fr%Lg(rL1Y}sG7=UR_Iye_)?GM= zcybB+J@h1ZjB`RME0nC|ZX_mRm}!~t=DJ|}(x_{R;ImQ4{`>koXKs1E4mtUy8|rSs z<{VSyVlZ`ZLIz7nN;tdI=9gdu<<)`yom{ z{~9c%D-^v5Zkh7sAKwH?%@qNMBwbmZCYJIv3uhM(IknoW9-yRedjj zUjE2m?0{`p7r}L`vxxK*@UFKZi?_ZnZw%fK$m9e1_UFH+ zpZYg(eRXoS5~>;n2u^GLnB&G9qBioZcy0&*1!P|Na@IDRKLj!HsXY#_kp)?i@^*;0NgO|}qAx0j@#9yYmam!qFt z*V=prwRC}oWV7W-rYuaG@7|(*WJi_=_!M74j2&bNJ$w~>IjZ~5D8IQcE`>{g0)8j) zqAK8vLOQd{bNuAEO`xV-Pt|~LQU5*@8*qwC)}aJC$mcW?A!M-@D z)l2A5h9>!AdW*pFM2*0K5W6LNC^lPtN06uhyJ1}BslfyEcfLYGeXzjFG;zM_)RLB= zF;5piRXVs4TYMS?5(T0>PEa?8~m~ayRF(>d)ssNV?ZZ+-XpSCQySID*&1Nk(UwtFLENqOI^1rd1R1{pHgVowGSm4f zN;|>eInpA$eugyI@6I;*3E_A5xvXlgSBjC^Wz;R)_bML?5RcZn(k3sR3ZX1m?Pnrb z8Xe7gs!Y1N2yz=7oeBgO8e;Wmp;V#jyU_A#(UfP2W8w=M`LmHXcdKYjeiF92--*JW zo2y`s#*xM74}T(J&-d60f4!bc`+)n7&17JV{6ZeB9DrY6jA?f&`MoXwQtQs}vq!pS zD4Y)QRd&uNzTcB6Au>o08}lB0$3EbiAuq#>Dz%p+pn?3jseEgdF+sb2I%)WB(OJ)!(6?kPdSWYy~8f?{gM7D%zXtw{Er~egH?7^IuuF0k-M@ z)Sj%*Du8noA*Z|cM%|r1z737Ha9!rdms-QB4By`b!D0Gf34>fOeBcR+ca{~H{q!eo zFcecKvfKBki~k-};RjqgO_&dJs^!n+?T=duP}L8BcExwl+ylwjAI=XhQg{R0RY;xN zqN0~u0Y888a?Id+0HDYVnd7-X-`$`78#EXG22TGuL4U*=^c$w*0Fa;Fz8f`XemtHZ zZ&1ep*Eh{)Mb>y+;lI%B2HZ4l1Ea{wZ_i-q57$S=QopaWUw)uvgdVm2t+f;!9@k_l z!S%_ikeNEaUOD+{4XXuDV=$Pt$BrJ-#{aQ)fphic-AVUCk8FfrVvZ@OA5A@c`j7tw D)li$g diff --git a/_freeze/R/kolmogorov-smirnov_test/execute-results/html.json b/_freeze/R/kolmogorov-smirnov_test/execute-results/html.json index 17015c42e..2f4c80c52 100644 --- a/_freeze/R/kolmogorov-smirnov_test/execute-results/html.json +++ b/_freeze/R/kolmogorov-smirnov_test/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "8d88ef01e73e0cf3432d8343bf86fb2b", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"R: Kolmogorov-Smirnov test\"\n---\n\n# Introduction\n\nKolmogorov-Smirnov (K-S) test is a non-parametric test employed to check whether the probability distributions of a sample and a control distribution, or two samples are equal. It is constructed based on the cumulative distribution function (CDF) and calculates the greatest difference between the empirical distribution function (EDF) of the sample and the theoretical or empirical distribution of the control sample.\n\nThe Kolmogorov-Smirnov test is mostly used for two purposes:\n\n1. **One-sample K-S test**: To compare the sample distribution to a known reference distribution.\n\n2. **Two-sample K-S test**: To compare the two independent samples' distributions.\n\nThe K-S test is formulated on the basis of the maximum difference between the observed and expected cumulative distribution functions (CDFs). The test is non-parametric, as it does not assume any specific distribution for the sample data. This makes it especially helpful in testing the goodness-of-fit for continuous distributions.\n\n# Libraries or Extensions Needed\n\nTo perform the Kolmogorov-Smirnov test in R, we will use the `ks.test()` function from the `dgof` package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(dgof)\n```\n:::\n\n\n# Data Sources for the Analysis\n\nWe will use the `lung` dataset from the `survival` package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(survival)\nattach(lung)\n```\n:::\n\n\nDetails about the lung dataset can be found in the documentation for the `survival` package, which is available at .\n\n# Statistical Method\n\n## One-sample K-S test\n\nFor this example, we will test whether the Karnofsky performance score rated by physician (`ph.karno`), and Karnofsky performance score rated by patient (`pat.karno`) follow a normal distribution.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nks.test(ph.karno, \"pnorm\")\nks.test(pat.karno, \"pnorm\")\n```\n:::\n\n\nBoth tests have p-values \\< 2.2e-16, which indicates that the distributions of `ph.karno` and `pat.karno` are significantly different from a normal distribution.\n\n## Two-sample K-S test\n\nNext, we will compare the distributions of `ph.karno` and `pat.karno` using the two-sample K-S test.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nks.test(ph.karno, pat.karno)\n```\n:::\n\n\nThe p-value of 0.2084 suggests that there is no significant difference between the distributions of `ph.karno` and `pat.karno`. This indicates that the Karnofsky performance scores rated by physicians and patients are not significantly different in terms of their distribution.\n\n# Conclusion\n\nWe demonstrated the use of the Kolmogorov-Smirnov test in R using the ks.test() function from the dgof package, which is straightforward and handy to use. As far as we are aware of, this is the most widely used function in R to perform the Kolmogorov-Smirnov test.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-26\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n R dgof [?] \n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n R ── Package was removed from disk.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::\n", + "markdown": "---\ntitle: \"R: Kolmogorov-Smirnov test\"\n---\n\n# Introduction\n\nKolmogorov-Smirnov (K-S) test is a non-parametric test employed to check whether the probability distributions of a sample and a control distribution, or two samples are equal. It is constructed based on the cumulative distribution function (CDF) and calculates the greatest difference between the empirical distribution function (EDF) of the sample and the theoretical or empirical distribution of the control sample.\n\nThe Kolmogorov-Smirnov test is mostly used for two purposes:\n\n1. **One-sample K-S test**: To compare the sample distribution to a known reference distribution.\n\n2. **Two-sample K-S test**: To compare the two independent samples' distributions.\n\nThe K-S test is formulated on the basis of the maximum difference between the observed and expected cumulative distribution functions (CDFs). The test is non-parametric, as it does not assume any specific distribution for the sample data. This makes it especially helpful in testing the goodness-of-fit for continuous distributions.\n\n# Libraries or Extensions Needed\n\nTo perform the Kolmogorov-Smirnov test in R, we will use the `ks.test()` function from the `dgof` package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(dgof)\n```\n:::\n\n\n# Data Sources for the Analysis\n\nWe will use the `lung` dataset from the `survival` package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(survival)\nattach(lung)\n```\n:::\n\n\nDetails about the lung dataset can be found in the documentation for the `survival` package, which is available at .\n\n# Statistical Method\n\n## One-sample K-S test\n\nFor this example, we will test whether the Karnofsky performance score rated by physician (`ph.karno`), and Karnofsky performance score rated by patient (`pat.karno`) follow a normal distribution.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nks.test(ph.karno, \"pnorm\")\nks.test(pat.karno, \"pnorm\")\n```\n:::\n\n\nBoth tests have p-values \\< 2.2e-16, which indicates that the distributions of `ph.karno` and `pat.karno` are significantly different from a normal distribution.\n\n## Two-sample K-S test\n\nNext, we will compare the distributions of `ph.karno` and `pat.karno` using the two-sample K-S test.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nks.test(ph.karno, pat.karno)\n```\n:::\n\n\nThe p-value of 0.2084 suggests that there is no significant difference between the distributions of `ph.karno` and `pat.karno`. This indicates that the Karnofsky performance scores rated by physicians and patients are not significantly different in terms of their distribution.\n\n# Conclusion\n\nWe demonstrated the use of the Kolmogorov-Smirnov test in R using the ks.test() function from the dgof package, which is straightforward and handy to use. As far as we are aware of, this is the most widely used function in R to perform the Kolmogorov-Smirnov test.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n R dgof [?] \n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n R ── Package was removed from disk.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n:::\n", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/R/logistic_regr/execute-results/html.json b/_freeze/R/logistic_regr/execute-results/html.json index 87fdf980f..28780b445 100644 --- a/_freeze/R/logistic_regr/execute-results/html.json +++ b/_freeze/R/logistic_regr/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "39852b48cd43fbaa59d09c59cadeb8bb", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Logistic Regression in R\"\n---\n\n\n\nIn binary logistic regression, there is a single binary dependent variable, coded by an indicator variable. For example, if we represent a response as 1 and non-response as 0, then the corresponding probability of response, can be between 0 (certainly not a response) and 1 (certainly a response) - hence the labeling !\n\nThe logistic model models the log-odds of an event as a linear combination of one or more independent variables (explanatory variables). If we observed $(y_i, x_i),$ where $y_i$ is a Bernoulli variable and $x_i$ a vector of explanatory variables, the model for $\\pi_i = P(y_i=1)$ is\n\n$$\n\\text{logit}(\\pi_i)= \\log\\left\\{ \\frac{\\pi_i}{1-\\pi_i}\\right\\} = \\beta_0 + \\beta x_i, i = 1,\\ldots,n \n$$\n\nThe model is especially useful in case-control studies and leads to the effect of risk factors by odds ratios.\n\n# Example: Lung Cancer Data\n\n*Data source: Loprinzi CL. Laurie JA. Wieand HS. Krook JE. Novotny PJ. Kugler JW. Bartel J. Law M. Bateman M. Klatt NE. et al. Prospective evaluation of prognostic variables from patient-completed questionnaires. North Central Cancer Treatment Group. Journal of Clinical Oncology. 12(3):601-7, 1994.*\n\nwgt_catn consists of: 1= patients a weight loss of zero or less, 0= patients with a weight loss of more than zero\n\ntrt01pn consists of 1= active treatment, 0 = placebo\n\n# Model Fit\n\nWe analyze the event of weight gain (or staying the same weight) in lung cancer patients in dependency of treatment (active or placebo), age, sex, ECOG performance score and calories consumed at meals. One of the most important things to remember is to ensure you tell R what your event is and what treatment comparison you are doing Active / Placebo or Placebo/Active! The easiest way to do this is to have event (or non-reference treatment) as 1, and non-event (reference treatment) as 0.\n\nBelow we are using wt_catn (0,1) and trt01pn (1,2) and sex (1,2). Let's see what happens !\n\nhead(lung)\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlung <- read.csv(\"../data/lung_cancer.csv\")\nhead(lung)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n inst time status age sex ph.ecog ph.karno pat.karno meal.cal wt.loss trt01p\n1 3 306 2 74 1 1 90 100 1175 NA Active\n2 3 455 2 68 1 0 90 90 1225 15 Active\n3 3 1010 1 56 1 0 90 90 NA 15 Active\n4 5 210 2 57 1 1 90 60 1150 11 Active\n5 1 883 2 60 1 0 100 90 NA 0 Active\n6 12 1022 1 74 1 1 50 80 513 0 Active\n trt01pn dose_mg dose_id wt_cat wt_catn cnsr\n1 1 10 1 NA 0\n2 1 10 1 loss 0 0\n3 1 10 1 loss 0 1\n4 1 10 1 loss 0 0\n5 1 10 1 gain 1 0\n6 1 10 1 gain 1 1\n```\n\n\n:::\n\n```{.r .cell-code}\nm1 <- glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\")\n)\nsummary(m1)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nglm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal, \n family = binomial(link = \"logit\"), data = lung)\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -3.8623998 1.7675776 -2.185 0.0289 *\ntrt01pn 0.3887667 0.3781566 1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n:::\n\n\nR by default sets the first category as the baseline category, hence trt01pn =1 and sex =1 are the baseline level, and other levels of the variable are contrasted to this level. This is using `contr.treatment` option (more information on this later!). The estimate for those variables (0.3887 and 0.8321) are the increase in the log-odds of being of receiving treatment 2 vs 1, and of being of sex=2 vs 1. The exponential of the estimate is the odds ratio. For example, exp(0.3887)=1.475, hence Treatment 2 is 1.475 times as likely to have weight gain compared to Treatment 1.\n\nThe intercept represents the baseline log odds of the outcome when all predictor variables are set to zero. In the above model, we have variables treatment and sex, each coded as 1 and 2. Currently R is not treating these as binary factors, instead R thinks they are a continuous variable, and hence the intercept is where treatment=0 and sex=0. In the context of our model, this doesn't make sense as you can't have a zero gender and zero treatment (but it may not matter as we rarely look at the intercept term anyway!)\n\nHowever, if you want to have an interpretable intercept term (and if you want to match SAS output for intercept!), then it's important to ensure any factors in your model as fitted as such in R. You can do this using: `lung$trt01pn<-as.factor(lung$trt01pn)` or by changing the variable to a 0 and 1. Note if you are using the same `contr.treatment` option, then this only affects the intercept estimate not the variable estimates.\n\n## Modelling factors correctly and Interpretation of the Intercept\n\nSo far we've learnt that it is good practice to always have binary and categorical variables set as factors, and that we should specify what method we want R to use for doing any contrasts (e.g. `contr.treatment`). You can specify different contrast methods for each variable by including them in a list in the model. It can sometimes be hard to see what contrast method R is using, so best to always specify this in your model (e.g. `contrasts = list(trt01pn = \"contr.treatment\", sex=\"contr.sum\")`. It is helpful to view what the contrasts look like before you select which to use.\n\nA factor with 2 levels (2 treatments) using `contr.treatment` would set the first level to be the reference (0) and contrast the second level to the baseline.\n\nA factor with 4 levels (4 treatments) using `contr.treatment` would set the first level to be the reference (0), you would see 3 parameters in the model with 3 estimates corresponding to the increase in log-odds attributable to contrasting the second, third or fourth level to the baseline.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncontr.treatment(2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 2\n1 0\n2 1\n```\n\n\n:::\n\n```{.r .cell-code}\ncontr.treatment(4)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 2 3 4\n1 0 0 0\n2 1 0 0\n3 0 1 0\n4 0 0 1\n```\n\n\n:::\n:::\n\n\nBelow we apply `contr.treatment` to our trt01pn and sex variables. Note: how the variables trt01pn2 and sex2 are now shown in the output, this is indicating that these rows relates to treatment=2 and sex=2. See the next section for how to interpret the estimates and change this `contr.` option.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Good practice to have binary variables (eg. treatment) identified as a factor\n# And to specify what contrast option you are using ! more on this below\nlung$trt01pn <- as.factor(lung$trt01pn)\nlung$sex <- as.factor(lung$sex)\n\nm1 <- stats::glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.treatment\", sex = \"contr.treatment\")\n)\nsummary(m1)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + \n meal.cal, family = binomial(link = \"logit\"), data = lung, \n contrasts = list(trt01pn = \"contr.treatment\", sex = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -2.6415326 1.5140191 -1.745 0.0810 .\ntrt01pn2 0.3887667 0.3781566 1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex2 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n:::\n\n\n# Model parameter estimates\n\nThe model summary contains the parameter estimates $\\beta_j$ for each explanatory variable $x_j$, corresponding to the log-odds for the response variable to take the value $1$, conditional on all other explanatory variables remaining constant. For better interpretation, we can exponentiate these estimates, to obtain estimates for the odds instead and provide 95% confidence intervals.\n\n## How R parameterizes your variables in the model\n\n- contr.treatment - \\[default\\] sets the first level to be the reference and contrasts each other level with the baseline. For example, based on the model shown below. Log-odds for Treatment 2 = -2.64153 + 0.38876 =-2.25277 , Log-odds for Treatment 1=-2.64153\n\n- contr.sum - sets the last level to be the reference and compares the mean of the dependent variable for a given level to the overall mean of the dependent variable. For treatment 1, the intercept + trt01pn1 estimate. For Treatment 2, the intercept - trt01pn1 estimate.\\\n Log-odds for Treatment 2 = -2.44715 - - 0.19438 = -2.25277, Log-odds for Treatment 1=-2.44715 - 0.19438 = -2.64153\\\n use `contr.sum(2)` to show the contrast you are using and hence how to interpret your estimates.\n\n- exponential (ratio log-odds) = odds ratio. eg. -2.25277/ -2.64153 = 0.85283 Treatment 2 is 0.85 times as likely (eg. 15% less likely) to have weight gain compared to Treatment 1.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nma <- stats::glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.treatment\", sex = \"contr.treatment\")\n)\nsummary(ma)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + \n meal.cal, family = binomial(link = \"logit\"), data = lung, \n contrasts = list(trt01pn = \"contr.treatment\", sex = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -2.6415326 1.5140191 -1.745 0.0810 .\ntrt01pn2 0.3887667 0.3781566 1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex2 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n\n```{.r .cell-code}\ncontr.treatment(2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 2\n1 0\n2 1\n```\n\n\n:::\n\n```{.r .cell-code}\nmb <- stats::glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.sum\", sex = \"contr.treatment\")\n)\nsummary(mb)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + \n meal.cal, family = binomial(link = \"logit\"), data = lung, \n contrasts = list(trt01pn = \"contr.sum\", sex = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -2.4471493 1.4929863 -1.639 0.1012 \ntrt01pn1 -0.1943833 0.1890783 -1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex2 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n\n```{.r .cell-code}\ncontr.sum(2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [,1]\n1 1\n2 -1\n```\n\n\n:::\n:::\n\n\n## Calculation of confidence intervals\n\nUsing the above model, you can output the estimates and confidence intervals using coef() and confint() and exponential back transforming using exp(). NOTE: that there are two types of confidence intervals that you can calculate. Function `confint.default` gives the Wald confidence limits, which is the default option in SAS `PROC LOGISTIC` procedure; whereas `confint` gives the profile-likelihood limits.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# model coefficients summary\nsummary(m1)$coefficients\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error z value Pr(>|z|)\n(Intercept) -2.6415326252 1.5140190574 -1.7447156 0.08103439\ntrt01pn2 0.3887666677 0.3781565596 1.0280574 0.30392281\nage 0.0122549015 0.0211552875 0.5792831 0.56239813\nsex2 0.8321005169 0.3743792762 2.2226137 0.02624186\nph.ecog -0.3763592487 0.2638321918 -1.4265100 0.15372119\nmeal.cal 0.0008499918 0.0004486401 1.8945961 0.05814593\n```\n\n\n:::\n\n```{.r .cell-code}\n# Wald confidence limits\ncbind(est = exp(coef(m1)), exp(confint.default(m1)))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n est 2.5 % 97.5 %\n(Intercept) 0.07125198 0.003664896 1.385263\ntrt01pn2 1.47516031 0.702994248 3.095470\nage 1.01233030 0.971213751 1.055188\nsex2 2.29814096 1.103327506 4.786840\nph.ecog 0.68635572 0.409236994 1.151128\nmeal.cal 1.00085035 0.999970674 1.001731\n```\n\n\n:::\n\n```{.r .cell-code}\n# profile-likelihood limits\ncbind(est = exp(coef(m1)), exp(confint(m1)))\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nWaiting for profiling to be done...\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n est 2.5 % 97.5 %\n(Intercept) 0.07125198 0.003312288 1.302587\ntrt01pn2 1.47516031 0.696210092 3.085089\nage 1.01233030 0.971670916 1.056194\nsex2 2.29814096 1.107651762 4.836770\nph.ecog 0.68635572 0.405659156 1.147452\nmeal.cal 1.00085035 0.999978126 1.001761\n```\n\n\n:::\n:::\n\n\n# Comparing 2 models\n\nTo compare two logistic models, the `residual deviances` (-2 \\* log likelihoods) are compared against a $\\chi^2$-distribution with degrees of freedom calculated using the difference in the two models' parameters. Below, the only difference is the inclusion/exclusion of age in the model, hence we test using $\\chi^2$ with 1 df. Here testing at the 5% level.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nm2 <- stats::glm(\n wt_catn ~ trt01pn + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.treatment\")\n)\nsummary(m2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + sex + ph.ecog + meal.cal, \n family = binomial(link = \"logit\"), data = lung, contrasts = list(trt01pn = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -1.8350766 0.5810727 -3.158 0.00159 **\ntrt01pn2 0.3681563 0.3761976 0.979 0.32777 \nsex2 0.7919227 0.3674936 2.155 0.03117 * \nph.ecog -0.3312031 0.2527586 -1.310 0.19008 \nmeal.cal 0.0007896 0.0004378 1.803 0.07131 . \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.80 on 165 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 200.8\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n\n```{.r .cell-code}\nanova(m1, m2, test = \"LRT\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnalysis of Deviance Table\n\nModel 1: wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal\nModel 2: wt_catn ~ trt01pn + sex + ph.ecog + meal.cal\n Resid. Df Resid. Dev Df Deviance Pr(>Chi)\n1 164 190.46 \n2 165 190.80 -1 -0.33867 0.5606\n```\n\n\n:::\n:::\n\n\nStackexchange [here](https://stats.stackexchange.com/questions/59879/logistic-regression-anova-chi-square-test-vs-significance-of-coefficients-ano) has a good article describing this method and the difference between comparing 2 models using the likelihood ratio tests versus using wald tests and Pr\\>chisq (from the maximum likelihood estimate). Note: `anova(m1, m2, test = \"Chisq\")` and using `test=\"LRT\"` as above are synonymous in this context.\n\n# Predicting likelihood of response for new patients\n\nPredictions from the model for the log-odds of a patient with new data to experience a weight loss are derived using `predict()`:\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# new female, symptomatic but completely ambulatory patient consuming 2500 calories\nnew_pt <- data.frame(\n trt01pn = 1,\n age = 48,\n sex = 2,\n ph.ecog = 1,\n meal.cal = 2500\n)\nnew_pt$trt01pn <- as.factor(new_pt$trt01pn)\nnew_pt$sex <- as.factor(new_pt$sex)\npredict(m1, new_pt, type = \"response\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 1 \n0.628882 \n```\n\n\n:::\n:::\n\n\n# Creating Treatment Contrasts for 2 treatments\n\n## {emmeans}\n\nHere we will use {emmeans} to output the log-odds of weight gain for treatment 1 and treatment 2.\n\nNOTE as per the output, these are on the logit scale, you need to exponentiate to get the odds (or use the type=\"response\" option).\n\nThe treatment comparison can also be output on the log-odds or back transformed scale as shown below.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nm3 <- stats::glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.treatment\")\n)\nsummary(m3)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + \n meal.cal, family = binomial(link = \"logit\"), data = lung, \n contrasts = list(trt01pn = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -2.6415326 1.5140191 -1.745 0.0810 .\ntrt01pn2 0.3887667 0.3781566 1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex2 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n\n```{.r .cell-code}\ncontr.treatment(2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 2\n1 0\n2 1\n```\n\n\n:::\n\n```{.r .cell-code}\n# log-odds for each treatment\nlsm <- emmeans::emmeans(m3, \"trt01pn\")\nlsm\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n trt01pn emmean SE df asymp.LCL asymp.UCL\n 1 -1.034 0.225 Inf -1.47 -0.5935\n 2 -0.645 0.302 Inf -1.24 -0.0529\n\nResults are averaged over the levels of: sex \nResults are given on the logit (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n\n```{.r .cell-code}\n# log-odds ratios (treatment comparison): This does all pairwise comparisons\n# However as seen below, this is TRT 1 - TRT 2\npairs(lsm)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df z.ratio p.value\n trt01pn1 - trt01pn2 -0.389 0.378 Inf -1.028 0.3039\n\nResults are averaged over the levels of: sex \nResults are given on the log odds ratio (not the response) scale. \n```\n\n\n:::\n\n```{.r .cell-code}\n# the below creates tests and CI's prior to back transformation (ratios of geometric means)\npairs(lsm, type = \"response\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast odds.ratio SE df null z.ratio p.value\n trt01pn1 / trt01pn2 0.678 0.256 Inf 1 -1.028 0.3039\n\nResults are averaged over the levels of: sex \nTests are performed on the log odds ratio scale \n```\n\n\n:::\n\n```{.r .cell-code}\n# see coefficients of the linear functions\ncoef(pairs(lsm))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n trt01pn c.1\n1 1 1\n2 2 -1\n```\n\n\n:::\n\n```{.r .cell-code}\n# Output treatment contrasts 2 vs 1 and 95% CIs, the type=\"response\" option back transforms the results\ntrtdiff <- contrast(lsm, \"poly\")\ntrtdiff\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df z.ratio p.value\n linear 0.389 0.378 Inf 1.028 0.3039\n\nResults are averaged over the levels of: sex \nResults are given on the log odds ratio (not the response) scale. \n```\n\n\n:::\n\n```{.r .cell-code}\nconfint(trtdiff, type = \"response\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast odds.ratio SE df asymp.LCL asymp.UCL\n linear 1.48 0.558 Inf 0.703 3.1\n\nResults are averaged over the levels of: sex \nConfidence level used: 0.95 \nIntervals are back-transformed from the log odds ratio scale \n```\n\n\n:::\n:::\n\n\nIn Summary: Treatment 2 is on average 1.48 times as likely to have weight gain compared to treatment 1, however this is not statistically significant (95% Confidence interval = 0.703-3.100, p-value= 0.3039).\n\n# Creating Treatment Contrasts for 2 or more treatments\n\n{emmeans} can also be used to do specific contrasts, instead of investigating treatment (active vs placebo), suppose we now want to look at 2 dose groups vs placebo.\n\nWe have a 3 level treatment variable (dose_id), where 1=10mg, 2=20mg doses for active treatment and 3= placebo which is 0mg. We want to test the null hypothesis that 0.5\\**dose10mg + 0.5*\\*dose20mg - placebo = 0. That there is not difference between the doses.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlung2 <- lung |>\n mutate(dose_id2 = as.factor(lung$dose_id))\n\nm3 <- stats::glm(\n wt_catn ~ dose_id2 + age + sex + ph.ecog + meal.cal,\n data = lung2,\n family = binomial(link = \"logit\"),\n contrasts = list(dose_id2 = \"contr.treatment\")\n)\n\n# log-odds for each treatment\nlsm3 <- emmeans::emmeans(m3, \"dose_id2\")\nlsm3\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n dose_id2 emmean SE df asymp.LCL asymp.UCL\n 1 -0.903 0.296 Inf -1.48 -0.3228\n 2 -1.201 0.348 Inf -1.88 -0.5201\n 3 -0.643 0.302 Inf -1.23 -0.0518\n\nResults are averaged over the levels of: sex \nResults are given on the logit (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n\n```{.r .cell-code}\ncontrast(lsm3, list(AveDose_vs_pbo = c(0.5, 0.5, -1)))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df z.ratio p.value\n AveDose_vs_pbo -0.41 0.38 Inf -1.078 0.2813\n\nResults are averaged over the levels of: sex \nResults are given on the log odds ratio (not the response) scale. \n```\n\n\n:::\n\n```{.r .cell-code}\nconfint(contrast(lsm3, list(AveDose_vs_pbo = c(0.5, 0.5, -1))))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df asymp.LCL asymp.UCL\n AveDose_vs_pbo -0.41 0.38 Inf -1.15 0.335\n\nResults are averaged over the levels of: sex \nResults are given on the log odds ratio (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n:::\n\n\nHere we found that on average there is -0.41 times the risk of weight gain on active vs placebo but that this is not statisticallly significantly different (95% CI -1.15 to 0.335, p-value = 0.2813).\n\nSee the emmeans vignette on creating bespoke contrasts [here](Comparisons%20and%20contrasts%20in%20emmeans).\n\n## {gmodels}\n\n{gmodels} is an alternative package to create contrasts instead of \\, you can use the `fit.contrast()` function from the `gmodels` package. The same result is obtained as \\.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlung2 <- lung |>\n mutate(dose_id2 = as.factor(lung$dose_id))\n\nm3 <- stats::glm(\n wt_catn ~ dose_id2 + age + sex + ph.ecog + meal.cal,\n data = lung2,\n family = binomial(link = \"logit\"),\n contrasts = list(dose_id2 = \"contr.treatment\")\n)\n\ngmodels::fit.contrast(m3, 'dose_id2', c(0.5, 0.5, -1), conf.int = 0.95)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error z value Pr(>|z|) lower CI\ndose_id2 c=( 0.5 0.5 -1 ) -0.4096323 0.3801683 -1.077502 0.2812558 -1.160322\n upper CI\ndose_id2 c=( 0.5 0.5 -1 ) 0.3410574\n```\n\n\n:::\n:::\n\n\n# Reference\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-23\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P cli 3.6.5 2025-04-23 [?] RSPM\n P dplyr * 1.2.0 2026-02-03 [?] RSPM\n P emmeans * 2.0.1 2025-12-16 [?] RSPM\n P estimability 1.5.1 2024-05-12 [?] RSPM\n P gdata 3.0.1 2024-10-22 [?] RSPM\n P generics 0.1.4 2025-05-09 [?] RSPM\n P glue 1.8.0 2024-09-30 [?] RSPM\n P gmodels * 2.19.1 2024-03-06 [?] RSPM\n P gtools 3.9.5 2023-11-20 [?] RSPM\n P lifecycle 1.0.5 2026-01-08 [?] RSPM\n P magrittr 2.0.4 2025-09-12 [?] RSPM\n MASS 7.3-65 2025-02-28 [2] CRAN (R 4.5.2)\n P mvtnorm 1.3-3 2025-01-10 [?] RSPM\n numDeriv 2016.8-1.1 2019-06-06 [1] RSPM\n P pillar 1.11.1 2025-09-17 [?] RSPM\n P pkgconfig 2.0.3 2019-09-22 [?] RSPM\n P R6 2.6.1 2025-02-15 [?] RSPM\n P rlang 1.1.7 2026-01-09 [?] RSPM\n P tibble 3.3.1 2026-01-11 [?] RSPM\n P tidyselect 1.2.1 2024-03-11 [?] RSPM\n utf8 1.2.6 2025-06-08 [1] RSPM\n P vctrs 0.7.1 2026-01-23 [?] RSPM\n withr 3.0.2 2024-10-28 [1] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n", + "markdown": "---\ntitle: \"Logistic Regression in R\"\n---\n\n\n\nIn binary logistic regression, there is a single binary dependent variable, coded by an indicator variable. For example, if we represent a response as 1 and non-response as 0, then the corresponding probability of response, can be between 0 (certainly not a response) and 1 (certainly a response) - hence the labeling !\n\nThe logistic model models the log-odds of an event as a linear combination of one or more independent variables (explanatory variables). If we observed $(y_i, x_i),$ where $y_i$ is a Bernoulli variable and $x_i$ a vector of explanatory variables, the model for $\\pi_i = P(y_i=1)$ is\n\n$$\n\\text{logit}(\\pi_i)= \\log\\left\\{ \\frac{\\pi_i}{1-\\pi_i}\\right\\} = \\beta_0 + \\beta x_i, i = 1,\\ldots,n \n$$\n\nThe model is especially useful in case-control studies and leads to the effect of risk factors by odds ratios.\n\n# Example: Lung Cancer Data\n\n*Data source: Loprinzi CL. Laurie JA. Wieand HS. Krook JE. Novotny PJ. Kugler JW. Bartel J. Law M. Bateman M. Klatt NE. et al. Prospective evaluation of prognostic variables from patient-completed questionnaires. North Central Cancer Treatment Group. Journal of Clinical Oncology. 12(3):601-7, 1994.*\n\nwgt_catn consists of: 1= patients a weight loss of zero or less, 0= patients with a weight loss of more than zero\n\ntrt01pn consists of 1= active treatment, 0 = placebo\n\n# Model Fit\n\nWe analyze the event of weight gain (or staying the same weight) in lung cancer patients in dependency of treatment (active or placebo), age, sex, ECOG performance score and calories consumed at meals. One of the most important things to remember is to ensure you tell R what your event is and what treatment comparison you are doing Active / Placebo or Placebo/Active! The easiest way to do this is to have event (or non-reference treatment) as 1, and non-event (reference treatment) as 0.\n\nBelow we are using wt_catn (0,1) and trt01pn (1,2) and sex (1,2). Let's see what happens !\n\nhead(lung)\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlung <- read.csv(\"../data/lung_cancer.csv\")\nhead(lung)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n inst time status age sex ph.ecog ph.karno pat.karno meal.cal wt.loss trt01p\n1 3 306 2 74 1 1 90 100 1175 NA Active\n2 3 455 2 68 1 0 90 90 1225 15 Active\n3 3 1010 1 56 1 0 90 90 NA 15 Active\n4 5 210 2 57 1 1 90 60 1150 11 Active\n5 1 883 2 60 1 0 100 90 NA 0 Active\n6 12 1022 1 74 1 1 50 80 513 0 Active\n trt01pn dose_mg dose_id wt_cat wt_catn cnsr\n1 1 10 1 NA 0\n2 1 10 1 loss 0 0\n3 1 10 1 loss 0 1\n4 1 10 1 loss 0 0\n5 1 10 1 gain 1 0\n6 1 10 1 gain 1 1\n```\n\n\n:::\n\n```{.r .cell-code}\nm1 <- glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\")\n)\nsummary(m1)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nglm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal, \n family = binomial(link = \"logit\"), data = lung)\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -3.8623998 1.7675776 -2.185 0.0289 *\ntrt01pn 0.3887667 0.3781566 1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n:::\n\n\nR by default sets the first category as the baseline category, hence trt01pn =1 and sex =1 are the baseline level, and other levels of the variable are contrasted to this level. This is using `contr.treatment` option (more information on this later!). The estimate for those variables (0.3887 and 0.8321) are the increase in the log-odds of being of receiving treatment 2 vs 1, and of being of sex=2 vs 1. The exponential of the estimate is the odds ratio. For example, exp(0.3887)=1.475, hence Treatment 2 is 1.475 times as likely to have weight gain compared to Treatment 1.\n\nThe intercept represents the baseline log odds of the outcome when all predictor variables are set to zero. In the above model, we have variables treatment and sex, each coded as 1 and 2. Currently R is not treating these as binary factors, instead R thinks they are a continuous variable, and hence the intercept is where treatment=0 and sex=0. In the context of our model, this doesn't make sense as you can't have a zero gender and zero treatment (but it may not matter as we rarely look at the intercept term anyway!)\n\nHowever, if you want to have an interpretable intercept term (and if you want to match SAS output for intercept!), then it's important to ensure any factors in your model as fitted as such in R. You can do this using: `lung$trt01pn<-as.factor(lung$trt01pn)` or by changing the variable to a 0 and 1. Note if you are using the same `contr.treatment` option, then this only affects the intercept estimate not the variable estimates.\n\n## Modelling factors correctly and Interpretation of the Intercept\n\nSo far we've learnt that it is good practice to always have binary and categorical variables set as factors, and that we should specify what method we want R to use for doing any contrasts (e.g. `contr.treatment`). You can specify different contrast methods for each variable by including them in a list in the model. It can sometimes be hard to see what contrast method R is using, so best to always specify this in your model (e.g. `contrasts = list(trt01pn = \"contr.treatment\", sex=\"contr.sum\")`. It is helpful to view what the contrasts look like before you select which to use.\n\nA factor with 2 levels (2 treatments) using `contr.treatment` would set the first level to be the reference (0) and contrast the second level to the baseline.\n\nA factor with 4 levels (4 treatments) using `contr.treatment` would set the first level to be the reference (0), you would see 3 parameters in the model with 3 estimates corresponding to the increase in log-odds attributable to contrasting the second, third or fourth level to the baseline.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ncontr.treatment(2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 2\n1 0\n2 1\n```\n\n\n:::\n\n```{.r .cell-code}\ncontr.treatment(4)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 2 3 4\n1 0 0 0\n2 1 0 0\n3 0 1 0\n4 0 0 1\n```\n\n\n:::\n:::\n\n\nBelow we apply `contr.treatment` to our trt01pn and sex variables. Note: how the variables trt01pn2 and sex2 are now shown in the output, this is indicating that these rows relates to treatment=2 and sex=2. See the next section for how to interpret the estimates and change this `contr.` option.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Good practice to have binary variables (eg. treatment) identified as a factor\n# And to specify what contrast option you are using ! more on this below\nlung$trt01pn <- as.factor(lung$trt01pn)\nlung$sex <- as.factor(lung$sex)\n\nm1 <- stats::glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.treatment\", sex = \"contr.treatment\")\n)\nsummary(m1)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + \n meal.cal, family = binomial(link = \"logit\"), data = lung, \n contrasts = list(trt01pn = \"contr.treatment\", sex = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -2.6415326 1.5140191 -1.745 0.0810 .\ntrt01pn2 0.3887667 0.3781566 1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex2 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n:::\n\n\n# Model parameter estimates\n\nThe model summary contains the parameter estimates $\\beta_j$ for each explanatory variable $x_j$, corresponding to the log-odds for the response variable to take the value $1$, conditional on all other explanatory variables remaining constant. For better interpretation, we can exponentiate these estimates, to obtain estimates for the odds instead and provide 95% confidence intervals.\n\n## How R parameterizes your variables in the model\n\n- contr.treatment - \\[default\\] sets the first level to be the reference and contrasts each other level with the baseline. For example, based on the model shown below. Log-odds for Treatment 2 = -2.64153 + 0.38876 =-2.25277 , Log-odds for Treatment 1=-2.64153\n\n- contr.sum - sets the last level to be the reference and compares the mean of the dependent variable for a given level to the overall mean of the dependent variable. For treatment 1, the intercept + trt01pn1 estimate. For Treatment 2, the intercept - trt01pn1 estimate.\\\n Log-odds for Treatment 2 = -2.44715 - - 0.19438 = -2.25277, Log-odds for Treatment 1=-2.44715 - 0.19438 = -2.64153\\\n use `contr.sum(2)` to show the contrast you are using and hence how to interpret your estimates.\n\n- exponential (ratio log-odds) = odds ratio. eg. -2.25277/ -2.64153 = 0.85283 Treatment 2 is 0.85 times as likely (eg. 15% less likely) to have weight gain compared to Treatment 1.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nma <- stats::glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.treatment\", sex = \"contr.treatment\")\n)\nsummary(ma)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + \n meal.cal, family = binomial(link = \"logit\"), data = lung, \n contrasts = list(trt01pn = \"contr.treatment\", sex = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -2.6415326 1.5140191 -1.745 0.0810 .\ntrt01pn2 0.3887667 0.3781566 1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex2 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n\n```{.r .cell-code}\ncontr.treatment(2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 2\n1 0\n2 1\n```\n\n\n:::\n\n```{.r .cell-code}\nmb <- stats::glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.sum\", sex = \"contr.treatment\")\n)\nsummary(mb)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + \n meal.cal, family = binomial(link = \"logit\"), data = lung, \n contrasts = list(trt01pn = \"contr.sum\", sex = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -2.4471493 1.4929863 -1.639 0.1012 \ntrt01pn1 -0.1943833 0.1890783 -1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex2 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n\n```{.r .cell-code}\ncontr.sum(2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [,1]\n1 1\n2 -1\n```\n\n\n:::\n:::\n\n\n## Calculation of confidence intervals\n\nUsing the above model, you can output the estimates and confidence intervals using coef() and confint() and exponential back transforming using exp(). NOTE: that there are two types of confidence intervals that you can calculate. Function `confint.default` gives the Wald confidence limits, which is the default option in SAS `PROC LOGISTIC` procedure; whereas `confint` gives the profile-likelihood limits.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# model coefficients summary\nsummary(m1)$coefficients\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error z value Pr(>|z|)\n(Intercept) -2.6415326252 1.5140190574 -1.7447156 0.08103439\ntrt01pn2 0.3887666677 0.3781565596 1.0280574 0.30392281\nage 0.0122549015 0.0211552875 0.5792831 0.56239813\nsex2 0.8321005169 0.3743792762 2.2226137 0.02624186\nph.ecog -0.3763592487 0.2638321918 -1.4265100 0.15372119\nmeal.cal 0.0008499918 0.0004486401 1.8945961 0.05814593\n```\n\n\n:::\n\n```{.r .cell-code}\n# Wald confidence limits\ncbind(est = exp(coef(m1)), exp(confint.default(m1)))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n est 2.5 % 97.5 %\n(Intercept) 0.07125198 0.003664896 1.385263\ntrt01pn2 1.47516031 0.702994248 3.095470\nage 1.01233030 0.971213751 1.055188\nsex2 2.29814096 1.103327506 4.786840\nph.ecog 0.68635572 0.409236994 1.151128\nmeal.cal 1.00085035 0.999970674 1.001731\n```\n\n\n:::\n\n```{.r .cell-code}\n# profile-likelihood limits\ncbind(est = exp(coef(m1)), exp(confint(m1)))\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\nWaiting for profiling to be done...\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n est 2.5 % 97.5 %\n(Intercept) 0.07125198 0.003312288 1.302587\ntrt01pn2 1.47516031 0.696210092 3.085089\nage 1.01233030 0.971670916 1.056194\nsex2 2.29814096 1.107651762 4.836770\nph.ecog 0.68635572 0.405659156 1.147452\nmeal.cal 1.00085035 0.999978126 1.001761\n```\n\n\n:::\n:::\n\n\n# Comparing 2 models\n\nTo compare two logistic models, the `residual deviances` (-2 \\* log likelihoods) are compared against a $\\chi^2$-distribution with degrees of freedom calculated using the difference in the two models' parameters. Below, the only difference is the inclusion/exclusion of age in the model, hence we test using $\\chi^2$ with 1 df. Here testing at the 5% level.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nm2 <- stats::glm(\n wt_catn ~ trt01pn + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.treatment\")\n)\nsummary(m2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + sex + ph.ecog + meal.cal, \n family = binomial(link = \"logit\"), data = lung, contrasts = list(trt01pn = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -1.8350766 0.5810727 -3.158 0.00159 **\ntrt01pn2 0.3681563 0.3761976 0.979 0.32777 \nsex2 0.7919227 0.3674936 2.155 0.03117 * \nph.ecog -0.3312031 0.2527586 -1.310 0.19008 \nmeal.cal 0.0007896 0.0004378 1.803 0.07131 . \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.80 on 165 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 200.8\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n\n```{.r .cell-code}\nanova(m1, m2, test = \"LRT\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nAnalysis of Deviance Table\n\nModel 1: wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal\nModel 2: wt_catn ~ trt01pn + sex + ph.ecog + meal.cal\n Resid. Df Resid. Dev Df Deviance Pr(>Chi)\n1 164 190.46 \n2 165 190.80 -1 -0.33867 0.5606\n```\n\n\n:::\n:::\n\n\nStackexchange [here](https://stats.stackexchange.com/questions/59879/logistic-regression-anova-chi-square-test-vs-significance-of-coefficients-ano) has a good article describing this method and the difference between comparing 2 models using the likelihood ratio tests versus using wald tests and Pr\\>chisq (from the maximum likelihood estimate). Note: `anova(m1, m2, test = \"Chisq\")` and using `test=\"LRT\"` as above are synonymous in this context.\n\n# Predicting likelihood of response for new patients\n\nPredictions from the model for the log-odds of a patient with new data to experience a weight loss are derived using `predict()`:\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# new female, symptomatic but completely ambulatory patient consuming 2500 calories\nnew_pt <- data.frame(\n trt01pn = 1,\n age = 48,\n sex = 2,\n ph.ecog = 1,\n meal.cal = 2500\n)\nnew_pt$trt01pn <- as.factor(new_pt$trt01pn)\nnew_pt$sex <- as.factor(new_pt$sex)\npredict(m1, new_pt, type = \"response\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 1 \n0.628882 \n```\n\n\n:::\n:::\n\n\n# Creating Treatment Contrasts for 2 treatments\n\n## {emmeans}\n\nHere we will use {emmeans} to output the log-odds of weight gain for treatment 1 and treatment 2.\n\nNOTE as per the output, these are on the logit scale, you need to exponentiate to get the odds (or use the type=\"response\" option).\n\nThe treatment comparison can also be output on the log-odds or back transformed scale as shown below.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nm3 <- stats::glm(\n wt_catn ~ trt01pn + age + sex + ph.ecog + meal.cal,\n data = lung,\n family = binomial(link = \"logit\"),\n contrasts = list(trt01pn = \"contr.treatment\")\n)\nsummary(m3)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nCall:\nstats::glm(formula = wt_catn ~ trt01pn + age + sex + ph.ecog + \n meal.cal, family = binomial(link = \"logit\"), data = lung, \n contrasts = list(trt01pn = \"contr.treatment\"))\n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -2.6415326 1.5140191 -1.745 0.0810 .\ntrt01pn2 0.3887667 0.3781566 1.028 0.3039 \nage 0.0122549 0.0211553 0.579 0.5624 \nsex2 0.8321005 0.3743793 2.223 0.0262 *\nph.ecog -0.3763592 0.2638322 -1.427 0.1537 \nmeal.cal 0.0008500 0.0004486 1.895 0.0581 .\n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 202.36 on 169 degrees of freedom\nResidual deviance: 190.46 on 164 degrees of freedom\n (58 observations deleted due to missingness)\nAIC: 202.46\n\nNumber of Fisher Scoring iterations: 4\n```\n\n\n:::\n\n```{.r .cell-code}\ncontr.treatment(2)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n 2\n1 0\n2 1\n```\n\n\n:::\n\n```{.r .cell-code}\n# log-odds for each treatment\nlsm <- emmeans::emmeans(m3, \"trt01pn\")\nlsm\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n trt01pn emmean SE df asymp.LCL asymp.UCL\n 1 -1.034 0.225 Inf -1.47 -0.5935\n 2 -0.645 0.302 Inf -1.24 -0.0529\n\nResults are averaged over the levels of: sex \nResults are given on the logit (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n\n```{.r .cell-code}\n# log-odds ratios (treatment comparison): This does all pairwise comparisons\n# However as seen below, this is TRT 1 - TRT 2\npairs(lsm)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df z.ratio p.value\n trt01pn1 - trt01pn2 -0.389 0.378 Inf -1.028 0.3039\n\nResults are averaged over the levels of: sex \nResults are given on the log odds ratio (not the response) scale. \n```\n\n\n:::\n\n```{.r .cell-code}\n# the below creates tests and CI's prior to back transformation (ratios of geometric means)\npairs(lsm, type = \"response\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast odds.ratio SE df null z.ratio p.value\n trt01pn1 / trt01pn2 0.678 0.256 Inf 1 -1.028 0.3039\n\nResults are averaged over the levels of: sex \nTests are performed on the log odds ratio scale \n```\n\n\n:::\n\n```{.r .cell-code}\n# see coefficients of the linear functions\ncoef(pairs(lsm))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n trt01pn c.1\n1 1 1\n2 2 -1\n```\n\n\n:::\n\n```{.r .cell-code}\n# Output treatment contrasts 2 vs 1 and 95% CIs, the type=\"response\" option back transforms the results\ntrtdiff <- contrast(lsm, \"poly\")\ntrtdiff\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df z.ratio p.value\n linear 0.389 0.378 Inf 1.028 0.3039\n\nResults are averaged over the levels of: sex \nResults are given on the log odds ratio (not the response) scale. \n```\n\n\n:::\n\n```{.r .cell-code}\nconfint(trtdiff, type = \"response\")\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast odds.ratio SE df asymp.LCL asymp.UCL\n linear 1.48 0.558 Inf 0.703 3.1\n\nResults are averaged over the levels of: sex \nConfidence level used: 0.95 \nIntervals are back-transformed from the log odds ratio scale \n```\n\n\n:::\n:::\n\n\nIn Summary: Treatment 2 is on average 1.48 times as likely to have weight gain compared to treatment 1, however this is not statistically significant (95% Confidence interval = 0.703-3.100, p-value= 0.3039).\n\n# Creating Treatment Contrasts for 2 or more treatments\n\n{emmeans} can also be used to do specific contrasts, instead of investigating treatment (active vs placebo), suppose we now want to look at 2 dose groups vs placebo.\n\nWe have a 3 level treatment variable (dose_id), where 1=10mg, 2=20mg doses for active treatment and 3= placebo which is 0mg. We want to test the null hypothesis that 0.5\\**dose10mg + 0.5*\\*dose20mg - placebo = 0. That there is not difference between the doses.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlung2 <- lung |>\n mutate(dose_id2 = as.factor(lung$dose_id))\n\nm3 <- stats::glm(\n wt_catn ~ dose_id2 + age + sex + ph.ecog + meal.cal,\n data = lung2,\n family = binomial(link = \"logit\"),\n contrasts = list(dose_id2 = \"contr.treatment\")\n)\n\n# log-odds for each treatment\nlsm3 <- emmeans::emmeans(m3, \"dose_id2\")\nlsm3\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n dose_id2 emmean SE df asymp.LCL asymp.UCL\n 1 -0.903 0.296 Inf -1.48 -0.3228\n 2 -1.201 0.348 Inf -1.88 -0.5201\n 3 -0.643 0.302 Inf -1.23 -0.0518\n\nResults are averaged over the levels of: sex \nResults are given on the logit (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n\n```{.r .cell-code}\ncontrast(lsm3, list(AveDose_vs_pbo = c(0.5, 0.5, -1)))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df z.ratio p.value\n AveDose_vs_pbo -0.41 0.38 Inf -1.078 0.2813\n\nResults are averaged over the levels of: sex \nResults are given on the log odds ratio (not the response) scale. \n```\n\n\n:::\n\n```{.r .cell-code}\nconfint(contrast(lsm3, list(AveDose_vs_pbo = c(0.5, 0.5, -1))))\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n contrast estimate SE df asymp.LCL asymp.UCL\n AveDose_vs_pbo -0.41 0.38 Inf -1.15 0.335\n\nResults are averaged over the levels of: sex \nResults are given on the log odds ratio (not the response) scale. \nConfidence level used: 0.95 \n```\n\n\n:::\n:::\n\n\nHere we found that on average there is -0.41 times the risk of weight gain on active vs placebo but that this is not statisticallly significantly different (95% CI -1.15 to 0.335, p-value = 0.2813).\n\nSee the emmeans vignette on creating bespoke contrasts [here](Comparisons%20and%20contrasts%20in%20emmeans).\n\n## {gmodels}\n\n{gmodels} is an alternative package to create contrasts instead of \\, you can use the `fit.contrast()` function from the `gmodels` package. The same result is obtained as \\.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlung2 <- lung |>\n mutate(dose_id2 = as.factor(lung$dose_id))\n\nm3 <- stats::glm(\n wt_catn ~ dose_id2 + age + sex + ph.ecog + meal.cal,\n data = lung2,\n family = binomial(link = \"logit\"),\n contrasts = list(dose_id2 = \"contr.treatment\")\n)\n\ngmodels::fit.contrast(m3, 'dose_id2', c(0.5, 0.5, -1), conf.int = 0.95)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Estimate Std. Error z value Pr(>|z|) lower CI\ndose_id2 c=( 0.5 0.5 -1 ) -0.4096323 0.3801683 -1.077502 0.2812558 -1.160322\n upper CI\ndose_id2 c=( 0.5 0.5 -1 ) 0.3410574\n```\n\n\n:::\n:::\n\n\n# Reference\n\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P cli 3.6.5 2025-04-23 [?] RSPM (R 4.5.0)\n P dplyr * 1.2.0 2026-02-03 [?] RSPM (R 4.5.0)\n P emmeans * 2.0.1 2025-12-16 [?] RSPM (R 4.5.0)\n P estimability 1.5.1 2024-05-12 [?] RSPM (R 4.5.0)\n P gdata 3.0.1 2024-10-22 [?] RSPM (R 4.5.0)\n P generics 0.1.4 2025-05-09 [?] RSPM (R 4.5.0)\n P glue 1.8.0 2024-09-30 [?] RSPM (R 4.5.0)\n P gmodels * 2.19.1 2024-03-06 [?] RSPM (R 4.5.0)\n P gtools 3.9.5 2023-11-20 [?] RSPM (R 4.5.0)\n P lifecycle 1.0.5 2026-01-08 [?] RSPM (R 4.5.0)\n P magrittr 2.0.4 2025-09-12 [?] RSPM (R 4.5.0)\n MASS 7.3-65 2025-02-28 [2] CRAN (R 4.5.2)\n P mvtnorm 1.3-3 2025-01-10 [?] RSPM (R 4.5.0)\n numDeriv 2016.8-1.1 2019-06-06 [1] RSPM (R 4.5.0)\n P pillar 1.11.1 2025-09-17 [?] RSPM (R 4.5.0)\n P pkgconfig 2.0.3 2019-09-22 [?] RSPM (R 4.5.0)\n P R6 2.6.1 2025-02-15 [?] RSPM (R 4.5.0)\n P rlang 1.1.7 2026-01-09 [?] RSPM (R 4.5.0)\n P tibble 3.3.1 2026-01-11 [?] RSPM (R 4.5.0)\n P tidyselect 1.2.1 2024-03-11 [?] RSPM (R 4.5.0)\n utf8 1.2.6 2025-06-08 [1] RSPM (R 4.5.0)\n P vctrs 0.7.1 2026-01-23 [?] RSPM (R 4.5.0)\n withr 3.0.2 2024-10-28 [1] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/R/marginal_homogeneity_tests/execute-results/html.json b/_freeze/R/marginal_homogeneity_tests/execute-results/html.json index ea82e2577..d21db9fa0 100644 --- a/_freeze/R/marginal_homogeneity_tests/execute-results/html.json +++ b/_freeze/R/marginal_homogeneity_tests/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "95381b7b1b29c8f4dfbee846e4825255", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Marginal Homogeneity Tests\"\n---\n\nThis page is solely based on **coin** package documentation including data samples which are generated inline.\n\n`coin::mh_test()` provides the McNemar test, the Cochran Q test, the Stuart(-Maxwell) test and the Madansky test of interchangeability. A general description of these methods is given by Agresti (2002).\n\nThe null hypothesis of marginal homogeneity is tested. If formula interface is used, the response variable and the measurement conditions are given by `y` and `x`, respectively, and `block` is a factor where each level corresponds to exactly one subject with repeated measurements: `coin::mh_test(y ~ x | block, data, subset = NULL, ...)`. We can also directly pass an object of class `\"table\"`.\n\n`coin::mh_test()` computes different tests depending on `x` and `y`:\n\n- McNemar test (McNemar, 1947) when both `y` and `x` are binary factors;\n\n- Cochran Q test (Cochran, 1950) when `y` is a binary factor and `x` is a factor with an arbitrary number of levels;\n\n- Stuart-Maxwell test (Stuart, 1955; Maxwell, 1970) when `y` is a factor with an arbitrary number of levels and `x` is a binary factor;\n\n- Madansky test of interchangeability (Madansky, 1963), which implies marginal homogeneity, when both `y` and `x` are factors with an arbitrary number of levels.\n\nThe conditional null distribution of the test statistic is used to obtain p-values and an asymptotic approximation of the exact distribution is used by default (`distribution = \"asymptotic\"`). Alternatively, the distribution can be approximated via Monte Carlo resampling or computed exactly for univariate two-sample problems (McNemar test) by setting distribution to `\"approximate\"` or `\"exact\"`, respectively.\n\n## McNemar test\n\nFor more information on the McNemar see the [McNemar’s test](https://psiaims.github.io/CAMIS/R/r_mcnemar.html) page.\n\n\n## Cochran Q test\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## Effectiveness of different media for the growth of diphtheria\n## Cochran (1950, Tab. 2)\ncases <- c(4, 2, 3, 1, 59)\nn <- sum(cases)\ncochran <- data.frame(\n diphtheria = factor(\n unlist(rep(\n list(\n c(1, 1, 1, 1),\n c(1, 1, 0, 1),\n c(0, 1, 1, 1),\n c(0, 1, 0, 1),\n c(0, 0, 0, 0)\n ),\n cases\n ))\n ),\n media = factor(rep(LETTERS[1:4], n)),\n case = factor(rep(seq_len(n), each = 4))\n)\n\nhead(cochran)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n diphtheria media case\n1 1 A 1\n2 1 B 1\n3 1 C 1\n4 1 D 1\n5 1 A 2\n6 1 B 2\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Cochran Q test (Cochran, 1950, p. 260)\ncoin::mh_test(\n diphtheria ~ media | case,\n data = cochran\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test\n\ndata: diphtheria by media (A, B, C, D) \n\t stratified by case\nchi-squared = 8.0526, df = 3, p-value = 0.04494\n```\n\n\n:::\n\n```{.r .cell-code}\n## Approximative Cochran Q test\nmt <- coin::mh_test(\n diphtheria ~ media | case,\n data = cochran,\n distribution = coin::approximate(nresample = 10000)\n)\ncoin::pvalue(mt) # standard p-value\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.0512\n99 percent confidence interval:\n 0.04568760 0.05714443 \n```\n\n\n:::\n\n```{.r .cell-code}\ncoin::midpvalue(mt) # mid-p-value\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.0429\n99 percent confidence interval:\n 0.03789480 0.04833997 \n```\n\n\n:::\n\n```{.r .cell-code}\ncoin::pvalue_interval(mt) # p-value interval\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n p_0 p_1 \n0.0346 0.0512 \n```\n\n\n:::\n\n```{.r .cell-code}\ncoin::size(mt, alpha = 0.05) # test size at alpha = 0.05 using the p-value\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.0346\n```\n\n\n:::\n:::\n\n\n## Stuart-Maxwell test\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## Opinions on Pre- and Extramarital Sex\n## Agresti (2002, p. 421)\nopinions <- c(\n \"Always wrong\",\n \"Almost always wrong\",\n \"Wrong only sometimes\",\n \"Not wrong at all\"\n)\n# fmt: skip\nPreExSex <- matrix(\n c(144, 33, 84, 126,\n 2, 4, 14, 29,\n 0, 2, 6, 25,\n 0, 0, 1, 5),\n nrow = 4,\n dimnames = list(\n \"Premarital Sex\" = opinions,\n \"Extramarital Sex\" = opinions\n )\n)\nPreExSex <- as.table(PreExSex)\n\nPreExSex\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Extramarital Sex\nPremarital Sex Always wrong Almost always wrong Wrong only sometimes\n Always wrong 144 2 0\n Almost always wrong 33 4 2\n Wrong only sometimes 84 14 6\n Not wrong at all 126 29 25\n Extramarital Sex\nPremarital Sex Not wrong at all\n Always wrong 0\n Almost always wrong 0\n Wrong only sometimes 1\n Not wrong at all 5\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Stuart test\ncoin::mh_test(PreExSex)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test\n\ndata: response by\n\t conditions (Premarital.Sex, Extramarital.Sex) \n\t stratified by block\nchi-squared = 271.92, df = 3, p-value < 2.2e-16\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Stuart-Birch test\n## Note: response as ordinal\ncoin::mh_test(\n PreExSex,\n scores = list(response = 1:length(opinions))\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test for Ordered Data\n\ndata: response (ordered) by\n\t conditions (Premarital.Sex, Extramarital.Sex) \n\t stratified by block\nZ = 16.454, p-value < 2.2e-16\nalternative hypothesis: two.sided\n```\n\n\n:::\n:::\n\n\n## Madansky test of interchangeability\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## Vote intention\n## Madansky (1963, pp. 107-108)\n# fmt: skip\nvote <- array(\n c(120, 1, 8, 2, 2, 1, 2, 1, 7,\n 6, 2, 1, 1, 103, 5, 1, 4, 8,\n 20, 3, 31, 1, 6, 30, 2, 1, 81),\n dim = c(3, 3, 3),\n dimnames = list(\n \"July\" = c(\"Republican\", \"Democratic\", \"Uncertain\"),\n \"August\" = c(\"Republican\", \"Democratic\", \"Uncertain\"),\n \"June\" = c(\"Republican\", \"Democratic\", \"Uncertain\")\n )\n)\nvote <- as.table(vote)\n\nvote\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n, , June = Republican\n\n August\nJuly Republican Democratic Uncertain\n Republican 120 2 2\n Democratic 1 2 1\n Uncertain 8 1 7\n\n, , June = Democratic\n\n August\nJuly Republican Democratic Uncertain\n Republican 6 1 1\n Democratic 2 103 4\n Uncertain 1 5 8\n\n, , June = Uncertain\n\n August\nJuly Republican Democratic Uncertain\n Republican 20 1 2\n Democratic 3 6 1\n Uncertain 31 30 81\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Madansky test (Q = 70.77)\ncoin::mh_test(vote)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test\n\ndata: response by\n\t conditions (July, August, June) \n\t stratified by block\nchi-squared = 70.763, df = 4, p-value = 1.565e-14\n```\n\n\n:::\n\n```{.r .cell-code}\n## Cross-over study\n## http://www.nesug.org/proceedings/nesug00/st/st9005.pdf (link is dead now)\n# fmt: skip\ndysmenorrhea <- array(\n c(6, 2, 1, 3, 1, 0, 1, 2, 1,\n 4, 3, 0, 13, 3, 0, 8, 1, 1,\n 5, 2, 2, 10, 1, 0, 14, 2, 0),\n dim = c(3, 3, 3),\n dimnames = list(\n \"Placebo\" = c(\"None\", \"Moderate\", \"Complete\"),\n \"Low dose\" = c(\"None\", \"Moderate\", \"Complete\"),\n \"High dose\" = c(\"None\", \"Moderate\", \"Complete\")\n )\n)\ndysmenorrhea <- as.table(dysmenorrhea)\n\ndysmenorrhea\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n, , High dose = None\n\n Low dose\nPlacebo None Moderate Complete\n None 6 3 1\n Moderate 2 1 2\n Complete 1 0 1\n\n, , High dose = Moderate\n\n Low dose\nPlacebo None Moderate Complete\n None 4 13 8\n Moderate 3 3 1\n Complete 0 0 1\n\n, , High dose = Complete\n\n Low dose\nPlacebo None Moderate Complete\n None 5 10 14\n Moderate 2 1 2\n Complete 2 0 0\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Madansky-Birch test (Q = 53.76)\n## Note: response as ordinal\ncoin::mh_test(\n dysmenorrhea,\n scores = list(response = 1:3)\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test for Ordered Data\n\ndata: response (ordered) by\n\t conditions (Placebo, Low.dose, High.dose) \n\t stratified by block\nchi-squared = 53.762, df = 2, p-value = 2.117e-12\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Madansky-Birch test (Q = 47.29)\n## Note: response and measurement conditions as ordinal\ncoin::mh_test(\n dysmenorrhea,\n scores = list(response = 1:3, conditions = 1:3)\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test for Ordered Data\n\ndata: response (ordered) by\n\t conditions (Placebo < Low.dose < High.dose) \n\t stratified by block\nZ = 6.8764, p-value = 6.138e-12\nalternative hypothesis: two.sided\n```\n\n\n:::\n:::\n\n\n## Reference\n\nHothorn T, Hornik K, van de Wiel MA, Zeileis A (2006). A Lego system for conditional inference. The American Statistician, 60 (3), 257-263. doi:10.1198/000313006X118430 \n\nAgresti, A. (2002). Categorical Data Analysis, Second Edition. Hoboken, New Jersey: John Wiley & Sons.\n\nBirch, M. W. (1965). The detection of partial association, II: The general case. Journal of the Royal Statistical Society B 27(1), 111–124. doi:10.1111/j.2517-6161.1965.tb00593.x\n\nCochran, W. G. (1950). The comparison of percentages in matched samples. Biometrika 37(3/4), 256–266. doi:10.1093/biomet/37.3-4.256\n\nMadansky, A. (1963). Tests of homogeneity for correlated samples. Journal of the American Statistical Association 58(301), 97–119. doi:10.1080/01621459.1963.10500835\n\nMaxwell, A. E. (1970). Comparing the classification of subjects by two independent judges. British Journal of Psychiatry 116(535), 651–655. doi:10.1192/bjp.116.535.651\n\nMcNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157. doi:10.1007/BF02295996\n\nStuart, A. (1955). A test for homogeneity of the marginal distributions in a two-way classification. Biometrika 42(3/4), 412–416. doi:10.1093/biomet/42.3-4.412\n\nWhite, A. A., Landis, J. R. and Cooper, M. M. (1982). A note on the equivalence of several marginal homogeneity test criteria for categorical data. International Statistical Review 50(1), 27–34. doi:10.2307/1402457\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-23\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n codetools 0.2-20 2024-03-31 [2] CRAN (R 4.5.2)\n P coin 1.4-3 2023-09-27 [?] RSPM\n lattice 0.22-7 2025-04-02 [2] CRAN (R 4.5.2)\n P libcoin 1.0-10 2023-09-27 [?] RSPM\n MASS 7.3-65 2025-02-28 [2] CRAN (R 4.5.2)\n Matrix 1.7-4 2025-08-28 [2] CRAN (R 4.5.2)\n P matrixStats 1.5.0 2025-01-07 [?] RSPM\n P modeltools 0.2-24 2025-05-02 [?] RSPM\n P multcomp 1.4-29 2025-10-20 [?] RSPM\n P mvtnorm 1.3-3 2025-01-10 [?] RSPM\n P sandwich 3.1-1 2024-09-15 [?] RSPM\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n P TH.data 1.1-5 2025-11-17 [?] RSPM\n P zoo 1.8-15 2025-12-15 [?] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n", + "markdown": "---\ntitle: \"Marginal Homogeneity Tests\"\n---\n\nThis page is solely based on **coin** package documentation including data samples which are generated inline.\n\n`coin::mh_test()` provides the McNemar test, the Cochran Q test, the Stuart(-Maxwell) test and the Madansky test of interchangeability. A general description of these methods is given by Agresti (2002).\n\nThe null hypothesis of marginal homogeneity is tested. If formula interface is used, the response variable and the measurement conditions are given by `y` and `x`, respectively, and `block` is a factor where each level corresponds to exactly one subject with repeated measurements: `coin::mh_test(y ~ x | block, data, subset = NULL, ...)`. We can also directly pass an object of class `\"table\"`.\n\n`coin::mh_test()` computes different tests depending on `x` and `y`:\n\n- McNemar test (McNemar, 1947) when both `y` and `x` are binary factors;\n\n- Cochran Q test (Cochran, 1950) when `y` is a binary factor and `x` is a factor with an arbitrary number of levels;\n\n- Stuart-Maxwell test (Stuart, 1955; Maxwell, 1970) when `y` is a factor with an arbitrary number of levels and `x` is a binary factor;\n\n- Madansky test of interchangeability (Madansky, 1963), which implies marginal homogeneity, when both `y` and `x` are factors with an arbitrary number of levels.\n\nThe conditional null distribution of the test statistic is used to obtain p-values and an asymptotic approximation of the exact distribution is used by default (`distribution = \"asymptotic\"`). Alternatively, the distribution can be approximated via Monte Carlo resampling or computed exactly for univariate two-sample problems (McNemar test) by setting distribution to `\"approximate\"` or `\"exact\"`, respectively.\n\n## McNemar test\n\nFor more information on the McNemar see the [McNemar’s test](https://psiaims.github.io/CAMIS/R/r_mcnemar.html) page.\n\n\n## Cochran Q test\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## Effectiveness of different media for the growth of diphtheria\n## Cochran (1950, Tab. 2)\ncases <- c(4, 2, 3, 1, 59)\nn <- sum(cases)\ncochran <- data.frame(\n diphtheria = factor(\n unlist(rep(\n list(\n c(1, 1, 1, 1),\n c(1, 1, 0, 1),\n c(0, 1, 1, 1),\n c(0, 1, 0, 1),\n c(0, 0, 0, 0)\n ),\n cases\n ))\n ),\n media = factor(rep(LETTERS[1:4], n)),\n case = factor(rep(seq_len(n), each = 4))\n)\n\nhead(cochran)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n diphtheria media case\n1 1 A 1\n2 1 B 1\n3 1 C 1\n4 1 D 1\n5 1 A 2\n6 1 B 2\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Cochran Q test (Cochran, 1950, p. 260)\ncoin::mh_test(\n diphtheria ~ media | case,\n data = cochran\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test\n\ndata: diphtheria by media (A, B, C, D) \n\t stratified by case\nchi-squared = 8.0526, df = 3, p-value = 0.04494\n```\n\n\n:::\n\n```{.r .cell-code}\n## Approximative Cochran Q test\nmt <- coin::mh_test(\n diphtheria ~ media | case,\n data = cochran,\n distribution = coin::approximate(nresample = 10000)\n)\ncoin::pvalue(mt) # standard p-value\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.0527\n99 percent confidence interval:\n 0.04710889 0.05872162 \n```\n\n\n:::\n\n```{.r .cell-code}\ncoin::midpvalue(mt) # mid-p-value\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.04285\n99 percent confidence interval:\n 0.03780066 0.04823421 \n```\n\n\n:::\n\n```{.r .cell-code}\ncoin::pvalue_interval(mt) # p-value interval\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n p_0 p_1 \n0.0330 0.0527 \n```\n\n\n:::\n\n```{.r .cell-code}\ncoin::size(mt, alpha = 0.05) # test size at alpha = 0.05 using the p-value\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 0.033\n```\n\n\n:::\n:::\n\n\n## Stuart-Maxwell test\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## Opinions on Pre- and Extramarital Sex\n## Agresti (2002, p. 421)\nopinions <- c(\n \"Always wrong\",\n \"Almost always wrong\",\n \"Wrong only sometimes\",\n \"Not wrong at all\"\n)\n# fmt: skip\nPreExSex <- matrix(\n c(144, 33, 84, 126,\n 2, 4, 14, 29,\n 0, 2, 6, 25,\n 0, 0, 1, 5),\n nrow = 4,\n dimnames = list(\n \"Premarital Sex\" = opinions,\n \"Extramarital Sex\" = opinions\n )\n)\nPreExSex <- as.table(PreExSex)\n\nPreExSex\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n Extramarital Sex\nPremarital Sex Always wrong Almost always wrong Wrong only sometimes\n Always wrong 144 2 0\n Almost always wrong 33 4 2\n Wrong only sometimes 84 14 6\n Not wrong at all 126 29 25\n Extramarital Sex\nPremarital Sex Not wrong at all\n Always wrong 0\n Almost always wrong 0\n Wrong only sometimes 1\n Not wrong at all 5\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Stuart test\ncoin::mh_test(PreExSex)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test\n\ndata: response by\n\t conditions (Premarital.Sex, Extramarital.Sex) \n\t stratified by block\nchi-squared = 271.92, df = 3, p-value < 2.2e-16\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Stuart-Birch test\n## Note: response as ordinal\ncoin::mh_test(\n PreExSex,\n scores = list(response = 1:length(opinions))\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test for Ordered Data\n\ndata: response (ordered) by\n\t conditions (Premarital.Sex, Extramarital.Sex) \n\t stratified by block\nZ = 16.454, p-value < 2.2e-16\nalternative hypothesis: two.sided\n```\n\n\n:::\n:::\n\n\n## Madansky test of interchangeability\n\n\n::: {.cell}\n\n```{.r .cell-code}\n## Vote intention\n## Madansky (1963, pp. 107-108)\n# fmt: skip\nvote <- array(\n c(120, 1, 8, 2, 2, 1, 2, 1, 7,\n 6, 2, 1, 1, 103, 5, 1, 4, 8,\n 20, 3, 31, 1, 6, 30, 2, 1, 81),\n dim = c(3, 3, 3),\n dimnames = list(\n \"July\" = c(\"Republican\", \"Democratic\", \"Uncertain\"),\n \"August\" = c(\"Republican\", \"Democratic\", \"Uncertain\"),\n \"June\" = c(\"Republican\", \"Democratic\", \"Uncertain\")\n )\n)\nvote <- as.table(vote)\n\nvote\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n, , June = Republican\n\n August\nJuly Republican Democratic Uncertain\n Republican 120 2 2\n Democratic 1 2 1\n Uncertain 8 1 7\n\n, , June = Democratic\n\n August\nJuly Republican Democratic Uncertain\n Republican 6 1 1\n Democratic 2 103 4\n Uncertain 1 5 8\n\n, , June = Uncertain\n\n August\nJuly Republican Democratic Uncertain\n Republican 20 1 2\n Democratic 3 6 1\n Uncertain 31 30 81\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Madansky test (Q = 70.77)\ncoin::mh_test(vote)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test\n\ndata: response by\n\t conditions (July, August, June) \n\t stratified by block\nchi-squared = 70.763, df = 4, p-value = 1.565e-14\n```\n\n\n:::\n\n```{.r .cell-code}\n## Cross-over study\n## http://www.nesug.org/proceedings/nesug00/st/st9005.pdf (link is dead now)\n# fmt: skip\ndysmenorrhea <- array(\n c(6, 2, 1, 3, 1, 0, 1, 2, 1,\n 4, 3, 0, 13, 3, 0, 8, 1, 1,\n 5, 2, 2, 10, 1, 0, 14, 2, 0),\n dim = c(3, 3, 3),\n dimnames = list(\n \"Placebo\" = c(\"None\", \"Moderate\", \"Complete\"),\n \"Low dose\" = c(\"None\", \"Moderate\", \"Complete\"),\n \"High dose\" = c(\"None\", \"Moderate\", \"Complete\")\n )\n)\ndysmenorrhea <- as.table(dysmenorrhea)\n\ndysmenorrhea\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n, , High dose = None\n\n Low dose\nPlacebo None Moderate Complete\n None 6 3 1\n Moderate 2 1 2\n Complete 1 0 1\n\n, , High dose = Moderate\n\n Low dose\nPlacebo None Moderate Complete\n None 4 13 8\n Moderate 3 3 1\n Complete 0 0 1\n\n, , High dose = Complete\n\n Low dose\nPlacebo None Moderate Complete\n None 5 10 14\n Moderate 2 1 2\n Complete 2 0 0\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Madansky-Birch test (Q = 53.76)\n## Note: response as ordinal\ncoin::mh_test(\n dysmenorrhea,\n scores = list(response = 1:3)\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test for Ordered Data\n\ndata: response (ordered) by\n\t conditions (Placebo, Low.dose, High.dose) \n\t stratified by block\nchi-squared = 53.762, df = 2, p-value = 2.117e-12\n```\n\n\n:::\n\n```{.r .cell-code}\n## Asymptotic Madansky-Birch test (Q = 47.29)\n## Note: response and measurement conditions as ordinal\ncoin::mh_test(\n dysmenorrhea,\n scores = list(response = 1:3, conditions = 1:3)\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\n\tAsymptotic Marginal Homogeneity Test for Ordered Data\n\ndata: response (ordered) by\n\t conditions (Placebo < Low.dose < High.dose) \n\t stratified by block\nZ = 6.8764, p-value = 6.138e-12\nalternative hypothesis: two.sided\n```\n\n\n:::\n:::\n\n\n## Reference\n\nHothorn T, Hornik K, van de Wiel MA, Zeileis A (2006). A Lego system for conditional inference. The American Statistician, 60 (3), 257-263. doi:10.1198/000313006X118430 \n\nAgresti, A. (2002). Categorical Data Analysis, Second Edition. Hoboken, New Jersey: John Wiley & Sons.\n\nBirch, M. W. (1965). The detection of partial association, II: The general case. Journal of the Royal Statistical Society B 27(1), 111–124. doi:10.1111/j.2517-6161.1965.tb00593.x\n\nCochran, W. G. (1950). The comparison of percentages in matched samples. Biometrika 37(3/4), 256–266. doi:10.1093/biomet/37.3-4.256\n\nMadansky, A. (1963). Tests of homogeneity for correlated samples. Journal of the American Statistical Association 58(301), 97–119. doi:10.1080/01621459.1963.10500835\n\nMaxwell, A. E. (1970). Comparing the classification of subjects by two independent judges. British Journal of Psychiatry 116(535), 651–655. doi:10.1192/bjp.116.535.651\n\nMcNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157. doi:10.1007/BF02295996\n\nStuart, A. (1955). A test for homogeneity of the marginal distributions in a two-way classification. Biometrika 42(3/4), 412–416. doi:10.1093/biomet/42.3-4.412\n\nWhite, A. A., Landis, J. R. and Cooper, M. M. (1982). A note on the equivalence of several marginal homogeneity test criteria for categorical data. International Statistical Review 50(1), 27–34. doi:10.2307/1402457\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n codetools 0.2-20 2024-03-31 [2] CRAN (R 4.5.2)\n P coin 1.4-3 2023-09-27 [?] RSPM (R 4.5.0)\n lattice 0.22-7 2025-04-02 [2] CRAN (R 4.5.2)\n P libcoin 1.0-10 2023-09-27 [?] RSPM (R 4.5.0)\n MASS 7.3-65 2025-02-28 [2] CRAN (R 4.5.2)\n Matrix 1.7-4 2025-08-28 [2] CRAN (R 4.5.2)\n P matrixStats 1.5.0 2025-01-07 [?] RSPM (R 4.5.0)\n P modeltools 0.2-24 2025-05-02 [?] RSPM (R 4.5.0)\n P multcomp 1.4-29 2025-10-20 [?] RSPM (R 4.5.0)\n P mvtnorm 1.3-3 2025-01-10 [?] RSPM (R 4.5.0)\n P sandwich 3.1-1 2024-09-15 [?] RSPM (R 4.5.0)\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n P TH.data 1.1-5 2025-11-17 [?] RSPM (R 4.5.0)\n P zoo 1.8-15 2025-12-15 [?] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n", "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" diff --git a/_freeze/R/rbmi_continuous_joint/execute-results/html.json b/_freeze/R/rbmi_continuous_joint/execute-results/html.json index 25c2983e4..0c93310c9 100644 --- a/_freeze/R/rbmi_continuous_joint/execute-results/html.json +++ b/_freeze/R/rbmi_continuous_joint/execute-results/html.json @@ -2,7 +2,7 @@ "hash": "78c3df12dbad314967a5905b5d4100a6", "result": { "engine": "knitr", - "markdown": "---\ntitle: \"Reference-Based Multiple Imputation (joint modelling): Continuous Data\"\n---\n\n## Libraries\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# General\nlibrary(dplyr)\nlibrary(tidyr)\nlibrary(gt)\nlibrary(labelled)\n\n# Methodlolgy specific\nlibrary(mmrm)\nlibrary(emmeans)\nlibrary(rbmi)\nlibrary(mice) # only used md.pattern()\n```\n:::\n\n\n## Reference-based multiple imputation (rbmi)\n\n### Methodology introduction\nReference-based multiple imputation methods have become popular for handling missing data, as well as for conducting sensitivity analyses, in randomized clinical trials. In the context of a repeatedly measured continuous endpoint assuming a multivariate normal model, [Carpenter et al. (2013)](https://www.tandfonline.com/doi/full/10.1080/10543406.2013.834911) proposed a framework to extend the usual MAR-based MI approach by postulating assumptions about the joint distribution of pre- and post-deviation data. Under this framework, one makes qualitative assumptions about how individuals’ missing outcomes relate to those observed in relevant groups in the trial, based on plausible clinical scenarios. Statistical analysis then proceeds using the method of multiple imputation ([Rubin 1976](https://doi.org/10.1093/biomet/63.3.581), [Rubin 1987]((https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316696))).\n\nIn general, multiple imputation of a repeatedly measured continuous outcome can be done via 2 computational routes ([Roger 2022](https://baselbiometrics.github.io/home/docs/talks/20221208/5_JamesRoger%2020121118.pdf)):\n\n 1. Stepwise: split problem into separate imputations of data at each visit\n \n + requires monotone missingness, such as missingness due to withdrawal\n \n + conditions on the imputed values at previous visit\n \n + Bayesian linear regression problem is much simpler with monotone missing, as one can sample directly using conjugate priors\n\n 2. One-step approach (joint modelling): Fit a Bayesian full multivariate normal repeated measures model using MCMC and then draw a sample.\n\nHere, we illustrate reference-based multiple imputation of a continuous outcome measured repeatedly via the so-called one-step approach.\n\n### rbmi package\nThe `rbmi` package [Gower-Page et al. (2022)](https://joss.theoj.org/papers/10.21105/joss.04251) will be used for the one-step approach of the reference-based multiple imputation using R. The package implements standard and reference based multiple imputation methods for continuous longitudinal endpoints . In particular, this package supports deterministic conditional mean imputation and jackknifing as described in [Wolbers et al. (2022)](https://onlinelibrary.wiley.com/doi/full/10.1002/pst.2234), convential MI based on Bayesian posterior draws as described in [Carpenter et al. (2013)](https://www.tandfonline.com/doi/full/10.1080/10543406.2013.834911), and bootstrapped maximum likelihood imputation as described in [von Hippel and Bartlett (2021)](https://doi.org/10.1214/20-STS793).\n\nThe following standard and reference-based multiple imputation approaches will be illustrated here: \n \n * MAR (Missing At Random)\n \n * CIR (Copy Increment from Reference)\n \n * J2R (Jump to Reference)\n \n * CR (Copy Reference)\n\n\n## Data used\nA publicly available example [dataset](https://r-packages.io/datasets/antidepressant_data) from an antidepressant clinical trial of an active drug versus placebo is used. Overall, data of 172 patients is available with 88 patients receiving placebo and 84 receiving active drug. This data is also used in the `rbmi` package [quickstart vignette](https://cran.r-project.org/web/packages/rbmi/vignettes/quickstart.html).\n\nThe relevant endpoint is the Hamilton 17-item depression rating scale (HAMD17) which was assessed at baseline and at weeks 1, 2, 4, and 6 (visits 4-7). Study drug discontinuation occurred in 24% (20/84) of subjects from the active drug and 26% (23/88) of subjects from placebo. All data after study drug discontinuation are missing. \n\n\n::: {.cell}\n\n```{.r .cell-code}\ndata(\"antidepressant_data\")\ndat <- antidepressant_data |>\n dplyr::select(\n PATIENT,\n GENDER,\n THERAPY,\n RELDAYS,\n VISIT,\n BASVAL,\n HAMDTL17,\n CHANGE\n ) |>\n dplyr::mutate(THERAPY = factor(THERAPY, levels = c(\"PLACEBO\", \"DRUG\"))) |>\n labelled::remove_labels()\n\ngt(head(dat, n = 10))\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n \n
PATIENTGENDERTHERAPYRELDAYSVISITBASVALHAMDTL17CHANGE
1503FDRUG743221-11
1503FDRUG1453220-12
1503FDRUG2863219-13
1503FDRUG4273217-15
1507FPLACEBO741411-3
1507FPLACEBO15514140
1507FPLACEBO296149-5
1507FPLACEBO427145-9
1509FDRUG742120-1
1509FDRUG1452118-3
\n
\n```\n\n:::\n:::\n\n\nThe number of patients per visit and arm are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat |>\n group_by(VISIT, THERAPY) |>\n dplyr::summarise(N = n())\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\n`summarise()` has regrouped the output.\nℹ Summaries were computed grouped by VISIT and THERAPY.\nℹ Output is grouped by VISIT.\nℹ Use `summarise(.groups = \"drop_last\")` to silence this message.\nℹ Use `summarise(.by = c(VISIT, THERAPY))` for per-operation grouping\n (`?dplyr::dplyr_by`) instead.\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 8 × 3\n# Groups: VISIT [4]\n VISIT THERAPY N\n \n1 4 PLACEBO 88\n2 4 DRUG 84\n3 5 PLACEBO 81\n4 5 DRUG 77\n5 6 PLACEBO 76\n6 6 DRUG 73\n7 7 PLACEBO 65\n8 7 DRUG 64\n```\n\n\n:::\n:::\n\n\nThe mean change from baseline of the endpoint (Hamilton 17-item depression rating scale, HAMD17) per visit per treatment group using only the complete cases are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat |>\n group_by(VISIT, THERAPY) |>\n dplyr::summarise(N = n(), MEAN = mean(CHANGE))\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\n`summarise()` has regrouped the output.\nℹ Summaries were computed grouped by VISIT and THERAPY.\nℹ Output is grouped by VISIT.\nℹ Use `summarise(.groups = \"drop_last\")` to silence this message.\nℹ Use `summarise(.by = c(VISIT, THERAPY))` for per-operation grouping\n (`?dplyr::dplyr_by`) instead.\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 8 × 4\n# Groups: VISIT [4]\n VISIT THERAPY N MEAN\n \n1 4 PLACEBO 88 -1.51\n2 4 DRUG 84 -1.82\n3 5 PLACEBO 81 -2.70\n4 5 DRUG 77 -4.71\n5 6 PLACEBO 76 -4.07\n6 6 DRUG 73 -6.79\n7 7 PLACEBO 65 -5.14\n8 7 DRUG 64 -8.34\n```\n\n\n:::\n:::\n\n\nThe missingness pattern is show below (1=observed data point (blue), 0=missing data point (red)). The incomplete data is primarily monotone in nature. 128 patients have complete data for all visits (all 1's at each visit). 20, 10 and 13 patients have 1, 2 or 3 monotone missing data, respectively. Further, there is a single additional intermittent missing observation (patient 3618).\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_wide = dat |>\n dplyr::select(PATIENT, VISIT, CHANGE) |>\n pivot_wider(\n id_cols = PATIENT,\n names_from = VISIT,\n names_prefix = \"VISIT_\",\n values_from = CHANGE\n )\n\ndat_wide |>\n dplyr::select(starts_with(\"VISIT_\")) |>\n mice::md.pattern(plot = TRUE, rotate.names = TRUE)\n```\n\n::: {.cell-output-display}\n![](rbmi_continuous_joint_files/figure-html/explore data 2-1.png){width=672}\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n VISIT_4 VISIT_5 VISIT_6 VISIT_7 \n128 1 1 1 1 0\n20 1 1 1 0 1\n10 1 1 0 0 2\n1 1 0 1 1 1\n13 1 0 0 0 3\n 0 14 23 43 80\n```\n\n\n:::\n:::\n\n\n## Complete case analysis\n\nA complete case analysis is performed using mixed model for repeated measures (MMRM) with covariates: treatment [THERAPY], gender [GENDER], visit [VISIT] as factors; baseline score [BASVAL] as continuous; and visit-by-treatment [THERAPY * VISIT] interaction, and visit-by-baseline [BASVAL * VISIT] interaction. An unstructured covariance matrix is used.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm_fit = mmrm::mmrm(\n CHANGE ~\n 1 +\n THERAPY +\n GENDER +\n VISIT +\n BASVAL +\n THERAPY * VISIT +\n BASVAL * VISIT +\n us(VISIT | PATIENT),\n data = dat,\n reml = TRUE\n)\nsummary(mmrm_fit)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nmmrm fit\n\nFormula: \nCHANGE ~ 1 + THERAPY + GENDER + VISIT + BASVAL + THERAPY * VISIT + \n BASVAL * VISIT + us(VISIT | PATIENT)\nData: dat (used 608 observations from 172 subjects with maximum 4 \ntimepoints)\nCovariance: unstructured (10 variance parameters)\nMethod: Satterthwaite\nVcov Method: Asymptotic\nInference: REML\n\nModel selection criteria:\n AIC BIC logLik deviance \n 3512.9 3544.4 -1746.5 3492.9 \n\nCoefficients: \n Estimate Std. Error df t value Pr(>|t|) \n(Intercept) 3.16355 1.20260 168.64000 2.631 0.00931 ** \nTHERAPYDRUG 0.06603 0.68662 168.11000 0.096 0.92350 \nGENDERM 0.31961 0.68216 168.46000 0.469 0.64001 \nVISIT5 -0.50646 1.22706 157.16000 -0.413 0.68036 \nVISIT6 -0.39390 1.41983 149.35000 -0.277 0.78184 \nVISIT7 -2.29237 1.62198 142.91000 -1.413 0.15974 \nBASVAL -0.27866 0.06222 168.05000 -4.479 1.38e-05 ***\nTHERAPYDRUG:VISIT5 -1.49495 0.73342 156.86000 -2.038 0.04320 * \nTHERAPYDRUG:VISIT6 -2.31710 0.85860 151.23000 -2.699 0.00775 ** \nTHERAPYDRUG:VISIT7 -2.89468 0.96582 139.86000 -2.997 0.00323 ** \nVISIT5:BASVAL -0.03429 0.06567 157.48000 -0.522 0.60231 \nVISIT6:BASVAL -0.11482 0.07646 150.73000 -1.502 0.13527 \nVISIT7:BASVAL -0.04656 0.08679 142.04000 -0.537 0.59244 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\nCovariance estimate:\n 4 5 6 7\n4 19.7877 16.6237 15.4265 16.4578\n5 16.6237 34.3231 25.4682 26.2897\n6 15.4265 25.4682 38.4094 33.9331\n7 16.4578 26.2897 33.9331 45.3625\n```\n\n\n:::\n:::\n\n\nUsing the `emmeans` package/function least square means and contrast can be obtained.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nem = emmeans::emmeans(\n mmrm_fit,\n specs = trt.vs.ctrl ~ THERAPY * VISIT,\n at = list(VISIT = \"7\"),\n level = 0.95,\n adjust = \"none\",\n mode = \"df.error\"\n)\n\nem_contrast = broom::tidy(em$contrasts, conf.int = TRUE, conf.level = 0.95)\nem_contrast |>\n gt() |>\n fmt_number(decimals = 3)\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n \n \n
termcontrastnull.valueestimatestd.errordfconf.lowconf.highstatisticp.value
THERAPY*VISITDRUG VISIT7 - PLACEBO VISIT70.000−2.8291.117150.711−5.035−0.622−2.5330.012
\n
\n```\n\n:::\n:::\n\n\nThe treatment difference at visit 7 is of interest, and is estimated to be -2.829 (se=1.117) with 95% CI of [-5.035 to -0.622] (p=0.0123).\n\n\n## rbmi: MAR approach\nThe code presented here is based on the `rbmi` package [quickstart vignette](https://cran.r-project.org/web/packages/rbmi/vignettes/quickstart.html).\n\n### Create needed datasets and specify imputation strategy\n`rbmi` expects its input dataset to be complete; that is, there must be one row per subject for each visit (note: in clinical trials ADAMs typically do not have this required complete data structure). Missing outcome values should be coded as `NA`, while missing covariate values are not allowed. If the dataset is incomplete, then the `expand_locf()` function can be used to add any missing rows, using LOCF imputation to carry forward the observed baseline covariate values to visits with missing outcomes.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_expand <- rbmi::expand_locf(\n dat,\n PATIENT = levels(dat$PATIENT), # expand by PATIENT and VISIT\n VISIT = levels(dat$VISIT),\n vars = c(\"BASVAL\", \"THERAPY\", \"GENDER\"), # complete covariates using LOCF\n group = c(\"PATIENT\"),\n order = c(\"PATIENT\", \"VISIT\") # sort\n)\n```\n:::\n\n\nFor example, the data of patient 1513 in the original data and expanded data are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat |>\n dplyr::filter(PATIENT == \"1513\") |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n \n \n
PATIENTGENDERTHERAPYRELDAYSVISITBASVALHAMDTL17CHANGE
1513MDRUG7419245
\n
\n```\n\n:::\n\n```{.r .cell-code}\ndat_expand |>\n dplyr::filter(PATIENT == \"1513\") |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n \n
PATIENTGENDERTHERAPYRELDAYSVISITBASVALHAMDTL17CHANGE
1513MDRUG7419245
1513MDRUGNA519NANA
1513MDRUGNA619NANA
1513MDRUGNA719NANA
\n
\n```\n\n:::\n:::\n\n\nNext, a dataset must be created specifying which data points should be imputed with the specified imputation strategy. The dataset `dat_ice` is created which specifies the first visit affected by an intercurrent event (ICE) and the imputation strategy for handling missing outcome data after the ICE. At most one ICE which is to be imputed is allowed per subject. In the example, the subject’s first visit affected by the ICE “study drug discontinuation” corresponds to the first terminal missing observation\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_expand |>\n arrange(PATIENT, VISIT) |>\n filter(is.na(CHANGE)) |>\n group_by(PATIENT) |>\n slice(1) |>\n ungroup() |>\n select(PATIENT, VISIT) |>\n mutate(strategy = \"MAR\")\n\ngt(head(dat_ice))\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n \n
PATIENTVISITstrategy
15135MAR
15145MAR
15175MAR
18047MAR
21047MAR
21185MAR
\n
\n```\n\n:::\n:::\n\n\nIn this dataset, subject 3618 has an intermittent missing values which does not correspond to a study drug discontinuation. We therefore remove this subject from `dat_ice`. In the later imputation step, it will automatically be imputed under the default MAR assumption.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_ice[-which(dat_ice$PATIENT == 3618), ]\n```\n:::\n\n\n### Fit imputation model and draw posterior parameters\nThe `vars` object using using `set_vars()` defines the names of key variables in the dataset and the covariates included in the imputation model. If you wish to include interaction terms these need to be added in the covariates input.\n\nThe `method` object specifies the statistical method used to fit the imputation models and to create imputed datasets.\n\nThe `draws()` function fits the imputation model and stores the corresponding parameter estimates and Bayesian posterior parameter draws.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nvars <- rbmi::set_vars(\n outcome = \"CHANGE\",\n visit = \"VISIT\",\n subjid = \"PATIENT\",\n group = \"THERAPY\",\n covariates = c(\"GENDER\", \"BASVAL*VISIT\", \"THERAPY*VISIT\")\n)\n\nmethod <- rbmi::method_bayes(\n n_samples = 500,\n control = rbmi::control_bayes(warmup = 500, thin = 10)\n)\n\nset.seed(12345)\ndrawObj <- draws(\n data = dat_expand,\n data_ice = dat_ice,\n vars = vars,\n method = method,\n quiet = TRUE\n)\n\ndrawObj\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nDraws Object\n------------\nNumber of Samples: 500\nNumber of Failed Samples: 0\nModel Formula: CHANGE ~ 1 + THERAPY + VISIT + GENDER + BASVAL * VISIT + THERAPY * VISIT\nImputation Type: random\nMethod:\n name: Bayes\n covariance: us\n same_cov: TRUE\n n_samples: 500\n prior_cov: default\nControls:\n warmup: 500\n thin: 10\n chains: 1\n init: mmrm\n seed: 2245663\n```\n\n\n:::\n:::\n\n\n### Generate imputed datasets\n\nThe next step is to use the parameters from the imputation model to generate the imputed datasets. This is done via the `impute()` function. The function only has two key inputs: the imputation model output from `draws()` and the `references` groups relevant to reference-based imputation methods. Since we are using the MAR approach here, we can set it to NULL.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nimputeObj <- rbmi::impute(draws = drawObj, references = NULL)\n```\n:::\n\n\nIn case we would like to access the imputed datasets, we can use the `extract_imputed_dfs()` function. For example, the imputed values in the 10th imputed dataset for patient 1513 are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nimputed_dfs = rbmi::extract_imputed_dfs(imputeObj)\nMI_10 = imputed_dfs[[10]]\nMI_10$PATIENT_ID = dat_expand$PATIENT\n\nMI_10 |>\n dplyr::filter(PATIENT_ID == \"1513\") |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n
PATIENTGENDERTHERAPYRELDAYSVISITBASVALHAMDTL17CHANGEPATIENT_ID
new_pt_5MDRUG7419245.0000001513
new_pt_5MDRUGNA519NA4.1156421513
new_pt_5MDRUGNA619NA-8.2959281513
new_pt_5MDRUGNA719NA-8.7947781513
\n
\n```\n\n:::\n:::\n\n\n### Analyse imputed datasets\nThe next step is to run the analysis model on each imputed dataset. This is done by defining an analysis function and then calling the `analyse()` function to apply this function to each imputed dataset. The `ancova()` function provided by the `rbmi` package which fits a separate ANCOVA model for the outcomes from each visit is used.\n\nThe `ancova()` function uses the `set_vars()` function which determines the names of the key variables within the data and the covariates (in addition to the treatment group) for which the analysis model will be adjusted.\n\nNote: In Appendix 1 below we show how you can easily use a different analysis method (e.g., mmrm).\n\n\n::: {.cell}\n\n```{.r .cell-code}\nvars_analyse <- rbmi::set_vars(\n outcome = \"CHANGE\",\n visit = \"VISIT\",\n subjid = \"PATIENT\",\n group = \"THERAPY\",\n covariates = c(\"BASVAL\", \"GENDER\")\n)\n\nanaObj <- rbmi::analyse(\n imputations = imputeObj,\n fun = ancova,\n vars = vars_analyse\n)\n```\n:::\n\n\n### Pool results\n\nFinally, the `pool()` function can be used to summarise the analysis results across multiple imputed datasets to provide an overall statistic with a standard error, confidence intervals and a p-value for the hypothesis test of the null hypothesis that the effect is equal to 0. Since we used `method_bayes()`, pooling and inference are based on Rubin’s rules.\n\nHere, the treatment difference at visit 7 is of interest. Since we set PLACEBO as the first factor in the variable `THERAPY` this corresponds to `ref`, whereas DRUG corresponds to `alt`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\npoolObj <- rbmi::pool(anaObj, conf.level = 0.95, alternative = \"two.sided\")\n\npoolObj |>\n data.frame() |>\n dplyr::filter(grepl(\"7\", parameter)) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
parameterestselciucipval
trt_7-2.8158561.1230680-5.036010-0.5957011.329608e-02
lsm_ref_7-4.8233090.7823422-6.369977-3.2766417.022688e-09
lsm_alt_7-7.6391650.7931535-9.206925-6.0714043.025813e-17
\n
\n```\n\n:::\n:::\n\n\n\n## rbmi: MNAR CR approach\nThe following changes need to be made in the code above to apply the Copy Reference (CR) approach in `rbmi`. For `dat_ice` the strategy need to be changed to CR. In the `impute()` step the `references` need to be specified. Here we set the reference for the DRUG group to PLACEBO.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_expand |>\n arrange(PATIENT, VISIT) |>\n filter(is.na(CHANGE)) |>\n group_by(PATIENT) |>\n slice(1) |>\n ungroup() |>\n select(PATIENT, VISIT) |>\n mutate(strategy = \"CR\")\n\nimputeObj <- rbmi::impute(\n drawObj,\n references = c(\"PLACEBO\" = \"PLACEBO\", \"DRUG\" = \"PLACEBO\")\n)\n```\n:::\n\n\n\n\nThe results for M=500 imputed datasets using the MNAR CR approach are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\npoolObj |>\n data.frame() |>\n dplyr::filter(grepl(\"7\", parameter)) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
parameterestselciucipval
trt_7-2.3897801.1101922-4.583669-0.19589113.297369e-02
lsm_ref_7-4.8298340.7876150-6.386855-3.27281348.189424e-09
lsm_alt_7-7.2196140.7942274-8.789278-5.64995046.533123e-16
\n
\n```\n\n:::\n:::\n\n\n## rbmi: MNAR JR approach\nThe following changes need to be made in the code above to apply the Jump to Reference (JR) approach in `rbmi`. For `dat_ice` the strategy need to be changed to JR. In the `impute()` step the `references` need to be specified. Here we set the reference for the DRUG group to PLACEBO.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_expand |>\n arrange(PATIENT, VISIT) |>\n filter(is.na(CHANGE)) |>\n group_by(PATIENT) |>\n slice(1) |>\n ungroup() |>\n select(PATIENT, VISIT) |>\n mutate(strategy = \"JR\")\n\nimputeObj <- rbmi::impute(\n drawObj,\n references = c(\"PLACEBO\" = \"PLACEBO\", \"DRUG\" = \"PLACEBO\")\n)\n```\n:::\n\n\n\n\nThe results for M=500 imputed datasets using the MNAR JR approach are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\npoolObj |>\n data.frame() |>\n dplyr::filter(grepl(\"7\", parameter)) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
parameterestselciucipval
trt_7-2.1420011.1307406-4.3769240.092921146.017815e-02
lsm_ref_7-4.8269990.7859242-6.380406-3.273592807.507619e-09
lsm_alt_7-6.9690010.8224064-8.595170-5.342831803.220372e-14
\n
\n```\n\n:::\n:::\n\n\n\n\n## rbmi: MNAR CIR approach\nThe following changes need to be made in the code above to apply the Copy Increments in Reference (CIR) approach in `rbmi`. For `dat_ice` the strategy need to be changed to CIR. In the `impute()` step the `references` need to be specified. Here we set the reference for the DRUG group to PLACEBO.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_expand |>\n arrange(PATIENT, VISIT) |>\n filter(is.na(CHANGE)) |>\n group_by(PATIENT) |>\n slice(1) |>\n ungroup() |>\n select(PATIENT, VISIT) |>\n mutate(strategy = \"CIR\")\n\nimputeObj <- rbmi::impute(\n drawObj,\n references = c(\"PLACEBO\" = \"PLACEBO\", \"DRUG\" = \"PLACEBO\")\n)\n```\n:::\n\n\n\n\nThe results for M=500 imputed datasets using the MNAR CIR approach are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\npoolObj |>\n data.frame() |>\n dplyr::filter(grepl(\"7\", parameter)) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
parameterestselciucipval
trt_7-2.4733641.1062706-4.659453-0.2872742.686046e-02
lsm_ref_7-4.8198110.7792744-6.360108-3.2795146.080274e-09
lsm_alt_7-7.2931740.7932565-8.860947-5.7254023.597819e-16
\n
\n```\n\n:::\n:::\n\n\n\n## Summary of results\nIn the table we present the results of the different imputation strategies (and with varying number, *M*, of multiple imputation draws). Note that the results can be (slightly) different from the results above due to a possible different seed. The table show the contrast at Visit 7 between DRUG and PLACEBO [DRUG - PLACEBO]:\n\n| Method | Estimate | SE | 95% CI | p-value |\n|-------------------|----------|-------|------------------|---------|\n| Complete Case | -2.829 | 1.117 | -5.035 to -0.622 | 0.0123 |\n| MI - MAR (M=500) | -2.833 | 1.120 | -5.046 to -0.620 | 0.0125 |\n| MI - MAR (M=2000) | -2.837 | 1.118 | -5.047 to -0.627 | 0.0122 |\n| MI - MAR (M=5000) | -2.830 | 1.123 | -5.040 to -0.610 | 0.0128 |\n| MI - MNAR CR (M=500) | -2.377 | 1.119 | -4.588 to -0.167 | 0.0352 |\n| MI - MNAR CR (M=2000) | -2.391 | 1.110 | -4.585 to -0.198 | 0.0328 |\n| MI - MNAR CR (M=5000) | -2.394 | 1.112 | -4.592 to -0.197 | 0.0329 |\n| MI - MNAR JR (M=500) | -2.169 | 1.134 | -4.411 to 0.072 | 0.0577 |\n| MI - MNAR JR (M=2000) | -2.146 | 1.135 | -4.389 to 0.097 | 0.0606 |\n| MI - MNAR JR (M=5000) | -2.148 | 1.135 | -4.390 to 0.095 | 0.0603 |\n| MI - MNAR CIR (M=500) | -2.495 | 1.113 | -4.695 to -0.295 | 0.0265 |\n| MI - MNAR CIR (M=2000) | -2.469 | 1.116 | -4.674 to -0.263 | 0.0285 |\n| MI - MNAR CIR (M=5000) | -2.479 | 1.112 | -4.676 to -0.282 | 0.0273 |\n\n## Approximate Bayesian\n\nIn the `draws()` function it is possible to specify other methods. For example, the approximate Bayesian MI `method_approxbayes()` which is based on bootstrapping. `draws()` returns the draws from the posterior distribution of the parameters using an approximate Bayesian approach, where the sampling from the posterior distribution is simulated by fitting the MMRM model on bootstrap samples of the original dataset.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmethod <- rbmi::method_approxbayes(\n covariance = \"us\",\n threshold = 0.01,\n REML = TRUE,\n n_samples = 500\n)\n```\n:::\n\n\nIn the table we present the results of the approximate Bayesian approach for a CR imputation strategy. The table show the contrast at Visit 7 between DRUG and PLACEBO [DRUG - PLACEBO]:\n\n| Method | Estimate | SE | 95% CI | p-value |\n|-------------------|----------|-------|------------------|---------|\n| MI - MNAR CR (M=500) | -2.415 | 1.109 | -4.617 to -0.210 | 0.0320 |\n| MI - MNAR CR (M=2000) | -2.403 | 1.112 | -4.600 to -0.205 | 0.0323 |\n\n## Discussion\n\nA note on computational time: The total running time (including data loading, setting up data sets, MCMC run, imputing data and analysis MI data) for M=500 was about 26 seconds on a personal laptop. It increased to about 92 seconds for M=2000. Computational time was similar across different imputation strategies.\n\nWith a small number of `n_samples` in `method_bayes()` a warning could pop-up \"The largest R-hat is 1.08, indicating chains have not mixed. Running the chains for more iterations may help\". Increasing the number of `n_samples` will mostly solve this warning. For example, for this data example, this message is received when setting `n_samples` equal to a number below 100.\n\n## Appendix 1: mmrm as analysis model\n\nIn the `analyse()` function (at the moment of writing) the only available analysis function is `ancova`. However, the user is able to specify its own analysis function. See the `analyse()` function for more details.\n\nAnother possibility (although, not the most efficient) is to implement a for loop in which the model is fit on each imputed dataset. The obtained results could then be pooled using Rubin's rule. For example, suppose an MMRM should be fit on each imputed dataset:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm_analyse_mi_function <- function(Impute_Obj) {\n # create all imputed datasets\n imputed_dfs = rbmi::extract_imputed_dfs(Impute_Obj)\n\n # create empty vectors to store mmrm analysis results\n est_vec = sd_vec = df_vec = NULL\n\n # for loop to save estimates per imputation\n for (k in 1:length(imputed_dfs)) {\n temp_dat = imputed_dfs[[k]]\n mmrm_fit_temp = mmrm::mmrm(\n CHANGE ~\n 1 +\n THERAPY +\n VISIT +\n BASVAL * VISIT +\n THERAPY * VISIT +\n GENDER +\n us(VISIT | PATIENT),\n data = temp_dat,\n reml = TRUE\n )\n em = emmeans::emmeans(\n mmrm_fit_temp,\n specs = trt.vs.ctrl ~ THERAPY * VISIT,\n at = list(VISIT = \"7\"),\n level = 0.95,\n adjust = \"none\",\n mode = \"df.error\"\n )\n est_vec[k] = summary(em$contrasts)$estimate\n sd_vec[k] = summary(em$contrasts)$SE\n df_vec[k] = summary(em$contrasts)$df\n }\n\n # summarize results using rubin's rule\n rr = rbmi:::rubin_rules(ests = est_vec, ses = sd_vec, v_com = mean(df_vec))\n rr$se_t = sqrt(rr$var_t)\n rr$t.stat = rr$est_point / sqrt(rr$var_t)\n rr$p_value = 2 * pt(q = rr$t.stat, df = rr$df, lower.tail = TRUE)\n\n return(rr = rr)\n}\n```\n:::\n\n\nThe following code then performs the analysis and pooling\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm_analyse_mi_function(Impute_Obj = imputeObj)\n```\n:::\n\n\nIn the table we present the results of the Bayesian approach for a CR imputation strategy with an MMRM analysis model. The table show the contrast at Visit 7 between DRUG and PLACEBO [DRUG - PLACEBO]:\n\n| Method | Estimate | SE | 95% CI | p-value |\n|-------------------|----------|-------|------------------|---------|\n| MI - MNAR CR (M=500) | -2.415 | 1.109 | -4.607 to -0.223 | 0.0310 |\n| MI - MNAR CR (M=2000) | -2.388 | 1.111 | -4.584 to -0.193 | 0.0332 | \n\n\n## Reference\n\n[Carpenter JR, Roger JH & Kenward MG (2013)](https://doi.org/10.1080/10543406.2013.834911). Analysis of Longitudinal Trials with Protocol Deviation: A Framework for Relevant, Accessible Assumptions, and Inference via MI. *Journal of Biopharmaceutical Statistics* 23: 1352-1371.\n\n[Gower-Page C, Noci A & Wolbers M (2022)](https://doi.org/10.21105/joss.04251). rbmi: A R package for standard and reference-based multiple imputation methods. *Journal of Open Source Software* 7(74): 4251. \n\n[rbmi: Reference Based Multiple Imputation](https://cran.r-project.org/web/packages/rbmi/index.html)\n\n[rbmi: Quickstart](https://cran.r-project.org/web/packages/rbmi/vignettes/quickstart.html)\n\n[Roger J (2022, Dec 8)](https://baselbiometrics.github.io/home/docs/talks/20221208/5_JamesRoger%2020121118.pdf). Other statistical software for continuous longitudinal endpoints: SAS macros for multiple imputation. *Addressing intercurrent events: Treatment policy and hypothetical strategies*. Joint EFSPI and BBS virtual event.\n\n[Rubin DB (1976)](https://doi.org/10.1093/biomet/63.3.581). Inference and Missing Data. *Biometrika* 63: 581–592.\n\n[Rubin DB (1987)](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316696). *Multiple Imputation for Nonresponse in Surveys*. New York: John Wiley & Sons.\n\n[von Hippel PT & Bartlett JW (2021)](https://doi.org/10.1214/20-STS793). Maximum likelihood multiple imputation: Faster imputations and consistent standard errors without posterior draws. *Statistical Science* 36(3): 400–420. \n\n[Wolbers M, Noci A, Delmar P, Gower-Page C, Yiu S & Bartlett JW (2022)](https://onlinelibrary.wiley.com/doi/full/10.1002/pst.2234). Standard and reference-based conditional mean imputation. *Pharmaceutical Statistics* 21(6): 1246-1257.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os macOS Tahoe 26.3\n system aarch64, darwin20\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-02-23\n pandoc 3.6.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)\n quarto 1.8.27 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/quarto\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P assertthat 0.2.1 2019-03-21 [?] RSPM\n P backports 1.5.0 2024-05-23 [?] RSPM\n boot 1.3-32 2025-08-29 [2] CRAN (R 4.5.2)\n P broom 1.0.12 2026-01-27 [?] RSPM\n P callr 3.7.6 2024-03-25 [?] RSPM\n P checkmate 2.3.4 2026-02-03 [?] RSPM\n P cli 3.6.5 2025-04-23 [?] RSPM\n codetools 0.2-20 2024-03-31 [2] CRAN (R 4.5.2)\n P curl 7.0.0 2025-08-19 [?] RSPM\n P digest 0.6.39 2025-11-19 [?] RSPM\n P dplyr * 1.2.0 2026-02-03 [?] RSPM\n P emmeans * 2.0.1 2025-12-16 [?] RSPM\n P estimability 1.5.1 2024-05-12 [?] RSPM\n P evaluate 1.0.5 2025-08-27 [?] RSPM\n P farver 2.1.2 2024-05-13 [?] RSPM\n P fastmap 1.2.0 2024-05-15 [?] RSPM\n P forcats 1.0.1 2025-09-25 [?] RSPM\n P foreach 1.5.2 2022-02-02 [?] RSPM\n P fs 1.6.6 2025-04-12 [?] RSPM\n P generics 0.1.4 2025-05-09 [?] RSPM\n P ggplot2 4.0.2 2026-02-03 [?] RSPM\n P glmnet 4.1-10 2025-07-17 [?] RSPM\n P glue 1.8.0 2024-09-30 [?] RSPM\n P gridExtra 2.3 2017-09-09 [?] RSPM\n P gt * 1.3.0 2026-01-22 [?] RSPM\n P gtable 0.3.6 2024-10-25 [?] RSPM\n P haven 2.5.5 2025-05-30 [?] RSPM\n P hms 1.1.4 2025-10-17 [?] RSPM\n P htmltools 0.5.9 2025-12-04 [?] RSPM\n P htmlwidgets 1.6.4 2023-12-06 [?] RSPM\n P inline 0.3.21 2025-01-09 [?] RSPM\n P iterators 1.0.14 2022-02-05 [?] RSPM\n P jinjar 0.3.2 2025-03-13 [?] RSPM\n P jomo 2.7-6 2023-04-15 [?] RSPM\n P jsonlite 2.0.0 2025-03-27 [?] RSPM\n P knitr 1.51 2025-12-20 [?] RSPM\n P labelled * 2.16.0 2025-10-22 [?] RSPM\n lattice 0.22-7 2025-04-02 [2] CRAN (R 4.5.2)\n P lifecycle 1.0.5 2026-01-08 [?] RSPM\n P lme4 1.1-38 2025-12-02 [?] RSPM\n P loo 2.9.0 2025-12-23 [?] RSPM\n P magrittr 2.0.4 2025-09-12 [?] RSPM\n MASS 7.3-65 2025-02-28 [2] CRAN (R 4.5.2)\n Matrix 1.7-4 2025-08-28 [2] CRAN (R 4.5.2)\n P matrixStats 1.5.0 2025-01-07 [?] RSPM\n P mice * 3.19.0 2025-12-10 [?] RSPM\n P minqa 1.2.8 2024-08-17 [?] RSPM\n P mitml 0.4-5 2023-03-08 [?] RSPM\n P mmrm * 0.3.17 2026-01-08 [?] RSPM\n P multcomp 1.4-29 2025-10-20 [?] RSPM\n P mvtnorm 1.3-3 2025-01-10 [?] RSPM\n nlme 3.1-168 2025-03-31 [2] CRAN (R 4.5.2)\n P nloptr 2.2.1 2025-03-17 [?] RSPM\n nnet 7.3-20 2025-01-01 [2] CRAN (R 4.5.2)\n P otel 0.2.0 2025-08-29 [?] RSPM\n P pan 1.9 2023-12-07 [?] RSPM\n P pillar 1.11.1 2025-09-17 [?] RSPM\n P pkgbuild 1.4.8 2025-05-26 [?] RSPM\n P pkgconfig 2.0.3 2019-09-22 [?] RSPM\n P processx 3.8.6 2025-02-21 [?] RSPM\n P ps 1.9.1 2025-04-12 [?] RSPM\n P purrr 1.2.1 2026-01-09 [?] RSPM\n P QuickJSR 1.9.0 2026-01-25 [?] RSPM\n P R6 2.6.1 2025-02-15 [?] RSPM\n P rbibutils 2.4.1 2026-01-21 [?] RSPM\n P rbmi * 1.6.0 2026-01-23 [?] RSPM\n P RColorBrewer 1.1-3 2022-04-03 [?] RSPM\n P Rcpp 1.1.1 2026-01-10 [?] RSPM\n P RcppParallel 5.1.11-1 2025-08-27 [?] RSPM\n P Rdpack 2.6.6 2026-02-08 [?] RSPM\n P reformulas 0.4.4 2026-02-02 [?] RSPM\n renv 1.0.10 2024-10-05 [1] RSPM (R 4.5.2)\n P rlang 1.1.7 2026-01-09 [?] RSPM\n P rmarkdown 2.30 2025-09-28 [?] RSPM\n rpart 4.1.24 2025-01-07 [2] CRAN (R 4.5.2)\n P rstan 2.32.7 2025-03-10 [?] RSPM\n P S7 0.2.1 2025-11-14 [?] RSPM\n P sandwich 3.1-1 2024-09-15 [?] RSPM\n P sass 0.4.10 2025-04-11 [?] RSPM\n P scales 1.4.0 2025-04-24 [?] RSPM\n P sessioninfo 1.2.3 2025-02-05 [?] RSPM\n P shape 1.4.6.1 2024-02-23 [?] RSPM\n P StanHeaders 2.32.10 2024-07-15 [?] RSPM\n P stringi 1.8.7 2025-03-27 [?] RSPM\n P stringr 1.6.0 2025-11-04 [?] RSPM\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n P TH.data 1.1-5 2025-11-17 [?] RSPM\n P tibble 3.3.1 2026-01-11 [?] RSPM\n P tidyr * 1.3.2 2025-12-19 [?] RSPM\n P tidyselect 1.2.1 2024-03-11 [?] RSPM\n P TMB 1.9.19 2025-12-15 [?] RSPM\n P utf8 1.2.6 2025-06-08 [?] RSPM\n P V8 8.0.1 2025-10-10 [?] RSPM\n P vctrs 0.7.1 2026-01-23 [?] RSPM\n P withr 3.0.2 2024-10-28 [?] RSPM\n P xfun 0.56 2026-01-18 [?] RSPM\n P xml2 1.5.2 2026-01-17 [?] RSPM\n P xtable 1.8-4 2019-04-21 [?] RSPM\n P yaml 2.3.12 2025-12-10 [?] RSPM\n P zoo 1.8-15 2025-12-15 [?] RSPM\n\n [1] /Users/christinafillmore/Documents/GitHub/CAMIS/renv/library/macos/R-4.5/aarch64-apple-darwin20\n [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library\n\n * ── Packages attached to the search path.\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n\n", + "markdown": "---\ntitle: \"Reference-Based Multiple Imputation (joint modelling): Continuous Data\"\n---\n\n## Libraries\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# General\nlibrary(dplyr)\nlibrary(tidyr)\nlibrary(gt)\nlibrary(labelled)\n\n# Methodlolgy specific\nlibrary(mmrm)\nlibrary(emmeans)\nlibrary(rbmi)\nlibrary(mice) # only used md.pattern()\n```\n:::\n\n\n## Reference-based multiple imputation (rbmi)\n\n### Methodology introduction\nReference-based multiple imputation methods have become popular for handling missing data, as well as for conducting sensitivity analyses, in randomized clinical trials. In the context of a repeatedly measured continuous endpoint assuming a multivariate normal model, [Carpenter et al. (2013)](https://www.tandfonline.com/doi/full/10.1080/10543406.2013.834911) proposed a framework to extend the usual MAR-based MI approach by postulating assumptions about the joint distribution of pre- and post-deviation data. Under this framework, one makes qualitative assumptions about how individuals’ missing outcomes relate to those observed in relevant groups in the trial, based on plausible clinical scenarios. Statistical analysis then proceeds using the method of multiple imputation ([Rubin 1976](https://doi.org/10.1093/biomet/63.3.581), [Rubin 1987]((https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316696))).\n\nIn general, multiple imputation of a repeatedly measured continuous outcome can be done via 2 computational routes ([Roger 2022](https://baselbiometrics.github.io/home/docs/talks/20221208/5_JamesRoger%2020121118.pdf)):\n\n 1. Stepwise: split problem into separate imputations of data at each visit\n \n + requires monotone missingness, such as missingness due to withdrawal\n \n + conditions on the imputed values at previous visit\n \n + Bayesian linear regression problem is much simpler with monotone missing, as one can sample directly using conjugate priors\n\n 2. One-step approach (joint modelling): Fit a Bayesian full multivariate normal repeated measures model using MCMC and then draw a sample.\n\nHere, we illustrate reference-based multiple imputation of a continuous outcome measured repeatedly via the so-called one-step approach.\n\n### rbmi package\nThe `rbmi` package [Gower-Page et al. (2022)](https://joss.theoj.org/papers/10.21105/joss.04251) will be used for the one-step approach of the reference-based multiple imputation using R. The package implements standard and reference based multiple imputation methods for continuous longitudinal endpoints . In particular, this package supports deterministic conditional mean imputation and jackknifing as described in [Wolbers et al. (2022)](https://onlinelibrary.wiley.com/doi/full/10.1002/pst.2234), convential MI based on Bayesian posterior draws as described in [Carpenter et al. (2013)](https://www.tandfonline.com/doi/full/10.1080/10543406.2013.834911), and bootstrapped maximum likelihood imputation as described in [von Hippel and Bartlett (2021)](https://doi.org/10.1214/20-STS793).\n\nThe following standard and reference-based multiple imputation approaches will be illustrated here: \n \n * MAR (Missing At Random)\n \n * CIR (Copy Increment from Reference)\n \n * J2R (Jump to Reference)\n \n * CR (Copy Reference)\n\n\n## Data used\nA publicly available example [dataset](https://r-packages.io/datasets/antidepressant_data) from an antidepressant clinical trial of an active drug versus placebo is used. Overall, data of 172 patients is available with 88 patients receiving placebo and 84 receiving active drug. This data is also used in the `rbmi` package [quickstart vignette](https://cran.r-project.org/web/packages/rbmi/vignettes/quickstart.html).\n\nThe relevant endpoint is the Hamilton 17-item depression rating scale (HAMD17) which was assessed at baseline and at weeks 1, 2, 4, and 6 (visits 4-7). Study drug discontinuation occurred in 24% (20/84) of subjects from the active drug and 26% (23/88) of subjects from placebo. All data after study drug discontinuation are missing. \n\n\n::: {.cell}\n\n```{.r .cell-code}\ndata(\"antidepressant_data\")\ndat <- antidepressant_data |>\n dplyr::select(\n PATIENT,\n GENDER,\n THERAPY,\n RELDAYS,\n VISIT,\n BASVAL,\n HAMDTL17,\n CHANGE\n ) |>\n dplyr::mutate(THERAPY = factor(THERAPY, levels = c(\"PLACEBO\", \"DRUG\"))) |>\n labelled::remove_labels()\n\ngt(head(dat, n = 10))\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n \n
PATIENTGENDERTHERAPYRELDAYSVISITBASVALHAMDTL17CHANGE
1503FDRUG743221-11
1503FDRUG1453220-12
1503FDRUG2863219-13
1503FDRUG4273217-15
1507FPLACEBO741411-3
1507FPLACEBO15514140
1507FPLACEBO296149-5
1507FPLACEBO427145-9
1509FDRUG742120-1
1509FDRUG1452118-3
\n
\n```\n\n:::\n:::\n\n\nThe number of patients per visit and arm are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat |>\n group_by(VISIT, THERAPY) |>\n dplyr::summarise(N = n())\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\n`summarise()` has regrouped the output.\nℹ Summaries were computed grouped by VISIT and THERAPY.\nℹ Output is grouped by VISIT.\nℹ Use `summarise(.groups = \"drop_last\")` to silence this message.\nℹ Use `summarise(.by = c(VISIT, THERAPY))` for per-operation grouping\n (`?dplyr::dplyr_by`) instead.\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 8 × 3\n# Groups: VISIT [4]\n VISIT THERAPY N\n \n1 4 PLACEBO 88\n2 4 DRUG 84\n3 5 PLACEBO 81\n4 5 DRUG 77\n5 6 PLACEBO 76\n6 6 DRUG 73\n7 7 PLACEBO 65\n8 7 DRUG 64\n```\n\n\n:::\n:::\n\n\nThe mean change from baseline of the endpoint (Hamilton 17-item depression rating scale, HAMD17) per visit per treatment group using only the complete cases are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat |>\n group_by(VISIT, THERAPY) |>\n dplyr::summarise(N = n(), MEAN = mean(CHANGE))\n```\n\n::: {.cell-output .cell-output-stderr}\n\n```\n`summarise()` has regrouped the output.\nℹ Summaries were computed grouped by VISIT and THERAPY.\nℹ Output is grouped by VISIT.\nℹ Use `summarise(.groups = \"drop_last\")` to silence this message.\nℹ Use `summarise(.by = c(VISIT, THERAPY))` for per-operation grouping\n (`?dplyr::dplyr_by`) instead.\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 8 × 4\n# Groups: VISIT [4]\n VISIT THERAPY N MEAN\n \n1 4 PLACEBO 88 -1.51\n2 4 DRUG 84 -1.82\n3 5 PLACEBO 81 -2.70\n4 5 DRUG 77 -4.71\n5 6 PLACEBO 76 -4.07\n6 6 DRUG 73 -6.79\n7 7 PLACEBO 65 -5.14\n8 7 DRUG 64 -8.34\n```\n\n\n:::\n:::\n\n\nThe missingness pattern is show below (1=observed data point (blue), 0=missing data point (red)). The incomplete data is primarily monotone in nature. 128 patients have complete data for all visits (all 1's at each visit). 20, 10 and 13 patients have 1, 2 or 3 monotone missing data, respectively. Further, there is a single additional intermittent missing observation (patient 3618).\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_wide = dat |>\n dplyr::select(PATIENT, VISIT, CHANGE) |>\n pivot_wider(\n id_cols = PATIENT,\n names_from = VISIT,\n names_prefix = \"VISIT_\",\n values_from = CHANGE\n )\n\ndat_wide |>\n dplyr::select(starts_with(\"VISIT_\")) |>\n mice::md.pattern(plot = TRUE, rotate.names = TRUE)\n```\n\n::: {.cell-output-display}\n![](rbmi_continuous_joint_files/figure-html/explore data 2-1.png){width=672}\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n VISIT_4 VISIT_5 VISIT_6 VISIT_7 \n128 1 1 1 1 0\n20 1 1 1 0 1\n10 1 1 0 0 2\n1 1 0 1 1 1\n13 1 0 0 0 3\n 0 14 23 43 80\n```\n\n\n:::\n:::\n\n\n## Complete case analysis\n\nA complete case analysis is performed using mixed model for repeated measures (MMRM) with covariates: treatment [THERAPY], gender [GENDER], visit [VISIT] as factors; baseline score [BASVAL] as continuous; and visit-by-treatment [THERAPY * VISIT] interaction, and visit-by-baseline [BASVAL * VISIT] interaction. An unstructured covariance matrix is used.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm_fit = mmrm::mmrm(\n CHANGE ~\n 1 +\n THERAPY +\n GENDER +\n VISIT +\n BASVAL +\n THERAPY * VISIT +\n BASVAL * VISIT +\n us(VISIT | PATIENT),\n data = dat,\n reml = TRUE\n)\nsummary(mmrm_fit)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nmmrm fit\n\nFormula: \nCHANGE ~ 1 + THERAPY + GENDER + VISIT + BASVAL + THERAPY * VISIT + \n BASVAL * VISIT + us(VISIT | PATIENT)\nData: dat (used 608 observations from 172 subjects with maximum 4 \ntimepoints)\nCovariance: unstructured (10 variance parameters)\nMethod: Satterthwaite\nVcov Method: Asymptotic\nInference: REML\n\nModel selection criteria:\n AIC BIC logLik deviance \n 3512.9 3544.4 -1746.5 3492.9 \n\nCoefficients: \n Estimate Std. Error df t value Pr(>|t|) \n(Intercept) 3.16355 1.20260 168.64000 2.631 0.00931 ** \nTHERAPYDRUG 0.06603 0.68662 168.11000 0.096 0.92350 \nGENDERM 0.31961 0.68216 168.46000 0.469 0.64001 \nVISIT5 -0.50646 1.22706 157.16000 -0.413 0.68036 \nVISIT6 -0.39390 1.41983 149.35000 -0.277 0.78184 \nVISIT7 -2.29237 1.62198 142.91000 -1.413 0.15974 \nBASVAL -0.27866 0.06222 168.05000 -4.479 1.38e-05 ***\nTHERAPYDRUG:VISIT5 -1.49495 0.73342 156.86000 -2.038 0.04320 * \nTHERAPYDRUG:VISIT6 -2.31710 0.85860 151.23000 -2.699 0.00775 ** \nTHERAPYDRUG:VISIT7 -2.89468 0.96582 139.86000 -2.997 0.00323 ** \nVISIT5:BASVAL -0.03429 0.06567 157.48000 -0.522 0.60231 \nVISIT6:BASVAL -0.11482 0.07646 150.73000 -1.502 0.13527 \nVISIT7:BASVAL -0.04656 0.08679 142.04000 -0.537 0.59244 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\nCovariance estimate:\n 4 5 6 7\n4 19.7877 16.6237 15.4265 16.4578\n5 16.6237 34.3231 25.4682 26.2897\n6 15.4265 25.4682 38.4094 33.9331\n7 16.4578 26.2897 33.9331 45.3625\n```\n\n\n:::\n:::\n\n\nUsing the `emmeans` package/function least square means and contrast can be obtained.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nem = emmeans::emmeans(\n mmrm_fit,\n specs = trt.vs.ctrl ~ THERAPY * VISIT,\n at = list(VISIT = \"7\"),\n level = 0.95,\n adjust = \"none\",\n mode = \"df.error\"\n)\n\nem_contrast = broom::tidy(em$contrasts, conf.int = TRUE, conf.level = 0.95)\nem_contrast |>\n gt() |>\n fmt_number(decimals = 3)\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n \n \n
termcontrastnull.valueestimatestd.errordfconf.lowconf.highstatisticp.value
THERAPY*VISITDRUG VISIT7 - PLACEBO VISIT70.000−2.8291.117150.711−5.035−0.622−2.5330.012
\n
\n```\n\n:::\n:::\n\n\nThe treatment difference at visit 7 is of interest, and is estimated to be -2.829 (se=1.117) with 95% CI of [-5.035 to -0.622] (p=0.0123).\n\n\n## rbmi: MAR approach\nThe code presented here is based on the `rbmi` package [quickstart vignette](https://cran.r-project.org/web/packages/rbmi/vignettes/quickstart.html).\n\n### Create needed datasets and specify imputation strategy\n`rbmi` expects its input dataset to be complete; that is, there must be one row per subject for each visit (note: in clinical trials ADAMs typically do not have this required complete data structure). Missing outcome values should be coded as `NA`, while missing covariate values are not allowed. If the dataset is incomplete, then the `expand_locf()` function can be used to add any missing rows, using LOCF imputation to carry forward the observed baseline covariate values to visits with missing outcomes.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_expand <- rbmi::expand_locf(\n dat,\n PATIENT = levels(dat$PATIENT), # expand by PATIENT and VISIT\n VISIT = levels(dat$VISIT),\n vars = c(\"BASVAL\", \"THERAPY\", \"GENDER\"), # complete covariates using LOCF\n group = c(\"PATIENT\"),\n order = c(\"PATIENT\", \"VISIT\") # sort\n)\n```\n:::\n\n\nFor example, the data of patient 1513 in the original data and expanded data are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat |>\n dplyr::filter(PATIENT == \"1513\") |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n \n \n
PATIENTGENDERTHERAPYRELDAYSVISITBASVALHAMDTL17CHANGE
1513MDRUG7419245
\n
\n```\n\n:::\n\n```{.r .cell-code}\ndat_expand |>\n dplyr::filter(PATIENT == \"1513\") |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n \n \n
PATIENTGENDERTHERAPYRELDAYSVISITBASVALHAMDTL17CHANGE
1513MDRUG7419245
1513MDRUGNA519NANA
1513MDRUGNA619NANA
1513MDRUGNA719NANA
\n
\n```\n\n:::\n:::\n\n\nNext, a dataset must be created specifying which data points should be imputed with the specified imputation strategy. The dataset `dat_ice` is created which specifies the first visit affected by an intercurrent event (ICE) and the imputation strategy for handling missing outcome data after the ICE. At most one ICE which is to be imputed is allowed per subject. In the example, the subject’s first visit affected by the ICE “study drug discontinuation” corresponds to the first terminal missing observation\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_expand |>\n arrange(PATIENT, VISIT) |>\n filter(is.na(CHANGE)) |>\n group_by(PATIENT) |>\n slice(1) |>\n ungroup() |>\n select(PATIENT, VISIT) |>\n mutate(strategy = \"MAR\")\n\ngt(head(dat_ice))\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n \n
PATIENTVISITstrategy
15135MAR
15145MAR
15175MAR
18047MAR
21047MAR
21185MAR
\n
\n```\n\n:::\n:::\n\n\nIn this dataset, subject 3618 has an intermittent missing values which does not correspond to a study drug discontinuation. We therefore remove this subject from `dat_ice`. In the later imputation step, it will automatically be imputed under the default MAR assumption.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_ice[-which(dat_ice$PATIENT == 3618), ]\n```\n:::\n\n\n### Fit imputation model and draw posterior parameters\nThe `vars` object using using `set_vars()` defines the names of key variables in the dataset and the covariates included in the imputation model. If you wish to include interaction terms these need to be added in the covariates input.\n\nThe `method` object specifies the statistical method used to fit the imputation models and to create imputed datasets.\n\nThe `draws()` function fits the imputation model and stores the corresponding parameter estimates and Bayesian posterior parameter draws.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nvars <- rbmi::set_vars(\n outcome = \"CHANGE\",\n visit = \"VISIT\",\n subjid = \"PATIENT\",\n group = \"THERAPY\",\n covariates = c(\"GENDER\", \"BASVAL*VISIT\", \"THERAPY*VISIT\")\n)\n\nmethod <- rbmi::method_bayes(\n n_samples = 500,\n control = rbmi::control_bayes(warmup = 500, thin = 10)\n)\n\nset.seed(12345)\ndrawObj <- draws(\n data = dat_expand,\n data_ice = dat_ice,\n vars = vars,\n method = method,\n quiet = TRUE\n)\n\ndrawObj\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n\nDraws Object\n------------\nNumber of Samples: 500\nNumber of Failed Samples: 0\nModel Formula: CHANGE ~ 1 + THERAPY + VISIT + GENDER + BASVAL * VISIT + THERAPY * VISIT\nImputation Type: random\nMethod:\n name: Bayes\n covariance: us\n same_cov: TRUE\n n_samples: 500\n prior_cov: default\nControls:\n warmup: 500\n thin: 10\n chains: 1\n init: mmrm\n seed: 1763605658\n```\n\n\n:::\n:::\n\n\n### Generate imputed datasets\n\nThe next step is to use the parameters from the imputation model to generate the imputed datasets. This is done via the `impute()` function. The function only has two key inputs: the imputation model output from `draws()` and the `references` groups relevant to reference-based imputation methods. Since we are using the MAR approach here, we can set it to NULL.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nimputeObj <- rbmi::impute(draws = drawObj, references = NULL)\n```\n:::\n\n\nIn case we would like to access the imputed datasets, we can use the `extract_imputed_dfs()` function. For example, the imputed values in the 10th imputed dataset for patient 1513 are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nimputed_dfs = rbmi::extract_imputed_dfs(imputeObj)\nMI_10 = imputed_dfs[[10]]\nMI_10$PATIENT_ID = dat_expand$PATIENT\n\nMI_10 |>\n dplyr::filter(PATIENT_ID == \"1513\") |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n
PATIENTGENDERTHERAPYRELDAYSVISITBASVALHAMDTL17CHANGEPATIENT_ID
new_pt_5MDRUG7419245.0000001513
new_pt_5MDRUGNA519NA5.0547791513
new_pt_5MDRUGNA619NA-7.9878851513
new_pt_5MDRUGNA719NA-8.0940471513
\n
\n```\n\n:::\n:::\n\n\n### Analyse imputed datasets\nThe next step is to run the analysis model on each imputed dataset. This is done by defining an analysis function and then calling the `analyse()` function to apply this function to each imputed dataset. The `ancova()` function provided by the `rbmi` package which fits a separate ANCOVA model for the outcomes from each visit is used.\n\nThe `ancova()` function uses the `set_vars()` function which determines the names of the key variables within the data and the covariates (in addition to the treatment group) for which the analysis model will be adjusted.\n\nNote: In Appendix 1 below we show how you can easily use a different analysis method (e.g., mmrm).\n\n\n::: {.cell}\n\n```{.r .cell-code}\nvars_analyse <- rbmi::set_vars(\n outcome = \"CHANGE\",\n visit = \"VISIT\",\n subjid = \"PATIENT\",\n group = \"THERAPY\",\n covariates = c(\"BASVAL\", \"GENDER\")\n)\n\nanaObj <- rbmi::analyse(\n imputations = imputeObj,\n fun = ancova,\n vars = vars_analyse\n)\n```\n:::\n\n\n### Pool results\n\nFinally, the `pool()` function can be used to summarise the analysis results across multiple imputed datasets to provide an overall statistic with a standard error, confidence intervals and a p-value for the hypothesis test of the null hypothesis that the effect is equal to 0. Since we used `method_bayes()`, pooling and inference are based on Rubin’s rules.\n\nHere, the treatment difference at visit 7 is of interest. Since we set PLACEBO as the first factor in the variable `THERAPY` this corresponds to `ref`, whereas DRUG corresponds to `alt`.\n\n\n::: {.cell}\n\n```{.r .cell-code}\npoolObj <- rbmi::pool(anaObj, conf.level = 0.95, alternative = \"two.sided\")\n\npoolObj |>\n data.frame() |>\n dplyr::filter(grepl(\"7\", parameter)) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
parameterestselciucipval
trt_7-2.8330371.1210994-5.049255-0.61681921.260028e-02
lsm_ref_7-4.8182260.7799966-6.360190-3.27626256.549593e-09
lsm_alt_7-7.6512630.7932569-9.219250-6.08327632.821595e-17
\n
\n```\n\n:::\n:::\n\n\n\n## rbmi: MNAR CR approach\nThe following changes need to be made in the code above to apply the Copy Reference (CR) approach in `rbmi`. For `dat_ice` the strategy need to be changed to CR. In the `impute()` step the `references` need to be specified. Here we set the reference for the DRUG group to PLACEBO.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_expand |>\n arrange(PATIENT, VISIT) |>\n filter(is.na(CHANGE)) |>\n group_by(PATIENT) |>\n slice(1) |>\n ungroup() |>\n select(PATIENT, VISIT) |>\n mutate(strategy = \"CR\")\n\nimputeObj <- rbmi::impute(\n drawObj,\n references = c(\"PLACEBO\" = \"PLACEBO\", \"DRUG\" = \"PLACEBO\")\n)\n```\n:::\n\n\n\n\nThe results for M=500 imputed datasets using the MNAR CR approach are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\npoolObj |>\n data.frame() |>\n dplyr::filter(grepl(\"7\", parameter)) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
parameterestselciucipval
trt_7-2.3933181.1092180-4.585247-0.20138973.256562e-02
lsm_ref_7-4.8176610.7796982-6.358748-3.27657406.224847e-09
lsm_alt_7-7.2109790.7955835-8.783369-5.63858917.801168e-16
\n
\n```\n\n:::\n:::\n\n\n## rbmi: MNAR JR approach\nThe following changes need to be made in the code above to apply the Jump to Reference (JR) approach in `rbmi`. For `dat_ice` the strategy need to be changed to JR. In the `impute()` step the `references` need to be specified. Here we set the reference for the DRUG group to PLACEBO.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_expand |>\n arrange(PATIENT, VISIT) |>\n filter(is.na(CHANGE)) |>\n group_by(PATIENT) |>\n slice(1) |>\n ungroup() |>\n select(PATIENT, VISIT) |>\n mutate(strategy = \"JR\")\n\nimputeObj <- rbmi::impute(\n drawObj,\n references = c(\"PLACEBO\" = \"PLACEBO\", \"DRUG\" = \"PLACEBO\")\n)\n```\n:::\n\n\n\n\nThe results for M=500 imputed datasets using the MNAR JR approach are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\npoolObj |>\n data.frame() |>\n dplyr::filter(grepl(\"7\", parameter)) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
parameterestselciucipval
trt_7-2.1434371.1269939-4.3708220.083947795.915888e-02
lsm_ref_7-4.8217020.7905712-6.384460-3.258944779.511139e-09
lsm_alt_7-6.9651390.8185475-8.583534-5.346744322.496563e-14
\n
\n```\n\n:::\n:::\n\n\n\n\n## rbmi: MNAR CIR approach\nThe following changes need to be made in the code above to apply the Copy Increments in Reference (CIR) approach in `rbmi`. For `dat_ice` the strategy need to be changed to CIR. In the `impute()` step the `references` need to be specified. Here we set the reference for the DRUG group to PLACEBO.\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndat_ice <- dat_expand |>\n arrange(PATIENT, VISIT) |>\n filter(is.na(CHANGE)) |>\n group_by(PATIENT) |>\n slice(1) |>\n ungroup() |>\n select(PATIENT, VISIT) |>\n mutate(strategy = \"CIR\")\n\nimputeObj <- rbmi::impute(\n drawObj,\n references = c(\"PLACEBO\" = \"PLACEBO\", \"DRUG\" = \"PLACEBO\")\n)\n```\n:::\n\n\n\n\nThe results for M=500 imputed datasets using the MNAR CIR approach are:\n\n\n::: {.cell}\n\n```{.r .cell-code}\npoolObj |>\n data.frame() |>\n dplyr::filter(grepl(\"7\", parameter)) |>\n gt()\n```\n\n::: {.cell-output-display}\n\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\n \n \n
parameterestselciucipval
trt_7-2.4743821.1080177-4.663971-0.28479182.703976e-02
lsm_ref_7-4.8145500.7842418-6.364835-3.26426567.837117e-09
lsm_alt_7-7.2889320.7952036-8.860613-5.71725164.368826e-16
\n
\n```\n\n:::\n:::\n\n\n\n## Summary of results\nIn the table we present the results of the different imputation strategies (and with varying number, *M*, of multiple imputation draws). Note that the results can be (slightly) different from the results above due to a possible different seed. The table show the contrast at Visit 7 between DRUG and PLACEBO [DRUG - PLACEBO]:\n\n| Method | Estimate | SE | 95% CI | p-value |\n|-------------------|----------|-------|------------------|---------|\n| Complete Case | -2.829 | 1.117 | -5.035 to -0.622 | 0.0123 |\n| MI - MAR (M=500) | -2.833 | 1.120 | -5.046 to -0.620 | 0.0125 |\n| MI - MAR (M=2000) | -2.837 | 1.118 | -5.047 to -0.627 | 0.0122 |\n| MI - MAR (M=5000) | -2.830 | 1.123 | -5.040 to -0.610 | 0.0128 |\n| MI - MNAR CR (M=500) | -2.377 | 1.119 | -4.588 to -0.167 | 0.0352 |\n| MI - MNAR CR (M=2000) | -2.391 | 1.110 | -4.585 to -0.198 | 0.0328 |\n| MI - MNAR CR (M=5000) | -2.394 | 1.112 | -4.592 to -0.197 | 0.0329 |\n| MI - MNAR JR (M=500) | -2.169 | 1.134 | -4.411 to 0.072 | 0.0577 |\n| MI - MNAR JR (M=2000) | -2.146 | 1.135 | -4.389 to 0.097 | 0.0606 |\n| MI - MNAR JR (M=5000) | -2.148 | 1.135 | -4.390 to 0.095 | 0.0603 |\n| MI - MNAR CIR (M=500) | -2.495 | 1.113 | -4.695 to -0.295 | 0.0265 |\n| MI - MNAR CIR (M=2000) | -2.469 | 1.116 | -4.674 to -0.263 | 0.0285 |\n| MI - MNAR CIR (M=5000) | -2.479 | 1.112 | -4.676 to -0.282 | 0.0273 |\n\n## Approximate Bayesian\n\nIn the `draws()` function it is possible to specify other methods. For example, the approximate Bayesian MI `method_approxbayes()` which is based on bootstrapping. `draws()` returns the draws from the posterior distribution of the parameters using an approximate Bayesian approach, where the sampling from the posterior distribution is simulated by fitting the MMRM model on bootstrap samples of the original dataset.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmethod <- rbmi::method_approxbayes(\n covariance = \"us\",\n threshold = 0.01,\n REML = TRUE,\n n_samples = 500\n)\n```\n:::\n\n\nIn the table we present the results of the approximate Bayesian approach for a CR imputation strategy. The table show the contrast at Visit 7 between DRUG and PLACEBO [DRUG - PLACEBO]:\n\n| Method | Estimate | SE | 95% CI | p-value |\n|-------------------|----------|-------|------------------|---------|\n| MI - MNAR CR (M=500) | -2.415 | 1.109 | -4.617 to -0.210 | 0.0320 |\n| MI - MNAR CR (M=2000) | -2.403 | 1.112 | -4.600 to -0.205 | 0.0323 |\n\n## Discussion\n\nA note on computational time: The total running time (including data loading, setting up data sets, MCMC run, imputing data and analysis MI data) for M=500 was about 26 seconds on a personal laptop. It increased to about 92 seconds for M=2000. Computational time was similar across different imputation strategies.\n\nWith a small number of `n_samples` in `method_bayes()` a warning could pop-up \"The largest R-hat is 1.08, indicating chains have not mixed. Running the chains for more iterations may help\". Increasing the number of `n_samples` will mostly solve this warning. For example, for this data example, this message is received when setting `n_samples` equal to a number below 100.\n\n## Appendix 1: mmrm as analysis model\n\nIn the `analyse()` function (at the moment of writing) the only available analysis function is `ancova`. However, the user is able to specify its own analysis function. See the `analyse()` function for more details.\n\nAnother possibility (although, not the most efficient) is to implement a for loop in which the model is fit on each imputed dataset. The obtained results could then be pooled using Rubin's rule. For example, suppose an MMRM should be fit on each imputed dataset:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm_analyse_mi_function <- function(Impute_Obj) {\n # create all imputed datasets\n imputed_dfs = rbmi::extract_imputed_dfs(Impute_Obj)\n\n # create empty vectors to store mmrm analysis results\n est_vec = sd_vec = df_vec = NULL\n\n # for loop to save estimates per imputation\n for (k in 1:length(imputed_dfs)) {\n temp_dat = imputed_dfs[[k]]\n mmrm_fit_temp = mmrm::mmrm(\n CHANGE ~\n 1 +\n THERAPY +\n VISIT +\n BASVAL * VISIT +\n THERAPY * VISIT +\n GENDER +\n us(VISIT | PATIENT),\n data = temp_dat,\n reml = TRUE\n )\n em = emmeans::emmeans(\n mmrm_fit_temp,\n specs = trt.vs.ctrl ~ THERAPY * VISIT,\n at = list(VISIT = \"7\"),\n level = 0.95,\n adjust = \"none\",\n mode = \"df.error\"\n )\n est_vec[k] = summary(em$contrasts)$estimate\n sd_vec[k] = summary(em$contrasts)$SE\n df_vec[k] = summary(em$contrasts)$df\n }\n\n # summarize results using rubin's rule\n rr = rbmi:::rubin_rules(ests = est_vec, ses = sd_vec, v_com = mean(df_vec))\n rr$se_t = sqrt(rr$var_t)\n rr$t.stat = rr$est_point / sqrt(rr$var_t)\n rr$p_value = 2 * pt(q = rr$t.stat, df = rr$df, lower.tail = TRUE)\n\n return(rr = rr)\n}\n```\n:::\n\n\nThe following code then performs the analysis and pooling\n\n\n::: {.cell}\n\n```{.r .cell-code}\nmmrm_analyse_mi_function(Impute_Obj = imputeObj)\n```\n:::\n\n\nIn the table we present the results of the Bayesian approach for a CR imputation strategy with an MMRM analysis model. The table show the contrast at Visit 7 between DRUG and PLACEBO [DRUG - PLACEBO]:\n\n| Method | Estimate | SE | 95% CI | p-value |\n|-------------------|----------|-------|------------------|---------|\n| MI - MNAR CR (M=500) | -2.415 | 1.109 | -4.607 to -0.223 | 0.0310 |\n| MI - MNAR CR (M=2000) | -2.388 | 1.111 | -4.584 to -0.193 | 0.0332 | \n\n\n## Reference\n\n[Carpenter JR, Roger JH & Kenward MG (2013)](https://doi.org/10.1080/10543406.2013.834911). Analysis of Longitudinal Trials with Protocol Deviation: A Framework for Relevant, Accessible Assumptions, and Inference via MI. *Journal of Biopharmaceutical Statistics* 23: 1352-1371.\n\n[Gower-Page C, Noci A & Wolbers M (2022)](https://doi.org/10.21105/joss.04251). rbmi: A R package for standard and reference-based multiple imputation methods. *Journal of Open Source Software* 7(74): 4251. \n\n[rbmi: Reference Based Multiple Imputation](https://cran.r-project.org/web/packages/rbmi/index.html)\n\n[rbmi: Quickstart](https://cran.r-project.org/web/packages/rbmi/vignettes/quickstart.html)\n\n[Roger J (2022, Dec 8)](https://baselbiometrics.github.io/home/docs/talks/20221208/5_JamesRoger%2020121118.pdf). Other statistical software for continuous longitudinal endpoints: SAS macros for multiple imputation. *Addressing intercurrent events: Treatment policy and hypothetical strategies*. Joint EFSPI and BBS virtual event.\n\n[Rubin DB (1976)](https://doi.org/10.1093/biomet/63.3.581). Inference and Missing Data. *Biometrika* 63: 581–592.\n\n[Rubin DB (1987)](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316696). *Multiple Imputation for Nonresponse in Surveys*. New York: John Wiley & Sons.\n\n[von Hippel PT & Bartlett JW (2021)](https://doi.org/10.1214/20-STS793). Maximum likelihood multiple imputation: Faster imputations and consistent standard errors without posterior draws. *Statistical Science* 36(3): 400–420. \n\n[Wolbers M, Noci A, Delmar P, Gower-Page C, Yiu S & Bartlett JW (2022)](https://onlinelibrary.wiley.com/doi/full/10.1002/pst.2234). Standard and reference-based conditional mean imputation. *Pharmaceutical Statistics* 21(6): 1246-1257.\n\n::: {.callout-note collapse=\"true\" title=\"Session Info\"}\n\n::: {.cell}\n::: {.cell-output .cell-output-stdout}\n\n```\n─ Session info ───────────────────────────────────────────────────────────────\n setting value\n version R version 4.5.2 (2025-10-31)\n os Ubuntu 24.04.3 LTS\n system x86_64, linux-gnu\n ui X11\n language (EN)\n collate en_US.UTF-8\n ctype en_US.UTF-8\n tz Europe/London\n date 2026-03-10\n pandoc 3.6.3 @ /home/michael/.positron-server/bin/f3aae65e0a1a11d39226cd884520f49301daef82/quarto/bin/tools/x86_64/ (via rmarkdown)\n\n─ Packages ───────────────────────────────────────────────────────────────────\n ! package * version date (UTC) lib source\n P assertthat 0.2.1 2019-03-21 [?] RSPM (R 4.5.0)\n P backports 1.5.0 2024-05-23 [?] RSPM (R 4.5.0)\n boot 1.3-32 2025-08-29 [2] CRAN (R 4.5.2)\n P broom 1.0.12 2026-01-27 [?] RSPM (R 4.5.0)\n P callr 3.7.6 2024-03-25 [?] RSPM (R 4.5.0)\n P checkmate 2.3.4 2026-02-03 [?] RSPM (R 4.5.0)\n P cli 3.6.5 2025-04-23 [?] RSPM (R 4.5.0)\n codetools 0.2-20 2024-03-31 [2] CRAN (R 4.5.2)\n P curl 7.0.0 2025-08-19 [?] RSPM (R 4.5.0)\n P digest 0.6.39 2025-11-19 [?] RSPM (R 4.5.0)\n P dplyr * 1.2.0 2026-02-03 [?] RSPM (R 4.5.0)\n P emmeans * 2.0.1 2025-12-16 [?] RSPM (R 4.5.0)\n P estimability 1.5.1 2024-05-12 [?] RSPM (R 4.5.0)\n P evaluate 1.0.5 2025-08-27 [?] RSPM (R 4.5.0)\n P farver 2.1.2 2024-05-13 [?] RSPM (R 4.5.0)\n P fastmap 1.2.0 2024-05-15 [?] RSPM (R 4.5.0)\n P forcats 1.0.1 2025-09-25 [?] RSPM (R 4.5.0)\n P foreach 1.5.2 2022-02-02 [?] RSPM (R 4.5.0)\n P fs 1.6.6 2025-04-12 [?] RSPM (R 4.5.0)\n P generics 0.1.4 2025-05-09 [?] RSPM (R 4.5.0)\n P ggplot2 4.0.2 2026-02-03 [?] RSPM (R 4.5.0)\n P glmnet 4.1-10 2025-07-17 [?] RSPM (R 4.5.0)\n P glue 1.8.0 2024-09-30 [?] RSPM (R 4.5.0)\n P gridExtra 2.3 2017-09-09 [?] RSPM (R 4.5.0)\n P gt * 1.3.0 2026-01-22 [?] RSPM (R 4.5.0)\n P gtable 0.3.6 2024-10-25 [?] RSPM (R 4.5.0)\n P haven 2.5.5 2025-05-30 [?] RSPM (R 4.5.0)\n P hms 1.1.4 2025-10-17 [?] RSPM (R 4.5.0)\n P htmltools 0.5.9 2025-12-04 [?] RSPM (R 4.5.0)\n P htmlwidgets 1.6.4 2023-12-06 [?] RSPM (R 4.5.0)\n P inline 0.3.21 2025-01-09 [?] RSPM (R 4.5.0)\n P iterators 1.0.14 2022-02-05 [?] RSPM (R 4.5.0)\n P jinjar 0.3.2 2025-03-13 [?] RSPM (R 4.5.0)\n P jomo 2.7-6 2023-04-15 [?] RSPM (R 4.5.0)\n P jsonlite 2.0.0 2025-03-27 [?] RSPM (R 4.5.0)\n P knitr 1.51 2025-12-20 [?] RSPM (R 4.5.0)\n P labelled * 2.16.0 2025-10-22 [?] RSPM (R 4.5.0)\n lattice 0.22-7 2025-04-02 [2] CRAN (R 4.5.2)\n P lifecycle 1.0.5 2026-01-08 [?] RSPM (R 4.5.0)\n P lme4 1.1-38 2025-12-02 [?] RSPM (R 4.5.0)\n P loo 2.9.0 2025-12-23 [?] RSPM (R 4.5.0)\n P magrittr 2.0.4 2025-09-12 [?] RSPM (R 4.5.0)\n MASS 7.3-65 2025-02-28 [2] CRAN (R 4.5.2)\n Matrix 1.7-4 2025-08-28 [2] CRAN (R 4.5.2)\n P matrixStats 1.5.0 2025-01-07 [?] RSPM (R 4.5.0)\n P mice * 3.19.0 2025-12-10 [?] RSPM (R 4.5.0)\n P minqa 1.2.8 2024-08-17 [?] RSPM (R 4.5.0)\n P mitml 0.4-5 2023-03-08 [?] RSPM (R 4.5.0)\n P mmrm * 0.3.17 2026-01-08 [?] RSPM (R 4.5.0)\n P multcomp 1.4-29 2025-10-20 [?] RSPM (R 4.5.0)\n P mvtnorm 1.3-3 2025-01-10 [?] RSPM (R 4.5.0)\n nlme 3.1-168 2025-03-31 [2] CRAN (R 4.5.2)\n P nloptr 2.2.1 2025-03-17 [?] RSPM (R 4.5.0)\n nnet 7.3-20 2025-01-01 [2] CRAN (R 4.5.2)\n P otel 0.2.0 2025-08-29 [?] RSPM (R 4.5.0)\n P pan 1.9 2023-12-07 [?] RSPM (R 4.5.0)\n P pillar 1.11.1 2025-09-17 [?] RSPM (R 4.5.0)\n P pkgbuild 1.4.8 2025-05-26 [?] RSPM (R 4.5.0)\n P pkgconfig 2.0.3 2019-09-22 [?] RSPM (R 4.5.0)\n P processx 3.8.6 2025-02-21 [?] RSPM (R 4.5.0)\n P ps 1.9.1 2025-04-12 [?] RSPM (R 4.5.0)\n P purrr 1.2.1 2026-01-09 [?] RSPM (R 4.5.0)\n P QuickJSR 1.9.0 2026-01-25 [?] RSPM (R 4.5.0)\n P R6 2.6.1 2025-02-15 [?] RSPM (R 4.5.0)\n P rbibutils 2.4.1 2026-01-21 [?] RSPM (R 4.5.0)\n P rbmi * 1.6.0 2026-01-23 [?] RSPM (R 4.5.0)\n P RColorBrewer 1.1-3 2022-04-03 [?] RSPM (R 4.5.0)\n P Rcpp 1.1.1 2026-01-10 [?] RSPM (R 4.5.0)\n P RcppParallel 5.1.11-1 2025-08-27 [?] RSPM (R 4.5.0)\n P Rdpack 2.6.6 2026-02-08 [?] RSPM (R 4.5.0)\n P reformulas 0.4.4 2026-02-02 [?] RSPM (R 4.5.0)\n renv 1.0.10 2024-10-05 [1] RSPM (R 4.5.2)\n P rlang 1.1.7 2026-01-09 [?] RSPM (R 4.5.0)\n P rmarkdown 2.30 2025-09-28 [?] RSPM (R 4.5.0)\n rpart 4.1.24 2025-01-07 [2] CRAN (R 4.5.2)\n P rstan 2.32.7 2025-03-10 [?] RSPM (R 4.5.0)\n P S7 0.2.1 2025-11-14 [?] RSPM (R 4.5.0)\n P sandwich 3.1-1 2024-09-15 [?] RSPM (R 4.5.0)\n P sass 0.4.10 2025-04-11 [?] RSPM (R 4.5.0)\n P scales 1.4.0 2025-04-24 [?] RSPM (R 4.5.0)\n P sessioninfo 1.2.2 2021-12-06 [?] RSPM (R 4.5.0)\n P shape 1.4.6.1 2024-02-23 [?] RSPM (R 4.5.0)\n P StanHeaders 2.32.10 2024-07-15 [?] RSPM (R 4.5.0)\n P stringi 1.8.7 2025-03-27 [?] RSPM (R 4.5.0)\n P stringr 1.6.0 2025-11-04 [?] RSPM (R 4.5.0)\n survival 3.8-3 2024-12-17 [2] CRAN (R 4.5.2)\n P TH.data 1.1-5 2025-11-17 [?] RSPM (R 4.5.0)\n P tibble 3.3.1 2026-01-11 [?] RSPM (R 4.5.0)\n P tidyr * 1.3.2 2025-12-19 [?] RSPM (R 4.5.0)\n P tidyselect 1.2.1 2024-03-11 [?] RSPM (R 4.5.0)\n P TMB 1.9.19 2025-12-15 [?] RSPM (R 4.5.0)\n P utf8 1.2.6 2025-06-08 [?] RSPM (R 4.5.0)\n P V8 8.0.1 2025-10-10 [?] RSPM (R 4.5.0)\n P vctrs 0.7.1 2026-01-23 [?] RSPM (R 4.5.0)\n P withr 3.0.2 2024-10-28 [?] RSPM (R 4.5.0)\n P xfun 0.56 2026-01-18 [?] RSPM (R 4.5.0)\n P xml2 1.5.2 2026-01-17 [?] RSPM (R 4.5.0)\n P xtable 1.8-4 2019-04-21 [?] RSPM (R 4.5.0)\n P yaml 2.3.12 2025-12-10 [?] RSPM (R 4.5.0)\n P zoo 1.8-15 2025-12-15 [?] RSPM (R 4.5.0)\n\n [1] /home/michael/source/personal/CAMIS/renv/library/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu\n [2] /opt/R/4.5.2/lib/R/library\n\n P ── Loaded and on-disk path mismatch.\n\n──────────────────────────────────────────────────────────────────────────────\n```\n\n\n:::\n:::\n\n\n:::\n\n", "supporting": [ "rbmi_continuous_joint_files" ], diff --git a/_freeze/R/rbmi_continuous_joint/figure-html/explore data 2-1.png b/_freeze/R/rbmi_continuous_joint/figure-html/explore data 2-1.png index 43f1472a2024bc1b8fc374840da4af2b99a366bf..89dc44f52331be4a208ceffb0d09a3e0e1fe66b9 100644 GIT binary patch literal 30156 zcmeFZcT|(>w=NoFDV9Z1M3z)p78C?jq!XebAR-_FA}y#irAiAuK?PBeA__=XkzRtd z&_Yy7=uvtLy+{HHEeYxOW9{D=_ndL=82g`n@7Q;r?HG=Px1i+PWx}@AHQ)e2_9rWnN1VC`2KXxlFLaNlZ~51Z$H%0YGP)tq^F-x zSODe+mCf88m)d)?f&W7rcsUap$p->O{vosh{E-COxA)z#Yu|W5plkaM1_OWG1n~$0 zFNaRe@&HTa`R|+lTM+-P5&!*Y96JS>IuHtGx1zl%f#sC&}^!D@F+E~3M>H{ z@pql&xvw`&RLgqWdl}2xV$H9vdM3Jn>%u$6h-PGzUQ7XhMveQ-SG`GBegbN(3j5KY ztMw4(Qin~ANZFna{oP@#)ZjxXQTh}-9+Svg+Kp&Lxepb&Gzo~Ko-$(c{u_6c#_V#SJxDyFj|j$$`7#o=aaM6N7bu$*X_(Eirr@7 zu)SrS)CZvZK8J!M?ca8N@le>dvM;Ae-B-rvW_(%D7%uaLeby^8gx_cqp5dV5YCtF0c~Lvx z324(>1kWcVsCIbw6Eklf5-jk!67M-Zlz3KQAbN)mrwQaY{825YaCuQ{3LfW4AC6)^wg zwA;OQnc+%eBxK5O)Mm20e#*3Yu?vzDIP->}eoKW+A!8%iLwC;kx>lU%pi8GET0K~KjS`H9VgqO!|HXgTe0q!;*u?BYXh}7jiBbskkN#F zJe9D0Qm3q0B*J&QtGk3_tLE2x`s&7E<#ppahb%u6?HQp9Rm*O#9#OiD6p+F6bRo1#-Cm?f6^AZF zr`=i*8lE8st*lQ3_gR}y8yK>df|>wnf{9(&nJ)aJc(DNRR>q1fCSdUOqzdjx{vejM zJ|?M(cmF0$4s)s)4+c#c2%HM?4C@9X*MF^K<(Xnu`}ETZE-79N$B;oX)!kx{?|OS@ zYv!L96>-F;JL^i^5G`pDd1C};)y)b}6PRW_`GAL3)=`zomB{Q^18w`d#{*5u1ALxu zp|mH4^j-fm-`(u4t;tMmtE&`Z~LCHR}HNNzvcMhK6|`oRYa!H zF|gQmQi_|KCZIv08uxQr`-GP-re7fpT+ic9HBYt}N=5Wu))Sbhi*M@VS-hyc0n+$# zee@GIBvA+2u#m1n^Jt8qE*-4nOZooC)S!jzGt8ZE54C$&T@XqkfjqF6Fq6?}E8hIRW zsQ0-W(cn7rlh?a)cTMUfp8j$WF2H5%{HC~3nC%DTPh4(W9T+Erty$!reu)Y4{MUw% zIY{Kx=~*5>%aUYr{b@UlwiZFs8|=X_Mg2_~t%e*rxk*}rNqC=r zq(@l>gWbeT7aGZd7ZQv)+t>T}z$H2@>;XHS5&`pUXao(~b72NzW=?FyL7v1tKVQAr zzD*k^Pnlb!MMx6qldR>~>+4%r9H)XhUZZY1tT)F8&-J7U>MVSxrwge}FhB6DM`#rr z!K*9yvS1WmpT4Y3+s$D&$p)rt-(kPi+WprK0*`;8K7XSmCHIi9AsaS&FJ!eriADKY z9;7{27nEij-xNzoG~jN&f5;A_5$b&74)EIsj1gXit*G=S&Y#6)2mQVv7|}9ufwZKI z2SY`I)c7Gj~CNX|h?fG_#;(@O~x~7e|uz{pH=~U!sp` zT+ll|HivvCn^pZpvBeg&D2eTQU zxT2_FcZJ!ih6U@)j}R;BXLlT>rh`!BY95pIm#i#YT$fEU87?joy6;inp=sbyu91cf zStjZL*tC(0?=rIw+(}p`4A6Zw64-x zTHJE06Ll6+QK^eR(KrsO=;Omold!t+nw1K9?o|g>#4>hJCxiOr`#pN8PLt7!166D7 zw_~pFj;Yr@v~Xxe@uV+iX1Zm@My)rTU5v%HutpXIA#~6Fw7AsNi^#J6sOBGy{9i6_J{qS+=lQj+N_96&)&Vsykkur4=&2(_IIKfA!@pzNfXB zyece<`1PQAy{V@2SNoqclcQS8u6do1}2sGhIv2SD4H0Gl}B_e#c;p0-bbla-A z!%YjT{{3(|UcRmtQ~@i=(JHqf+toGrPruL;a}B$AXp@paAMay|0PjJpMZbXWVnD2V zChq^sn07YCNaTdb@P!SHE$|}nPGot~c}7nTV`^^x5Ly@W_6+QHWNtMOetRbJLM@${pH!JzNe=6us5mwG!YDHMyOXE{6Nf6@QkB?1aV19e^+t3lO=?nMoYRi3hlA9a zIdNfXoz$nvAKh6ZyoxBNeRquMdtp4WbJ8sSq=l4iuJXD~~1|y0&ifaR0v((pTsTCfUt*SDiIZJiUATrj_;3m*29ICV$_cQ!;BG z@A9tvv``Rdf3y1a`hp7)e!1)^&oKtr*Se4Pp=3H2TwUVMv>EID_I z6AHCemylY!-&j(I!yK8M(}rq&)Jw^2d#hiZzU;j>=c#{}{~*QkCdpGF^qPBC=4?q# zb(ozN1+^B`{luZvLVRti`&?XZ3_?@Bv_& zSuOR6BYTJP;1Kv)g?fogD_=KaD)&4AFPa>>UNNXfM{8g>(JkHHI(&+8b#a-23W>*U+_U$hWYsFoGnNt7IzE*Kw>B0s2 z+w40IQ1;-Pvq$Gkmd>6%3%N4eBR2r~_4JsGgZaI?cehUS!b7YSv+>qmdn>&7=^uXR z=~1mnd-J-0uEgx|H~(mZvIlCuq{_tw;`dI!Sd$`tVu^j@v@JKkCoSckGkj51zG$=( zdOe}+qu{4T&nLx_!jZPe{*7b;%*p>mFUj8KsVv!fZe#}8WM#h7;aowwtdxh*v>q_) z^v9(JLo59H#hr2_xYR|nwBxm_ReHH_CgpLuyjlYD(-mV#_}u0SG|Y#P6;Pc3hCvpS z9JGdT>4VdVzqHPQ`k(Q2>bnf)HZSD{X;V97qhnYeJL8!9tM|Q!Wl!TIf`4FU>{)$* zou8kpHgOmenTp`>ZHCQBB0~mxW$AmSw*+U%u6jZOL0-qg?muDMj(@Q;l?d+GD9)b= z&oa=LWMt8hh-?gk`NQ)~V1zlH9v3vdF%tu2JRPqgZrHFdICIgjeeyETuIXiv#+mBfS`R_|m+9DB-Nc>D%@lbEbuHZ?&i!!F z&;cfk-Iu%;uUR_as2%1@=09TXnf830!-7*#i7v};Bqnq)FHKH}cxrD@6XmMZ2y1fL ze$20Ss;2rf0HhLQ+@aq}R6uNRLHRwxQ>sp09{HSE>Fks{$;E{GO0xmdG6R4BRcA zQ(solZf)h`5YfR;1_Hwy`@Mp+IEkb7+_dVLwb~~OmdCGb=V%0VwhMQU&?BoAp;tJx z2!Y1Mbez1$g!5S^B@O0u`Y-^Hf1P9I!p0#S-*F@#sm7OmzRgP6AjfY=diTFxf$#xg>?Hyd;|~LP~2t-1)jd z#h*36lI_NIo`b$M-c?yuLKA>37MU{H{Z0N;eA~B5S|V!Jx`ZbSbPvk-w6>DdH~V;d zAdUWaVxFImDw>{F>%ZydG-YF{$w4_B{w$;moJcF@>U_%TkyJ#}p({s78a2MhbzdD; zh0tGzI6{(a7*rLePtswTFB#I8A3dz-J74I=p+y!H3ju+m`^q4Pz4K+#z;2DJL2@Xa7w|*_K!kPWA|ik!#@)%! zJD?K3cfX!NM8mDZLI{TP4G2Tb^Ylnrxpw;<=^gtWw)GAccZ6})(MzQeTkxuxooJNM zXU$0m4s=OQPyw-;rAU{FVkxemQ|{v?$9KXCLuz5@ z4=jW%IDWV*scSjr@jI!a6k3FN+;p9{X))#wIO2zKuGR33XVNs}LI(#nQ#TQuQ z+E-?WN6Wz@wsr0~hh;3Po<3}E{A-OdS8B572k!mj2QIwUyQ%VI#JLZ}DM3MlL!LP$$=`Q9@hWfa3T*F^W&?;)KNZ+`6Ugd$72m&tuGa=Jq+CTBfNAzNz`+DL~cd z2a*S-#+@Cx{&@%bGES6jK49dD%J!PZB)QiOdgDiXnL=CszM1X$doJAU8}DLlV<13} z1P*5esTw18=67)zxl#oWxl)2lPihRHoBDz7jp7uXg=Nj7xag7+l{8svH06h}@q*Bn z7OBdgDW+q*;9?9!j8R4B(^PF<5k?ejCrdBx$!ZahRU;ZUQ(?-FVfCq&T39Za;1D@2 z@bq!uEco}%!uG0hbW+ZCSMO3OI4PYyr7Yj=9)m|Ai^~R-3voBGGGA;_4hwg^Ax?pI zSCP4gKZP#uoGcX$r@a4)>$?SPsxh_d7LMiYsu`8!o??-Cak$K-+VWt!OD1N zY+A+Eb}(r{5PkP(OcG|vTczRvcX?ti5H(iiZ&J%pPMJZw(Ex39)MoBzmG&nTK!O+F z(n|G!M|3QLIjrV9w&ZOr9nU3KMBmIpQzxsYBkcn_dG5AL>-QlKoNDchU0WyNo)jX^nHrA z*S*V|j%qD;q+*h0d{GWsi{G#-XO9d`w=x#TMBJ&zn|=>h2(IQ|6?Umg!2qE}uv?5uxz1|W)wp8Uj{ zky9hE+1D{Ox1;NwS&0_(cG|ZH130WBV$BHEe=|l%#53gD zOoE5!+p1!eM9@^9@_@`@EreCi7I$-693QO5y?&v!7+vo-Fe5-j{CZ^#vAJv2@SAPE z@@La-doo}It0M#^T;k)Xg)3kt^SWru9kG}qPCr&))mE>Z6P~|^Poc-L6?TV{PG3>> z20Kp*OR@dho&DaYrFDN9l-Jq+IaG@sIA1>lqswJqadCuGS7U^wnWJaZ1Q9+)=LF1K z9gv|eYvT?cAGFvurQu{`7%5=d`0O4pYz~#{{-P|eYEv@)9o5YIcG(ek!di*>Xt_|j z`3fN;4jKA85AeK}J$l~R>WtrFN>UvE6A1cGCDh5M*j|Fje9Rb8gBIIC*tav1xJHdf6H)Q&PRufJ3uZB#CZb8{-cFdo%I~ks#x&{rYS}g zlv!y`__(UPzNIAwyXv&;Ql++5CKo|_311(rR}39`%shfsvU!sEK2d7ulfib+)JD)N;cX6*E!Zma~VdT?t7}!;I9eJqg`a|gT7bEykX#~YWX-2uGK=_^B zLGu@ihW1+21`nD?dMS!pf?B$eHxok1fk>gPQT&<5mzIW~O!&sM_BXlxP);LF6?J$W z6RDY|lpS3#%$}-Uww}Y~?CJ5>D5*%5wz%Bp#V?a?H4~mh2c{J4r+}A#O-V~)Z|~6+ z7U|PjuO`nX>~~&<=kSl@W%51B*03RW0bh_Ex>hsui47y(hgc^D4FThwS^@GHd#PLF z-~W5k_UYx+mLdDC<)pB5dH}_{ z!Oy6v#Mdnd^1EO>U7BKeI0)l9a2kv1dTTskr5$Y9^l;C*8|_*5t>oln>G4CuOT&=9 z`^RJ|Y@z>z6D?(bvxWI(j}C~q{9C+{X1z2(2`pU)PAKQ>4zSWBkoRf6B>C=CMfz9e zO!Mn(1B##P`aOHAy0SGx@2c}wh49wT&AE?n5HSJts)5+v#3|SP6tNNc&bJ~vGL~ry z3blJN@xNuSzx^v#+%x$F%}jm$I&9MUC1qJNhdU|j=4WoPmiWm1a@95Du+HAEz-?h6 z06FDbhXoMjQ+KRuAzm3i@#A(YzgDf!J?zmbvJg&O=W$#FPMR`NXqZ+C<#`eFr^ePt$;r{64JQRT1F31}kN=9a8s@FNh*MV!{g%|& z=P|3UVLQ@SgEv0v(Zil8pjqfSa&ksLX5V6vHI-_8< zdTYSA0KI$D*LSm%FDKN)lrD#?`jKh}kJ!l5q?FdcnUaE< zkAVG`1gNa$OPd8dI|Ez$J!}lD1ceq9*LLoF8?!d`>w97fpd$B=@CaTzIDDi~hkIx9 z)%-kPZuV8oqx`mRkEswE+)zMFd$$tZc(o5Ve{fU_m}z^57Z7%PXj5jv{;(kZE{Ha%Ro_b9des0epuP1$xnBp4<-eU$@i#C_ zl%b&raF}qv`L(K)TP+_$MuC<1ZTTSlHwyZ9q0xX23O$eH`{+6xrxQFk2Tb)=P-mMY z>d2?}Z!}On#;Q+l1)T@*44dLX3$Vi%RR|m_6g?% zc8QV#7i4i+QzO?C#Q;p_XE)t0>X@~D8C1zJ8ut3NDyl`Qw<=s7&TZz5l!>Sv=pn;) zLwK8PEA6VA8Re>Y*A*y!LhWKUgYiYgS43rkQ>4s*1&n{q&=1V0Be%4g+68&1WU!aT zb1$022r4A9VNelGc`fNW9uO`{$?Dm;7K)n(AS=3t*dD&H*_OXu;yU zLxOm<@Llxj-1Hg(zK~c$8mDzW1TG~s1tvzTmJX17!{l!bk zz@a&2fY-7z?CjILCGrtV?SG| zzo$C=mQyU1G8k(`B3R7=1ta9WPZ0@!X?l)*)35uf^Bbl#jZGqlkf0h3`UXZCG9*4vW_@)CYH}Teuq0uBn+s}us2PEsg#6=n#gZoXGfsEln zk6tP1tB-$IJ~UjLL^s3@mVa`eI+;yt^mYx+m)76k*SH3L`r)ucTrA%K6-Y0D-WAdT z9!F#qxvZCGiRMQ^9&cO{VMxvgFiPn^&m$jV9hibAVMCRz)UlecnT<%iTITKOk$V&W zilejMj#dszIhH@120pz7uU4Q+Tmp_wh7FspZy8^hctxls4hQ`N7Md{T({nI@CM6|hTuMLrBDA^QJkOx`{fqwE+Hg8h0D*4|tOh5wbvj=$ zPE+(6_+nJYmwYBVI(mJgwugEisTq0Smnn&f4)JCpKy#UYJlBP5|CX z{wtAF5_a$*vZup2?9}Vc#c?g`6vb8TSz6{=r4R7^pMyE|Ap|{c=iaIlwh#U9Sn|Jo z)!;u>PWP?sHM!MFX`)x33>Nraufy*9#`ye*WQWsH+sv~$GRSWiwD#4G43!l7@M@o5 zf|||)_FXn3zsYmp;X8Ikjy>DKto(z@Tm>X|!e@n)={eR6H^+AB!Do`YqpC7&B|J^+jkLv631bAUJL=9 zxi>S}VIy6Jv>BIoBbk!BF@^&`n-x>zLn7Io6u1)WI0`GBxOg z+q@6_-er%~wCd^Hjh_haG?0UQ=@@b|X4q|GAz#=>8~&4Q0Jo&}DJkSa7SJnh&9v~% zcxzhAMn#j-ls_+4g=ggSurdh(Bj=Im^;e+uLf?X&1*{?w;yWLRd!w0eXI^)Jud;$( zaJbB{1cF|uvV)i+Mh6FS@j*X$WT&Xq`P&koL&ev%Hx~UBn>W@L_}1{{F4vFG?>tXK zf~&~sPRqzGFC|Mv!1l+MLQ#a}iI}2FD;#ekcwy>DKeO*p4C!Zu_wo4>Ty9IfBL-q- z&*4%Nd1sj=oLR<0?98^Wi^Njx^a3gc;AU{ z@5mXpP!0!SieCJQ;x+`eBBe9JxFMo%($fx^UypgFrW4w;i5ClJ^T`(^vMASEIa>MQ ztMUjPX2I!>oL?Gb?o#f-Z?t1L$xWvGx%X`{?)BSCW;(v!i6R#3lsb)_&%f+YTAs`W z=^RJ;bl}QVz}na#+LyQC`=8oLR>NexZ^y3~5;`#EM0ulRz;sb!Dt3kL=u$Ch%MknT_=^DbCmAf1ghJLOc%2)9N6J-rfr6GmMxQIBe-#^i=Ro>ZA> zJ6xwe_l6f>vD@;M5+Z81N^|cau7FDIf~#CfpA2L#gE zAP(1MSzD8lzoj1aob)2b+s=&=v#lv8R_Bp@K8kQbP+ML9J0PHAp{Gi>8h5l-noUYt zIJlh}L!`7yF$+->PN^v#ELjlx{28bi@{T9_17Vn00P)uaF6XsXda_1!THC`H=K(IW zH#VPM{QO78h>d~1zP|ZE8RvoY2}vW+EZ1sS3yF3B$hpzbG`uQuE?djJw#VCHD>tO6 zgz>^!RxttP$;lpt?7ykaBj{`er+rW3&?mWN!He6Oh9(Th(bKVMiVo2Gmd95De-E`h zRWQ3C?;r*6okgbEKaefBC9h_q7}jG0O~o*;De)yX%w|2gebubPZ1S!rQ>UI*Ft!PMGpjoZ>)A{QP7JzQF(A5y(-kfgh?zl8l7PHtI z4LuoCG}n*i$KU+Hy=HiH<)2;JqMto5+nUJQj##!oB)M09lluBc@5BnYD1Qn(MI zl#Y9ui|F%6@qKkTOu{!f(Pw(`uGax}Slf_Bj00!jxmv{pqGyG{zlyONRi%_HBQ^$F zTjONI?@1Z*hp`A1O=gq5DQ|d2czsu^0KQB}#e2e8A#NeB!$b_o@TgnZ-W7`p9r?)% z3EL;Nd$Rb0&ejfG7IH~V_%ka33-z@{r8sH3w=nZ@%~kD25s3}NJSC{~CT-NPZ0A15 zAcnoYIcd5!{c{u&Nzn~7G6J;jRctJwOkp;TP*ERlWPi{0M9i$D#O`=?l+v{pY|mx<`LYWw7m1hIMgVsT(+yZey);c~I2C@vwEc(`VYv~sCgUrJ? z>4kmOpZS)+VJeIHur|yWbVJ8D7uks64N}VsV zMqWBB)3&rE;?baW)BbCCyO;5220tMa&gb_mk}tVztJo}5dFouF~n zH%4WrgZyyAsXyFY@tdW>-rP#<1hMsiO=V<4~%zhD=oOYI45EkzJ%zEHLeU?Ihtpw394dEXefX1Ta+# zu{e|r6lfAOXUzoBYD=r732SxBA#1C}Kz4KyFh6t$vnEorq*(c8R3qLDJ@-62Zm&4a zZ@~olA&RfFt~p+-7I#F(IklG&Zqv6vH%Hk6DEdosAN&+y9*+aTo{&#YKAd)R?WBZ3l`#J#I;10kZ$Nm z+v{p&6&=3Fz;j5v>z;{>A|#$e++R3JT0Kcd+&>Wm}o|1hyJ@pnJnh=3?73WGov_9J@vx9*3o^ z166ke%8u|5SAMM3LOZhMY6Gwl%y@#o!l@nR&?Y$dq}p<9 z%eqI71^1pMqBTAvA7J;>$%s2ubr9y<`zNbF&ax11ARg@VM`v9Tr6mG&;gxPt=t9d; z=?dS#p{zdn*&%RCghe2Kd!S^%9p)YxO#&E8%LlHsA%&>2Fef50foYkGYmm?;s=QQbQppERqVL1qLpm31Z>YPT;G=pXi_F)E2!unc&*i?&J zJI@Q?q?bF4vuKtUY=RAB-%s({%$Eq$Gff6ONlv*L`o`0*tE~v zNGk~dZfh9-IMIRwAIv=vP?5RBv#$&wJZ~oNu#{SEkA`c_l252G+Fu8?;K@60xoEGU z52Daa#yG28WPiME4c;>oPhCgs9C`tpuIx9{ybppOjuxjh?0gl(rE><97i9ZBRG%5F zv|=ivsncj<3;C62S*K^8QH-)*knG<#_^&6fc!l=&q^z($aP__)m=tJyFnL{+^#Q~4 ztufVo>`59Y8`ed=1gh%M2YAf?<2r>-#V)8%I)OT606wLONovab5&G$)HxNV#rSvS6 zR3%vpiKui_?;Ja==&;A#s^O^DfcJKP&8y3~AKyA$1x}4BvX!z<<(Jglsc0ermxBCm z!t3}zNp>d`Y=Mdd`5s*t8(_Tit0f}lx7NoW%~;#FT%FXr`9Z|zF{n&=pVWiw3HQ=+ zsj#)L)Fz3#IVvAe6xVf$8Gl*11D`w7%!U~{CIg{kM-Kv`8dbSqgL2u6`WW$rR%vND z_3hr1o;7LYX;)~7b9IGK0zWxMh@pitolqi&4m z5;qKepC+yQd#GqhwOng&!i|2`$#JQD*A)GZYWMLP?c4o_at?q#c-ugVS=K) znAJ4xa17b>iFboH1sv##GnH)0+Ik#PXSgqEKVTg`6U;~uIJ2DT@RT@cal$gfZrXS0 zN3j`~*c*YG-fjlyQlZ6krH-sxf1I9TbJ&kJ1HIOX*gZmfVg%B`X(el-BWP?sUsJ~_aBpmGUlUWI9nq|F21j}?5Gk?fB3S4rkg`tNkb^d z(47RW+^l=a(2XrC!#_Rg^Mh_dO7^R?k^+0>*$s~w`x7yit%`gc!u!)f>r+MH%N~N9 zm6yyJ=pD}H#sagJyh9Mt8tI#W{29r;puCWSc#|QIe>Qb?_Su(WI9b}wo@r_$@mUjZ z+RWV~3O;eHoPUNEl<*_?n9^}S*`^V{w|C|@)3U=xk0wftl8Dg;={OPtFQTLnr#JKz z8C(U}5a(5D|k?zUmpzl(qk{PS9W6(nCFK~lg zR1wKrR-&o<2SZ5rW-Vmv3vXt=?o9+|qv(+0n;ituWBS2ziM=tm(!2I!AoKWzyCPa2 zU)>kdU)!(pS^JQ|(;;91Y$(CeE^87-_T6F}N?=jGygip6^e-HeG$^_|4SbAtMWA zplG9C)j`d~IwbA6>bh-VYniA0*QcCh^&y~7X{C94WGq%l_xnf~Wqda#MltU|(|WB_ zAJ)YvIIOhvfo6Ut#O){2bzD2U&Ep*d; zmOVf0A$hpW@dL_T4*hsEdkJV(oXIIrc5HEes0D{(G!Gu_M5g=X)4)J|W zJi`OJ!#)`N;6-Klk8hlqfc4N_FKMOzmqk!iwN%)SU?yVw+^5*s*c8P1^Y<4fb^}6} zO!D(}ZVG@R-EV&5Jp?3pB%gM1?0hE4u`3;EZ89Z;^jDM9;S0+V84<+WVV0$)>nV<@ zDJjU+mG>!oRXKM`VZW5X${-LcK=BdZnX%Z{I7k0wX)d+<{QyweXNB0->u;Lt*cMpZ zU?;{T5{8O}qlI)~A-+bcSIY=vE-pq!PY%j_AqH(@$< z$^0^Mo-GT9(x#ubUqIWA6cJjA3(D0hPDXF3GlzSTRiCUxp5BY>lX++l<%MIb!x{cx$FraR2J z;C7bnNnnVzW_pKeJP2e7q27Jg*3}%Ecsl5m{{<-O7 z;1i%uP?5g);;4w#l#Fu7n4y1_%ng+~Jlw*j$)748b{#52HQ z-Ll)S++mpCGopdWrUbz_F_M4pb)bmG(zKr2QRwP|@pozt-st0s-2>ROX%bdqWaQ=1 zy%FGYN#{44t@&CU#bWN%x0)MzbA(444dVqHcg^q~`Z8Ly^y>dxDJ#tE-A~J z0l!n0ksDikZ=5*U1_K6h?@9m$5jhQ^H(wQyd5`fojS1Mv8&EKfShp)(I#Q>(!XYwM zjiT3@)s)upPkO+DkWH?sb~w^Yb4PuQ4nuE8# z>>sb4LAtBd?Y3GVMRI)C=!pr6P;F9tVMlq3j2z)7xrC3bQ;2P>=;Q<3r|CCdqho0! z-ZLIFy!_~Df3ZQfIKn60eCR>bpcH4V(%&(3t;<^HB*cldC1zq9xHdtvHk595Dz*Kk zbmrDfRY0k=RdqR{&eAf(f?5k@ShbOo$sW8Ky;U`nOxRycz*D-VE37~^CB)aGOQET4_fKwEuQPz7PyUPCFM+D@Y(FX)^m{U-30U`^w1mI5fFb48 z05q=l|B_<-XH|xO<$?A$NP%YEgm8%Ui8$%Zq50V!hIC`sMB;7NRMTJ@CCJc}Q}U(5 zz)O=h=1=p=7`ZQKv2O$O3aBI4e9M;`yj=Tm!L2u^>8f4n!)>Yqj1czQs8D*e&|%%) zjIUDaQ1Am@A*En??kIbOE@FqGw=Q`c+D-N~qi|d4+;AtLJ+ql$XZ~=IgmnnS@)rj9 zcv>C1MoG?QGpuBD0@ghSd*+cCYfZO-&!gue7!!FYe83nbae6yOQp`XjOXvak*v7Rj zmvi87+{bzKFQa~#;4VG{_%XL<2Kt^&Ldh>j?BouH35u)CQtYT+1#>(l@h;NPz~2wq z5CF&uiCmrbL%v|bJS;rA7T@MJD?5)3&9B8C;^owEs~-Q}MvPUtRv7NTFTGa0?~tg{ zN!Y>#YfSauP@<$L#&7zqbu;?B_-9Q!sldTF$6njU=3vs88cDOSaQ*Y$$l9+HAVuQq&!dhn-0m`oHtifnP5P`NU%L;BcJ)Wmq$_8B??SeK>(Y0h6q~ z24vde9K|m^Hc~wB$E?K=f-`g7=3uIvQ9Bw-+m z`j`;|Zv3(%ZhU}M_dU_^nYz~cf>{ed^N7HRXbW};e4dI9DiMzsQer!geTa2*bq9cC zb8y#|7?4TO51a^kjD86o*;?w6*E=XvVpCmkK&V?gd^sM5SuCCd&u{PB7pc}g^rAlJ0jr_G8 zW;mlBY!D)muQsvNTsTv|GV0OTCi(3p_JCj~ulc%W?Ov08&n@H8r=eg)2Xo-z`>Yx0 zVfXsb9dw(q4q~Hh+HOaEF7;D&_riJn7E@UP&aKH97%G-41otN(tiQIu2penY=pa)t=#mo%|!#rKpPIN-* z!mml)RPh?EfXZF$bage!O*yn)azFfiI@k$zPC(|19m>Ok5>4Q_Pril42eIIVV3h_y zx%=wEa>E7{L?!D&z*w7S+AfE{NrHzp14&5&GR|)CCsFm?DmzOH*xP_!4JfFR*B*6% zhILuP)rIRp1dDe)3q{mrLoN|Lr^KaL6C^v`01@)7Qne_a=>jr}%*E_PYfIml?zd!k z{TXpxOC+{`(l+ytJN#|i2eSMp^xnxY6p^Sa{Nc>G881|U;J*ExqQD48R6->xgMcdb z@en0#)yJ!X-ip%Hw#Y*%KyN{qbd4LY)+!p^zs-bCrl2eH>q@3Z7h|BBNFe+FcLjxdAx zDjVm4bb)@dQ$9m$@DUMA1NW7oTM8W3CxfY=e3L{Fs*TJKx>Qf2nOVX(gp-lP-FPO{ zZ&Qcj4?Yh99$yIKL4XU8p3`Yr{i(y{{BkOdp5)AZvV#v#ZCXEgAv^ml{HO|d7$U6^ zwDSxh4!yF_9>;_=H$EYrs7=GAUq;qn5s)cq7&oHf5$=4%WKhYA-nPpNd;P=qS19j3 zSs+;bDpz9OcgXC;PHxsBozmv5c%<}Jzk@9KXPLoq>(t)tE56-R^WExlk8AqO)i89D z10Sk#o9_6yopGF&e~^r&D4(KWwc${Ql<0IN8$zApIV#@+)Vhw zyG`X~MdYt9R@jOr+Uj&kDvYnlz+LJo_s1H$A!pym3?R+1m9M6?_@hUFen_zBS6Ppa zA#8j_RQwfE!xJ=Ib0W z4y?lP=9{YSSfL-4Cl^G^&v#J(64Z}0R2ll>z_(E0cHb=GNzhg&G;E#1AR<>n4@c%+ zpJ|EU;}?5Rya3vI1$3AatqM)Ep$InRxozN)Si*8>s&mmr`C|K)vQxKIimjCP;APg5 zvF<_rc-zI$q_*>R+Oi1?&o%jvxE}{#B5<)ng}wPyu|=W1;JzeBpr9voxZW2~uR^pL zsb=QFKLAvq(Wf^uQ**?f)rrLQQ!c*##r_Tq8G)=FadJS1k`zVG8mod?RS8<8>^(60 zqbDdaf+YQaGQ0#Vj#Iv)KlVrOZ}8Hpt5&sSr2)o!e{4XEUq-<{Bep8a7ma{ zOh~IQHuvd(V@{4MMCjDfs&~>F%-_PM>Pu9?9zXEmYXS#@PdT>7LE*pY4deESJl`}+ zCD>!9rLFF(>|a#i5nL<`{yy8IB!0;e!wVE!)gy8dJx|#Ji(jAH3xX=)R=`h)u*%bN z>$eLKXzj04&{NkaX%0GWYn9(|;HN!(f{5mC&#X`79tu9?^Iov!0rnl~q?VKKlY8~N zXRf>cz$B_~R9VaRD^%3X9T3{MCiPm>9K-7+FP;T&-Sl^EbWQ~uq7^#du z8pQ`GB@9YVjK=$c{{(r64o^5Ur<+DA{#2=RPwl5%0%-u?G~B9wyPbEYUVYR8RM~&m zE1}Wpr#WB+PX+y}1^GX0VgJd0*X|#=gZ`mh;k`jSda(^)IH7Sk7B)yz4QiHu2&xZ2vQRGZDPABvK zHf=x^qBZ+Mte@u1&uu_&&|x4z7vaN=nN>4R+OefaT$hxoeFyk~B8hLjW!CGHC_x#| zX>wvVRqO0t=-zgn_J6f^UU5xk?b^q26lWNUucN*Sf{Y`m2pL5{TB4#dfCxwvLJ<{E zdJ_nN#Dbt8V8aYG3L=mQL3&G~AVma1hfosfkdTBHLi%33``|nI4!)!BWbcz4CBNji z@;qzZ*L`2t|4BIzVn08sY`V&<2sgF58G&AYsnlfN=AOem|F3+?9%qO)Jd~A6X8a#n z9MC0SfBV;Ph0L<<_cxuQDPk0i zA2uf{n0?GO@T*#kH0Cr0X4NmcQFfH&0?BJ;%kNEZV23ro*VYj)^lLRpMiR7jGBz1R zY0LuO0Qzr3@9wHMz@Ka@eeE8(q*F)NBiE zSH?P(X)`XVAz`4RcF)LQKXawGqt{P~K5XaJ|LEf5$4o5?>_heUw1q}e?B26b1v&ooXZWV|K;z-j^m1Mah$K6uI%)jjb#g2?Ap@n^0OBoTUU?H*H-A%H%h(d zbNYfC_UYv*Id`f=w7ese?q1}_95`^lN=rGC`w7!%2vauS(S^)G%;S&vK(tGad7eUx zdW4jTd~Xmb_Ew`ukMAIyd^#+tkkQ`-q>zPE-JMP?DQP`h0)cdv&gNQq5?80@eSFOv zdNz9C*g9Tikh~L?@HE+c9I+(vg&G{G*v#?Qi34tcRR2i_g>&64a@YobX|Ff11Jamp z-fgEWxZN6*Pcddiis&9R5;fNp5>xZ1uDBO%f|w;01_XSUpV{vFcmh@* zKd!CPj(nzbL!K9PQXV=)wzrs2H z#;Thgd&C^k$3NnWJP7%MDlYAh`+k7l!%u1GhKU6ft*N57;!l{|#^NJ{bGhIYh9)H! zqE^4f@37k6l>=o@533?Wek`<5i^{sx(d23AC^PCl<4%h*My(IMJau%<*E+`}+ybFu zUVIs_YmZF+B;^p zWf>iYTv}{8sF%z*b7v@M%rup>`6Y@jfMYKgD?x;t39yM%t+HRXJB07(iZtZBkY?YZ}$CT(#G~-u^_o2jB4G zVkKqI&i1N^l$Il&$k}vf4-_NiQ@VexO=Pd=G3c1Q4jQR_kmGt-ZDeShGAV*-Z9MoC z5EK*9XKu<1A7uQJ-Ft+(piUlk zH3YGmXYMzGUt?lhF`wHolzgEmdvj!hpk3;>S6p7hj#*9y!_M>b&^Wh_e;*#s(~wmm z1~)tGO5ek-o56xxAtbTSE-D+z+`ku0kQzjSGHn&aj@5@mhr@~&yA+MNJ#%2v3~bok z5jf0sJ*URfXO4_svul(1-_(eAXeLTG(PNwUt?y?iXRaIb2DJSzsy9Tp?}!s*M~=y1 zufpBl$WzN(>|rQ2N0T?oS{mMe+sq-q|M+n3rvjAh>Z1vV!;k^kj}B!;1S{Ux$Jd7x zvT^m0q-uQ_o&p97>c-S5N1lWsCOtaT`T1loL2{Z2$1n zYcw+R<0H0_)ys@5koZ(izOr zjFxFyZ0vXzon&ZfK$~1go5nUz9tzmE4VUaXCRwRd)F)1gl7kSDGff?vaF%KTqmmQe zw8Pmmv?Ep7aOu|>MI`PV=?Np!RJ<}Nl(Hw$u;ul)Jg6G%mZbbjiXbJcX6(bfs_+o1 z@2PCyV9(XV=k)9=CL|bWm6RWTnIvVmi$w1L1?826wK4dBsb@ z+<8Tv4ruMwsFJ}*V8^O&JB}7V*8J(myu&~&(^O$BCJhws&;1qL0QUKR8bkTq`BGV# zc60j#sJ^h`{8sJS@kaYM^()%$u}jHOGcj-CSDuwLyj(3;Y6?6;kG|0)yQPKqeZ2mL zxm#xS#nw1zwDK+(2f3TnM68T&je`oP9jD}c=3XAYZeuli}LV$qotg8 z<|>upS)+Bic)>AoSPz;m#i20U6&Ryu1y9r5p{#xz4Tzj|&A9jX56YrbYpZ;)kZkI3 zLJIa2lHC_jlF&64MJ)~{d$Ne$`Rkc#m9eXEiYbpVhK)50m-MYR36!fwBO2U!>|sJw zVm*voN0WK2KxWOX<>sa{!seRMu#eP(x%Zb|hCaf^p?VscHM=~@PTCK=b;;6h;8b(e zWw`vG`wEl;?hU}8ZnlH? z;)p4G^k_i~&l9ypwYA{cbA&hVMiQl}hK?dy%N7OmLJF6QOa1i=vTNo$l<&c~lHL9SWlF5xOud;wiC8S}IN3_1AbqY; z1>(IjVOK!H8{fGuRVCl;x--Z|ze;sSE|po2B|OHH>U+dZdIoA5l`l@Omt7K=nyfbvF*GjQQ@EKOJgg9aXlw{Rn_k!n}Q98(BVp zhUC!WCuJsdx;a`NedW0sPDp?8oE_cDvmUIOMZc?Sa(8|Tt3 zTJYGe&Z!q^xu3)x1vABS=1{|$7}conSBOaRWi4q3sH3Q#fAOS{Zwj)|#_!#|fZ@`Q z70qhrh;`wo7kG#Iw1lYPWZA1HT=)~^E@lMS+k5AK2mTP4XoY~A=lr%T2f7o8;-!gj z1CSlgI|%pOq(x1iZus%|)mTA32%o49r#vd{=dtJAl5uh9cxGcNu>On;*)^yx3#>h$WZALA5_@g`>acas8pv;;yh-er)#gj$wV(nk3{VW}6D!$K-^pub z%oxA{j*l1v|1Rz2+bN#T=m#IL1W`uwIU*PQwi#oL1Z@4@9McWDMTSz|PFXQulzWd9 z4j+gnTXoZ_wrDqnX)wbRukv??7sM{_m>AVBrPrFJJwl`Vt@dErU}AFz6k`7PJn<|< z=gvUwmjsEF+F=JFniQXow;h?~VGhw4vgDxNA@l-zTw|-KO!J6l;Tz%rqNY7kND@pE;s7QN6Ox`=$YH6(6Jb|# zD~;`>bfNu3rL#pc85VzdDv5j{+M(!qC zR!Gfi2@cRzj;5zcw4GZx6Gn?@or8smx$;e`y+?oo+O>nl8^tp`tDOLNsOZ3BCFd&{ zXG=~47rKui9{4iBgs`iIVLSM!Pa9qXOoUTQFE3e1=gChmwqL2Ak3z?6;A`JDTSSS) z&*%K>m5@HGo8L0FV5&~=0EfXGzp8-2KlfgEf;s8>vSeyQ<@-K}R(Gp9U@-ew<3{*b z3<61E-z?o4BdEjZ=?sYHzt7@;=4!o~s5Z1AtC*ar5)w_10fgiXh>6tho=L#A<+Zg_ zVKRKpT!X3P4Ddx>826Zgk|~AaeWp>2e^B8O-m|q7^U_^#r+Zj=M=yfaPEy$Fu(|dL zXyZz|)=tdF@QbCJo1=RO(NQi%RTDd;b+H(Ar1-R%lQQ*j_&mV<#II`{(syb8lH_py zO9#B1>S*IX5K?F_@Fbp99^}xf%5aId=Q?w<D1i$d+QhR7RSXtuLtg4S;n3}h@<`s7IE_BpJj4xQEdySb0$d_;k z3`rQl7kn>YHOF1)#O);`TP%iA?Fb-?YQtrmpqo8>~t-Lf? z;J(t5h%I?y_T*QifFZA0x#NRA1YI4EwNELV_<0V>$4Mi;=!>fClB;n4fkxM3LQ$}$ zr%{B3TR6UgUH2il`Oah}CzJM(&PTIrTC=n?p|KSI?^_WF9*r!tiEb%1(2Fu%zweV~ zYNTn5{;Y?F;VA`UXkm)+>@W z=JC8l6gE)X(c@>$IVUp`GSjPbKnCy7nL#|P49u(n#Fc|rT?T`wIJ@6{+NcDdxODlk z(&3etCp<)g7EN*7ZI_~9W|B0UUUF=vIZM&6YP{b3yIP&8@!C^mwwfon%?H+g#0}M6 zFo{=kF1>ylJ@kHeFicGG4?>`PfTM4H5#6;Lw(A^i8dW;dnHWecW2j)XE+AqLWwBAVPs>tM#_|uTuxx!iAwo)cqjj}sjo(Bxvj-zhqK)O zV&cn{el{ta+yjmjO`YQ&b$^-w*ed(r>MPq823VLPSo}=QqTW@J? zxItQ({t}Eu_cz|-Px?ZQ4V3dDgnM)hC5fDy9tE2#o_fXtfN&!B<>d`EyQ-pUH*?9c9HW=a!^YepQzPkON%^3S%WK?NsI(%8BrbnsP+a=G z-CkRvF>ZmGl@r^K zl4eGf-w09s>^FE%OYt*AL+7{)B9L2DR5X{lk?ddEc)axF@#FBcildH^8)%9V>%q5l zyH|0gWFar-kgB4V1~J;3d}xEuL?&0L&q`|zP7TzbVPr9qgUj)$Dx zXINaNPra*{BT3Zs@wwI|n24B{IwIV;vDa2-D~7R1nlT0JLS)xFlGC?1o`g5Zw9t8cW9co9rj7Uc3TlS9^|MExJL#IO}GIm5-m)8rPS_l-P|yWK?$LS`SY(hVPkJ<-&y= zbGKExr2bW1cuN`Q*tR{d-QxIR2(a8W?innGBWEq}<$dLP*_8ka2#|ai9In>{={>81Xm+Yj-}Qcw+Hwa!>=eDQ>KER_}FC zZ&h%zBN{3gSc4!q3^Ro#=|-L*LCXKijFiscSomp2s>B&sU9o&&Vxf0lE7B9mQ`47v zqN}!S-c{dlFg2ktSt44k6N&}Z6pRf3jUOk*$V2h=dd5kwb7DBZm`{jSUIsW6j;Uc| zCR@eeb|cfh+N3L@8iADY98dQ+!^oXZTdJf!m#fypuAp`55q*GYpoi`_vqv3^h&9Gk3i7 zC*~H$f=%-n$%6UEo#Qp7oSML4x|@(O(0A6-fGrqP_B>!ZRcAHq%7H!iYh1%}ObBq$ zzh3Q4gBk^xx&DpK+4#KZEB{Z9wcdoZwc#pjJv_(!_((LF8Dv^@%P78flb9$qepj#k z9)je(@%M0lE^pVXfH7HT#dsbcjHm%So)>nf-IHWk?P3DvO{pOA(PjB->hVi?eE(5f z@BQFbJ5fa6GD&?o5Pe@{--GIpTwK27y@6lU)w)nLrE(+kOJ|1Glv0y{n)p(>d z*)gl>A`c2(bu@YjZq|lkRxdrX09!w>NvW-M_q+pjp~I`LsL^_U0`J*AbR@G^+>Q2% zL8#<4r9_#s7yzpql=wj^Nt0{N=QQOgvVVy`doz8@!vKyn1 zG&kZFkOnBdyg%ZVPQb{lc2jKW0RRwWR0@MlZ{|R0jYFwO&d}!2FKbS5d9L$}h+m<) zsP>-4twMr#ByBv8p&0MK#=IrS7I)~jPFK|JQfk zdm#^3U3BMsfN#G#E3(XCSG$}t4f?Z;zkQEavUPGEBx$WLZMpo)%mwnS-l!T%L$o zp!t%auq}6=3PLQd$)V+cOC#QS0}Brrqf2Ve>MWtSZD!pQA3OStbiur8ED=y*%1NA5Bu6mY_r>;b<(b)D&h0(&{W2T~umEFE-^Q zz`;gW-S7C+Yz|tS7xnbV${99EvLihsBim-)88vJLXcrAEb@wEY^t2n|5;>e}&cN>4 z{V-sc9nk~1G4h;-?xYZwru|dvLY74VfbO_P*e*p*rpgaTRySU5*w8*ZjRIp3dE)?j#e5!af4pzgCF*k=y_ESz+b~FMrM&-@gd#gt)sEDc8KNX>@l2 z51)P|p?uQHr_1d>HJd*D&fp=x2u?p<;%=kU-HrBbf6_s0K%)tI#$r~D#%&<`mU?`L z1>_};%+~PzzR7F9I?j3Hf!H-xUy_$1^vtRvCRjLe__}0cdm0+J^&EnEW`a_^wpBO_ z>ik>m>^hDfmYu^NFeD2pBOSK4Fv+=%5j-H%a8!Nn5a&L*Z0V!9P&&QVclNz;_G-^7 zrY(3tz(eb(5mKn%%=l8?aBi&kMZkMWBf(4GjwaLB(t94^?KqlSn4Zxrt4V5AnUUXeJfzM!;!VDP`rq>L5k=FMc&oTf7d2VAre_;_gIl%pa-~ zbm-J_?q`DhNj|T{p3J-LvmbhI28=0y-0jMMP)bEKGfZ5l@PXFwhr+R3LB~t3{;H9d zbc(^=0zH6MOB$qmlLjm3e8?j#R;&(dEyQ*M*C6BdZ({qx2=I<7kf4MMSi?<8FAR8F zF^~7IqR7vpMnt*{f&#YD#^3dzo2%b>DDE@oIS69yAxUUWW?U{r`50SV)vVfY2b=(s73aRfCpvSby|3ObRts|xqEvZs!Sn{ryQ4l& zYr}QLnc5qH=9STK~gPIj!{D0z*5kZ58-Hp(Fg#;KMo9Kac_!j`HOxPK?lbrS3u6WdT zVS~uad}66e^~aeCxbWVAuyza0^|5>SA#VPZ%Pm+m{{<2&3W)^UdY(2KEj-!~i^SU_ zhviC5R~L_{^1Yq zE}0q2NE&`Vi2fEgm-MUaI&m`JyH2&tC}gLk&0(uoaSyuJHh)-~Ze)RV|7*S6-N3S> zxFIi-rBKgrZb97h;I?&tZVOd$?v9oyx(|zrQ|)rJQ|qDzR=F@IIH?%Bw zvAET=JI2S-QQH0n71G7->T@*F0ce8INoF@`JEBE7L1xvdvX_QN3a7VW)INc92tFzm zAhZ-3uo2UEY-22nS8te;`g41ChN-W-S`Sg*ZW<81RH2*Ix*KElckR}#voGl#3?3D3 z;<2KIeqRc^_cLNJ*rLX0p_&IpP6}QL0)~o~;gsvRBC3-xJ8~_#85XnrRU5s@-t;>% zn8$G>pjX)9{ZgN{YA$(k6W}f#YaUTZYh4t&piMmxZ=?zmGs&L9nOUaI!UTyuY#y8E zS`wpruSHi)kr-s-OMlS#ua;0fVbN}xp@u8p#ux2BaF3 zWADC!IxDST8Tx)h^0pt%`Ta!MPzLRDoaNo!j#&GFYCwqeIeh5 z{AnNDvp!VesVy7ZhQRc2D_HZqu&s8BI_2`ekDdH~f9&KYnl-%b7qEQy-%1H%yBFlZ n3hw-Q>%pG?`}%)bfuBxX6TD@fKFc!FvHy-{MPGt>3 literal 47614 zcmeEvc{r8p_qU`nrxGQiQ%EwDsmxIc85)o|nP(E2*++&*bqJZM44E^}WK0TWp67Yo z$dKt>&wg}%$N74Pf8OhQhyLig*t`Ag=eh57f7V)`^;yePd07cEVj5yRJUlYV8&_}R z;q4Q}!`oX*xF23w)N5?V!y_O!6c?8_7MBn=H!**B$3jb6S3=iJ_o1QoZ3!_vyi2~m z3hD;53Y5a3c^Rx{>fD-QWBtfCd0u6j-2V2YV{B75!?f$Xze(nmmo-z};f73HgX`}M zS%fLCP<)K<^Y7bI{m6XEC~?|{?My2^pL#FJ+B(r!-#*LCWURdaUi35q@sfV(MP8SGN%SM*5?7|0+{xk7%VI4n4V>* z>SDp0Op%w^UOSW<{shfzi`TDg+0t<~pAX^28^z{84+YhBq-p_H<*B+nP)E zwQl2zPB*z18Jtd(y?b+)g3dh*?DwRIR2b@d9m{CcwrJH9+gguR?UQp=%kQggm zp{?LhNIUlEeFERNQr5VpV10^SH(TqEf}g8quC8ZY&TG9_Xx%`hXK}jef+UBepW6!U zAO1s}$4|Xw@MEsIqw`GX8^07ePj%oQRyqrL-5Zb%bNs6il=--o8=hQT&NdUyJ9IXInt1zh8BI zXt_qj&$wpUkDke8uxCiPzs@T1VWefR>nHU-_J^7eH7)6_A|;HP>}I)*XHwsOARaLc zy44jarqaeHcvi<6W--DLxT^9Q!k#QA{{P`Iy;#^RzUkLmOLxS zU>GmW;o_w-bIHvO>Q1{DU8~6bm)4mDmQ}JZAM8uo+puO(%j>E?STu{5BphqIblaA> z;z2L|rDnJCk4kbh^@;S8KU8ha7<=&A9E&uz);8>&7WOvv;^jtd?cHLuC8^YIsRe+z zq(> zuU@%pyJxDKC`>_jLu5|>-Zg?F*D9Q($*4<{?^f*Nqjdkwv)&&OxbfwtOQ51k?ECjD zicE&*h*gwF{tWax!IF|2J(SWk=9X0p(wv|@5jkG+-~8Lxo>SFOs8`y*e1`&UOJ57rAUtF6qHA1tRXS*c8G*9&!*;cY*Y zY1*0XZr{)8OLQ}om+ie=QX~Il(h=ScgSQPIw5zLla|(m;GxY1p;hM(jt?UI;hwYc9 z+P_el$fC~kZieXB0meH#dyfgAu+flKT21dlTRi8^afY>CaN*0`LgZFqKN%aFWM7ZN zELpwqno0UBU&-p%v;l8c-nP0zyM;2l(eNP0uaP1pJ-nf2`G&cDf0C5-eRP{l&bYry z5Gxx_V^)Z*vKS42>=C)S7^k&1os-&lKdHKsT9ek}ApOZ6r{!+`$$|R1D+lN(Wd)`) z&m^T}eD~BNDco4<)SAv}m0j}bAN|VLOKa0*S*@3&eE?myDE@t2_?qF?d`8>92khHi z9~JSOb((dkX=RRd{C?lO$8q6n+E?DrGySUK{rX+w@SUXM9;8HY51RMV?q6}6zSri6 zvHf8++f!67$sA6(xw%~8`MAd}PWCj`s0%iOe|^qVB#GL&iY!hp@}%?DS}(Jynq}ot z>!lxBs{{|)3OmgT+z*|;eqK|2i3zo|gzKeja|(Bi_A7p$B`(ZQ&G%l{$D5M)N0@FB7R(kqghz&UoU$(bx?!xjreZrD6XdsKY-A)BDeUA| zeJCmY^SSdVs--`F-Bq#2>+bd0*n`ohragi?u$e&|Yn@!AXtbA;`K*W8A;(`P!2Q^_P~ zEw2_$pEt-C-P+vnsW0)39zN(i=hl#@l=@&VbC`u#?)+y~|9Vx^EQQFw%2*a5;Ya?E7mM=emXo<moE3Z zxLgn|8uF7qr|T6wrSDjCw|`hug3kKxvC9v2!sob(X_MqbPYT&TahS?1KgRwY!ow^d1|7iU1QupIt3b!m>87x&}$>xvL z8Yb*Sj%`fndtHJDpQzk&FVQ*9ebj=3rZIw1(scS^0XE?weo%-zk4@Oa6zeU1XJH@l z3wtjSdTRAYw~F=;1Y~2q>n?>`Rvkap!C~yp^Ryn{m!rtO4|Bjz--F%QVKRg%Z6YC& zi6hP%v(>I2NhiFq*f>{p!u*M?uC=3}pVW>N{<`s*jItPq6f^SDtQ3?$l@9-0GT4EzwYG zY3i8DY?PJ6Q!yxLa}6Fo>~+d|eOQ(>SWf6ilSQKpALDdo=dhgc>d;U^!^)p&=8l6W zv;b5|r#n{T>K|e{gK?4L-85I#KGs~@d=wx<&QYZHIWB&a@K1cxk!g5j6Z^vD30qFr z#k{Qi_oi(r!vq7#&F-v^L}rQ0#Zm_{7zZ;b2G1C<#&d72!77DKQHp08HpVs^-|sZd zNzR#CprVZU*x97wtl`kodi0)ga#ooyU(ahbtrKo_Wt6GJMg1Np{8)PiIUEKS)Hnc0 zIRP|=^eecYk=!mmf|vr`*yd`zi-&cKGBa|w{V&DI zi9G3@Z69Va$4(Je@Of0TQVeoCudgs!CoL(wJz<)8%YW&^Ehmgv>G9NeMS!!DrWBgz z6bU8i9i~3mPX=jFh-V+mtuzcZ%~rP0s%Igt2vX_>oYpFPLR6y=4Q_`mTUht=t23m& z9GNuS2J-h@Hhz?UW=#!KW1;IkWBvzXt;&nHZ<|$BPfLpToyYxus||MIr;q+YcqTC%0OAlsbbQqyCJ238FUXm9gkaS z!=bZ=xaQW+R)1Y^h8$(^r1#97ruSik$lZB~`;Df5j5Jb@b>^5qyk_vI1pHK)>z=(d zg&RV`iLbd*MqM`c?X5rSn*jxRe`Yyy zSQ3d#!>{g4?6YKROHL5F>lkY7cv%5(w(LSRt^X5V&DGCJVy!9a;*KnGj>THqcSt1$ zeeOSKI1;yVz9BdA{UZ!=w>5PfCF>98@e>4Q3o~T3MJ zUtK82R@2iD9QCy9Fv<8@w&wt?yu*`Vn@qK@5zC$2XXZ!xAV|cw?F;QwrEgPw*z?8WzrT?zcvw2iuAFael+tz+T(6 zSw&S-g|51rhWK%j&i3=4yvlmOy4Ag|{6GSX0fjfLDY*~kAAGPJta4wmeOk2C!CDa+ zeyK8%7;!WModbi}Sew;k;PBwM1Fg*PJP9G)8mWUqv$z+SAc71B<${?mn)ol03YktEamuCwX z%BI>g%FUe@LMh7Aj~P^@TGd^S(hAiDre~reh7!HEczrkuJ`G1!`C)pAl^K z*ouPfNn_f(j%He?NPT+;o5NF!e;T#jcO#p4mS9Bg6v81UaK+E^`1Mq_uEQyvD4bn@f7S|hcthF5AqB0wxcr6xAqd*d*S8t#uim` zy8Yy|uV?mTUPmL$Ar(=)kjZ%(THLOkRqsnFn2}1c_hFj?GnTQ(NEg~;{pEy-j8pPr zsj;tQBY#Fr`0*Q_Bdby!)r6%!)AhW*i$_;nR6YIHfg;pV8bU#@v8jm)cggsN7&mhM zvt1DA)pQOICNIXC5XMXMPNY5J7+A2$X?x(4hOIi`xl(0!hnJv<@q~2)3F=>CMMJqy zG@=*TcEEoZ_0rD(Zr$ctbVo5_$J>~@`v{`mKQ!8{28lUOL@1hZm(1xDihqA!5cz>Y zVb30egfN=j>W~c5P;Wf1Ag;ON6X+6B+)#iuJQr8B+k2n<0KTM~8>qLpea8*oK1$&l z`}zHa-QN5E?h#*Ixu2MvO>CU>;d+~Xq|@gH`);O#LW>PZh9kHLVdM;PiAzT20gmPv5@BpJwICB2-(}yXkPo7ysBNmS=`^LbUmW{OWir$x zUDC)@!AgAM_Sa@U8{A^chq&Tpa`QQ$m+(H)W;^IxKAqF6VYyM~;MvSraCEaq&iU|u zGMZQ23G8>5F!Q4fsles9KD^(r$*5js#yePK#*p^(&!6zaKM95fADNmFFANQs^q` zsNG**!D{ss+C@tTv1?5x=hWGnC&oE|*{YPR%@p*<1Zha|=L`p`dYEt?)FXkoQ} zpZ@0wcAUA~PzH>;7(cSx<9&J#o0f5hL}*u3)Wz7aq56BeQY@WMi zl-C?{H*bE1+C}k!>E{bshK)=W9FWB{#jNTrZNswgkvYD!`tx7VEmI>-f`}v9X>Cb& z^)ur=5;7@u6~g>r&*K*!Tvc{f9H{TRJQ~*crKhcsA&Pw;QSWvZcpDZnCcfQsw;k{X z#fRe6;75MDy?Gls8FFz-*`};F{!mp{2r_AY+Yq{vRCCRT>Zow+ zPa)y0jM!MFX5Pe@3)b%grA{#8mhbEfFgw8%m0e9|FTNS3Ui8o-%x>R54=28MFhOBA z3%JP8z`ob1Vc+)Jy5T?WwpOy8`*@(d)4YVPMa4KdZSc8(#}&8zb{<-+t*}ygc)qlo z8_T^1-(L46<|tly;8ma4^{7#%JlgYQ$odp_x=$q=tQ(u_6a{*$@d*U?X5xFl^TGYf zz=q_VJnNA669cx&ALG0)nmhK!XM$$~LId}YZU;PxjxaT;<{$(XDYr(8ds+-U zW%khAoDb4CrYNDeDjZ)szuF#(Tco>hVLfRQ&AuSFB_DtfT#DAE!Dn1z{nLY^CZFKe z`a-3+Ob9nig(iMe#Dfi4k&VR;pNX&4tZ`dQIVD6_CTqA|F!|iQUryrI!&Q_pV$Z~R z)KC=|EH+n1nNP`_T7M=UA?##d^j9@Q9EEeXd9SeJ#-yfP%8KAn1*|7tB^AKtmd*8-`*a4g$cU;-S!Gb8(4KbnehruR&Fx!!zAvy#qbOIq`nk^_ z{TkM|uN31`{hY4Awq`ot`ufJK%a((sjEszB0b|Zai?Yb$nXXDz^CD}@4Zps)I(zyM z<|y{B3r01`F9c?C2kd%cq=PDh)NR@aS~XpYG^mTQzUGuUlZ|op19Bo8nRUF7fLK{> zNUr8*-*4{X@!_13jcHsqyc#Jj z_Kp8^2}E`NK{*xf5U-Mmdy*$-2n*M9YP2inCa?*g_;&Y~MGZ<@rGWb$qDV|im$IGu zP+f-5d@Y(T8~mQWN@CTW9VB-gBg`BHuTHaQTycWznb{}s$=g#JL`LU!EZUsro;~qg z=ouVRi=xP{;2Z@q_S#!>%T&4%T1XPf5-Rvqt8+e9v-E7PK`zXl8}1&-5yc*9-bAB{ zP#{|0-dwARE?F*`ZFU&ZkCt2RvXXnvW2`uT!Q(A9<^h=uLN8fj!yor3hw(Mywmc|# z8=&&Rf8?JO-Ud0M3ch^mi1EuhpI$a(8Mp4QUhem#_kgg5S2<^hcO|XYd5tGmUG1JN zl|h)xpfBG5-yJRh&EWfNxvL0sH1G|{BL>#Lb1ip2*0lV#xi+gArfDAwp1bmzvxYb2 z*N9s|dj9R(kQ2|yfS2s96^J`qLoy6Kyeh8)i`{e51f5-gEb`7NG0$pN)PjGBMfr7( z(|DOSYu3E;TEX~P-A%f|ciD>5bH2&kFx4CjOl2gL2=K+74OmUON!`2OHse(HC3@B6tZ7DTgM?}HCwbyMLnI} zX;!)Fp{G|Lp{_<7Nn&JF>Gl-%^{G{icIOvZ8?n>l2}Y zg+jtNBZY;Or$=Kh#MEo1fNAKU>4<2QB}8X~ye+67@uo%+LGStvM_{T?(8`e-43PK^ zP^QXvf*JX*R&6dc-H|)0l>DKJrqYf}hFU%-RyAue(CO!*B>83Yo z)CyZQ9FC5j2*ECVe<`&P^j>=5>(qAa=*Yx?xACP6&CNcAi#kT%a*Wjv15C`O5D$-u z69ML+@)|JmcgDK%tl$D%*vpRoOfphbkg87j7N2q$-*8M}jbgVJN?&~cc($i<2Fb%J zSIcfXE;vl*y~a*4j^6ca0qoS-FDn_URkKYg0!aywpd}DC@EYR;s+rZlhLn^CG22@tsREfP3Qo zmWQ!YJ#d`z(h_=>ZFPSHC}}9G70(v0QH_4(hVV3GRM!}Fj!`M1U;Rd@Hpn zibB!Q)jo)Nh*eEKTzf6z;(YPV=@)_*X)Q|k^dl)N;!|w<-Kl{V@ol}3Hs199PSIcW zVVAkqp$c&j31riO!Wsrb=OA{~Gxj?{Qw(=bU>ob^Gd1wTHPK)eMRNeJA{F}bh>IZw z)KtEgX>co@y)N?6cS$lE7Gc(99JjylmCyeNQn$$IdMNGIsnW&`?jXef7p46Zu;C{Y z%GqPOh(L;)*DoRR^VZuzZ>2)t<6KOnw3pKAc%i;Tg7wa^tu-(Fk;CaUD$q+5r0Hze z|8|yKI@>=y~d22QU9?H<0B=m2t8e0|lcg!Jr9tI;|#gl!XZOuE`nO=79; zm{fsqE)QNCol^(rl9Pc(GVwN;`pqiE>BChhCGzk*&S^`t=e_%wC@^?5lCQ^ByJ(K< zVVCpT^eN#3A8c0aR{A}eO^076R(iy;vt92_+FKdqAhbSLt_6l@Y*Dgj@BT9-obf=h zlTov)9drKK7}-K3QeZtvH6ZQ(B$DhluW@ufw@+==wnJmvqvYCB#wm^4#Cdj#ULz+U zyqNdMkQxspW6M3rNV*Fin_G-gYdTt9`q6a9-}*x7Wgxc8iI~IW9j`xd6Hsn5fd%GO z_L-(AuXUI$;sc&?zXkT(xX^M}#<#)8tX14pw_=6xq823NdCHx26y|w7=_uPnq?9?<`sT zE-g+fRWcut`s7EeX33(*RB(c8{<~i(yu2lBv^Q%oIgScO0qgI4t>NyTdOXvFSJS(N z;PICLT`B`^dhVJSCb%B3<5Bo*gIm8Sv3&!z>x)B<}Pm<+~~CT{1eUtWGD!uo-C z`1OG5V34zw49+=}H_w3ZCu%BmXB^rdD~d8Oiz zb;7%TE9w3F_eGx^kAP=#8zI_Bt_Lf6C^653#E<8*5?R* zC@3tBZh!(u8h8uO6!iiRfQr+O^EJ#*B?_~a5gb+0u+zeF@xch~9q+4xprIW7*vL5_*C#Y9tjexq3Prx~*H?k)TPk^o~O*LT+Cgi*hT2 zDB#4;ACpQB0jn1QK95@lWsEQPIB-0Jvc_t#-Avx-L>85J-pay5fHzObnVVJ3uh_wk z{+;I8Q9+?BFTV<XN~JGPSQbzT_rB&Ih}wUB+qjd3cMrtltXmdNBl^f;e{jM^^# z%>~pxDG3JdSi%~*Fn+85ofnxb(u-D6zBPYi}Vj<_OVV2wPF*j2% z+tTiooYg{%^=>#Kd+DQ(|HP+PTWUR`TAwoRGo`drVFU7{0aK9R`fi$6`hB=aU1|p~-;v5Af^{lM?f!r*3YO3oVdV|R` zz>zB*18&0>lHszKr^GKd6--Qf=ifD-_6@^@>sK*0$9}m4U!wE((iplRm#t|w$?3LH zMI{@v2_U0lAc$AQ9v1URgdiAHFW8byGC6MPwWwCq7m zF_B3Dty=d>UC&fJ%~`aWYWo0*=x2z*hcnb80!Qn@PgFfS?!W$q!8wx8TqAs><({^$ zMa3bBCxA=*Ch0Xy)-9{@JFJxvST@!?`=4mtH}DR?r|^xG`DaQgLQ8GhKU7p9WGlZV zK^NeN7%jkx{7JCW7nAeZG@3tul%*Bkvy7>cGRXRy`z0GoB-o7i z>)_#?wIxs@Ab`EkTsJ`D$_6veeSM{Tec4KH1+*D~Pc9)Tp;x_sl92US!zt2}f|roQ zQtO4#5+Qa%?#?QBs`aXvll^(45z^@CXOfFhOASWikD7o8tap8ckOKsk{{k9tX(JyJ zs?e}D@bJA{3oS1#{ldzPKvgrM3nr@e9S?li{8(ED7KD^QauK7nrnVm*Z?BP*Offv` zsr?aH{Sm1DC$}0Qg%u!yS=Z6-m)Go-C*Rx-9* z8_Sc*k05ZOwI7R?^vVDfZ-x6r&u?wxJVeyT5%JHE;xbTW&BjJhQD$w(6| zj(s}~#nAIa!hGG28e>|1wyEGxd{>#T6*dD_1o6uA3-isX&ODE%+AP}z?sfhFZ~yUy zTnq;e519dR)g{TiaA&dUsF!(+aG&emvZyO=g#r2OGKtQ&gA|9&7yA-Zi|toHUJzE; zWB`uJ{E+~wiXu4Smd#%^PU~tjYf49r8IEcjUe2EZwFPNk*xHpYq%Tw z1qEoV?8+({z8C!wcZ0YkK(J>>3Apsta^u%pBr>JVV=&iCybqdYe`10Zb|p4-Zc?5S(ll+@1yrrEjkJlzj>2 z5)Wcm56SnLR=L5jnY0MCfA{AjoVdfl5DDpqV@q>9V&PHl1St?6TYxXjY4}F#f$UMj zErv~HD^Qat4mNC;rr!O*fG6XA>ojr!`DA=V>v3^$i8YCGAc*Az{=(=NQus$0EmM&8 zptU*u8%xm*0zDZQj@9A;>J|pSQy69X68f@%%IFPmO~Pq`q?geV3XhdW>Q^7tY}E>r zxr-PlsFshRAUh6s_kDUw5_gdNRP9ceGi_`!n`9TTUdebURk+wk)}JT9K|+^rdTf6L z|2fv~?=KkfT~DgS&hU`JjF1;6|Ka%nb=4TzYC?ZPM zyeF=6+M{mxPgl+mL z#%(C?#QuK7-@Eu9#)0{N)lk%2+X9Y$YidLZR3 zLv*1_o#yn4%DMp!ZzWzJY=y--w%QZqW%)IxNc^qUmZpV}fh6h$He{1Y8M1AV!6Q~zXnC;>=+j0 zQ=guef=06pV*8_1Y${DdhaiwNmi%-XvI_e((9<5!>dLk3pGZt4M-TVwA7wl85Y-{5_Zse>pu!2T_TZ0`AVhZmGp;(q zh0%HrUSY%U?#O=W=4m{|!^l_-WtwIYBj;RIzv+Wn7zayqWeaY8k~0=NsFHs^KE z=2_?E0v&az>y-nkOPvh`Sc)}k$frGzr4@rjqCd)woESsk+E3<{ftJyJ|N zY~M&9sv*UBiyO-Kq3TmZNBf~2(ud1kAo`PRP`>m9QI!a0kHm(WNo}faw%0pTf!?o| zQgIE<{8ZFSOOB})5%41cYYVkk{S^P|kiR&j6(PVsC-SF)b?8uB$b6<`!wxS5fP_cZ z%vg?BZ9dXAvFm$d@7CsWnk{xf2MZ~hD)ahz69*uQJwaVxTT_iF`ngA?n#E3_V%|OY zGSs=SujLSX_|vh2-j}~$U#$%kcAQ2{@k4ptSgZAGaii7}L`RTk(zyn95g;{&l&VxD z8G5c4BWjFV_JZ?vBYK4)7kfO4t7we5-%B9L4mRe1+kt}`BW&hw`^nA;uUUa|_eV!+ zkx8rilGUR7Gj(dEU3+;;nbee?GovCcyBdG)!UJB zgnk>;h5Yrc*^N6_Uc6m#Y`9d}IX+Ss-Z?h%p}Q*9orIc73DlUusPo3m6GF$xiGHX? zNYlKiU!DSpczb{bNws`F=UyrNqY1s?bO^b=N--|F!Ldm`izwASj@eh`lf%kEDPr|Z46W#R2OjI3wiSle3R7-E9%seyq+5(~t2OV_+v$WYDf*rDZfHXPS{%BV6-x%w{pL4yPt);IIQ4W8}9a`?#E z^>(@!4IpH#7m}Yg1tj~C*V%~%b(v3GT`p%oN!LcgaPNIk=7s*;+=D?59&U(>`v^OeWu#+oNkMTs^?MsWLuhoRip zkz=lD!dC1xw-DRW$<4^QQMiPkyaalmHF|oJUK?rN#)c;9atou!gG$zQu`$Hpv1L<^{f!NHnBgO19a+X7FdZCM z&#|i+#V+w?Evp-G(f_hqfR0z4AK|3Gbit^aout3_ zC2aPpu1vVVV{*`VNb@QKYlYNX1GDn1(SsuT6d!42u&l4~XQm!K5z1oH2xSBQ`dKp* zhwVcmjUbg+`vv$qc}9GsesF27bg#H0_Ey+H@&rPGdSQk~W$2w(v@HFRc9cuKZ8r<) zia9IwJwmq8XmOwYj|l~nObdMp%a6CYzw(L1Um>mc;V8be4hkVM&?A_B7ez$)12(|w-LowI5WFJxt}GKA z@=-k})EEeEQ;tE2@@!%I1{4C+^lGv7vw2pdOr4IawVE0u*^0Q30uu=I~=~ATK97tJQ5!%i4yhh}90XlnH^^M^iSme}y1W zv#(A21sm@}B;XrwjH4Ez)41ShAzb4ir56@d9u?VM2Rnj+$Vu$E1|%lyhl-QjX43N)`Rzy@+|gqh(VAI6BZS8a zX~W8643xh`>}X}}?BAm5y9P&q#b4@*EP|BW0q=phV`^2bq9Oc8H3ILp6$4}1-&(S1 zJ^vzJIYZAK0g_d3CEKc;qUt4(A+b)&@O4w>+f1w68`^HjlxM=+dVAg7w)L9hO__D~ zA?RfB6X1L>rtIMTaSWV`lEYtUzgWWrs07V_93DeL$aAWF&=hpT8V~ywTQ9QZ`0%J< zfz8&2kotPH0;tcFysHvL2dzG3ciVK{eUZGa50BEqKq%%8wIkM5u1j^*?`>9&_x=_Gp{*~ij(;*YIpy-}Y*ggPLR8V&Ie6M=pcRCG5rq^;Cj$cNL?=wrt~^|G zkn;S#9m)Me{>~tSDLc_qG%O^eV2X1fm|%7vh z)f9g#I#xYSzOncYRl&UgClzjUMM9F%Ck?Uc28|z#6=-y<8+`O!UOh^di8E84exR_F zC?6C;t}Z6ixujtDvhoT?gowa-4Nidy5#yAS&7bFOZbjnoL-ZP9W}=tI#2Fl*e2On! z9B=fGsC0>{;Enhx*w-ti+8u6juf`DkO^KY$y#b$fV@#q(L)N$d{$6|Djq`Z%lC z%#K&L9mC5AV1@d+|Z0ZE#nD?nHf2q&47Jx~;|M!;vg<941YS%%B28poxj=#x-e<6 zI>^pjj;taAxm?S^GMJG;Co71pu;{THb%z#^*3Oi)zBfMX#QI^D*AdyT)-7ag#r)CE z6%gyvKWL!YtBBGd3=RDFfa4@M^%CfICm)rfjd!9)Afc9dl8$Rt=4R*AuM6V@rT!eq zsu)k;nh6!6unAw06achM7oVLJ;zx8jNGgKJ!x4Ef8Sk-e;?Uf@OrVBZfzt&8zJ z9jwmX52cXjsgQ~cig-L|YX?;X{~vK-52fZ0#{4G>`}Gye9Y5BqgJu8VQrpIoGU|vC zIV}hy?mR4v+?dkuhtV)HT3NH`!4M@OtkArA((22Fn(*Un7MDCxps!`?J04Xl6OTaY z-5k-vgass5rWjxw$9}GhDX-5>2PwH$h1a1P8>|RcaVaKFnxlyO3-nGZJAT|p-2|;K zWgso}L=q#=|FgFd-9gB6SElNb?xeRr%jq{?o}c_4BRzlfgZ+u~$_=1gqDW=( zZG`MGL`^2dagH8mUZPPZ_(jP!vL7F(*f;EtK`g9_lEja2^xO&vV20F)c&BlX4 z1N{UjAK(gqk)MRGlo*6P*C5%JBTpX*G4@#bUXG-@na0b#aS2B^)!5BBGEq`n>c|{v z!}qQ!oC0J^^FEN+#MnfZMr72_gNEaAkaco{dgVg!i!BiINq8?n<$$G*kGm1rZA9Za z2b3Ywjm4Y!bek|J-O`@b=$j59U&o||9uPYd5dV&p={19n{13&Dih_=bCgzjjx9Vw( z3iUIqo;|OZNaTvCjN&&vboavb_pP>Beg_YI^RurI=&hX3O{ZS8XWNTT$dt} zLy#Sg@)1?rn^{yOWM>UiP-Y%2!ZaFoVS13qS(e0wBNLM@Q?}yyUWxZbLx3?M4Ct|$ zODdiP6_JQI)GL2Q47Wi50*W3>1y;mBA04uVDv7`J@IWp0_6jl_AY7_KdYq&)O{<)^ zKuEYsIm+D>hAO$M<47PdEe^tpA0h|HY1>G66492y#bSu>mVL93nnED8S5Ps%Z6D^8 za23>5WJ#b_CBYn2_F-iJa(jNd$r(w*U|@}`l`4#Px)%mQQ71wq^odg12l8y%f$sh| zP}F5|fwFhtX2W-n!I&v`QxGnmn_61Am+u^pmnp%tV^DR&JUm$779k!IHO&$s{N<^h z4;w&oOVT`@wCRDSobI~0sooNo52tH`I5=jOGw-Ox`juwJ*%zXg^egg8%yBI!P3;++ z?1tbe$`C)ba9-+RyMyC1)xroYH&ntgd|4TMCAn>@Ltwo%6#LhSj z^QY_asC`5zea49adlW?2z-S6*1mm>khiZg8B0uzh7x*05L-9UNHXx8iC?{@SW1A*J z?*`L?@L(gk4y~|WV>}q>Ul4jmC&2q9goo8|5b9b|>==&6`pYmus<16Cm`k5(*`b5w z)p%>F<}v<926C1EdNww8&~alAy5>Lw1Liuhn$wTw{s!@bWY{VROM&rs{$;?zA8&y= zQue~1OKpuZoP61>7SVq8Qi|DqIp zQ7AzKt6~zNsi|rIfDZ>b!cHP%ORhDW0WleEp}!1gE@eIp>W+V$SI_5W$$(lkS*v+kwylJj?PYT;Qqu&#gRE)2@}uxuJD8M%+Kn96U@Z9 z`(jk?HD|Q#6u3e5na+Pn%GBXb)vpO6%Zy3bCks<^(i~UkhYAtKi`Kg7mO|?xxl1sb z%4akjbf;P~U3rzG?%~KFBsD0t)-;;tbn{m5TsPfuqoPsF7-u26Iey%t#9w8os|o|h zdYij$GYncW;~y!GKyy>;`2OqI!b>EJ$jm69gvg-4E6E=7;_<63#ACyisp_EjhXRhV zsX7!?YlfEs7VjMY=Q>ju$N9td?r=sO#@H>4 zDBMm;xQLx!BH<|HTnz>x_(Sn`2<8I`>jBfn&IFYxDMXi{wUy*j4dnv@~)lO zYH(4j3pe>u<~PV%V@%=Otn-dfMj=X4Hlw;q7+%~cL>91 zrN<Z zV&Dba9ek)~*57q`gqD|vWB`Y94mD~bSo7&sDN_GD3EUm$3fJlKzWz}C1oPysV7u93 zs*$1}4-8bTy3?@nLaM=kSj}$K%XNn?8ut2Te^u*M$rC?VvZ}UnDbw?t8!u^jE8X|c zl)?71Tq7ma8p?ATD^vO+q3@l?(dxpdP&jYqr(;WyC4I?)f#i~Fyk7I!N$PAG2b0*J zEuX{f3igP-GNZskW33DDVGVZEa>Q$&s5;L_7C9XzSZ`q*H;AoahwlEQ4^z%Ki(d3C=+mlH$!6hMf?F{x3VA-9}6+7K6}mdB;aF z-Vh`ly~FY^*M_qQ(H<~}-;elv7k6=u@xKlAZsivOBIO$M!*TdR0sG)lzYCIF@bSmm4V#Dn?<{V&y!k;X-q%nZ#OQeVhPSFQk zRKnDgE!Ly@wwr?@o7(dxNS~c2oz1;A2<#C}emTs(JqmOS6+(0^!N_4yYaD!{Rqmmn zP_q4eqCfC~z)-ma{7pG@Y2|*efi6}J97iYp88?#Fp?6IKX(v=XH;T@_X*dtanK6_l z4FPiVJ%;_S5TV3H?TnC|%g`@MvDHtRgUqIZ0Vb!3_(`mB%EI`!&xuQZHm8uiUTkV?-uxN)|h4|3He?D=_OT;=Ejj!M8VaFt$}V_-H<4O&0CC zLGGa5@BkH83bp+kDa5@o$cMV}h#Fg2;pWdhI!)0!f@C4(5e_rT)+@&o*Ps)K7Lxq1 zrgb;^^{6{5$2F0(JQt{;a_EwJ^(vTCOAI#%VfL4FVU{oq#AG0<)aX7G^bUsQNc&~w zA=6LKOd&nAK1#Y3v!h#^vs*Lab@8aDZeoCNa?e*5sjxtl|33@P=OA<-LBHvySZ4UHBu zTVbqq*1uV?qoYG{l^n*k{6JW!Gr9vmBe1c=LZ-+t1NWZ;&*V>_jtfW|OFjLTltO?+NWg2 z2S*^s;{)w-Ev|BwF|$nP+?gkZ3C{W(Q}(w zs?Wmko0RlIG(l%z+rxK7N?HX;`0st)hfGqegV|G@7Ja25J>wvbP!lu6kjy zAPmc(cm$KXsh`}hf5waJCh>otq4FmfpoyBEo|c@1QKVM>_R(l7c3L`&ThTp|id1|f5s5hPp_5JRUG7(2pBFl~G zIf@wR2MjXzCAYUxc3jsQzaiq)HsfNi$*_5N0S-8+yQW?Q2y~D}$FU6$geXJM5O^I$ z!Wcx~NVu4S8x&1;UbDCmxsYM76n>AC zxQRP#4fo*qYYR2wmX(M2RSJE+PK{rB)+-z<)jm(W?WTN>B5jK0NL2tPefZs_HJ7&q z<&^t(T_c*#Hy!BXnc~mjYlenqAiqU@#7ES%E%)rj-wpVv+ep>Eis+1a9k+cNI0y?t zXIG;;i@jh58gW8oB-VTz$=CNNE!f0(f4_mcqbzPD-gP3A;?SDGwfz6Fd|Mja07y=hPLI zTdv?fA{uo8A4Y*$Ujogxcp>&ymn{GIJKPL-7*ZT}fh-Rca)WIrGTKS^K2>pj)ogK0xniVcEM)CcsfgAm*@b%DoSNWM*Z#BkDReyWs#t z^Iy~$rGkr(MfZXaBCEK26fvJBX2f$+v*Mfv!m`(s2q3Mtxj>xz#liEKdIj~g<2D$K z_J~)>Ye5*dE9T(ffau}9w8lW^-tIt_fKDOgh7hLXzI-!MI2N78x8&8=q8fwoFXc!( z12jI<0_}Mi89k4~esS{4XDWC^2SrNR^T*!kP)SX*@%y6z^;u>_TID%h8^l~$C$hC} z(R529K%mkRw}g<4I0sMo*5OvYmZS?N5%znXeolxJ?EK+$N9 zV{4wHpIKy8^0knIO&kdIe+bpBKN@%7ApbJ~x4NZUI9lzR4*-suckK6Mi;Q!X@Jfh0JXtH|FlP>`-T&V+#aJQo7<9yj*xCnY>B@thQn#SM0s)co7b+h