diff --git a/doc/api/utility.rst b/doc/api/utility.rst index f6ee96f4..811943ac 100644 --- a/doc/api/utility.rst +++ b/doc/api/utility.rst @@ -14,6 +14,7 @@ Utility Classes utils.DMLDummyRegressor utils.DMLDummyClassifier + utils.DMLOptunaResult utils.DoubleMLBLP utils.DoubleMLPolicyTree utils.GlobalRegressor diff --git a/doc/examples/index.rst b/doc/examples/index.rst index 479c8df5..03f4e713 100644 --- a/doc/examples/index.rst +++ b/doc/examples/index.rst @@ -24,7 +24,8 @@ General Examples py_double_ml_irm_vs_apo.ipynb py_double_ml_lplr.ipynb py_double_ml_ssm.ipynb - py_double_ml_learner.ipynb + learners/py_optuna.ipynb + learners/py_learner.ipynb py_double_ml_firststage.ipynb py_double_ml_multiway_cluster.ipynb py_double_ml_sensitivity_booking.ipynb diff --git a/doc/examples/py_double_ml_learner.ipynb b/doc/examples/learners/py_learner.ipynb similarity index 99% rename from doc/examples/py_double_ml_learner.ipynb rename to doc/examples/learners/py_learner.ipynb index a63b468a..f6d0d106 100644 --- a/doc/examples/py_double_ml_learner.ipynb +++ b/doc/examples/learners/py_learner.ipynb @@ -7,9 +7,10 @@ "source": [ "# Python: Choice of learners\n", "\n", - "This notebooks contains some practical recommendations to choose the right learner and evaluate different learners for the corresponding nuisance components.\n", + "This notebook contains some practical recommendations to choose the right learner and evaluate different learners for the corresponding nuisance components.\n", + "This notebook mainly highlights the differences in using different learners, i.e. linear or tree-based methods. Generally, we recommend to tune hyperparameters for the chosen learners, see [Example Gallery](https://docs.doubleml.org/stable/examples/index.html).\n", "\n", - "For the example, we will work with a IRM, but all of the important components are directly usable for all other models too.\n", + "For the example, we will work with a IRM, but all of the important components are directly usable for all other models, too.\n", "\n", "To be able to compare the properties of different learners, we will start by setting the true treatment parameter to zero, fix some other parameters of the data generating process and generate several datasets \n", "to obtain some information about the distribution of the estimators." diff --git a/doc/examples/learners/py_optuna.ipynb b/doc/examples/learners/py_optuna.ipynb new file mode 100644 index 00000000..c861bf8a --- /dev/null +++ b/doc/examples/learners/py_optuna.ipynb @@ -0,0 +1,8286 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "50de5573", + "metadata": {}, + "source": [ + "# Python: Hyperparametertuning with Optuna\n", + "\n", + "This notebook explains how to use the implmented `tune_ml_models()` method to tune hyperparameters using the [Optuna](https://optuna.org/) package.\n", + "\n", + "In this example, we will focus on the [DoubleMLAPO](https://docs.doubleml.org/stable/api/generated/doubleml.irm.DoubleMLAPO.html#doubleml.irm.DoubleMLAPO) model to estimate average potential outcomes (APOs) in an interactive regression model (see [DoubleMLIRM](https://docs.doubleml.org/stable/guide/models.html#binary-interactive-regression-model-irm)).\n", + "\n", + "The goal is to estimate the average potential outcome\n", + "\n", + " $$\\theta_0 =\\mathbb{E}[Y(d)]$$\n", + "\n", + "for a given treatment level $d$ and and discrete valued treatment $D$.\n", + "\n", + "For a more detailed description of the DoubleMLAPO model, see [Average Potential Outcome Model](https://docs.doubleml.org/stable/guide/models.html#average-potential-outcomes-apos) or [Example Gallery](https://docs.doubleml.org/stable/examples/index.html).\n", + "\n", + "**Remark** that the untuned settings and hyperparameter spaces are mainly chosen for display of the tuning possibilities and not a blueprint for standard tuning." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "7e56470f", + "metadata": {}, + "outputs": [], + "source": [ + "import optuna\n", + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "import seaborn as sns\n", + "from plotly.io import show\n", + "\n", + "from sklearn.pipeline import Pipeline\n", + "from sklearn.preprocessing import RobustScaler\n", + "from sklearn.linear_model import LinearRegression, Ridge, LogisticRegression\n", + "from sklearn.ensemble import StackingRegressor, StackingClassifier\n", + "from lightgbm import LGBMRegressor, LGBMClassifier\n", + "\n", + "from doubleml.data import DoubleMLData\n", + "from doubleml.irm import DoubleMLAPO, DoubleMLAPOS\n", + "from doubleml.irm.datasets import make_irm_data_discrete_treatments\n", + "\n", + "palette = sns.color_palette(\"colorblind\")\n", + "\n", + "import warnings\n", + "warnings.filterwarnings(\"ignore\")" + ] + }, + { + "cell_type": "markdown", + "id": "433cbe4c", + "metadata": {}, + "source": [ + "## Data Generating Process (DGP)" + ] + }, + { + "cell_type": "markdown", + "id": "ebdfa1ad", + "metadata": {}, + "source": [ + "At first, let us generate data according to the [make_irm_data_discrete_treatments](https://docs.doubleml.org/dev/api/datasets.html#dataset-generators) data generating process. The process generates data with a continuous treatment variable and contains the true individual treatment effects (ITEs) with respect to option of not getting treated.\n", + "\n", + "According to the continuous treatment variable, the treatment is discretized into multiple levels, based on quantiles. Using the *oracle* ITEs, enables the comparison to the true APOs and averate treatment effects (ATEs) for the different levels of the treatment variable.\n", + "\n", + "**Remark:** The average potential outcome model does not require an underlying continuous treatment variable. The model will work identically if the treatment variable is discrete by design." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "e246dbc8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Average Individual effects in each group:\n", + "[ 0. 3.44 9.32 10.49]\n", + "\n", + "Average Potential Outcomes in each group:\n", + "[210.05 213.49 219.38 220.54]\n", + "\n", + "Levels and their counts:\n", + "(array([0., 1., 2., 3.]), array([171, 110, 109, 110]))\n" + ] + } + ], + "source": [ + "# Parameters\n", + "n_obs = 500\n", + "n_levels = 3\n", + "treatment_lvl = 1.0\n", + "\n", + "np.random.seed(42)\n", + "data_apo = make_irm_data_discrete_treatments(n_obs=n_obs,n_levels=n_levels, linear=False)\n", + "\n", + "y0 = data_apo['oracle_values']['y0']\n", + "cont_d = data_apo['oracle_values']['cont_d']\n", + "ite = data_apo['oracle_values']['ite']\n", + "d = data_apo['d']\n", + "potential_level = data_apo['oracle_values']['potential_level']\n", + "level_bounds = data_apo['oracle_values']['level_bounds']\n", + "\n", + "average_ites = np.full(n_levels + 1, np.nan)\n", + "apos = np.full(n_levels + 1, np.nan)\n", + "mid_points = np.full(n_levels, np.nan)\n", + "\n", + "for i in range(n_levels + 1):\n", + " average_ites[i] = np.mean(ite[d == i]) * (i > 0)\n", + " apos[i] = np.mean(y0) + average_ites[i]\n", + "\n", + "print(f\"Average Individual effects in each group:\\n{np.round(average_ites,2)}\\n\")\n", + "print(f\"Average Potential Outcomes in each group:\\n{np.round(apos,2)}\\n\")\n", + "print(f\"Levels and their counts:\\n{np.unique(d, return_counts=True)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "1cd46dba", + "metadata": {}, + "source": [ + "As for all [DoubleML](https://docs.doubleml.org/stable/index.html) models, we specify a [DoubleMLData](https://docs.doubleml.org/stable/api/generated/doubleml.data.DoubleMLData.html) object to handle the data." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "a98bf812", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "================== DoubleMLData Object ==================\n", + "\n", + "------------------ Data summary ------------------\n", + "Outcome variable: y\n", + "Treatment variable(s): ['d']\n", + "Covariates: ['x0', 'x1', 'x2', 'x3', 'x4']\n", + "Instrument variable(s): None\n", + "No. Observations: 500\n", + "\n", + "------------------ DataFrame info ------------------\n", + "\n", + "RangeIndex: 500 entries, 0 to 499\n", + "Columns: 7 entries, y to x4\n", + "dtypes: float64(7)\n", + "memory usage: 27.5 KB\n", + "\n" + ] + } + ], + "source": [ + "y = data_apo['y']\n", + "x = data_apo['x']\n", + "d = data_apo['d']\n", + "df_apo = pd.DataFrame(\n", + " np.column_stack((y, d, x)),\n", + " columns=['y', 'd'] + ['x' + str(i) for i in range(data_apo['x'].shape[1])]\n", + ")\n", + "\n", + "dml_data = DoubleMLData(df_apo, 'y', 'd')\n", + "print(dml_data)" + ] + }, + { + "cell_type": "markdown", + "id": "397fd594", + "metadata": {}, + "source": [ + "## Basic Tuning Example\n", + "\n", + "At first, we will take a look at a very basic tuning example without much customization." + ] + }, + { + "cell_type": "markdown", + "id": "ce8478ec", + "metadata": {}, + "source": [ + "### Define Nuisance Learners\n", + "\n", + "For our example, we will choose [LightGBM](https://lightgbm.readthedocs.io/en/stable/) learners, which are typical non-parametric choice." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "5ae487e1", + "metadata": {}, + "outputs": [], + "source": [ + "ml_g = LGBMRegressor(random_state=314, verbose=-1)\n", + "ml_m = LGBMClassifier(random_state=314, verbose=-1)" + ] + }, + { + "cell_type": "markdown", + "id": "f3a336e1", + "metadata": {}, + "source": [ + "### Untuned Model\n", + "\n", + "Now let us take a look at the standard workflow, focusing on a single treatment level and using default hyperparameters." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "68ea7d50", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
coefstd errtP>|t|2.5 %97.5 %
d211.23265915.65743113.4908881.769555e-41180.544657241.920661
\n", + "
" + ], + "text/plain": [ + " coef std err t P>|t| 2.5 % 97.5 %\n", + "d 211.232659 15.657431 13.490888 1.769555e-41 180.544657 241.920661" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_obj_untuned = DoubleMLAPO(\n", + " dml_data,\n", + " ml_g,\n", + " ml_m,\n", + " treatment_level=treatment_lvl,\n", + ")\n", + "\n", + "dml_obj_untuned.fit()\n", + "dml_obj_untuned.summary" + ] + }, + { + "cell_type": "markdown", + "id": "affc29fc", + "metadata": {}, + "source": [ + "### Hyperparameter Tuning\n", + "\n", + "Now, let us take a look at the basic hyperparameter tuning. We will initialize a separate model to compare the results." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "76ff8ca0", + "metadata": {}, + "outputs": [], + "source": [ + "dml_obj_tuned = DoubleMLAPO(\n", + " dml_data,\n", + " ml_g,\n", + " ml_m,\n", + " treatment_level=treatment_lvl,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "087a8a52", + "metadata": {}, + "source": [ + "The required input for tuning is a parameter space dictionary for the hyperparameters for each learner that should be tuned.\n", + "This dictionary should include a callable for each learner you want to have (only a subset is also possible, i.e. only tuning `ml_g`)." + ] + }, + { + "cell_type": "markdown", + "id": "09d36a4a", + "metadata": {}, + "source": [ + "The parameter spaces should be a callable and suggest the search spaces via a `trial` object.\n", + "\n", + "Generally, the hyperparameter structure should follow the definitions in [Optuna](https://optuna.org/#key_features), but instead of the objective the hyperparameters have to be specified as a callable. The corresponding DoubleML object then assigns a corresponding objective for each learning using the supplied parameter space.\n", + "\n", + "To keep this example relatively fast and simple, we keep the `n_estimators` fix and only tune a small number of other hyperparameters." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "e74257e8", + "metadata": {}, + "outputs": [], + "source": [ + "# parameter space for the outcome regression tuning\n", + "def ml_g_params(trial):\n", + " return {\n", + " 'n_estimators': 100,\n", + " 'learning_rate': trial.suggest_float('learning_rate', 0.001, 0.1, log=True),\n", + " 'max_depth': 5,\n", + " 'min_child_samples': trial.suggest_int('min_child_samples', 20, 50, step=10),\n", + " 'lambda_l1': trial.suggest_float('lambda_l1', 1e-2, 10.0, log=True),\n", + " 'lambda_l2': trial.suggest_float('lambda_l2', 1e-2, 10.0, log=True),\n", + " }\n", + "\n", + "# parameter space for the propensity score tuning\n", + "def ml_m_params(trial):\n", + " return {\n", + " 'n_estimators': 100,\n", + " 'learning_rate': trial.suggest_float('learning_rate', 0.001, 0.1, log=True),\n", + " 'max_depth': 5,\n", + " 'min_child_samples': trial.suggest_int('min_child_samples', 20, 50, step=10),\n", + " 'lambda_l1': trial.suggest_float('lambda_l1', 1e-2, 10.0, log=True),\n", + " 'lambda_l2': trial.suggest_float('lambda_l2', 1e-2, 10.0, log=True),\n", + " }\n", + "\n", + "param_space = {\n", + " 'ml_g': ml_g_params,\n", + " 'ml_m': ml_m_params\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "33a2ff6c", + "metadata": {}, + "source": [ + "To tune the hyperparameters the `tune_ml_models()` with the `ml_param_space` argument should be called.\n", + "Further, to define the number of trials and other optuna options you can use the `optuna_setttings` argument." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "fe81ad26", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "584ba3a562d34a238f8bfdd41ae5fdb6", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/200 [00:00" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "optuna_settings = {\n", + " 'n_trials': 200,\n", + " 'show_progress_bar': True,\n", + " 'verbosity': optuna.logging.WARNING, # Suppress Optuna logs\n", + "}\n", + "\n", + "dml_obj_tuned.tune_ml_models(\n", + " ml_param_space=param_space,\n", + " optuna_settings=optuna_settings,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "f345dc41", + "metadata": {}, + "source": [ + "Per default, the model will set the best hyperparameters automatically (identical hyperparameters for each fold), and you can directly call the `fit()` method afterwards." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
coefstd errtP>|t|2.5 %97.5 %
d215.2734332.48586.629160.0210.402924220.143943
\n", + "
" + ], + "text/plain": [ + " coef std err t P>|t| 2.5 % 97.5 %\n", + "d 215.273433 2.485 86.62916 0.0 210.402924 220.143943" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_obj_tuned.fit()\n", + "dml_obj_tuned.summary" + ] + }, + { + "cell_type": "markdown", + "id": "db48b7d7", + "metadata": {}, + "source": [ + "**Remark**: Even if the initialization and tuning only requires the learners `ml_g` and `ml_m`, the models in the `irm` submodule generally, copy the learner for `ml_g` and fit different response surfaces for treatment and control (or not-treatment) groups. These different learners are tuned separately but with the same parameter space. To see which parameter spaces can be tuned you can take a look at the `params_names` property.\n", + "\n", + "In this example, we specified the parameter spaces for `ml_m` and `ml_g`, but actually three sets of hyperparameters were tuned, i.e. `ml_m`, `ml_g_d_lvl0` and `ml_g_d_lvl1` (two response surfaces for the outcome)." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "31a70f0a", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['ml_g_d_lvl0', 'ml_g_d_lvl1', 'ml_m']" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_obj_tuned.params_names" + ] + }, + { + "cell_type": "markdown", + "id": "3b6aa5f6", + "metadata": {}, + "source": [ + "Each hyperparameter combination is set for each fold." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "35aaf050", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'ml_g_d_lvl0': {'d': [[{'learning_rate': 0.09801463544687879,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 3.1247297280098136,\n", + " 'lambda_l2': 0.1676705013704926},\n", + " {'learning_rate': 0.09801463544687879,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 3.1247297280098136,\n", + " 'lambda_l2': 0.1676705013704926},\n", + " {'learning_rate': 0.09801463544687879,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 3.1247297280098136,\n", + " 'lambda_l2': 0.1676705013704926},\n", + " {'learning_rate': 0.09801463544687879,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 3.1247297280098136,\n", + " 'lambda_l2': 0.1676705013704926},\n", + " {'learning_rate': 0.09801463544687879,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 3.1247297280098136,\n", + " 'lambda_l2': 0.1676705013704926}]]},\n", + " 'ml_g_d_lvl1': {'d': [[{'learning_rate': 0.09957868943595276,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 1.0312190390696285,\n", + " 'lambda_l2': 0.058903541934281406},\n", + " {'learning_rate': 0.09957868943595276,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 1.0312190390696285,\n", + " 'lambda_l2': 0.058903541934281406},\n", + " {'learning_rate': 0.09957868943595276,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 1.0312190390696285,\n", + " 'lambda_l2': 0.058903541934281406},\n", + " {'learning_rate': 0.09957868943595276,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 1.0312190390696285,\n", + " 'lambda_l2': 0.058903541934281406},\n", + " {'learning_rate': 0.09957868943595276,\n", + " 'min_child_samples': 20,\n", + " 'lambda_l1': 1.0312190390696285,\n", + " 'lambda_l2': 0.058903541934281406}]]},\n", + " 'ml_m': {'d': [[{'learning_rate': 0.0732109605604835,\n", + " 'min_child_samples': 30,\n", + " 'lambda_l1': 9.590467244372398,\n", + " 'lambda_l2': 0.078138007883929},\n", + " {'learning_rate': 0.0732109605604835,\n", + " 'min_child_samples': 30,\n", + " 'lambda_l1': 9.590467244372398,\n", + " 'lambda_l2': 0.078138007883929},\n", + " {'learning_rate': 0.0732109605604835,\n", + " 'min_child_samples': 30,\n", + " 'lambda_l1': 9.590467244372398,\n", + " 'lambda_l2': 0.078138007883929},\n", + " {'learning_rate': 0.0732109605604835,\n", + " 'min_child_samples': 30,\n", + " 'lambda_l1': 9.590467244372398,\n", + " 'lambda_l2': 0.078138007883929},\n", + " {'learning_rate': 0.0732109605604835,\n", + " 'min_child_samples': 30,\n", + " 'lambda_l1': 9.590467244372398,\n", + " 'lambda_l2': 0.078138007883929}]]}}" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_obj_tuned.params" + ] + }, + { + "cell_type": "markdown", + "id": "6bccb213", + "metadata": {}, + "source": [ + "### Comparison\n", + " \n", + "Let us compare the results for both models. If we take a look at the predictive performance of the learners, the main difference can be observed in the log loss of the propensity score `ml_m`" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "20a95719", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'ml_g_d_lvl0': array([[14.49934441]]),\n", + " 'ml_g_d_lvl1': array([[23.50828321]]),\n", + " 'ml_m': array([[0.44349955]])}" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_obj_untuned.evaluate_learners()" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "68e08448", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'ml_g_d_lvl0': array([[14.73579164]]),\n", + " 'ml_g_d_lvl1': array([[23.25821083]]),\n", + " 'ml_m': array([[0.4101895]])}" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_obj_tuned.evaluate_learners()" + ] + }, + { + "cell_type": "markdown", + "id": "3b58e994", + "metadata": {}, + "source": [ + "As a result the standard error is reduced and confidence intervals are much tighter." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "f594a9f7", + "metadata": { + "tags": [ + "hide-input" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "True APO at treatment level 1.0: 213.4904\n", + "\n", + " Model theta se ci_lower ci_upper\n", + "Untuned 211.232659 15.657431 180.544657 241.920661\n", + " Tuned 215.273433 2.485000 210.402924 220.143943\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA90AAAJOCAYAAACqS2TfAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjYsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvq6yFwwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAd61JREFUeJzt3XlcVGX///H3DDvCgCiIuOKWa2pWhnvlbgtlt9ldLlnZgpqZdWeLqVnelXfar8zKSsx2u/XOFk1zNzVLU9PU1FxKRXADBEFgrt8ffpkcB2RQjoC9no8HD53rnHOdz5kzM/Ce6yw2Y4wRAAAAAAAocfbSLgAAAAAAgEsVoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGygnOnXqpE6dOpV2GSVqz549stlsSkxMLO1Syr2ZM2eqYcOG8vPzU3h4uKv95ZdfVp06deTj46MWLVpIkmrXrq2BAwcWq/+/274aOHCgateu7fW8ISEh1hZUApYuXSqbzaalS5da0n9xPqM6deqkpk2bWlLHpcBms2nMmDGlsu5Dhw7ptttuU6VKlWSz2TR58uRSqQN/H5fi3zfA2QjdwAVKTEyUzWYr9GfNmjVe9/Xrr79qzJgx2rNnj3UFn4c33nijzIStb775RjabTTExMXI6nQXOU7t2bbd9EBUVpfbt22vOnDke8xpjNHPmTHXo0EHh4eEKDg5Ws2bNNG7cOGVkZBSrtg0bNuiuu+5SjRo1FBAQoIiICHXu3FnTp09XXl7eeW2vN7Zt26aBAweqbt26mjZtmt5++21J0oIFC/T444+rbdu2mj59ul544QXLaigpZem1dqbMzEyNGTPGssB6KTpw4IDGjBmjDRs2lHYpbr755huvAm1Rn+35P95+OVNePPLII/r22281atQozZw5U927d7dkPQMHDvTq+S3uF4Tna9WqVRozZoyOHz9+UdZ3oV544QX973//82re/C9NJ06caG1RFluwYIHuueceNW3aVD4+Puf13ps7d66uuOIKBQYGqmbNmnr22WeVm5tb8sUCZ/Et7QKAS8W4ceMUGxvr0V6vXj2v+/j11181duxYderUyeOXyYIFCy60xPP2xhtvqHLlyhftj59z+fDDD1W7dm3t2bNHixcvVufOnQucr0WLFnr00Uclnf7j/6233tKtt96qqVOn6oEHHpAk5eXl6Z///Kc+++wztW/fXmPGjFFwcLBWrFihsWPHatasWfruu+9UpUqVIut655139MADD6hKlSrq16+f6tevr/T0dC1atEj33HOPDh48qCeffLLknogzLF26VE6nU6+++qrb623x4sWy2+1699135e/v72rfvn277Pbifedaq1YtnTx5Un5+fiVWd0HKymtt2rRpbl/qZGZmauzYsZLEiEwhzv6MOnDggMaOHavatWu7jrIoC7755htNmTKlyODdoUMHzZw5063t3nvv1dVXX63Bgwe72qw4yuHkyZPy9S2dP9EWL16sm2++WSNHjrR0Pffff7/b5/fu3bs1evRoDR48WO3bt3e1161b19I68q1atUpjx47VwIED3Y4WKqteeOEF3XbbbYqPjy/tUi6ajz76SJ9++qmuuOIKxcTEFHv5efPmKT4+Xp06ddJrr72mX375RePHj1dycrKmTp1qQcXAXwjdQAnp0aOHrrzySsv6PzM0/V1lZGToiy++0IQJEzR9+nR9+OGHhYbuatWq6a677nI97t+/v+rVq6dJkya5QvdLL72kzz77TCNHjtTLL7/smnfw4MHq06eP4uPjNXDgQM2bN++cda1Zs0YPPPCA4uLi9M033yg0NNQ1bfjw4frpp5+0efPmC9n0c0pOTpYkjz8Uk5OTFRQU5PHaCQgIKPY6bDabAgMDz7vG8sbqLxcuRZfaZ1SdOnVUp04dt7YHHnhAderUcftssUJpvteSk5NLNHRmZWXJ39/f44u+uLg4xcXFuR7/9NNPGj16tOLi4s75/GZkZKhChQolVh/KjxdeeEHTpk2Tn5+fbrjhhmL/Xh05cqQuv/xyLViwwPWllsPh0AsvvKCHH35YDRs2tKJs4DQD4IJMnz7dSDI//vhjkfN+/PHH5oorrjAhISEmNDTUNG3a1EyePNmtn7N/lixZYowxpmPHjqZjx46uvpYsWWIkmU8//dSMGTPGxMTEmJCQENO7d29z/Phxk5WVZR5++GETGRlpKlSoYAYOHGiysrLc6nnvvffMtddeayIjI42/v79p1KiReeONN9zmqVWrlkdNZ9Zx7Ngx8/DDD5vq1asbf39/U7duXfPvf//b5OXlufVz7NgxM2DAAONwOExYWJjp37+/+fnnn40kM336dK+e65kzZxq73W4OHjxoXnzxReNwOMzJkyc95qtVq5bp1auXR/uVV15p/Pz8jDHGZGZmmooVK5oGDRqYnJycAtd39913G0lm9erV56yre/fuxtfX1+zdu9er7Thx4oQZMWKE6zlr0KCBefnll43T6fSYd+bMmeaKK64wgYGBpmLFiub22283+/btc9vWs/fPs88+W+BrKf95rlWrlhkwYIDbeo4dO2aGDx9uatWqZfz9/U21atVMv379TEpKijHGmN27dxe4r7Zu3Wp69+5tKlasaAICAkyrVq3MF1984TZP/mt75cqV5pFHHjGVK1c2wcHBJj4+3iQnJ59zW/Jfa6dOnTJjxowx9erVMwEBASYiIsK0bdvWLFiwoNDn+dixY8Zut5tXX33V1ZaSkmJsNpuJiIhwe74feOABU6VKFdfjAQMGmFq1arlte0HPc/68FSpUMH/++ae5+eabTYUKFUzlypXNo48+anJzcwutL9///vc/07NnT1O1alXj7+9v6tSpY8aNG+exbMeOHU2TJk3Mli1bTKdOnUxQUJCJiYkxL774okeff/zxh7n55ptNcHCwiYyMNMOHDzfz5893+0wpyMaNG40kt334008/GUmmZcuWbvN2797dXH311W715e+v/M+nwl6DxdmWQ4cOmUGDBpmoqCgTEBBgLr/8cpOYmOg2T/76zt62s1+3AwYMKLAub1WoUMHtvePtevPX7e3r5MzXlzHG9Z7esWOHGTBggAkLCzMOh8MMHDjQZGRkuC2bmZlphg4daipVqmRCQkLMjTfeaP7880+PPs9W2O+gfLt27TK33XabqVixogkKCjKtW7c2X331lVsf+c/Hxx9/bJ566ikTExNjbDabOXbs2DmfV2OM+fHHHz2es/yali5dah588EETGRlpwsPDXdO/+eYb065dOxMcHGxCQkJMz549zebNm9363bhxoxkwYICJjY01AQEBpkqVKubuu+82hw8f9nh+z/7ZvXu3Meb0/khISDCfffaZadSokQkMDDTXXHON2bRpkzHGmDfffNPUrVvXBAQEmI4dO7qWO9OaNWtMt27djMPhMEFBQaZDhw5m5cqVbvN4u58LqvXsz/Qz5b8eX3755XPug6ysLDN69GhTt25d4+/vb6pXr24ee+wxt78dmjRpYjp16uSxbF5enomJiTG9e/d2a5s0aZJp3LixCQgIMFFRUWbw4MHm6NGjbsue/feNN3r16uX6jPbGli1bjCQzZcoUt/b9+/cbSea5554r1vqB4mKkGyghqampOnz4sFubzWZTpUqVJEkLFy7UHXfcoeuvv14vvviiJGnr1q36/vvv9fDDD6tDhw4aNmyY/t//+3968skn1ahRI0ly/VuYCRMmKCgoSE888YR27typ1157TX5+frLb7Tp27JjGjBmjNWvWKDExUbGxsRo9erRr2alTp6pJkya66aab5Ovrqy+//FIPPfSQnE6nEhISJEmTJ0/W0KFDFRISoqeeekqSXIdbZ2ZmqmPHjtq/f7/uv/9+1axZU6tWrdKoUaN08OBB1wV4jDG6+eabtXLlSj3wwANq1KiR5syZowEDBhTrOf7www917bXXKjo6Wn379tUTTzyhL7/8Uv/4xz+KXDYnJ0d//PGHa3+sXLlSx44d08MPP1zoYZz9+/fX9OnT9dVXX+maa64pcJ7MzEwtWrRIHTp0UM2aNYuswxijm266SUuWLNE999yjFi1a6Ntvv9Vjjz2m/fv3a9KkSa55n3/+eT3zzDPq06eP7r33XqWkpOi1115Thw4d9PPPPys8PFyTJ0/W+++/rzlz5mjq1KkKCQnR5Zdfrnr16untt9/W2rVr9c4770iS2rRpU2BNJ06cUPv27bV161YNGjRIV1xxhQ4fPqy5c+fqzz//VOXKlQtcbsuWLWrbtq2qVaumJ554QhUqVNBnn32m+Ph4/fe//9Utt9ziNv/QoUNVsWJFPfvss9qzZ48mT56sIUOG6NNPP5V07tfamDFjNGHCBNfhvWlpafrpp5+0fv16denSpcD6wsPD1bRpUy1fvlzDhg2TdHq/22w2HT16VL/++quaNGkiSVqxYoXbIa1nioyM1NSpU/Xggw/qlltu0a233ipJuvzyy13z5OXlqVu3bmrdurUmTpyo7777Tv/5z39Ut25dPfjggwX2my8xMVEhISEaMWKEQkJCtHjxYo0ePVppaWluR2BI0rFjx9S9e3fdeuut6tOnjz7//HP961//UrNmzdSjRw9Jpw9Nvv7667Vv3z4NGzZMMTExmjlzphYvXnzOOiSpadOmCg8P1/Lly3XTTTe5nhu73a6NGzcqLS1NDodDTqdTq1atcjvM+kyNGjXSuHHjPA4XPvM16O22dOrUSTt37tSQIUMUGxurWbNmaeDAgTp+/LgefvjhIrfpTPfff78OHDighQsXehw6fjFcyOtEkvr06aPY2FhNmDBB69ev1zvvvKOoqCjX7xTp9LnSn332mfr166drrrlGy5YtU69evYrsO/9w+n79+qlLly7q37+/a9qhQ4fUpk0bZWZmatiwYapUqZJmzJihm266SZ9//rnHe/25556Tv7+/Ro4cqezs7As+CuKhhx5SZGSkRo8e7brWxsyZMzVgwAB169ZNL774ojIzMzV16lS1a9dOP//8s+sUrYULF+r333/X3XffrejoaG3ZskVvv/22tmzZojVr1shms+nWW2/Vb7/9po8//liTJk1yfeZFRka6alixYoXmzp3r+t04YcIE3XDDDXr88cf1xhtv6KGHHtKxY8f00ksvadCgQW7vt8WLF6tHjx5q1aqVnn32Wdntdk2fPl3XXXedVqxYoauvvtpte4vazzNnzvQ41eFCD8N3Op266aabtHLlSg0ePFiNGjXSL7/8okmTJum3335znT9+++23a8yYMUpKSlJ0dLRr+ZUrV+rAgQPq27evq+3+++9XYmKi7r77bg0bNky7d+/W66+/rp9//lnff//9RT2i6Oeff5YkjyMSY2JiVL16ddd0wDKlnfqB8q6w0QFJJiAgwDXfww8/bBwOxzlHvmbNmlXoSFRhI91NmzY1p06dcrXfcccdxmazmR49ergtHxcX5/GtcGZmpsd6unXrZurUqePW1qRJkwK/hX7uuedMhQoVzG+//ebW/sQTTxgfHx/XiOz//vc/I8m89NJLrnlyc3NN+/btvR7pPnTokPH19TXTpk1ztbVp08bcfPPNHvPWqlXLdO3a1aSkpJiUlBSzceNG07dvXyPJDB061BhjzOTJk40kM2fOnELXefToUSPJ3HrrrYXOkz8y+PDDDxe5Dcb89VyMHz/erf22224zNpvN7Ny50xhjzJ49e4yPj495/vnn3eb75ZdfjK+vr1t7/uhI/qh0vvyRtbOdPdI9evRoI8nMnj3bY9780eCCRu6uv/5606xZM7dREKfTadq0aWPq16/vast/j3Tu3NltdPmRRx4xPj4+5vjx4662wl5rzZs3L/DohaIkJCS4jWCPGDHCdOjQwURFRZmpU6caY4w5cuSIsdlsbiPiZ450G3N6hFyFjBTmj56OGzfOrb1ly5amVatWRdZY0Pvw/vvvN8HBwW7PbceOHY0k8/7777vasrOzTXR0tNvoUv5r+7PPPnO1ZWRkmHr16hU50m3M6RGkM0ewb731VnPrrbcaHx8fM2/ePGOMMevXr/cYET/7M6qgkcvz3ZYPPvjA1Xbq1CkTFxdnQkJCTFpamjGmeCPOCQkJxRrdPtOFjnR7+zo5+7WW/x4fNGiQ23y33HKLqVSpkuvxunXrjCQzfPhwt/kGDhxY5Ej3metOSEhwaxs+fLiRZFasWOFqS09PN7GxsaZ27dquI5vyn486deoU+Lo+l3ONdLdr187td2d6eroJDw839913n1sfSUlJJiwszK29oDo+/vhjI8ksX77c1fbyyy+7jW6fKf/3+ZnT3nrrLSPJREdHu16HxhgzatQot36cTqepX7++6datm9vnX2ZmpomNjTVdunRxtXm7n43xfC2eizcj3flHkp25j405PYovyXz//ffGGGO2b99uJJnXXnvNbb6HHnrIhISEuJ7vFStWGEnmww8/dJsv/4ibM9svxkh3/v4980ixfFdddZW55pprirV+oLi4ejlQQqZMmaKFCxe6/Zx5LnB4eLgyMjK0cOHCEl1v//793b4tbt26tYwxGjRokNt8rVu31h9//OF2lc6goCDX//NH6jt27Kjff/9dqampRa571qxZat++vSpWrKjDhw+7fjp37qy8vDwtX75c0ukLF/n6+rqN5Pj4+Gjo0KFeb+cnn3wiu92u3r17u9ruuOMOzZs3T8eOHfOYf8GCBYqMjFRkZKSaN2+uWbNmqV+/fq6RgvT0dElyO//6bPnT0tLSCp0nf9q5+jnTN998Ix8fH9fIa75HH31UxhjXa2b27NlyOp3q06eP23MbHR2t+vXra8mSJV6tzxv//e9/1bx5c4/RKun00RoFOXr0qBYvXqw+ffooPT3dVd+RI0fUrVs37dixQ/v373dbZvDgwW79tW/fXnl5edq7d2+RNYaHh2vLli3asWNHsbatffv2OnTokLZv3y7p9GhVhw4d1L59e61YsULS6REaY0yhI93eyr9WwJnr/v3334tc7sz3Yf5z2b59e2VmZmrbtm1u84aEhLid7+rv76+rr77abT3ffPONqlatqttuu83VFhwcXOio9Nnat2+v9evXu0YUV65cqZ49e6pFixau52zFihWy2Wxq166dV30WxNttiY6O1h133OFq8/Pz07Bhw3TixAktW7bsvNdfWs73dVLYskeOHHF9Ds2fP1/S6ZHhMxXns7Yg33zzja6++mq3/R0SEqLBgwdrz549+vXXX93mHzBggNvr+kLdd9998vHxcT1euHChjh8/rjvuuMPt89HHx0etW7d2+3w8s46srCwdPnzYdeTS+vXrva7h+uuvd7vAaevWrSVJvXv3dvv8z2/P36cbNmzQjh079M9//lNHjhxx1ZqRkaHrr79ey5cv97gTR1H72QqzZs1So0aN1LBhQ7fn9LrrrpMk13PaoEEDtWjRwnWEknT6CI7PP/9cN954o+v5njVrlsLCwtSlSxe3/lq1aqWQkJAS/R3mjZMnT0oq+JomgYGBrumAVTi8HCghV1999TkvpPbQQw/ps88+U48ePVStWjV17dpVffr0ueDbsZx9SHNYWJgkqUaNGh7tTqdTqamprkOsv//+ez377LNavXq1MjMz3eZPTU119VWYHTt2aNOmTW6H4J0p/wJfe/fuVdWqVT2u8nvZZZcVsXV/+eCDD3T11VfryJEjOnLkiCSpZcuWOnXqlGbNmuURKFq3bq3x48fLZrMpODhYjRo1crs4UP4fSfnhuyDeBHOHw1FkP2fau3evYmJiPPrMP40gP4Du2LFDxhjVr1+/wH5K8rC8Xbt2uX2Z4Y2dO3fKGKNnnnlGzzzzTIHzJCcnq1q1aq7HZ79WK1asKEkFfmlytnHjxunmm29WgwYN1LRpU3Xv3l39+vVzO8S7IPlBesWKFa5DCMePH6/IyEjX7XNWrFghh8Oh5s2bF1lHYQIDAz3eBxUrVvRq27Zs2aKnn35aixcv9vij+uwvv6pXr+7xRUjFihW1adMm1+O9e/eqXr16HvN5+35r3769cnNztXr1atWoUUPJyclq3769tmzZ4ha6GzdurIiICK/6LIi321K/fn2Pi3Cd/X4pLy7kdSKd+z3kcDi0d+9e2e12jztpFOcuGgXZu3evK0ye6cz9cOZ91wu6k8eFOLu//C/f8gPh2fI/l6XTXxCOHTtWn3zyiet3Uj5vvlzOV5zftdJfn2v5tZ7rdKrU1FTXvixoXWfvZyvs2LFDW7duLfL3uXT6EPMnn3xS+/fvV7Vq1bR06VIlJyfr9ttvd+svNTVVUVFRRfZ3MeR/GZCdne0xLSsrq0S/JAIKQugGLpKoqCht2LBB3377rebNm6d58+Zp+vTp6t+/v2bMmHHe/Z757b837cYYSaeD1vXXX6+GDRvqlVdeUY0aNeTv769vvvlGkyZNKvQe2GdyOp3q0qWLHn/88QKnN2jQwMutOLcdO3boxx9/lKQCQ+iHH37oEborV65c6JXNpb/+WNy0aVOht1zJ/+O/cePGhfZTr149+fr66pdffjnnNhSX0+mUzWbTvHnzCtyXVtymqDjyXx8jR45Ut27dCpzn7D/0i3pNnkuHDh20a9cuffHFF1qwYIHeeecdTZo0SW+++abuvffeQpeLiYlRbGysli9frtq1a8sYo7i4OEVGRurhhx/W3r17tWLFCrVp06bYt1HzZtuKcvz4cXXs2FEOh0Pjxo1T3bp1FRgYqPXr1+tf//qXx/vwQp5Db1155ZUKDAzU8uXLVbNmTUVFRalBgwZq37693njjDWVnZ2vFihUFHhlRHCW5LYUdkZGXl1fsvqxc7/m+TopaviT3f0ko6QBzdn/574uZM2e6nVec78zrdPTp00erVq3SY489phYtWigkJEROp1Pdu3f36vdcvvP9XZu/jpdffrnQW+ed/XleGvvZ6XSqWbNmeuWVVwqcfuaXC7fffrtGjRqlWbNmafjw4frss88UFhbmNojgdDoVFRWlDz/8sMD+Cgv3Vqlataok6eDBgx5flBw8eNDjvHqgpBG6gYvI399fN954o2688UY5nU499NBDeuutt/TMM88UODJlpS+//FLZ2dmaO3eu27fqBR3yVVhddevW1YkTJ84ZbqXT93hetGiRTpw44fbHRf4hv0X58MMP5efnp5kzZ3r8MbJy5Ur9v//3/7Rv3z6vLmSWr127dgoPD9dHH32kp556qsA/ct5//31J0g033FBoP8HBwbruuuu0ePFi/fHHHx6/zM9Wq1Ytfffdd0pPT3cb7c4/jLhWrVqSTj+3xhjFxsaW2JcXhalbt26xb72SfyslPz+/Ivd/cZzrPRAREaG7775bd999t06cOKEOHTpozJgx5wzd0umR2+XLlys2NlYtWrRQaGiomjdvrrCwMM2fP1/r16933YP7fOq6EEuXLtWRI0c0e/ZsdejQwdW+e/fu8+6zVq1a2rx5s4wxbnV7+37LP8x7xYoVqlmzputogfbt2ys7O1sffvihDh065FZvQUriOatVq5Y2bdokp9Pp9qXI2e+X/JHA48ePuy1f0Eh4Se7L4qz3YqhVq5acTqd2797t9gXlzp07L7jfgl4/Z++HiyX/omFRUVHn/Pw5duyYFi1apLFjx7pdRLSg01Sseo/n1+pwOC7aZ+X5qFu3rjZu3Kjrr7++yL5jY2N19dVX69NPP9WQIUM0e/ZsxcfHux26XbduXX333Xdq27ZtmRhFzv/C46effnIL2AcOHNCff/7p9ek3wPninG7gIsk/JDqf3W53HRqbf7hT/r1Hz/4Dzgr5IfPMb85TU1M1ffp0j3krVKhQYE19+vTR6tWr9e2333pMO378uOv88Z49eyo3N1dTp051Tc/Ly9Nrr73mVa0ffvih2rdvr9tvv1233Xab289jjz0mSfr444+96itfcHCwRo4cqe3bt7uulH2mr7/+WomJierWrVuhVy7P9+yzz8oYo379+unEiRMe09etW+c6mqFnz57Ky8vT66+/7jbPpEmTZLPZXFdtvvXWW+Xj46OxY8d6jG4YYzxeTxeid+/e2rhxo+bMmeMxrbCRlaioKHXq1ElvvfWWDh486DE9JSXlvGop7LV29vaGhISoXr16BR4qeLb27dtrz549+vTTT10B0m63q02bNnrllVeUk5NT5PncwcHBkkr+vVnQ+/DUqVN64403zrvPnj176sCBA/r8889dbZmZmXr77be97qN9+/b64YcftGTJEtdzU7lyZTVq1Mh1XYSinrOS+Dzr2bOnkpKS3M4fzc3N1WuvvaaQkBB17NhR0unQ5+Pj47qORL6CnseS/JwtznovhvyjTs5ev7eftYXp2bOn1q5dq9WrV7vaMjIy9Pbbb6t27drnPBrICt26dXPdXzknJ8djev7nT0HvL0muO2ucyarfv61atVLdunU1ceLEAn8/lPRn5fnq06eP9u/fr2nTpnlMO3nypOsaD/luv/12rVmzRu+9954OHz7sdmh5fn95eXl67rnnPPrLzc219O+cnJwcbdu2ze13U5MmTdSwYUO9/fbbbkeiTJ06VTabze0aGIAVGOkGSsi8efM8Lnoknb5FTp06dXTvvffq6NGjuu6661S9enXt3btXr732mlq0aOE61LlFixby8fHRiy++qNTUVAUEBOi6664r9JyoC9G1a1fXyPv999+vEydOaNq0aYqKivIIUa1atdLUqVM1fvx41atXT1FRUbruuuv02GOPae7cubrhhhs0cOBAtWrVShkZGfrll1/0+eefa8+ePapcubJuvPFGtW3bVk888YT27Nmjxo0ba/bs2V6dT/fDDz+4bhdUkGrVqumKK67Qhx9+qH/961/Feg6eeOIJ/fzzz3rxxRe1evVq9e7dW0FBQVq5cqU++OADNWrUyKtD/9u0aaMpU6booYceUsOGDdWvXz/Vr19f6enpWrp0qebOnavx48dLkm688UZde+21euqpp7Rnzx41b95cCxYs0BdffKHhw4e7RkXq1q2r8ePHa9SoUdqzZ4/i4+MVGhqq3bt3a86cORo8eLBGjhxZrO0tzGOPPabPP/9c//jHPzRo0CC1atVKR48e1dy5c/Xmm28Weq7zlClT1K5dOzVr1kz33Xef6tSpo0OHDmn16tX6888/tXHjxmLXUthrrXHjxurUqZNatWqliIgI/fTTT/r8888LfV2cKT8cbt++XS+88IKrvUOHDpo3b54CAgJ01VVXnbOPoKAgNW7cWJ9++qkaNGigiIgINW3a1O081vPRpk0bVaxYUQMGDNCwYcNks9k0c+bMCzqM9L777tPrr7+u/v37a926dapatapmzpzp+uLAG+3bt9fzzz+vP/74wy1cd+jQQW+99ZZq166t6tWrn7OPunXrKjw8XG+++aZCQ0NVoUIFtW7duljn+w4ePFhvvfWWBg4cqHXr1ql27dr6/PPP9f3332vy5Mmuo0XCwsL0j3/8Q6+99ppsNpvq1q2rr776qsDzRlu1aiVJGjZsmLp16yYfHx+32xwVR3HWezG0atVKvXv31uTJk3XkyBHXLcN+++03Sec/OvrEE0/o448/Vo8ePTRs2DBFRERoxowZ2r17t/773/9e0KkZ58PhcGjq1Knq16+frrjiCvXt21eRkZHat2+fvv76a7Vt21avv/66HA6HOnTooJdeekk5OTmqVq2aFixYUOCRJPmvi6eeekp9+/aVn5+fbrzxRlcYP192u13vvPOOevTooSZNmujuu+9WtWrVtH//fi1ZskQOh0Nffvllsftt1aqVvvvuO73yyiuu02gKOu/+TIsWLVJWVpZHe3x8vPr166fPPvtMDzzwgJYsWaK2bdsqLy9P27Zt02effaZvv/3W7bo1ffr00ciRIzVy5EhFRER4jOJ37NhR999/vyZMmKANGzaoa9eu8vPz044dOzRr1iy9+uqrxQ66mzZt0ty5cyWdPnojNTXV9bu1efPmuvHGGyVJ+/fvV6NGjTRgwAAlJia6ln/55Zd10003qWvXrurbt682b96s119/Xffee2+Rt2cFLthFvFI6cEk61y3DdMbtTz7//HPTtWtXExUVZfz9/U3NmjXN/fffbw4ePOjW37Rp00ydOnWMj4+P261oCrtl2KxZswqs58cff3RrL+i2UnPnzjWXX365CQwMNLVr1zYvvviiee+99zxum5KUlGR69eplQkNDjSS3OtLT082oUaNMvXr1jL+/v6lcubJp06aNmThxotutzI4cOWL69etnHA6HCQsLM/369TM///xzkbcMGzp0qJFkdu3aVeg8Y8aMMZLMxo0bjTGnb4nl7e2l8vLyzPTp003btm2Nw+EwgYGBpkmTJmbs2LHmxIkTXvWRb926deaf//yniYmJMX5+fqZixYrm+uuvNzNmzHDdUseY08/ZI4884pqvfv365uWXX3a7nUy+//73v6Zdu3amQoUKpkKFCqZhw4YmISHBbN++3TXPhd4yzJjT+2fIkCGmWrVqxt/f31SvXt0MGDDAHD582BhT8C2QjDFm165dpn///iY6Otr4+fmZatWqmRtuuMF8/vnnrnkKe00WdLulwl5r48ePN1dffbUJDw83QUFBpmHDhub55593e42dS1RUlJFkDh065GpbuXKlkWTat2/vMf/ZtwwzxphVq1aZVq1aGX9/f7fbLxX2POfvl6J8//335pprrjFBQUEmJibGPP744+bbb7/1eG46duxomjRp4lWte/fuNTfddJMJDg42lStXNg8//LDrVj1F3TLMGGPS0tKMj4+PCQ0NdbtV0wcffGAkmX79+nksU9Btf7744gvTuHFj4+vr6/b6Kc62HDp0yNx9992mcuXKxt/f3zRr1qzAz4yUlBTTu3dvExwcbCpWrGjuv/9+s3nzZo/XbW5urhk6dKiJjIw0NputWLcPK+g2Td6utzivkzNfX2fOc/Z7PP+9debndUZGhklISDAREREmJCTExMfHu27z9O9//7vIbVQBtwwz5vR7/bbbbjPh4eEmMDDQXH311earr75ym6ew30veONctw87+7Dhzfd26dTNhYWEmMDDQ1K1b1wwcOND89NNPrnn+/PNPc8stt5jw8HATFhZm/vGPf5gDBw4UeAu15557zlSrVs3Y7Xa357Wg56Sw23AV9hz8/PPP5tZbbzWVKlUyAQEBplatWqZPnz5m0aJFrnmKs5+3bdtmOnToYIKCgoykc94+LL/Wwn5mzpxpjDl9O74XX3zRNGnSxAQEBJiKFSuaVq1ambFjx5rU1FSPftu2bWskmXvvvbfQdb/99tumVatWJigoyISGhppmzZqZxx9/3Bw4cMA1j7e3DDvX31tnbn/+9hb0nMyZM8e0aNHCBAQEmOrVq5unn37a698jwIWwGVPGrr4BAACAErNhwwa1bNlSH3zwge68887SLgcA/nY4pxsAAOASUdD9hidPniy73V7kxe8AANbgnG4AAIBLxEsvvaR169bp2muvla+vr+sWlYMHDy7y7goAAGtweDkAAMAlYuHChRo7dqx+/fVXnThxQjVr1lS/fv301FNPud2/GgBw8RC6AQAAAACwCOd0AwAAAABgEUI3AAAAAAAW4eQeSU6nUwcOHFBoaKhsNltplwMAAAAAKOOMMUpPT1dMTIzs9sLHswndkg4cOMAVPQEAAAAAxfbHH3+oevXqhU4ndEsKDQ2VdPrJcjgcpVxNwZxOp1JSUhQZGXnOb1EAAAAAoDwrL9knLS1NNWrUcOXJwhC6Jdch5Q6Ho0yH7qysLDkcjjL9wgMAAACAC1Hesk9RpyiX/S0AAAAAAKCcInQDAAAAAGARQjcAAAAAABbhnG4AAAAAsFheXp5ycnJKu4xywel0KicnR1lZWaV6Trefn598fHwuuB9CNwAAAABYxBijpKQkHT9+vLRLKTeMMXI6nUpPTy/yImVWCw8PV3R09AXVQegGAAAAAIvkB+6oqCgFBweXeogsD4wxys3Nla+vb6k9X8YYZWZmKjk5WZJUtWrV8+6L0A0AAAAAFsjLy3MF7kqVKpV2OeVGWQjdkhQUFCRJSk5OVlRU1Hkfas6F1AAAAADAAvnncAcHB5dyJThf+fvuQs7HJ3QDAAAAgIU4pLz8Kol9R+gGAAAAAMAihG4AAAAAACxC6AYAAAAAuNhstnP+jBkz5qLX9PHHH8vHx0cJCQke05YuXepWX5UqVdS7d2/9/vvvbvOtWrVKPXv2VMWKFRUYGKhmzZrplVdeUV5enqW1E7oBAAAAAC4HDx50/UyePFkOh8OtbeTIka558680brV3331Xjz/+uD7++GNlZWUVOM/27dt14MABzZo1S1u2bNGNN97oCtRz5sxRx44dVb16dS1ZskTbtm3Tww8/rPHjx6tv374yxlhWO6EbAAAAAOASHR3t+gkLC5PNZnM93rZtm0JDQzVv3jy1atVKAQEBWrlypQYOHKj4+Hi3foYPH65OnTq5HjudTk2YMEGxsbEKCgpS8+bN9fnnnxdZz+7du7Vq1So98cQTatCggWbPnl3gfFFRUapatao6dOig0aNH69dff9XOnTuVkZGh++67TzfddJPefvtttWjRQrVr19a9996rGTNm6PPPP9dnn312IU/ZORG6AQAAAADF8sQTT+jf//63tm7dqssvv9yrZSZMmKD3339fb775prZs2aJHHnlEd911l5YtW3bO5aZPn65evXopLCxMd911l959990i15V/j+1Tp05pwYIFOnLkiNsIfb4bb7xRDRo00Mcff+zVNpwPX8t6BgAAAAAU6NCKfTq0cl+R8wVXC1W9/s3d2na+v1GZ+9OLXLZKu5qq0r7medd4LuPGjVOXLl28nj87O1svvPCCvvvuO8XFxUmS6tSpo5UrV+qtt95Sx44dC1zO6XQqMTFRr732miSpb9++evTRR7V7927FxsYWuMzBgwc1ceJEVatWTZdddpm++eYbSVKjRo0KnL9hw4b67bffvN6W4iJ0AwAAAMBFlpedq5y07CLnyw0L8Gw7ccqrZfOyrTvX+sorryzW/Dt37lRmZqZHUD916pRatmxZ6HILFy5URkaGevbsKUmqXLmyunTpovfee0/PPfec27zVq1eXMUaZmZlq3ry5/vvf/8rf39813crzts+F0A0AAAAAF5lPgK/8HJ6B+my+If4FtnmzrE+AdXGvQoUKbo/tdrtHqM3JyXH9/8SJE5Kkr7/+WtWqVXObLyCg8G159913dfToUdfh4tLp0e9NmzZp7Nixstv/OmN6xYoVcjgcioqKUmhoqKu9QYMGkqStW7eqTZs2HuvYunWrGjduXGgNF4rQDQAAAAAXWZX253/o99mHm5cFkZGR2rx5s1vbhg0b5OfnJ0lq3LixAgICtG/fvkIPJT/bkSNH9MUXX+iTTz5RkyZNXO15eXlq166dFixYoO7du7vaY2NjFR4e7tFP165dFRERof/85z8eoXvu3LnasWOHx6h5SSJ0AwAAAAAuyHXXXaeXX35Z77//vuLi4vTBBx9o8+bNrkPHQ0NDNXLkSD3yyCNyOp1q166dUlNT9f3338vhcGjAgAEefc6cOVOVKlVSnz59ZLPZ3Kb17NlT7777rlvoLkyFChX01ltvqW/fvho8eLCGDBkih8OhRYsW6bHHHtNtt92mPn36lMwTUQBCdxnzyrJdmrT89wKnOfPyZPfxKXDaIx3qaETHulaWBgAAAAAF6tatm5555hk9/vjjysrK0qBBg9S/f3/98ssvrnmee+45RUZGasKECfr9998VHh6uK664Qk8++WSBfU6fPl233HKLR+CWpN69e6tfv346fPiwV/XddtttWrJkiZ5//nm1b99eWVlZql+/vp566ikNHz68wHWUFJsprbPJy5C0tDSFhYUpNTVVDoejVGsZ8+12jVtY/Cvnje7SQGO6XWZBRQAAAADOR1ZWlusq24GBgaVdTrlhjFFubq58fX0tDcPeONc+9DZHMtJdxjgCfVUtzH1nGmN04P+uThjjCCjwhecIZFcCAAAAQFlDUitjRnSs63GYeEZ2rkKfmidJ2vp4J4UGel7BEAAAAABQ9tiLngUAAAAAAJwPQjcAAAAAABYhdAMAAAAAYBFCNwAAAAAAFiF0AwAAAABgEa5eDgAAAABlyCvLdmnS8t/d2owxchrJyMgmm+w2edxK+JEOdTzuhITSR+gGAAAAgDIkLStX+1Ozzms5lD0cXg4AAAAAZYgj0FfVwgJVKdhPtiLmtUmqFOynamGBcgQyplqUxMREhYeHX9R1lmronjBhgq666iqFhoYqKipK8fHx2r59e4HzGmPUo0cP2Ww2/e9//3Obtm/fPvXq1UvBwcGKiorSY489ptxcvuUBAAAAUP6M6FhX7/RpruMnc2UrInXbbNLxk7l6p0/zEj20vFOnTho+fLhHe3FDa+3atTV58uQSq6s8KtXQvWzZMiUkJGjNmjVauHChcnJy1LVrV2VkZHjMO3nyZI9zFiQpLy9PvXr10qlTp7Rq1SrNmDFDiYmJGj169MXYBAAAAAAoUcdP5ui2GT/J6PR53OeSf573bTN+0vGTORenQBRLqYbu+fPna+DAgWrSpImaN2+uxMRE7du3T+vWrXObb8OGDfrPf/6j9957z6OPBQsW6Ndff9UHH3ygFi1aqEePHnruuec0ZcoUnTp16mJtCgAAAACUiBk//aHMU3lFBu58TiNlnsrT+z/9YW1hZxk4cKDi4+M1ceJEVa1aVZUqVVJCQoJyck6H/06dOmnv3r165JFHZLPZXIOoY8aMUYsWLdz6mjx5smrXru16fM899+iWW24ptG9Jys7O1siRI1WtWjVVqFBBrVu31tKlS936TUxMVM2aNRUcHKxbbrlFR44cseS5OJcyddB/amqqJCkiIsLVlpmZqX/+85+aMmWKoqOjPZZZvXq1mjVrpipVqrjaunXrpgcffFBbtmxRy5YtPZbJzs5Wdna263FaWpokyel0yul0ltj2lBSn+aumslojAAAAAHdOp1PGGNePN4wxen3l7vNa32srd2tI29oFHiF8PgqqO/9x/r9LlixRdHS0Fi9erJ07d6pv375q3ry57rvvPv33v/9VixYtdN999+m+++7z6PPMvgtqO1ffkpSQkKCtW7fq448/VkxMjObMmaPu3btr06ZNql+/vn744Qfdc889euGFFxQfH6/58+drzJgxHuvx5jkoKId5m8vKTOh2Op0aPny42rZtq6ZNm7raH3nkEbVp00Y333xzgcslJSW5BW5JrsdJSUkFLjNhwgSNHTvWoz0lJUVZWcW/SqDVMk/luf6fkpKikwF+pVgNAAAAAG/k5OTI6XQqNzfX62tOHc44pV1HMou9LiNp15FMJaedVKUK/sVe3qO//wubZ9edHzRzc3PldDpVsWJFTZ48WT4+PqpXr5569Oih7777TnfffbccDod8fHxUoUIFVa5c2W25s/s+s9/8dZ+r73379ikxMVG7du1STEyMJGn48OGaP3++3n33XY0fP16TJ09Wt27dNGLECEnSQw89pO+//14LFizwen/k13vkyBH5+bnnsPT0dK/6KDOhOyEhQZs3b9bKlStdbXPnztXixYv1888/l+i6Ro0a5XripdMj3TVq1FBkZKQcDkeJrqskZJz66wURGRmp0MALfxMBAAAAsFZWVpbS09Pl6+srX1/voldW3oWdInsyT16v61zyDwc/uy+7/fQZyr6+vrLb7WrSpIkCAgJc02NiYrR582a35ex2u8fjs/s+s9/89Z+r761btyovL09NmjRxqy87O1uVK1eWr6+vtm/frvj4eLf1tGnTRgsWLPD6OcrfzkqVKikwMNBt2tmPC+3Dq7ksNmTIEH311Vdavny5qlev7mpfvHixdu3a5XF1vN69e6t9+/ZaunSpoqOjtXbtWrfphw4dkqQCD0eXpICAALedl89ut7t2dllit/1VU1mtEQAAAIC7/HB55vnMRQm9wNt+OQL9SuTwcofDobS0NI++UlNTFRYW5mr383Nfn91ul9PpdGs7e/t9fHxkjHFryx95ttlsrkO/z9V3RkaGfHx8tG7dOvn4+LjVGBIS4lru7HWf2e6N/OULymHe5rJSDd3GGA0dOlRz5szR0qVLFRsb6zb9iSee0L333uvW1qxZM02aNEk33nijJCkuLk7PP/+8kpOTFRUVJUlauHChHA6HGjdufHE2BAAAAABKQKVgf9WtFKzfj2TKy+uoSTp9v+46lYIVEVwyp6JedtllWrBggUf7+vXr1aBBA6/78ff3V15enltbZGSkkpKS3IL3hg0bilVfy5YtlZeXp+TkZLVv377AeRo1aqQffvjBrW3NmjXFWk9JKNUh04SEBH3wwQf66KOPFBoaqqSkJCUlJenkyZOSTo9UN23a1O1HkmrWrOkK6F27dlXjxo3Vr18/bdy4Ud9++62efvppJSQkFDiaDQAAAABllc1m05B2sUXPWICh7WJL7CJqDz74oH777TcNGzZMmzZt0vbt2/XKK6/o448/1qOPPup1P7Vr19by5cu1f/9+HT58WNLpq5qnpKTopZde0q5duzRlyhTNmzevWPU1aNBAd955p/r376/Zs2dr9+7dWrt2rSZMmKCvv/5akjRs2DDNnz9fEydO1I4dO/T6669r/vz5xVpPSSjV0D116lSlpqaqU6dOqlq1quvn008/9boPHx8fffXVV/Lx8VFcXJzuuusu9e/fX+PGjbOwcgAAAACwxoArayjY30d2L/Oz3SYF+/uo/5U1SqyGOnXqaPny5dq2bZs6d+6s1q1b67PPPtOsWbPUvXt3r/sZN26c9uzZo7p16yoyMlLS6RHoN954Q1OmTFHz5s21du1ajRw5stg1Tp8+Xf3799ejjz6qyy67TPHx8frxxx9Vs2ZNSdI111yjadOm6dVXX1Xz5s21YMECPf3008Vez4WyGW+vlX4JS0tLU1hYmFJTU8vmhdSycxX61OlvflLHd+NCagAAAEA5kJWVpd27dys2Ntbri27l+3Z7sm54Z62MzDnv1223STbZ9PW9V6vrZVEXWHHZkH9lc19f3xIbuT9f59qH3uZIrsgFAAAAAGXIK8t26d7PNio8yFdFDZEaI4UH+eqezzbqlWW7Lk6BKJYycfVyAAAAAMBpaVm52p+a5dW8RtKRzBxJOUrL8u7e07i4CN0AAAAAUIY4An1VLcz9UGZjTh9mbmRkk+30YeVnHXrtuMDbjcEa7BUAAAAAKENGdKyrER3rlnYZKCGc0w0AAAAAgEUI3QAAAAAAWITQDQAAAACARTinGwAAAADKkNT1k5W6/lW3NmOMJOfpe4TZbJLsHhdSC7viYYVdMfyi1QnvELoBAAAAoAxxZqcp78T+81oOZQ+HlwMAAABAGWIPcMgnpJrsgZUk2YqY2yZ7YKXT8wc4LkZ5pSYxMVHh4eGlXUaxEboBAAAAoAwJu2K4Knd+S87s4/ImdDuzj6ty57dK7NBym812zp8xY8aUyHr+Lji8HAAAAADKkLys40r++vbT52/LWcTcTsnYlfz17apxz275BIZf8PoPHjzo+v+nn36q0aNHa/v27a62kJCQC17H3wkj3QAAAABQhpzYOlMmJ1NFB+58TpmcTJ3Y+kGJrD86Otr1ExYWJpvN5nr85ptvql27dm7zT548WbVr13Y9HjhwoOLj4zVx4kRVrVpVlSpVUkJCgnJyclzzZGdna+TIkapWrZoqVKig1q1ba+nSpW79JiYmqmbNmgoODtYtt9yiI0eOlMj2XWyEbgAAAAAoI4wxStsw5byWTdvw+v9d5bz0LVmyRLt27dKSJUs0Y8YMJSYmKjEx0TV9yJAhWr16tT755BNt2rRJ//jHP9S9e3ft2LFDkrR27Vrde++9GjJkiDZs2KBrr71W48ePL6WtuTAcXg4AAAAAZYQz64hyU38/jyWNclN/lzPrqHyCKpV4XcVVsWJFvf766/Lx8VHDhg3Vq1cvLVq0SPfdd5/27dun6dOna9++fYqJiZEkjRw5UvPnz9f06dP1/PPP67XXXlP37t31+OOPS5IaNGigVatWaf78+aW5WeeFkW4AAAAAKCOcp05c4PLpJVTJhWnSpIl8fHxcj6tWrark5GRJ0i+//KK8vDw1aNBAISEhrp9ly5Zp165dkqRt27bp6quvduszLi7u4m1ACWKkGwAAAADKCLv/hV2kzO4fWkKVFNK/3e5xCPuZ52rn8/Pzc3tss9nkdJ4+R/3EiRPy8fHRunXr3IK5dGlepI3QDQAAAABlhD2wknzD6ig3dbek4pyfbZNvWKzsgRFWlSZJioyMVFJSkowxstlO385sw4YNxeqjZcuWysvLU3Jystq3b+8x3Rijhg0bau3atW7ta9asOe+6SxOHlwMAAABAGWGz2eRokXBeyzpaDHEFYat06tRJKSkpeumll7Rr1y5NmTJF8+bNK1YfDRo00J133qn+/ftr9uzZ2r17t9auXasJEybo66+/lnT6Qmvz58/XxIkTtWPHDr3++uvl8nxuidANAAAAAGVKSKN+svkFy/u4ZpfNL1ghje6ysixJUqNGjfTGG29oypQpat68udauXauRI0cWu5/p06erf//+evTRR3XZZZcpPj5eP/74o2rWrClJat26td5++229+uqrat68uRYsWKCnn366pDfnorCZsnJN+VKUlpamsLAwpaamyuFwlHY5HjKycxX61Olvj1LHd1NooH8pVwQAAACgKFlZWdq9e7diY2MVGBhYrGUz9yzQoS9ulozRue/XbZdsNlWJn6vgWl0uqN6ywhij3Nxc+fr6Wj5yX5Rz7UNvcyQj3QAAAABQhqSun6zD390ve0C4ij6v28geEK7DCwcrdf1k64tDsXEhNQAAAAAoQ5zZaco7sd/LuY2cWUdcy6HsIXQDAAAAQBliD3DIJ6SaW5vJP8zcGMlmk2T3OPTaHlD2TpUFoRsAAAAAypSwK4Yr7IrhpV0GSgjndAMAAACAhbh2dflVEvuO0A0AAAAAFvDz85MkZWZmlnIlOF/5+y5/X54PDi8HAAAAAAv4+PgoPDxcycnJkqTg4OBSvwVWeVAWbhlmjFFmZqaSk5MVHh4uHx+f8+6L0A0AAAAAFomOjpYkV/BG0Ywxcjqdsts9LxZ3sYWHh7v24fkidAMAAACARWw2m6pWraqoqCjl5OSUdjnlgtPp1JEjR1SpUiXZ7aV3RrSfn98FjXDnI3QDAAAAgMV8fHxKJMD9HTidTvn5+SkwMLBUQ3dJKf9bAAAAAABAGUXoBgAAAADAIoRuAAAAAAAsQugGAAAAAMAihG4AAAAAACxC6AYAAAAAwCKEbgAAAAAALELoBgAAAADAIoRuAAAAAAAsQugGAAAAAMAihG4AAAAAACxC6AYAAAAAwCKEbgAAAAAALELoBgAAAADAIoRuAAAAAAAsQugGAAAAAMAihG4AAAAAACxC6AYAAAAAwCKEbgAAAAAALELoBgAAAADAIoRuAAAAAAAsQugGAAAAAMAihG4AAAAAACxC6AYAAAAAwCKEbgAAAAAALELoBgAAAADAIoRuAAAAAAAsQugGAAAAAMAihG4AAAAAACxC6AYAAAAAwCKEbgAAAAAALELoBgAAAADAIoRuAAAAAAAsQugGAAAAAMAihG4AAAAAACxSqqF7woQJuuqqqxQaGqqoqCjFx8dr+/btbvPcf//9qlu3roKCghQZGambb75Z27Ztc5tn37596tWrl4KDgxUVFaXHHntMubm5F3NTAAAAAADwUKqhe9myZUpISNCaNWu0cOFC5eTkqGvXrsrIyHDN06pVK02fPl1bt27Vt99+K2OMunbtqry8PElSXl6eevXqpVOnTmnVqlWaMWOGEhMTNXr06NLaLAAAAAAAJEk2Y4wp7SLypaSkKCoqSsuWLVOHDh0KnGfTpk1q3ry5du7cqbp162revHm64YYbdODAAVWpUkWS9Oabb+pf//qXUlJS5O/vX+R609LSFBYWptTUVDkcjhLdppKQkZ2r0KfmSZJSx3dTaGDR2wQAAAAA5ZHT6VRycrKioqJkt5fdM6K9zZFlagtSU1MlSREREQVOz8jI0PTp0xUbG6saNWpIklavXq1mzZq5ArckdevWTWlpadqyZYv1RQMAAAAAUAjf0i4gn9Pp1PDhw9W2bVs1bdrUbdobb7yhxx9/XBkZGbrsssu0cOFC1wh2UlKSW+CW5HqclJRU4Lqys7OVnZ3tepyWluaqwel0ltg2lRSn+aumslojAAAAAJQEp9MpY0yZzz3e1ldmQndCQoI2b96slStXeky788471aVLFx08eFATJ05Unz599P333yswMPC81jVhwgSNHTvWoz0lJUVZWVnn1aeVMk/luf6fkpKikwF+pVgNAAAAAFjH6XQqNTVVxpgyfXh5enq6V/OVidA9ZMgQffXVV1q+fLmqV6/uMT0sLExhYWGqX7++rrnmGlWsWFFz5szRHXfcoejoaK1du9Zt/kOHDkmSoqOjC1zfqFGjNGLECNfjtLQ01ahRQ5GRkWXznO5Tf12JPTIyknO6AQAAAFyynE6nbDabIiMjy3To9nYQuFRDtzFGQ4cO1Zw5c7R06VLFxsZ6tYwxxnV4eFxcnJ5//nnXifaStHDhQjkcDjVu3LjAPgICAhQQEODRbrfby+ROtdv+qqms1ggAAAAAJcVms5X57ONtbaUauhMSEvTRRx/piy++UGhoqOsc7LCwMAUFBen333/Xp59+qq5duyoyMlJ//vmn/v3vfysoKEg9e/aUJHXt2lWNGzdWv3799NJLLykpKUlPP/20EhISCgzWAAAAAABcLKX6tcHUqVOVmpqqTp06qWrVqq6fTz/9VNLp4foVK1aoZ8+eqlevnm6//XaFhoZq1apVrlFtHx8fffXVV/Lx8VFcXJzuuusu9e/fX+PGjSvNTQMAAAAAoPQPLz+XmJgYffPNN0X2U6tWLa/mAwAAAADgYiq7B8gDAAAAAFDOEboBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLlGronjBhgq666iqFhoYqKipK8fHx2r59u2v60aNHNXToUF122WUKCgpSzZo1NWzYMKWmprr1s2/fPvXq1UvBwcGKiorSY489ptzc3Iu9OQAAAAAAuCnV0L1s2TIlJCRozZo1WrhwoXJyctS1a1dlZGRIkg4cOKADBw5o4sSJ2rx5sxITEzV//nzdc889rj7y8vLUq1cvnTp1SqtWrdKMGTOUmJio0aNHl9ZmAQAAAAAgSbIZY0xpF5EvJSVFUVFRWrZsmTp06FDgPLNmzdJdd92ljIwM+fr6at68ebrhhht04MABValSRZL05ptv6l//+pdSUlLk7+9f5HrT0tIUFham1NRUORyOEt2mkpCRnavQp+ZJklLHd1NoYNHbBAAAAABlWer6yUpd/6rnBCPlOfPkY/eRbJ6Tw654WGFXDLe8vqJ4myN9L2JNRco/bDwiIuKc8zgcDvn6ni599erVatasmStwS1K3bt304IMPasuWLWrZsqVHH9nZ2crOznY9TktLkyQ5nU45nc4S2ZaS5DR/1VRWawQAAACA4sjLSlXeif2FTz/HcmUhE3lbQ5kJ3U6nU8OHD1fbtm3VtGnTAuc5fPiwnnvuOQ0ePNjVlpSU5Ba4JbkeJyUlFdjPhAkTNHbsWI/2lJQUZWVlne8mWCbz1F8vt5SUFJ0M8CvFagAAAADgwuWcsktB0We1GunkodP/Dawi2TyHujNO2XUqOdn6AouQnp7u1XxlJnQnJCRo8+bNWrlyZYHT09LS1KtXLzVu3Fhjxoy5oHWNGjVKI0aMcOu7Ro0aioyMLJuHl5/666JwkZGRHF4OAAAAoPyLekrq8JRbkzMnQ39MrSRJqjZgs3wDQkujMq8EBgZ6NV+ZCN1DhgzRV199peXLl6t69eoe09PT09W9e3eFhoZqzpw58vP7a6Q3Ojpaa9eudZv/0KFDrmkFCQgIUEBAgEe73W6X3V727qJmt/1VU1mtEQAAAAAumL38ZB9vayvVLTDGaMiQIZozZ44WL16s2NhYj3nS0tLUtWtX+fv7a+7cuR7fJsTFxemXX35R8hmHFyxcuFAOh0ONGze2fBsAAAAAAChMqY50JyQk6KOPPtIXX3yh0NBQ1znYYWFhCgoKcgXuzMxMffDBB0pLS3Nd9CwyMlI+Pj7q2rWrGjdurH79+umll15SUlKSnn76aSUkJBQ4mg0AAAAAwMVSqqF76tSpkqROnTq5tU+fPl0DBw7U+vXr9cMPP0iS6tWr5zbP7t27Vbt2bfn4+Oirr77Sgw8+qLi4OFWoUEEDBgzQuHHjLso2AAAAAABQmFIN3UXdIrxTp05FziNJtWrV0jfffFNSZQEAAAAAUCLK7lnpAAAAAACUc4RuAAAAAAAsQugGAAAAAMAihG4AAAAAACxyXqF7165devrpp3XHHXe47o89b948bdmypUSLAwAAAACgPCt26F62bJmaNWumH374QbNnz9aJEyckSRs3btSzzz5b4gUCAAAAAFBeFTt0P/HEExo/frwWLlwof39/V/t1112nNWvWlGhxAAAAAACUZ8UO3b/88otuueUWj/aoqCgdPny4RIoCAAAAAOBSUOzQHR4eroMHD3q0//zzz6pWrVqJFAUAAAAAwKWg2KG7b9+++te//qWkpCTZbDY5nU59//33GjlypPr3729FjQAAAAAAlEvFDt0vvPCCGjZsqBo1aujEiRNq3LixOnTooDZt2ujpp5+2okYAAAAAAMol3+Iu4O/vr2nTpumZZ57R5s2bdeLECbVs2VL169e3oj4AAAAAAMqtYofufDVr1lTNmjVLshYAAAAAAC4pxQ7dgwYNOuf0995777yLAQAAAADgUlLs0H3s2DG3xzk5Odq8ebOOHz+u6667rsQKAwAAAACgvCt26J4zZ45Hm9Pp1IMPPqi6deuWSFEAAAAAAFwKin318gI7sds1YsQITZo0qSS6AwAAAADgklAioVuSdu3apdzc3JLqDgAAAACAcq/Yh5ePGDHC7bExRgcPHtTXX3+tAQMGlFhhAAAAAACUd8UO3T///LPbY7vdrsjISP3nP/8p8srmAAAAAAD8nRQ7dC9ZssSKOgAAAAAAuOSU2DndAAAAAADAnVcj3S1btpTNZvOqw/Xr119QQQAAAACAvydjjOv/eSePyMc/xOssWlZ5Fbrj4+MtLgMAAAAA8HeVl3VcJ7bOVNrPr7vaDiQ2kG9YHTlaJCikUT/5BIaXXoEXwKvQ/eyzz1pdBwAAAADgbyhzzwIlf327TE6mx7Tc1N06umykjq0arahenyq4dtdSqPDCcE43AAAAAKBUZO5ZoENf3CyTc1KS+b+fM51uMzkndeiLm5W5Z8HFL/ICFTt05+XlaeLEibr66qsVHR2tiIgItx8AAAAAAIqSl3VcyV/fLhkjyVnE3E7JGCV/fbvyso5fhOpKTrFD99ixY/XKK6/o9ttvV2pqqkaMGKFbb71VdrtdY8aMsaBEAAAAAMCl5sTWmf93SHlRgTufUyYnUye2fmBlWSWu2KH7ww8/1LRp0/Too4/K19dXd9xxh9555x2NHj1aa9assaJGAAAAAMAlxBijtA1TzmvZtA2vu13lvKwrduhOSkpSs2bNJEkhISFKTU2VJN1www36+uuvS7Y6AAAAAMAlx5l1RLmpv8vzHO6iGOWm/i5n1lEryrJEsUN39erVdfDgQUlS3bp1tWDB6RPZf/zxRwUEBJRsdQAAAACAS47z1IkLXD69hCqxXrFD9y233KJFixZJkoYOHapnnnlG9evXV//+/TVo0KASLxAAAAAAcGmx+4dc4PKhJVSJ9by6T7ckvf7667rrrrv073//29V2++23q2bNmlq9erXq16+vG2+80ZIiAQAAAACXDntgJfmG1VFu6m4V7xBzm3zDYmUPLD93zvJ6pPupp55STEyM7rzzTi1evNjVHhcXpxEjRhC4AQAAAABesdlscrRIOK9lHS2GyGazlXBF1vE6dCclJenNN9/UgQMH1KVLF8XGxuq5557TH3/8YWV9AAAAAIBLUEijfrL5Bcv7WGqXzS9YIY3usrKsEud16A4KClL//v21ZMkS7dixQ/369dO7776r2NhYde/eXbNmzVJOTo6VtQIAAAAALhE+geGK6vWpZLOp6Ghql2w2Rd3wmXwCwy9CdSWn2BdSk6Q6depo3Lhx2r17t+bNm6dKlSpp4MCBqlatWknXBwAAAAC4RAXX7qoqN38hm1+QJNv//ZzpdJvNL0hV4ucquFaXi1/kBfL6QmoFsdls8vX1lc1mkzGGkW4AAAAAQLEE1+6qGvfs1omtHyjt59eUm7bbNc03LFaOFkMU2rif7AFhpVjl+Tuvke4//vhD48aNU506ddSlSxcdOHBA06ZNc92/GwAAAAAAb/kEhius5RDF3LXO1RZz9w5VH7hVYS2HlNvALRVjpPvUqVOaPXu23nvvPS1evFhVq1bVgAEDNGjQINWpU8fKGgEAAAAAfwNnXpXcJzCiXF2lvDBeh+7o6GhlZmbqhhtu0Jdffqlu3brJbj+vgXIAAAAAAP4WvA7dTz/9tPr166fIyEgr6wEAAAAA4JLhdegeMWKElXUAAAAAAHDJuaCrl19qNv9ntUICK5xznuBqoarXv7lb2873Nypzf3qR/VdpV1NV2td0Pc7LztWWV9YUudxJYzzajm89rH3/21bksvYAHzUdEefW9uc3O3R046Eilw1rWEm1bmnk1rb19bXKST9V5LLVe9RTRIto1+OslAz99s7PRS4nSY0SrpKfI8D1OGXtfh1ctPscS5wWWDlYDe67wq1t9yeblb77eJHLVr4qRjGd3a9NsGnCSq/qjb29iULrVHQ9Tv/9mHZ/usWrZS8f1c7t8YHvftfhHw8UuVxobLhi+zZ1a/tt2nplHc4sctmq18cq8uq/bu+Xk5atrVN+9KreBve2VGDkX++RoxuS9Oe8nUUu5xfqr0ZDrnZr2ztnq1K3HSly2YjmVVS9Z323ts2vrJYzO6/IZWvGN1R4o8quxxn707Tr/U1FLidJTUZcI5+Avz4iD63Yp0Mr9xW5XGl8RkhS3f6Xq0I1h+sxnxGe+IzgM+JsfEbwGXEmPiP4jDgbnxGl9RmRqtCg04+3vLJaNgV7LFtWPiP2fP6rV+skdJ8hJz1bOafO/ZTkhgV4tp04pZy07CL7z8vOdW8w8mq5nAJCt8nN82pZe4CPR1vuyVyvls3NzPVoy0n3bludOe4fZMZpvFpOksxZ2+vM9m5bfQI9911uZo53+yargG31sl5nrtPjsbfLFlSHd/vG8/Z8OV6+Ds/+JWNMMfaN86x9k+PdvilIbqaX23qy4H3jzS9Lk3vWtuZ6v606622Xl+1lvaXwGSGd3jb3x3xGnI3PCD4jzsZnBJ8RZ+Izgs+Is/EZUVqfEaek/wvdOWmnZJPnOsrKZ0TeSe9umU3oPoNfaID8Aj1f6GfyDfEvsO3Mb0sKc+a3XZIkm7xaLtcY6ehJ90V9fbxatqA3gm+Qr1fL+gZ7vjz8Qj23v8D1+rmv12a3ebVOSR5XKLQHeLetfgXtm2A/7/ZNAW8ib+u1+9o9Hnu7bEF1eLdv/Dza/EL8C/ylf7azXxM2WzH2jf2sfePn5b4p4HXjG+zltgYVvG+8+WVp8z1rW32931addaFMnwAv6y2Fzwjp9La5P+YzwqM2PiP4jDgLnxF8RrjVxmcEnxFn4TOitD4j/noO/Bz+ssmzr7LyGeET5PleKnCd5uyoX4Rx48Zp5MiRCg52H+Y/efKkXn75ZY0ePbo43ZUJaWlpCgsLU2pqqhwOR9ELXGQZ2bkKfWqeJCl1fDeFBnr3CwsAAAAAyhNnTob2Tjl9ykWNB4/INyC0lCsqnLc5stj3/Bo7dqxOnDjh0Z6ZmamxY8cWtzsAAAAAAC5ZxQ7dxpgCb1C+ceNGRURElEhRAAAAAABcCrw+p7tixYqy2Wyy2Wxq0KCBW/DOy8vTiRMn9MADD1hSJAAAAAAA5ZHXoXvy5MkyxmjQoEEaO3aswsLCXNP8/f1Vu3ZtxcXFnaMHAAAAAAD+XrwO3QMGDJAkxcbGqk2bNvLz8+5KbQAAAAAA/F0V+5ZhHTt2lNPp1G+//abk5GQ5ne73FuzQoUOJFQcAAAAAQHlW7NC9Zs0a/fOf/9TevXs9bixus9mUl1f0Pe9QPGc+z0cyTikkwK/Ai9kBAAAAAMqWYofuBx54QFdeeaW+/vprVa1alfBnoeMnczTjpz/0/1bsdrXVmbBEdSsFa0i7WA24sobCvbwhOwAAAADg4rOZs4eri1ChQgVt3LhR9erVs6qmi87bm5pfTN9uT9ZtM35S5qnTRw6cuZPyv+YI9vfR5wOuVLfLoi56fQAAAABQ0pw5Gdo7paIkqcaDR+QbEFrKFRXO2xxZ7Pt0t27dWjt37ryg4nBu325P1g3vrNXJnDwZuQdu/d9jI+lkTp5ueGetvt2efPGLBAAAAAAUqdiHlw8dOlSPPvqokpKS1KxZM4+rmF9++eUlVtzf0fGTObptxk8yMnIWcQyC00h2m9FtM37SH8904VBzAAAAAChjih26e/fuLUkaNGiQq81ms8kYw4XUSsCMn/5Q5qk8j9HtwjiNlHkqT+//9IeGta9jaW0AAAAAgOIpdujevXt30TPhvBhj9PrK83t+X1u5W0PbxXJhOwAAAAAoQ4odumvVqmVFHZB0JPOUdh3JLPZyRtKuI5k6mpmjShX8S74wAAAAAMB5KfaF1CRp5syZatu2rWJiYrR3715J0uTJk/XFF1+UaHF/NyeyL+zQ/PTs3BKqBAAAAABQEooduqdOnaoRI0aoZ8+eOn78uOsc7vDwcE2ePLmk6/tbCQnwuaDlQwOKfeACAAAAAMBCxQ7dr732mqZNm6annnpKPj5/hcQrr7xSv/zyS4kW93dTKdhfdSsFq7hnZdsk1a0UrIhgrl4OAAAAAGVJsUP37t271bJlS4/2gIAAZWRklEhRf1c2m01D2sWe17JcRA0AAAAAyp5ih+7Y2Fht2LDBo33+/Plq1KhRSdT0tzbgyhoK9veR3cv8bLdJwf4+6n9lDWsLAwAAAAAUW7FD94gRI5SQkKBPP/1UxhitXbtWzz//vEaNGqXHH3+8WH1NmDBBV111lUJDQxUVFaX4+Hht377dbZ63335bnTp1ksPhkM1m0/Hjxz36OXr0qO688045HA6Fh4frnnvu0YkTJ4q7aWVCeJCfPh9wpWyyFRm87TbJJpv+O+BKhQdxaDkAAAAAlDU2Y4wp7kIffvihxowZo127dkmSYmJiNHbsWN1zzz3F6qd79+7q27evrrrqKuXm5urJJ5/U5s2b9euvv6pChQqSTl8VPSsrS5I0atQoHTt2TOHh4W799OjRQwcPHtRbb72lnJwc3X333brqqqv00UcfeVVHWlqawsLClJqaKofDUaxtsMq325N124yflHnq9IXqztxJ+Vk82N9H/x1wpbpeFnXR6wMAAACAC5G6frJS17/q3miM8jIOSJJ8KlSVbJ7jxGFXPKywK4ZfhArPzdsceV6hO19mZqZOnDihqKiSCX0pKSmKiorSsmXL1KFDB7dpS5cu1bXXXusRurdu3arGjRvrxx9/1JVXXinp9KHuPXv21J9//qmYmJgi11sWQ7ckHT+Zo/d/+kOvrtit3Uf/un933UrBGtouVgOurKEwRrgBAAAAlEPHVo/T8R/GF3u58NZPq2LcaAsqKh5vc+QF3WMqODhYwcHBF9KFm9TUVElSRESE18usXr1a4eHhrsAtSZ07d5bdbtcPP/ygW265pcTqu9jCg/w0rH0dDbqqhhxPz5ck7X7yWtWsWIGLpgEAAAAo1+wBDvmEVPOcYKQ8Z5587D4q6NZO9oCyM1DqDa9C9xVXXKFFixapYsWKatmy5TkD3/r168+rEKfTqeHDh6tt27Zq2rSp18slJSV5jLT7+voqIiJCSUlJBS6TnZ2t7Oxs1+O0tDRXDU6n8zyqt5Y54+Dy8EBfGWN0AQcoAAAAAECpC20xTKEthnm0O51OpaSkKDIyUnZ7wZchKwu5zdsavArdN998swICAiRJ8fHx513UuSQkJGjz5s1auXKlJf2facKECRo7dqxHe0pKiuv88bIk/7xu6XSNJwM4pBwAAADApcnpdCo1NVXGmEJDd1mQnp7u1Xxehe5nn322wP+XlCFDhuirr77S8uXLVb169WItGx0dreTkZLe23NxcHT16VNHR0QUuM2rUKI0YMcL1OC0tTTVq1FBkZGSZOqc7X8apXNf/IyMjFRroX4rVAAAAAIB1nE6nbDbbOUe6y4LAwECv5iv2Od0//vijnE6nWrdu7db+ww8/yMfHx+3c6qIYYzR06FDNmTNHS5cuVWxsbHHLUVxcnI4fP65169apVatWkqTFixcXWGO+gIAA18j9mex2e5ncqfYzrthXVmsEAAAAgJJis9nKfPbxtrZib0FCQoL++OMPj/b9+/crISGh2H198MEH+uijjxQaGqqkpCQlJSXp5MmTrnmSkpK0YcMG7dy5U5L0yy+/aMOGDTp69KgkqVGjRurevbvuu+8+rV27Vt9//72GDBmivn37enXlcgAAAAAArFLs0P3rr7/qiiuu8Ghv2bKlfv3112L1NXXqVKWmpqpTp06qWrWq6+fTTz91zfPmm2+qZcuWuu+++yRJHTp0UMuWLTV37lzXPB9++KEaNmyo66+/Xj179lS7du309ttvF3fTAAAAAAAoUcU+vDwgIECHDh1SnTp13NoPHjwoX9/idefNFbjHjBmjMWPGnHOeiIgIffTRR8VaNwAAAAAAViv2SHfXrl01atQo1z21Jen48eN68skn1aVLlxItDgAAAACA8qzYI90TJ05Uhw4dVKtWLbVs2VKStGHDBlWpUkUzZ84s8QIBAAAAACivih26q1Wrpk2bNunDDz/Uxo0bFRQUpLvvvlt33HGH/Py4fzQAAAAAAPmKHbolqUKFCho8eHBJ1wIAAAAAwCXFq9A9d+5c9ejRQ35+fm5XDS/ITTfdVCKFAQAAAABQ3nkVuuPj45WUlKSoqCjFx8cXOp/NZlNeXl5J1QYAAAAAQLnmVeh2Op0F/h8AAAAAABTOq1uGRURE6PDhw5KkQYMGKT093dKiAAAAAAC4FHgVuk+dOqW0tDRJ0owZM5SVlWVpUQAAAAAAXAq8Orw8Li5O8fHxatWqlYwxGjZsmIKCggqc97333ivRAgEAAAAAKK+8Ct0ffPCBJk2apF27dkmSUlNTGe0GAAAAAKAIXoXuKlWq6N///rckKTY2VjNnzlSlSpUsLQwAAAAAgPKu2BdSu/baa+Xv729pUQAAAAAAXAq4kBoAAAAAABbhQmoAAAAAAFik2BdSs9lsXEgNAAAAAAAvcCE1AAAAAAAs4lXoPtPu3btd/8/KylJgYGCJFgQAAAAAwKXCqwupncnpdOq5555TtWrVFBISot9//12S9Mwzz+jdd98t8QIBAAAAACivih26x48fr8TERL300ktutw5r2rSp3nnnnRItDgAAAACA8qzYofv999/X22+/rTvvvFM+Pj6u9ubNm2vbtm0lWhwAAAAAAOVZsUP3/v37Va9ePY92p9OpnJycEikKAAAAAIBLQbFDd+PGjbVixQqP9s8//1wtW7YskaIAAAAAALgUFPvq5aNHj9aAAQO0f/9+OZ1OzZ49W9u3b9f777+vr776yooaAQAAAAAol4o90n3zzTfryy+/1HfffacKFSpo9OjR2rp1q7788kt16dLFihoBAAAAACiXij3SLUnt27fXwoULS7oWAAAAAAAuKecVuiVp3bp12rp1qySpSZMmnM8NAAAAAMBZih26k5OT1bdvXy1dulTh4eGSpOPHj+vaa6/VJ598osjIyJKuEQAAAACAcqnY53QPHTpU6enp2rJli44ePaqjR49q8+bNSktL07Bhw6yoEQAAAACAcqnYI93z58/Xd999p0aNGrnaGjdurClTpqhr164lWhwAAAAAAOVZsUe6nU6n/Pz8PNr9/PzkdDpLpCgAAAAAAC4FxQ7d1113nR5++GEdOHDA1bZ//3498sgjuv7660u0OAAAAAAAyrNih+7XX39daWlpql27turWrau6desqNjZWaWlpeu2116yoEQAAAACAcqnY53TXqFFD69ev13fffadt27ZJkho1aqTOnTuXeHEAAAAAAJRn53WfbpvNpi5duqhLly4lXQ8AAAAAAJcMrw8vX7x4sRo3bqy0tDSPaampqWrSpIlWrFhRosUBAAAAAFCeeR26J0+erPvuu08Oh8NjWlhYmO6//3698sorJVocAAAAAADlmdehe+PGjerevXuh07t27ap169aVSFEAAAAAAFwKvA7dhw4dKvD+3Pl8fX2VkpJSIkUBAAAAAHAp8Dp0V6tWTZs3by50+qZNm1S1atUSKQoAAAAAgEuB16G7Z8+eeuaZZ5SVleUx7eTJk3r22Wd1ww03lGhxAAAAAACUZ17fMuzpp5/W7Nmz1aBBAw0ZMkSXXXaZJGnbtm2aMmWK8vLy9NRTT1lWKAAAAAAA5Y3XobtKlSpatWqVHnzwQY0aNUrGGEmn79ndrVs3TZkyRVWqVLGsUAAAAAAAyhuvQ7ck1apVS998842OHTumnTt3yhij+vXrq2LFilbVBwAAAABAuVWs0J2vYsWKuuqqq0q6FgAAAAAALileX0gNAAAAAAAUD6EbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIuUauieMGGCrrrqKoWGhioqKkrx8fHavn272zxZWVlKSEhQpUqVFBISot69e+vQoUNu8+zbt0+9evVScHCwoqKi9Nhjjyk3N/dibgoAAAAAAB5KNXQvW7ZMCQkJWrNmjRYuXKicnBx17dpVGRkZrnkeeeQRffnll5o1a5aWLVumAwcO6NZbb3VNz8vLU69evXTq1CmtWrVKM2bMUGJiokaPHl0amwQAAAAAgIvNGGNKu4h8KSkpioqK0rJly9ShQwelpqYqMjJSH330kW677TZJ0rZt29SoUSOtXr1a11xzjebNm6cbbrhBBw4cUJUqVSRJb775pv71r38pJSVF/v7+Ra43LS1NYWFhSk1NlcPhsHQbz0dGdq5Cn5onSUod302hgUVvEwAAAACUR06nU8nJyYqKipLdXnbPiPY2R/pexJqKlJqaKkmKiIiQJK1bt045OTnq3Lmza56GDRuqZs2artC9evVqNWvWzBW4Jalbt2568MEHtWXLFrVs2dJjPdnZ2crOznY9TktLk3R65zqdTku27UI4zV81ldUaAQAAAKAkOJ1OGWPKfO7xtr4yE7qdTqeGDx+utm3bqmnTppKkpKQk+fv7Kzw83G3eKlWqKCkpyTXPmYE7f3r+tIJMmDBBY8eO9WhPSUlRVlbWhW5Kics8lef6f0pKik4G+JViNQAAAABgHafTqdTUVBljyvRId3p6ulfzlZnQnZCQoM2bN2vlypWWr2vUqFEaMWKE63FaWppq1KihyMjIsnl4+am/LgoXGRnJ4eUAAAAALllOp1M2m02RkZFlOnQHBgZ6NV+ZCN1DhgzRV199peXLl6t69equ9ujoaJ06dUrHjx93G+0+dOiQoqOjXfOsXbvWrb/8q5vnz3O2gIAABQQEeLTb7fYyuVPttr9qKqs1AgAAAEBJsdlsZT77eFtbqW6BMUZDhgzRnDlztHjxYsXGxrpNb9Wqlfz8/LRo0SJX2/bt27Vv3z7FxcVJkuLi4vTLL78oOTnZNc/ChQvlcDjUuHHji7MhAAAAAAAUoFRHuhMSEvTRRx/piy++UGhoqOsc7LCwMAUFBSksLEz33HOPRowYoYiICDkcDg0dOlRxcXG65pprJEldu3ZV48aN1a9fP7300ktKSkrS008/rYSEhAJHswEAAAAAuFhKNXRPnTpVktSpUye39unTp2vgwIGSpEmTJslut6t3797Kzs5Wt27d9MYbb7jm9fHx0VdffaUHH3xQcXFxqlChggYMGKBx48ZdrM0AAAAAAKBAZeo+3aWF+3QDAAAAQNlwqd2nu+xuAQAAAAAA5RyhGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsAihGwAAAAAAixC6AQAAAACwCKEbAAAAAACLELoBAAAAALAIoRsAAAAAAIsQugEAAAAAsEiphu7ly5frxhtvVExMjGw2m/73v/+5TT906JAGDhyomJgYBQcHq3v37tqxY4fbPFlZWUpISFClSpUUEhKi3r1769ChQxdxKwAAAAAAKFiphu6MjAw1b95cU6ZM8ZhmjFF8fLx+//13ffHFF/r5559Vq1Ytde7cWRkZGa75HnnkEX355ZeaNWuWli1bpgMHDujWW2+9mJsBAAAAAECBfEtz5T169FCPHj0KnLZjxw6tWbNGmzdvVpMmTSRJU6dOVXR0tD7++GPde++9Sk1N1bvvvquPPvpI1113nSRp+vTpatSokdasWaNrrrnmom0LAAAAAABnK9XQfS7Z2dmSpMDAQFeb3W5XQECAVq5cqXvvvVfr1q1TTk6OOnfu7JqnYcOGqlmzplavXl1o6M7Oznb1L0lpaWmSJKfTKafTacXmXBCn+aumslojAAAAAJQEp9MpY0yZzz3e1ldmQ3d+eB41apTeeustVahQQZMmTdKff/6pgwcPSpKSkpLk7++v8PBwt2WrVKmipKSkQvueMGGCxo4d69GekpKirKysEt2OkpB5Ks/1/5SUFJ0M8CvFagAAAADAOk6nU6mpqTLGyG4vu9f+Tk9P92q+Mhu6/fz8NHv2bN1zzz2KiIiQj4+POnfurB49esgYc0F9jxo1SiNGjHA9TktLU40aNRQZGSmHw3GhpZe4jFO5rv9HRkYqNNC/FKsBAAAAAOs4nU7ZbDZFRkaW6dB95lHZ51JmQ7cktWrVShs2bFBqaqpOnTqlyMhItW7dWldeeaUkKTo6WqdOndLx48fdRrsPHTqk6OjoQvsNCAhQQECAR7vdbi+TO9Vu+6umslojAAAAAJQUm81W5rOPt7WV3S04Q1hYmCIjI7Vjxw799NNPuvnmmyWdDuV+fn5atGiRa97t27dr3759iouLK61yAQAAAACQVMoj3SdOnNDOnTtdj3fv3q0NGzYoIiJCNWvW1KxZsxQZGamaNWvql19+0cMPP6z4+Hh17dpV0ukwfs8992jEiBGKiIiQw+HQ0KFDFRcXx5XLAQAAAAClrlRD908//aRrr73W9Tj/POsBAwYoMTFRBw8e1IgRI3To0CFVrVpV/fv31zPPPOPWx6RJk2S329W7d29lZ2erW7dueuONNy7qdgAAAAAAUBCbudCrkl0C0tLSFBYWptTU1LJ5IbXsXIU+NU+SlDq+GxdSAwAAAHDJcjqdSk5OVlRUVJk+p9vbHFl2twAAAAAAgHKO0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYxLe0C4C7V5bt0qTlv7u1GWNc/2/00lLZbDaP5R7pUEcjOta1vD4AAAAAgPdKdaR7+fLluvHGGxUTEyObzab//e9/btNPnDihIUOGqHr16goKClLjxo315ptvus2TlZWlhIQEVapUSSEhIerdu7cOHTp0EbeiZKVl5Wp/apbbz4G0bNf0A2nZHtP3p2YpLSu3FKsGAAAAABSkVEe6MzIy1Lx5cw0aNEi33nqrx/QRI0Zo8eLF+uCDD1S7dm0tWLBADz30kGJiYnTTTTdJkh555BF9/fXXmjVrlsLCwjRkyBDdeuut+v777y/25pQIR6CvqoUFFjjNmZcnu49PocsBAAAAAMoWmznz2OVSZLPZNGfOHMXHx7vamjZtqttvv13PPPOMq61Vq1bq0aOHxo8fr9TUVEVGRuqjjz7SbbfdJknatm2bGjVqpNWrV+uaa67xat1paWkKCwtTamqqHA5HiW5XSXE6nUpOTlZUVJTsdk7FBwAAAHBpKi/Zx9scWXa3QFKbNm00d+5c7d+/X8YYLVmyRL/99pu6du0qSVq3bp1ycnLUuXNn1zINGzZUzZo1tXr16tIqGwAAAAAASWX8QmqvvfaaBg8erOrVq8vX11d2u13Tpk1Thw4dJElJSUny9/dXeHi423JVqlRRUlJSof1mZ2crO/uv86TT0tIknf5Gxel0lvyGlACn0yljTJmtDwAAAABKQnnJPt7WV+ZD95o1azR37lzVqlVLy5cvV0JCgmJiYtxGt4trwoQJGjt2rEd7SkqKsrKyLqRkyzidTqWmpsoYU6YPsQAAAACAC1Fesk96erpX85XZ0H3y5Ek9+eSTmjNnjnr16iVJuvzyy7VhwwZNnDhRnTt3VnR0tE6dOqXjx4+7jXYfOnRI0dHRhfY9atQojRgxwvU4LS1NNWrUUGRkZJk+p9tmsykyMrJMv/AAAAAA4EKUl+wTGFjwBbDPVmZDd05OjnJycjyeZB8fH9cwfqtWreTn56dFixapd+/ekqTt27dr3759iouLK7TvgIAABQQEeLTb7fYyvVNtNluZrxEAAAAALlR5yD7e1laqofvEiRPauXOn6/Hu3bu1YcMGRUREqGbNmurYsaMee+wxBQUFqVatWlq2bJnef/99vfLKK5KksLAw3XPPPRoxYoQiIiLkcDg0dOhQxcXFeX3lcgAAAAAArFKqofunn37Stdde63qcf8j3gAEDlJiYqE8++USjRo3SnXfeqaNHj6pWrVp6/vnn9cADD7iWmTRpkux2u3r37q3s7Gx169ZNb7zxxkXfFgAAAAAAzlZm7tNdmrhPNwAAAACUDeUl+1wS9+kGAAAAAKA8I3QDAAAAAGARQjcAAAAAABYhdAMAAAAAYBFCNwAAAAAAFiF0AwAAAABgEUI3AAAAAAAWIXQDAAAAAGARQjcAAAAAABYhdAMAAAAAYBFCNwAAAAAAFiF0AwAAAABgEd/SLqAsMMZIktLS0kq5ksI5nU6lp6crMDBQdjvflQAAAAC4NJWX7JOfH/PzZGEI3ZLS09MlSTVq1CjlSgAAAAAA5Ul6errCwsIKnW4zRcXyvwGn06kDBw4oNDRUNputtMspUFpammrUqKE//vhDDoejtMsBAAAAAEuUl+xjjFF6erpiYmLOOSLPSLcku92u6tWrl3YZXnE4HGX6hQcAAAAAJaE8ZJ9zjXDnK7sHyAMAAAAAUM4RugEAAAAAsAihu5wICAjQs88+q4CAgNIuBQAAAAAsc6llHy6kBgAAAACARRjpBgAAAADAIoRuAAAAAAAsQugGAAAAAMAihG54SExMVHh4eGmXAQAAAAAXpCxkG0L3BejUqZOGDx/u0V7cHVu7dm1Nnjy5xOoCAAAAgIvJZrOd82fMmDGlXWKp8S3tAgAAAAAA5dvBgwdd///00081evRobd++3dUWEhJSGmWVCYx0W2zgwIGKj4/XxIkTVbVqVVWqVEkJCQnKycmRdHq0fO/evXrkkUdc3wJJ0pgxY9SiRQu3viZPnqzatWt73bckZWdna+TIkapWrZoqVKig1q1ba+nSpW79JiYmqmbNmgoODtYtt9yiI0eOWPJcAAAAALg0RUdHu37CwsJks9lcj9988021a9fObf6/U7YhdF8ES5Ys0a5du7RkyRLNmDFDiYmJSkxMlCTNnj1b1atX17hx43Tw4EG3b4gutG9JGjJkiFavXq1PPvlEmzZt0j/+8Q91795dO3bskCT98MMPuueeezRkyBBt2LBB1157rcaPH19Smw4AAAAAXrlUsw2Hl18EFStW1Ouvvy4fHx81bNhQvXr10qJFi3TfffcpIiJCPj4+Cg0NVXR0dIn2vW/fPk2fPl379u1TTEyMJGnkyJGaP3++pk+frhdeeEGvvvqqunfvrscff1yS1KBBA61atUrz588v0ecAAAAAAM7lUs02jHRfBE2aNJGPj4/rcdWqVZWcnGx537/88ovy8vLUoEEDhYSEuH6WLVumXbt2SZK2bt2q1q1bu/UZFxdXIrUBAAAAgLcu1WzDSPcFcDgcSk1N9Wg/fvy4wsLCXI/9/PzcpttsNjmdznP2bbfbZYxxazvzfAZv+j5x4oR8fHy0bt06txev9Pe+kAEAAACAi+fvnm0I3Rfgsssu04IFCzza169frwYNGnjdj7+/v/Ly8tzaIiMjlZSUJGOM6+JqGzZsKFZ9LVu2VF5enpKTk9W+ffsC52nUqJF++OEHt7Y1a9YUaz0AAAAAUJi/e7bh8PIL8OCDD+q3337TsGHDtGnTJm3fvl2vvPKKPv74Yz366KNe91O7dm0tX75c+/fv1+HDhyWdvqp5SkqKXnrpJe3atUtTpkzRvHnzilVfgwYNdOedd6p///6aPXu2du/erbVr12rChAn6+uuvJUnDhg3T/PnzNXHiRO3YsUOvv/56qZ/zAAAAAODS8XfPNoTuC1CnTh0tX75c27ZtU+fOndW6dWt99tlnmjVrlrp37+51P+PGjdOePXtUt25dRUZGSjr9Lc0bb7yhKVOmqHnz5lq7dq1GjhxZ7BqnT5+u/v3769FHH9Vll12m+Ph4/fjjj6pZs6Yk6ZprrtG0adP06quvqnnz5lqwYIGefvrpYq8HAAAAAAryd882NnP2wfUAAAAAAKBEMNINAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABYhNANAAAAAIBFCN0AAAAAAFiE0A0AAAAAgEUI3QAAAAAAWITQDQAAAACARQjdAAAAAABY5P8DdaYkBbaWzDgAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "ci_untuned = dml_obj_untuned.confint()\n", + "ci_tuned = dml_obj_tuned.confint()\n", + "\n", + "# Create comparison dataframe\n", + "comparison_data = {\n", + " 'Model': ['Untuned', 'Tuned'],\n", + " 'theta': [dml_obj_untuned.coef[0], dml_obj_tuned.coef[0]],\n", + " 'se': [dml_obj_untuned.se[0], dml_obj_tuned.se[0]],\n", + " 'ci_lower': [ci_untuned.iloc[0, 0], ci_tuned.iloc[0, 0]],\n", + " 'ci_upper': [ci_untuned.iloc[0, 1], ci_tuned.iloc[0, 1]]\n", + "}\n", + "df_comparison = pd.DataFrame(comparison_data)\n", + "\n", + "print(f\"\\nTrue APO at treatment level {treatment_lvl}: {apos[int(treatment_lvl)]:.4f}\\n\")\n", + "print(df_comparison.to_string(index=False))\n", + "\n", + "plt.figure(figsize=(10, 6))\n", + "plt.errorbar(0, df_comparison.loc[0, 'theta'], \n", + " yerr=[[df_comparison.loc[0, 'theta'] - df_comparison.loc[0, 'ci_lower']], \n", + " [df_comparison.loc[0, 'ci_upper'] - df_comparison.loc[0, 'theta']]], \n", + " fmt='o', capsize=5, capthick=2, ecolor=palette[0], color=palette[0], \n", + " label='Untuned', markersize=10, zorder=2)\n", + "plt.errorbar(1, df_comparison.loc[1, 'theta'], \n", + " yerr=[[df_comparison.loc[1, 'theta'] - df_comparison.loc[1, 'ci_lower']], \n", + " [df_comparison.loc[1, 'ci_upper'] - df_comparison.loc[1, 'theta']]], \n", + " fmt='o', capsize=5, capthick=2, ecolor=palette[1], color=palette[1], \n", + " label='Tuned', markersize=10, zorder=2)\n", + "plt.axhline(y=apos[int(treatment_lvl)], color=palette[4], linestyle='--', \n", + " linewidth=2, label='True APO', zorder=1)\n", + "\n", + "plt.title(f'Estimated APO Coefficients with and without Tuning for Treatment Level {treatment_lvl}')\n", + "plt.ylabel('Coefficient Value')\n", + "plt.xticks([0, 1], ['Untuned', 'Tuned'])\n", + "plt.legend()\n", + "plt.grid(True, alpha=0.3)\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "4279c7a0", + "metadata": {}, + "source": [ + "## Detailed Hyperparameter Tuning Guide\n", + "\n", + "In this section, we explore tuning options in more detail and employ a more complicated learning pipeline." + ] + }, + { + "cell_type": "markdown", + "id": "86c7d75f", + "metadata": {}, + "source": [ + "### Define Nuisance Learners\n", + "\n", + "For our example, we will choose [Pipelines](https://scikit-learn.org/stable/modules/compose.html#pipeline) to generate a complex [StackingRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingRegressor.html) or [StackingClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html)." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "76937e73", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
Pipeline(steps=[('scaler', RobustScaler()),\n",
+       "                ('stacking',\n",
+       "                 StackingRegressor(estimators=[('linear_regression',\n",
+       "                                                LinearRegression()),\n",
+       "                                               ('lgbm',\n",
+       "                                                LGBMRegressor(random_state=42,\n",
+       "                                                              verbose=-1))],\n",
+       "                                   final_estimator=Ridge()))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" + ], + "text/plain": [ + "Pipeline(steps=[('scaler', RobustScaler()),\n", + " ('stacking',\n", + " StackingRegressor(estimators=[('linear_regression',\n", + " LinearRegression()),\n", + " ('lgbm',\n", + " LGBMRegressor(random_state=42,\n", + " verbose=-1))],\n", + " final_estimator=Ridge()))])" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "base_regressors = [\n", + " ('linear_regression', LinearRegression()),\n", + " ('lgbm', LGBMRegressor(random_state=42, verbose=-1))\n", + "]\n", + "\n", + "stacking_regressor = StackingRegressor(\n", + " estimators=base_regressors,\n", + " final_estimator=Ridge()\n", + ")\n", + "\n", + "ml_g_pipeline = Pipeline([\n", + " ('scaler', RobustScaler()),\n", + " ('stacking', stacking_regressor)\n", + "])\n", + "ml_g_pipeline" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "395bb00f", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
Pipeline(steps=[('scaler', RobustScaler()),\n",
+       "                ('stacking',\n",
+       "                 StackingRegressor(estimators=[('linear_regression',\n",
+       "                                                LinearRegression()),\n",
+       "                                               ('lgbm',\n",
+       "                                                LGBMRegressor(random_state=42,\n",
+       "                                                              verbose=-1))],\n",
+       "                                   final_estimator=Ridge()))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" + ], + "text/plain": [ + "Pipeline(steps=[('scaler', RobustScaler()),\n", + " ('stacking',\n", + " StackingRegressor(estimators=[('linear_regression',\n", + " LinearRegression()),\n", + " ('lgbm',\n", + " LGBMRegressor(random_state=42,\n", + " verbose=-1))],\n", + " final_estimator=Ridge()))])" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "base_classifiers = [\n", + " ('logistic_regression', LogisticRegression(max_iter=1000, random_state=42)),\n", + " ('lgbm', LGBMClassifier(random_state=42, verbose=-1))\n", + "]\n", + "\n", + "stacking_classifier = StackingClassifier(\n", + " estimators=base_classifiers,\n", + " final_estimator=LogisticRegression(),\n", + ")\n", + "\n", + "ml_m_pipeline = Pipeline([\n", + " ('scaler', RobustScaler()),\n", + " ('stacking', stacking_classifier)\n", + "])\n", + "ml_g_pipeline" + ] + }, + { + "cell_type": "markdown", + "id": "d49756f0", + "metadata": {}, + "source": [ + "### Untuned Model with Pipeline\n", + "\n", + "Now let us take a look at the standard workflow, focusing on a single treatment level and using default hyperparameters." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "cbf47ae5", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
coefstd errtP>|t|2.5 %97.5 %
d213.5128852.33512591.4353060.0208.936124218.089647
\n", + "
" + ], + "text/plain": [ + " coef std err t P>|t| 2.5 % 97.5 %\n", + "d 213.512885 2.335125 91.435306 0.0 208.936124 218.089647" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_obj_untuned_pipeline = DoubleMLAPO(\n", + " dml_data,\n", + " ml_g_pipeline,\n", + " ml_m_pipeline,\n", + " treatment_level=treatment_lvl,\n", + ")\n", + "\n", + "dml_obj_untuned_pipeline.fit()\n", + "dml_obj_untuned_pipeline.summary" + ] + }, + { + "cell_type": "markdown", + "id": "fb8fb3b6", + "metadata": {}, + "source": [ + "### Hyperparameter Tuning with Pipelines\n", + "\n", + "Now, let us take a look at more complex. Again, we will initialize a separate model to compare the results." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "c33bf5f1", + "metadata": {}, + "outputs": [], + "source": [ + "dml_obj_tuned_pipeline = DoubleMLAPO(\n", + " dml_data,\n", + " ml_g_pipeline,\n", + " ml_m_pipeline,\n", + " treatment_level=treatment_lvl,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "ea0febfc", + "metadata": {}, + "source": [ + "As before the tuning input is a parameter space dictionary for the hyperparameters for each learner that should be tuned.\n", + "This dictionary should include a callable for each learner you want to have (only a subset is also possible, i.e. only tuning `ml_g`).\n", + "\n", + "Since we have now a much more complicated learner the tuning inputs have to passed correctly into the pipeline." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "d88d60a0", + "metadata": {}, + "outputs": [], + "source": [ + "# parameter space for the outcome regression tuning\n", + "def ml_g_params_pipeline(trial):\n", + " return {\n", + " 'stacking__lgbm__n_estimators': 100,\n", + " 'stacking__lgbm__learning_rate': trial.suggest_float('stacking__lgbm__learning_rate', 0.001, 0.1, log=True),\n", + " 'stacking__lgbm__max_depth': 5,\n", + " 'stacking__lgbm__min_child_samples': trial.suggest_int('stacking__lgbm__min_child_samples', 20, 50, step=10),\n", + " 'stacking__lgbm__lambda_l1': trial.suggest_float('stacking__lgbm__lambda_l1', 1e-3, 10.0, log=True),\n", + " 'stacking__lgbm__lambda_l2': trial.suggest_float('stacking__lgbm__lambda_l2', 1e-3, 10.0, log=True),\n", + " 'stacking__final_estimator__alpha': trial.suggest_float('stacking__final_estimator__alpha', 0.001, 10.0, log=True),\n", + " }\n", + "\n", + "# parameter space for the propensity score tuning\n", + "def ml_m_params_pipeline(trial):\n", + " return {\n", + " 'stacking__lgbm__n_estimators': 100,\n", + " 'stacking__lgbm__learning_rate': trial.suggest_float('stacking__lgbm__learning_rate', 0.001, 0.1, log=True),\n", + " 'stacking__lgbm__max_depth': 5,\n", + " 'stacking__lgbm__min_child_samples': trial.suggest_int('stacking__lgbm__min_child_samples', 20, 50, step=10),\n", + " 'stacking__lgbm__lambda_l1': trial.suggest_float('stacking__lgbm__lambda_l1', 1e-3, 10.0, log=True),\n", + " 'stacking__lgbm__lambda_l2': trial.suggest_float('stacking__lgbm__lambda_l2', 1e-3, 10.0, log=True),\n", + " 'stacking__final_estimator__C': trial.suggest_float('stacking__final_estimator__C', 0.01, 100.0, log=True),\n", + " 'stacking__final_estimator__max_iter': 1000,\n", + " }\n", + "\n", + "param_space_pipeline = {\n", + " 'ml_g': ml_g_params_pipeline,\n", + " 'ml_m': ml_m_params_pipeline\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "6cf12fb1", + "metadata": {}, + "source": [ + "As before, we can pass the arguments for optuna via `optuna_settings`. For possible option please take a look at the [Optuna Documenation](https://optuna.readthedocs.io/en/stable/index.html). For each learner you can pass local settings which will override the settings.\n", + "\n", + "Here, we will reduce the number of trials for `ml_g` as it did already perform quite well before.\n", + "In principle, we could also use different [samplers](https://optuna.readthedocs.io/en/stable/reference/samplers/index.html), but generally we recommend to use the [TPESampler](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.TPESampler.html), which is used by default." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "742d903f", + "metadata": {}, + "outputs": [], + "source": [ + "optuna_settings_pipeline = {\n", + " 'n_trials': 200,\n", + " 'show_progress_bar': True,\n", + " 'verbosity': optuna.logging.WARNING, # Suppress Optuna logs\n", + " 'ml_g': {\n", + " 'n_trials': 100\n", + " }\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "aca5cc9a", + "metadata": {}, + "source": [ + "As before, we can tune the hyperparameters via the `tune_ml_models()` method. If we would like to inspect the optuna.study results, we can return all tuning results via the `return_tune_res` argument.\n", + "\n", + "We will have a detailed look at the returned results later in the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "8468d9fd", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "a36e9e3c08cb4b0db31f600c3229f236", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/100 [00:00\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
coefstd errtP>|t|2.5 %97.5 %
d213.863432.27677493.9326510.0209.401035218.325825
\n", + "" + ], + "text/plain": [ + " coef std err t P>|t| 2.5 % 97.5 %\n", + "d 213.86343 2.276774 93.932651 0.0 209.401035 218.325825" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_obj_tuned_pipeline.fit()\n", + "dml_obj_tuned_pipeline.summary" + ] + }, + { + "cell_type": "markdown", + "id": "c4037bd4", + "metadata": {}, + "source": [ + "**Remark**: All settings (`optuna_settings` and `ml_param_space`) can also be set on the `params_names` level instead of the `learner_names` level, i.e. `ml_g_d_lvl1` instead of `ml_g`. Generally, more specific settings will override more general settings." + ] + }, + { + "cell_type": "markdown", + "id": "e24e73bb", + "metadata": {}, + "source": [ + "### Comparison" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "deb5359c", + "metadata": { + "tags": [ + "nbsphinx-thumbnail" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "True APO at treatment level 1.0: 213.4904\n", + "\n", + " Model theta se ci_lower ci_upper\n", + " Untuned 211.232659 15.657431 180.544657 241.920661\n", + " Tuned 215.273433 2.485000 210.402924 220.143943\n", + "Untuned Pipeline 213.512885 2.335125 208.936124 218.089647\n", + " Tuned Pipeline 213.863430 2.276774 209.401035 218.325825\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABKUAAAJOCAYAAABm7rQwAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjYsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvq6yFwwAAAAlwSFlzAAAPYQAAD2EBqD+naQAArvBJREFUeJzs3Xd4VFX+x/HPzKR3iAmhE3oVUFwMXaWXFURZUCmi4rIBpOnKigrIioUVVBAbBlSwgLBgAUFBFKUoiBQBMYC4QEgoaYTUub8/+GVkmITMhMmk8H49Tx5yzz333O+de+cM880955oMwzAEAAAAAAAAeJC5tAMAAAAAAADAtYekFAAAAAAAADyOpBQAAAAAAAA8jqQUAAAAAAAAPI6kFAAAAAAAADyOpBQAAAAAAAA8jqQUAAAAAAAAPI6kFAAAAAAAADyOpBQAAAAAAAA8jqQUAMDtunTpoi5dupR2GG519OhRmUwmLVq0qLRDKffeffddNW7cWN7e3goLC7OVv/DCC6pbt64sFotatWolSapTp45GjBjhUvucq4qHc1p+Xf4e/vrrr2UymfT111+XWkxFMZlMmjZtmsvbcZ0CgOtISgHANWTRokUymUyF/mzdutXptn755RdNmzZNR48eLbmAi+HVV18tM18IPv/8c5lMJlWrVk1Wq7XAOnXq1LE7B5GRkerYsaNWrlzpUNcwDL377rvq1KmTwsLCFBAQoBYtWmjGjBk6f/68S7Ht2rVL9957r2rWrClfX19VrlxZXbt2VVxcnPLy8op1vM44cOCARowYoXr16unNN9/UG2+8IUlat26dHn30UbVv315xcXF65plnSiwGd/H0tVZa5wzOy8vLU7Vq1WQymbRmzZrSDqfEOdPHFdeln1ebN292WG8YhmrWrCmTyaS+ffu6dd8AAM/xKu0AAACeN2PGDEVHRzuU169f3+k2fvnlF02fPl1dunRRnTp17NatW7fuakMstldffVXXXXedy3fXlIQlS5aoTp06Onr0qDZs2KCuXbsWWK9Vq1aaNGmSJOnEiRN6/fXXdccdd2jBggX6+9//Lunil927775bH330kTp27Khp06YpICBA3377raZPn65ly5bpyy+/VJUqVYqM66233tLf//53ValSRUOHDlWDBg2Ulpamr776Svfff79Onjypf/3rX+57IS7x9ddfy2q16qWXXrK73jZs2CCz2ayFCxfKx8fHVn7w4EGZza79Da127dq6cOGCvL293RZ3QTx5rZXmOSsLPHVOr9aGDRt08uRJ1alTR0uWLFGvXr1KO6QS5WwfdzX8/Py0dOlSdejQwa5806ZN+t///idfX1+37xMA4DkkpQDgGtSrVy+1adOmxNq/NKlwrTp//rxWrVqlWbNmKS4uTkuWLCn0C1v16tV177332paHDRum+vXra86cObak1PPPP6+PPvpIkydP1gsvvGCrO2rUKA0aNEj9+/fXiBEjirw7Y+vWrfr73/+umJgYff755woODratGz9+vH788Uft3bv3ag79ihITEyXJbthefrm/v7/DtVOcL5wmk0l+fn7FjrGsKe1zVppyc3NltVrl4+NTLs7pe++9pxtuuEHDhw/Xv/71L50/f16BgYFuaTsjI0MBAQFuacsdXOnjrkbv3r21bNkyvfzyy/Ly+vOry9KlS3XjjTfq9OnTbt8nAMBzGL4HACjQBx98oBtvvFHBwcEKCQlRixYt9NJLL0m6OKzirrvukiTdcssttiEW+XOEXD6nVP4cIh999JGmT5+u6tWrKzg4WHfeeadSUlKUlZWl8ePHKzIyUkFBQbrvvvuUlZVlF09cXJxuvfVWRUZGytfXV02bNtWCBQvs6tSpU0f79u3Tpk2bbDFdGkdycrLGjx9vG/5Uv359Pffccw7DTpKTkzVixAiFhoYqLCxMw4cPV3Jyskuv38qVK3XhwgXdddddGjx4sFasWKHMzEynto2KilKTJk105MgRSdKFCxf0wgsvqGHDhpo1a5ZD/X79+mn48OFau3ZtkUMwp0+fLpPJpCVLltglN/K1adPG7s6f8+fPa9KkSbbXrFGjRpo9e7YMw3DY9r333tONN94of39/Va5cWYMHD9Yff/xhW1+nTh099dRTkqSIiAjbvC0mk0lxcXE6f/687bzlD4sraE6p5ORkTZgwQXXq1JGvr69q1KihYcOG2b6cFjavy4EDB3TnnXeqcuXK8vPzU5s2bbR69Wq7OvlDhr777jtNnDhRERERCgwM1IABA5SUlGR3LIVdazk5OZo+fboaNGggPz8/hYeHq0OHDlq/fr1t+5ycHB04cEAnT54s+ERdoqTOmclk0pgxY7Rs2TI1bdpU/v7+iomJ0Z49eyRJr7/+uurXry8/Pz916dLFYahuly5d1Lx5c+3YsUPt2rWTv7+/oqOj9dprr9nVy87O1pNPPqkbb7xRoaGhCgwMVMeOHbVx40a7evnnbfbs2Zo7d67q1asnX19f/fLLLwWe04SEBN13332qUaOGfH19VbVqVd1+++0Ocb766qtq1qyZfH19Va1aNcXGxjq8n/OP5ZdfftEtt9yigIAAVa9eXc8///wVzoy9CxcuaOXKlRo8eLAGDRqkCxcuaNWqVQXWXbNmjTp37mzrX2+66SYtXbq0wNe2U6dOCggIsN0Jl5iYqPvvv19VqlSRn5+fWrZsqcWLFzvs40p9uOTcdXolV9PHuWLIkCE6c+aMXVzZ2dlavny57r777gK3cfY9kJWVpQkTJigiIkLBwcH661//qv/9738Ftnn8+HGNHDlSVapUka+vr5o1a6a33367yPidvU4B4FrFnVIAcA1KSUlx+OuyyWRSeHi4JGn9+vUaMmSIbrvtNj333HOSpP379+u7777Tww8/rE6dOmncuHF6+eWX9a9//UtNmjSRJNu/hZk1a5b8/f312GOP6bffftMrr7wib29vmc1mnTt3TtOmTdPWrVu1aNEiRUdH68knn7Rtu2DBAjVr1kx//etf5eXlpU8++UT/+Mc/ZLVaFRsbK0maO3euxo4dq6CgID3++OOSZBvOlpGRoc6dO+v48eN66KGHVKtWLX3//feaMmWKTp48qblz50q6OE/J7bffrs2bN+vvf/+7mjRpopUrV2r48OEuvcZLlizRLbfcoqioKA0ePFiPPfaYPvnkE1sy70pycnL0xx9/2M7H5s2bde7cOT388MN2dwpcatiwYYqLi9Onn36qm2++ucA6GRkZ+uqrr9SpUyfVqlWryDgMw9Bf//pXbdy4Uffff79atWqlL774Qo888oiOHz+uOXPm2Or++9//1hNPPKFBgwbpgQceUFJSkl555RV16tRJP/30k8LCwjR37ly98847WrlypRYsWKCgoCBdf/31ql+/vt544w1t375db731liSpXbt2BcaUnp6ujh07av/+/Ro5cqRuuOEGnT59WqtXr9b//vc/XXfddQVut2/fPrVv317Vq1fXY489psDAQH300Ufq37+/Pv74Yw0YMMCu/tixY1WpUiU99dRTOnr0qObOnasxY8boww8/lHTla23atGmaNWuWHnjgAf3lL39RamqqfvzxR+3cuVPdunWTdPELbpMmTTR8+PArzktVkudMkr799lutXr3a9h6aNWuW+vbtq0cffVSvvvqq/vGPf+jcuXN6/vnnNXLkSG3YsMFu+3Pnzql3794aNGiQhgwZoo8++kijR4+Wj4+PRo4cKUlKTU3VW2+9pSFDhujBBx9UWlqaFi5cqB49emj79u22Se3zxcXFKTMzU6NGjbLNnVXQfEUDBw7Uvn37NHbsWNWpU0eJiYlav369jh07ZhtSPG3aNE2fPl1du3bV6NGjdfDgQS1YsEA//PCDvvvuO7vhgOfOnVPPnj11xx13aNCgQVq+fLn++c9/qkWLFk4Nw1u9erXS09M1ePBgRUVFqUuXLlqyZIlD4mTRokUaOXKkmjVrpilTpigsLEw//fST1q5da1f3zJkz6tWrlwYPHqx7771XVapU0YULF9SlSxf99ttvGjNmjKKjo7Vs2TKNGDFCycnJevjhhyUV3YfnvzZFXadXcjV9nCvq1KmjmJgYvf/++7bzsGbNGqWkpGjw4MF6+eWX7eq78h544IEH9N577+nuu+9Wu3bttGHDBvXp08chhlOnTunmm2+2JXIjIiK0Zs0a3X///UpNTdX48eMLjd+Z6xQArmkGAOCaERcXZ0gq8MfX19dW7+GHHzZCQkKM3NzcQttatmyZIcnYuHGjw7rOnTsbnTt3ti1v3LjRkGQ0b97cyM7OtpUPGTLEMJlMRq9evey2j4mJMWrXrm1XlpGR4bCfHj16GHXr1rUra9asmd2+8z399NNGYGCg8euvv9qVP/bYY4bFYjGOHTtmGIZh/Pe//zUkGc8//7ytTm5urtGxY0dDkhEXF+fQ9uVOnTpleHl5GW+++aatrF27dsbtt9/uULd27dpG9+7djaSkJCMpKcn4+eefjcGDBxuSjLFjxxqGYRhz5841JBkrV64sdJ9nz541JBl33HFHoXV+/vlnQ5Lx8MMPF3kMhvHnazFz5ky78jvvvNMwmUzGb7/9ZhiGYRw9etSwWCzGv//9b7t6e/bsMby8vOzKn3rqKUOSkZSUZFd3+PDhRmBgoEMMtWvXNoYPH25bfvLJJw1JxooVKxzqWq1WwzAM48iRIw7n6rbbbjNatGhhZGZm2tVv166d0aBBA1tZ/nuka9eutvYMwzAmTJhgWCwWIzk52VZW2LXWsmVLo0+fPg7ll8qP8dJjK0hJnTPDMGzv+yNHjtjKXn/9dUOSERUVZaSmptrKp0yZYkiyq9u5c2dDkvGf//zHVpaVlWW0atXKiIyMtL3Xc3NzjaysLLt4zp07Z1SpUsUYOXKkrSz/NQkJCTESExPt6l9+Ts+dO2dIMl544YVCX4vExETDx8fH6N69u5GXl2crnzdvniHJePvttx2O5Z133rE7lqioKGPgwIGF7uNSffv2Ndq3b29bfuONNwwvLy+7Y0lOTjaCg4ONtm3bGhcuXLDb/tLrLT+e1157za5Ofl/w3nvv2cqys7ONmJgYIygoyHbOnOnDnblOC+NqH3fpdZ7/eVDQZ8el8t+LP/zwgzFv3jwjODjY9jlw1113Gbfccout/UuPw9n3wK5duwxJxj/+8Q+7enfffbchyXjqqadsZffff79RtWpV4/Tp03Z1Bw8ebISGhtriKs51CgDXOobvAcA1aP78+Vq/fr3dz6VzEYWFhen8+fNOD+Nw1rBhw+zuTGjbtq0Mw7DdUXFp+R9//KHc3Fxbmb+/v+33/Du9OnfurMOHDyslJaXIfS9btkwdO3ZUpUqVdPr0adtP165dlZeXp2+++UbSxadJeXl5afTo0bZtLRaLxo4d6/RxfvDBBzKbzRo4cKCtbMiQIVqzZo3OnTvnUH/dunWKiIhQRESEWrZsqWXLlmno0KG2OxzS0tIkqcChW/ny16WmphZaJ3/dldq51Oeffy6LxaJx48bZlU+aNEmGYdiumRUrVshqtWrQoEF2r21UVJQaNGjgMEzranz88cdq2bKlw51N0sW7/Qpy9uxZbdiwQYMGDVJaWpotvjNnzqhHjx46dOiQjh8/brfNqFGj7Nrr2LGj8vLy9PvvvxcZY1hYmPbt26dDhw4VWqdOnToyDKPIp/eV1DnLd9ttt9ndrdG2bVtJF+/uuHSf+eWHDx+2297Ly0sPPfSQbdnHx0cPPfSQEhMTtWPHDkkX3z/5c4VZrVadPXtWubm5atOmjXbu3OlwDAMHDlRERMQVjzN//rGvv/66wPeUJH355ZfKzs7W+PHj7SbLf/DBBxUSEqLPPvvMrn5QUJDd3G4+Pj76y1/+4nDMBTlz5oy++OILDRkyxO448oct51u/fr3S0tL02GOPOcyRdfn16+vrq/vuu8+u7PPPP1dUVJTdfry9vTVu3Dilp6dr06ZNkpzrw525Tgvjah93tfKHQ3766adKS0vTp59+WujQPWffA59//rkkOdS7/K4nwzD08ccfq1+/fjIMw66P69Gjh1JSUgq8jiXnrlMAuNYxfA8ArkF/+ctfrjjR+T/+8Q999NFH6tWrl6pXr67u3btr0KBB6tmz51Xt9/LhR6GhoZKkmjVrOpRbrValpKTYhrB99913euqpp7RlyxZlZGTY1U9JSbG1VZhDhw5p9+7dhX7ZzZ+A+/fff1fVqlUVFBRkt75Ro0ZFHN2f3nvvPf3lL3/RmTNndObMGUlS69atlZ2drWXLlmnUqFF29du2bauZM2fKZDIpICBATZo0sZsIPD85kJ+cKogziauQkJAi27nU77//rmrVqjm0mT9MMz9Bc+jQIRmGoQYNGhTYjjufmBYfH2/3RdgZv/32mwzD0BNPPKEnnniiwDqJiYmqXr26bfnya7VSpUqS5NQXyxkzZuj2229Xw4YN1bx5c/Xs2VNDhw7V9ddf71LcUsmds3yuvCclx+OvVq2aw0TeDRs2lHRxjqj8oaSLFy/Wf/7zHx04cEA5OTm2ugU9BbSgssv5+vrqueee06RJk1SlShXdfPPN6tu3r4YNG6aoqCi7Y738vevj46O6des6vBY1atRwSAxVqlRJu3fvLjKeDz/8UDk5OWrdurV+++03W3nbtm21ZMkS2/DI+Ph4SVLz5s2LbLN69eoOE////vvvatCggcMTKS8/v8704Vdznbrax12tiIgIde3aVUuXLlVGRoby8vJ05513FljX2ffA77//LrPZrHr16tnVu/x6SUpKUnJyst544w298cYbBe4z//Pjcs5cpwBwrSMpBQBwEBkZqV27dumLL77QmjVrtGbNGsXFxWnYsGEFTqjrLIvF4lK58f+T0sbHx+u2225T48aN9eKLL6pmzZry8fHR559/rjlz5hQ438zlrFarunXrpkcffbTA9flfpK/WoUOH9MMPP0hSgUmaJUuWOHxhu+6666741Kr8L1O7d+9W//79C6yT/8W5adOmhbZTv359eXl52Saydher1SqTyaQ1a9YUeC4vT/B5Wv71MXnyZPXo0aPAOvXr17dbLuqavJJOnTopPj5eq1at0rp16/TWW29pzpw5eu211/TAAw+4FHtJnbN8xX1PuuK9997TiBEj1L9/fz3yyCOKjIyUxWLRrFmzbEmaS116V+SVjB8/Xv369dN///tfffHFF3riiSc0a9YsbdiwQa1bt3Y5zqs55iVLlkiS2rdvX+D6w4cPq27dui7F4+zrUBBn+vDiXqfF6ePc4e6779aDDz6ohIQE9erVy+EJniUlv/+49957C51b8EqJPHdfpwBQ0ZCUAgAUyMfHR/369VO/fv1ktVr1j3/8Q6+//rqeeOIJ1a9fv9ChUiXhk08+UVZWllavXm13Z0dBw8IKi6tevXpKT08v8pHltWvX1ldffaX09HS7ZMrBgwedinXJkiXy9vbWu+++6/Ald/PmzXr55Zd17NgxpyatztehQweFhYVp6dKlevzxxwv88vzOO+9Ikvr27VtoOwEBAbr11lu1YcMG/fHHHw53w1yudu3a+vLLL5WWlmZ318GBAwds66WLr61hGIqOjnZbcq8w9erV0969e13aJj8Z4O3t7dZH1l/pPVC5cmXdd999uu+++5Senq5OnTpp2rRpLielSuqcucuJEyd0/vx5u7ulfv31V0myDQtcvny56tatqxUrVti9ZvlPYrwa9erV06RJkzRp0iQdOnRIrVq10n/+8x+99957tmM9ePCgXUIoOztbR44ccdu1cOTIEX3//fcaM2aMOnfubLfOarVq6NChWrp0qaZOnWq7K2fv3r0OiVBn1K5dW7t375bVarW7W6qg81tUHy4V7zotiT7OGQMGDNBDDz2krVu32h44UBBn3wO1a9eW1WpVfHy83d1Rl/f1+U/my8vLK/Y1c6XrFACudcwpBQBwkD8cI5/ZbLb9JTgrK0uSbF9CL3+0eknI/+Jz6R0LKSkpiouLc6gbGBhYYEyDBg3Sli1b9MUXXzisS05Ots1f1bt3b+Xm5mrBggW29Xl5eXrllVecinXJkiXq2LGj/va3v+nOO++0+3nkkUckSe+//75TbeULCAjQ5MmTdfDgQduT3i712WefadGiRerRo0ehT97L99RTT8kwDA0dOlTp6ekO63fs2GG7k6J3797Ky8vTvHnz7OrMmTNHJpPJ9iSsO+64QxaLRdOnT3e4q8QwDIfr6WoMHDhQP//8s1auXOmwrrA7WiIjI9WlSxe9/vrrOnnypMP6pKSkYsVS2LV2+fEGBQWpfv36tveOdPEJiwcOHCgwnsuVxDlzl9zcXL3++uu25ezsbL3++uuKiIjQjTfeKKng9++2bdu0ZcuWYu83IyNDmZmZdmX16tVTcHCw7XXu2rWrfHx89PLLL9vte+HChUpJSSnwKWvFkX+X1KOPPurwnh80aJA6d+5sq9O9e3cFBwdr1qxZDvE7c0dW7969lZCQYJeUyc3N1SuvvKKgoCBbUsyZPtyZ67Sw43V3H+eMoKAgLViwQNOmTVO/fv0KrefseyD/38uf3pf/JNZ8FotFAwcO1Mcff1xgQvxK/Ycz1ykAXOu4UwoArkFr1qyx/dX4Uu3atVPdunX1wAMP6OzZs7r11ltVo0YN/f7773rllVfUqlUr21CyVq1ayWKx6LnnnlNKSop8fX116623KjIy0u3xdu/e3fZX/4ceekjp6el68803FRkZ6fCl/sYbb9SCBQs0c+ZM1a9fX5GRkbr11lv1yCOPaPXq1erbt69GjBihG2+8UefPn9eePXu0fPlyHT16VNddd5369eun9u3b67HHHtPRo0fVtGlTrVixwqnJ1Ldt22Z7VHtBqlevrhtuuEFLlizRP//5T5deg8cee0w//fSTnnvuOW3ZskUDBw6Uv7+/Nm/erPfee09NmjRxamhlu3btNH/+fP3jH/9Q48aNNXToUDVo0EBpaWn6+uuvtXr1as2cOVOS1K9fP91yyy16/PHHdfToUbVs2VLr1q3TqlWrNH78eNtdH/Xq1dPMmTM1ZcoUHT16VP3791dwcLCOHDmilStXatSoUZo8ebJLx1uYRx55RMuXL9ddd92lkSNH6sYbb9TZs2e1evVqvfbaa2rZsmWB282fP18dOnRQixYt9OCDD6pu3bo6deqUtmzZov/973/6+eefXY6lsGutadOm6tKli2688UZVrlxZP/74o5YvX253XRw/flxNmjTR8OHDi5zsvCTOmbtUq1ZNzz33nI4ePaqGDRvqww8/1K5du/TGG2/Y5hLr27evVqxYoQEDBqhPnz46cuSIXnvtNTVt2rTAJJszfv31V912220aNGiQmjZtKi8vL61cuVKnTp3S4MGDJV28w2XKlCmaPn26evbsqb/+9a86ePCgXn31Vd100012k5pfjSVLlqhVq1aF3sX217/+VWPHjtXOnTt1ww03aM6cOXrggQd000036e6771alSpX0888/KyMjo8j38KhRo/T6669rxIgR2rFjh+rUqaPly5fru+++09y5c213BjnThztznV6uJPs4ZxQ2fO5Szr4HWrVqpSFDhujVV19VSkqK2rVrp6+++spuTrB8zz77rDZu3Ki2bdvqwQcfVNOmTXX27Fnt3LlTX375pc6ePVtgLM5cpwBwzfPgk/4AAKUs/xHbhf3kP8Z6+fLlRvfu3Y3IyEjDx8fHqFWrlvHQQw8ZJ0+etGvvzTffNOrWrWtYLBa7R3x37tzZ6Ny5s61e/iPAly1bVmA8P/zwg135U089ZUgykpKSbGWrV682rr/+esPPz8+oU6eO8dxzzxlvv/22w2PqExISjD59+hjBwcGGJLs40tLSjClTphj169c3fHx8jOuuu85o166dMXv2bNvj6w3DMM6cOWMMHTrUCAkJMUJDQ42hQ4caP/30k91rVJCxY8cakoz4+PhC60ybNs2QZPz888+GYTg+zvxK8vLyjLi4OKN9+/ZGSEiI4efnZzRr1syYPn26kZ6e7lQb+Xbs2GHcfffdRrVq1Qxvb2+jUqVKxm233WYsXrzYyMvLs9VLS0szJkyYYKvXoEED44UXXrB7fH2+jz/+2OjQoYMRGBhoBAYGGo0bNzZiY2ONgwcP2uoUdG4NwzCGDx9uBAYGOrR5+ePkDePi+RkzZoxRvXp1w8fHx6hRo4YxfPhw2+PaL38se774+Hhj2LBhRlRUlOHt7W1Ur17d6Nu3r7F8+XJbncKuyYIeY1/YtTZz5kzjL3/5ixEWFmb4+/sbjRs3Nv7973/bXWP5MV5+bFfi7nMmyYiNjbUry4/r8kfYF/Qe7ty5s9GsWTPjxx9/NGJiYgw/Pz+jdu3axrx58+y2tVqtxjPPPGPUrl3b8PX1NVq3bm18+umnxvDhw43atWsXue9L1+Wf09OnTxuxsbFG48aNjcDAQCM0NNRo27at8dFHHzlsO2/ePKNx48aGt7e3UaVKFWP06NHGuXPn7OrkH8vlLo/xcjt27DAkGU888UShdY4ePWpIMiZMmGArW716tdGuXTvD39/fCAkJMf7yl78Y77//fpHxGIZhnDp1yrjvvvuM6667zvDx8TFatGjhcK0704c7c51errh93KXXeUHvpYIU9l68XEF9qLPvgQsXLhjjxo0zwsPDjcDAQKNfv37GH3/8YUgynnrqKbu6p06dMmJjY42aNWsa3t7eRlRUlHHbbbcZb7zxhq3O1VynAHCtMhlGMWasBAAAwDWtS5cuOn36tMtzfAEAAORjTikAAAAAAAB4HEkpAAAAAAAAeBxJKQAAAAAAAHgcc0oBAAAAAADA47hTCgAAAAAAAB5HUgoAAAAAAAAe51XaAZQFVqtVJ06cUHBwsEwmU2mHAwAAAAAAUG4ZhqG0tDRVq1ZNZnPh90ORlJJ04sQJ1axZs7TDAAAAAAAAqDD++OMP1ahRo9D1JKUkBQcHS7r4YoWEhJRyNMVntVqVlJSkiIiIK2YiAQAFox8FgKtDPwoAxVeR+tDU1FTVrFnTlm8pDEkpyTZkLyQkpNwnpTIzMxUSElLuL2AAKA30owBwdehHAaD4KmIfWtQUSRXjKAEAAAAAAFCukJQCAAAAAACAx5GUAgAAAAAAgMcxpxQAAAAAAGVcXl6ecnJySjsMlCCr1aqcnBxlZmaW+TmlvL29ZbFYrrodklIAAAAAAJRRhmEoISFBycnJpR0KSphhGLJarUpLSytygvCyICwsTFFRUVcVK0kpAAAAAADKqPyEVGRkpAICAspFsgLFYxiGcnNz5eXlVabPs2EYysjIUGJioiSpatWqxW6LpBQAAAAAAGVQXl6eLSEVHh5e2uGghJWXpJQk+fv7S5ISExMVGRlZ7KF8ZXuQIgAAAAAA16j8OaQCAgJKORLAUf51eTVznZGUAgAAAACgDCvrd83g2uSO65KkFAAAAAAAADyOpBQAAAAAAAA8jqQUAAAAAABwG5PJdMWfadOmeTym999/XxaLRbGxsQ7rvv76a7v4qlSpooEDB+rw4cN29b7//nv17t1blSpVkp+fn1q0aKEXX3xReXl5njqMCoekFAAAAAAAcJuTJ0/afubOnauQkBC7ssmTJ9vq5j9xrqQtXLhQjz76qN5//31lZmYWWOfgwYM6ceKEli1bpn379qlfv362hNPKlSvVuXNn1ahRQxs3btSBAwf08MMPa+bMmRo8eLAMwyjxY6iISEoBAAAAAAC3iYqKsv2EhobKZDLZlg8cOKDg4GCtWbNGN954o3x9fbV582aNGDFC/fv3t2tn/Pjx6tKli23ZarVq1qxZio6Olr+/v1q2bKnly5cXGc+RI0f0/fff67HHHlPDhg21YsWKAutFRkaqatWq6tSpk5588kn98ssv+u2333T+/Hk9+OCD+utf/6o33nhDrVq1Up06dfTAAw9o8eLFWr58uT766KOrecmuWSSlAAAAAACARz322GN69tlntX//fl1//fVObTNr1iy98847eu2117Rv3z5NmDBB9957rzZt2nTF7eLi4tSnTx+Fhobq3nvv1cKFC4vcl7+/vyQpOztb69at05kzZ+zu8MrXr18/NWzYUO+//75TxwB7XqUdAAAAAAAAcM2pb4/p1OZjRdYLqB6s+sNa2pX99s7PyjieVuS2VTrUUpWOtYod45XMmDFD3bp1c7p+VlaWnnnmGX355ZeKiYmRJNWtW1ebN2/W66+/rs6dOxe4ndVq1aJFi/TKK69IkgYPHqxJkybpyJEjio6OLnCbkydPavbs2apevboaNWqkzz//XJLUpEmTAus3btxYv/76q9PHgj+RlAIAAAAAoJzJy8pVTmpWkfVyQ30dy9Kzndo2L6vk5npq06aNS/V/++03ZWRkOCSysrOz1bp160K3W79+vc6fP6/evXtLkq677jp169ZNb7/9tp5++mm7ujVq1JBhGMrIyFDLli318ccfy8fHx7aeeaPcj6QUAAAAAADljMXXS94hjgmny3kF+RRY5sy2Ft+SSxkEBgbaLZvNZoekT05Oju339PR0SdJnn32m6tWr29Xz9S38WBYuXKizZ8/ahuNJF++e2r17t6ZPny6z+c9Zjb799luFhIQoMjJSwcHBtvKGDRtKkvbv36927do57GP//v1q2rRpoTGgcCSlAAAAAAAoZ6p0LP7QusuH85UFERER2rt3r13Zrl275O3tLUlq2rSpfH19dezYsUKH6l3uzJkzWrVqlT744AM1a9bMVp6Xl6cOHTpo3bp16tmzp608OjpaYWFhDu10795dlStX1n/+8x+HpNTq1at16NAhh7uu4BySUuXQi5viNeebwwWus+blyWyxFLhuQqe6mti5XkmGBgAAAACAy2699Va98MILeueddxQTE6P33ntPe/futQ3NCw4O1uTJkzVhwgRZrVZ16NBBKSkp+u677xQSEqLhw4c7tPnuu+8qPDxcgwYNkslkslvXu3dvLVy40C4pVZjAwEC9/vrrGjx4sEaNGqUxY8YoJCREX331lR555BHdeeedGjRokHteiGsMSalyKDUzV8dTMq9QI6fA0tTMkhsPDAAAAABAcfXo0UNPPPGEHn30UWVmZmrkyJEaNmyY9uzZY6vz9NNPKyIiQrNmzdLhw4cVFhamG264Qf/6178KbPPtt9/WgAEDHBJSkjRw4EANHTpUp0+fdiq+O++8Uxs3btS///1vdezYUZmZmWrQoIEef/xxjR8/vsB9oGgmg5m6lJqaqtDQUKWkpCgkJKS0wylSQXdKGYahE/8/UV21EN8C3xDcKQUAV2a1WpWYmKjIyEi7+QUAAM6hHwXcKzMz0/aUOD8/v9IOByXMMAzl5ubKy8urXCS5rnR9Optn4U6pcmhi53oOyaXzWbkKfnyNJGn/o10U7Oc4mR0AAAAAAEBZwZ8vAAAAAAAA4HEkpQAAAAAAAOBxJKUAAAAAAADgcSSlAAAAAAAA4HEkpQAAAAAAAOBxJKUAAAAAAADgcV6lufNZs2ZpxYoVOnDggPz9/dWuXTs999xzatSokUNdwzDUu3dvrV27VitXrlT//v1t644dO6bRo0dr48aNCgoK0vDhwzVr1ix5eZXq4QEAAAAA4HEvborXnG8O25UZhiGrIRkyZJJJZpNkMpns6kzoVFcTO9fzZKi4xpVq1mbTpk2KjY3VTTfdpNzcXP3rX/9S9+7d9csvvygwMNCu7ty5cx3eMJKUl5enPn36KCoqSt9//71OnjypYcOGydvbW88884ynDgUAAAAAgDIhNTNXx1Myi7Ud4EmlOnxv7dq1GjFihJo1a6aWLVtq0aJFOnbsmHbs2GFXb9euXfrPf/6jt99+26GNdevW6ZdfftF7772nVq1aqVevXnr66ac1f/58ZWdne+pQAAAAAAAoE0L8vFQ91E/hAd5yvLXDnklSeIC3qof6KcSP0UZFWbRokcLCwko7jAqjTM0plZKSIkmqXLmyrSwjI0N333235s+fr6ioKIdttmzZohYtWqhKlSq2sh49eig1NVX79u0r+aABAAAAAChDJnaup7cGtVTyhVwVMODIjskkJV/I1VuDWrp16F6XLl00fvx4h3JXkzp16tTR3Llz3RYXypYykwa1Wq0aP3682rdvr+bNm9vKJ0yYoHbt2un2228vcLuEhAS7hJQk23JCQkKB22RlZSkrK8u2nJqaaovBarVe1XGUFqvxZ9zl+TgAoDRZrdaL8y3QhwJAsdCPAu6V/57K/3FW8oUc3bn4Rxm6OI/UFfdhSGaToTsX/6hjU7sqzN/7KqP+U0Fx5y+7cjyuHn9JKk78Zal9d8o/LwXlIJz9HCgzSanY2Fjt3btXmzdvtpWtXr1aGzZs0E8//eTWfc2aNUvTp093KE9KSlJmpuvjbsuCjOw82+9JSUm64Ou+jgQArhVWq1UpKSkyDENmc5m6mRgAygX6UcC9cnJyZLValZubq9xc5+d7itv+uzKy8+RsWsNqXPxOuWj77xrTvk6xYr1cfsLi8rjzkxW5ubm6//77lZycrPbt22vu3LnKzs7WoEGD9J///Efe3t7q2rWrfv/9d02cOFETJ06UJGVnZ2vGjBlavXq1fvzxR1u7L7/8sl555RUdOnRIkopsW7p4w8qTTz6pDz/8UMnJyWrWrJmeeeYZde7c2dbuO++8o+nTp+v06dPq1q2b2rdvb4vf3QzDUF7exe/2Bc2pXdbk5ubKarXqzJkzttc0X1pamlNtlImk1JgxY/Tpp5/qm2++UY0aNWzlGzZsUHx8vMOtfQMHDlTHjh319ddfKyoqStu3b7dbf+rUKUkqcLifJE2ZMsV2QUsX75SqWbOmIiIiFBIS4qaj8qzz2X++ISIiIhTs51OK0QBA+WS1WmUymRQREcGXKQAoBvpRwL0yMzOVlpYmLy8vp58ubxiGFmw5Vqz9vbrlmB7uVM8tCRGTySSTyeQQd37f4OXlJbPZrE2bNqlatWrasGGDfvvtNw0ePFitW7fWgw8+qBUrVqhVq1Z68MEH9eCDD9ptd3nbl7abv3yltiVp9OjR2r9/v95//31Vq1ZNK1euVN++fbV79241aNBA27Zt06hRo/TMM8+of//+Wrt2raZNm2a3n5JweYKnrMo/F+Hh4fLz87Nbd/lyoW2URGDOMgxDY8eO1cqVK/X1118rOjrabv1jjz2mBx54wK6sRYsWmjNnjvr16ydJiomJ0b///W8lJiYqMjJSkrR+/XqFhISoadOmBe7X19dXvr6+DuVms7ncfniaTX/GXZ6PAwBKm8lkoh8FgKtAPwq4T37yJf/HGWcyshV/JsPlfRmS4s9k6NyFXIUHuucmh4Lizl/O/7dSpUqaP3++LBaLmjRpoj59+mjDhg0aNWqUwsPDZbFYFBISoqpVqxbaRmFlV2r72LFjtoetVatWTZL0yCOP6IsvvtCiRYv0zDPP6OWXX1bPnj31z3/+U5LUqFEjbdmyRWvXri2RO5kMwyjwOMqq/PNbUJ/v7GdAqSalYmNjtXTpUq1atUrBwcG2OaBCQ0Pl7++vqKioAu92qlWrli2B1b17dzVt2lRDhw7V888/r4SEBE2dOlWxsbEFJp4AAAAAAKio0rPyiq50BWlZ7ktKOaNZs2ayWCy25apVq2rPnj0l3vaePXuUl5enhg0b2m2TlZWl8PBwSdL+/fs1YMAAu/UxMTFau3atW+JDKSelFixYIOnirPyXiouL04gRI5xqw2Kx6NNPP9Xo0aMVExOjwMBADR8+XDNmzHBztAAAAAAAlG1BvpaiK11BsK970gQhISFKSUlxKE9OTlZoaKht+fKhaiaTqchJss1ms8NE4Dk5OQ71rtR2enq6LBaLduzYYZe4kqSgoKAr7h/uU+rD99yxTe3atfX555+7IyQAAAAAAMqt8AAf1QsP0OEzGU5PdC5JJkl1wwNUOcA98xk1atRI69atcyjfuXOnw91JV+Lj42Ob/DtfRESEEhIS7Ia77dq1y6X4Wrdurby8PCUmJqpjx44F1mnSpIm2bdtmV7Z161aX9oMrY6A3AAAAAAAVhMlk0pgO0UVXLMDYDtFum8to9OjR+vXXXzVu3Djt3r1bBw8e1Isvvqj3339fkyZNcrqdOnXq6JtvvtHx48d1+vRpSRdHWyUlJen5559XfHy85s+frzVr1rgUX8OGDXXPPfdo2LBhWrFihY4cOaLt27dr1qxZ+uyzzyRJ48aN09q1azV79mwdOnRI8+bNY+iem5GUAgAAAACgAhnepqYCfCwyO5lfMpukAB+LhrWp6bYY6tatq2+++UYHDhxQ165d1bZtW3300UdatmyZevbs6XQ7M2bM0NGjR1WvXj1FRERIungH06uvvqr58+erZcuW2r59uyZPnuxyjHFxcRo2bJgmTZqkRo0aqX///vrhhx9Uq1YtSdLNN9+sN998Uy+99JJatmypdevWaerUqS7vB4UzGcUZQ1fBpKamKjQ0VCkpKQoJCSntcIrlfFaugh+/mBlOmdlDwX6em5gOACoKq9Vqe5orT40CANfRjwLulZmZqSNHjig6Olp+fn4ubfvFwUT1fWu7DBmyXuFbv9kkmWTSZw/8Rd0bRV5lxLgahmEoNzdXXl5e5eLpe1e6Pp3Ns/BJAQAAAABABfLipng98NHPCvP3UlG3oRiGFObvpfs/+lkvbor3TIDA/yvVic4BAAAAAIB7pWbm6nhKplN1DUlnMnIk5Sg1M7dE4wIuR1IKAAAAAIAKJMTPS9VD7YdTGcbFYXyGDJlkujhs77IhYiF+pAjgWVxxAAAAAABUIBM719PEzvVKOwygSMwpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAj2NOKQAAAAAAKpCUnXOVsvMluzLDMCRZJcOQTCZJZoeJzkNveFihN4z3WJwASSkAAAAAACoQa1aq8tKPF2s7wJMYvgcAAAAAQAVi9g2RJai6zH7hkkxF1DbJ7Bd+sb5viCfCKzWLFi1SWFhYaYeBS5CUAgAAAACgAgm9Ybyu6/q6rFnJciYpZc1K1nVdX3fb0D2TyXTFn2nTprllPyj/GL4HAAAAAEAFkpeZrMTP/nZx/ihZi6htlQyzEj/7m2ref0QWv7Cr3v/Jkydtv3/44Yd68skndfDgQVtZUFDQVe8DFQN3SgEAAAAAUIGk739XRk6Gik5I5bPKyMlQ+v733LL/qKgo209oaKhMJpNt+bXXXlOHDh3s6s+dO1d16tSxLY8YMUL9+/fX7NmzVbVqVYWHhys2NlY5OTm2OllZWZo8ebKqV6+uwMBAtW3bVl9//bVdu4sWLVKtWrUUEBCgAQMG6MyZM245PrgPSSkAAAAAACoIwzCUumt+sbZN3TXv/5/SV/o2btyo+Ph4bdy4UYsXL9aiRYu0aNEi2/oxY8Zoy5Yt+uCDD7R7927ddddd6tmzpw4dOiRJ2rZtm+6//36NGTNGu3bt0i233KKZM2eW0tGgMAzfAwAAAACggrBmnlFuyuFibGkoN+WwrJlnZfEPd3tcrqpUqZLmzZsni8Wixo0bq0+fPvrqq6/04IMP6tixY4qLi9OxY8dUrVo1SdLkyZO1du1axcXF6ZlnntFLL72knj176tFHH5UkNWzYUN9//73Wrl1bmoeFy3CnFAAAAAAAFYQ1O/0qt09zUyRXp1mzZrJYLLblqlWrKjExUZK0Z88e5eXlqWHDhgoKCrL9bNq0SfHx8ZKk/fv3q23btnZtxsTEeO4A4BTulAIAAAAAoIIw+1zdJOJmn2A3RVJI+2azwxDBS+eKyuft7W23bDKZZLVenCMrPT1dFotFO3bssEtcSUyiXt6QlAIAAAAAoIIw+4XLK7SuclOOSHJlfiiTvEKjZfarXFKhSZIiIiKUkJAgwzBkMpkkSbt27XKpjdatWysvL0+JiYnq2LFjgXWaNGmibdu22ZVt3bq1WDGj5DB8DwAAAACACsJkMimkVWyxtg1pNcaWKCopXbp0UVJSkp5//nnFx8dr/vz5WrNmjUttNGzYUPfcc4+GDRumFStW6MiRI9q+fbtmzZqlzz77TJI0btw4rV27VrNnz9ahQ4c0b9485pMqg0hKAQAAAABQgQQ1GSqTd4Cc/8pvlsk7QEFN7i3JsCRdvIPp1Vdf1fz589WyZUtt375dkydPdrmduLg4DRs2TJMmTVKjRo3Uv39//fDDD6pVq5Yk6eabb9abb76pl156SS1bttS6des0depUdx8OrpLJKCvPeyxFqampCg0NVUpKikJCQko7nGI5n5Wr4McvZpdTZvZQsJ9PKUcEAOWP1WpVYmKiIiMjZTbzdxsAcBX9KOBemZmZOnLkiKKjo+Xn5+fSthlH1+nUqtslw5BkvUJNs2QyqUr/1Qqo3e2q4sXVMQxDubm58vLyKvE71tzhStens3kWPikAAAAAAKhAUnbO1ekvH5LZN0xFzytlyOwbptPrRyll59ySDw64BBOdAwAAAABQgVizUpWXftzJ2oasmWds2wGeRFIKAAAAAIAKxOwbIktQdbsyI38Yn2FIJpMks8MQMbNv+ZzOBuUXSSkAAAAAACqQ0BvGK/SG8aUdBlAk5pQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHMacUAAAAAAAVyJy9mzRn3zd2ZYZhyCrDNs+5WSaHic4nNOukCc07ezJUXONISgEAAAAAUIGk5mTqeEZKsbYDPInhewAAAAAAVCAh3n6qHhCqcN8AmYqoa5IU7hug6gGhCvH280R45d6iRYsUFhZ2VW2YTCb997//dUs8+bp06aLx48fbluvUqaO5c+e6dR/uRlIKAAAAAIAKZELzznqzwyAlZ1+QqYi0lEkmJWdf0JsdBrl16N7lCZJ8xUnolIfkyuVMJpPtJzQ0VO3bt9eGDRts60+ePKlevXqVaAw//PCDRo0aVaL7uFokpQAAAAAAqECSsy7org2LZRiSVcYV6+bPM3XXhsVKzrrgoQivDXFxcTp58qS+++47XXfdderbt68OHz4sSYqKipKvr2+J7j8iIkIBAQEluo+rRVIKAAAAAIAK5J3fflRGbnaRCal8VhnKyM3Wu/E/lnBkjkaMGKH+/ftr9uzZqlq1qsLDwxUbG6ucnBxJF++4+v333zVhwgTbnUeSNG3aNLVq1cqurblz56pOnTpOty1JWVlZmjx5sqpXr67AwEC1bdtWX3/9tV27ixYtUq1atRQQEKABAwbozJkzTh1bWFiYoqKi1Lx5cy1YsEAXLlzQ+vXrJdkP3zt69KhMJpM++OADderUSf7+/mrevLk2bdpk197evXvVq1cvBQUFqUqVKho6dKhOnz5d6P4vv8PMZDLprbfe0oABAxQQEKAGDRpo9erVV7WPq0VSCgAAAACACsIwDM3bv7lY277yy2YZhnOJLHfauHGj4uPjtXHjRi1evFiLFi3SokWLJEkrVqxQjRo1NGPGDJ08eVInT550W9uSNGbMGG3ZskUffPCBdu/erbvuuks9e/bUoUOHJEnbtm3T/fffrzFjxmjXrl265ZZbNHPmTJeP0d/fX5KUnZ1daJ1HH31U48eP186dOxUTE6N+/frZEmDJycm69dZb1bp1a/34449au3atTp06pUGDBrkUx/Tp0zVo0CDt3r1bvXv31j333KOzZ8+6dR+uICkFAAAAAEAFcSYrQ/FpZ5y8R+pPhqT4tDM6m5VREmFdUaVKlTRv3jw1btxYffv2VZ8+ffTVV19JkipXriyLxaLg4GBFRUUpKirKbW0fO3ZMcXFxWrZsmTp27Kh69epp8uTJ6tChg+Li4iRJL730knr27KlHH31UDRs21Lhx49SjRw+XYsjIyNDUqVNlsVjUuXPh83bFxsbqjjvuUJMmTbRgwQKFhoZq4cKFkqR58+apdevWeuaZZ9S4cWO1bt1ab7/9tjZu3Khff/3V6VhGjBihIUOGqH79+nrmmWeUnp6u7du3u3UfrvAqkVYBAAAAAIDHpedkXdX2aTlZCvcLdFM0zmnWrJksFottuWrVqtqzZ0+Jt71nzx7l5eWpYcOGdttkZWUpPDxckrR//34NGDDAbn1MTIzWrl1b5L6HDBkii8WiCxcuKCIiQgsXLtT1119faP2YmBjb715eXmrTpo32798vSfr555+1ceNGBQUFOWwXHx/vcAyFuXT/gYGBCgkJUWJiolv34QqSUgAAAAAAVBBB3lc3eXbwVW6fLyQkRCkpKQ7lycnJCg0NtSvz9va2WzaZTLJarVds32w2Oww1vHSuKGfaTk9Pl8Vi0Y4dO+wSV5IKTMy4as6cOeratatCQ0MVERFxVW2lp6erX79+eu655xzWVa1a1el2ino93LEPV5CUAgAAAACgggj3DVC94HAddnEIn0lS3eBwVfZ1z9PaGjVqpHXr1jmU79y50+U7bnx8fJSXl2dXFhERoYSEBBmGYZv8fNeuXS6127p1a+Xl5SkxMVEdO3YssE6TJk20bds2u7KtW7c61X5UVJTq16/vdDxbt25Vu3btJEm5ubnasWOHxowZI0m64YYb9PHHH6tOnTry8iqZVI4n9nE55pQCAAAAAKCCMJlMGtOkQ7G2Hdu0gy3Bc7VGjx6tX3/9VePGjdPu3bt18OBBvfjii3r//fc1adIkl9qqU6eOvvnmGx0/ftz2JLguXbooKSlJzz//vOLj4zV//nytWbPGpXYbNmyoe+65R8OGDdOKFSt05MgRbd++XbNmzdJnn30mSRo3bpzWrl2r2bNn69ChQ5o3b55TQ/eK49VXX9V///tfHThwQLGxsTp37pxGjhwp6eJ8U2fPntWQIUP0ww8/KD4+Xl988YXuu+8+h4RdcXliH5cjKQUAAAAAQAUyrH4bBXj5yCznEkxmmRTg5aOh9dq4LYa6devqm2++0YEDB9S1a1e1bdtWH330kZYtW6aePXu61NaMGTN09OhR1atXzzYMrkmTJnr11Vc1f/58tWzZUtu3b9fkyZNdjjMuLk7Dhg3TpEmT1KhRI/Xv318//PCDatWqJUm6+eab9eabb+qll15Sy5YttW7dOk2dOtXl/Thj1qxZeuGFF9SqVStt3rxZq1ev1nXXXSdJqlatmr777jvl5eWpe/fuatGihcaPH6+wsDCZze5J7XhiH5czGaXxvMcyJjU1VaGhoUpJSVFISEhph1Ms57NyFfz4xaxwysweCvbzKeWIAKD8sVqtSkxMVGRkZIl98AJARUY/CrhXZmamjhw5oujoaPn5+bm07RfHD6rf+rdkGJL1CgP5zDLJZJI+7faAuldvdLUhoxiOHj2q6Oho7dy5U82bN5eXl5fb7lgrSVe6Pp3Ns/BJAQAAAABABTJn7yY9uPkjhfn4yyhiZilDhsJ8/PXA5o80Z+8mD0UIXMRE5wAAAAAAVCCpOZk6nuH45LuCGJLOZGXYtgM8iaQUAAAAAAAVSIi3n6oHhNqVGYYhqwwZhmQy5Q/bMzlsB8+rU6eODMOQYRjKzc0t7XA8iqQUAAAAAAAVyITmnTWheefSDgMoEnNKAQAAAAAAwONISgEAAAAAAMDjSEoBAAAAAADA45hTCgAAAACACuTM2hd1Zu2LdmWGYUiGVbaZzk1mh4nOw3tOVHjPiZ4MFdc4klIAAAAAAFQgeRdSlXvueLG2AzyJ4XsAAAAAAFQgFv8QeVWqLnNQuCRTEbVNMgeFy6tSdVn8QzwRXqlZtGiRwsLCrqoNk8mk//73v26JJ1+XLl00fvx423J0dLTmzp3r1n2UVSSlAAAAAACoQMJ7TlS1+xfKmpF8cajelZhMsmYkq9r9C902dM9kMl3xZ9q0aW7ZT0m4NM7Q0FC1b99eGzZssK0/efKkevXqVaIxbN++XaNGjSrRfZQVJKUAAAAAAKhA8s4n649XBl6cP8qwXrny/88z9ccrA5V3Ptkt+z958qTtZ+7cuQoJCbErmzx5slv2U1Li4uJ08uRJfffdd7ruuuvUt29fHT58WJIUFRUlX1/fEt1/RESEAgICSnQfZQVJKQAAAAAAKpDk7xbLyMooOiGVz7DKyM5Q8nfvuGX/UVFRtp/Q0FCZTCbb8muvvaYOHTrY1Z87d67q1KljWx4xYoT69++v2bNnq2rVqgoPD1dsbKxycnJsdbKysjR58mRVr15dgYGBatu2rb7++mu7dhctWqRatWopICBAAwYM0JkzZ5yKPywsTFFRUWrevLkWLFigCxcuaP369ZLsh+8dPXpUJpNJH3zwgdq1ayc/Pz81b95cmzZtsmtv79696tWrl4KCglSlShUNHTpUp0+fLnT/lw/fM5lMeuuttzRgwAAFBASoQYMGWr169VXto6wgKQUAAAAAQAVhGIbOrn+lWNueXf/yxaf0lQEbN25UfHy8Nm7cqMWLF2vRokVatGiRbf2YMWO0ZcsWffDBB9q9e7fuuusu9ezZU4cOHZIkbdu2Tffff7/GjBmjXbt26ZZbbtHMmTNdjsPf31+SlJ2dXWidRx55RJMmTdJPP/2kmJgY9evXz5YAS05O1q233qrWrVvrxx9/1Nq1a3Xq1CkNGjTIpTimT5+uQYMGaffu3erdu7fuuecenT171q37KA0kpQAAAAAAqCDy0s8oJzFekovJJcNQTmK88s6fLZG4XFWpUiXNmzdPjRs3Vt++fdWnTx999dVXkqRjx44pLi5Oy5YtU8eOHVWvXj1NnjxZHTp0UFxcnCTppZdeUs+ePfXoo4+qYcOGGjdunHr06OFSDBkZGZo6daosFos6d+5caL0xY8Zo4MCBatKkiRYsWKDQ0FAtXLhQkjRv3jy1bt1azzzzjBo3bqzWrVvr7bff1saNG/Xrr786HcuIESM0ZMgQ1a9fX88884zS09O1fft2t+6jNHiVdgAAAAAAAMA9rJnpV7f9hTQpKNxN0RRfs2bNZLFYbMtVq1bVnj17JEl79uxRXl6eGjZsaLdNVlaWwsMvxr5//34NGDDAbn1MTIzWrl1b5L6HDBkii8WiCxcuKCIiQgsXLtT1119faP2YmBjb715eXmrTpo32798vSfr555+1ceNGBQUFOWwXHx/vcAyFuXT/gYGBCgkJUWJiolv3URpISgEAAAAAUEGY/RwTEy5t7x/spkgKad9sdhgieOlcUfm8vb3tlk0mk6zWi3Nkpaeny2KxaMeOHXaJK0kFJmZcNWfOHHXt2lWhoaGKiIi4qrbS09PVr18/Pffccw7rqlat6nQ7Rb0e7thHaSjV4XuzZs3STTfdpODgYEVGRqp///46ePCgXZ2HHnpI9erVk7+/vyIiInT77bfrwIEDdnWOHTumPn36KCAgQJGRkXrkkUeUm5vryUMBAAAAAKDUWYLC5R1ZTzKZXNvQZJJ3ZD1ZAiuXTGD/LyIiQgkJCXaJqV27drnURuvWrZWXl6fExETVr1/f7icqKkqS1KRJE23bts1uu61btzrVflRUlOrXr+90QurSdnNzc7Vjxw41adJEknTDDTdo3759qlOnjkOsgYGBTrVfFE/so6SUalJq06ZNio2N1datW7V+/Xrl5OSoe/fuOn/+vK3OjTfeqLi4OO3fv19ffPGFDMNQ9+7dlZeXJ0nKy8tTnz59lJ2dre+//942AdqTTz5ZWocFAAAAAECpMJlMqtxtrMtTSklS5W7jZHI1meWiLl26KCkpSc8//7zi4+M1f/58rVmzxqU2GjZsqHvuuUfDhg3TihUrdOTIEW3fvl2zZs3SZ599JkkaN26c1q5dq9mzZ+vQoUOaN2+eU0P3imP+/PlauXKlDhw4oNjYWJ07d04jR46UJMXGxurs2bMaMmSIfvjhB8XHx+uLL77QfffdZ8trXC1P7KOklGpSau3atRoxYoSaNWumli1batGiRTp27Jh27NhhqzNq1Ch16tRJderU0Q033KCZM2fqjz/+0NGjRyVJ69at0y+//KL33ntPrVq1Uq9evfT0009r/vz5V5wdHwAAAACAiiis/XCZfAMkk5Nf+U1mmXwCFNZ+WMkGpot3ML366quaP3++WrZsqe3bt2vy5MkutxMXF6dhw4Zp0qRJatSokfr3768ffvhBtWrVkiTdfPPNevPNN/XSSy+pZcuWWrdunaZOneruw5EkPfvss3r22WfVsmVLbd68WatXr9Z1110nSapWrZq+++475eXlqXv37mrRooXGjx+vsLAwmc3uScl4Yh8lxWSUlec9Svrtt9/UoEED7dmzR82bN3dYf/78eU2dOlWrVq3SgQMH5OPjoyeffFKrV6+2u93vyJEjqlu3rnbu3KnWrVsXud/U1FSFhoYqJSVFISEh7jwkjzmflavgxy9ml1Nm9lCwn08pRwQA5Y/ValViYqIiIyPL/Ac4AJRF9KOAe2VmZurIkSOKjo6Wn5+fS9um7/lCx17sIxmGZFgLr2gySyaTak38XEEtul9lxNeWo0ePKjo6Wj/99JNatWp11e0ZhqHc3Fx5eXmV+B1r7nCl69PZPEuZmejcarVq/Pjxat++vUNC6tVXX9Wjjz6q8+fPq1GjRlq/fr18fC4mXRISElSlShW7+vnLCQkJBe4rKytLWVlZtuXU1FRbDPkThZU31ks6mfJ8HABQmqxWqwzDoA8FgGKiHwXcK/89lf/jrDNrX9TZL+bIHBAma/rZK1c2DJkDK+nEwpGq3GOCwntOvMqorx3558TV8+Nsm2Vd/nEXlINw9nOgzCSlYmNjtXfvXm3evNlh3T333KNu3brp5MmTmj17tgYNGqTvvvvO5UxxvlmzZmn69OkO5UlJScrMzCxWm6UtI/vPcaJJSUm64Ot9hdoAgIJYrValpKTIMAz+wg8AxUA/CrhXTk6OrFarcnNzXXqYV+75ZOWeO+5kbUPW9DOy5m/HQ8Oclv9auXp+CmMYhm0OqPJwp1Rubq6sVqvOnDnj8HTAtLQ0p9ooE0mpMWPG6NNPP9U333yjGjVqOKwPDQ1VaGioGjRooJtvvlmVKlXSypUrNWTIEEVFRWn79u129U+dOiVJtln3LzdlyhRNnPhn9jc1NVU1a9ZURERE+R2+l/3nGyAiIoLhewBQDFarVSaTSREREXyZAoBioB8F3CszM1NpaWny8vKSl5fzX9+9AsPkVam6XZmRP4zPMC4+mc9kdkh8eAWGubSfa139+vVL5M7QyxM8ZZWXl5fMZrPCw8Mdbhpy9iaiUr3aDMPQ2LFjtXLlSn399deKjo52ahvDMGzD72JiYvTvf//bNnZdktavX6+QkBA1bdq0wDZ8fX3l6+vrUG42m8vth6f5kgnsyvNxAEBpM5lM9KMAcBXoRwH3MZsvJo7yf5x1Xa9Juq7XpBKMDCXBMAzbeS4Pd0rlX5cF9fnOfgaUalIqNjZWS5cu1apVqxQcHGybAyo0NFT+/v46fPiwPvzwQ3Xv3l0RERH63//+p2effVb+/v7q3bu3JKl79+5q2rSphg4dqueff14JCQmaOnWqYmNjC0w8AQAAAABQnpSH+YVw7XHHdVmqf75YsGCBUlJS1KVLF1WtWtX28+GHH0q6eLvXt99+q969e6t+/fr629/+puDgYH3//fe2u6IsFos+/fRTWSwWxcTE6N5779WwYcM0Y8aM0jw0AAAAAACuSv4wroyMjFKOBHCUf11ezXDDUh++dyXVqlXT559/XmQ7tWvXdqoeAAAAAADlhcViUVhYmBITEyVJAQEB5WJYF4rHMAzl5ubKy8urTJ9nwzCUkZGhxMREhYWFyWKxFLstZjADAAAAAKCMyn+AV35iChWXYRiyWq22ucTKurCwsEIfMOcsklIAAAAAAJRRJpNJVatWVWRkpHJycko7HJQgq9WqM2fOKDw8vMw/LMLb2/uq7pDKR1IKAAAAAIAyzmKxuCUJgLLLarXK29tbfn5+ZT4p5S7XxlECAAAAAACgTCEpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPIykFAAAAAAAAjyMpBQAAAAAAAI8jKQUAAAAAAACPK9Wk1KxZs3TTTTcpODhYkZGR6t+/vw4ePGhbf/bsWY0dO1aNGjWSv7+/atWqpXHjxiklJcWunWPHjqlPnz4KCAhQZGSkHnnkEeXm5nr6cAAAAAAAAOCkUk1Kbdq0SbGxsdq6davWr1+vnJwcde/eXefPn5cknThxQidOnNDs2bO1d+9eLVq0SGvXrtX9999vayMvL099+vRRdna2vv/+ey1evFiLFi3Sk08+WVqHBQAAAAAAgCKYDMMwSjuIfElJSYqMjNSmTZvUqVOnAussW7ZM9957r86fPy8vLy+tWbNGffv21YkTJ1SlShVJ0muvvaZ//vOfSkpKko+PT5H7TU1NVWhoqFJSUhQSEuLWY/KU81m5Cn58jSQpZWYPBfsVfdwAcK1K2TlXKTtfclxhSHnWPFnMFsnkuDr0hocVesP4Eo8PAMorq9WqxMRERUZGymxmphAAcEVF6kOdzbN4eTCmIuUPy6tcufIV64SEhMjL62LoW7ZsUYsWLWwJKUnq0aOHRo8erX379ql169YObWRlZSkrK8u2nJqaKuniBWC1Wt1yLJ5mNf6MuzwfBwB4Ql5mivLSjxe+/grb0b8CQOGsVqsMw6CvBIBiqEh9qLPHUGaSUlarVePHj1f79u3VvHnzAuucPn1aTz/9tEaNGmUrS0hIsEtISbItJyQkFNjOrFmzNH36dIfypKQkZWZmFvcQSlVG9p9foZKSknTB17sUowGAsi0n2yz5R11WakgXTl381a+KZHK8Vep8tlnZiYklHyAAlFNWq1UpKSkyDKPc/5UfADytIvWhaWlpTtUrM0mp2NhY7d27V5s3by5wfWpqqvr06aOmTZtq2rRpV7WvKVOmaOLEiXZt16xZUxEREeV3+F72nxO7R0REMHwPAK4k8nGp0+N2Rdac8/pjQbgkqfrwvfLyDS6NyACgXLNarTKZTIqIiCj3X6gAwNMqUh/q5+fnVL0ykZQaM2aMPv30U33zzTeqUaOGw/q0tDT17NlTwcHBWrlypby9/7wLKCoqStu3b7erf+rUKdu6gvj6+srX19eh3Gw2l9sTbzb9GXd5Pg4AKDVm+lEAcAeTyUQ/CgDFVFH6UGfjL9WjNAxDY8aM0cqVK7VhwwZFR0c71ElNTVX37t3l4+Oj1atXO2TbYmJitGfPHiVeMpxi/fr1CgkJUdOmTUv8GAAAAAAAAOC6Ur1TKjY2VkuXLtWqVasUHBxsmwMqNDRU/v7+toRURkaG3nvvPaWmptomJY+IiJDFYlH37t3VtGlTDR06VM8//7wSEhI0depUxcbGFng3FAAAAAAAAEpfqSalFixYIEnq0qWLXXlcXJxGjBihnTt3atu2bZKk+vXr29U5cuSI6tSpI4vFok8//VSjR49WTEyMAgMDNXz4cM2YMcMjxwAAAAAAAADXlWpSyjCMK67v0qVLkXUkqXbt2vr888/dFRYAAAAAAABKWLHmlIqPj9fUqVM1ZMgQ21xOa9as0b59+9waHAAAAAAAAComl5NSmzZtUosWLbRt2zatWLFC6enpkqSff/5ZTz31lNsDBAAAAAAAQMXjclLqscce08yZM7V+/Xr5+PjYym+99VZt3brVrcEBAAAAAACgYnI5KbVnzx4NGDDAoTwyMlKnT592S1AAAAAAAACo2Fye6DwsLEwnT55UdHS0XflPP/2k6tWruy0wAAAAAACAiubM2hd1Zu2LBa7Ly7Mq1VLw/UPhPScqvOfEkgzN41xOSg0ePFj//Oc/tWzZMplMJlmtVn333XeaPHmyhg0bVhIxAgAAAAAAVAh5F1KVe+54oetzr7BdReNyUuqZZ55RbGysatasqby8PDVt2lR5eXm6++67NXXq1JKIEQAAAEAZMmfvJs3Z902B6/Ly8mSxWApcN6FZJ01o3rkkQwOAMs/iHyKvSpeNNDMM5SafkCR5hVWTTKYCt6toXE5K+fj46M0339QTTzyhvXv3Kj09Xa1bt1aDBg1KIj4AAAAAZUxqTqaOZ6QUazsAuNYVNAzPmnVeB0YFSZLqPntAXv7BpRGax7mclMpXq1Yt1apVy52xAAAAACgHQrz9VD0g1K7MkKETGReHllTzD5GpgL/yh3j7eSQ+AED54HJSauTIkVdc//bbbxc7GAAAAABl34TmnR2G4Z3PyVLIe49Lkn4Z8KiCfUlAAQCuzOWk1Llz5+yWc3JytHfvXiUnJ+vWW291W2AAAAAAAACouFxOSq1cudKhzGq1avTo0apXr55bggIAAAAAAEDFZnZLI2azJk6cqDlz5rijOQAAAAAAAFRwbklKSVJ8fLxyc3Pd1RwAAAAAAAAqMJeH702caP/YQsMwdPLkSX322WcaPny42wIDAAAAAABAxeVyUuqnn36yWzabzYqIiNB//vOfIp/MBwAAAAAAAEjFSEpt3LixJOIAAAAAAADANcRtc0oBAAAAAAAAznLqTqnWrVvLZDI51eDOnTuvKiAAAAAAAABUfE4lpfr371/CYQAAAAAAAOBa4lRS6qmnnirpOAAAKFWGYdh+z7twRhafIKfvEgYAAADgOuaUAgBc0/Iyk5Xy0ys68d6NtrITixrqf4uaKOWnV5SXmVx6wQEAAAAVmMtP38vLy9OcOXP00Ucf6dixY8rOzrZbf/bsWbcFBwBASco4uk6Jn/1NRk6Gw7rclCM6u2myzn3/pCL7fKiAOt1LIUIAAACg4nL5Tqnp06frxRdf1N/+9jelpKRo4sSJuuOOO2Q2mzVt2rQSCBEAAPfLOLpOp1bdLiPngiTj/38udbHMyLmgU6tuV8bRdZ4PEgAAAKjAXE5KLVmyRG+++aYmTZokLy8vDRkyRG+99ZaefPJJbd26tSRiBADArfIyk5X42d8kw5BkLaK2VTIMJX72N4byAQAAAG7kclIqISFBLVq0kCQFBQUpJSVFktS3b1999tln7o0OAIASkL7/3f8fsldUQiqfVUZOhtL3v1eSYQEAAADXFJeTUjVq1NDJkyclSfXq1dO6dReHM/zwww/y9fV1b3QAALiZYRhK3TW/WNum7ppn95Q+AAAAAMXnclJqwIAB+uqrryRJY8eO1RNPPKEGDRpo2LBhGjlypNsDBADAnayZZ5SbcliOc0gVxVBuymFZM3mgBwAAAOAOTj99b968ebr33nv17LPP2sr+9re/qVatWtqyZYsaNGigfv36lUiQAAC4izU7/Sq3T5PFP9xN0QAAAADXLqfvlHr88cdVrVo13XPPPdqwYYOtPCYmRhMnTiQhBQAoF8w+QVe5fbCbIgEAAACubU4npRISEvTaa6/pxIkT6tatm6Kjo/X000/rjz/+KMn4AABwK7NfuLxC60oyubilSV6hdWX2q1wSYQEAAOAadum8pXnpZ66ZeUydTkr5+/tr2LBh2rhxow4dOqShQ4dq4cKFio6OVs+ePbVs2TLl5OSUZKwAAFw1k8mkkFaxxdo2pNUYmUyuJrMAAACAguWdT9aZdS/p8NTrbWXxk6P126MNdGbdS8o7n1x6wXmAyxOdS1LdunU1Y8YMHTlyRGvWrFF4eLhGjBih6tWruzs+AADcLqjJUJm8A+T8x6BZJu8ABTW5tyTDAgAAwDUkfc8X+nVCDZ1aMkE5SUft1uUkHdapJRP064QaSt/zRekE6AHFSkrlM5lM8vLykslkkmEY3CkFACgXLH5hiuzzoWQyqeiPQrNkMimy70ey+IV5IDoAKJ8uHWpyJvP8NTP0BACKI33PFzr2Yh8Z2Rd08anQl/WZxsUyI/uCjr3Yp8ImpoqVlPrjjz80Y8YM1a1bV926ddOJEyf05ptv6uTJk+6ODwCAEhFQp7uq3L5KJm9/XZxf6vJheRfLTN7+qtJ/tQJqd/N8kABQDiRnXdDL+75Vq1Uv2srqrZilRh8/q5f3favkrAulGB0AlD1555P1xysDLyaeDOuVKxtWyTD0xysDK+RQPi9nK2ZnZ2vFihV6++23tWHDBlWtWlXDhw/XyJEjVbdu3ZKMEQCAEhFQp7tq3n9E6fvfU+pPryg39YhtnVdotEJajVFw06Ey+4aWYpQAUHZ9cfyg7tqwWBm52Q7rDqed0cTtqzR15xotu3W4elRvVAoRAkDZk/zdYhlZGXK4O6owhlVGdoaSv3tH4d3HlWhsnuZ0UioqKkoZGRnq27evPvnkE/Xo0UNm81WN/gMAoNRZ/MIU2nqMgpqN0LFXLz5Zr9p9h+QTUotJzQHgCr44flD91r918Q/9BazPL7uQm6N+69/SJ90eIDEF4JpnGIbOrn+lWNueXf+yKncbW6H+j+p0Vmnq1Kn6448/tHz5cvXq1YuEFACgQrn0w93iV7lCfdgDgLslZ13QXRsWyzAkaxF/6bfKkGFId21YzFA+ANe8vPQzykmMl9N3SeUzDOUkxivv/NkSiau0OJ1ZmjhxoiIiIkoyFgAAAADlwDu//aiM3OwiE1L5rDKUkZutd+N/LOHIAKBss2amX932F9LcFEnZwO1OAAAAAJxmGIbm7d9crG1f+WUzT+UDcE0z+wVd3fb+wW6KpGxwek6pa8He/2xRkF/gFesEVA9W/WEt7cp+e+dnZRwvOltZpUMtVelYy7acl5WrfS9udSq2esOuV2D1ENty8v7TOvbfA7blC5d8uO97casCTBfzjWZfi5pPjLFr63+fH9LZn08Vuc/QxuGqPaCJXdn+eduVk+Y4keXlavSqr8qtomzLmUnn9etbPxW5nSQ1ib1J3iG+tuWk7cd18qsjV9jiIr/rAtTwwRvsyo58sFdpR5KL3Pa6m6qpWlf7Cft3z3LuP1vRf2um4LqVbMtph8/pyIf7nNr2+ikd7JZPfHlYp384UeR2wdFhih7c3K7s1zd3KvN0RpHbVr0tWhF/qW5bzknN0v75PzgVb8MHWssv4s/3yNldCfrfmt+K3M472EdNxvzFruz3lfuVcuBMkdtWbllFNXo3sCvb++IWWbPyity2Vv/GCmtynW35/PFUxb+zu8jtJKnZxJtl8f2zizz17TGd2nysyO3Kah9RGPqIS/uICwr2v/jbvhe3yKQAWz36CPqIy9FHXIt9ROGutT7CdH2Y4tOKfn9ezpAUn3ZGZ7MyFO4XSB9RAPoI+ojLlcc+gv9HFMEw5B9RVzmnj1x8+p6TDJlk8q8uS2Blu/Ky2kekZ553qm2SUpfISctSTvaVX5LcUF/HsvRs5aRmFdl+XlaufYEhp7aTJCPXuGw5z27bnEsu5pzULOVckpRyiPdCrlP7zc3IdSjLSXPuWK059m9iw2o4f6yXvTGtWXlObWvxczx3uRk5zp2bzAKO1cl4rblWh2Vnty0oDufOTY5DWY6T1+HlHaxhuHBurJedmxznzk1BcjOcPNYLBZ8bZz4ojNzLjjXX+WO9fDRCXpaT8ZbRPqIw9BGX9hHZ0v8npXJSs2XSn68NfQR9hOPG9ov0EY4qXh9RuGutj7iQU7z48qXlZCncL5A+ogD0EfQRlyuPfcSl+H9EwardEqvEjyY7VfdSpsi/Ocx7Wlb7iJws59p2OSk1Y8YMTZ48WQEBAXblFy5c0AsvvKAnn3zS1SbLDO9gX3n7OX4QXMoryKfAskuz7YW59K8lkiSTnNpOkkxepsuWLXbb5hqGdPbixJHeIb7yvkJSysvfy6n9egU4Xh7ewY7HXxCzt/1+TWaT88d62ZvM7Gtxalvvgs5NgLdz56aADxln4zV7mR2Wnd22oDicOzfeDmXeQT4FfuBd7vJrwmRy4dyYLzs33k6emwKuG68AJ4/Vv+Bz48wHhcnrsmP1cv5Yddkc1xZfJ+Mto31EYegjLu0j/vxPn3eIj0z6s036CPoIx43tF+kjHFW8PqJw11ofEeRdvPjyBf//9vQRjugj6CMuVx77iEvx/4iChbQbpqRVT8rIviAZ1iLrGzJJJj951b7dYV1Z7SO8nbheJMlkuDio22Kx6OTJk4qMjLQrP3PmjCIjI5WXV/TJK2tSU1MVGhqqlJQUhYSEFL1BGXQ+K1fBj6+RJKXM7KFgP+c6dADARdac8/p9/sXb42uOPiMv34o1Xh8A3MUwDDX6+FkdTjvj0rOjTJLqBofr4MDHeMIpgGte+p4vdOzFPheH8F0pMWUySyaTak38XEEtunsuwKvkbJ7F5YnODcMo8EPk559/VuXKlQvYAgAAAEBFYTKZNKZJh6IrFmBs0w4kpABAUlCLHqo18TOZfPwlk0kOt1iaTJLJJJOPf7lLSLnC6eF7lSpVkslkkslkUsOGDe0+TPLy8pSenq6///3vJRIkAAAAgLJjWP02mrpzjS7k5sjqxP1SZpnk7+WtofXaeCA6ACgfglr0UMM5/1Pyd+/o7Lq5ykn6c9J974i6qtxtnMI6DJclILQUoyxZTiel5s6dK8MwNHLkSE2fPl2hoX++KD4+PqpTp45iYmKu0AIAAACAiiDM11/Lbh2ufuvfktkwXTExZZZJJpO0/NbhCvP192CUAFD2WQLDFN59nMI6jdTBhy5OH1Fv9hH5XFf7mriz1Omk1PDhwyVJ0dHRateunby9HSc/AwAAAHBt6FG9kT7p9oDu2rBYGbnZkuwfOpf/Vcrfy1vLbx2u7tUbeTxGACgvLk1AWYLCr4mElFSMp+917txZVqtVv/76qxITE2W12k/I1alTJ7cFBwAAAKDs6lG9kY4NekLvxv+ol/Z9qyPpZ23r6gaHa2zTDhpWv41CfbhDCgDgyOWk1NatW3X33Xfr999/1+UP7jOZTOXy6XsVwaXn4sz5bAX5el8zmVUAAACUnjBff41t2lH31b9JoUumSpIOD/yXagVX4v+jAIArcjkp9fe//11t2rTRZ599pqpVq/JBU8qSL+Ro8Y9/6OVv/5wQre6sjaoXHqAxHaI1vE1Nhfkz1BIAAAAl69LvBZV9A/ieAAAokstJqUOHDmn58uWqX79+ScQDF3xxMFF3Lv5RGdmOd6cdPpOhiav2aeqaA1o+vI16NIoshQgBAAAAAAAKZnZ1g7Zt2+q3334riVjggi8OJqrvW9t1ISdPhuTwvJP8sgs5eer71nZ9cTDR80ECAAAAAAAUwuU7pcaOHatJkyYpISFBLVq0cHgK3/XXX++24FCw5As5unPxjzJkyFr403clSVZDMpsM3bn4R/3xRDeG8gEAAAAAgDLB5aTUwIEDJUkjR460lZlMJhmGwUTnHrL4xz+UkZ3ncHdUYayGlJGdp3d+/EPjOtYt0dgAAAAAAACc4XJS6siRI0VXQokxDEPzNhfvHLyy+YjGdohm0kkAAAAAAFDqXE5K1a5duyTigJPOZGQr/kyGy9sZkuLPZOhsRo7CA33cHxgAAAAAAIALXJ7oXJLeffddtW/fXtWqVdPvv/8uSZo7d65WrVrl1uDgKD3r6oZHpmXluikSAAAAAACA4nM5KbVgwQJNnDhRvXv3VnJysm0OqbCwMM2dO9fd8eEyQb6Wq9o+2Nflm+MAAAAAAADczuWk1CuvvKI333xTjz/+uCyWPxMkbdq00Z49e9waHByFB/ioXniAXJ0VyiSpXniAKgfw9D0AAAAAAFD6XE5KHTlyRK1bt3Yo9/X11fnz590SFApnMpk0pkN0sbZlknMAAAAAAFBWuJyUio6O1q5duxzK165dqyZNmrgjJhRheJuaCvCxyOxkfslskgJ8LBrWpmbJBgYAAAAAAOAklycYmjhxomJjY5WZmSnDMLR9+3a9//77mjVrlt566y2X2po1a5ZWrFihAwcOyN/fX+3atdNzzz2nRo0a2eq88cYbWrp0qXbu3Km0tDSdO3dOYWFhdu2cPXtWY8eO1SeffCKz2ayBAwfqpZdeUlBQkKuHVy6E+Xtr+fA26vvWdplNhqxG4XXNJskkkz4e3kZh/gzdAwBJStk5Vyk7X7IvNP7sTE+801wyOf7dJvSGhxV6w/gSjg4AAAC4NriclHrggQfk7++vqVOnKiMjQ3fffbeqVauml156SYMHD3aprU2bNik2NlY33XSTcnNz9a9//Uvdu3fXL7/8osDAQElSRkaGevbsqZ49e2rKlCkFtnPPPffo5MmTWr9+vXJycnTfffdp1KhRWrp0qauHV270aBSpTx/4i+5c/KMysi9ONn9pbir/Jip/b4s+Ht5G3RtFejxGACirrFmpyks/Xuj6vPMnC90OAAAAgHsU61Fs99xzj+655x5lZGQoPT1dkZHFS3isXbvWbnnRokWKjIzUjh071KlTJ0nS+PHjJUlff/11gW3s379fa9eu1Q8//KA2bdpIujgZe+/evTV79mxVq1atWLGVBz0aReqPJ7rpnR//0EvfHtGRsxm2dXXDAzS2Q7SGt6mpUO6QAgA7Zt8QWYKqO64wpDxrnixmiwp6ooTZN6TkgwMAAACuEcVKSuULCAhQQECAu2JRSkqKJKly5cpOb7NlyxaFhYXZElKS1LVrV5nNZm3btk0DBgxw2CYrK0tZWVm25dTUi3/5tlqtslqtxQ2/VIT4WjSmfR2NaFNdYU+skyTFP9ZZtSoF2iY1L2/HBAAlLbjVOAW3GudQbrValZSUpIiICJnNBU+7SJ8KAAWzXjKnRHn8fzUAlKZL+8yK0Ic6G79TSakbbrhBX331lSpVqqTWrVtf8QluO3fudC7Cy1itVo0fP17t27dX8+bNnd4uISHB4U4tLy8vVa5cWQkJCQVuM2vWLE2fPt2hPCkpSZmZma4FXkbkD+GTpLzzKUrKzbhCbQBAQaxWq1JSUmQYRqFJKQBAwTJys22/JyUl6YKPbylGAwDli5H953f4pKQkWfwulGI0Vy8tLc2pek4lpW6//Xb5+l78UOnfv3+xg7qS2NhY7d27V5s3by6R9i81ZcoUTZw40bacmpqqmjVrKiIiQiEh5XNoxvnsXNvvERERCvbzKcVoAKB8slqtMplMV7xTCgBQsPM5fyalIiIiFOzrV4rRAED5Ys06r3P//3tERIS8/INLNZ6r5efn3GeAU0mpp556qsDf3WXMmDH69NNP9c0336hGjRoubRsVFaXExES7stzcXJ09e1ZRUVEFbuPr62tLsl3KbDaX2y8h5kueElWejwMASpvJZKIfBYBiMJtNl/xOPwoALjFXrO/0zsbv8lH+8MMP2rZtm0P5tm3b9OOPP7rUlmEYGjNmjFauXKkNGzYoOjra1XAUExOj5ORk7dixw1a2YcMGWa1WtW3b1uX2AAAAAAAAUPJcnug8NjZWjz76qEPC5/jx43ruuecKTFhdqa2lS5dq1apVCg4Ots0BFRoaKn9/f0kX54xKSEjQb7/9Jknas2ePgoODVatWLVWuXFlNmjRRz5499eCDD+q1115TTk6OxowZo8GDB1foJ+8BAAAAAIDy58zaF3Vm7Yv2hcafD4s4/FhjqYC5vMN7TlR4z4kO5eWZy0mpX375RTfccINDeevWrfXLL7+41NaCBQskSV26dLErj4uL04gRIyRJr732mt2k5J06dXKos2TJEo0ZM0a33XabzGazBg4cqJdfftmlWAAAAAAAAEpa3oVU5Z47Xuj63OQThW5X0biclPL19dWpU6dUt25du/KTJ0/Ky8u15oxLMoGFmTZtmqZNm3bFOpUrV9bSpUtd2jcAAAAAAICnWfxD5FWpeoHr8vKsslgKnmnJ4l8+H8x2JS4npbp3764pU6Zo1apVCg0NlSQlJyfrX//6l7p16+b2AAEAAACULXP2btKcfd/YlRn68w/OTVc+L1MBQ08mNOukCc07l3h8AFCWFTYMz2q1KjExUZGRkeV+onNnuZyUmj17tjp16qTatWurdevWkqRdu3apSpUqevfdd90eIAAAAICyJTUnU8czUgpdf6KQISapOZklFRIAoBxyOSlVvXp17d69W0uWLNHPP/8sf39/3XfffRoyZIi8vb1LIkYAAAAAZUiIt5+qB4QWuC4vL08Wi6XQ7QAAyOdyUkqSAgMDNWrUKHfHAgAAAKAcmNC8c4HD8K7FoScAgOJzKim1evVq9erVS97e3lq9evUV6/71r391S2AAAAAAAACouJxKSvXv318JCQmKjIxU//79C61nMpmUl5fnrtgAAAAAAABQQTmVlLJarQX+DgAAAAAAABSHUwO9K1eurNOnT0uSRo4cqbS0tBINCgAAAAAAABWbU0mp7OxspaZefKzr4sWLlZnJo1wBAAAAAABQfE4N34uJiVH//v114403yjAMjRs3Tv7+/gXWffvtt90aIAAAAAAAACoep5JS7733nubMmaP4+HhJUkpKCndLAQAAAAAAoNicSkpVqVJFzz77rCQpOjpa7777rsLDw0s0MAAAAAAAAFRcLk90fsstt8jHx6dEgwIAAAAAAEDFxkTnAAAAAAAA8DgmOgcAAAAAAIDHuTzRuclkYqJzAAAAAAAAXBUmOgcAAAAAAIDHOZWUutSRI0dsv2dmZsrPz8+tAQEAAAAAAKDic2qi80tZrVY9/fTTql69uoKCgnT48GFJ0hNPPKGFCxe6PUAAAAAAAABUPC4npWbOnKlFixbp+eefl4+Pj628efPmeuutt9waHAAAAAAAAComl5NS77zzjt544w3dc889slgstvKWLVvqwIEDbg0OAAAAAAAAFZPLSanjx4+rfv36DuVWq1U5OTluCQoAAAAAAAAVm8tJqaZNm+rbb791KF++fLlat27tlqAAAAAAAABQsbn89L0nn3xSw4cP1/Hjx2W1WrVixQodPHhQ77zzjj799NOSiBEAAAAAAAAVjMt3St1+++365JNP9OWXXyowMFBPPvmk9u/fr08++UTdunUriRgBAAAAAABQwbh8p5QkdezYUevXr3d3LAAAAAAAALhGFCspJUk7duzQ/v37JUnNmjVjPikAAAAAAAA4zeWkVGJiogYPHqyvv/5aYWFhkqTk5GTdcsst+uCDDxQREeHuGAEAAAAAAFDBuDyn1NixY5WWlqZ9+/bp7NmzOnv2rPbu3avU1FSNGzeuJGIEAAAAAABABePynVJr167Vl19+qSZNmtjKmjZtqvnz56t79+5uDQ4AAAAAAAAVk8t3SlmtVnl7ezuUe3t7y2q1uiUoAAAAAAAAVGwuJ6VuvfVWPfzwwzpx4oSt7Pjx45owYYJuu+02twYHAAAAAACAisnlpNS8efOUmpqqOnXqqF69eqpXr56io6OVmpqqV155pSRiBAAAAAAAQAXj8pxSNWvW1M6dO/Xll1/qwIEDkqQmTZqoa9eubg8OAAAAAAAAFZPLSSlJMplM6tatm7p16+bueAAAAAAAAHANcHr43oYNG9S0aVOlpqY6rEtJSVGzZs307bffujU4AAAAAAAAVExOJ6Xmzp2rBx98UCEhIQ7rQkND9dBDD+nFF190a3AAAAAAAAComJxOSv3888/q2bNnoeu7d++uHTt2uCUoAAAAAAAAVGxOJ6VOnTolb2/vQtd7eXkpKSnJLUEBAAAAAACgYnM6KVW9enXt3bu30PW7d+9W1apV3RIUAAAAAAAAKjank1K9e/fWE088oczMTId1Fy5c0FNPPaW+ffu6NTgAAAAAAABUTF7OVpw6dapWrFihhg0basyYMWrUqJEk6cCBA5o/f77y8vL0+OOPl1igAAAAAAAAqDicTkpVqVJF33//vUaPHq0pU6bIMAxJkslkUo8ePTR//nxVqVKlxAIFAAAAAABAxeF0UkqSateurc8//1znzp3Tb7/9JsMw1KBBA1WqVKmk4gMAAAAAAEAF5FJSKl+lSpV00003uTsWAAAAAAAAXCOcnugcAAAAAAAAcBeSUgAAAAAAAPA4klIAAAAAAADwOJJSAAAAAAAA8DiSUgAAAAAAAPA4klIAAAAAAADwOJJSAAAAAAAA8DiSUgAAAAAAAPA4klIAAAAAAADwOJJSAAAAAAAA8DiSUgAAAAAAAPA4klIAAAAAAADwOJJSAAAAAAAA8DiSUgAAAAAAAPA4klIAAAAAAADwOJJSAAAAAAAA8DiSUgAAAAAAAPC4Uk1KzZo1SzfddJOCg4MVGRmp/v376+DBg3Z1MjMzFRsbq/DwcAUFBWngwIE6deqUXZ1jx46pT58+CggIUGRkpB555BHl5uZ68lAAAAAAAADgglJNSm3atEmxsbHaunWr1q9fr5ycHHXv3l3nz5+31ZkwYYI++eQTLVu2TJs2bdKJEyd0xx132Nbn5eWpT58+ys7O1vfff6/Fixdr0aJFevLJJ0vjkAAAAAAAAOAEk2EYRmkHkS8pKUmRkZHatGmTOnXqpJSUFEVERGjp0qW68847JUkHDhxQkyZNtGXLFt18881as2aN+vbtqxMnTqhKlSqSpNdee03//Oc/lZSUJB8fnyL3m5qaqtDQUKWkpCgkJKREj7GknM/KVfDjayRJKTN7KNiv6OMGANizWq1KTExUZGSkzGZGuAOAq+hHAaD4KlIf6myexcuDMRUpJSVFklS5cmVJ0o4dO5STk6OuXbva6jRu3Fi1atWyJaW2bNmiFi1a2BJSktSjRw+NHj1a+/btU+vWrR32k5WVpaysLNtyamqqpIsXgNVqLZFjK2lW48+4y/NxAEBpslqtMgyDPhQAiol+FACKryL1oc4eQ5lJSlmtVo0fP17t27dX8+bNJUkJCQny8fFRWFiYXd0qVaooISHBVufShFT++vx1BZk1a5amT5/uUJ6UlKTMzMyrPZRSkZGdZ/s9KSlJF3y9SzEaACifrFarUlJSZBhGuf/rFACUBvpRACi+itSHpqWlOVWvzCSlYmNjtXfvXm3evLnE9zVlyhRNnDjRtpyamqqaNWsqIiKi/A7fy/5zYveIiAiG7wFAMVitVplMJkVERJT7/wgAQGmgHwWA4qtIfaifn59T9cpEUmrMmDH69NNP9c0336hGjRq28qioKGVnZys5OdnubqlTp04pKirKVmf79u127eU/nS+/zuV8fX3l6+vrUG42m8vtiTeb/oy7PB8HAJQ2k8lEPwoAV4F+FACKr6L0oc7GX6pHaRiGxowZo5UrV2rDhg2Kjo62W3/jjTfK29tbX331la3s4MGDOnbsmGJiYiRJMTEx2rNnjxITE2111q9fr5CQEDVt2tQzBwIAAAAAAACXlOqdUrGxsVq6dKlWrVql4OBg2xxQoaGh8vf3V2hoqO6//35NnDhRlStXVkhIiMaOHauYmBjdfPPNkqTu3buradOmGjp0qJ5//nklJCRo6tSpio2NLfBuKAAAAAAAAJS+Uk1KLViwQJLUpUsXu/K4uDiNGDFCkjRnzhyZzWYNHDhQWVlZ6tGjh1599VVbXYvFok8//VSjR49WTEyMAgMDNXz4cM2YMcNThwEAAAAAAAAXlWpSyjCMIuv4+flp/vz5mj9/fqF1ateurc8//9ydoQEAAAAAAKAEle+ZswAAAAAAAFAukZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5GUAgAAAAAAgMeRlAIAAAAAAIDHkZQCAAAAAACAx5VqUuqbb75Rv379VK1aNZlMJv33v/+1W3/q1CmNGDFC1apVU0BAgHr27KlDhw7Z1cnMzFRsbKzCw8MVFBSkgQMH6tSpUx48CgAAAAAAALiqVJNS58+fV8uWLTV//nyHdYZhqH///jp8+LBWrVqln376SbVr11bXrl11/vx5W70JEybok08+0bJly7Rp0yadOHFCd9xxhycPAwAAAAAAAC7yKs2d9+rVS7169Spw3aFDh7R161bt3btXzZo1kyQtWLBAUVFRev/99/XAAw8oJSVFCxcu1NKlS3XrrbdKkuLi4tSkSRNt3bpVN998s8eOBQAAAAAAAM4rs3NKZWVlSZL8/PxsZWazWb6+vtq8ebMkaceOHcrJyVHXrl1tdRo3bqxatWppy5Ytng0YAAAAAAAATivVO6WuJD+5NGXKFL3++usKDAzUnDlz9L///U8nT56UJCUkJMjHx0dhYWF221apUkUJCQmFtp2VlWVLeklSamqqJMlqtcpqtbr/YDzAavwZd3k+DgAoTVarVYZh0IcCQDHRjwJA8VWkPtTZYyizSSlvb2+tWLFC999/vypXriyLxaKuXbuqV69eMgzjqtqeNWuWpk+f7lCelJSkzMzMq2q7tGRk59l+T0pK0gVf71KMBgDKJ6vVqpSUFBmGIbO5zN5MDABlFv0oABRfRepD09LSnKpXZpNSknTjjTdq165dSklJUXZ2tiIiItS2bVu1adNGkhQVFaXs7GwlJyfb3S116tQpRUVFFdrulClTNHHiRNtyamqqatasqYiICIWEhJTY8ZSk89m5tt8jIiIU7OdTitEAQPlktVplMpkUERFR7v8jAAClgX4UAIqvIvWhl07FdCVlOimVLzQ0VNLFyc9//PFHPf3005IuJq28vb311VdfaeDAgZKkgwcP6tixY4qJiSm0PV9fX/n6+jqUm83mcnvizaY/4y7PxwEApc1kMtGPAsBVoB8FgOKrKH2os/GXalIqPT1dv/32m235yJEj2rVrlypXrqxatWpp2bJlioiIUK1atbRnzx49/PDD6t+/v7p37y7pYrLq/vvv18SJE1W5cmWFhIRo7NixiomJ4cl7AAAAAAAAZVipJqV+/PFH3XLLLbbl/CF1w4cP16JFi3Ty5ElNnDhRp06dUtWqVTVs2DA98cQTdm3MmTNHZrNZAwcOVFZWlnr06KFXX33Vo8cBAAAAAAAA15iMq501vAJITU1VaGioUlJSyu+cUlm5Cn58jSQpZWYP5pQCgGKwWq1KTExUZGRkub9lGgBKA/0oABRfRepDnc2zlO+jBAAAAAAAQLlEUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHkdSCgAAAAAAAB5HUgoAAAAAAAAeR1IKAAAAAAAAHvd/7d15fIzn4v//12TXRCIRIgtJLIksgiD2fT9aGkstXdBF0VZRbakq6oMuUnq0nFLFsbRHv4e2dmonYok1sWuUIhFBEhHZZn5/5Dd3xdJzTpfE8n7+w8zc9z3XPZnHNff1vq9FoZSIiIiIiIiIiBQ7hVIiIiIiIiIiIlLsFEqJiIiIiIiIiEixUyglIiIiIiIiIiLFTqGUiIiIiIiIiIgUO4VSIiIiIiIiIiJS7BRKiYiIiIiIiIhIsVMoJSIiIiIiIiIixU6hlIiIiIiIiIiIFDuFUiIiIiIiIiIiUuwUSomIiIiIiIiISLFTKCUiIiIiIiIiIsVOoZSIiIiIiIiIiBQ7hVIiIiIiIiIiIlLsFEqJiIiIiIiIiEixUyglIiIiIiIiIiLFTqGUiIiIiIiIiIgUO4VSIiIiIiIiIiJS7BRKiYiIiIiIiIhIsVMoJSIiIiIiIiIixU6hlIiIiIiIiIiIFDuFUiIiIiIiIiIiUuwUSomIiIiIiIiISLFTKCUiIiIiIiIiIsVOoZSIiIiIiIiIiBQ7hVIiIiIiIiIiIlLsFEqJiIiIiIiIiEixUyglIiIiIiIiIiLFTqGUiIiIiIiIiIgUO4VSIiIiIiIiIiJS7BRKiYiIiIiIiIhIsVMoJSIiIiIiIiIixc6uJN9869atfPzxx8THx3Px4kWWLVvGk08+abx+/fp1Ro4cyXfffUdaWhqBgYEMGTKEgQMHGtvcvHmTN954g2+++YacnBzat2/PjBkz8PLyKoEzKh6fbDnN1K0/FXnOYrEY/w/5aDMmk+mO/YY1q8zw5lX+8vKJiIiIiIiIiPwnJRpKZWVlUbNmTZ5//nm6du16x+vDhw9n48aNLFy4kICAANatW8fgwYPx8fGhc+fOAAwbNoyVK1fy7bff4ubmxquvvkrXrl3ZsWNHcZ9Oscm4mc/59Jv3fP1CRs499xMRERERERERuR+UaCjVsWNHOnbseM/XY2Nj6du3Ly1atABgwIABfPHFF+zevZvOnTuTnp7OnDlzWLx4Ma1atQJg7ty5hISEEBcXR4MGDYrjNIqdq5Mdvm5Od33NXFCAja3tPfcTEREREREREbkf3NcpRaNGjfjhhx94/vnn8fHxYfPmzZw4cYKpU6cCEB8fT15eHm3atDH2qV69OpUqVWLnzp0PbSg1vHmVuw7DM5vNXLp0ifLly2Njo+nCREREREREROT+dV+HUtOnT2fAgAH4+flhZ2eHjY0Ns2fPplmzZgAkJyfj4OBAmTJliuzn5eVFcnLyPY+bk5NDTs6vQ9wyMjKAwlDHbDb/+SdSTMxmMxaL5YE+BxGRkqR6VETkj1E9KiLy+z1Mdeh/ew73fSgVFxfHDz/8gL+/P1u3buWVV17Bx8enSO+o/9XkyZMZP378Hc+npqZy8+a952q635nNZtLT07FYLOopJSLyO6geFRH5Y1SPioj8fg9THZqZmflfbXffhlLZ2dm88847LFu2jE6dOgEQERHBgQMHmDJlCm3atKFChQrk5uZy7dq1Ir2lUlJSqFChwj2PPWrUKIYPH248zsjIoGLFipQrVw5XV9e/7Jz+amazGZPJRLly5R74L7CISElQPSoi8seoHhUR+f0epjrUyenu82Df7r4NpfLy8sjLy7vjD2Fra2t0A6tTpw729vZs2LCBbt26AXD8+HHOnj1Lw4YN73lsR0dHHB0d73jexsbmgf/Dm0ymh+I8RERKiupREZE/RvWoiMjv97DUof9t+Us0lLp+/TqnTp0yHiclJXHgwAE8PDyoVKkSzZs3580336RUqVL4+/uzZcsW/vnPf/LJJ58A4ObmxgsvvMDw4cPx8PDA1dWV1157jYYNGz60k5yLiIiIiIiIiDwMSjSU2rt3Ly1btjQeW4fU9e3bl3nz5vHNN98watQonn76aa5cuYK/vz8TJ05k4MCBxj5Tp07FxsaGbt26kZOTQ/v27ZkxY0axn4uIiIiIiIiIiPz3TBaLxVLShShpGRkZuLm5kZ6e/sDPKXXp0iXKly//wHf1ExEpCapHRUT+GNWjIiK/38NUh/63OcuDfZYiIiIiIiIiIvJAUiglIiIiIiIiIiLFTqGUiIiIiIiIiIgUO4VSIiIiIiIiIiJS7BRKiYiIiIiIiIhIsVMoJSIiIiIiIiIixU6hlIiIiIiIiIiIFDuFUiIiIiIiIiIiUuwUSomIiIiIiIiISLFTKCUiIiIiIiIiIsVOoZSIiIiIiIiIiBQ7u5IuwP3AYrEAkJGRUcIl+WPMZjOZmZk4OTlhY6O8UUTkf6V6VETkj1E9KiLy+z1Mdag1X7HmLfeiUArIzMwEoGLFiiVcEhERERERERGRh0NmZiZubm73fN1k+U+x1SPAbDZz4cIFSpcujclkKuni/G4ZGRlUrFiRc+fO4erqWtLFERF54KgeFRH5Y1SPioj8fg9THWqxWMjMzMTHx+c3e32ppxRgY2ODn59fSRfjT+Pq6vrAf4FFREqS6lERkT9G9aiIyO/3sNShv9VDyurBHqQoIiIiIiIiIiIPJIVSIiIiIiIiIiJS7BRKPUQcHR0ZO3Ysjo6OJV0UEZEHkupREZE/RvWoiMjv9yjWoZroXEREREREREREip16SomIiIiIiIiISLFTKCUiIiIiIiIiIsVOoZSIiIiIiIiIiBQ7hVIiIvJIycnJYdKkSTz33HMlXRQRERERkUeaQqkHUFxcHBUqVOCnn34q6aKIiDxwLBYLN27cYO3atSVdFBGRB5bWShIR+X0sFgsFBQWYzeaSLsp9QaHUA8T6pXV0dMTR0ZEff/wR0EWBiMh/YrFYjDrUycmJpk2bkpOTw5EjR0q4ZCIiDw5rQwrAZDIZz4mIyH/PZDJha2uLjc2vccyjXJcqlHpAWCwW40sbEhJC3bp1+eGHH4zXRESk0N3qRJPJhI2NDdnZ2cTGxmIymahatSorV64sgRKKiDyYrA0pgN27d7Nu3TojnBIRkV+ZzeZ7ttPPnTtHTEwMffv2ZdasWVy5cgWTyfTItusVSt1nLBbLPRtUWVlZLFu2jG+++YbGjRsTHx8PUCRhFRF5VN1+9/721z7++GPc3NwYNGgQX331FYcOHSI2Nra4iykiUqKsvUZv3LjB5cuXgaJh/n8aUvLhhx/i6elJt27deOONN+jatSuJiYl/baFFRO4zR48eZfbs2aSkpACFdeutdaeNjc1dr0kPHDjAE088wbfffoubmxuzZ8+mQ4cOZGdnP7Ihv9KM+4T1C2wyme76ZdyxYwfh4eGMGDGCjRs3MnfuXFJSUjT0RETk/2e9e3/q1Cl27txphFQACQkJfPzxx8yaNYu9e/fy2muvERERwfbt20uquCIixS4lJQUbGxuSk5MJCgpiw4YNQNEw3zqk5MqVK0Zjy2r58uUsXbqUmTNncu7cOdavX4+dnR0jR458ZO/wi8ijISMjg2XLlnHgwAEAdu7cyd69e43608bGxugsYrFYWLduHX//+985duxYkeP079+fzp07ExcXx9///nf27NnDyZMn+fTTT4tcuz5KFErdJ6xf4MTERL766iuOHTtWJGkdN24cERERHDlyhDlz5tC/f3+cnJxYvnw5gCZJE5GHntlsLvJjfXsDaM6cOQQEBFC3bl2GDh1Ku3btuHTpEgD79+/H1dWVrl27Ym9vT+PGjXn77bfJzc1l27ZtxXoeIiIloUOHDowYMYKbN29SoUIFDh48SM+ePYFf69czZ86wceNGIiMj8fb2pnfv3vzrX/8yjrF161Zq1apFjx49OHXqFGvXruXIkSPExsaSkJBQUqcmIvKXS0xM5LXXXmPjxo0APP/883zxxReUL1/e2OaTTz5h69atPPfcc7z88svMmjWLDh06GDdBf/rpJ1xcXGjdujWbN2/m6aefJjQ0lPT0dHJycrh582aJnFtJUyhVDKxD8n4rOIqPj6dx48Y0adKEL774gujoaMaOHQsUjjm9evUq7du3x9HREXt7e/r27cvjjz/OihUrius0RET+VPcarnwvNjY22NraYjabSU5OLnJn/8iRI8yYMYO33nqLixcvsmbNGiwWC6NHjyY7O5uTJ09SpUoVrl27ZuwTERFBQEAAq1at+jNPS0TkvmK9/qxatSpXr17ll19+AaBs2bKcOnWKtLQ0bGxsGD9+PMHBwSxatIjXX3+dXbt24evry6BBg0hOTiYrK4tz586xYcMGKleuTJ06dfj888/p3Lkz33//PSEhISV5miIif6m6detSvXp1zp49a9wk/f7775k/fz6ZmZlAYSjVvn17fH19OXHiBDt27CAwMJDx48eTn59PSkoKly5dok2bNrz44os4ODgwfvx4fvnlF8aOHYuzs3NJnmKJUSj1F7i9kWUdkmdjY0NOTg779+8nPT0dKLxQyM7O5vPPP8ff35+UlBR27drFxx9/zLp161iwYAEODg5kZWXh6OhoHNPV1ZX69euTmJhIXl6e5pUSkfueNZz/T8OV7+aXX37h0KFDtG3bFg8PD7p06cLKlSvJzc0F4NNPP6VVq1YMHjyY7OxsDh48SHp6Ov/+9785evQoNWrU4PLlyyQlJRnHLF++PPb29hrCJyIPFbPZTH5+vnE9aq1zGzduTEpKitGjac2aNdSpU8foLdqrVy/y8vK4evUqTz/9NLVq1WLBggXk5eWxYsUKnJ2dcXZ2xsHBgffee4+ff/6Z3bt3M2nSJJo0aVIyJysi8ie7vWe+9Tl7e3uqVq3KqVOnOH36NACLFy/m888/5/z58wC88sor5OTk0KZNG+zt7XFzc+Ptt9/mwIEDJCQkUKVKFVxdXenXrx+nTp1i7ty59OjRAx8fH06ePEleXl6xn+/9QEnGf+l/uaNvbWRZLwJOnz7Nrl27GDduHC4uLjz55JO8+OKLnDx5EhsbG44cOcKOHTtYtGgR169fZ8mSJSxevJg9e/awbds2vLy8KF26dJEvqr29PWazmWvXrrF7926jjCIi95tbJyC3jre/du0a//znP+nSpQsxMTF37HPx4kWysrIAmDx5Mn/729+YMGECbdu2Zfv27ZQvX55x48axY8cOANLT0/n666+JiorC39+fl19+mcjISBYsWEBkZCT169cHYOnSpcZ7JCYmkpSURFxcnHGHS0TkQWdjY4OdnR0mk4n8/Hzs7OwAaNCgARaLxZiPtHHjxpQtW5ZffvkFi8VCaGgoLi4u1KxZ09gHIDIykq1btwJQp04dbGxsKF++PGXKlDG2mTdvHtOnTwd0PSoiDzZrz3wonJP08uXLRgeQBg0acOnSJWOeqF69epGRkcG5c+cAaN26NSaTiVKlShnHa968Ofn5+ezbt4/y5cvTtGlTNm3axPbt242bqwcOHGDo0KHs27evOE/1vqFQ6r90rzv6d/vhTUlJYfPmzdjY2JCYmMjTTz/NSy+9xNWrVzl+/Dhz585l7969xo+3s7MzJ0+eJCgoiICAAMaNG4ebmxvLly9n0qRJALRs2ZJNmzaxfv16ALKysoiLiwMKE1rQvFIicn+y/rCvW7eOV199lVq1auHh4cGkSZOoXLkyffr0KVJ/xcfHExUVxZw5cwAIDQ0lKyuLtLQ0Xn/9dcLDw/nggw9wd3c3ht5FRESQmprK8OHDiY+P58iRI8yePZtOnTphsVgIDAxkxIgRfPHFF7z00ktMnDiRjz76iMGDBxMeHq6Vo0TkgfJbK+Tt27ePgQMHEhUVxQsvvGD0hAoMDMTLy4sTJ05w7do1Spcujb+/P/v27SM1NRWA+vXrc+jQIa5fv24cr127dsTHx3P+/Hl69+5NkyZN6NmzJ6NHj2b27Nn06NGDadOmGY2rR3X1KBG5/9yt55HFYinSm/R2SUlJdO3a1ehM8swzzzB37lygsI4sKCgwQqnmzZuTlZXFyZMnAYiKisLBwYEjR44YnVpKlSpFcHAwcXFx5ObmMmHCBMLDw3n22Wfp2rUroaGhtGvXDk9PT8qWLfsXfRL3N4VS/6UzZ84wbdq0O2bPN5lM3Lhxw3h88+ZNJk6cSK9evQAIDg4mLCyMs2fP0qdPHypXrkyrVq148cUXiYuLIykpCXt7ewICAmjTpg3nz5/n8OHDzJw5k06dOuHq6grAG2+8QVBQEP369ePll1+mS5cuuLm5MWHCBFxcXIBfG34iIveTIUOGYGNjQ//+/UlMTOTYsWPMmDGDY8eO8e677zJq1CgjpIfCYXV16tRhz549QGHg5OPjg6enpzGMuUqVKlSpUoX9+/cDheP88/LyCA0NJSgoyKgP16xZw7x587h27Rq9e/fm66+/Ji0tjX//+9+0adOGN954g/3799OgQYNi/lRERH51e4/8hIQEnn32WWP+J4vFUmQ4iXWFvNslJSUxYsQILl68yODBg7G3t6dPnz4sW7YMgNq1a/Pzzz8bDahmzZqRmJhoDD158skniYuL4/Lly8Yxn3jiCS5dukRiYiLu7u7ExMQwffp0Dh8+zIwZM6hQoQKzZs3i7bff/vM/GBGR/8Ld5m/u3LkzL730kjF5+K3TR1h7k97ajgfIycnhww8/NKZ3iI2N5YknnmDo0KGcOXOGkJAQypUrx4kTJ8jMzMTDwwN/f38OHjxo1Jt169Zl06ZNZGdnGyF9+/btjba/s7MzixYt4uuvvyYyMpLRo0dz/Phx5s+fT9WqVf/qj+q+9EiHUv/LkLyrV6/yzjvvcPHiRaDwjj9ATEyMMSwEwMnJifr16+Pk5MSpU6ews7OjevXqlClThscee8zYrmbNmpjNZnbt2kWVKlUICgri5MmT2NraGo2pY8eOMWXKFOLi4ihfvjzTp09nwoQJJCcn06ZNGyZOnMjo0aP58MMP/6yPRETkTzds2DASEhI4f/48q1evpn379mzZsgUorFtTU1PZuXOnsb2XlxcRERHs3bsXKLy7HxISQlZWljHMzsnJierVq5OWlsbhw4dp164dXbp04cknn2T06NH8+9//ZvDgwYwaNYq0tDQjzIqOjmbp0qXs27ePIUOGFBl+IiJSnH5rjr0yZcpQpkwZSpcubbxuvT5MS0vjs88+o1+/fqxatYrs7Gxjv7feegtvb2++//57+vXrx5dffkn16tWZPHkyUHhX/9q1a8ZN1g4dOnDhwgXOnDkDFDbiUlJS+Omnn4xjRkREkJaWxp49eygoKMDZ2Zl+/fqxdOlS9u/fz/Tp04mKivrrPigRkbu4tR1vnSICfg2fJk6cyNSpU3FycgIKh+Xl5uaSkZHBm2++ScWKFWnWrBmffvqpcZxr166xcuVKpk+fTq1atcjIyMDFxYXMzEwWLFgAQK1atThz5owxr1STJk1ISEjgwoULQOG15u7du4uE++3atePq1avk5OQAhSOlGjRowPvvv8/TTz+Nu7v7f1wY7WH2SIdSt18AWEOqW7/g1v87OTnh4uJCr169cHBw4MknnyQrK4uyZcuSl5dXZPxn1apVcXFxMYKrkJAQfHx8jOF21ufKly9vjNF///33SUpKonHjxkydOpUXXniB7t27s3v3bsqVKwcUXqC8/PLLfP/994wcORJvb++/7sMREfmTBAYGEhoaajyOjIwkNjYWgICAAOrVq8exY8eMH2oHBwdq1KhBZmYmBw4cAKB69epcuXKlyJLj4eHhODs7GxOVT58+nZEjR3LgwAHefvttUlNTmTBhAkOGDCkytl9E5H5gnWMPYNu2bcyZM4fjx49jsVjw8/Nj+vTpuLm5GduPHj2av//974wdO5aFCxeSlpZG3759efPNN7FYLFy7dg2z2UxAQABz586lcePGeHp6cuDAAZo2bUpWVha1a9fG0dHRCKUaNmzIY489xvHjx8nNzaVixYrY29uzevXqIsNbfvzxRwYPHlykV76dnZ0xqfqj2pASkZJjbcefOHGCyZMn8/jjjzNgwAA2btwIQI0aNXB3dzfmKD179ixOTk5MnDiRlJQUpkyZQpcuXRg2bBhTpkzBbDazbds23N3d6d+/P/7+/tStW5d//OMfjB49mu7duwOF9ebVq1c5evQoAB07diQ5OdkI9zt16sRPP/1UJNxv3Lgx586dIyIiosg53H5z4lFdvOyROOt7/VCeOnWKN99803hsDalMJhNnzpxh586dxvj4BQsWYG9vj5+fH7Gxsdy4cQNnZ2ejF9SmTZuM41SqVInq1auzYcMGoLDhVKZMmSLBlb+/P8HBwRw/fpy8vDyioqJYvXo13bt3Z9WqVdy8eZNJkyaxZMkSqlSp8ld8LCIixc7JyYk6deoYc+zZ2dlRrVo1cnJy2LVrl7FdQEAAbm5uRj1ao0YN7Ozs7rgBYDKZjIsPPz8/BgwYwJIlSzh16hTffvstjz/+OA4ODsV7kiIi/4V58+ZRt25dHBwc6NOnDzExMbRo0YIvvvgCgCVLljBo0CCuXbsGFC4AMXToUNLS0ti2bRvLly9n5syZfPfdd8ybNw8nJyeuXLnC1KlTmTNnDh06dGDVqlWcPXuWmJgYnJ2d8fT0JCAggKSkJKP3f0BAAD/++KNxl3/UqFHUq1cPW1tbTCYTFouFVq1a4e7ufsc5WCdVf1QbUiJSclJSUnjssceoUaMGq1atombNmpw6dYp27doZc476+voydepU8vPz8fPzIyIigo8//pg+ffrQs2dPxowZw8SJE1m8eDHx8fH4+fmRkZFBfn4+X331FceOHWPnzp1MmDCBkJAQoHDeKEdHR06cOAFAo0aNSE5OJiEhgYKCAoKDg9myZQvNmze/o8y35xK33px4lD0Sn8C9/tBHjhwhJibGuBO/c+dOtm7dSvfu3QkJCaF379688cYbJCcnM2nSJMaNG0dBQUGR3lWBgYFUrlzZWAEKwNPTk6pVq3L48GGg8Mc+MDCQs2fPkp6eDhTeXfL29iYpKclYBSUoKIh33nmH9evXs2jRIjp37qzGlIg8dKpUqYK3tzcrV64ECsMlT09PNm/ebGxjvbNlnVcqPDycUqVKGQEUFIb777///h1DmJ2dnf/6kxAR+Q0FBQWsXLmSK1euAEWHmVjnhlqyZAlJSUnExsZy7tw5Vq5cSdOmTXn//fc5f/48ZrOZWbNmcenSJQB69uwJFM7xZG9vD0D37t1p2LAh69atw8nJCV9fX5o0acL69esZM2YMUVFRlCpVir179xo9TWvUqMHBgweN69S+ffvSqVMnY4Ld9957j6eeesq43rX+q1X1RKS4WCwW1q9fb/ROulv94+XlRUFBAR988AHbtm1j4sSJrFy5klq1ajFt2jSgcH6nPXv2cOPGDWxsbKhfvz5BQUHUrVvXOE7Hjh1xc3Njx44dBAcH4+PjQ6VKlWjdujUVKlQAIDU1lVGjRpGVlYW3tzf29vbs37+ftLQ07O3tWbNmDa+//jq2traYzWaaNm1610UfFEDd3SPxqcyZM4cePXoYSzVaLwZCQkKoUaMGsbGxxMfH88wzzzB8+HDq1atHamoq7777LmvXrjVm22/UqBGpqakcP37cOLaXlxdhYWEcP37cOK69vT1paWlcuHCBgwcPAoXh1fHjx40GFkCfPn3YtGkTNWvWNJ7TiiUi8rCrUKECtWvXZs2aNUBhSBUREcGiRYuMCSfXr19PdnY2q1atIjc3F29vb1q1akX79u3Jz883jtWyZUsqV65cIuchInIviYmJ9OrVi/nz5wN377Xfp08fnJyc8PT0BAqvFSdOnEhycjKHDx+mU6dOAMbd+MjISBwcHO6YNyUoKIiLFy9y5coVnnrqKc6cOcPbb79NUlISN2/eZOvWrUyYMMHoedq2bVueeuopgoKCjHIMGzbMmL8KKFLPWukaVUSKi8lkIjk5maVLl5KcnHxH/WOto1q1asXWrVu5cuUKubm5lCpViqZNmxqLOURHR7Nz505jhdHWrVtz/vx5srOzjaCrVq1a5OXlceHCBdzd3Rk8eDBff/01zz//POvWrePLL79k0KBBxMXFGYtCxMTEMHPmTMqWLWuEUNabogqe/nePxCdWtWpVzp49a0xGZlW+fHkiIiJYvnw5kZGRVKxYkatXr/Lqq6/i4uLCCy+8QJs2bdi0aRMFBQWEhYXh4eHBoUOHjFn84ddxpVOmTAHg4MGDnDlzhhs3bvD1118D0KJFC9566y3CwsKM/fz8/PD39y+GT0BE5P7h6upKgwYNjNDe09OTV155hbS0NDp27EiDBg1YsWIFn3zyCa+++qoxdGXkyJEMGDAAOzu7Eiy9iMi9WZcf9/X1pUOHDmzbtu2ObazzMnXs2JGLFy8aPZbg1wDKycmJ0qVLU6lSJbZt20Zubi7lypUjPDycjRs3UlBQYDR8ypQpw7Vr13B0dKRz586MGTOGTZs20a1bN/z9/enatSteXl60bt0aKBx6MnbsWAICAoz3LSgoKNITQfWsiJSEgoICI3CvU6cO/v7+rF271njNyhpSRUdHs2vXLiwWCw4ODuTn57N//36qVasGFNazly9fNkKqpk2bkpOTw6ZNm4oEXadPn6ZixYoAPPPMM8yaNYuCggIGDRpETEwMQUFBzJw5k6CgICwWC3Xr1jXmd1YI9cc9Er84tWvXJj8/n4SEBFq0aGF8cVxdXWnUqBHvvvsuJpOJGjVqcOXKFeN1k8lEWFgYu3btIj4+nqioKOrVq0diYiIXL14kMDAQKExo+/Xrx4wZM5gzZw4Wi4VXXnmFwYMH4+XlBRR2Hby1m6CIyKPK1taWmjVrkpOTw44dO2jcuDGhoaGsW7eOH374gVKlStGnTx/j4kBE5EFhHVbn4eFBkyZNjFXvbp0gHAp7OZUtWxZfX18OHz5MVFQUW7Zs4a233iI6OtpYHKJDhw5s3ryZjIwMPD096dixIzNmzKB169b07NmT69ev8+OPP+Lr62vcpX/22WeJjo5m/fr1VKxY8a7Xn9ZGn/Wa9/byiYiUBGtdlJKSQnp6Oo0aNeKHH36gb9++d93u8ccfZ+DAgcybN49z586xbNkycnNz+eabb4DCUU2+vr7s2LGD1q1b4+3tTe3atRk1ahQATz75JLNmzaJ06dJF6so+ffoQHR2Nra3tHdPp3DqkWT1I/xyPRCjl6upK2bJlOXbsGHl5edjb2xtfoho1alBQUMD+/ftp2LAhu3btYu/evTRt2hSAsLAwHB0diY2NJSoqik6dOvH2228za9YsqlSpwuHDh/n000959913admyJefOnaN169bqASUichfWutfPz4/KlSuTkJBA48aNgcKhKZGRkSVcQhGRe7OulGSdBPxWeXl5LFq0iNmzZ+Ph4UHNmjW5evUqiYmJhIWFFWnAmM1mbGxsiI6O5t133yUmJoYKFSrwzDPPMGTIEGNIX7du3Zg/fz4XLlzA09OTv/3tb0ycOJHJkyezadMmtm3bRnZ2Nt99912Rsri4uBAdHW08ts6Jag2hdGdfREqCtbfTvYLwjRs38sILL5CVlUWLFi04fPiwsfDY7ftYLBZ8fHyoUqUKo0aNonPnzvzf//0fvXr1wt7envz8fOzs7GjTpg1bt24lMzMTDw8POnXqxLhx49izZw9jxowhPT2d9957j4YNGxY5vnXl5nvV+wqk/jyPRCgFhSnqmjVr+Pnnn6latarxxfLx8SE4OJgNGzYQHR2NyWRiz549RigVHBxMxYoViY+PBwq7AKanpzN16lRMJhNt27bFYrHw2GOP0bZt25I8RRGR+571B7xKlSrs2bMHW1tb3WkSkQfGrSsl3V53rVq1ig8++IAuXboQFRXFV199RV5eHuvXrycsLMy49rQeBwrv0n/22WcsXryY9u3b3/F+rVu3xmw2c/DgQWrUqEFERARubm4MHjzYWL25S5cuxiTlt7IOxzOZTOoJJSL3hVvroqNHj1K1alWjh2lWVhajRo2iXbt2xoIPI0aMYPPmzRw/fpzg4OAi9W5BQQF2dnY0adIEb29v5s+fb/QYNZvNxnbdu3fniSee4OzZs3h4eNCwYUOcnJwYPHgwQ4cO/Y8r3WuFvL/eI/PptmrVisuXL5OYmAj8+kNdrlw5KlasyLFjx6hUqRI+Pj7GNgA+Pj54eHhw7NgxsrKyeOyxx+jfvz+HDh3i0KFDxMTEqDElIvI/srGxMS5MVIeKyP3CYrEUmdPk1uehcKXmF198kaioKMaPH8/+/fuNbRYsWICPjw8ffvgh3bp1Y8aMGTRp0oRly5YBRes6awOnefPmmEwmrl69esd7WXs3eXt7ExcXR25uLs7OztStW5etW7fyyiuv8Pzzz981kLK+n+pXESlOFouF/Pz8e67WuWvXLjp06EDp0qXp1q0b3bp1M1YGPXHiBGfPnqV37954eXkRGRnJhAkTqFSpEitWrACKLhphrd+6dOnCoUOHjEXNoOh1Ztu2balVq5YxV179+vWxt7cnMTHRCKR+q8zy13tkQqlq1arh5ORkLDlu/ZJev36d7du306BBA+zt7alQoQK7d+/mwoULxr7vvPMOmzZtMpJX/ciLiIiIPBwsFovR0LH2Krr1rrg1HNq+fTuDBg3i5s2bPPfcc+zfv5+nn36aI0eOkJ6eTnJyMi1btjT28/Pzo0ePHhw+fJj8/Pw77rQXFBRga2tL7dq12bp1q7GIzu3XmLGxsXz++ec4OjoC8MQTT7BlyxYyMjIwm81qSInIX+puq4fe7taemXZ2dphMJrKysoocIycnhylTphjt7fXr12Nra8vYsWM5d+4cZ86coUKFCkZdB4Vt+Hr16hkrNt/K2p5v3749GRkZRW4S3MrBwYF9+/YRHh4OFE7tExQUxL/+9S9jG2uZpWQ8MqGUo6Mjjz/+OIsWLWL+/PmkpqZy8uRJPvnkE0JCQmjRogVQOHb//fffx8PDw9i3UqVKuLi4lFDJRUREROTPdPsqTtbA6MqVK8TExDBo0CC+//57oLDhk56ezpQpU+jcuTMLFy7k1Vdf5euvv8bR0ZHx48fj5uZGQUEBN2/e5MaNG8Z+fn5+3Lx5k+3btwNFG3fWRlzDhg1Zs2YN169fL1JGa4PLusKTVbNmzUhJSSE1NRUbGxs1pETkT2XtMWp1t6Frt4fhJpOJ3NxcLly4wHvvvYePjw+tW7dm8eLF3Lx5ExsbG5YsWUJaWhrz5s0jJCQER0dH/Pz8WLVqFatXryYsLIysrCx+/vln47jly5fH1dWVAwcOAHfOK1VQUECpUqWoU6cON27c+M2QPj8/3/j/Sy+9RM2aNRXq3ydMlkfsLzFo0CD2799v3NGKiIjg/fffp3nz5iVdNBERERH5ne41P91/mrcuMzOTuXPnkpqaSm5uLjt27MDb25v169fz0UcfMWDAADIyMvDy8mL16tXEx8fz//7f/+P48eOULl2afv36MX78eAYOHMiZM2f47LPPqFq1KgAzZszg1VdfZezYsYwdO9aYePfWcl29epW8vDzKly//h89VROT3KCgouGvInZCQwDfffIODgwPDhw+/o6PGtWvXKFOmDMuXL+fjjz/G29ubChUq0KZNG1auXMmaNWuYPHkyvXv35rPPPmPUqFF0796d1atXk5ubS0hICNHR0fTo0QN/f3+aNWuGv78/X375JY6OjqSlpdG2bVsOHDjAjh07aNiwobFQBKgufFg8cqEUwKlTp7h48SL169e/Y4lHEREREXmw3NowuXnzJk5OTsbwuLttu3btWubMmcO3335LXl4eTz31FFu2bKFXr15MmzYNBwcHXnjhBX766Se++uorPD09iYqK4uTJk7Rp04amTZvSoUMHwsPDjaEmmzZtYvTo0bi7uzNr1iwyMzOZPHkysbGx3Lhxg/Pnz6sBJSL3teTkZFasWMHGjRtZuXIlFouFGjVqMGnSJJo2bWqEQRcvXqR///6UK1eOBQsWsH//fgYOHEhycjIrV64kPDycK1euMGjQICwWC0uWLOG7776jZ8+evPTSS3Tu3JnIyEhjpVGr5cuXM3jwYKKioujZsyebN28mLy+P+Ph4+vTpw4gRI+5Z9nvV+XL/e2SG792qatWqNG3aVIGUiIiIyEPAZDJx7NgxatSowdq1a4Ffh3mkpqayaNGiItueO3eOdevWcfLkSezt7WnWrBkFBQX07NnTuD6Mjo7m+vXr7Nq1i1KlSuHr60unTp1Ys2YNo0ePpk6dOtjb25OQkMCJEydo2bIlkyZN4pdffqF+/frUrVsXV1dXli1bxnfffWe8t4jI/Wbjxo3Y2NgQGBjIBx98wA8//EDLli3JyMhg+/btHDlyhLfeesvYvnTp0gQHB3P8+HEAQkNDqVatGqVLlzbmbvLw8KBmzZqcPn2a8+fPExoaioODAw0bNqRdu3ZGIHX06FHmz5/Pzz//zBNPPMGsWbOwsbHhtddeIzMzk9dff524uDhGjBjxm8PtFEg9uB7JUEpERERE7m+3TkB+O7PZTEFBQZEGSvXq1bl+/To7d+5k2rRpREZGkpKSwtq1axk8eDB79uwxtq1Zsyb+/v78+OOPAAQHB1O5cmWOHj1a5HjlypVj+/bt2NnZ8dxzz7Fu3Tq++OILUlNTyc7OZsWKFUyaNImkpCQsFgstWrRg1apVLFy4kAsXLjB9+nTCw8OpV6/eX/QpiYj8cbVq1WLx4sVcvnyZU6dOMXToUH766SegcGGwffv2sWHDBmN7FxcXateuzYULFzh79iyOjo4EBwfj5ubG4cOHje1CQ0MB2LFjB0FBQbz44ouMHDmSN954gz179jBz5kyGDh3Kjh07KFWqFAAdO3ZkwYIFpKSksGDBAsLDw3FwcMBsNivYf0gplBIRERGR+86tE5BDYRBlDamsy33f2kBZtGgRly9f5qOPPmLBggU0a9YMFxcXqlWrRkREhDHZOBSujFetWjUjlAoNDcXb25tDhw4Z2wQEBFClShWOHj1Kfn4+zz33HK+99hpTp06lS5cu+Pr6MnDgQLy9vQkPDzfK4uvrS4sWLXB1dTVWx3sEZ8sQkQeIh4cHvXr1MlabDwsL48yZM6SmplK6dGkaNGjAzZs3iwRO1p5R69evByA8PByz2cy+ffuMbUJDQ6lQoQKxsbEAvPfee7zzzjskJiYSHR3N9OnTadKkCSNHjiwyr56TkxNQODn5rfW+PJz0lxURERGREnOvwObMmTNER0dz+vRpoLBBYm2UJCQk8Nprr/HSSy+xceNGAOrUqUO/fv0ICQlh/vz5TJs2DWdnZwICAvDz82Pbtm3GsT09PalVq5bRwAoICMDf35/Tp0+Tnp4OFC4RHhISwi+//MLu3bsB+Oijj1i+fDlDhgxhw4YNXLhwgZiYGHx9fe96TtaJg3V3X0QeJNZeT9bh0FWqVMHFxYUtW7YY21SsWJGAgACjDg4PD8fNzc1YKQ/A398fd3d3Nm/eDIC7uzuDBg1i4cKFJCUlceTIEcaMGUPlypXvWg47OzuFUY8A/YVFREREpMSYTCauX79OTk4OgHFXPCMjg9jYWFasWAHAyZMnGTVqFGPGjCEmJobz589z8eJFnnvuOZYuXUr16tUZNmwY6enpRRpFXl5ehIaGcvr0aTIzMwFwcHCgXLlypKWlGXf1w8LCyMzMLLJv9erVqVu3bpHyVqtWjV69elG7dm2AO4YRWs9JRORB5evrS3h4uFH/BgYGEhgYyNatW4tsYzabSUhIAArrRl9fX/bu3WvUtY6Ojjz77LN89NFHRt1usVjw9PTE3t6egoIC8vPz1Zv0EadQSkRERERKzFdffUWLFi2M4XXWxknFihVp06YNq1atMrbdsmUL//jHP6hTpw5Lly5l4cKF1K1blwULFlBQUEDlypVxd3cnISGB7OxsY78aNWqQlZXFzp07jecOHjzI1atX2bRpE1DYoLpx4wbHjh0ztmnVqhWLFi2iUaNGRcp865C824cRiog86Nzd3WnYsKFRZ/r5+VGvXj22bt3K+fPnAdi9ezenTp3i8OHDRrhfs2ZN6tSpw40bN4xjtW/fnnbt2hk9nm6tL21tbbGzs1Md+ohTKCUiIiIiJSY4OBiLxUJSUhLwa4PF1dWVevXqsX//fqAwNIqMjASgf//+AJQpU4bWrVtz9uxZ4uPjgcJhfAkJCaSmphrv0bBhQ8LCwhg8eDCrV69m5syZZGVl0aZNG2PoSfPmzVm/fj0vv/zyHWUsKCgo8lhD8kTkYebg4EBkZCRXr17lxIkT2Nra0rt3b1xcXOjevTt9+vRh3LhxDBo0iAEDBpCVlQXAkCFDmDZtGl5eXkWOd69FK0RAoZSIiIiIlKC6devi7OxMYmIiZrPZuJtua2tLREQE+fn5xiS5wcHB+Pn5GQEUYKzMZL2j3759e06ePGmEWVlZWXh5eTFp0iTq1atH3759+fLLL+nRowdLlixh5cqVQOHEuuXKlbtrGbXUuIg8aqpWrYqfnx/Lly8HCntLfffdd7Rs2RJbW1uGDh3KsGHD+Mc//kHTpk2N/W5dlMJK80LJb7Er6QKIiIiIyKPLupT46dOnOXfuHP7+/hQUFGBra0tgYCABAQGsWLGCRo0aERYWhrOzM/Hx8TRr1gyAoKAgfHx8jBCqXbt2LFy4kDFjxjB+/HgOHjxIdnY2YWFhzJo1i1KlSmFnp0tgEZHfUq5cOSpXrkxcXJzxXHh4OJMmTbpjW2udDQqg5H+nX2QRERERKVH169dn9uzZHD9+HH9/f+N5d3d3atWqxZ49e4DCBpGnpycHDx40tvH19SUwMJC1a9dy7do1ypYty6xZs/jXv/6Fh4cHbdu2xcHBAbPZTOnSpYFfh+OpB5SIyN2VLVuWBQsW4O7ufsdrt9ehqkvlj1CMKSIiIiIlqlGjRhQUFBhhk/VOe5kyZShbtiwFBQXk5eVRvnx5qlWrxs8//8yFCxeM/WvWrEmzZs2MeU18fX0ZPnw4/fr1w9fXt8gxobABpUaUiMhvu1sgBapD5c+lUEpERERESlRoaChVqlRh8+bNwK+Tnd+4cYNly5ZRv359I1QKDAzk+vXrxsToAM8++yyff/65EUBB4Qp5t09QLiIiIvcXk8W6nq2IiIiISAlZtmwZw4YNo2nTpgwdOpScnBwWLlzIyZMn+fTTTwkNDQUgMzMTR0dHHBwciuxvnVhX85mIiIg8OBRKiYiIiMh9YcmSJSxevJijR4+SmZlJ/fr1GTZsmDGp+a0sFovRo0pEREQeTAqlREREROS+cePGDVJTU4tMeC4iIiIPJ4VSIiIiInJf0ip5IiIiDzeFUiIiIiIiIiIiUuw0E6SIiIiIiIiIiBQ7hVIiIiIiIiIiIlLsFEqJiIiIiIiIiEixUyglIiIiIiIiIiLFTqGUiIiIiIiIiIgUO4VSIiIiIiIiIiJS7BRKiYiIiIiIiIhIsVMoJSIiIiIiIiIixU6hlIiIiIiIiIiIFDuFUiIiIiIiIiIiUuwUSomIiIiIiIiISLH7/wA0qxw7MXJWNAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "ci_untuned = dml_obj_untuned.confint()\n", + "ci_tuned = dml_obj_tuned.confint()\n", + "ci_untuned_pipeline = dml_obj_untuned_pipeline.confint()\n", + "ci_tuned_pipeline = dml_obj_tuned_pipeline.confint()\n", + "\n", + "# Create comparison dataframe\n", + "comparison_data = {\n", + " 'Model': ['Untuned', 'Tuned', 'Untuned Pipeline', 'Tuned Pipeline'],\n", + " 'theta': [dml_obj_untuned.coef[0], dml_obj_tuned.coef[0], dml_obj_untuned_pipeline.coef[0], dml_obj_tuned_pipeline.coef[0]],\n", + " 'se': [dml_obj_untuned.se[0], dml_obj_tuned.se[0], dml_obj_untuned_pipeline.se[0], dml_obj_tuned_pipeline.se[0]],\n", + " 'ci_lower': [ci_untuned.iloc[0, 0], ci_tuned.iloc[0, 0], \n", + " ci_untuned_pipeline.iloc[0, 0], ci_tuned_pipeline.iloc[0, 0]],\n", + " 'ci_upper': [ci_untuned.iloc[0, 1], ci_tuned.iloc[0, 1],\n", + " ci_untuned_pipeline.iloc[0, 1], ci_tuned_pipeline.iloc[0, 1]]\n", + "}\n", + "df_comparison = pd.DataFrame(comparison_data)\n", + "\n", + "print(f\"\\nTrue APO at treatment level {treatment_lvl}: {apos[int(treatment_lvl)]:.4f}\\n\")\n", + "print(df_comparison.to_string(index=False))\n", + "\n", + "# Create plot with all 4 models\n", + "plt.figure(figsize=(12, 6))\n", + "\n", + "for i in range(len(df_comparison)):\n", + " plt.errorbar(i, df_comparison.loc[i, 'theta'], \n", + " yerr=[[df_comparison.loc[i, 'theta'] - df_comparison.loc[i, 'ci_lower']], \n", + " [df_comparison.loc[i, 'ci_upper'] - df_comparison.loc[i, 'theta']]], \n", + " fmt='o', capsize=5, capthick=2, ecolor=palette[i], color=palette[i], \n", + " label=df_comparison.loc[i, 'Model'], markersize=10, zorder=2)\n", + "\n", + "plt.axhline(y=apos[int(treatment_lvl)], color=palette[4], linestyle='--', \n", + " linewidth=2, label='True APO', zorder=1)\n", + "\n", + "plt.title('Estimated APO Coefficients: Comparison Across All Models')\n", + "plt.ylabel('Coefficient Value')\n", + "plt.xticks(range(4), df_comparison['Model'], rotation=15, ha='right')\n", + "plt.legend()\n", + "plt.grid(True, alpha=0.3)\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "e0c8d51b", + "metadata": {}, + "source": [ + "### Detailed Tuning Result Analysis\n", + "\n", + "The `tune_ml_models()` method creates several [Optuna Studies](https://optuna.readthedocs.io/en/stable/reference/study.html), which can be inspected in detail via the returned results.\n", + "\n", + "The results are a list of dictionaries, which contain a corresponding `DMLOptunaResult` for each treatment variable on the `param_names` level, i.e. for each [Optuna Study](https://optuna.readthedocs.io/en/stable/reference/study.html) object a separate `DMLOptunaResult` is constructed." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "41cbd33f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "dict_keys(['ml_g_d_lvl0', 'ml_g_d_lvl1', 'ml_m'])\n" + ] + } + ], + "source": [ + "# Optuna results for the single treatment\n", + "print(tuning_results[0].keys())" + ] + }, + { + "cell_type": "markdown", + "id": "45acd97b", + "metadata": {}, + "source": [ + "In this example, we take a more detailed look in the tuning of `ml_m`" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "94e92730", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "================== DMLOptunaResult ==================\n", + "Learner name: ml_m\n", + "Params name: ml_m\n", + "Tuned: True\n", + "Best score: -0.5210215656500876\n", + "Scoring method: neg_log_loss\n", + "\n", + "------------------ Best parameters ------------------\n", + "{'stacking__final_estimator__C': 99.9143065172164,\n", + " 'stacking__lgbm__lambda_l1': 9.771463014326052,\n", + " 'stacking__lgbm__lambda_l2': 0.0013978426220758982,\n", + " 'stacking__lgbm__learning_rate': 0.0011563701553192595,\n", + " 'stacking__lgbm__min_child_samples': 30}\n", + "\n" + ] + } + ], + "source": [ + "print(tuning_results[0]['ml_m'])" + ] + }, + { + "cell_type": "markdown", + "id": "d85cfecd", + "metadata": {}, + "source": [ + "As we have access to the saved [Optuna Study](https://optuna.readthedocs.io/en/stable/reference/study.html) object, it is possible to access all trials and hyperparameter combinations" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "fe4c6e84", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
numbervaluedatetime_startdatetime_completedurationparams_stacking__final_estimator__Cparams_stacking__lgbm__lambda_l1params_stacking__lgbm__lambda_l2params_stacking__lgbm__learning_rateparams_stacking__lgbm__min_child_samplesstate
00-0.5285532025-11-26 15:53:57.4895112025-11-26 15:53:57.8905480 days 00:00:00.4010370.0373010.4985750.0017380.06070630COMPLETE
11-0.5234532025-11-26 15:53:57.8915472025-11-26 15:53:58.2674440 days 00:00:00.37589724.4511582.0075960.0138490.06545350COMPLETE
22-0.5289912025-11-26 15:53:58.2684062025-11-26 15:53:58.6004450 days 00:00:00.3320390.0149141.5402990.2426040.03085840COMPLETE
33-0.5219942025-11-26 15:53:58.6011582025-11-26 15:53:58.9296360 days 00:00:00.32847843.6463187.8671870.0048680.01072620COMPLETE
44-0.5280752025-11-26 15:53:58.9304972025-11-26 15:53:59.3431170 days 00:00:00.4126200.2354190.6366396.2499700.01459330COMPLETE
....................................
195195-0.5216442025-11-26 15:55:14.1298812025-11-26 15:55:14.4491720 days 00:00:00.31929197.7608687.0856970.0010970.00100830COMPLETE
196196-0.5219902025-11-26 15:55:14.4497152025-11-26 15:55:14.8101180 days 00:00:00.36040399.4473756.0057340.0013530.00118130COMPLETE
197197-0.5211182025-11-26 15:55:14.8109602025-11-26 15:55:15.1115260 days 00:00:00.30056699.6754517.5891840.0017790.00108030COMPLETE
198198-0.5253162025-11-26 15:55:15.1124152025-11-26 15:55:15.5432870 days 00:00:00.43087299.9824560.0167860.0011590.00118430COMPLETE
199199-0.5210442025-11-26 15:55:15.5440582025-11-26 15:55:15.8263660 days 00:00:00.28230880.4531899.9303570.0015760.00107330COMPLETE
\n", + "

200 rows × 11 columns

\n", + "
" + ], + "text/plain": [ + " number value datetime_start datetime_complete \\\n", + "0 0 -0.528553 2025-11-26 15:53:57.489511 2025-11-26 15:53:57.890548 \n", + "1 1 -0.523453 2025-11-26 15:53:57.891547 2025-11-26 15:53:58.267444 \n", + "2 2 -0.528991 2025-11-26 15:53:58.268406 2025-11-26 15:53:58.600445 \n", + "3 3 -0.521994 2025-11-26 15:53:58.601158 2025-11-26 15:53:58.929636 \n", + "4 4 -0.528075 2025-11-26 15:53:58.930497 2025-11-26 15:53:59.343117 \n", + ".. ... ... ... ... \n", + "195 195 -0.521644 2025-11-26 15:55:14.129881 2025-11-26 15:55:14.449172 \n", + "196 196 -0.521990 2025-11-26 15:55:14.449715 2025-11-26 15:55:14.810118 \n", + "197 197 -0.521118 2025-11-26 15:55:14.810960 2025-11-26 15:55:15.111526 \n", + "198 198 -0.525316 2025-11-26 15:55:15.112415 2025-11-26 15:55:15.543287 \n", + "199 199 -0.521044 2025-11-26 15:55:15.544058 2025-11-26 15:55:15.826366 \n", + "\n", + " duration params_stacking__final_estimator__C \\\n", + "0 0 days 00:00:00.401037 0.037301 \n", + "1 0 days 00:00:00.375897 24.451158 \n", + "2 0 days 00:00:00.332039 0.014914 \n", + "3 0 days 00:00:00.328478 43.646318 \n", + "4 0 days 00:00:00.412620 0.235419 \n", + ".. ... ... \n", + "195 0 days 00:00:00.319291 97.760868 \n", + "196 0 days 00:00:00.360403 99.447375 \n", + "197 0 days 00:00:00.300566 99.675451 \n", + "198 0 days 00:00:00.430872 99.982456 \n", + "199 0 days 00:00:00.282308 80.453189 \n", + "\n", + " params_stacking__lgbm__lambda_l1 params_stacking__lgbm__lambda_l2 \\\n", + "0 0.498575 0.001738 \n", + "1 2.007596 0.013849 \n", + "2 1.540299 0.242604 \n", + "3 7.867187 0.004868 \n", + "4 0.636639 6.249970 \n", + ".. ... ... \n", + "195 7.085697 0.001097 \n", + "196 6.005734 0.001353 \n", + "197 7.589184 0.001779 \n", + "198 0.016786 0.001159 \n", + "199 9.930357 0.001576 \n", + "\n", + " params_stacking__lgbm__learning_rate \\\n", + "0 0.060706 \n", + "1 0.065453 \n", + "2 0.030858 \n", + "3 0.010726 \n", + "4 0.014593 \n", + ".. ... \n", + "195 0.001008 \n", + "196 0.001181 \n", + "197 0.001080 \n", + "198 0.001184 \n", + "199 0.001073 \n", + "\n", + " params_stacking__lgbm__min_child_samples state \n", + "0 30 COMPLETE \n", + "1 50 COMPLETE \n", + "2 40 COMPLETE \n", + "3 20 COMPLETE \n", + "4 30 COMPLETE \n", + ".. ... ... \n", + "195 30 COMPLETE \n", + "196 30 COMPLETE \n", + "197 30 COMPLETE \n", + "198 30 COMPLETE \n", + "199 30 COMPLETE \n", + "\n", + "[200 rows x 11 columns]" + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "ml_m_study = tuning_results[0]['ml_m'].study\n", + "ml_m_study.trials_dataframe()" + ] + }, + { + "cell_type": "markdown", + "id": "047f6d0f", + "metadata": {}, + "source": [ + " Additionally, we can access all [Optuna visualization options](https://optuna.readthedocs.io/en/stable/reference/visualization/index.html)" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "7d35c2c0", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + " \n", + " " + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.plotly.v1+json": { + "config": { + "plotlyServerURL": "https://plot.ly" + }, + "data": [ + { + "mode": "markers", + "name": "Objective Value", + "type": "scatter", + "x": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99, + 100, + 101, + 102, + 103, + 104, + 105, + 106, + 107, + 108, + 109, + 110, + 111, + 112, + 113, + 114, + 115, + 116, + 117, + 118, + 119, + 120, + 121, + 122, + 123, + 124, + 125, + 126, + 127, + 128, + 129, + 130, + 131, + 132, + 133, + 134, + 135, + 136, + 137, + 138, + 139, + 140, + 141, + 142, + 143, + 144, + 145, + 146, + 147, + 148, + 149, + 150, + 151, + 152, + 153, + 154, + 155, + 156, + 157, + 158, + 159, + 160, + 161, + 162, + 163, + 164, + 165, + 166, + 167, + 168, + 169, + 170, + 171, + 172, + 173, + 174, + 175, + 176, + 177, + 178, + 179, + 180, + 181, + 182, + 183, + 184, + 185, + 186, + 187, + 188, + 189, + 190, + 191, + 192, + 193, + 194, + 195, + 196, + 197, + 198, + 199 + ], + "y": [ + -0.5285533485050931, + -0.5234530668488695, + -0.5289909251122416, + -0.5219937268019332, + -0.5280748025960298, + -0.5247511460603084, + -0.5233214137141129, + -0.5220375732865283, + -0.5247813848560081, + -0.525607889744429, + -0.521793167139381, + -0.52393449917208, + -0.5298415565612141, + -0.5220455269882187, + -0.5276474146430548, + -0.5211422010586271, + -0.5234576734117896, + -0.5250917115953776, + -0.5250459972012218, + -0.5282228598898202, + -0.5286311804687133, + -0.5236289553218089, + -0.5278280671703078, + -0.5214534923277736, + -0.52333337223033, + -0.5266448206550504, + -0.5287432505873271, + -0.5216159770110647, + -0.5231285226559483, + -0.5252325946633133, + -0.5246742907506108, + -0.5212396228009438, + -0.5215788071674636, + -0.5227829570708368, + -0.5238941790594186, + -0.5221730257565227, + -0.5211825412229547, + -0.529038145700957, + -0.5211873756002623, + -0.5255312743119468, + -0.5283849099629242, + -0.5212260637508805, + -0.5220110516231451, + -0.5243179511231577, + -0.5211868227445543, + -0.5239550045078102, + -0.5211038183747249, + -0.5245986747710893, + -0.5260468805287009, + -0.523581622748035, + -0.528929594330532, + -0.5211632204501271, + -0.5228882185475336, + -0.5243646096018659, + -0.5213680427544362, + -0.5214083051906739, + -0.5275319940416372, + -0.5290532240159138, + -0.5234568612226449, + -0.5215587395876351, + -0.5267350163473619, + -0.5220388977232652, + -0.521138524589729, + -0.5223215756901927, + -0.5220351872327159, + -0.5217445102620349, + -0.5216155030981015, + -0.5216599008112297, + -0.526674366479631, + -0.5265681628732654, + -0.5264127625718814, + -0.5214203050441409, + -0.5210925736706045, + -0.5221850923191912, + -0.5230542089481144, + -0.5226203638565847, + -0.5216952956820158, + -0.5211353530107499, + -0.5210862027236057, + -0.5221826773706424, + -0.5242012491840791, + -0.521049709425719, + -0.5210562002466338, + -0.5217631654403333, + -0.5226823461083122, + -0.5262873202598376, + -0.5222101770259606, + -0.524000195350726, + -0.5210408867514058, + -0.528209156527186, + -0.528938906838646, + -0.5211111322450857, + -0.5211257729697464, + -0.5212051574033424, + -0.5217351269684029, + -0.5210749594779168, + -0.5211064760917907, + -0.5255552939962225, + -0.5246290064701352, + -0.5219718941997751, + -0.521120623531767, + -0.5211157279849528, + -0.5245660174712901, + -0.5210837426239671, + -0.5243798463210467, + -0.5217587721342211, + -0.5210761611559112, + -0.5224955422583203, + -0.5228915290060185, + -0.5211089220703624, + -0.5215180088987477, + -0.521080876295949, + -0.5222140446779496, + -0.522531270439395, + -0.5214297184900827, + -0.5245020594282019, + -0.5211829173788083, + -0.5220518585052518, + -0.5210734007415885, + -0.5214676555545281, + -0.5211515144484844, + -0.5211050851709785, + -0.5210625470079859, + -0.5210604114901434, + -0.5234712251894624, + -0.521593032601775, + -0.525234558823649, + -0.5210804260125815, + -0.5210785424739072, + -0.525505371052851, + -0.5221445235887108, + -0.5210593879073832, + -0.5210737023669363, + -0.521083761853172, + -0.5215853405268017, + -0.5227768063008479, + -0.5211065140705554, + -0.5215101602536663, + -0.5288199238810922, + -0.5273214638606539, + -0.5257277842329631, + -0.5210695412795339, + -0.5210753071327717, + -0.5210703536387976, + -0.5218009417179299, + -0.5211164503065419, + -0.5226296090871653, + -0.5218783527595913, + -0.5210395304903896, + -0.5210700655263808, + -0.521145559792451, + -0.5217375844208683, + -0.5210457209511343, + -0.5210895780964436, + -0.5216443499814831, + -0.5228431433329648, + -0.5210832218278626, + -0.5274268727732995, + -0.5211216755913307, + -0.5219205044901309, + -0.524766867769203, + -0.5251195052420332, + -0.5210758044384366, + -0.521089097440652, + -0.521749878026454, + -0.5210958011908401, + -0.5274189971923838, + -0.521237938312581, + -0.5236942419543406, + -0.521062418359218, + -0.5210373652682448, + -0.5210387651008384, + -0.5210721524878251, + -0.5210426125489409, + -0.5210217893863571, + -0.5210228318027557, + -0.5210260738428435, + -0.5210216027586888, + -0.5210252358628062, + -0.5210313416300331, + -0.5210345712133734, + -0.5210229672517228, + -0.521783480697217, + -0.5210361216790312, + -0.5210267851893767, + -0.5210368512770732, + -0.521031021928863, + -0.5217844559373256, + -0.5210215656500876, + -0.5210279207137334, + -0.5218414039206085, + -0.5217026562163004, + -0.5210235606243971, + -0.5210222268319301, + -0.5210315702789767, + -0.5216440218826598, + -0.5219900384499969, + -0.521118206357774, + -0.525315872071635, + -0.5210436227523191 + ] + }, + { + "mode": "lines", + "name": "Best Value", + "type": "scatter", + "x": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99, + 100, + 101, + 102, + 103, + 104, + 105, + 106, + 107, + 108, + 109, + 110, + 111, + 112, + 113, + 114, + 115, + 116, + 117, + 118, + 119, + 120, + 121, + 122, + 123, + 124, + 125, + 126, + 127, + 128, + 129, + 130, + 131, + 132, + 133, + 134, + 135, + 136, + 137, + 138, + 139, + 140, + 141, + 142, + 143, + 144, + 145, + 146, + 147, + 148, + 149, + 150, + 151, + 152, + 153, + 154, + 155, + 156, + 157, + 158, + 159, + 160, + 161, + 162, + 163, + 164, + 165, + 166, + 167, + 168, + 169, + 170, + 171, + 172, + 173, + 174, + 175, + 176, + 177, + 178, + 179, + 180, + 181, + 182, + 183, + 184, + 185, + 186, + 187, + 188, + 189, + 190, + 191, + 192, + 193, + 194, + 195, + 196, + 197, + 198, + 199 + ], + "y": [ + -0.5285533485050931, + -0.5234530668488695, + -0.5234530668488695, + -0.5219937268019332, + -0.5219937268019332, + -0.5219937268019332, + -0.5219937268019332, + -0.5219937268019332, + -0.5219937268019332, + -0.5219937268019332, + -0.521793167139381, + -0.521793167139381, + -0.521793167139381, + -0.521793167139381, + -0.521793167139381, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211422010586271, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5211038183747249, + -0.5210925736706045, + -0.5210925736706045, + -0.5210925736706045, + -0.5210925736706045, + -0.5210925736706045, + -0.5210925736706045, + -0.5210862027236057, + -0.5210862027236057, + -0.5210862027236057, + -0.521049709425719, + -0.521049709425719, + -0.521049709425719, + -0.521049709425719, + -0.521049709425719, + -0.521049709425719, + -0.521049709425719, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210408867514058, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210395304903896, + -0.5210373652682448, + -0.5210373652682448, + -0.5210373652682448, + -0.5210373652682448, + -0.5210217893863571, + -0.5210217893863571, + -0.5210217893863571, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210216027586888, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876, + -0.5210215656500876 + ] + }, + { + "marker": { + "color": "#cccccc" + }, + "mode": "markers", + "name": "Infeasible Trial", + "showlegend": false, + "type": "scatter", + "x": [], + "y": [] + } + ], + "layout": { + "template": { + "data": { + "bar": [ + { + "error_x": { + "color": "#2a3f5f" + }, + "error_y": { + "color": "#2a3f5f" + }, + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + }, + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "bar" + } + ], + "barpolar": [ + { + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + }, + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "barpolar" + } + ], + "carpet": [ + { + "aaxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "baxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "type": "carpet" + } + ], + "choropleth": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "choropleth" + } + ], + "contour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "contour" + } + ], + "contourcarpet": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "contourcarpet" + } + ], + "heatmap": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmap" + } + ], + "heatmapgl": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmapgl" + } + ], + "histogram": [ + { + "marker": { + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "histogram" + } + ], + "histogram2d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2d" + } + ], + "histogram2dcontour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2dcontour" + } + ], + "mesh3d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "mesh3d" + } + ], + "parcoords": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "parcoords" + } + ], + "pie": [ + { + "automargin": true, + "type": "pie" + } + ], + "scatter": [ + { + "fillpattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + }, + "type": "scatter" + } + ], + "scatter3d": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatter3d" + } + ], + "scattercarpet": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattercarpet" + } + ], + "scattergeo": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergeo" + } + ], + "scattergl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergl" + } + ], + "scattermapbox": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattermapbox" + } + ], + "scatterpolar": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolar" + } + ], + "scatterpolargl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolargl" + } + ], + "scatterternary": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterternary" + } + ], + "surface": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "surface" + } + ], + "table": [ + { + "cells": { + "fill": { + "color": "#EBF0F8" + }, + "line": { + "color": "white" + } + }, + "header": { + "fill": { + "color": "#C8D4E3" + }, + "line": { + "color": "white" + } + }, + "type": "table" + } + ] + }, + "layout": { + "annotationdefaults": { + "arrowcolor": "#2a3f5f", + "arrowhead": 0, + "arrowwidth": 1 + }, + "autotypenumbers": "strict", + "coloraxis": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "colorscale": { + "diverging": [ + [ + 0, + "#8e0152" + ], + [ + 0.1, + "#c51b7d" + ], + [ + 0.2, + "#de77ae" + ], + [ + 0.3, + "#f1b6da" + ], + [ + 0.4, + "#fde0ef" + ], + [ + 0.5, + "#f7f7f7" + ], + [ + 0.6, + "#e6f5d0" + ], + [ + 0.7, + "#b8e186" + ], + [ + 0.8, + "#7fbc41" + ], + [ + 0.9, + "#4d9221" + ], + [ + 1, + "#276419" + ] + ], + "sequential": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "sequentialminus": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ] + }, + "colorway": [ + "#636efa", + "#EF553B", + "#00cc96", + "#ab63fa", + "#FFA15A", + "#19d3f3", + "#FF6692", + "#B6E880", + "#FF97FF", + "#FECB52" + ], + "font": { + "color": "#2a3f5f" + }, + "geo": { + "bgcolor": "white", + "lakecolor": "white", + "landcolor": "#E5ECF6", + "showlakes": true, + "showland": true, + "subunitcolor": "white" + }, + "hoverlabel": { + "align": "left" + }, + "hovermode": "closest", + "mapbox": { + "style": "light" + }, + "paper_bgcolor": "white", + "plot_bgcolor": "#E5ECF6", + "polar": { + "angularaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "radialaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "scene": { + "xaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "yaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "zaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + } + }, + "shapedefaults": { + "line": { + "color": "#2a3f5f" + } + }, + "ternary": { + "aaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "baxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "caxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "title": { + "x": 0.05 + }, + "xaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + }, + "yaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + } + } + }, + "title": { + "text": "Optimization History Plot" + }, + "xaxis": { + "title": { + "text": "Trial" + } + }, + "yaxis": { + "title": { + "text": "Objective Value" + } + } + } + }, + "text/html": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig = optuna.visualization.plot_optimization_history(ml_m_study)\n", + "show(fig)" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "f97a88d6", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.plotly.v1+json": { + "config": { + "plotlyServerURL": "https://plot.ly" + }, + "data": [ + { + "dimensions": [ + { + "label": "Objective Value", + "range": [ + -0.5298415565612141, + -0.5210215656500876 + ], + "values": [ + -0.5285533485050931, + -0.5234530668488695, + -0.5289909251122416, + -0.5219937268019332, + -0.5280748025960298, + -0.5247511460603084, + -0.5233214137141129, + -0.5220375732865283, + -0.5247813848560081, + -0.525607889744429, + -0.521793167139381, + -0.52393449917208, + -0.5298415565612141, + -0.5220455269882187, + -0.5276474146430548, + -0.5211422010586271, + -0.5234576734117896, + -0.5250917115953776, + -0.5250459972012218, + -0.5282228598898202, + -0.5286311804687133, + -0.5236289553218089, + -0.5278280671703078, + -0.5214534923277736, + -0.52333337223033, + -0.5266448206550504, + -0.5287432505873271, + -0.5216159770110647, + -0.5231285226559483, + -0.5252325946633133, + -0.5246742907506108, + -0.5212396228009438, + -0.5215788071674636, + -0.5227829570708368, + -0.5238941790594186, + -0.5221730257565227, + -0.5211825412229547, + -0.529038145700957, + -0.5211873756002623, + -0.5255312743119468, + -0.5283849099629242, + -0.5212260637508805, + -0.5220110516231451, + -0.5243179511231577, + -0.5211868227445543, + -0.5239550045078102, + -0.5211038183747249, + -0.5245986747710893, + -0.5260468805287009, + -0.523581622748035, + -0.528929594330532, + -0.5211632204501271, + -0.5228882185475336, + -0.5243646096018659, + -0.5213680427544362, + -0.5214083051906739, + -0.5275319940416372, + -0.5290532240159138, + -0.5234568612226449, + -0.5215587395876351, + -0.5267350163473619, + -0.5220388977232652, + -0.521138524589729, + -0.5223215756901927, + -0.5220351872327159, + -0.5217445102620349, + -0.5216155030981015, + -0.5216599008112297, + -0.526674366479631, + -0.5265681628732654, + -0.5264127625718814, + -0.5214203050441409, + -0.5210925736706045, + -0.5221850923191912, + -0.5230542089481144, + -0.5226203638565847, + -0.5216952956820158, + -0.5211353530107499, + -0.5210862027236057, + -0.5221826773706424, + -0.5242012491840791, + -0.521049709425719, + -0.5210562002466338, + -0.5217631654403333, + -0.5226823461083122, + -0.5262873202598376, + -0.5222101770259606, + -0.524000195350726, + -0.5210408867514058, + -0.528209156527186, + -0.528938906838646, + -0.5211111322450857, + -0.5211257729697464, + -0.5212051574033424, + -0.5217351269684029, + -0.5210749594779168, + -0.5211064760917907, + -0.5255552939962225, + -0.5246290064701352, + -0.5219718941997751, + -0.521120623531767, + -0.5211157279849528, + -0.5245660174712901, + -0.5210837426239671, + -0.5243798463210467, + -0.5217587721342211, + -0.5210761611559112, + -0.5224955422583203, + -0.5228915290060185, + -0.5211089220703624, + -0.5215180088987477, + -0.521080876295949, + -0.5222140446779496, + -0.522531270439395, + -0.5214297184900827, + -0.5245020594282019, + -0.5211829173788083, + -0.5220518585052518, + -0.5210734007415885, + -0.5214676555545281, + -0.5211515144484844, + -0.5211050851709785, + -0.5210625470079859, + -0.5210604114901434, + -0.5234712251894624, + -0.521593032601775, + -0.525234558823649, + -0.5210804260125815, + -0.5210785424739072, + -0.525505371052851, + -0.5221445235887108, + -0.5210593879073832, + -0.5210737023669363, + -0.521083761853172, + -0.5215853405268017, + -0.5227768063008479, + -0.5211065140705554, + -0.5215101602536663, + -0.5288199238810922, + -0.5273214638606539, + -0.5257277842329631, + -0.5210695412795339, + -0.5210753071327717, + -0.5210703536387976, + -0.5218009417179299, + -0.5211164503065419, + -0.5226296090871653, + -0.5218783527595913, + -0.5210395304903896, + -0.5210700655263808, + -0.521145559792451, + -0.5217375844208683, + -0.5210457209511343, + -0.5210895780964436, + -0.5216443499814831, + -0.5228431433329648, + -0.5210832218278626, + -0.5274268727732995, + -0.5211216755913307, + -0.5219205044901309, + -0.524766867769203, + -0.5251195052420332, + -0.5210758044384366, + -0.521089097440652, + -0.521749878026454, + -0.5210958011908401, + -0.5274189971923838, + -0.521237938312581, + -0.5236942419543406, + -0.521062418359218, + -0.5210373652682448, + -0.5210387651008384, + -0.5210721524878251, + -0.5210426125489409, + -0.5210217893863571, + -0.5210228318027557, + -0.5210260738428435, + -0.5210216027586888, + -0.5210252358628062, + -0.5210313416300331, + -0.5210345712133734, + -0.5210229672517228, + -0.521783480697217, + -0.5210361216790312, + -0.5210267851893767, + -0.5210368512770732, + -0.521031021928863, + -0.5217844559373256, + -0.5210215656500876, + -0.5210279207137334, + -0.5218414039206085, + -0.5217026562163004, + -0.5210235606243971, + -0.5210222268319301, + -0.5210315702789767, + -0.5216440218826598, + -0.5219900384499969, + -0.521118206357774, + -0.525315872071635, + -0.5210436227523191 + ] + }, + { + "label": "stacking__final_e...", + "range": [ + -1.9540203809464836, + 1.9999238004545299 + ], + "ticktext": [ + "0.0111", + "0.1", + "1", + "10", + "100" + ], + "tickvals": [ + -1.9540203809464836, + -1, + 0, + 1, + 1.9999238004545299 + ], + "values": [ + -1.428275465118931, + 1.388299436654305, + -1.8264043471386533, + 1.639947616464307, + -0.6281578547065815, + 1.4906898290879995, + 0.4474621824120747, + 1.0542904185214255, + 0.08058033402010757, + -0.4246276359156199, + 1.9063945668147069, + 1.919542172969506, + 1.9453107254358493, + 0.7429808325938154, + 1.2993697757471268, + 1.975116095021122, + 0.874398446498588, + 0.3962592870269903, + 1.982761885092244, + -0.17283640584760057, + -1.1750172036623419, + 1.6336306104552636, + 1.6657904979301488, + 1.150869566786412, + 1.1730833138617016, + 0.7776835134987934, + 1.7309077740391476, + 1.1150963376519347, + 0.44142040876128763, + 1.0639989727648547, + 0.1378664219553568, + 1.3802824586126912, + 1.315291453817139, + 1.442714131896899, + 1.3688466604750704, + 0.8498794636988568, + 1.590185544368145, + -1.9540203809464836, + 1.6069658826034035, + 1.723202949187454, + -0.9869491592799305, + 1.5188661783384936, + 1.5721654653085386, + 1.7881944190525147, + 1.4806351227779433, + 1.5024982159894589, + 1.7837973719507216, + 1.830384894361559, + 1.9910069094939802, + 0.6079770254675666, + -1.5944719702568473, + 1.58500199464666, + 1.7620453294596177, + 0.954925469221835, + 1.2478858643841182, + 1.6196006137578314, + 1.855095958745426, + 1.9928915062053474, + 1.5150984232023856, + 1.2702017576875693, + -0.3902442563181051, + 1.5551095789784672, + 1.6674481587550065, + 1.666345222144125, + 1.8432936170511212, + 1.4148376790141004, + 1.0205033507738808, + 1.2066226374478297, + 1.8878364423093303, + 1.693759777326588, + 1.4033319428437447, + 1.5540851380870202, + 1.7225124199535287, + 1.7820547335889627, + 1.9173540625884666, + 1.6418875734168812, + 1.332327573708942, + 1.7341368307378862, + 1.756884316964199, + 1.7236042692373699, + 0.18745942417761682, + 1.9176434544677856, + 1.8878791351504418, + 1.9108396578072668, + 1.9875538577152612, + 1.8169277576195628, + 1.8172753048434211, + 1.705240444384836, + 1.9353986825665341, + -0.8808595796982928, + 1.88056494654855, + 1.757695190935565, + 1.765686959891216, + 1.7650994262336983, + 1.444088780972229, + 1.9147113251525618, + 1.922745539065941, + 1.9140065069777032, + 1.930482544616497, + 1.8109517253855694, + 1.98340416904632, + 1.9742814243242441, + 1.9990795318583228, + 1.8842820159068137, + 1.8796941107436227, + 1.6212789554282272, + 1.8481445881392662, + 1.8422369161817185, + 1.4907102881493457, + 1.6613049577926204, + 1.6674299757737698, + 1.765160656885965, + 1.8795299176955127, + 1.5950326372664467, + 1.6866524795205948, + 1.8154218515124858, + 1.5063508370029355, + 1.9139170476099039, + 1.737894318069052, + 1.746263323068957, + 1.5640903637166241, + 1.653734118806375, + 1.8516024360810166, + 1.8302343663125051, + 1.8850764673261626, + 1.8007565078098162, + -0.02718630771524897, + 1.9984048079378356, + 1.9842537746454882, + 1.9783152968570559, + 1.9984511560749614, + 1.8638341928484348, + 1.8479606998586637, + 1.8537684860174433, + 1.846609295944224, + 1.9204156804880734, + 1.8392007851366352, + 1.7272968530021728, + -1.399525391055773, + 1.9127003660387882, + 1.997255559759418, + 1.8421854753944065, + 1.794876932500625, + 1.7646112925086161, + 1.612273919343768, + 1.8056640483364088, + 0.6154113574802901, + 1.699543896514574, + 1.9270049336832336, + 1.8114112406495535, + 1.558646240863841, + 1.8276750943361868, + 1.9150708710211097, + 1.727548299803447, + 1.9113752451738186, + 1.8014002088190144, + 1.6872173764273795, + 1.8824473965872317, + 1.6209436215289257, + 1.771610526718376, + 1.9220046070112582, + 1.8533811366269901, + 1.9403337358196564, + 1.7813843093344661, + 1.9170833516902281, + 1.6939048229307039, + -0.5413193435098012, + 1.8471654529457344, + 0.29961778519025656, + 1.9278497063266886, + 1.9379623961185335, + 1.9325568094616439, + 1.776340183332222, + 1.9305900859829979, + 1.9985508551297635, + 1.9919689028702752, + 1.9706999258010027, + 1.9963844318401716, + 1.9864243505418593, + 1.979002094640689, + 1.9735954202048103, + 1.9942458166613966, + 1.987569201404533, + 1.9770932202542144, + 1.9988384156685715, + 1.973768636628579, + 1.9887849394150052, + 1.9887378161422216, + 1.9996276783824456, + 1.9959614922074524, + 1.997542013763729, + 1.974799737751547, + 1.9891500642563442, + 1.996749478298116, + 1.9850683519720822, + 1.9901650469749896, + 1.9975933246128286, + 1.9985882098283994, + 1.9999238004545299, + 1.9055432624369466 + ] + }, + { + "label": "stacking__lgbm__l...", + "range": [ + -2.9776214437064583, + 0.9999101193527662 + ], + "ticktext": [ + "0.00105", + "0.01", + "0.1", + "1", + "10" + ], + "tickvals": [ + -2.9776214437064583, + -2, + -1, + 0, + 0.9999101193527662 + ], + "values": [ + -0.30226931559461456, + 0.30267623446783837, + 0.18760506769272564, + 0.8958194992926735, + -0.19610658575621737, + -0.36432975657062994, + 0.3307034222532508, + -0.43623689917528924, + -1.3853675357984627, + -0.9995703043892183, + 0.9835440203115525, + 0.8124971363141833, + -2.5456599202966643, + 0.9784232905751841, + -2.347564403003742, + 0.9677413312402854, + -0.9454255118515486, + -1.9175193390434886, + 0.4719109445428733, + -1.6401181844422321, + 0.5729038615770077, + 0.8364474863819551, + -0.023632738221451084, + 0.9683264102087153, + 0.5616631728598944, + -0.6330769725718075, + 0.06277736893496247, + 0.6848446113414313, + 0.6358731620391738, + -0.7544901166789716, + -0.14885725997624083, + 0.957816319010129, + 0.6839223426917744, + 0.30199994264331803, + 0.35303524308855544, + 0.7172704527849758, + 0.9709617544027448, + 0.13636084048675554, + 0.961442439561776, + 0.4003857810367042, + 0.2136860620875969, + 0.9498150767697143, + 0.8414153535998146, + 0.478067046082465, + 0.9855598894661758, + 0.7040617458093046, + 0.9944087447895524, + 0.7801934155074667, + -0.14294094641017963, + -1.3034997773731194, + 0.503642174313158, + 0.9828261831878767, + -2.9776214437064583, + 0.8205011493599196, + 0.9782498951955889, + 0.8037253146517674, + 0.5095341200290852, + 0.2716902829247605, + 0.6004282819037452, + 0.876256783721201, + 0.977830651026532, + 0.8222390186074271, + 0.9817358488100751, + 0.6038744502728678, + 0.7292042096863166, + 0.44270642425625417, + 0.979379736383386, + 0.8620144141153261, + 0.6064045094230678, + -2.211677900795369, + -0.40979541321077073, + 0.8941874172635288, + 0.9984068774469372, + 0.7102542434741106, + 0.7366530795010114, + 0.8946406988102966, + 0.38385496358753257, + 0.9861296192644955, + 0.9999007495903237, + 0.6435243928014074, + 0.7777098988304306, + 0.9995607887342942, + 0.999786068318505, + 0.8596185841903166, + 0.7329770512226806, + 0.5392045515123555, + 0.841481992097953, + -0.5471272732811878, + 0.9916950227702455, + 0.6593083196708868, + -1.3534421776043797, + 0.8971811639596519, + 0.9212142640725165, + 0.8850541301935306, + 0.7994163784163542, + 0.9055711342855066, + 0.8990517031905784, + 0.5494484075645342, + -1.103800646974293, + 0.7683304478135322, + 0.8944551681731564, + 0.8970729267691534, + 0.6662765178057519, + 0.8935841058885534, + -1.6522633232714876, + 0.7915465747481626, + 0.8978570952590489, + 0.6537082528135871, + 0.44770054489015765, + 0.9974545678020077, + 0.7833238564359768, + 0.9089140421058703, + 0.71774122897892, + 0.9987310442284573, + 0.8419466855564851, + 0.5856492833483625, + 0.9047783657471652, + 0.7484145320678612, + 0.9286206133144338, + 0.8278954082876587, + 0.9141025139364752, + 0.9910412889587855, + 0.9976567005954509, + 0.991741183523153, + 0.6808339289625083, + 0.8099041470057483, + 0.741850640446547, + 0.9243981159096792, + 0.9197532553382812, + 0.5956078592600101, + 0.8528142321630104, + 0.926101196467322, + 0.9138331420461707, + 0.9007630188873036, + 0.7864654828125519, + 0.7128465639706683, + 0.9173965281764905, + 0.8345525505877229, + 0.641158835591117, + -0.28423483575521835, + 0.5207740101773077, + 0.9174564343568251, + 0.9162682520853603, + 0.9332271272753574, + 0.8089984171543908, + 0.9346999386962157, + 0.7457059195962578, + 0.8474938080991243, + 0.9932907175048181, + 0.9426957721545249, + 0.9976191127350238, + 0.8286677023911764, + 0.9994864364446497, + 0.9990692986777945, + 0.7804389287339608, + 0.6941385001427273, + 0.9285656115036356, + 0.8472354026478142, + 0.9209995053574964, + 0.7414335767107454, + -0.03671573266616603, + -0.8303130670801795, + 0.929766983048848, + 0.9837331055469595, + 0.8425183471854116, + 0.998087900392848, + -2.680205881992988, + 0.8760273064634068, + 0.7802831000857429, + 0.9175528239011297, + 0.9953384760108726, + 0.9965063724027852, + 0.9954326379165434, + 0.9985453424042402, + 0.9800207654803109, + 0.9807534692148185, + 0.9879500893821984, + 0.9910761343802041, + 0.992875067978677, + 0.9967758617198897, + 0.9988726413092274, + 0.9917219652809726, + 0.8284275722016563, + 0.999735552630182, + 0.9947343431555035, + 0.9999101193527662, + 0.997623764220608, + 0.8209153605740726, + 0.9899595925295096, + 0.9955008449363947, + 0.8474174404397266, + 0.8545249205207118, + 0.9834935853623056, + 0.9882526698949397, + 0.9987057937862694, + 0.8503825667926431, + 0.7785660993674642, + 0.8801950558566273, + -1.7750586552775391, + 0.9969648695187711 + ] + }, + { + "label": "stacking__lgbm__l...", + "range": [ + -2.9917255946519212, + 0.9756382215167431 + ], + "ticktext": [ + "0.00102", + "0.01", + "0.1", + "1", + "9.45" + ], + "tickvals": [ + -2.9917255946519212, + -2, + -1, + 0, + 0.9756382215167431 + ], + "values": [ + -2.7599792316937015, + -1.858585637163399, + -0.6151019925337775, + -2.3126727072618367, + 0.7958779001875315, + 0.6740766622276123, + -1.2950287938358243, + -0.07281279981899948, + -1.512021915744928, + -0.5508645711255508, + -2.94918803951068, + -2.7580354386304604, + -2.1633622744216034, + -2.893646937972425, + -2.2844470237924046, + -2.2660159268363023, + -1.7677123579398382, + -2.4392034615712967, + -2.978167435361675, + -1.035795378870038, + -1.94187026281305, + -2.386451394141801, + -2.4595287138442896, + -2.6635318892187883, + -2.6844490300709323, + -2.9917255946519212, + -2.025108608899019, + -1.6756302443440474, + -1.5658346620516757, + -1.0678921134546935, + -1.6554812569927349, + -2.5171859410460056, + -2.608246824993184, + -2.6187150637715604, + -2.0863599974165052, + -2.6403143963751186, + -2.511720084023074, + -2.250573919836963, + -0.12697864121121227, + 0.44398416863157153, + -0.09928474519492766, + -0.5266755454196996, + -0.4626926261246023, + -0.7824688405335575, + -0.10629558150399016, + -0.27814651304462007, + 0.3120803888152767, + 0.3019674519306464, + 0.2882242463217561, + 0.8444392512333262, + 0.11270294857096226, + -0.2843212742502914, + 0.5697877948295691, + -0.11339495256084821, + -0.7637037280109892, + 0.9756382215167431, + -0.28877430672639537, + -1.28466418225098, + 0.084869718470547, + -0.318849134339815, + 0.22848648703898505, + -0.5272615205098761, + -0.6966107072216029, + -0.8714595047077642, + 0.014569112018173527, + -0.1902958196521778, + -0.6608502452522572, + -2.823948313697063, + -1.1802553001834792, + -0.4073333817541456, + 0.46948704487379767, + -0.4340184719072161, + 0.029010342517191947, + -0.029131041234814286, + -0.18773784245150296, + 0.17767736338061432, + 0.4463662399495661, + -0.690228980664112, + -0.9790614934727326, + -0.9027531499804428, + -0.6854272622385275, + -1.1201031207098342, + -1.1004179596808017, + -1.3599491508908692, + -0.9872276370712887, + -1.1386633120745784, + -1.3224721307926017, + -1.407966755688873, + -1.860842541628295, + -1.7654970569968178, + -1.442539055600756, + -1.1997176338701898, + -1.2083001546782495, + -1.237214044858792, + -1.0792677382639326, + -0.9957049274375819, + -1.553680786472974, + -1.4921243940801396, + -1.898970519579208, + -0.9775522521105969, + -1.598862568558039, + -1.5861193631751043, + -1.6994749107100866, + -1.6111410861049047, + -2.0240121393202295, + -1.7987252849761088, + -1.536668281083171, + -1.108529062304656, + -1.5317118150787616, + -0.8938086462355058, + -0.8685730405599688, + -1.2728174273430732, + -1.4275997017776425, + -1.668502911872225, + -1.299904665752178, + -0.9455374682450113, + -0.8286458951844307, + -1.0293381232191723, + -1.962977642016504, + -2.1838911196560504, + -2.01199664957726, + -1.8757753415810228, + -1.9029744021656976, + -1.7988535571080284, + -2.13254647328034, + -1.9496030137674636, + -1.8188779070934074, + -1.946074096567349, + -1.746865218315063, + -1.7486078656590478, + -1.985208090644419, + -2.345951298888164, + -2.3605346813046504, + -2.380400519472774, + -2.2354535973236294, + -2.0823597490447043, + -1.871723386515737, + -2.3103448563942015, + -1.7593837038956632, + -2.4822667789408612, + -1.620410977474957, + -2.3836827956510174, + -2.1023864381082844, + -2.352960522480358, + -2.4004161994069237, + -2.595162483415522, + -2.141593226290036, + -2.233067319972739, + -2.0766128936369546, + -2.320328788916386, + -2.3743941100380273, + -2.0592935123788796, + -2.3200548769200657, + -2.550432232562551, + -2.1971090477155912, + -2.3109556360512604, + -2.7004340068139476, + -2.2794431486907656, + -2.4125498606518407, + -2.1158942797886775, + -2.3465316266854166, + -2.4447038397207272, + -1.8883589847745812, + -2.0542536709991444, + -2.1701337871033815, + -1.9295606343646239, + -2.5396344359781557, + -1.880697884623651, + -1.9987663619187797, + -2.286574782942853, + -2.272958689079292, + -2.2978031869835633, + -2.269723795777713, + -2.2922656743892413, + -2.2528251887142092, + -2.2706775366353957, + -2.2678033722976685, + -2.4888382206888164, + -2.193713107143374, + -2.2329105622264573, + -2.2081165128615186, + -2.225450476139785, + -2.218892811563502, + -2.4928357422148433, + -2.4684526970825456, + -2.4990022009081896, + -2.762738159289246, + -2.6932875596436516, + -2.854541721445424, + -2.888170933645917, + -2.8866941633264096, + -2.8065778931992247, + -2.9334543656766807, + -2.864831261895104, + -2.9228289838430737, + -2.959786836279779, + -2.868666991622764, + -2.749782102911928, + -2.935806856924173, + -2.802510355967725 + ] + }, + { + "label": "stacking__lgbm__l...", + "range": [ + -2.9996278167102197, + -1.0489166031705979 + ], + "ticktext": [ + "0.001", + "0.01", + "0.0893" + ], + "tickvals": [ + -2.9996278167102197, + -2, + -1.0489166031705979 + ], + "values": [ + -1.2167681972926578, + -1.1840729644853332, + -1.5106252342079907, + -1.9695808270623587, + -1.8358452114914205, + -1.4136466191682504, + -2.497218679847063, + -2.944403265812129, + -1.2238987364395104, + -1.188642769476512, + -2.1976176380801977, + -2.151737541474684, + -2.17903075661944, + -1.87990994493694, + -2.461578936493054, + -2.4603168832403854, + -2.735839554596071, + -2.4217728975555453, + -2.708450431518637, + -1.7409160269461368, + -2.254127954814008, + -2.0083119152483224, + -2.0227796268904505, + -2.347521434602191, + -2.6520121763338635, + -2.3091837822854475, + -2.3855336923883126, + -2.9356604705175555, + -2.957782606643845, + -2.5721068110050584, + -2.7975899913535223, + -2.8485888068931744, + -2.8478594945505065, + -2.846015419857415, + -2.628402978948562, + -2.5840003654846466, + -2.8597025324349263, + -2.3508249056125567, + -2.5326324738841257, + -2.5389349659431035, + -2.991090918649589, + -2.743656325225274, + -2.753599294105137, + -2.709532148707997, + -2.5043971515094223, + -2.510368405326435, + -2.6531012611637195, + -2.1253732934261484, + -2.455591762425454, + -1.5545594210489477, + -2.657178285431277, + -2.8845200916947764, + -2.8919866762132975, + -1.0489166031705979, + -2.8021619260232393, + -2.608557419688905, + -2.492470309243549, + -2.252406033984395, + -2.6962667135377947, + -2.425294705061753, + -2.0881410798388798, + -2.767644410959755, + -2.9179416798514293, + -2.87376550050674, + -2.916126805838479, + -2.991547435705009, + -2.6518441356988545, + -2.8043107497564144, + -2.5525719303898797, + -2.264012910965086, + -1.9232056450861388, + -2.7122423520716423, + -2.927410398214455, + -2.925756933111645, + -2.8318669001672347, + -1.7503939033965932, + -2.948382475580811, + -2.670006315029245, + -2.8812135641608356, + -2.9993023946081103, + -2.78709328990403, + -2.8779468908808576, + -2.8715129116298823, + -2.905409465449753, + -2.8743592323590006, + -2.6737320252006054, + -2.750507920109499, + -2.9528544303292525, + -2.826097975288204, + -2.8202803539862784, + -2.607410834654463, + -2.886256818938404, + -2.7394977356133796, + -2.743138509228953, + -2.8405600739251122, + -2.9482003402016965, + -2.804993449719542, + -2.78543404966465, + -2.961548567071792, + -2.8738035109405766, + -2.8272992472987317, + -2.8220287811432736, + -2.8251298499161286, + -2.9497324033412657, + -2.9609985051592296, + -2.917379733474881, + -2.9974698865968508, + -2.986304483616805, + -2.88621183624595, + -2.937520305804515, + -2.9602341430175136, + -2.9231114582187447, + -2.928681409050559, + -1.3733495337554085, + -2.9919213283273067, + -2.7756457263902155, + -2.8577814945649878, + -2.929219963762523, + -2.9996278167102197, + -2.993679029505583, + -2.8935910283012416, + -2.9207061314332923, + -2.85349968329173, + -2.850178559897722, + -2.8628201491228293, + -2.958438582283406, + -2.7137224159163496, + -2.8498316126978676, + -2.855823637408871, + -2.7783412104158836, + -2.8468185039399527, + -2.8998049988222236, + -2.8931756722479167, + -2.9078331700701927, + -2.992603981856915, + -2.8590907803757184, + -2.798122931127673, + -2.959889757180055, + -2.8951179984594146, + -2.767423873598842, + -2.83434378806186, + -2.9060908064739466, + -2.902696701808303, + -2.8942441402065486, + -2.871772782806638, + -2.7145010071708118, + -2.8045235973016327, + -2.8405739753008663, + -2.8972352184914536, + -2.9984891052591105, + -2.9920126607541335, + -2.893927579705559, + -2.9533214681896793, + -2.9481119032781518, + -2.9988629762836356, + -2.9068133233003013, + -2.9588023893526816, + -1.6014179276389615, + -2.9206940906559593, + -2.959708756237333, + -2.8995988395464996, + -2.873417407757244, + -2.8000904927841113, + -2.8189953970329844, + -2.9321274548605696, + -2.9966385478490487, + -2.7513302566761255, + -2.7996631238890433, + -2.8965623147809985, + -2.9558539793624385, + -2.939399192827881, + -2.9547774453091393, + -2.941833696669759, + -2.9407084134148316, + -2.957153136658949, + -2.95569786262459, + -2.9432617164343835, + -2.955901561688776, + -2.9556787271334333, + -2.951891114922058, + -2.956241501684827, + -2.9521371175951137, + -2.9553986589324412, + -2.939575611050511, + -2.9367450363101506, + -2.939854930521867, + -2.95007805285527, + -2.9573055790866576, + -2.936903125534256, + -2.939771856318346, + -2.9297360996037556, + -2.9607872099633554, + -2.9357847146310703, + -2.9292860914555114, + -2.975295535682108, + -2.996328586409733, + -2.927885183628377, + -2.9666655833665336, + -2.926735401648633, + -2.9695324400037975 + ] + }, + { + "label": "stacking__lgbm__m...", + "range": [ + 20, + 50 + ], + "values": [ + 30, + 50, + 40, + 20, + 30, + 30, + 50, + 40, + 40, + 50, + 20, + 20, + 20, + 20, + 20, + 30, + 30, + 30, + 20, + 30, + 40, + 20, + 20, + 20, + 30, + 20, + 30, + 20, + 20, + 30, + 30, + 20, + 20, + 20, + 20, + 20, + 40, + 40, + 40, + 40, + 50, + 40, + 40, + 40, + 40, + 40, + 50, + 50, + 50, + 50, + 40, + 40, + 40, + 40, + 40, + 50, + 40, + 40, + 30, + 50, + 40, + 40, + 40, + 40, + 30, + 40, + 40, + 30, + 40, + 50, + 40, + 40, + 40, + 40, + 40, + 30, + 40, + 40, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30, + 30 + ] + } + ], + "labelangle": 30, + "labelside": "bottom", + "line": { + "color": [ + -0.5285533485050931, + -0.5234530668488695, + -0.5289909251122416, + -0.5219937268019332, + -0.5280748025960298, + -0.5247511460603084, + -0.5233214137141129, + -0.5220375732865283, + -0.5247813848560081, + -0.525607889744429, + -0.521793167139381, + -0.52393449917208, + -0.5298415565612141, + -0.5220455269882187, + -0.5276474146430548, + -0.5211422010586271, + -0.5234576734117896, + -0.5250917115953776, + -0.5250459972012218, + -0.5282228598898202, + -0.5286311804687133, + -0.5236289553218089, + -0.5278280671703078, + -0.5214534923277736, + -0.52333337223033, + -0.5266448206550504, + -0.5287432505873271, + -0.5216159770110647, + -0.5231285226559483, + -0.5252325946633133, + -0.5246742907506108, + -0.5212396228009438, + -0.5215788071674636, + -0.5227829570708368, + -0.5238941790594186, + -0.5221730257565227, + -0.5211825412229547, + -0.529038145700957, + -0.5211873756002623, + -0.5255312743119468, + -0.5283849099629242, + -0.5212260637508805, + -0.5220110516231451, + -0.5243179511231577, + -0.5211868227445543, + -0.5239550045078102, + -0.5211038183747249, + -0.5245986747710893, + -0.5260468805287009, + -0.523581622748035, + -0.528929594330532, + -0.5211632204501271, + -0.5228882185475336, + -0.5243646096018659, + -0.5213680427544362, + -0.5214083051906739, + -0.5275319940416372, + -0.5290532240159138, + -0.5234568612226449, + -0.5215587395876351, + -0.5267350163473619, + -0.5220388977232652, + -0.521138524589729, + -0.5223215756901927, + -0.5220351872327159, + -0.5217445102620349, + -0.5216155030981015, + -0.5216599008112297, + -0.526674366479631, + -0.5265681628732654, + -0.5264127625718814, + -0.5214203050441409, + -0.5210925736706045, + -0.5221850923191912, + -0.5230542089481144, + -0.5226203638565847, + -0.5216952956820158, + -0.5211353530107499, + -0.5210862027236057, + -0.5221826773706424, + -0.5242012491840791, + -0.521049709425719, + -0.5210562002466338, + -0.5217631654403333, + -0.5226823461083122, + -0.5262873202598376, + -0.5222101770259606, + -0.524000195350726, + -0.5210408867514058, + -0.528209156527186, + -0.528938906838646, + -0.5211111322450857, + -0.5211257729697464, + -0.5212051574033424, + -0.5217351269684029, + -0.5210749594779168, + -0.5211064760917907, + -0.5255552939962225, + -0.5246290064701352, + -0.5219718941997751, + -0.521120623531767, + -0.5211157279849528, + -0.5245660174712901, + -0.5210837426239671, + -0.5243798463210467, + -0.5217587721342211, + -0.5210761611559112, + -0.5224955422583203, + -0.5228915290060185, + -0.5211089220703624, + -0.5215180088987477, + -0.521080876295949, + -0.5222140446779496, + -0.522531270439395, + -0.5214297184900827, + -0.5245020594282019, + -0.5211829173788083, + -0.5220518585052518, + -0.5210734007415885, + -0.5214676555545281, + -0.5211515144484844, + -0.5211050851709785, + -0.5210625470079859, + -0.5210604114901434, + -0.5234712251894624, + -0.521593032601775, + -0.525234558823649, + -0.5210804260125815, + -0.5210785424739072, + -0.525505371052851, + -0.5221445235887108, + -0.5210593879073832, + -0.5210737023669363, + -0.521083761853172, + -0.5215853405268017, + -0.5227768063008479, + -0.5211065140705554, + -0.5215101602536663, + -0.5288199238810922, + -0.5273214638606539, + -0.5257277842329631, + -0.5210695412795339, + -0.5210753071327717, + -0.5210703536387976, + -0.5218009417179299, + -0.5211164503065419, + -0.5226296090871653, + -0.5218783527595913, + -0.5210395304903896, + -0.5210700655263808, + -0.521145559792451, + -0.5217375844208683, + -0.5210457209511343, + -0.5210895780964436, + -0.5216443499814831, + -0.5228431433329648, + -0.5210832218278626, + -0.5274268727732995, + -0.5211216755913307, + -0.5219205044901309, + -0.524766867769203, + -0.5251195052420332, + -0.5210758044384366, + -0.521089097440652, + -0.521749878026454, + -0.5210958011908401, + -0.5274189971923838, + -0.521237938312581, + -0.5236942419543406, + -0.521062418359218, + -0.5210373652682448, + -0.5210387651008384, + -0.5210721524878251, + -0.5210426125489409, + -0.5210217893863571, + -0.5210228318027557, + -0.5210260738428435, + -0.5210216027586888, + -0.5210252358628062, + -0.5210313416300331, + -0.5210345712133734, + -0.5210229672517228, + -0.521783480697217, + -0.5210361216790312, + -0.5210267851893767, + -0.5210368512770732, + -0.521031021928863, + -0.5217844559373256, + -0.5210215656500876, + -0.5210279207137334, + -0.5218414039206085, + -0.5217026562163004, + -0.5210235606243971, + -0.5210222268319301, + -0.5210315702789767, + -0.5216440218826598, + -0.5219900384499969, + -0.521118206357774, + -0.525315872071635, + -0.5210436227523191 + ], + "colorbar": { + "title": { + "text": "Objective Value" + } + }, + "colorscale": [ + [ + 0, + "rgb(247,251,255)" + ], + [ + 0.125, + "rgb(222,235,247)" + ], + [ + 0.25, + "rgb(198,219,239)" + ], + [ + 0.375, + "rgb(158,202,225)" + ], + [ + 0.5, + "rgb(107,174,214)" + ], + [ + 0.625, + "rgb(66,146,198)" + ], + [ + 0.75, + "rgb(33,113,181)" + ], + [ + 0.875, + "rgb(8,81,156)" + ], + [ + 1, + "rgb(8,48,107)" + ] + ], + "reversescale": false, + "showscale": true + }, + "type": "parcoords" + } + ], + "layout": { + "template": { + "data": { + "bar": [ + { + "error_x": { + "color": "#2a3f5f" + }, + "error_y": { + "color": "#2a3f5f" + }, + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + }, + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "bar" + } + ], + "barpolar": [ + { + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + }, + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "barpolar" + } + ], + "carpet": [ + { + "aaxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "baxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "type": "carpet" + } + ], + "choropleth": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "choropleth" + } + ], + "contour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "contour" + } + ], + "contourcarpet": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "contourcarpet" + } + ], + "heatmap": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmap" + } + ], + "heatmapgl": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmapgl" + } + ], + "histogram": [ + { + "marker": { + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "histogram" + } + ], + "histogram2d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2d" + } + ], + "histogram2dcontour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2dcontour" + } + ], + "mesh3d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "mesh3d" + } + ], + "parcoords": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "parcoords" + } + ], + "pie": [ + { + "automargin": true, + "type": "pie" + } + ], + "scatter": [ + { + "fillpattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + }, + "type": "scatter" + } + ], + "scatter3d": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatter3d" + } + ], + "scattercarpet": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattercarpet" + } + ], + "scattergeo": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergeo" + } + ], + "scattergl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergl" + } + ], + "scattermapbox": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattermapbox" + } + ], + "scatterpolar": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolar" + } + ], + "scatterpolargl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolargl" + } + ], + "scatterternary": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterternary" + } + ], + "surface": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "surface" + } + ], + "table": [ + { + "cells": { + "fill": { + "color": "#EBF0F8" + }, + "line": { + "color": "white" + } + }, + "header": { + "fill": { + "color": "#C8D4E3" + }, + "line": { + "color": "white" + } + }, + "type": "table" + } + ] + }, + "layout": { + "annotationdefaults": { + "arrowcolor": "#2a3f5f", + "arrowhead": 0, + "arrowwidth": 1 + }, + "autotypenumbers": "strict", + "coloraxis": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "colorscale": { + "diverging": [ + [ + 0, + "#8e0152" + ], + [ + 0.1, + "#c51b7d" + ], + [ + 0.2, + "#de77ae" + ], + [ + 0.3, + "#f1b6da" + ], + [ + 0.4, + "#fde0ef" + ], + [ + 0.5, + "#f7f7f7" + ], + [ + 0.6, + "#e6f5d0" + ], + [ + 0.7, + "#b8e186" + ], + [ + 0.8, + "#7fbc41" + ], + [ + 0.9, + "#4d9221" + ], + [ + 1, + "#276419" + ] + ], + "sequential": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "sequentialminus": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ] + }, + "colorway": [ + "#636efa", + "#EF553B", + "#00cc96", + "#ab63fa", + "#FFA15A", + "#19d3f3", + "#FF6692", + "#B6E880", + "#FF97FF", + "#FECB52" + ], + "font": { + "color": "#2a3f5f" + }, + "geo": { + "bgcolor": "white", + "lakecolor": "white", + "landcolor": "#E5ECF6", + "showlakes": true, + "showland": true, + "subunitcolor": "white" + }, + "hoverlabel": { + "align": "left" + }, + "hovermode": "closest", + "mapbox": { + "style": "light" + }, + "paper_bgcolor": "white", + "plot_bgcolor": "#E5ECF6", + "polar": { + "angularaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "radialaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "scene": { + "xaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "yaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "zaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + } + }, + "shapedefaults": { + "line": { + "color": "#2a3f5f" + } + }, + "ternary": { + "aaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "baxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "caxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "title": { + "x": 0.05 + }, + "xaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + }, + "yaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + } + } + }, + "title": { + "text": "Parallel Coordinate Plot" + } + } + }, + "text/html": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig = optuna.visualization.plot_parallel_coordinate(ml_m_study)\n", + "show(fig)" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "e7b5f011", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.plotly.v1+json": { + "config": { + "plotlyServerURL": "https://plot.ly" + }, + "data": [ + { + "cliponaxis": false, + "hovertemplate": [ + "stacking__lgbm__lambda_l2 (FloatDistribution): 0.01634668278708586", + "stacking__lgbm__min_child_samples (IntDistribution): 0.023292997200844767", + "stacking__lgbm__learning_rate (FloatDistribution): 0.09462288774062386", + "stacking__final_estimator__C (FloatDistribution): 0.1285616425560594", + "stacking__lgbm__lambda_l1 (FloatDistribution): 0.7371757897153861" + ], + "name": "Objective Value", + "orientation": "h", + "text": [ + "0.02", + "0.02", + "0.09", + "0.13", + "0.74" + ], + "textposition": "outside", + "type": "bar", + "x": [ + 0.01634668278708586, + 0.023292997200844767, + 0.09462288774062386, + 0.1285616425560594, + 0.7371757897153861 + ], + "y": [ + "stacking__lgbm__lambda_l2", + "stacking__lgbm__min_child_samples", + "stacking__lgbm__learning_rate", + "stacking__final_estimator__C", + "stacking__lgbm__lambda_l1" + ] + } + ], + "layout": { + "template": { + "data": { + "bar": [ + { + "error_x": { + "color": "#2a3f5f" + }, + "error_y": { + "color": "#2a3f5f" + }, + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + }, + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "bar" + } + ], + "barpolar": [ + { + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + }, + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "barpolar" + } + ], + "carpet": [ + { + "aaxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "baxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "type": "carpet" + } + ], + "choropleth": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "choropleth" + } + ], + "contour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "contour" + } + ], + "contourcarpet": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "contourcarpet" + } + ], + "heatmap": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmap" + } + ], + "heatmapgl": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmapgl" + } + ], + "histogram": [ + { + "marker": { + "pattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + } + }, + "type": "histogram" + } + ], + "histogram2d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2d" + } + ], + "histogram2dcontour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2dcontour" + } + ], + "mesh3d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "mesh3d" + } + ], + "parcoords": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "parcoords" + } + ], + "pie": [ + { + "automargin": true, + "type": "pie" + } + ], + "scatter": [ + { + "fillpattern": { + "fillmode": "overlay", + "size": 10, + "solidity": 0.2 + }, + "type": "scatter" + } + ], + "scatter3d": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatter3d" + } + ], + "scattercarpet": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattercarpet" + } + ], + "scattergeo": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergeo" + } + ], + "scattergl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergl" + } + ], + "scattermapbox": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattermapbox" + } + ], + "scatterpolar": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolar" + } + ], + "scatterpolargl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolargl" + } + ], + "scatterternary": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterternary" + } + ], + "surface": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "surface" + } + ], + "table": [ + { + "cells": { + "fill": { + "color": "#EBF0F8" + }, + "line": { + "color": "white" + } + }, + "header": { + "fill": { + "color": "#C8D4E3" + }, + "line": { + "color": "white" + } + }, + "type": "table" + } + ] + }, + "layout": { + "annotationdefaults": { + "arrowcolor": "#2a3f5f", + "arrowhead": 0, + "arrowwidth": 1 + }, + "autotypenumbers": "strict", + "coloraxis": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "colorscale": { + "diverging": [ + [ + 0, + "#8e0152" + ], + [ + 0.1, + "#c51b7d" + ], + [ + 0.2, + "#de77ae" + ], + [ + 0.3, + "#f1b6da" + ], + [ + 0.4, + "#fde0ef" + ], + [ + 0.5, + "#f7f7f7" + ], + [ + 0.6, + "#e6f5d0" + ], + [ + 0.7, + "#b8e186" + ], + [ + 0.8, + "#7fbc41" + ], + [ + 0.9, + "#4d9221" + ], + [ + 1, + "#276419" + ] + ], + "sequential": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "sequentialminus": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ] + }, + "colorway": [ + "#636efa", + "#EF553B", + "#00cc96", + "#ab63fa", + "#FFA15A", + "#19d3f3", + "#FF6692", + "#B6E880", + "#FF97FF", + "#FECB52" + ], + "font": { + "color": "#2a3f5f" + }, + "geo": { + "bgcolor": "white", + "lakecolor": "white", + "landcolor": "#E5ECF6", + "showlakes": true, + "showland": true, + "subunitcolor": "white" + }, + "hoverlabel": { + "align": "left" + }, + "hovermode": "closest", + "mapbox": { + "style": "light" + }, + "paper_bgcolor": "white", + "plot_bgcolor": "#E5ECF6", + "polar": { + "angularaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "radialaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "scene": { + "xaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "yaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "zaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + } + }, + "shapedefaults": { + "line": { + "color": "#2a3f5f" + } + }, + "ternary": { + "aaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "baxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "caxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "title": { + "x": 0.05 + }, + "xaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + }, + "yaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + } + } + }, + "title": { + "text": "Hyperparameter Importances" + }, + "xaxis": { + "title": { + "text": "Hyperparameter Importance" + } + }, + "yaxis": { + "title": { + "text": "Hyperparameter" + } + } + } + }, + "text/html": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig = optuna.visualization.plot_param_importances(ml_m_study)\n", + "show(fig)" + ] + }, + { + "cell_type": "markdown", + "id": "07034102", + "metadata": {}, + "source": [ + "## `DoubleMLAPOS` Tuning Example\n", + "\n", + "We will repeat the tuning procedure for all treatment level with the `DoubleMLAPOS` object.\n", + "Combined `DoubleML` object, such as `DoubleMLAPOS` or `DoubleMLDIDMulti` just pass the tuning arguments to the underlying submodels." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "404c6918", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[0. 1. 2. 3.]\n" + ] + } + ], + "source": [ + "treatment_lvls = np.unique(d)\n", + "print(treatment_lvls)" + ] + }, + { + "cell_type": "markdown", + "id": "b22e18fa", + "metadata": {}, + "source": [ + "Consequently, we are tuning $3$ learners for each treatment level." + ] + }, + { + "cell_type": "markdown", + "id": "d78a4d36", + "metadata": {}, + "source": [ + "### Untuned Model\n", + "\n", + "Again, let's start with the untuned model. We will focus on just on the boosted trees to highlight the ideas." + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "39f51bd6", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
coefstd errtP>|t|2.5 %97.5 %
0.0199.2446279.33549621.3426930.0180.947391217.541863
1.0206.48855110.46801619.7256620.0185.971617227.005485
2.0210.4408679.12806123.0542790.0192.550196228.331538
3.0236.29607719.48669212.1260230.0198.102863274.489292
\n", + "
" + ], + "text/plain": [ + " coef std err t P>|t| 2.5 % 97.5 %\n", + "0.0 199.244627 9.335496 21.342693 0.0 180.947391 217.541863\n", + "1.0 206.488551 10.468016 19.725662 0.0 185.971617 227.005485\n", + "2.0 210.440867 9.128061 23.054279 0.0 192.550196 228.331538\n", + "3.0 236.296077 19.486692 12.126023 0.0 198.102863 274.489292" + ] + }, + "execution_count": 31, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_apos_untuned = DoubleMLAPOS(\n", + " dml_data,\n", + " ml_g,\n", + " ml_m,\n", + " treatment_levels=treatment_lvls,\n", + ")\n", + "dml_apos_untuned.fit()\n", + "dml_apos_untuned.summary" + ] + }, + { + "cell_type": "markdown", + "id": "470632fe", + "metadata": {}, + "source": [ + "### Hyperparameter Tuning\n", + "\n", + "Let's initialize a second `DoubleMLAPOS` object." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "d22135ec", + "metadata": {}, + "outputs": [], + "source": [ + "dml_apos_tuned = DoubleMLAPOS(\n", + " dml_data,\n", + " ml_g,\n", + " ml_m,\n", + " treatment_levels=treatment_lvls,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "139c881d", + "metadata": {}, + "source": [ + "Again, we can directly call the `tune_ml_models()` method for tuning. This will take some time as each submodel is tuned." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "747d851f", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "af5772432f9842d6b56300dc208b2146", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/100 [00:00" + ] + }, + "execution_count": 33, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_apos_tuned.tune_ml_models(\n", + " ml_param_space=param_space,\n", + " optuna_settings=optuna_settings_pipeline,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "5015c34d", + "metadata": {}, + "source": [ + "Afterwards, all hyperparameters are set for each submodel and we can proceed as usual." + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "edf496de", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
coefstd errtP>|t|2.5 %97.5 %
0.0211.4222442.23393194.6413550.0207.043820215.800668
1.0215.3708892.58167583.4229330.0210.310899220.430878
2.0219.2335732.24318097.7334040.0214.837022223.630124
3.0215.9279562.44297688.3872680.0211.139811220.716100
\n", + "
" + ], + "text/plain": [ + " coef std err t P>|t| 2.5 % 97.5 %\n", + "0.0 211.422244 2.233931 94.641355 0.0 207.043820 215.800668\n", + "1.0 215.370889 2.581675 83.422933 0.0 210.310899 220.430878\n", + "2.0 219.233573 2.243180 97.733404 0.0 214.837022 223.630124\n", + "3.0 215.927956 2.442976 88.387268 0.0 211.139811 220.716100" + ] + }, + "execution_count": 34, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_apos_tuned.fit()\n", + "dml_apos_tuned.summary" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "287fc64b", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
coefstd errtP>|t|2.5 %97.5 %
1.0 vs 0.03.9486442.6151801.5098940.131071-1.1770159.074303
2.0 vs 0.07.8113292.3751063.2888330.0010063.15620512.466452
3.0 vs 0.04.5057122.5790801.7470230.080633-0.5491929.560616
\n", + "
" + ], + "text/plain": [ + " coef std err t P>|t| 2.5 % 97.5 %\n", + "1.0 vs 0.0 3.948644 2.615180 1.509894 0.131071 -1.177015 9.074303\n", + "2.0 vs 0.0 7.811329 2.375106 3.288833 0.001006 3.156205 12.466452\n", + "3.0 vs 0.0 4.505712 2.579080 1.747023 0.080633 -0.549192 9.560616" + ] + }, + "execution_count": 35, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dml_apos_tuned.causal_contrast(reference_levels=[0]).summary" + ] + }, + { + "cell_type": "markdown", + "id": "810fd697", + "metadata": {}, + "source": [ + "Finally, lets compare the estimates including confidence intervals." + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "04c8d4cb", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABKUAAAKyCAYAAAAEvm1SAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjYsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvq6yFwwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAmK9JREFUeJzs3Xd4FFXfxvF7U0gvlIRAqCH0DkrvJXQFUQSkClKeIF3A8jyCoiAioDQbTaqggKiABAEx0nvvTek1IQFCkp33j7xZWTaBBJINhO/nuvYyO3Nm5rezuxNzc84Zk2EYhgAAAAAAAAA7csjoAgAAAAAAAPDsIZQCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAoBnUJ06dVSnTp2MLiNNnTp1SiaTSTNnzszoUp4aJpNJw4cPz+gyMo2LFy/q5ZdfVvbs2WUymTRhwgStW7dOJpNJ69ate+j2mfF7mdml9D0zmUzq06dP+heEZx7XdQBPG0IpAHiCzJw5UyaTKdnHpk2bUryvAwcOaPjw4Tp16lT6FfwIpkyZ8sQER8uXL5fJZFLu3LllNpuTbFOgQAGr98Df3181a9bUkiVLbNoahqHZs2erVq1a8vX1lbu7u0qXLq0PPvhA0dHR6f1y0s3FixfVtWtX+fv7y83NTRUqVNCiRYts2g0fPjzJz62rq6tVu5iYGL355pvy8/NTnjx5NHLkSJt9/fPPP/L09NRff/2V6loHDx6sYsWKyd3dXR4eHqpYsaJGjhypGzdupGpfqTVgwAD99ttvevvttzV79mw1btw4XY/3pKpTp45KlSr1SNtu2LBBw4cPT/f3KrO5/zqV3MNe19558+ZpwoQJdjnW4zp37pyGDx+uXbt2pah94u/pbdu2pW9hAPCMcMroAgAAtj744AMVLFjQZnlwcHCK93HgwAGNGDFCderUUYECBazWrVq16nFLfGRTpkxRjhw51KVLlwyrIdHcuXNVoEABnTp1SmvWrFGDBg2SbFeuXDkNGjRIUsIfMF999ZVeeuklTZ06Vb169ZIkxcfHq3379lq4cKFq1qyp4cOHy93dXX/++adGjBihRYsWafXq1cqZM6fdXl9aiIyMVI0aNXTx4kX169dPAQEBWrhwodq0aaO5c+eqffv2NttMnTpVnp6elueOjo5W6z/99FN99913evfdd3Xz5k198MEHKlSokNq1a2dp89Zbb+mFF15Q9erVU1zr1q1b1bRpU0VFRalDhw6qWLGiJGnbtm0aPXq01q9fn66f/TVr1ujFF1/U4MGDLcuKFCmi27dvK0uWLOl23Mxkw4YNGjFihLp06SJfX9+MLuepMWHCBEVFRVmeL1++XPPnz9f48eOVI0cOy/Jq1arZpZ558+Zp37596t+/v12O9zjOnTunESNGqECBAipXrlxGlwMAzxxCKQB4AjVp0kTPPfdcuu2fP5Cl6Oho/fTTTxo1apRmzJihuXPnJhtKBQYGqkOHDpbnnTp1UnBwsMaPH28JpcaMGaOFCxdq8ODB+vTTTy1te/TooTZt2qhly5bq0qWLVqxYkb4vLI199dVXOnbsmH7//XfVq1dPktS7d29VqVJFgwYN0ssvv2zzeXr55Zet/hC+3y+//KJBgwZpyJAhkqS///5by5Yts4RS4eHh+vnnn3Xo0KEU13njxg21atVKjo6O2rlzp4oVK2a1/qOPPtI333yT4v09ikuXLtkEKQ4ODjY9xWB/t27dkru7e0aXkW5atmxp9fzChQuaP3++WrZsafOPEveKjo6Wh4dH+hYHAMADMHwPAJ5SCxYsUMWKFeXl5SVvb2+VLl1an3/+uaSE4QWvvPKKJKlu3bqWoRuJ89rcPw9K4rw3Cxcu1IgRIxQYGCgvLy+9/PLLioiIUExMjPr37y9/f395enqqa9euiomJsapnxowZqlevnvz9/eXi4qISJUpo6tSpVm0KFCig/fv3648//rDUdG8dN27cUP/+/ZU3b165uLgoODhYn3zyic3Quhs3bqhLly7y8fGRr6+vOnfunOrhPkuWLNHt27f1yiuvqG3btlq8eLHu3LmTom0DAgJUvHhxnTx5UpJ0+/ZtffrppypSpIhGjRpl075Fixbq3LmzVq5c+dAhmHv27FGXLl0UFBQkV1dXBQQE6PXXX9fVq1et2iUOlTt27JilV4mPj4+6du2qW7duWbWNiYnRgAED5OfnJy8vL73wwgv6559/UvRa//zzT/n5+VkCKSkhaGnTpo0uXLigP/74w2YbwzAUGRkpwzCS3Oft27eVNWtWy/Ns2bJZajabzerXr5+GDBmiPHnypKhGKSE8O3v2rMaNG2cTSElSzpw59d5771ktmzJlikqWLCkXFxflzp1boaGhNp+jxKFoBw4cUN26deXu7q7AwECNGTPG0iZxOI9hGJo8ebLlsy0p2Tmlvv76axUqVEhubm6qVKmS/vzzzyRfV0xMjN5//30FBwfLxcVFefPm1ZAhQ2y+f4lzFi1dulSlSpWSi4uLSpYsqZUrV9rs8+zZs+rWrZty584tFxcXFSxYUL1799bdu3ctbVL6XUyplNQ3fPhwvfXWW5KkggULWs7jvUOQ58yZo4oVK8rNzU3ZsmVT27Zt9ffff1sdK/E92759u2rVqiV3d3e98847at68uYKCgpKsr2rVqlb/EJCS69mjmDt3rooWLSpXV1dVrFhR69evt6xbu3atTCZTkkOD582bJ5PJpI0bNz7ysbt06SJPT08dP35cTZs2lZeXl1577TVJCd+7CRMmqGTJknJ1dVXOnDnVs2dPXb9+3WofP/30k5o1a2b57BQqVEgffvih4uPjLW3q1KmjX3/9VadPn7a8h4nBWFr8rpFS9zl40Hd33bp1ev755yVJXbt2TdNhjmfPntXrr7+unDlzWj7v06dPt6y/ePGinJycNGLECJttDx8+LJPJpEmTJlmWpfV3EgCeFPSUAoAnUEREhK5cuWK1zGQyKXv27JKksLAwtWvXTvXr19cnn3wiSTp48KD++usv9evXT7Vq1VLfvn31xRdf6J133lHx4sUlyfLf5IwaNUpubm4aNmyYjh07pokTJ8rZ2VkODg66fv26hg8frk2bNmnmzJkqWLCg/ve//1m2nTp1qkqWLKkXXnhBTk5O+vnnn/Wf//xHZrNZoaGhkhKGmLz55pvy9PTUu+++K0mW4Wy3bt1S7dq1dfbsWfXs2VP58uXThg0b9Pbbb+v8+fOW+UkMw9CLL76o8PBw9erVS8WLF9eSJUvUuXPnVJ3juXPnqm7dugoICFDbtm01bNgw/fzzz5Yw70FiY2P1999/W96P8PBwXb9+Xf369ZOTU9K/Wjt16qQZM2bol19+UZUqVZLdd1hYmE6cOKGuXbsqICBA+/fv19dff639+/dr06ZNlrAjUZs2bVSwYEGNGjVKO3bs0Lfffit/f3/L50KSunfvrjlz5qh9+/aqVq2a1qxZo2bNmqXkNCkmJkZubm42yxN7nWzfvl0NGza0WhcUFKSoqCh5eHioZcuW+uyzz6yGLT7//PP6+uuvVadOHUVFRWn+/PmWSaCnTZumK1euWMKJlFq2bJnc3Nz08ssvp6j98OHDNWLECDVo0EC9e/fW4cOHNXXqVG3dulV//fWXnJ2dLW2vX7+uxo0b66WXXlKbNm30ww8/aOjQoSpdurSaNGmiWrVqafbs2erYsaMaNmyoTp06PfDY06ZNU8+ePVWtWjX1799fJ06c0AsvvKBs2bIpb968lnZms1kvvPCCwsPD1aNHDxUvXlx79+7V+PHjdeTIES1dutRqv+Hh4Vq8eLH+85//yMvLS1988YVat26tM2fOWD6r586dU6VKlXTjxg316NFDxYoV09mzZ/XDDz/o1q1bypIlS4q/i6n1sPpeeuklHTlyxGbYmZ+fn6SE3m7//e9/1aZNG3Xv3l2XL1/WxIkTVatWLe3cudOql9rVq1fVpEkTtW3bVh06dFDOnDlVsWJFderUSVu3brUEEZJ0+vRpbdq0yaqHY0quZ6n1xx9/6Pvvv1ffvn3l4uKiKVOmqHHjxtqyZYtKlSqlOnXqKG/evJo7d65atWplte3cuXNVqFAhVa1a9ZGOnSguLk6NGjVSjRo1NHbsWMv3uGfPnpo5c6a6du2qvn376uTJk5o0aZJ27txp9X2YOXOmPD09NXDgQHl6emrNmjX63//+p8jISMv5e/fddxUREaF//vlH48ePlySr4bzS4/2uSc3n4GHf3eLFi+uDDz7Q//73P/Xo0UM1a9aU9PjDHC9evKgqVapYwlg/Pz+tWLFC3bp1U2RkpPr376+cOXOqdu3aWrhwod5//32r7b///ns5Ojpafh+l13cSAJ4IBgDgiTFjxgxDUpIPFxcXS7t+/foZ3t7eRlxcXLL7WrRokSHJWLt2rc262rVrG7Vr17Y8X7t2rSHJKFWqlHH37l3L8nbt2hkmk8lo0qSJ1fZVq1Y18ufPb7Xs1q1bNsdp1KiRERQUZLWsZMmSVsdO9OGHHxoeHh7GkSNHrJYPGzbMcHR0NM6cOWMYhmEsXbrUkGSMGTPG0iYuLs6oWbOmIcmYMWOGzb7vd/HiRcPJycn45ptvLMuqVatmvPjiizZt8+fPb4SEhBiXL182Ll++bOzevdto27atIcl48803DcMwjAkTJhiSjCVLliR7zGvXrhmSjJdeeumBtSV1HufPn29IMtavX29Z9v777xuSjNdff92qbatWrYzs2bNbnu/atcuQZPznP/+xate+fXtDkvH+++8/sJ4333zTcHBwME6dOmW1PPEc9OnTx7JswoQJRp8+fYy5c+caP/zwg9GvXz/DycnJKFy4sBEREWFp9/fffxslS5a0fLZr1qxp3Lx507hx44bh5+dnLFiw4IE1JSVr1qxG2bJlU9T20qVLRpYsWYyQkBAjPj7esnzSpEmGJGP69OmWZbVr1zYkGd99951lWUxMjBEQEGC0bt3aar+SjNDQUKtlid+txO/h3bt3DX9/f6NcuXJGTEyMpd3XX39tSLL6bsyePdtwcHAw/vzzT6t9fvnll4Yk46+//rI6dpYsWYxjx45Zlu3evduQZEycONGyrFOnToaDg4OxdetWm/NiNpsNw0j5dzE5tWvXNkqWLGm1LKX1ffrpp4Yk4+TJk1bbnzp1ynB0dDQ++ugjq+V79+41nJycrJYnvmdffvmlVduIiAjDxcXFGDRokNXyMWPGGCaTyTh9+rRlWUqvZ/dfS5OT+Fnftm2bZdnp06cNV1dXo1WrVpZlb7/9tuHi4mLcuHHDsuzSpUuGk5PTQ7+r90rqPHbu3NmQZAwbNsyq7Z9//mlIMubOnWu1fOXKlTbLkzovPXv2NNzd3Y07d+5YljVr1szmd4RhPP7vmkf5HDzsu7t169YU/+4wjH9/Tyf1HUrUrVs3I1euXMaVK1eslrdt29bw8fGxnMevvvrKkGTs3bvXql2JEiWMevXqWZ6n5juZkus6ADxJGL4HAE+gyZMnKywszOpx71xEvr6+io6OVlhYWJoet1OnTlY9RCpXrizDMPT6669btatcubL+/vtvxcXFWZbd25smsadX7dq1deLECUVERDz02IsWLVLNmjWVNWtWXblyxfJo0KCB4uPjLcNcli9fLicnJ/Xu3duyraOjo958880Uv84FCxbIwcFBrVu3tixr166dVqxYYTNcRUqYGN7Pz09+fn4qW7asFi1apI4dO1p6I928eVOS5OXllewxE9dFRkY+sLZ7z+OdO3d05coVS8+qHTt22LRPnNMqUc2aNXX16lXLcZYvXy5J6tu3r1W7lE5A3L17dzk6OqpNmzbasGGDjh8/rlGjRlmGGN2+fdvStl+/fpo4caLat2+v1q1ba8KECZo1a5aOHj2qKVOmWNrlyZNHO3fu1M6dO7V//36tW7dOnp6eGjFihIoWLapXX31V4eHhqly5svLmzau+fftaDS1LSmRk5APP/71Wr16tu3fvqn///nJw+Pd/hd544w15e3vr119/tWrv6elpNadYlixZVKlSJZ04cSJFx7vXtm3bdOnSJfXq1ctqLq7E4aj3WrRokYoXL65ixYpZfScSh1KuXbvWqn2DBg1UqFAhy/MyZcrI29vbUqfZbNbSpUvVokWLJOesS+yFl9LvYmo9rL4HWbx4scxms9q0aWNVU0BAgAoXLmxzLlxcXNS1a1erZd7e3mrSpIkWLlxoNbT0+++/V5UqVZQvXz7Lsse9niWlatWqlsn3JSlfvnx68cUX9dtvv1mGv3Xq1EkxMTH64YcfrOqLi4uz+gw+jnuvnVLC++3j46OGDRtanduKFSvK09PT6tzee15u3rypK1euqGbNmrp161aq5oB71N81qf0cpOV3N6UMw9CPP/6oFi1ayDAMqzobNWqkiIgIy7X8pZdekpOTk77//nvL9vv27dOBAwf06quvWpal13cSAJ4EDN8DgCdQpUqVHjjR+X/+8x8tXLhQTZo0UWBgoEJCQtSmTZvHvgX9vX+USbL8kXzvkKLE5WazWREREZZhQX/99Zfef/99bdy40WZOo4iICJs/uO939OhR7dmzxzJU536XLl2SlDDUJleuXDbDQYoWLfqQV/evOXPmqFKlSrp69aplrqby5cvr7t27WrRokXr06GHVvnLlyho5cqRMJpPc3d1VvHhxqyEiiWFIYjiVlJQEV5J07do1jRgxQgsWLLC85kRJ/TF8/3uWOFfT9evX5e3trdOnT8vBwcEqDJBSfr7KlCmjefPmqVevXpY74QUEBGjChAnq3bu3zftwv/bt22vQoEFavXq1hg0bZlnu7OxsdaerQ4cOacqUKdqwYYOuXbumZs2aadiwYapbt666du2qjz76KMm5VxJ5e3s/8Pzf6/Tp05Jsz0GWLFkUFBRkWZ8oT548NsMms2bNqj179qToeEkdu3DhwlbLnZ2dbeY7Onr0qA4ePPjQ70Si+z8LiXUmBq2XL19WZGSkSpUq9cAaU/pdTK2H1fewmgzDsDlvie4NOKSEmxMkdUOHV199VUuXLtXGjRtVrVo1HT9+XNu3b7cZ/vS417OkJFV7kSJFdOvWLV2+fFkBAQEqVqyYnn/+ec2dO1fdunWTlDB0r0qVKqm6+2pynJycbOZqO3r0qCIiIuTv75/kNve+3/v379d7772nNWvW2ATsqQnrHvV3TWo/B2n53U2py5cv68aNG/r666/19ddfJ9km8ZzmyJFD9evX18KFC/Xhhx9KSgghnZyc9NJLL1nap9d3EgCeBIRSAPAU8vf3165du/Tbb79pxYoVWrFihWbMmKFOnTpp1qxZj7xfR0fHVC1P7G1w/Phx1a9fX8WKFdO4ceOUN29eZcmSRcuXL9f48eNTNBGr2WxWw4YNLXdku1+RIkVS+Coe7OjRo9q6daukpP9InDt3rk0olSNHjmTvzCf9O1fXnj17bO6ClSjxj6ASJUo8sL7EHklvvfWWypUrJ09PT5nNZjVu3DjJ8/iw9yYtvPzyy3rhhRe0e/duxcfHq0KFCpaJu1PyvuTNm1fXrl17YJsBAwaoQ4cOqlChgmbPnq1s2bLp7bffliQNGTLkoaFUsWLFtGvXLt29ezfN7y5pj3OcFLPZrNKlS2vcuHFJrr//D/i0qjO9vouPU5/ZbJbJZNKKFSuS3M/94WhS86BJCTcdcHd318KFC1WtWjUtXLhQDg4OVnPJpcX17HF06tRJ/fr10z///KOYmBht2rTJasLrx+Hi4mLVO1BKOLf+/v6aO3duktskBiE3btxQ7dq15e3trQ8++ECFChWSq6urduzYoaFDh6bqvDzq75rUfg4y4rubeB46dOiQ7FyHZcqUsfzctm1bde3aVbt27VK5cuW0cOFC1a9f3+oOpvb6/QgAGYFQCgCeUlmyZFGLFi3UokULmc1m/ec//9FXX32l//73vwoODrb51+H09PPPPysmJkbLli2z+hfw+4dSSEq2rkKFCikqKuqB4Y8k5c+fX7///ruioqKs/gA5fPhwimqdO3eunJ2dNXv2bJs/WMLDw/XFF1/ozJkzSfbqSE6NGjXk6+urefPm6d13303yD6HvvvtOktS8efNk93P9+nX9/vvvGjFihNXEvkePHk1xLffLnz+/zGazjh8/btUzKKXnK1GWLFmsJodevXq1JD30/TIMQ6dOnVL58uWTbfPLL79ow4YNltd57tw55cqVy7I+d+7cOnv27AOP06JFC23cuFE//vij2rVr98C2+fPnl5RwDu7tnXT37l2dPHnyoa/pcSQe++jRo1Z3NIyNjdXJkydVtmxZy7JChQpp9+7dql+/fpp8n/38/OTt7a19+/Y9sF1Kv4vp4UHXB8MwVLBgwcf6A9zDw0PNmzfXokWLNG7cOH3//feqWbOmcufObWmTmutZaiT1PT5y5Ijc3d2tesC0bdtWAwcO1Pz583X79m05OztbDeVKa4UKFdLq1atVvXr1ZMM8KeFOdVevXtXixYtVq1Yty/LEu5DeK71+/6TV5+BeaV1r4l1O4+PjU/QdatmypXr27GkZwnfkyBFLIJ8oI7+TAJDemFMKAJ5CiUPOEjk4OFj+5TXx9tkeHh6SZHOL+/SQGMLc+6/PERERmjFjhk1bDw+PJGtq06aNNm7cqN9++81m3Y0bNyxzijRt2lRxcXFWt2ePj4/XxIkTU1Tr3LlzVbNmTb366qt6+eWXrR6Jd3ybP39+ivaVyN3dXYMHD9bhw4ctdxW816+//qqZM2eqUaNGD7zzXlLnUdJj3VmpSZMmkqQvvvgizfZ59OhRffnll2revLnVH4aXL1+2aTt16lRdvnw52aGld+/e1cCBA/Xee+9Zhg/lzJlTx44ds7znBw8eVEBAwANr6tWrl3LlyqVBgwbpyJEjNusvXbqkkSNHSkoI0rJkyaIvvvjC6lxPmzZNERERKb4z4aN47rnn5Ofnpy+//NJqnqyZM2fafC/atGmjs2fP6ptvvrHZz+3btxUdHZ2qYzs4OKhly5b6+eeftW3bNpv1iecipd/F9JDcdeull16So6OjRowYYfP9MAzD5pr4IK+++qrOnTunb7/9Vrt377YJfFJzPUuNjRs3Ws0L9/fff+unn35SSEiIVZCdI0cONWnSRHPmzNHcuXPVuHFjq14zaa1NmzaKj4+3DB+7V1xcnOW9SOq83L1712q+uEQeHh6PPPfWg6Tl5yBRWv+udHR0VOvWrfXjjz8mGQDff5309fVVo0aNtHDhQi1YsEBZsmSx6XGbkd9JAEhv9JQCgCfQihUrkpw0tlq1agoKClL37t117do11atXT3ny5NHp06c1ceJElStXzjKUrFy5cnJ0dNQnn3yiiIgIubi4qF69esnOG/I4QkJCLD23evbsqaioKH3zzTfy9/fX+fPnrdpWrFhRU6dO1ciRIxUcHCx/f3/Vq1dPb731lpYtW6bmzZurS5cuqlixoqKjo7V371798MMPOnXqlHLkyKEWLVqoevXqGjZsmE6dOqUSJUpo8eLFKfoDaPPmzTp27Jj69OmT5PrAwEBVqFBBc+fO1dChQ1N1DoYNG6adO3fqk08+0caNG9W6dWu5ubkpPDxcc+bMUfHixR86tNLb21u1atXSmDFjFBsbq8DAQK1atSrJnggpVa5cObVr105TpkxRRESEqlWrpt9//13Hjh1L8T5KlCihV155Rfny5dPJkyc1depUZcuWTV9++aVVu/z58+vVV19V6dKl5erqqvDwcC1YsEDlypVTz549k9z3559/LilhkvRETZs2VWhoqNq3b69q1arpww8/VPfu3R9YY9asWbVkyRI1bdpU5cqVU4cOHSyTSu/YsUPz589X1apVJSX0ZHj77bc1YsQINW7cWC+88IIOHz6sKVOm6Pnnn0+zCaWT4uzsrJEjR6pnz56qV6+eXn31VZ08eVIzZsywmVOqY8eOWrhwoXr16qW1a9eqevXqio+P16FDh7Rw4UL99ttvD5x7Likff/yxVq1apdq1a6tHjx4qXry4zp8/r0WLFik8PFy+vr4p/i6mh8T37N1331Xbtm3l7OysFi1aqFChQho5cqTefvttnTp1Si1btpSXl5dOnjypJUuWqEePHho8eHCKjtG0aVN5eXlp8ODBlgDhXqm5nqVGqVKl1KhRI/Xt21cuLi6WMCepYamdOnXSyy+/LElJhkVpqXbt2urZs6dGjRqlXbt2KSQkRM7Ozjp69KgWLVqkzz//XC+//LKqVaumrFmzqnPnzurbt69MJpNmz56d5FC4ihUr6vvvv9fAgQP1/PPPy9PTUy1atHjsWtPyc3DvPn19ffXll1/Ky8tLHh4eqly5sgoWLPjA7aZPn66VK1faLO/Xr59Gjx6ttWvXqnLlynrjjTdUokQJXbt2TTt27NDq1atthjO/+uqr6tChg6ZMmaJGjRpZzVkoKUO/kwCQ7uxzkz8AQEok3mo6uUfiLat/+OEHIyQkxPD39zeyZMli5MuXz+jZs6dx/vx5q/198803RlBQkOHo6Gh1W/r7b2OeeJvuRYsWJVnP/be+fv/99w1JxuXLly3Lli1bZpQpU8ZwdXU1ChQoYHzyySfG9OnTbW5LfuHCBaNZs2aGl5eXIcmqjps3bxpvv/22ERwcbGTJksXIkSOHUa1aNWPs2LFWtw+/evWq0bFjR8Pb29vw8fExOnbsaOzcufOht/V+8803DUnG8ePHk20zfPhwQ5Kxe/duwzAMI3/+/EazZs2SbX+v+Ph4Y8aMGUb16tUNb29vw9XV1ShZsqQxYsQIIyoqKkX7+Oeff4xWrVoZvr6+ho+Pj/HKK68Y586ds7nNd1LvgWH8+57de85v375t9O3b18iePbvh4eFhtGjRwvj7779TfOvwtm3bGnnz5jWyZMli5M6d2+jVq5dx8eJFm3bdu3c3SpQoYXh5eRnOzs5GcHCwMXToUCMyMjLJ/V64cMHw8vIyli1bZrNuxYoVRrFixQxfX1+jU6dORnR09EPrNAzDOHfunDFgwACjSJEihqurq+Hu7m5UrFjR+Oijj4yIiAirtpMmTTKKFStmODs7Gzlz5jR69+5tXL9+3apN7dq1jZIlS9ocp3Pnzja3vJdkhIaGWi1L/G4lfvcSTZkyxShYsKDh4uJiPPfcc8b69ettvpeGYRh37941PvnkE6NkyZKGi4uLkTVrVqNixYrGiBEjrF5PUsc2jITPb+fOna2WnT592ujUqZPh5+dnuLi4GEFBQUZoaKgRExNjaZPS72JSkjpnqanvww8/NAIDAw0HBwebz/KPP/5o1KhRw/Dw8DA8PDyMYsWKGaGhocbhw4cfePz7vfbaa4Yko0GDBkmuT+n1LKn3LCmJr3/OnDlG4cKFDRcXF6N8+fI2n4tEMTExRtasWQ0fHx/j9u3bD93//T799FObWjt37mx4eHgku83XX39tVKxY0XBzczO8vLyM0qVLG0OGDDHOnTtnafPXX38ZVapUMdzc3IzcuXMbQ4YMMX777Tebz3hUVJTRvn17w9fX15Bk+a6kxe8aw3i8z0FS392ffvrJKFGihOHk5PTQ3yMP+z39999/G4ZhGBcvXjRCQ0ONvHnzGs7OzkZAQIBRv3594+uvv7bZZ2RkpOHm5mZIMubMmZPkcVP6nUzpdR0AnhQmw0jnWToBAAAApFhcXJxy586tFi1aaNq0aRldDgAA6YY5pQAAAIAnyNKlS3X58mV16tQpo0sBACBd0VMKAAAAeAJs3rxZe/bs0YcffqgcOXJYTYwOAEBmRE8pAAAA4AkwdepU9e7dW/7+/vruu+8yuhwAANIdPaUAAAAAAABgd/SUAgAAAAAAgN0RSgEAAAAAAMDuCKUAAMiExowZo2LFislsNqdqu3Xr1slkMmndunXpU9j/K1CggLp06WL342Z2JpNJw4cPz+gy8ABdunRRgQIFLM+vXr0qDw8PLV++POOKAgAggxBKAQCQyURGRuqTTz7R0KFD5eDw7696k8lkeTg4OCh37twKCQkhCLrP8OHDrc5Vco86depkdKlPjVOnTslkMmns2LFJrh87dqxMJpNOnTqV6n0fOHBAw4cPf6RtnwTZs2dX9+7d9d///jejSwEAwO6cMroAAACQtqZPn664uDi1a9fOZl3Dhg3VqVMnGYahkydPasqUKapXr55+/fVXNWnSRLVq1dLt27eVJUsWu9acUcdNyksvvaTg4GDL86ioKPXu3VutWrXSSy+9ZFmeM2fOjCgP9zlw4IBGjBihOnXqWPVAepr06tVLX3zxhdasWaN69epldDkAANgNoRQAAJnMjBkz9MILL8jV1dVmXZEiRdShQwfL81atWqlMmTKaMGGCmjRpIgcHhyS3S28ZddyklClTRmXKlLE8v3Llinr37q0yZcpYnTsgrRQvXlylSpXSzJkzCaUAAM8Uhu8BAJCJnDx5Unv27FGDBg1S1L506dLKkSOHTp48KSnpuZ3q1KmjUqVKafv27apWrZrc3NxUsGBBffnllzb7i4mJ0fvvv6/g4GC5uLgob968GjJkiGJiYh5Yx4OOe+DAAdWtW1fu7u4KDAzUmDFjHvm4V65c0aFDh3Tr1q0UnZ/kzJw5M8nhZvZ6HTExMRowYID8/Pzk5eWlF154Qf/8889D67548aKcnJw0YsQIm3WHDx+WyWTSpEmTJEmxsbEaMWKEChcuLFdXV2XPnl01atRQWFhYCs7Q4ytQoICaN2+u8PBwVapUSa6urgoKCtJ3331naTNz5ky98sorkqS6detahlYmnv/k5ti6f06zxPfzr7/+0sCBA+Xn5ycPDw+1atVKly9fttl+xYoVqlmzpjw8POTl5aVmzZpp//79Nu2WLl2qUqVKydXVVaVKldKSJUuSfb0NGzbUzz//LMMwUniGAAB4+hFKAQCQiWzYsEGSVKFChRS1v379uq5fv67s2bM/tF3Tpk1VsWJFjRkzRnny5FHv3r01ffp0Sxuz2awXXnhBY8eOVYsWLTRx4kS1bNlS48eP16uvvvpIr+f69etq3LixypYtq88++0zFihXT0KFDtWLFikc67qRJk1S8eHFt2bLlkep5VGn9Orp3764JEyYoJCREo0ePlrOzs5o1a/bQOnLmzKnatWtr4cKFNuu+//57OTo6WkKe4cOHa8SIEapbt64mTZqkd999V/ny5dOOHTse82yk3LFjx/Tyyy+rYcOG+uyzz5Q1a1Z16dLFEgDVqlVLffv2lSS98847mj17tmbPnq3ixYs/0vHefPNN7d69W++//7569+6tn3/+WX369LFqM3v2bDVr1kyenp765JNP9N///lcHDhxQjRo1rELKVatWqXXr1jKZTBo1apRatmyprl27atu2bUkeu2LFirpx40aS4RYAAJkVw/cAAMhEDh06JEkqWLBgkuvv3LmjK1euWOaUeueddxQfH28JIpJz7tw5ffbZZxo4cKAkqWfPnqpcubLefvttdezYUc7Ozpo3b55Wr16tP/74QzVq1LBsW6pUKfXq1UsbNmxQtWrVUvV6zp07p++++04dO3aUJHXr1k358+fXtGnT1KRJE0lKl+OmtbR8Hbt379acOXP0n//8R5MnT5YkhYaG6rXXXtOePXseWsurr76qnj17at++fSpVqpRl+ffff6/atWtb5sr69ddf1bRpU3399ddpdh5S6/Dhw1q/fr1q1qwpSWrTpo3y5s2rGTNmaOzYsQoKClLNmjX1xRdfqGHDho89+Xz27Nm1atUqmUwmSQlB4RdffKGIiAj5+PgoKipKffv2Vffu3a3OS+fOnVW0aFF9/PHHluVDhw5Vzpw5FR4eLh8fH0lS7dq1FRISovz589scOygoSFLCHFn3vi8AAGRm9JQCACATuXr1qpycnOTp6Znk+mnTpsnPz0/+/v6qXLmyZbhS//79H7hfJycn9ezZ0/I8S5Ys6tmzpy5duqTt27dLkhYtWqTixYurWLFiunLliuWROEfO2rVrU/16PD09reZxypIliypVqqQTJ05YlqXmuMOHD5dhGHa/c15avo7ly5dLkqWHUKKHvYeJXnrpJTk5Oen777+3LNu3b58OHDhg1SPL19dX+/fv19GjR1P3YtNQiRIlLIGUJPn5+alo0aJW5y0t9ejRwxJISVLNmjUVHx+v06dPS5LCwsJ048YNtWvXzuo9cnR0VOXKlS3v0fnz57Vr1y517tzZEkhJCUP0SpQokeSxs2bNKilhiCkAAM8KekoBAPAMefHFF9WnTx+ZTCZ5eXmpZMmS8vDweOh2uXPntmlXpEgRSdKpU6dUpUoVHT16VAcPHpSfn1+S+7h06VKq682TJ49VSCAl/PF+b4+g9DhuWkvL13H69Gk5ODioUKFCVuuLFi2aolpy5Mih+vXra+HChfrwww8lJfSScnJysrq74AcffKAXX3xRRYoUUalSpdS4cWN17NjRahL4tHb/OcqXL59Nm6xZs+r69evpcvz7j5cYFCUeLzGgS24ycm9vb0myhFiFCxe2aVO0aNEkh0AmziV1/zkAACAzI5QCACATyZ49u+Li4nTz5k15eXnZrM+TJ0+KJ0FPLbPZrNKlS2vcuHFJrs+bN2+q9+no6Jjk8nsng06P4z5McsFBfHx8ksuftNfRtm1bde3aVbt27VK5cuW0cOFC1a9fXzly5LC0qVWrlo4fP66ffvpJq1at0rfffqvx48fryy+/VPfu3VN1vMQ7K96+fTvJ9YkTz99/B8aUnLdH8ajvk9lslpQwr1RAQIBNOyenR/9f68Tg6973AACAzI5QCgCATKRYsWKSEu7Cl5Y9Ws6dO6fo6Gir3lJHjhyRlHAnM0kqVKiQdu/erfr169u1t0dGHDexB82NGzeslif2kHkUKX0d+fPnl9ls1vHjx616Rx0+fDjFx2rZsqV69uxpGcJ35MgRvf322zbtsmXLpq5du6pr166KiopSrVq1NHz48FSHUn5+fnJ3d0+2xsOHD8vd3f2RApkHnausWbPavEd3797V+fPnU30cSZbeaf7+/g8MdxPnjEpq6GNy5yDxDpiPOkk7AABPI+aUAgAgE6lataokJXuHr0cVFxenr776yvL87t27+uqrr+Tn56eKFStKSpiE+uzZs/rmm29str99+7aio6PTtKZEqTnulStXdOjQIUvPnEeVGE6sX7/esiw+Pv6xJgVP6etInBj9iy++sGozYcKEFB/L19dXjRo10sKFC7VgwQJlyZJFLVu2tGpz9epVq+eenp4KDg5WTEyMZVlERIQOHTqkiIiIBx7P0dFRISEh+vnnn3XmzBmrdWfOnNHPP/+skJCQZHsqPUhiUHp/+CQlvE/3vkeS9PXXXyfbU+phGjVqJG9vb3388ceKjY21WX/58mVJUq5cuVSuXDnNmjXL6tyEhYXpwIEDSe57+/bt8vHxUcmSJR+pNgAAnkb0lAIAIBMJCgpSqVKltHr1ar3++utptt/cuXPrk08+0alTp1SkSBF9//332rVrl77++ms5OztLkjp27KiFCxeqV69eWrt2rapXr674+HgdOnRICxcu1G+//abnnnsuzWpKlJrjTpo0SSNGjNDatWsfa7LzkiVLqkqVKnr77bd17do1ZcuWTQsWLFBcXFy6v45y5cqpXbt2mjJliiIiIlStWjX9/vvvOnbsWKqO9+qrr6pDhw6aMmWKGjVqJF9fX6v1JUqUUJ06dVSxYkVly5ZN27Zt0w8//KA+ffpY2ixZskRdu3bVjBkz1KVLlwce7+OPP1aVKlVUoUIF9ejRQwUKFNCpU6f09ddfy2Qy6eOPP05V/YnKlSsnR0dHffLJJ4qIiJCLi4vq1asnf39/de/eXb169VLr1q3VsGFD7d69W7/99tsjD5Hz9vbW1KlT1bFjR1WoUEFt27aVn5+fzpw5o19//VXVq1fXpEmTJEmjRo1Ss2bNVKNGDb3++uu6du2aJk6cqJIlSyoqKspm32FhYWrRogVzSgEAnimEUgAAZDKvv/66/ve//+n27dtyc3NLk31mzZpVs2bN0ptvvqlvvvlGOXPm1KRJk/TGG29Y2jg4OGjp0qUaP368vvvuOy1ZskTu7u4KCgpSv379LBOjp7WMOu7cuXPVs2dPjR49Wr6+vurWrZvq1q2rhg0bPtL+UvM6pk+fLj8/P82dO1dLly5VvXr19Ouvv6Zq3qkXXnhBbm5uunnzptVd9xL17dtXy5Yt06pVqxQTE6P8+fNr5MiReuuttx7p9RUvXlybN2/W8OHDNW3aNEuY17BhQ73//vuWoaepFRAQoC+//FKjRo1St27dFB8fr7Vr18rf319vvPGGTp48qWnTpmnlypWqWbOmwsLCVL9+/Uc6liS1b99euXPn1ujRo/Xpp58qJiZGgYGBqlmzprp27Wpp17hxYy1atEjvvfee3n77bRUqVEgzZszQTz/9pHXr1lnt89ChQ9q3b1+qersBAJAZmIzHnSkSAAA8USIiIhQUFKQxY8aoW7duj72/OnXq6MqVK9q3b18aVAfgfv3799f69eu1fft2ekoBAJ4pzCkFAEAm4+PjoyFDhujTTz+13C0MwJPp6tWr+vbbbzVy5EgCKQDAM4eeUgAA4IHoKQUAAID0QE8pAAAAAAAA2B09pQAAAAAAAGB39JQCAAAAAACA3RFKAQAAAAAAwO6cMrqAJ4HZbNa5c+fk5eXFXU8AAAAAAAAeg2EYunnzpnLnzi0Hh+T7QxFKSTp37pzy5s2b0WUAAAAAAABkGn///bfy5MmT7HpCKUleXl6SEk6Wt7d3Blfz5IiNjdWqVasUEhIiZ2fnjC4HADIU10QASMD1EAD+xTUxaZGRkcqbN68lb0kOoZRkGbLn7e1NKHWP2NhYubu7y9vbmy8XgGce10QASMD1EAD+xTXxwR42RRITnQMAAAAAAMDuCKUAAAAAAABgd4RSAAAAAAAAsDvmlEohs9msu3fvZnQZdhUbGysnJyfduXNH8fHxGV0O8ERydnaWo6NjRpcBAAAAAE8dQqkUuHv3rk6ePCmz2ZzRpdiVYRgKCAjQ33///dDJyYBnma+vrwICAvieAAAAAEAqEEo9hGEYOn/+vBwdHZU3b145ODw7Ix7NZrOioqLk6en5TL1uIKUMw9CtW7d06dIlSVKuXLkyuCIAAAAAeHoQSj1EXFycbt26pdy5c8vd3T2jy7GrxCGLrq6uhFJAMtzc3CRJly5dkr+/P0P5AAAAACCFSBoeInEupSxZsmRwJQCeVImBdWxsbAZXAgAAAABPD0KpFGKuGADJ4foAAAAAAKlHKAUAAAAAAAC7I5QCAAAAAACA3RFKZUImk+mBj+HDh9u9pvnz58vR0VGhoaE269atW2dVX86cOdW6dWudOHHCqt2GDRvUtGlTZc2aVa6uripdurTGjRtnmfcLAAAAAAA8PQilMqHz589bHhMmTJC3t7fVssGDB1vaGoahuLi4dK9p2rRpGjJkiObPn687d+4k2ebw4cM6d+6cFi1apP3796tFixaWwGnJkiWqXbu28uTJo7Vr1+rQoUPq16+fRo4cqbZt28owjHR/DQAAAAAAIO0QSqWGYUgRERn7SEH4EhAQYHn4+PjIZDJZnh86dEheXl5asWKFKlasKBcXF4WHh6tLly5q2bKl1X4GDBig5s2bW56bzWaNGjVKBQsWlJubm8qWLasffvjhofWcPHlSGzZs0LBhw1SkSBEtXrw4yXb+/v7KlSuXatWqpf/97386cOCAjh07pujoaL3xxht64YUX9PXXX6tcuXIqUKCAunfvrlmzZumHH37QwoULH1oHAAAAAAB4cjhldAFPlchIqUOHjK1hzhzJx+exdzNs2DCNHTtWQUFBypo1a4q2GTVqlObMmaMvv/xShQsX1vr169WhQwf5+fmpdu3ayW43Y8YMNWvWTD4+PurQoYOmTZum9u3bP/BYbm5ukqS7d+9q1apVunr1qlUPr0QtWrRQkSJFNH/+fL366qspeh0AAAAAACDjEUo9oz744AM1bNgwxe1jYmL08ccfa/Xq1apataokKSgoSOHh4frqq6+SDaXMZrNmzpypiRMnSpLatm2rQYMG6eTJkypYsGCS25w/f15jx45VYGCgihYtquXLl0uSihcvnmT7YsWK6ciRIyl+LQAAAAAAIOMxfO8Z9dxzz6Wq/bFjx3Tr1i01bNhQnp6elsd3332n48ePJ7tdWFiYoqOj1bRpU0lSjhw51LBhQ02fPt2mbZ48eeTh4aHcuXMrOjpaP/74o7JkyWJZz7xRAAAAAABkHvSUekZ5eHhYPXdwcLAJfWJjYy0/R0VFSZJ+/fVXBQYGWrVzcXFJ9jjTpk3TtWvXLMPxpITeU3v27NGIESPk4PBvLvrnn3/K29tb/v7+8vLysiwvUqSIJOngwYOqVq2azTEOHjyoEiVKJFsDAAAAAAB48hBKpYa3d8KcThldQzrw8/PTvn37rJbt3r1bJpNJklSiRAm5uLjozJkzD5w/6l5Xr17VTz/9pAULFqhkyZKW5fHx8apRo4ZWrVqlxo0bW5YXLFhQvr6+NvsJCQlRtmzZ9Nlnn9mEUsuWLdPRo0f14YcfpvSlAgAAAACAJwChVGqYTGkyyfiTqF69evr000/13XffqWrVqpozZ4727dun0qVLS5K8vLw0ePBgDRgwQGazWTVq1FBERIT++usveXt7q3Pnzjb7nD17trJnz642bdpYwq1ETZs21bRp06xCqeR4eHjoq6++Utu2bdWjRw/16dNH3t7e+v333/XWW2/p5ZdfVps2bdLmRAAAAAAAALtgTilIkho1aqT//ve/GjJkiJ5//nndvHlTHTt2tGrz4Ycf6r///a9GjRql4sWLq3Hjxvr111+TnbB8+vTpatWqlU0gJUmtW7fWsmXLdOXKlRTV9/LLL2vt2rU6c+aMatasqaJFi2r8+PF69913tWDBgiSPAQAAAAAAnlwmg9mjFRkZKR8fH0VERMj7vuFxd+7csdwpztXVNYMqzBhms1mRkZHy9va2mvsJgLVn+TrxLImNjdXy5cvVtGlTOTs7Z3Q5AJBhuB4CwL+4JibtQTnLvUgaAAAAAAAAYHfMKQUAAAAAADKVcX8c1/j1J1K93YBaQRpYu1A6VISkEEoBAAAAAIBMJfJOnM5G3Hmk7WA/hFIAAAAAACBT8XZ1UqCP9XyvhmHoXGSMJCm3t0uSN8zydiUmsSfONgAAAAAAyFQG1i5kMwwvOiZOXu+ukCQdHlpPHi5EIhmNic4BAAAAAABgd4RSAAAAAAAAsDv6qqUDZvkHAAAAAAB4MEKpdMAs/wAAAAAAAA/G8L10kDjL/72P3N4ulvW5vV1s1gf6uDLLfwrNnDlTvr6+j7UPk8mkpUuXpkk9ierUqaP+/ftbnhcoUEATJkxI02MAAAAAAJBZEEqlg4G1C+nv/za0ehx4q65l/bevlNWpdxvYtEnLoXv3BySJHiXQCQoKeurCFZPJZHn4+PioevXqWrNmjWX9+fPn1aRJk3StYevWrerRo0e6HuNexYoVk4uLiy5cuGCzrk6dOpbz4erqqhIlSmjKlClWbW7fvq33339fRYoUkYuLi3LkyKFXXnlF+/fvT9M6Y2Nj9cEHH6hQoUJydXVV2bJltXLlSqs2w4cPt3oPTSaTihUrZtVm4MCBypYtm/Lmzau5c+darVu0aJFatGiRonru3r2rMWPGqGzZsnJ3d1eOHDlUvXp1zZgxQ7GxsZKkLl26qGXLlo/+ogEAAAAANgil7GDx3vMq+ek6y/Om07ao4EertXjv+Ywr6hkwY8YMnT9/Xn/99Zdy5Mih5s2b68SJhLm+AgIC5OLi8pA9PB4/Pz+5u7un6zEShYeH6/bt23r55Zc1a9asJNu88cYbOn/+vA4cOKA2bdooNDRU8+fPlyTFxMSoQYMGmj59ukaOHKkjR45o+fLliouLU+XKlbVp06YU17Ju3ToVKFAg2fXvvfeevvrqK02cOFEHDhxQr1691KpVK+3cudOqXcmSJXX+/HnLIzw83LLu559/1rx587Rq1SqNGTNG3bt315UrVyRJERERevfddzV58uSH1nr37l01atRIo0ePVo8ePbRhwwZt2bJFoaGhmjhxYpoHcgAAAACAfxFKpbPFe8/rlVnbdDbSeo6psxF39MqsbRkeTCX2ABk7dqxy5cql7NmzKzQ01NJDpHnz5jp9+rQGDBhg6bEiJfRkKVeunNW+JkyYYBVGPGzfUkIYMnjwYAUGBsrDw0OVK1fWunXrrPY7c+ZM5cuXT+7u7mrVqpWuXr2aotfm6+urgIAAlSpVSlOnTtXt27cVFhYmyXr43qlTp2QymbRgwQJVq1ZNrq6uKlWqlP744w+r/e3bt09NmjSRp6encubMqY4dO1qCkKTcP3zPZDLp22+/VatWreTu7q7ChQtr2bJlj3WMRNOmTVP79u3VsWNHTZ8+Pck27u7uCggIUFBQkIYPH251/AkTJmjjxo365Zdf1KZNG+XPn1+VKlXSjz/+qOLFi6tbt24yDOOhdaTE7Nmz9c4776hp06YKCgpS79691bRpU3322WdW7ZycnBQQEGB55MiRw7Lu4MGDqlOnjp577jm1a9dO3t7eOnnypCRpyJAh6t27t/Lly/fQWiZMmKD169fr999/V2hoqMqVK6egoCC1b99emzdvVuHChdPkNQMAAAAAbBFKpaN4s6H+S/cpqT/lE5cN+Gmf4s1p88f+o1q7dq2OHz+utWvXatasWZo5c6ZmzpwpKSFAyJMnjz744ANLj5W02rck9enTRxs3btSCBQu0Z88evfLKK2rcuLGOHj0qSdq8ebO6deumPn36aNeuXapbt65GjhyZ6tfo5uYmKaFnTHLeeustDRo0SDt37lTVqlXVokULSwB248YN1atXT+XLl9e2bdu0cuVKXbx4UW3atElVHSNGjFCbNm20Z88eNW3aVK+99pquXbv2WMe4efOmFi1apA4dOqhhw4aKiIjQn3/++dBa3NzcLOdj3rx5atiwocqWLWvVxsHBQQMGDNCBAwe0e/fuVL3W5MTExMjV1dWmlnt7QknS0aNHlTt3bgUFBem1117TmTNnLOvKli2rbdu26fr169q+fbtu376t4OBghYeHa8eOHerbt2+Kapk7d64aNGig8uXL26xzdnaWh4fHI7xCAAAAAEBKEEqloz9PXNU/D7gLnyHp7xt39OeJlPX8SS9Zs2bVpEmTVKxYMTVv3lzNmjXT77//blnn6OgoLy8vS4+VtNr3mTNnNGPGDC1atEg1a9ZUoUKFNHjwYNWoUUMzZsyQJH3++edq3LixhgwZoiJFiqhv375q1KhRqmq4deuW3nvvPTk6Oqp27drJtuvTp49at26t4sWLa+rUqfLx8dG0adMkSZMmTVL58uX18ccfq1ixYipfvrymT5+utWvX6siRIymupUuXLmrXrp2Cg4P18ccfKyoqSlu2bHmsYyxYsECFCxdWyZIl5ejoqLZt21rqTkp8fLzmzJmjPXv2qF69epKkI0eOqHjx4km2T1z+oBo8PT0tjyZNmujMmTNWy3r16mVp26hRI40bN05Hjx6V2WxWWFiYFi9ebBV4Vq5cWTNnztTKlSs1depUnTx5UjVr1tTNmzct++jQoYOef/55denSRbNmzZKHh4d69+6tL7/8UlOnTlXRokVVvXr1Bw7BO3r0qM1cVQAAAAAA++B2b+no/M2YNG2XXhLDjES5cuXS3r17033fe/fuVXx8vIoUKWK1TUxMjLJnzy4pYZhWq1atrNZXrVrVZmLspLRr106Ojo66ffu2/Pz8NG3aNJUpUybZ9lWrVrX87OTkpOeee04HDx6UJO3evVtr166Vp6enzXbHjx+3eQ3Juff4Hh4e8vb21qVLlx7rGNOnT1eHDh0szzt06KDatWtr4sSJ8vLysiyfMmWKvv32W929e1eOjo4aMGCAevfubVn/OMPzdu3aZfl58+bNGjp0qNUwTG9vb8vPn3/+ud544w0VK1ZMJpNJhQoVUteuXa2GHd47CX2ZMmVUuXJl5c+fXwsXLlS3bt0kJQwhHT58uKXdiBEj1KBBAzk7O2vkyJHau3evfvnlF3Xq1Enbt29Psu60GpIIAAAAAEg9Qql0lMsrZRNpp7Rdanh7eysiIsJm+Y0bN+Tj42O1zNnZ2eq5yWSS2Wx+4P4dHBxs/qC/d66olOw7KipKjo6O2r59u1VwJSnJYCa1xo8frwYNGsjHx0d+fn6Pta+oqCi1aNFCn3zyic26XLlypXg/DzsfqT3GgQMHtGnTJm3ZskVDhw61LI+Pj9eCBQv0xhtvWJa99tprevfdd+Xm5qZcuXLJweHfjpJFihSxBHD3S1z+oOAtODjY8vM///wjJycnq2X38vPz09KlS3Xnzh1dvXpVuXPn1rBhwxQUFJTs/n19fVWkSBEdO3YsyfWHDh3SnDlztHPnTk2fPl21atWSn5+f2rRpo9dff103b960Cujufd2HDh1K9rgAAAAAgPTD8L10VDMou/L4uMqUzHqTpLy+rqoZlD3Nj120aFHt2LHDZvmOHTtS3KsnUZYsWRQfH2+1zM/PTxcuXLAKpu7tLZMS5cuXV3x8vC5duqTg4GCrR+IwweLFi2vz5s1W26X0TnABAQEKDg5OcSB1737j4uK0fft2y9C1ChUqaP/+/SpQoIBNrWk179CjHGPatGmqVauWdu/erV27dlkeAwcOtBnC5+Pjo+DgYAUGBloFUpLUtm1brV692mbeKLPZrPHjx6tEiRI28009LldXVwUGBiouLk4//vijXnzxxWTbRkVF6fjx40mGc4ZhqGfPnho3bpw8PT0VHx9vCUgT/3v/5zdR+/bttXr1aps7/yVuGx0d/SgvDQAAAACQAoRS6cjRwaQJLUtJkk0wlfh8/Iul5OiQXGz16Hr37q0jR46ob9++2rNnjw4fPqxx48Zp/vz5GjRoUKr2lT9/fq1fv15nz5613AmuTp06unz5ssaMGaPjx49r8uTJWrFiRar2W6RIEb322mvq1KmTFi9erJMnT2rLli0aNWqUfv31V0lS3759tXLlSo0dO1ZHjx7VpEmTUjR071FMnjxZS5Ys0aFDhxQaGqrr16/r9ddflySFhobq2rVrateunbZu3arjx4/rt99+U9euXZMNPFIrtceIjY3V7Nmz1a5dO5UqVcrq0b17d23evPmB8ynda8CAAapUqZJatGihRYsW6cyZM9q6datat26tgwcPatq0aZY7LyblwoULlkexYsW0adMmq2X39trbvHmzFi9erBMnTujPP/9U48aNZTabNWTIEEubwYMH648//tCpU6e0YcMGtWrVSo6OjmrXrp3Nsb/99lv5+fmpRYsWkqTq1atrzZo12rRpkyVQ8/X1TbLu/v37q3r16qpfv74mT56s3bt368SJE1q4cKGqVKlimXAfAAAAAJD2CKXS2Uulc2lR5+eU29v6bmN5fF21qPNzeql0yod+pUZQUJDWr1+vQ4cOqUGDBqpcubIWLlyoRYsWqXHjxqna14gRI3Tq1CkVKlTI0uuoePHimjJliiZPnqyyZctqy5YtGjx4cKrrnDFjhjp16qRBgwapaNGiatmypbZu3ap8+fJJkqpUqaJvvvlGn3/+ucqWLatVq1bpvffeS/VxUmL06NEaPXq0ypYtq/DwcC1btkw5cuSQJOXOnVt//fWX4uPjFRISotKlS6t///7y9fW16XX0qFJ7jGXLlunq1as2c25JCe9P8eLFHzjh+b1cXV21Zs0aderUSe+8846Cg4PVuHFjOTo6atOmTapSpcoDt8+VK9cDH/369bO0vXPnjt577z2VKFFCrVq1UmBgoMLDw62Co3/++Uft2rVT0aJF1aZNG2XPnl2bNm2y6fV28eJFffTRR/riiy8syypVqqRBgwapWbNmWrhwoWXS/KS4uLgoLCxMQ4YM0VdffaUqVaro+eef1xdffKG+ffuqVKlSKTp/AAAAAIDUMxnM9KvIyEj5+PgoIiLCakJmKeEP6JMnT6pgwYI2t7FP1TFux8r3vwk9fJZ3q6SGRf3TpYdUWjKbzYqMjJS3t3eaBS9PolOnTqlgwYLauXOnypUrl9Hl4CmUVtcJPNliY2O1fPlyNW3a1GZ+OAB4lnA9BPC0io6Jk9e7CSN8bn7URB4ujz/NNtfEpD0oZ7kXE52ng3F/HNf49Seslt2b/XVftDvJoVADagVpYO1C6V4fAAAAAABARiOUSgeRd+J0NuJOsuvPRcYkux0AAAAAAMCzgFAqHXi7OinQJ/VDeLxdeTsyQoECBcQoVgAAAAAA7IsUJB0MrF2IYXgAAAAAAAAPkHlnrwYAAAAAAMATi1AKAAAAAAAAdkcoBQAAAAAAALtjTql0ELFjgiJ2fJ7q7Xwq9JNPhf5pXxAAAAAAAMAThlAqHZhjIhUfdfaRtgMAAAAAAHgWMHwvHTi4eMvRM9D64ZHbst7RI7ftes9AObh4Z2DV6W/mzJny9fV9rH2YTCYtXbo0TepJVKdOHfXv39/yvECBApowYUKaHgMAAAAAAFgjlEoHPhX6K1/3k1aPwE57LOtzNPxKeV8/ZtMmrYbumUymBz6GDx+eJsdJD/fW6ePjo+rVq2vNmjWW9efPn1eTJk3StYatW7eqR48e6XqMexUrVkwuLi66cOGCzbo6depYzoerq6tKlCihKVOmWLW5ffu23n//fRUpUkQuLi7KkSOHXnnlFe3fvz9N64yNjdUHH3ygQoUKydXVVWXLltXKlSut2gwfPtzm81asWDGrNgMHDlS2bNmUN29ezZ0712rdokWL1KJFixTVc/fuXY0ZM0Zly5aVu7u7cuTIoerVq2vGjBmKjY2VJHXp0kUtW7Z89BcNAAAAAEg3hFJ2EH1sic5+V8by/OLSFvp7erCijy1Jl+OdP3/e8pgwYYK8vb2tlg0ePDhdjptWZsyYofPnz+uvv/5Sjhw51Lx5c504cUKSFBAQIBcXl3Q9vp+fn9zd3dP1GInCw8N1+/Ztvfzyy5o1a1aSbd544w2dP39eBw4cUJs2bRQaGqr58+dLkmJiYtSgQQNNnz5dI0eO1JEjR7R8+XLFxcWpcuXK2rRpU4prWbdunQoUKJDs+vfee09fffWVJk6cqAMHDqhXr15q1aqVdu7cadWuZMmSVp+38PBwy7qff/5Z8+bN06pVqzRmzBh1795dV65ckSRFRETo3Xff1eTJkx9a6927d9WoUSONHj1aPXr00IYNG7RlyxaFhoZq4sSJaR7IAQAAAADSHqFUOos+tkSXfmmr+OhzVsvjo87p0i9t0yWYCggIsDx8fHxkMpksz7/88kvVqFHDqv2ECROswojE3iWfffaZihUrJj8/P4WGhlp6n0gJYcjgwYMVGBgoDw8PVa5cWevWrbPa78yZM5UvXz65u7urVatWunr1aorq9/X1VUBAgEqVKqWpU6fq9u3bCgsLk2Q9fO/UqVMymUxasGCBqlWrJldXV5UqVUp//PGH1f727dunJk2ayNPTUzlz5lTHjh0tQUhS7h++ZzKZ9O2336pVq1Zyd3dX4cKFtWzZssc6RqJp06apffv26tixo6ZPn55kG3d3dwUEBCgoKEjDhw+3Ov6ECRO0ceNG/fLLL2rTpo3y58+vSpUq6ccff1Tx4sXVrVs3GYbx0DpSYvbs2XrnnXfUtGlTBQUFqXfv3mratKk+++wzq3ZOTk5Wn8EcOXJY1h08eFB16tTRc889p3bt2snb21snT56UJA0ZMkS9e/dWvnz5HlrLhAkTtH79ev3+++8KDQ1VuXLlFBQUpPbt22vz5s0qXLhwmrxmAAAAAED6IZRKR4Y5XlfXDZSUVCiQsOzqukEyzPF2rSsl1q5dq+PHj2vZsmWaMWOGZs6cqZkzZ1rW9+nTRxs3btSCBQu0Z88evfLKK2rcuLGOHj0qSdq8ebO6deumPn36aNeuXapbt65GjhyZ6jrc3NwkJfSMSc5bb72lQYMGaefOnapatapatGhhCcBu3LihevXqqXz58tq2bZtWrlypixcvqk2bNqmqY8SIEWrTpo327Nmjpk2b6rXXXtO1a9ce6xg3b97UokWL1KFDBzVs2FARERH6888/H1qLm5ub5XzMmzdPDRs2VNmyZa3aODg4aMCAATpw4IB2796dqteanJiYGLm6utrUcm9PKEk6evSocufOraCgIL322ms6c+aMZV3ZsmW1bds2Xb9+Xdu3b9ft27cVHBys8PBw7dixQ3379k1RLXPnzlWDBg1Uvnx5m3XOzs7y8PB4hFcIAAAAALAnQql0dOds+EPuwmcoPuof3Tkb/oA2GSNr1qyaOHGiihQpoubNm6tZs2b6/fffJUlnzpzRjBkztGjRItWsWVOFChXS4MGDVaNGDc2YMUOS9Pnnn6tx48YaMmSIihQpor59+6pRo0apquHWrVt677335OjoqNq1ayfbrk+fPmrdurWKFy+uqVOnysfHR9OmTZMkTZo0SeXLl9fHH3+sYsWKqXz58po+fbrWrl2rI0eOpLiWLl26qF27dgoODtbHH3+sqKgobdmy5bGOsWDBAhUuXFglS5aUo6Oj2rZta6k7KfHx8ZozZ4727NmjevXqSZKOHDmi4sWLJ9k+cfmDavD09LQ8mjRpojNnzlgt69Wrl6Vto0aNNG7cOB09elRms1lhYWFavHixzp8/b2lTuXJlzZw5UytXrtTUqVN18uRJ1axZUzdv3rTso0OHDnr++efVpUsXzZo1Sx4eHurdu7e+/PJLTZ06VUWLFlX16tUfOATv6NGjNnNVAQAAAACeLk4ZXUBmFh99/uGNUtHOnhKDkkS5cuXS3r17JUl79+5VfHy8ihQpYrVNTEyMsmfPLilhmFarVq2s1letWtVmYuyktGvXTo6Ojrp9+7b8/Pw0bdo0lSlTJtn2VatWtfzs5OSk5557TgcPHpQk7d69W2vXrpWnp6fNdsePH7d5Dcm59/geHh7y9vbWpUuXHusY06dPV4cOHSzPO3TooNq1a2vixIny8vKyLJ8yZYq+/fZb3b17V46OjhowYIB69+5tWf84w/N27dpl+Xnz5s0aOnSo1TBMb+9/7wj5+eef64033lCxYsVkMplUqFAhde3a1WrY4b2T0JcpU0aVK1dW/vz5tXDhQnXr1k1SwmTo9062P2LECDVo0EDOzs4aOXKk9u7dq19++UWdOnXS9u3bk6w7rYYkAgAAAAAyDqFUOnL0yJWm7dKCg4ODzR/0984VlcjZ2dnquclkktlsliRFRUXJ0dFR27dvtwquJCUZzKTW+PHj1aBBA/n4+MjPz++x9hUVFaUWLVrok08+sVmXK1fKz/vDzkdqj3HgwAFt2rRJW7Zs0dChQy3L4+PjtWDBAr3xxhuWZa+99preffddubm5KVeuXHJw+LeDY5EiRSwB3P0Slz8oeAsODrb8/M8//8jJyclq2b38/Py0dOlS3blzR1evXlXu3Lk1bNgwBQUFJbt/X19fFSlSRMeOHUty/aFDhzRnzhzt3LlT06dPV61ateTn56c2bdro9ddf182bN60Cuntf96FDh5I9LgAAAADgycfwvXTkGlhDjp6BkkzJtDDJ0TOPXANrJLM+7fn5+enChQtWwdS9vWVSonz58oqPj9elS5cUHBxs9QgICJCUMHRs8+bNVtul9E5wAQEBCg4OTnEgde9+4+LitH37dsvQtQoVKmj//v0qUKCATa1pNe/Qoxxj2rRpqlWrlnbv3q1du3ZZHgMHDrQZwufj46Pg4GAFBgZaBVKS1LZtW61evdpm3iiz2azx48erRIkSNvNNPS5XV1cFBgYqLi5OP/74o1588cVk20ZFRen48eNJhnOGYahnz54aN26cPD09FR8fbwlIE/8bH5/0fGvt27fX6tWrbe78l7htdHT0o7w0AAAAAIAdEUqlI5ODo7LXGZf47P61kqTsdT6TycFR9lKnTh1dvnxZY8aM0fHjxzV58mStWLEiVfsoUqSIXnvtNXXq1EmLFy/WyZMntWXLFo0aNUq//vqrJKlv375auXKlxo4dq6NHj2rSpEkpGrr3KCZPnqwlS5bo0KFDCg0N1fXr1/X6669LkkJDQ3Xt2jW1a9dOW7du1fHjx/Xbb7+pa9euyQYeqZXaY8TGxmr27Nlq166dSpUqZfXo3r27Nm/e/MD5lO41YMAAVapUSS1atNCiRYt05swZbd26Va1bt9bBgwc1bdo0mUzJhaLShQsXLI9ixYpp06ZNVssiIiIsbTdv3qzFixfrxIkT+vPPP9W4cWOZzWYNGTLE0mbw4MH6448/dOrUKW3YsEGtWrWSo6Oj2rVrZ3Psb7/9Vn5+fmrRooUkqXr16lqzZo02bdpkCdR8fX2TrLt///6qXr266tevr8mTJ2v37t06ceKEFi5cqCpVqlgm3AcAAAAAPLkIpdKZR3Ar+TdfYDNEz9EzUP7NF8gjuFUyW6aP4sWLa8qUKZo8ebLKli2rLVu2aPDgwanez4wZM9SpUycNGjRIRYsWVcuWLbV161bly5dPklSlShV98803+vzzz1W2bFmtWrVK7733Xlq/HEnS6NGjNXr0aJUtW1bh4eFatmyZcuTIIUnKnTu3/vrrL8XHxyskJESlS5dW//795evra9Pr6FGl9hjLli3T1atXbebckhLen+LFiz9wwvN7ubq6as2aNerUqZPeeecdBQcHq3HjxnJ0dNSmTZtUpUqVB26fK1euBz769etnaXvnzh299957KlGihFq1aqXAwECFh4dbBUf//POP2rVrp6JFi6pNmzbKnj27Nm3aZNPr7eLFi/roo4/0xRdfWJZVqlRJgwYNUrNmzbRw4ULLpPlJcXFxUVhYmIYMGaKvvvpKVapU0fPPP68vvvhCffv2ValSpVJ0/gAAAAAAGcdkMGOwIiMj5ePjo4iICKuJnaWEP8RPnjypggULytXV9ZGPER8TqTNTE4KSnC1/llu+BnbtIfUozGazIiMj5e3tnWYBTlo6deqUChYsqJ07d6pcuXIZXQ6eYWl1ncCTLTY2VsuXL1fTpk1t5pkDgGcJ10MAT6vomDh5vZswUujmR03k4fL402xzTUzag3KWezHReTqI2DFBETs+t154T/Z3JaynlMSQKp8K/eRToX86VwcAAAAAAJDxCKXSgTkmUvFRZ5NdHx99LtntAAAAAAAAngWEUunAwcX7/++6l/rtkHIFChQQo08BAAAAAHg6EUqlA58K/RmGBwAAAAAA8ABP3uzVTyh65ABIDtcHAAAAAEg9QqmHcHRMuEPe3bt3M7gSAE+qW7duSRJ32wAAAACAVGD43kM4OTnJ3d1dly9flrOzsxwcnp0cz2w26+7du7pz584z9bqBlDIMQ7du3dKlS5fk6+trCbEBAAAAAA9HKPUQJpNJuXLl0smTJ3X69OmMLseuDMPQ7du35ebmJpPJlNHlAE8sX19fBQQEZHQZAAAAAPBUIZRKgSxZsqhw4cLP3BC+2NhYrV+/XrVq1WJYEpAMZ2dnekgBAAAAwCMglEohBwcHubq6ZnQZduXo6Ki4uDi5uroSSgEAAAAAgDTFREEAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAgAAAAAAgN0RSgEAAAAAAMDuCKUAAAAAAABgd4RSAAAAAAAAsDtCKQAAAAAAANgdoRQAAAAAAADsjlAKAAAAAAAAdkcoBQAAAAAAALsjlAIAAAAAAIDdEUoBAAAAAADA7gilAAAAAAAAYHeEUgAAAAAAALA7QikAAAAAAADYHaEUAAAAAAAA7I5QCgAAAAAAAHZHKAUAAAAAAAC7I5QCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOwuQ0OpUaNG6fnnn5eXl5f8/f3VsmVLHT582Kbdxo0bVa9ePXl4eMjb21u1atXS7du3LeuvXbum1157Td7e3vL19VW3bt0UFRVlz5cCAAAAAACAVMjQUOqPP/5QaGioNm3apLCwMMXGxiokJETR0dGWNhs3blTjxo0VEhKiLVu2aOvWrerTp48cHP4t/bXXXtP+/fsVFhamX375RevXr1ePHj0y4iUBAAAAAAAgBZwy8uArV660ej5z5kz5+/tr+/btqlWrliRpwIAB6tu3r4YNG2ZpV7RoUcvPBw8e1MqVK7V161Y999xzkqSJEyeqadOmGjt2rHLnzm2HVwIAAAAAAIDUeKLmlIqIiJAkZcuWTZJ06dIlbd68Wf7+/qpWrZpy5syp2rVrKzw83LLNxo0b5evrawmkJKlBgwZycHDQ5s2b7fsCAAAAAAAAkCIZ2lPqXmazWf3791f16tVVqlQpSdKJEyckScOHD9fYsWNVrlw5fffdd6pfv7727dunwoUL68KFC/L397fal5OTk7Jly6YLFy4keayYmBjFxMRYnkdGRkqSYmNjFRsbmx4v76mUeC44JwDANREAEnE9BPC0io2Lu+fnWMU6GI+/T66JSUrp+XhiQqnQ0FDt27fPqheU2WyWJPXs2VNdu3aVJJUvX16///67pk+frlGjRj3SsUaNGqURI0bYLF+1apXc3d0faZ+ZWVhYWEaXAABPDK6JAJCA6yGAp82deClxwNhvv62Sq2Pa7ZtrorVbt26lqN0TEUr16dPHMkF5njx5LMtz5colSSpRooRV++LFi+vMmTOSpICAAF26dMlqfVxcnK5du6aAgIAkj/f2229r4MCBlueRkZHKmzevQkJC5O3tnSavKTOIjY1VWFiYGjZsKGdn54wuBwAyFNdEAEjA9RDA0yr6bpy0YbUkqVGjEHlkefxIhGti0hJHpD1MhoZShmHozTff1JIlS7Ru3ToVLFjQan2BAgWUO3duHT582Gr5kSNH1KRJE0lS1apVdePGDW3fvl0VK1aUJK1Zs0Zms1mVK1dO8rguLi5ycXGxWe7s7MyHKAmcFwD4F9dEAEjA9RDA08bZbPr3ZydnOTunXSTCNdFaSs9FhoZSoaGhmjdvnn766Sd5eXlZ5oDy8fGRm5ubTCaT3nrrLb3//vsqW7asypUrp1mzZunQoUP64YcfJCX0mmrcuLHeeOMNffnll4qNjVWfPn3Utm1b7rwHAAAAAADwhMrQUGrq1KmSpDp16lgtnzFjhrp06SJJ6t+/v+7cuaMBAwbo2rVrKlu2rMLCwlSoUCFL+7lz56pPnz6qX7++HBwc1Lp1a33xxRf2ehkAAAAAAABIpQwfvpcSw4YN07Bhw5Jdny1bNs2bNy+tygIAAAAAAEA6c8joAgAAAAAAAPDsIZQCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAgAAAAAAgN0RSgEAAAAAAMDuCKUAAAAAAABgd4RSAAAAAAAAsDtCKQAAAAAAANgdoRQAAAAAAADsjlAKAAAAAAAAdkcoBQAAAAAAALsjlAIAAAAAAIDdEUoBAAAAAADA7gilAAAAAAAAYHeEUgAAAAAAALA7QikAAAAAAADYHaEUAAAAAAAA7I5QCgAAAAAAAHZHKAUAAAAAAAC7I5QCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAgAAAAAAgN0RSgEAAAAAAMDuCKUAAAAAAABgd4RSAAAAAAAAsDtCKQAAAAAAANgdoRQAAAAAAADsjlAKAAAAAAAAdkcoBQAAAAAAALsjlAIAAAAAAIDdEUoBAAAAAADA7gilAAAAAAAAYHeEUgAAAAAAALA7QikAAAAAAADYHaEUAAAAAAAA7I5QCgAAAAAAAHZHKAUAAAAAAAC7I5QCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAgAAAAAAgN0RSgEAAAAAAMDuCKUAAAAAAABgd4RSAAAAAAAAsDtCKQAAAAAAANgdoRQAAAAAAADsjlAKAAAAAAAAdkcoBQAAAAAAALsjlAIAAAAAAIDdEUoBAAAAAADA7gilAAAAAAAAYHeEUgAAAAAAALA7QikAAAAAAADYHaEUAAAAAAAA7I5QCgAAAAAAAHZHKAUAAAAAAAC7I5QCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAgAAAAAAgN0RSgEAAAAAAMDuCKUAAAAAAABgd4RSAAAAAAAAsDtCKQAAAAAAANgdoRQAAAAAAADsjlAKAAAAAAAAdkcoBQAAAAAAALsjlAIAAAAAAIDdEUoBAAAAAADA7gilAAAAAAAAYHeEUgAAAAAAALA7QikAAAAAAADYHaEUAAAAAAAA7I5QCgAAAAAAAHZHKAUAAAAAAAC7I5QCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOwuQ0OpUaNG6fnnn5eXl5f8/f3VsmVLHT58OMm2hmGoSZMmMplMWrp0qdW6M2fOqFmzZnJ3d5e/v7/eeustxcXF2eEVAAAAAAAA4FFkaCj1xx9/KDQ0VJs2bVJYWJhiY2MVEhKi6Ohom7YTJkyQyWSyWR4fH69mzZrp7t272rBhg2bNmqWZM2fqf//7nz1eAgAAAAAAAB6BU0YefOXKlVbPZ86cKX9/f23fvl21atWyLN+1a5c+++wzbdu2Tbly5bLaZtWqVTpw4IBWr16tnDlzqly5cvrwww81dOhQDR8+XFmyZLHLawEAAAAAAEDKZWgodb+IiAhJUrZs2SzLbt26pfbt22vy5MkKCAiw2Wbjxo0qXbq0cubMaVnWqFEj9e7dW/v371f58uVttomJiVFMTIzleWRkpCQpNjZWsbGxafZ6nnaJ54JzAgBcEwEgEddDAE+r2Hum+YmNi1Wsg/H4++SamKSUno8nJpQym83q37+/qlevrlKlSlmWDxgwQNWqVdOLL76Y5HYXLlywCqQkWZ5fuHAhyW1GjRqlESNG2CxftWqV3N3dH/UlZFphYWEZXQIAPDG4JgJAAq6HAJ42d+KlxFmMfvttlVwd027fXBOt3bp1K0XtnphQKjQ0VPv27VN4eLhl2bJly7RmzRrt3LkzTY/19ttva+DAgZbnkZGRyps3r0JCQuTt7Z2mx3qaxcbGKiwsTA0bNpSzs3NGlwMAGYprIgAk4HoI4GkVfTdO2rBaktSoUYg8sjx+JMI1MWmJI9Ie5okIpfr06aNffvlF69evV548eSzL16xZo+PHj8vX19eqfevWrVWzZk2tW7dOAQEB2rJli9X6ixcvSlKSw/0kycXFRS4uLjbLnZ2d+RAlgfMCAP/imggACbgeAnjaOJv/vXmas5OznJ3TLhLhmmgtpeciQ+++ZxiG+vTpoyVLlmjNmjUqWLCg1fphw4Zpz5492rVrl+UhSePHj9eMGTMkSVWrVtXevXt16dIly3ZhYWHy9vZWiRIl7PZaAAAAAAAAkHIZ2lMqNDRU8+bN008//SQvLy/LHFA+Pj5yc3NTQEBAkr2d8uXLZwmwQkJCVKJECXXs2FFjxozRhQsX9N577yk0NDTJ3lAAAAAAAADIeBnaU2rq1KmKiIhQnTp1lCtXLsvj+++/T/E+HB0d9csvv8jR0VFVq1ZVhw4d1KlTJ33wwQfpWDkAAAAAAAAeR4b2lDKM1N9+Malt8ufPr+XLl6dFSQAAAAAAALCDDO0pBQAAAAAAgGcToRQAAAAAAADsjlAKAAAAAAAAdkcoBQAAAAAAALsjlAIAAAAAAIDdEUoBAAAAAADA7gilAAAAAAAAYHeEUgAAAAAAALA7QikAAAAAAADYHaEUAAAAAADI9OLNhuXnP09ctXqOjEEoBQAAAAAAMrXFe8+r5KfrLM+bTtuigh+t1uK95zOuKBBKAQAAAACAzGvx3vN6ZdY2nY28Y7X8bMQdvTJrG8FUBiKUAgAAAAAAmVK82VD/pfuU1EC9xGUDftrHUL4MQigFAAAAAAAypT9PXNU/EXeSXW9I+vvGHf154qr9ioIFoRQAAAAAAMiUzt+MSdN2SFuEUgAAAAAAIFPK5eWSpu2QtgilAAAAAABAplQzKLvy+LjKlMx6k6S8vq6qGZTdnmXh/xFKAQAAAACATMnRwaQJLUtJkk0wlfh8/Iul5OiQXGyF9EQoBQAAAAAAMq2XSufSos7PKbe3q9XyPL6uWtT5Ob1UOlcGVQanjC4AAAAAAAAgPb1UOpcaBOeQ739XSpKWd6ukhkX96SGVwegpBQAAAAAAMr17A6iaQdkJpJ4AhFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAgAAAAAAgN0RSgEAAAAAAMDuCKUAAAAAAABgd4RSAAAAAAAAsDtCKQAAAAAAANgdoRQAAAAAAADsjlAKAAAAAAAAdkcoBQAAAAAAALsjlAIAAAAAAIDdEUoBAAAAAADA7gilAAAAAAAAYHeEUgAAAAAAALA7QikAAAAAAADYHaEUAAAAAAAA7I5QCgAAAAAAAHZHKAUAAAAAAAC7I5QCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAgAAAAAAgN2lKpSKi4vTp59+qgoVKsjT01Oenp6qUKGCxo4dq9jY2PSqEQAAAAAAAJmMU0ob3r59Ww0bNtTGjRvVoEED1apVS5J08OBBDR06VMuWLdOqVavk6uqabsUCAAAAAAAgc0hxKDV69Gj9/fff2rlzp8qUKWO1bvfu3XrhhRc0evRoDR8+PK1rBAAAAAAAQCaT4uF7CxYs0Lhx42wCKUkqW7asxo4dq3nz5qVpcQAAAAAAAMicUhxKnT59WpUqVUp2fZUqVXTmzJk0KQoAAAAAAACZW4pDKW9vb126dCnZ9RcuXJCXl1eaFAUAAAAAAIDMLcWhVN26dfXxxx8nu3706NGqW7dumhQFAAAAAACAzC3FE52///77qly5sqpUqaKBAweqWLFiMgxDBw8e1Pjx43XgwAFt2rQpPWsFAAAA8ADj/jiu8etPpHq7AbWCNLB2oXSoCACA5KU4lCpRooTCwsLUrVs3tW3bViaTSZJkGIaKFSumVatWqWTJkulWKAAAAIAHi7wTp7MRdx5pOwAA7C3FoZSUMJn5/v37tWvXLh05ckSSVKRIEZUrVy49agMAAACQCt6uTgr0cbVaZhiGzkXGSJJye7tY/nH5/u0AALC3VP32iYyM1ObNm3X37l3VrVtXfn5+6VUXAAAAgFQaWLuQzTC86Jg4eb27QpJ0eGg9ebgQQAEAngwp/o20a9cuNW3aVBcuXJAkeXl5aeHChWrUqFG6FQcAAAAAAIDMKcV33xs6dKgKFiyov/76S9u3b1f9+vXVp0+f9KwNAAAAAAAAmVSKe0pt375dq1atUoUKFSRJ06dPV7Zs2RQZGSlvb+90KxAAAAAAAACZT4p7Sl27dk158uSxPPf19ZWHh4euXr2aLoUBAAAAAAAg80rVLIcHDhywzCklJdzJ4+DBg7p586ZlWZkyZdKuOgAAHmLcH8c1fv2JVG83oFaQzWTAAAAAAOwnVaFU/fr1ZRiG1bLmzZtbfjaZTIqPj0+bygAASIHIO3E6G3HnkbYDAAAAkHFSHEqdPHnyoW3u7TEFAIA9eLs6KdDH1WqZYRg6FxkjScrt7SKTyZTkdgAAAAAyTor/jzx//vxJLr9586bmz5+vadOmadu2bfSUAgDY1cDahWyG4UXHxMnr3RWSpMND68nDhQAKAAAAeNKkeKLz+61fv16dO3dWrly5NHbsWNWtW1ebNm1Ky9oAAAAAAACQSaXqn44vXLigmTNnatq0aYqMjFSbNm0UExOjpUuXqkSJEulVIwAAAAAAADKZFPeUatGihYoWLao9e/ZowoQJOnfunCZOnJietQEAAAAAACCTSnFPqRUrVqhv377q3bu3ChcunJ41AQAAAAAAIJNLcU+p8PBw3bx5UxUrVlTlypU1adIkXblyJT1rAwAAAAAAQCaV4lCqSpUq+uabb3T+/Hn17NlTCxYsUO7cuWU2mxUWFqabN2+mZ50AAAAAAADIRFJ99z0PDw+9/vrrCg8P1969ezVo0CCNHj1a/v7+euGFF9KjRgAAAAAAAGQyqQ6l7lW0aFGNGTNG//zzj+bPn59WNQEAAAAAACCTe6xQKpGjo6NatmypZcuWpcXuAAAAAAAAkMmlSSgFAAAAAAAApAahFAAAAAAAAOyOUAoAAAAAAAB2RygFAAAAAAAAuyOUAgAAAAAAgN0RSgEAAAAAAMDunDK6AAAAAACZX8SOCYrY8Xmqt/Op0E8+FfqnfUEAgAxHKAUAAAAg3ZljIhUfdfaRtgMAZE6EUgAAAADSnYOLtxw9A60XGobio89Jkhw9cksmU5LbAQAyJ0IpAAAAAOnOp0J/m2F45thonZ6cVZKUp8t+OTh7ZEBlAICMwkTnAAAAAAAAsDtCKQAAAAAAANgdoRQAAAAAAADsjlAKAAAAAAAAdkcoBQAAAAAAALsjlAIAAAAAAIDdEUoBAAAAAADA7gilAAAAAAAAYHeEUgAAAAAAALA7QikAAAAAAADYHaEUAAAAAAAA7I5QCgAAAAAAAHbnlNEFAAAAAACAFDIMKTIyo6t4OsXEyTsmOuHniAjJJQ0ikdhYOUVHJ+zP2fnx95dZpPAzmqGh1KhRo7R48WIdOnRIbm5uqlatmj755BMVLVpUknTt2jW9//77WrVqlc6cOSM/Pz+1bNlSH374oXx8fCz7OXPmjHr37q21a9fK09NTnTt31qhRo+TkROYGAAAAAMhEIiOlDh0yuoqnkovZ0OxDlxJ+PjFHcjA99j4dzWZVvHRJjnPnSg4pG4wWG3FCcREnU30sJ5+CcvYJSvV2GSI2NkXNMjS1+eOPPxQaGqrnn39ecXFxeueddxQSEqIDBw7Iw8ND586d07lz5zR27FiVKFFCp0+fVq9evXTu3Dn98MMPkqT4+Hg1a9ZMAQEB2rBhg86fP69OnTrJ2dlZH3/8cUa+PADAMy5ixwRF7Pg81dv5VOgnnwr9074gAAAAZDjDHCcj7s4jbZfZZGgotXLlSqvnM2fOlL+/v7Zv365atWqpVKlS+vHHHy3rCxUqpI8++kgdOnRQXFycnJyctGrVKh04cECrV69Wzpw5Va5cOX344YcaOnSohg8frixZstj7ZQEAIEkyx0QqPursI20HAACAzMnk4CSTk6vN8sSgKql1idtlNk/UK4qIiJAkZcuW7YFtvL29LUPzNm7cqNKlSytnzpyWNo0aNVLv3r21f/9+lS9fPn2LBgAgGQ4u3nL0DLReaBiKjz4nSXL0yC2ZbLuNO7h426M8AAAAZABnnyCbYXiGOV63TyV03HHNU0cmB8eMKM3unphQymw2q3///qpevbpKlSqVZJsrV67oww8/VI8ePSzLLly4YBVISbI8v3DhQpL7iYmJUUxMjOV55P9PwBUbG6vYFI57fBYkngvOCYCnTWxc3D0/xyrWwXj8fT7CNdG9dKjcS4daLTPHRuvc1/6SpJyv7ZKDs8cDjwcAjyutr4lp+f+I8TH/Dl+JOr1OrnnrPzN/iAGPzM1NmjEjo6t4KkXfjVPHUeskSaferiOPLI8ficTGxmr72rWqW7eunB9jonNzbLQuzQyWJOXqMjXZ/0d8WsTevCkFPXz+qycmlAoNDdW+ffsUHh6e5PrIyEg1a9ZMJUqU0PDhwx/rWKNGjdKIESNslq9atUru7u6Pte/MKCwsLKNLAIBUuRMvSQkTTf722yq5puHfN497TTSZ76j0///822+/yXBIuns2gPsYhpxu3croKp5KMfGSd0zCNXHN4iVySYNropOktUuXPtY+vO5sU0DUXCX+CXf1l1aKdciqC56v6abrc49b4jMlzt09yZ63AKzdiZciXRLCnt82bU67/0f08FDYli2PtQuT+Y5KuyV8j3/btOmp/3/EWyn8nf1EhFJ9+vTRL7/8ovXr1ytPnjw262/evKnGjRvLy8tLS5YssUofAwICtOW+N//ixYuWdUl5++23NXDgQMvzyMhI5c2bVyEhIfL2ZshEotjYWIWFhalhw4aPlfgCgL1F342TNqyWJDVqFJJm/wqWFtfEhJ5S+v/aGj31/woG2E1EhBw7d87oKp5K8Yah2YevSJIaH88hx8cMLwzD0OXLl+Xn5yfTI+7rlv9FXS6zJ+HJPbtwjr+uvBGT5PdnGblfypn0xrARP2uWdM/dyQEkjf9HtJ/EEWkPk6GhlGEYevPNN7VkyRKtW7dOBQsWtGkTGRmpRo0aycXFRcuWLZOrq3VaWLVqVX300Ue6dOmS/P0ThkOEhYXJ29tbJUqUSPK4Li4ucnFxsVnu7OxM+JIEzguAp42z+d+/cJydnOXsnHa/7h73mmjWv9s6OzvLgesrkDLOzim+1Tasmc3/DtczmRzk8Ji3QDebzf+/L5McHuE9MWToWtHD/1/QfStNkgzpWtEj8rgSIJNNAyTFwdk54TsC4IH4f0T7Sem5yNBQKjQ0VPPmzdNPP/0kLy8vyxxQPj4+cnNzU2RkpEJCQnTr1i3NmTNHkZGRlrTNz89Pjo6OCgkJUYkSJdSxY0eNGTNGFy5c0HvvvafQ0NAkgycAAAAAGeeO73XFu8Yk38Akxbve0R3f63K7kfwNkAAAT78MDaWmTp0qSapTp47V8hkzZqhLly7asWOHNm/eLEkKDg62anPy5EkVKFBAjo6O+uWXX9S7d29VrVpVHh4e6ty5sz744AO7vAYAAAAAKRfv8oBA6hHaAQCeXhk+fO9B6tSp89A2kpQ/f34tX748rcoCAADAk8bbW5ozJ6OreCrFxMSp44cJN2k4+9+GcnJ5vD8B4mNjtX3VKoWEhDzS8BLH839JK1s+vN3gD6Rc1R+hwmcQ8+ICeEo9EROdAwAAAA9kMjGR86OKibPcbUo+PtJjhlKKjVWch0fCvh4hlHL1aiTH8EDFR52TlNQ/QJvk6Bko1yKNJIc0vH0qAOCJw2yRAAAAAOzG5OCo7HXGJT67f60kKXudz2QikAKATI9QCgAAAIBdeQS3kn/zBXL0yGW13NEzUP7NF8gjuFUGVQYAsCeG7wEAAACwO4/gVnLNW19npuaQJOVs+bPc8jWghxQAPEPoKQUAAAAgQ9wbQLkG1iCQAoBnDKEUAAAAAAAA7I5QCgAAAAAAAHbHnFJPoXF/HNf49SdSvd2AWkEaWLtQOlQEAAAAAACQOoRST6HIO3E6G3HnkbYDAAAAAAB4EhBKPYW8XZ0U6ONqtcwwDJ2LjJEk5fZ2kclkSnI7AAAAAACAJwEpxVNoYO1CNsPwomPi5PXuCknS4aH15OHCWwsAAAAAAJ5cTHQOAAAAAAAAuyOUAgAAAAAAyECGOd7y852z4VbPMzNCKQAAAAAAgAwSfWyJzn5XxvL84tIW+nt6sKKPLcnAquyDUAoAAAAAACADRB9boku/tFV89Dmr5fFR53Tpl7aZPpgilAIAwI6e1a7ZAAAAsGaY43V13UBJRlJrJUlX1w3K1P+/SCgFAICdPMtdswEAAGDtztlwxUedfUALQ/FR/+jO2XC71WRvThldADKXiB0TFLHj81Rv51Ohn3wq9E/7ggDgCZHYNfv+fwlL7Jrt33yBPIJbZUxxAAAAsLv46PNp2u5pRCiFNGWOiXxI0pv8dgCQWT28a7ZJV9cNknvQCzI5ONq5OgAAAGQER49cadruaUQohTTl4OItR89A64WGYZm0zdEjt2QyJbkdAGRWqema7Za3tt3qAgAAQMZxDawhR89AxUedU9L/eGmSo2egXANr2Ls0uyGUQpryqdDfZhieOTZapydnlSTl6bJfDs4eGVAZAGQcumYDAADgfiYHR2WvM+7/p3gwyTqYSujMkb3OZ5m6Jz0TnQMAkM7omg0AAICkeAS3kn/zBTb/H+joGfhMzDlKTykAANIZXbMBIJkb4hj/XhP/mVkyyWkeuCEOgMzOI7iVXPPW15mpOSRJOVv+LLd8DTJ1D6lEhFIAAKQzumYDwMNviJM4B2lS2wFAZnfv/we6BtZ4Zv6/kFAKAAA7SOyafXXtAKs/vBw9A5W9zmeZvms2ACR5Q5wUbgcAyJwIpQAAsJNnuWs2ACR1QxwAwLONic4BALCjZ7VrNgAAAHA/QikAAAAAAADYHcP3AOBJZRhSJJO7PpKYOHnHRCf8HBEhuaTBr7vYWDlFRyfsz9n5MfYTLdNt49/anOMev7Znkbd3knfpAgAAwNODUAoAnlSRkVKHDhldxVPJxWxo9qFLCT+fmCM5PH544Wg2q+KlS3KcO1dyePSOxiZzvPxPxSf8HN5dYvjeo5kzR/LxyegqAAAA8BgYvgcAAAAAAAC7I5QCAAAAAACA3RFKAQAAAAAAwO6YUwoAnlTe3gnz5iDVYmLi1PHDMEnS2f82lFMaTHQeHxur7atWKSQkRA6PMdG5ERutS9/klyTle+NbmZw9Hru2Z5K3d0ZXAAAAgMdEKAUATyqTiYmcH1VMnCJd/j/s8fFJs7vvxXl4JOzvse6+5yTDzfRvbYRSAAAAeEYxfA8AAAAAAAB2RygFAAAAAAAAuyOUQrozzPGWn++cDbd6DgAAAAAAnk2EUkhX0ceW6Ox3ZSzPLy5tob+nByv62JIMrAoAAAAAAGQ0Qimkm+hjS3Tpl7aKjz5ntTw+6pwu/dKWYAoAAAAAgGcYoRTShWGO19V1AyUZSa2VJF1dN4ihfAAAAAAAPKMIpZAu7pwNV3zU2Qe0MBQf9Y/unA23W00AAAAAAODJQSiFdBEffT5N2wEAAAAAgMzFKaMLQObk6JErTdsBAADg0cSb/51O4c8TV9WwqL8cHUwZWBEAPNsidkxQxI7PrRca/16r/5lZUjLZXqd9KvSTT4X+6VydfRFKIV24BtaQo2eg4qPOKel5pUxy9AyUa2ANe5cGAADwzFi897z6Ldlned502hbl8XHVhJal9FJp/nEQADKCOSbygdPd3H+zsHu3y2wIpZAuTA6Oyl5nnC790laSSdbBVELim73OZzI5OGZEeQAAAJne4r3n9cqsbTb/PHg24o5embVNizo/RzAFABnAwcVbjp6Bj7RdZkMohXTjEdxK/s0X6OraAVZJr6NnoLLX+Uwewa0ysDoAAIDMK95sqP/SfcneB9kkacBP+/RiyQCG8gGAnflU6J/phuE9KkKpe0VEWI3jfKrExMk7Jjrh54gIySUN3trYWDlFRyfsz9n5kXbh4VdPri/+qTMzgiSTlLPpArkF1k3oIRUR8fg1Pku8vZMcVwwAAHC/P09c1T8Rd5Jdb0j6+8Yd/XniquoE57BfYQAA3INQ6l5vvPHI4UtGczEbmn3oUsLPJ+ZIafAvXo5msypeuiTHuXMlh0e/UaODOV45T8VLktz+mCaTw8zHru2ZNGeO5OOT0VUASAUmsQSQUc7fjEnTdgAApAdCKQAA0gmTWALIKLm8XNK0HQAA6YFQCgCAdMIklgAySs2g7Mrj46qzEXeSuQ+ylMfXVTWDstu7NAAALAilAABIJ0xiCSCjODqYNKFlKb0ya1sy90GWxr9YiknOAQAZilDqXt98kzCZ9FMoJiZOHT8MkySd/W9DOaXBROfxsbHavmqVQkJC5PAYc20ZsdG69E1+SVK+N76VydnjsWt7Jj2ln00AAJAxXiqdS4s6P6d+S/bpbOS/k57n8XXV+BdL6aXSuTKwOgAACKWs+fg8vX/4x8Qp0uX/wx4fnzS7+16ch0fC/h5nAvhYJxlupn9rI5QCAACwi5dK51KD4Bzy/e9KSdLybpXUsKg/PaQAAE+ER7+lGgAAAIAn3r0BVM2g7ARSAIAnBqEUAAAAAAAA7I5QCgAAAAAAAHZHKAUAAAAAAAC7I5QCAAAAAACA3RFKAQAAAAAAwO4IpQAAAAAAAGB3hFIAAAAAAACwO0IpAAAAAAAA2B2hFAAAAAAAAOyOUAoAAAAAAAB2RygFAMh04s2G5ec/T1y1eg4AAADgyeCU0QUgc4nYMUEROz63Xmj8+8fgPzNLSiaTzXY+FfrJp0L/dK4OwLNg8d7z6rdkn+V502lblMfHVRNaltJLpXNlYGUAAAAA7kUohTRljolUfNTZZNfHR59LdjsAeFyL957XK7O26f5+UWcj7uiVWdu0qPNzBFMAAADAE4JQCmnKwcVbjp6Bj7QdADyOeLOh/kv32QRSkmRIMkka8NM+vVgy4P/au/sgrcr7buDfBZZdCOwyEAEJKGgiaEVEMLBYQSmCjmNCYWo0NoOWxIYstrIPMTGNKHEiUZOorZg4moA2MkZt0dR3ggGEQFUyOFEjPmhaUhXISwVBWRZ2nz98WLMB4gtwbhY+n5mduc91rnPdv3MGrr35cp1zp22bXVdsAgAAxRJKsU9Vn3Sp2/CAknjyld/nfzZu3eP+piS/eWNrnnzl9znt4x8trjAAAGC3POgcgIPC62/W79N+AADA/iWUAuCgcHjnin3aDwAA2L+EUgAcFE49qlt6V1dmT0+LKkvSp0tlTj2qW5FlAQAAeyCUAuCg0LZNWW4cf3yS7BJM7dy+4dPHe8g5AAAcIIRSABw0Jgw8PPdOGppeVZUt2nt3qcy9k4ZmwsDDS1QZAADwp3z7HgAHlQkDD8+Yj380Xa54NEny8ORP5oz+3a2QAgCAA4yVUgAcdP44gDr1qG4CKQAAOAAJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMIJpQAAAAAonFAKAAAAgMKVNJSaNWtWTj755HTu3Dndu3fP+PHjs3r16hZ9tm7dmtra2nTr1i2dOnXKxIkTs379+hZ91q5dm7PPPjsdO3ZM9+7d8+Uvfznbt28v8lQAAAAA+ABKGkotXrw4tbW1WbFiRRYsWJCGhoaMHTs2W7Zsae4zbdq0/Md//EfuvffeLF68OK+99lomTJjQvH/Hjh05++yzs23btvz85z/PHXfckblz52bGjBmlOCUAAAAA3od2pXzzRx99tMX23Llz071796xcuTIjR47Mxo0b84Mf/CDz5s3L6NGjkyRz5szJsccemxUrVmT48OF5/PHH88ILL+SnP/1pevTokRNPPDFXX311vvKVr+Sqq65K+/btS3FqAAAAAPwZJQ2l/tTGjRuTJF27dk2SrFy5Mg0NDRkzZkxznwEDBuSII47I8uXLM3z48CxfvjwDBw5Mjx49mvuMGzcuU6ZMyfPPP5/Bgwfv8j719fWpr69v3t60aVOSpKGhIQ0NDfvl3Pa3rdverftn/3d9xnzisLRtU7ZXY+68Fq31mgCHroY/uoW7YXtDGto07f2Y5kSgldrXc6L5EGitfEYszvu9HgdMKNXY2JhLL700p5xySo4//vgkybp169K+fft06dKlRd8ePXpk3bp1zX3+OJDauX/nvt2ZNWtWZs6cuUv7448/no4dO+7tqRRu+e+S218uS/JOCPWpub9It/ZNmXx0U2o+uvfjL1iwYO8HASjQ1h3JzjvUH3vs8VS23XdjmxOB1mZ/zYnmQ6C18RmxOG+99db76nfAhFK1tbV57rnnsnTp0v3+Xpdffnnq6uqatzdt2pQ+ffpk7Nixqaqq2u/vvy/d/9y6XP/kqvxpvvuHbWW5/ldlufuCEzP++J4fauyGhoYsWLAgZ5xxRsrLy/e+WICCbNm2Pfn5T5Mk48aNzUfa7/2vO3Mi0Frt6znRfAi0Vj4jFmfnHWnv5YAIpaZOnZoHH3wwS5YsSe/evZvbe/bsmW3btuWNN95osVpq/fr16dmzZ3Ofp556qsV4O7+db2efP1VRUZGKiopd2svLy1vVH6IdjU35Pw++uEsglSRNeWfd1PSHXsyEQb336la+1nZdAMob353zytuVp7x83/26MycCrc3+mhPNh0Br4zNicd7vtSjpt+81NTVl6tSpmT9/fp544on069evxf4hQ4akvLw8CxcubG5bvXp11q5dm5qamiRJTU1NfvnLX2bDhg3NfRYsWJCqqqocd9xxxZxIiTz5yu/zPxu37nF/U5LfvLE1T77y++KKAgAAAHgfSrpSqra2NvPmzcsDDzyQzp07Nz8Dqrq6Oh06dEh1dXUmT56curq6dO3aNVVVVbnkkktSU1OT4cOHJ0nGjh2b4447Lp/73Ody3XXXZd26dfn617+e2tra3a6GOpi8/mb9e3f6AP0AAAAAilLSUOp73/tekuS0005r0T5nzpxceOGFSZIbbrghbdq0ycSJE1NfX59x48bllltuae7btm3bPPjgg5kyZUpqamrykY98JJMmTco3vvGNok6jZA7v/P5Ct/fbDwAAAKAoJQ2lmpre++sXKysrM3v27MyePXuPfY488sg8/PDD+7K0VuHUo7qld3VlXt24dbfPlSpL0rtLZU49qlvRpQEAAAD8WSV9phR7p22bstw4/vgk7wRQf2zn9g2fPn6vHnIOAAAAsD8IpVq5CQMPz72ThqZXVWWL9t5dKnPvpKGZMPDwElUGAAAAsGclvX2PfWPCwMMz5uMfTZcrHk2SPDz5kzmjf3crpAAAAIADllDqIPHHAdSpR3UTSAEAAHDI+u7il3PDkldatP3xc637X/tEysp2/XfztJFHpW7U0fu9Pt4hlAIAAAAOKpu2bs+rG7fucf9rm+r3eBzFEUoBAAAAB5Wqynb5WHXle3fczXEUx9UGAAAADip1o452G14r4Nv3AAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAAChcu1IXAAAA7BvfXfxybljySou2pqam5tf9r30iZWVluxw3beRRqRt19H6vDwD+mFAKAAAOEpu2bs+rG7fucf9rm+r3eBwAFE0oBQAAB4mqynb5WHXlhzoOAIrmtw8AABwk6kYd7TY8AFoNDzoHAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAKJ5QCAAAAoHBCKQAAAAAK167UBQDA3vju4pdzw5JXWrQ1NTU1v+5/7RMpKyvb5bhpI49K3aij93t9AADA7gmlAGjVNm3dnlc3bt3j/tc21e/xOAAAoHSEUgC0alWV7fKx6soPdRwAAFA6PpED0KrVjTrabXgAANAKedA5AAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQuHalLoAP7ruLX84NS15p0dbU1NT8uv+1T6SsrGyX46aNPCp1o47e7/UBAAAAvBehVCu0aev2vLpx6x73v7apfo/HAQAAABwISnr73pIlS3LOOeekV69eKSsry/33399i/+bNmzN16tT07t07HTp0yHHHHZfvf//7Lfps3bo1tbW16datWzp16pSJEydm/fr1BZ5F8aoq2+Vj1ZUf+KeqUgYJAAAAHBhKmlJs2bIlgwYNyt/93d9lwoQJu+yvq6vLE088kR/96Efp27dvHn/88XzpS19Kr1698qlPfSpJMm3atDz00EO59957U11dnalTp2bChAlZtmxZ0adTmLpRR7sNDwAAAGjVShpKnXXWWTnrrLP2uP/nP/95Jk2alNNOOy1JcvHFF+fWW2/NU089lU996lPZuHFjfvCDH2TevHkZPXp0kmTOnDk59thjs2LFigwfPryI0wAAAADgAzqgv31vxIgR+clPfpJXX301TU1N+dnPfpaXXnopY8eOTZKsXLkyDQ0NGTNmTPMxAwYMyBFHHJHly5eXqmwAAAAA3sMB/ZChf/mXf8nFF1+c3r17p127dmnTpk1uu+22jBw5Mkmybt26tG/fPl26dGlxXI8ePbJu3bo9jltfX5/6+ncfBr5p06YkSUNDQxoaGvb9ibRSO6+FawJgTgTYyXwI8C5z4u693+txwIdSK1asyE9+8pMceeSRWbJkSWpra9OrV68Wq6M+qFmzZmXmzJm7tD/++OPp2LHj3pR8UFqwYEGpSwA4YJgTAd5hPgR4lzmxpbfeeut99TtgQ6m33347X/va1zJ//vycffbZSZITTjghq1atyre//e2MGTMmPXv2zLZt2/LGG2+0WC21fv369OzZc49jX3755amrq2ve3rRpU/r06ZOxY8emqqpqv51Ta9PQ0JAFCxbkjDPOSHl5eanLASgpcyLAO8yHAO8yJ+7ezjvS3ssBG0rtvJWuTZuWj71q27ZtGhsbkyRDhgxJeXl5Fi5cmIkTJyZJVq9enbVr16ampmaPY1dUVKSiomKX9vLycn+IdsN1AXiXORHgHeZDgHeZE1t6v9eipKHU5s2bs2bNmubtX//611m1alW6du2aI444IqNGjcqXv/zldOjQIUceeWQWL16cO++8M9/97neTJNXV1Zk8eXLq6urStWvXVFVV5ZJLLklNTY1v3gMAAAA4gJU0lHrmmWdy+umnN2/vvKVu0qRJmTt3bu6+++5cfvnlueCCC/KHP/whRx55ZL75zW/mi1/8YvMxN9xwQ9q0aZOJEyemvr4+48aNyy233FL4uQAAAADw/pU0lDrttNPS1NS0x/09e/bMnDlz/uwYlZWVmT17dmbPnr2vywMAAABgP2nz3l0AAAAAYN8SSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQOKEUAAAAAIUTSgEAAABQuHalLuBA0NTUlCTZtGlTiSs5sDQ0NOStt97Kpk2bUl5eXupyAErKnAjwDvMhwLvMibu3M1/ZmbfsiVAqyZtvvpkk6dOnT4krAQAAADg4vPnmm6murt7j/rKm94qtDgGNjY157bXX0rlz55SVlZW6nAPGpk2b0qdPn/zmN79JVVVVqcsBKClzIsA7zIcA7zIn7l5TU1PefPPN9OrVK23a7PnJUVZKJWnTpk169+5d6jIOWFVVVf5yAfx/5kSAd5gPAd5lTtzVn1shtZMHnQMAAABQOKEUAAAAAIUTSrFHFRUVufLKK1NRUVHqUgBKzpwI8A7zIcC7zIl7x4POAQAAACiclVIAAAAAFE4oBQAAAEDhhFIAAAAAFE4oxW7Nnj07ffv2TWVlZYYNG5annnqq1CUBlMSSJUtyzjnnpFevXikrK8v9999f6pIASmLWrFk5+eST07lz53Tv3j3jx4/P6tWrS10WQEl873vfywknnJCqqqpUVVWlpqYmjzzySKnLanWEUuzixz/+cerq6nLllVfmF7/4RQYNGpRx48Zlw4YNpS4NoHBbtmzJoEGDMnv27FKXAlBSixcvTm1tbVasWJEFCxakoaEhY8eOzZYtW0pdGkDhevfunW9961tZuXJlnnnmmYwePTqf/vSn8/zzz5e6tFbFt++xi2HDhuXkk0/OzTffnCRpbGxMnz59cskll+SrX/1qiasDKJ2ysrLMnz8/48ePL3UpACX329/+Nt27d8/ixYszcuTIUpcDUHJdu3bN9ddfn8mTJ5e6lFbDSila2LZtW1auXJkxY8Y0t7Vp0yZjxozJ8uXLS1gZAAAHko0bNyZ55x9hAIeyHTt25O67786WLVtSU1NT6nJalXalLoADy+9+97vs2LEjPXr0aNHeo0ePvPjiiyWqCgCAA0ljY2MuvfTSnHLKKTn++ONLXQ5ASfzyl79MTU1Ntm7dmk6dOmX+/Pk57rjjSl1WqyKUAgAAPpDa2to899xzWbp0aalLASiZ/v37Z9WqVdm4cWPuu+++TJo0KYsXLxZMfQBCKVr46Ec/mrZt22b9+vUt2tevX5+ePXuWqCoAAA4UU6dOzYMPPpglS5akd+/epS4HoGTat2+fj3/840mSIUOG5Omnn85NN92UW2+9tcSVtR6eKUUL7du3z5AhQ7Jw4cLmtsbGxixcuNC9sQAAh7CmpqZMnTo18+fPzxNPPJF+/fqVuiSAA0pjY2Pq6+tLXUarYqUUu6irq8ukSZMydOjQfPKTn8yNN96YLVu25KKLLip1aQCF27x5c9asWdO8/etf/zqrVq1K165dc8QRR5SwMoBi1dbWZt68eXnggQfSuXPnrFu3LklSXV2dDh06lLg6gGJdfvnlOeuss3LEEUfkzTffzLx587Jo0aI89thjpS6tVSlrampqKnURHHhuvvnmXH/99Vm3bl1OPPHE/PM//3OGDRtW6rIACrdo0aKcfvrpu7RPmjQpc+fOLb4ggBIpKyvbbfucOXNy4YUXFlsMQIlNnjw5CxcuzOuvv57q6uqccMIJ+cpXvpIzzjij1KW1KkIpAAAAAArnmVIAAAAAFE4oBQAAAEDhhFIAAAAAFE4oBQAAAEDhhFIAAAAAFE4oBQAAAEDhhFIAAAAAFE4oBQAAAEDhhFIAAJTUaaedlksvvbTUZQAABRNKAQAHlbKysj/7c9VVV+2X973wwgszfvz4/TL2hzF37tx06dJln/UDANjX2pW6AACAfen1119vfv3jH/84M2bMyOrVq5vbOnXq1Py6qakpO3bsSLt2PhIBABTNSikA4KDSs2fP5p/q6uqUlZU1b7/44ovp3LlzHnnkkQwZMiQVFRVZunRpGhsbM2vWrPTr1y8dOnTIoEGDct999zWPuWPHjkyePLl5f//+/XPTTTc177/qqqtyxx135IEHHmhekbVo0aL813/9V8rKynLPPffk1FNPTYcOHXLyySfnpZdeytNPP52hQ4emU6dOOeuss/Lb3/62xXncfvvtOfbYY1NZWZkBAwbklltuad63c9x///d/z+mnn56OHTtm0KBBWb58eZJk0aJFueiii7Jx48a9XiH2xhtv5POf/3wOO+ywVFVVZfTo0Xn22WeTJC+99FLKysry4osvtjjmhhtuyNFHH928/dxzz+Wss85Kp06d0qNHj3zuc5/L7373uw9VDwBw8BBKAQCHnK9+9av51re+lV/96lc54YQTMmvWrNx55535/ve/n+effz7Tpk3L3/7t32bx4sVJksbGxvTu3Tv33ntvXnjhhcyYMSNf+9rXcs899yRJpk+fnnPPPTdnnnlmXn/99bz++usZMWJE8/tdeeWV+frXv55f/OIXadeuXT772c/msssuy0033ZQnn3wya9asyYwZM5r733XXXZkxY0a++c1v5le/+lWuueaaXHHFFbnjjjtanMc//dM/Zfr06Vm1alWOOeaYnH/++dm+fXtGjBiRG2+8MVVVVc31TJ8+/UNdq7/5m7/Jhg0b8sgjj2TlypU56aST8ld/9Vf5wx/+kGOOOSZDhw7NXXfd1eKYu+66K5/97GeTvBNqjR49OoMHD84zzzyTRx99NOvXr8+55577oeoBAA4e1qoDAIecb3zjGznjjDOSJPX19bnmmmvy05/+NDU1NUmSo446KkuXLs2tt96aUaNGpby8PDNnzmw+vl+/flm+fHnuueeenHvuuenUqVM6dOiQ+vr69OzZc5f3mz59esaNG5ck+cd//Mecf/75WbhwYU455ZQkyeTJkzN37tzm/ldeeWW+853vZMKECc3v98ILL+TWW2/NpEmTWox79tlnJ0lmzpyZv/iLv8iaNWsyYMCAFqvEPqylS5fmqaeeyoYNG1JRUZEk+fa3v537778/9913Xy6++OJccMEFufnmm3P11VcneWf11MqVK/OjH/0oSXLzzTdn8ODBueaaa5rH/eEPf5g+ffrkpZdeyjHHHPOh6wMAWjehFABwyBk6dGjz6zVr1uStt95qDql22rZtWwYPHty8PXv27Pzwhz/M2rVr8/bbb2fbtm058cQT39f7nXDCCc2ve/TokSQZOHBgi7YNGzYkSbZs2ZKXX345kydPzhe+8IXmPtu3b091dfUexz388MOTJBs2bMiAAQPeV13v5dlnn83mzZvTrVu3Fu1vv/12Xn755STJeeedl+nTp2fFihUZPnx47rrrrpx00knNNTz77LP52c9+1uJZXju9/PLLQikAOIQJpQCAQ85HPvKR5tebN29Okjz00EP52Mc+1qLfztVBd999d6ZPn57vfOc7qampSefOnXP99dfnP//zP9/X+5WXlze/Lisr221bY2Nji3puu+22DBs2rMU4bdu2fc9xd46zL2zevDmHH354Fi1atMu+nd/Y17Nnz4wePTrz5s3L8OHDM2/evEyZMqXFGOecc06uvfbaXcbYGaQBAIcmoRQAcEg77rjjUlFRkbVr12bUqFG77bNs2bKMGDEiX/rSl5rbdq4U2ql9+/bZsWPHXtfTo0eP9OrVK6+88kouuOCCDz3OvqjnpJNOyrp169KuXbv07dt3j/0uuOCCXHbZZTn//PPzyiuv5Lzzzmsxxr/927+lb9++vuUQAGjBJwMA4JDWuXPnTJ8+PdOmTUtjY2P+8i//Mhs3bsyyZctSVVWVSZMm5ROf+ETuvPPOPPbYY+nXr1/+9V//NU8//XT69evXPE7fvn3z2GOPZfXq1enWrdsut9p9EDNnzsw//MM/pLq6OmeeeWbq6+vzzDPP5H//939TV1f3vsbo27dvNm/enIULF2bQoEHp2LFjOnbsuNu+O3bsyKpVq1q0VVRUZMyYMampqcn48eNz3XXX5Zhjjslrr72Whx56KH/913/dfBvkhAkTMmXKlEyZMiWnn356evXq1TxObW1tbrvttpx//vm57LLL0rVr16xZsyZ33313br/99l1WfwEAhw7fvgcAHPKuvvrqXHHFFZk1a1aOPfbYnHnmmXnooYeaQ6e///u/z4QJE/KZz3wmw4YNy+9///sWq6aS5Atf+EL69++foUOH5rDDDsuyZcs+dD2f//znc/vtt2fOnDkZOHBgRo0alblz57YIwd7LiBEj8sUvfjGf+cxncthhh+W6667bY9/Nmzdn8ODBLX7OOeeclJWV5eGHH87IkSNz0UUX5Zhjjsl5552X//7v/25+NlbyTrB3zjnn5Nlnn91ldVevXr2ybNmy7NixI2PHjs3AgQNz6aWXpkuXLmnTxkdRADiUlTU1NTWVuggAAAAADi3+ewoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACicUAoAAACAwgmlAAAAACjc/wMEz5ZXiAUbnwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# --- Plot APOs and 95% CIs for all treatment levels (tuned vs. untuned pipeline) ---\n", + "\n", + "plt.figure(figsize=(12, 7))\n", + "palette = sns.color_palette(\"colorblind\")\n", + "\n", + "# Collect results for each treatment level from both models\n", + "treatment_levels_plot = np.unique(d)\n", + "n_levels = len(treatment_levels_plot)\n", + "\n", + "# Prepare DataFrame for plotting\n", + "df_apo_plot = pd.DataFrame({\n", + " 'treatment_level': np.tile(treatment_levels_plot, 2),\n", + " 'APO': np.concatenate([dml_apos_untuned.coef, dml_apos_tuned.coef]),\n", + " 'ci_lower': np.concatenate([dml_apos_untuned.confint().iloc[:, 0], dml_apos_tuned.confint().iloc[:, 0]]),\n", + " 'ci_upper': np.concatenate([dml_apos_untuned.confint().iloc[:, 1], dml_apos_tuned.confint().iloc[:, 1]]),\n", + " 'Model': ['Untuned Pipeline'] * n_levels + ['Tuned Pipeline'] * n_levels\n", + "})\n", + "\n", + "jitter_strength = 0.12\n", + "models = df_apo_plot['Model'].unique()\n", + "n_models = len(models)\n", + "\n", + "for i, model in enumerate(models):\n", + " df = df_apo_plot[df_apo_plot['Model'] == model]\n", + " jitter = (i - (n_models - 1) / 2) * jitter_strength\n", + " x_jittered = df['treatment_level'] + jitter\n", + " plt.errorbar(\n", + " x_jittered,\n", + " df['APO'],\n", + " yerr=[df['APO'] - df['ci_lower'], df['ci_upper'] - df['APO']],\n", + " fmt='o',\n", + " capsize=5,\n", + " capthick=2,\n", + " ecolor=palette[i % len(palette)],\n", + " color=palette[i % len(palette)],\n", + " label=f\"{model} APO ±95% CI\",\n", + " zorder=2\n", + " )\n", + "\n", + "# Add true APOs as horizontal lines\n", + "x_range = plt.xlim()\n", + "total_width = x_range[1] - x_range[0]\n", + "for i, level in enumerate(treatment_levels_plot):\n", + " line_width = 0.6\n", + " x_center = level\n", + " x_start = x_center - line_width/2\n", + " x_end = x_center + line_width/2\n", + " xmin_rel = max(0, (x_start - x_range[0]) / total_width)\n", + " xmax_rel = min(1, (x_end - x_range[0]) / total_width)\n", + " plt.axhline(y=apos[int(level)], color='red', linestyle='-', alpha=0.7,\n", + " xmin=xmin_rel, xmax=xmax_rel,\n", + " linewidth=3, label='True APO' if i == 0 else \"\")\n", + "\n", + "plt.title('Estimated APO and 95% Confidence Interval by Treatment Level\\n(Pipeline: Tuned vs. Untuned)')\n", + "plt.xlabel('Treatment Level')\n", + "plt.ylabel('APO')\n", + "plt.xticks(treatment_levels_plot)\n", + "plt.legend()\n", + "plt.grid(True)\n", + "plt.tight_layout()\n", + "plt.show()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/examples/py_double_ml_irm_vs_apo.ipynb b/doc/examples/py_double_ml_irm_vs_apo.ipynb index e9be82b1..6c03ab5b 100644 --- a/doc/examples/py_double_ml_irm_vs_apo.ipynb +++ b/doc/examples/py_double_ml_irm_vs_apo.ipynb @@ -35,7 +35,7 @@ "source": [ "## Data\n", "\n", - "We rely on the [make_irm_data](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_irm_data.html) go generate data with a binary treatment." + "We rely on the [make_irm_data](https://docs.doubleml.org/stable/api/generated/doubleml.irm.datasets.make_irm_data.html#doubleml.irm.datasets.make_irm_data) go generate data with a binary treatment." ] }, { @@ -401,7 +401,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In this example the average potential outcome of the control group is zero (as can be seen in the outcome definition, see [documentation](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_irm_data.html#doubleml.datasets.make_irm_data)).\n", + "In this example the average potential outcome of the control group is zero (as can be seen in the outcome definition, see [documentation](https://docs.doubleml.org/stable/api/generated/doubleml.irm.datasets.make_irm_data.html)).\n", "Let us visualize the effects" ] }, diff --git a/doc/examples/py_double_ml_meets_flaml.ipynb b/doc/examples/py_double_ml_meets_flaml.ipynb index d846f003..2aee0b79 100644 --- a/doc/examples/py_double_ml_meets_flaml.ipynb +++ b/doc/examples/py_double_ml_meets_flaml.ipynb @@ -24,7 +24,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We create synthetic data using the [make_plr_CCDDHNR2018()](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_plr_CCDDHNR2018.html) process, with $1000$ observations of $50$ covariates as well as $1$ treatment variable and an outcome. We calibrate the process such that hyperparameter tuning becomes more important." + "We create synthetic data using the [make_plr_CCDDHNR2018](https://docs.doubleml.org/stable/api/generated/doubleml.plm.datasets.make_plr_CCDDHNR2018.html#doubleml.plm.datasets.make_plr_CCDDHNR2018) process, with $1000$ observations of $50$ covariates as well as $1$ treatment variable and an outcome. We calibrate the process such that hyperparameter tuning becomes more important." ] }, { diff --git a/doc/examples/py_double_ml_multiway_cluster.ipynb b/doc/examples/py_double_ml_multiway_cluster.ipynb index b79abfd8..d808a9e3 100644 --- a/doc/examples/py_double_ml_multiway_cluster.ipynb +++ b/doc/examples/py_double_ml_multiway_cluster.ipynb @@ -118,11 +118,11 @@ "\\end{aligned}$$\n", "and $\\alpha_{ij}^V, \\alpha_{i}^V, \\alpha_{j}^V \\sim \\mathcal{N}(0, 1)$.\n", "\n", - "Data from this DGP can be generated with the [make_pliv_multiway_cluster_CKMS2021()](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_pliv_multiway_cluster_CKMS2021.html#doubleml.datasets.make_pliv_multiway_cluster_CKMS2021) function from [DoubleML](https://docs.doubleml.org/stable/index.html).\n", + "Data from this DGP can be generated with the [make_pliv_multiway_cluster_CKMS2021()](https://docs.doubleml.org/stable/api/generated/doubleml.plm.datasets.make_pliv_multiway_cluster_CKMS2021.html) function from [DoubleML](https://docs.doubleml.org/stable/index.html).\n", "Analogously to [Chiang et al. (2021, Section 5)](https://doi.org/10.1080/07350015.2021.1895815)\n", "we use the following parameter setting:\n", "$\\theta=1.0$, $N=M=25$, $p_x=100$, $\\pi_{10}=1.0$, $\\omega_X = \\omega_{\\varepsilon} = \\omega_V = \\omega_v = (0.25, 0.25)$, $s_X = s_{\\varepsilon v} = 0.25$ and the $j$-th entries of the $p_x$-vectors $\\zeta_0 = \\pi_{20} = \\xi_0$ are $(\\zeta_{0})_j = 0.5^j$.\n", - "This are also the default values of [make_pliv_multiway_cluster_CKMS2021()](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_pliv_multiway_cluster_CKMS2021.html#doubleml.datasets.make_pliv_multiway_cluster_CKMS2021)." + "This are also the default values of [make_pliv_multiway_cluster_CKMS2021()](https://docs.doubleml.org/stable/api/generated/doubleml.plm.datasets.make_pliv_multiway_cluster_CKMS2021.html)." ] }, { @@ -444,7 +444,7 @@ "$$\n", "\\omega_2^X = \\omega_2^\\varepsilon = \\omega_2^v = \\omega_2^V = 0.\n", "$$\n", - "Again we can simulate this data with [make_pliv_multiway_cluster_CKMS2021()](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_pliv_multiway_cluster_CKMS2021.html#doubleml.datasets.make_pliv_multiway_cluster_CKMS2021). To prepare the data-backend for one-way clustering, we only have to alter the `cluster_cols` to be `'cluster_var_i'`." + "Again we can simulate this data with [make_pliv_multiway_cluster_CKMS2021()](https://docs.doubleml.org/stable/api/generated/doubleml.plm.datasets.make_pliv_multiway_cluster_CKMS2021.html#doubleml.plm.datasets.make_pliv_multiway_cluster_CKMS2021). To prepare the data-backend for one-way clustering, we only have to alter the `cluster_cols` to be `'cluster_var_i'`." ] }, { diff --git a/doc/examples/py_double_ml_sensitivity_booking.ipynb b/doc/examples/py_double_ml_sensitivity_booking.ipynb index 40363b77..60fcdb0b 100644 --- a/doc/examples/py_double_ml_sensitivity_booking.ipynb +++ b/doc/examples/py_double_ml_sensitivity_booking.ipynb @@ -36,7 +36,7 @@ "source": [ "## Load Data\n", "\n", - "We will simulate a data set according to a data generating process (DGP), which is available and documented in the [DoubleML](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_confounded_irm_data.html#doubleml.datasets.make_confounded_irm_data) for Python. We will parametrize the DGP in a way that it roughly mimics patterns of the data used in the original analysis." + "We will simulate a data set according to a data generating process (DGP), which is available and documented in the [DoubleML](https://docs.doubleml.org/stable/api/generated/doubleml.irm.datasets.make_confounded_irm_data.html#doubleml.irm.datasets.make_confounded_irm_data) for Python. We will parametrize the DGP in a way that it roughly mimics patterns of the data used in the original analysis." ] }, { diff --git a/doc/guide/learners.rst b/doc/guide/learners.rst index bb34f239..4489e545 100644 --- a/doc/guide/learners.rst +++ b/doc/guide/learners.rst @@ -16,778 +16,14 @@ separately for :ref:`Python ` and :ref:`R `. Python: Learners and hyperparameters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Minimum requirements for learners -################################# +.. include:: learners/python/learners_overview.inc -The minimum requirement for a learner to be used for nuisance models in the :ref:`DoubleML ` -package is - -* The implementation of a ``fit()`` and ``predict()`` method. - Some models, like :py:class:`doubleml.DoubleMLIRM` and :py:class:`doubleml.DoubleMLIIVM` require classifiers. -* In case of classifiers, the learner needs to come with a ``predict_proba()`` instead of, or in addition to, a - ``predict()`` method, see for example :py:meth:`sklearn.ensemble.RandomForestClassifier.predict_proba`. -* In order to be able to use the ``set_ml_nuisance_params()`` method of :ref:`DoubleML ` classes the - learner additionally needs to come with a ``set_params()`` method, - see for example :py:meth:`sklearn.ensemble.RandomForestRegressor.set_params`. -* We further rely on the function :py:func:`sklearn.base.clone` which adds the requirement of a ``get_params()`` - method for a learner in order to be used for nuisance models of :ref:`DoubleML ` model classes. - -Most learners from `scikit-learn `_ satisfy all these minimum requirements. - -Specifying learners and set hyperparameters -########################################### - -The learners are set during initialization of the :ref:`DoubleML ` model classes -:py:class:`doubleml.DoubleMLPLR`, :py:class:`doubleml.DoubleMLPLIV`, -:py:class:`doubleml.DoubleMLIRM` and :py:class:`doubleml.DoubleMLIIVM`. -Lets simulate some data and consider the partially linear regression model. -We need to specify learners for the nuisance functions :math:`g_0(X) = E[Y|X]` and :math:`m_0(X) = E[D|X]`, -for example :py:class:`sklearn.ensemble.RandomForestRegressor`. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - import doubleml as dml - from doubleml.plm.datasets import make_plr_CCDDHNR2018 - from sklearn.ensemble import RandomForestRegressor - - np.random.seed(1234) - ml_l = RandomForestRegressor() - ml_m = RandomForestRegressor() - data = make_plr_CCDDHNR2018(alpha=0.5, return_type='DataFrame') - obj_dml_data = dml.DoubleMLData(data, 'y', 'd') - dml_plr_obj = dml.DoubleMLPLR(obj_dml_data, ml_l, ml_m) - dml_plr_obj.fit().summary - -Without further specification of the hyperparameters, default values are used. To set hyperparameters: - -* We can also use pre-parametrized learners, like ``RandomForestRegressor(n_estimators=10)``. -* Alternatively, hyperparameters can also be set after initialization via the method - ``set_ml_nuisance_params(learner, treat_var, params)`` - - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - np.random.seed(1234) - dml_plr_obj = dml.DoubleMLPLR(obj_dml_data, - RandomForestRegressor(n_estimators=10), - RandomForestRegressor()) - print(dml_plr_obj.fit().summary) - - np.random.seed(1234) - dml_plr_obj = dml.DoubleMLPLR(obj_dml_data, - RandomForestRegressor(), - RandomForestRegressor()) - dml_plr_obj.set_ml_nuisance_params('ml_l', 'd', {'n_estimators': 10}); - print(dml_plr_obj.fit().summary) - -Setting treatment-variable-specific or fold-specific hyperparameters: - -* In the multiple-treatment case, the method ``set_ml_nuisance_params(learner, treat_var, params)`` can be used to set - different hyperparameters for different treatment variables. -* The method ``set_ml_nuisance_params(learner, treat_var, params)`` accepts dicts and lists for ``params``. - A dict should be provided if for each fold the same hyperparameters should be used. - Fold-specific parameters are supported. To do so, provide a nested list as ``params``, where the outer list is of - length ``n_rep`` and the inner list of length ``n_folds``. - - -Hyperparameter tuning -##################### - -Parameter tuning of learners for the nuisance functions of :ref:`DoubleML ` models can be done via -the ``tune()`` method. -To illustrate the parameter tuning, we generate data from a sparse partially linear regression model. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - import doubleml as dml - import numpy as np - - np.random.seed(3141) - n_obs = 200 - n_vars = 200 - theta = 3 - X = np.random.normal(size=(n_obs, n_vars)) - d = np.dot(X[:, :3], np.array([5, 5, 5])) + np.random.standard_normal(size=(n_obs,)) - y = theta * d + np.dot(X[:, :3], np.array([5, 5, 5])) + np.random.standard_normal(size=(n_obs,)) - dml_data = dml.DoubleMLData.from_arrays(X, y, d) - -The hyperparameter-tuning is performed using either an exhaustive search over specified parameter values -implemented in :class:`sklearn.model_selection.GridSearchCV` or via a randomized search implemented in -:class:`sklearn.model_selection.RandomizedSearchCV`. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - import doubleml as dml - from sklearn.linear_model import Lasso - - ml_l = Lasso() - ml_m = Lasso() - dml_plr_obj = dml.DoubleMLPLR(dml_data, ml_l, ml_m) - par_grids = {'ml_l': {'alpha': np.arange(0.05, 1., 0.1)}, - 'ml_m': {'alpha': np.arange(0.05, 1., 0.1)}} - dml_plr_obj.tune(par_grids, search_mode='grid_search'); - print(dml_plr_obj.params) - print(dml_plr_obj.fit().summary) - - np.random.seed(1234) - par_grids = {'ml_l': {'alpha': np.arange(0.05, 1., 0.01)}, - 'ml_m': {'alpha': np.arange(0.05, 1., 0.01)}} - dml_plr_obj.tune(par_grids, search_mode='randomized_search', n_iter_randomized_search=20); - print(dml_plr_obj.params) - print(dml_plr_obj.fit().summary) - -Hyperparameter tuning can also be done with more sophisticated methods, like for example an iterative fitting along -a regularization path implemented in :py:class:`sklearn.linear_model.LassoCV`. -In this case the tuning should be done externally and the parameters can then be set via the -``set_ml_nuisance_params()`` method. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - import doubleml as dml - from sklearn.linear_model import LassoCV - - np.random.seed(1234) - ml_l_tune = LassoCV().fit(dml_data.x, dml_data.y) - ml_m_tune = LassoCV().fit(dml_data.x, dml_data.d) - - ml_l = Lasso() - ml_m = Lasso() - dml_plr_obj = dml.DoubleMLPLR(dml_data, ml_l, ml_m) - dml_plr_obj.set_ml_nuisance_params('ml_l', 'd', {'alpha': ml_l_tune.alpha_}); - dml_plr_obj.set_ml_nuisance_params('ml_m', 'd', {'alpha': ml_m_tune.alpha_}); - print(dml_plr_obj.params) - print(dml_plr_obj.fit().summary) - - -.. TODO: Also discuss other specification options like `tune_on_folds` or `scoring_methods`. - -.. _eval_learners: - -Evaluate learners -################# - -To compare different learners it is possible to evaluate the out-of-sample performance of each learner. The ``summary`` -already displays either the root-mean-squared error (for regressions) or log-loss (for classifications) for each learner -and each corresponding repetition of cross-fitting (``n_rep`` argument). - -To illustrate the parameter tuning, we work with the following example. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - import doubleml as dml - from doubleml.plm.datasets import make_plr_CCDDHNR2018 - from sklearn.ensemble import RandomForestRegressor - - np.random.seed(1234) - ml_l = RandomForestRegressor() - ml_m = RandomForestRegressor() - data = make_plr_CCDDHNR2018(alpha=0.5, return_type='DataFrame') - obj_dml_data = dml.DoubleMLData(data, 'y', 'd') - dml_plr_obj = dml.DoubleMLPLR(obj_dml_data, ml_l, ml_m) - dml_plr_obj.fit() - print(dml_plr_obj) - -The loss of each learner are also stored in the ``nuisance_loss`` attribute. -Further, the ``evaluate_learners()`` method allows to evalute customized evaluation metrics as e.g. the mean absolute error. -The default option is still the root-mean-squared error for evaluation. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - print(dml_plr_obj.nuisance_loss) - print(dml_plr_obj.evaluate_learners()) - -To evaluate a customized metric one has to define a ``callable``. For some models (e.g. the IRM model) it is important that -the metric can handle ``nan`` values as not all target values are known. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - from sklearn.metrics import mean_absolute_error - - def mae(y_true, y_pred): - subset = np.logical_not(np.isnan(y_true)) - return mean_absolute_error(y_true[subset], y_pred[subset]) - - dml_plr_obj.evaluate_learners(learners=['ml_l'], metric=mae) - -A more detailed notebook on the choice of learners is available in the :ref:`example gallery `. - -.. _ext_pred: - -Advanced: External Predictions -############################## - -Since there might be cases where the user wants to use a learner that is not supported by :ref:`DoubleML ` -or do some extensive hyperparameter tuning, it is possible to use external predictions for the nuisance functions. -Remark that this requires the user to take care of the cross-fitting procedure and learner evaluation. - -To illustrate the use of external predictions, we work with the following example. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - import numpy as np - import doubleml as dml - from doubleml.irm.datasets import make_irm_data - from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier - - np.random.seed(3333) - data = make_irm_data(theta=0.5, n_obs=500, dim_x=10, return_type='DataFrame') - obj_dml_data = dml.DoubleMLData(data, 'y', 'd') - - # DoubleML with interal predictions - ml_g = RandomForestRegressor(n_estimators=100, max_features=10, max_depth=5, min_samples_leaf=2) - ml_m = RandomForestClassifier(n_estimators=100, max_features=10, max_depth=5, min_samples_leaf=2) - dml_irm_obj = dml.DoubleMLIRM(obj_dml_data, ml_g, ml_m) - dml_irm_obj.fit() - print(dml_irm_obj.summary) - -The :py:class:`doubleml.DoubleMLIRM` model class saves nuisance predictions in the ``predictions`` attribute as a nested dictionary. -To rely on external predictions, the user has to provide a nested dictionary, where the outer level keys correspond to the treatment -variable names and the inner level keys correspond to the nuisance learner names. Further the values have to be ``numpy`` arrays of shape -``(n_obs, n_rep)``. Here we generate an external predictions dictionary from the internal ``predictions`` attribute. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - pred_dict = {"d": { - "ml_g0": dml_irm_obj.predictions["ml_g0"][:, :, 0], - "ml_g1": dml_irm_obj.predictions["ml_g1"][:, :, 0], - "ml_m": dml_irm_obj.predictions["ml_m"][:, :, 0] - } - } - -The external predictions can be passed to the ``fit()`` method of the :py:class:`doubleml.DoubleML` class via the ``external_predictions`` argument. - -.. tab-set:: - - .. tab-item:: Python - :sync: py - - .. ipython:: python - - ml_g = dml.utils.DMLDummyRegressor() - ml_m = dml.utils.DMLDummyClassifier() - dml_irm_obj_ext = dml.DoubleMLIRM(obj_dml_data, ml_g, ml_m) - dml_irm_obj_ext.fit(external_predictions=pred_dict) - print(dml_irm_obj_ext.summary) - -Both model have identical estimates. Remark that :py:class:`doubleml.DoubleML` class usually require learners for initialization. -With external predictions these learners are not used. The ``DMLDummyRegressor`` and ``DMLDummyClassifier`` are dummy learners which -are used to initialize the :py:class:`doubleml.DoubleML` class. Both dummy learners raise errors if specific methods are called to safeguard against -undesired behavior. Further, the :py:class:`doubleml.DoubleMLData` class requires features (e.g. via the ``x_cols`` argument) which are not used. -This can be handled by adding a dummy column to the data. .. _learners_r: R: Learners and hyperparameters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Minimum requirements for learners -################################# - -The minimum requirement for a learner to be used for nuisance models in the :ref:`DoubleML ` package is - -* The implementation as a learner for regression or classification in the `mlr3 `_ package - or its extension packages `mlr3learners `_ and - `mlr3extralearners `_ . A guide on how to add a learner is provided in the - `chapter on extending learners in the mlr3 book `_ . -* The `mlr3 `_ package makes sure that the learners satisfy some core functionalities. - To specify a specific learner in :ref:`DoubleML ` users can pass objects of the class - `Learner `_. A fast way to construct these objects is to use the - `mlr3 `_ function `lrn() `_. - An introduction to learners in `mlr3 `_ is provided in the `chapter on learners of the mlr3 book `_. -* It is also possible to pass learners that have been constructed from a pipeline with the `mlr3pipelines `_ - package. -* The models `DoubleML::DoubleMLIRM `_ and - `DoubleML::DoubleMLIIVM `_ require classifiers. - Users can also specify classifiers in the `DoubleML::DoubleMLPLR `_ - in cases with binary treatment variables. -* Hyperparameters of learners can either be set at instantiation in `mlr3 `_ or after - instantiation using the ``set_ml_nuisance_params()`` method. - - -An interactive list of provided learners in the `mlr3 `_ and extension packages can be found on the -`website of the mlr3extralearners package `_. - - - -Specifying learners and set hyperparameters -########################################### - -The learners are set during initialization of the :ref:`DoubleML ` model classes -`DoubleML::DoubleMLPLR `_, -`DoubleML::DoubleMLPLIV `_ , -`DoubleML::DoubleMLIRM `_ -and `DoubleML::DoubleMLIIVM `_. -Lets simulate some data and consider the partially linear regression model. -We need to specify learners for the nuisance functions :math:`g_0(X) = E[Y|X]` and :math:`m_0(X) = E[D|X]`, -for example `LearnerRegrRanger `_ -(``lrn("regr.ranger")``) for regression with random forests based on the `ranger `_ -package for R. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - library(DoubleML) - library(mlr3) - library(mlr3learners) - library(data.table) - lgr::get_logger("mlr3")$set_threshold("warn") - - # set up a mlr3 learner - learner = lrn("regr.ranger") - ml_l = learner$clone() - ml_m = learner$clone() - set.seed(3141) - data = make_plr_CCDDHNR2018(alpha=0.5, return_type='data.table') - obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") - dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m) - dml_plr_obj$fit() - dml_plr_obj$summary() - -Without further specification of the hyperparameters, default values are used. To set hyperparameters: - -* We can also use pre-parametrized learners ``lrn("regr.ranger", num.trees=10)``. -* Alternatively, hyperparameters can be set after initialization via the method - ``set_ml_nuisance_params(learner, treat_var, params, set_fold_specific)``. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - set.seed(3141) - ml_l = lrn("regr.ranger", num.trees=10) - ml_m = lrn("regr.ranger") - obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") - dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m) - dml_plr_obj$fit() - dml_plr_obj$summary() - - set.seed(3141) - ml_l = lrn("regr.ranger") - dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l , ml_m) - dml_plr_obj$set_ml_nuisance_params("ml_l", "d", list("num.trees"=10)) - dml_plr_obj$fit() - dml_plr_obj$summary() - -Setting treatment-variable-specific or fold-specific hyperparameters: - -* In the multiple-treatment case, the method ``set_ml_nuisance_params(learner, treat_var, params, set_fold_specific)`` - can be used to set different hyperparameters for different treatment variables. -* The method ``set_ml_nuisance_params(learner, treat_var, params, set_fold_specific)`` accepts lists for ``params``. - The structure of the list depends on whether the same parameters should be provided for all folds or separate values - are passed for specific folds. -* Global parameter passing: The values in ``params`` are used for estimation on all folds. - The named list in the argument ``params`` should have entries with names corresponding to - the parameters of the learners. It is required that option ``set_fold_specific`` is set to ``FALSE`` (default). -* Fold-specific parameter passing: ``params`` is a nested list. The outer list needs to be of length ``n_rep`` and the inner - list of length ``n_folds``. The innermost list must have named entries that correspond to the parameters of the learner. - It is required that option ``set_fold_specific`` is set to ``TRUE``. Moreover, fold-specific - parameter passing is only supported, if all parameters are set fold-specific. -* External setting of parameters will override previously set parameters. To assert the choice of parameters, access the - fields ``$learner`` and ``$params``. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - set.seed(3141) - ml_l = lrn("regr.ranger") - ml_m = lrn("regr.ranger") - obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") - - n_rep = 2 - n_folds = 3 - dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m, n_rep=n_rep, n_folds=n_folds) - - # Set globally - params = list("num.trees"=10) - dml_plr_obj$set_ml_nuisance_params("ml_l", "d", params=params) - dml_plr_obj$set_ml_nuisance_params("ml_m", "d", params=params) - dml_plr_obj$learner - dml_plr_obj$params - dml_plr_obj$fit() - dml_plr_obj$summary() - - -The following example illustrates how to set parameters for each fold. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - learner = lrn("regr.ranger") - ml_l = learner$clone() - ml_m = learner$clone() - dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m, n_rep=n_rep, n_folds=n_folds) - - # Set values for each fold - params_exact = rep(list(rep(list(params), n_folds)), n_rep) - dml_plr_obj$set_ml_nuisance_params("ml_l", "d", params=params_exact, - set_fold_specific=TRUE) - dml_plr_obj$set_ml_nuisance_params("ml_m", "d", params=params_exact, - set_fold_specific=TRUE) - dml_plr_obj$learner - dml_plr_obj$params - dml_plr_obj$fit() - dml_plr_obj$summary() - - -Using pipelines to construct learners -##################################### - -Users can also specify learners that have been constructed from a pipeline using the `mlr3pipelines `_ -package. In general, pipelines can be used to perform data preprocessing, feature selection, combine learners and even -to perform hyperparameter tuning. In the following, we provide two examples on how to construct a single learner and how -to stack different learners via a pipeline. For a more detailed introduction to `mlr3pipelines `_, -we refer to the `Pipelines Chapter in the mlr3book `_. Moreover, a -notebook on how to use `mlr3pipelines `_ in combination with :ref:`DoubleML ` -is available in the example gallery. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - library(mlr3pipelines) - - set.seed(3141) - # Define random forest learner in a pipeline - single_learner_pipeline = po("learner", lrn("regr.ranger", num.trees = 10)) - - # Use pipeline to create a new instance of a learner - ml_g = as_learner(single_learner_pipeline) - ml_m = as_learner(single_learner_pipeline) - - obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") - - n_rep = 2 - n_folds = 3 - dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_g, ml_m, n_rep=n_rep, n_folds=n_folds) - dml_plr_obj$learner - dml_plr_obj$fit() - dml_plr_obj$summary() - - set.seed(3141) - # Define ensemble learner in a pipeline - ensemble_learner_pipeline = gunion(list( - po("learner", lrn("regr.cv_glmnet", s = "lambda.min")), - po("learner", lrn("regr.ranger")), - po("learner", lrn("regr.rpart", cp = 0.01)))) %>>% - po("regravg", 3) - - # Use pipeline to create a new instance of a learner - ml_g = as_learner(ensemble_learner_pipeline) - ml_m = as_learner(ensemble_learner_pipeline) - - obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") - - n_rep = 2 - n_folds = 3 - dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_g, ml_m, n_rep=n_rep, n_folds=n_folds) - dml_plr_obj$learner - dml_plr_obj$fit() - dml_plr_obj$summary() - - -Hyperparameter tuning -##################### - -Parameter tuning of learners for the nuisance functions of :ref:`DoubleML ` models can be done via the ``tune()`` method. -The ``tune()`` method passes various options and parameters to the tuning interface provided by the -`mlr3tuning `_ package. The `mlr3 book `_ provides a -`step-by-step introduction to parameter tuning `_. - -To illustrate the parameter tuning, we generate data from a sparse partially linear regression model. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - library(DoubleML) - library(mlr3) - library(data.table) - - set.seed(3141) - n_obs = 200 - n_vars = 200 - theta = 3 - X = matrix(stats::rnorm(n_obs * n_vars), nrow = n_obs, ncol = n_vars) - d = X[, 1:3, drop = FALSE] %*% c(5, 5, 5) + stats::rnorm(n_obs) - y = theta * d + X[, 1:3, drop = FALSE] %*% c(5, 5, 5) + stats::rnorm(n_obs) - dml_data = double_ml_data_from_matrix(X = X, y = y, d = d) - -The hyperparameter-tuning is performed according to options passed through a named list ``tune_settings``. -The entries in the list specify options during parameter tuning with `mlr3tuning `_: - -* ``terminator`` is a `Terminator object `_ passed to - `mlr3tuning `_ that manages the budget to solve the tuning problem. -* ``algorithm`` is an object of class - `Tuner `_ and specifies the tuning algorithm. - Alternatively, ``algorithm`` can be a ``character()`` that is used as an argument in the wrapper - `mlr3tuning `_ call - `tnr(algorithm) `_. - `The corresponding chapter in the mlr3book `_ illustrates - how the `Tuner `_ class supports grid search, random search, - generalized simulated annealing and non-linear optimization. -* ``rsmp_tune`` is an object of class `mlr3 resampling `_ - that specifies the resampling method for evaluation, for example `rsmp("cv", folds = 5)` implements 5-fold cross-validation. - `rsmp("holdout", ratio = 0.8)` implements an evaluation based on a hold-out sample that contains 20 percent of the observations. - By default, 5-fold cross-validation is performed. -* ``measure`` is a named list containing the measures used for tuning of the nuisance components. - The names of the entries must match the learner names (see method ``learner_names()``). The entries in the list must either be - objects of class `Measure `_ or keys passed to `msr() `_. - If ``measure`` is not provided by the user, default measures are used, i.e., mean squared error for regression models - and classification error for binary outcomes. - -In the following example, we tune the penalty parameter :math:`\lambda` (``lambda``) for lasso with the R package -`glmnet `_. To tune the value of ``lambda``, a grid search is performed over a grid of values that range from 0.05 -to 0.1 at a resolution of 10. Using a resolution of 10 splits the grid of values in 10 equally spaced values ranging from a minimum of 0.05 -to a maximum of 0.1. To evaluate the predictive performance in both nuisance parts, the cross-validated mean squared error is used. - -Setting the option ``tune_on_folds=FALSE``, the tuning is performed on the whole sample. Hence, the cross-validated errors -are obtained from a random split of the whole sample into 5 folds. As a result, one set of ``lambda`` values are obtained -which are later used in the fitting stage for all folds. - -Alternatively, setting the option ``tune_on_folds=TRUE`` would assign the tuning resampling scheme ``rsmp_tune`` to each fold. -For example, if we set ``n_folds=2`` at initialization of the ``DoubleMLPLR`` object and use a 5-fold cross-validated error -for tuning, each of the two folds would be split up into 5 subfolds and the error would be evaluated on these subfolds. - - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - library(DoubleML) - library(mlr3) - library(data.table) - library(mlr3learners) - library(mlr3tuning) - library(paradox) - lgr::get_logger("mlr3")$set_threshold("warn") - lgr::get_logger("bbotk")$set_threshold("warn") - - set.seed(1234) - ml_l = lrn("regr.glmnet") - ml_m = lrn("regr.glmnet") - dml_plr_obj = DoubleMLPLR$new(dml_data, ml_l, ml_m) - - par_grids = list( - "ml_l" = ps(lambda = p_dbl(lower = 0.05, upper = 0.1)), - "ml_m" = ps(lambda = p_dbl(lower = 0.05, upper = 0.1))) - - tune_settings = list(terminator = trm("evals", n_evals = 100), - algorithm = tnr("grid_search", resolution = 10), - rsmp_tune = rsmp("cv", folds = 5), - measure = list("ml_l" = msr("regr.mse"), - "ml_m" = msr("regr.mse"))) - dml_plr_obj$tune(param_set=par_grids, tune_settings=tune_settings, tune_on_fold=TRUE) - dml_plr_obj$params - - dml_plr_obj$fit() - dml_plr_obj$summary() - - -Hyperparameter tuning can also be done with more sophisticated methods, for example by using built-in tuning -paths of learners. For example, the learner `regr.cv_glmnet `_ -performs an internal cross-validated choice of the parameter ``lambda``. -Alternatively, the powerful functionalities of the `mlr3tuning `_ package can be used for -external parameter tuning of the nuisance parts. The optimally chosen parameters can then be passed to the -:ref:`DoubleML ` models using the ``set_ml_nuisance_params()`` method. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - library(DoubleML) - library(mlr3) - library(data.table) - library(mlr3learners) - library(mlr3tuning) - lgr::get_logger("mlr3")$set_threshold("warn") - lgr::get_logger("bbotk")$set_threshold("warn") - - set.seed(1234) - ml_l = lrn("regr.cv_glmnet", s="lambda.min") - ml_m = lrn("regr.cv_glmnet", s="lambda.min") - dml_plr_obj = DoubleMLPLR$new(dml_data, ml_l, ml_m) - - dml_plr_obj$fit() - dml_plr_obj$summary() - - -The following code chunk illustrates another example for global parameter tuning with random forests -as provided by the `ranger `_ package. In this example, we use random search to find optimal -parameters ``mtry`` and ``max.depth`` of a random forest. Evaluation is based on 3-fold cross-validation. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - library(DoubleML) - library(mlr3) - library(mlr3learners) - library(data.table) - library(mlr3tuning) - library(paradox) - lgr::get_logger("mlr3")$set_threshold("warn") - lgr::get_logger("bbotk")$set_threshold("warn") - - # set up a mlr3 learner - learner = lrn("regr.ranger") - ml_l = learner$clone() - ml_m = learner$clone() - - set.seed(3141) - obj_dml_data = make_plr_CCDDHNR2018(alpha=0.5) - dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m) - - # set up a list of parameter grids - param_grid = list("ml_l" = ps(mtry = p_int(lower = 2 , upper = 20), - max.depth = p_int(lower = 2, upper = 5)), - "ml_m" = ps(mtry = p_int(lower = 2 , upper = 20), - max.depth = p_int(lower = 2, upper = 5))) - - tune_settings = list(terminator = mlr3tuning::trm("evals", n_evals = 20), - algorithm = tnr("random_search"), - rsmp_tune = rsmp("cv", folds = 3), - measure = list("ml_l" = msr("regr.mse"), - "ml_m" = msr("regr.mse"))) - dml_plr_obj$tune(param_set=param_grid, tune_settings=tune_settings, tune_on_folds=FALSE) - dml_plr_obj$params - - dml_plr_obj$fit() - dml_plr_obj$summary() - - -Hyperparameter tuning with pipelines -#################################### - -As an alternative to the previously presented tuning approach, it is possible to base the parameter tuning on a pipeline -as provided by the `mlr3pipelines `_ package. The basic idea of this approach is to -define a learner via a pipeline and then perform the tuning via the ``tune()``. We will shortly repeat the lasso example -from above. In general, the pipeline-based approach can be used to find optimal values not only for the parameters of -one or multiple learners, but also for other parameters, which are, for example, involved in the data preprocessing. We -refer to more details provided in the `Pipelines Chapter in the mlr3book `_. - -.. tab-set:: - - .. tab-item:: R - :sync: r - - .. jupyter-execute:: - - library(DoubleML) - library(mlr3) - library(mlr3tuning) - library(mlr3pipelines) - lgr::get_logger("mlr3")$set_threshold("warn") - lgr::get_logger("bbotk")$set_threshold("warn") - - # Define learner in a pipeline - set.seed(1234) - lasso_pipe = po("learner", - learner = lrn("regr.glmnet")) - ml_g = as_learner(lasso_pipe) - ml_m = as_learner(lasso_pipe) - - # Instantiate a DoubleML object - dml_plr_obj = DoubleMLPLR$new(dml_data, ml_g, ml_m) - - # Parameter grid for lambda - par_grids = ps(regr.glmnet.lambda = p_dbl(lower = 0.05, upper = 0.1)) - - tune_settings = list(terminator = trm("evals", n_evals = 100), - algorithm = tnr("grid_search", resolution = 10), - rsmp_tune = rsmp("cv", folds = 5), - measure = list("ml_g" = msr("regr.mse"), - "ml_m" = msr("regr.mse"))) - dml_plr_obj$tune(param_set = list("ml_g" = par_grids, - "ml_m" = par_grids), - tune_settings=tune_settings, - tune_on_fold=TRUE) - dml_plr_obj$fit() - dml_plr_obj$summary() - -References -++++++++++ - -* Lang, M., Binder, M., Richter, J., Schratz, P., Pfisterer, F., Coors, S., Au, Q., Casalicchio, G., Kotthoff, L., Bischl, B. (2019), mlr3: A modern object-oriented machine learing framework in R. Journal of Open Source Software, `doi:10.21105/joss.01903 `_. +.. include:: learners/r/learners_overview.inc -* Becker, M., Binder, M., Bischl, B., Lang, M., Pfisterer, F., Reich, N.G., Richter, J., Schratz, P., Sonabend, R. (2020), mlr3 book, available at `https://mlr3book.mlr-org.com `_. diff --git a/doc/guide/learners/python/evaluate_learners.rst b/doc/guide/learners/python/evaluate_learners.rst new file mode 100644 index 00000000..defbc896 --- /dev/null +++ b/doc/guide/learners/python/evaluate_learners.rst @@ -0,0 +1,59 @@ +To compare different learners it is possible to evaluate the out-of-sample performance of each learner. The ``summary`` +already displays either the root-mean-squared error (for regressions) or log-loss (for classifications) for each learner +and each corresponding repetition of cross-fitting (``n_rep`` argument). + +To illustrate the parameter tuning, we work with the following example. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + import doubleml as dml + from doubleml.plm.datasets import make_plr_CCDDHNR2018 + from sklearn.ensemble import RandomForestRegressor + + np.random.seed(1234) + ml_l = RandomForestRegressor() + ml_m = RandomForestRegressor() + data = make_plr_CCDDHNR2018(alpha=0.5, return_type='DataFrame') + obj_dml_data = dml.DoubleMLData(data, 'y', 'd') + dml_plr_obj = dml.DoubleMLPLR(obj_dml_data, ml_l, ml_m) + dml_plr_obj.fit() + print(dml_plr_obj) + +The loss of each learner are also stored in the ``nuisance_loss`` attribute. +Further, the ``evaluate_learners()`` method allows to evalute customized evaluation metrics as e.g. the mean absolute error. +The default option is still the root-mean-squared error for evaluation. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + print(dml_plr_obj.nuisance_loss) + print(dml_plr_obj.evaluate_learners()) + +To evaluate a customized metric one has to define a ``callable``. For some models (e.g. the IRM model) it is important that +the metric can handle ``nan`` values as not all target values are known. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + from sklearn.metrics import mean_absolute_error + + def mae(y_true, y_pred): + subset = np.logical_not(np.isnan(y_true)) + return mean_absolute_error(y_true[subset], y_pred[subset]) + + dml_plr_obj.evaluate_learners(learners=['ml_l'], metric=mae) + +A more detailed notebook on the choice of learners is available in the :ref:`example gallery `. \ No newline at end of file diff --git a/doc/guide/learners/python/external_preds.rst b/doc/guide/learners/python/external_preds.rst new file mode 100644 index 00000000..6ae30d22 --- /dev/null +++ b/doc/guide/learners/python/external_preds.rst @@ -0,0 +1,68 @@ +Since there might be cases where the user wants to use a learner that is not supported by :ref:`DoubleML ` +or do some extensive hyperparameter tuning, it is possible to use external predictions for the nuisance functions. +Remark that this requires the user to take care of the cross-fitting procedure and learner evaluation. + +To illustrate the use of external predictions, we work with the following example. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + import numpy as np + import doubleml as dml + from doubleml.irm.datasets import make_irm_data + from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier + + np.random.seed(3333) + data = make_irm_data(theta=0.5, n_obs=500, dim_x=10, return_type='DataFrame') + obj_dml_data = dml.DoubleMLData(data, 'y', 'd') + + # DoubleML with interal predictions + ml_g = RandomForestRegressor(n_estimators=100, max_features=10, max_depth=5, min_samples_leaf=2) + ml_m = RandomForestClassifier(n_estimators=100, max_features=10, max_depth=5, min_samples_leaf=2) + dml_irm_obj = dml.DoubleMLIRM(obj_dml_data, ml_g, ml_m) + dml_irm_obj.fit() + print(dml_irm_obj.summary) + +The :py:class:`doubleml.DoubleMLIRM` model class saves nuisance predictions in the ``predictions`` attribute as a nested dictionary. +To rely on external predictions, the user has to provide a nested dictionary, where the outer level keys correspond to the treatment +variable names and the inner level keys correspond to the nuisance learner names. Further the values have to be ``numpy`` arrays of shape +``(n_obs, n_rep)``. Here we generate an external predictions dictionary from the internal ``predictions`` attribute. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + pred_dict = {"d": { + "ml_g0": dml_irm_obj.predictions["ml_g0"][:, :, 0], + "ml_g1": dml_irm_obj.predictions["ml_g1"][:, :, 0], + "ml_m": dml_irm_obj.predictions["ml_m"][:, :, 0] + } + } + +The external predictions can be passed to the ``fit()`` method of the :py:class:`doubleml.DoubleML` class via the ``external_predictions`` argument. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + ml_g = dml.utils.DMLDummyRegressor() + ml_m = dml.utils.DMLDummyClassifier() + dml_irm_obj_ext = dml.DoubleMLIRM(obj_dml_data, ml_g, ml_m) + dml_irm_obj_ext.fit(external_predictions=pred_dict) + print(dml_irm_obj_ext.summary) + +Both model have identical estimates. Remark that :py:class:`doubleml.DoubleML` class usually require learners for initialization. +With external predictions these learners are not used. The ``DMLDummyRegressor`` and ``DMLDummyClassifier`` are dummy learners which +are used to initialize the :py:class:`doubleml.DoubleML` class. Both dummy learners raise errors if specific methods are called to safeguard against +undesired behavior. Further, the :py:class:`doubleml.DoubleMLData` class requires features (e.g. via the ``x_cols`` argument) which are not used. +This can be handled by adding a dummy column to the data. \ No newline at end of file diff --git a/doc/guide/learners/python/learners_overview.inc b/doc/guide/learners/python/learners_overview.inc new file mode 100644 index 00000000..25648dac --- /dev/null +++ b/doc/guide/learners/python/learners_overview.inc @@ -0,0 +1,47 @@ +.. _py_learner_req: + +Minimum requirements for learners +################################# + +.. include:: /guide/learners/python/minimum_req.rst + + +.. _py_set_params: + +Specifying learners and set hyperparameters +########################################### + +.. include:: /guide/learners/python/set_hyperparams.rst + +.. _py_tune_params: + +Hyperparameter tuning +##################### + +.. include:: /guide/learners/python/tune_hyperparams.rst + + +Hyperparameter tuning (Grid Search) +################################### + +.. warning:: + **Deprecated:** The ``tune()`` method is deprecated and be removed in a future version. + Please use ``tune_ml_models()`` for hyperparameter tuning, see :ref:`Hyperparameter tuning `. + +.. include:: /guide/learners/python/tune_hyperparams_old.rst + +.. _py_eval_learners: + +Evaluate learners +################# + +.. include:: /guide/learners/python/evaluate_learners.rst + + +.. _py_ext_pred: + +Advanced: External Predictions +############################## + + +.. include:: /guide/learners/python/external_preds.rst \ No newline at end of file diff --git a/doc/guide/learners/python/minimum_req.rst b/doc/guide/learners/python/minimum_req.rst new file mode 100644 index 00000000..cb557ebc --- /dev/null +++ b/doc/guide/learners/python/minimum_req.rst @@ -0,0 +1,14 @@ +The minimum requirement for a learner to be used for nuisance models in the :ref:`DoubleML ` +package is + +* The implementation of a ``fit()`` and ``predict()`` method. + Some models, like :py:class:`doubleml.DoubleMLIRM` and :py:class:`doubleml.DoubleMLIIVM` require classifiers. +* In case of classifiers, the learner needs to come with a ``predict_proba()`` instead of, or in addition to, a + ``predict()`` method, see for example :py:meth:`sklearn.ensemble.RandomForestClassifier.predict_proba`. +* In order to be able to use the ``set_ml_nuisance_params()`` method of :ref:`DoubleML ` classes the + learner additionally needs to come with a ``set_params()`` method, + see for example :py:meth:`sklearn.ensemble.RandomForestRegressor.set_params`. +* We further rely on the function :py:func:`sklearn.base.clone` which adds the requirement of a ``get_params()`` + method for a learner in order to be used for nuisance models of :ref:`DoubleML ` model classes. + +Most learners from `scikit-learn `_ satisfy all these minimum requirements. diff --git a/doc/guide/learners/python/set_hyperparams.rst b/doc/guide/learners/python/set_hyperparams.rst new file mode 100644 index 00000000..039ba2c5 --- /dev/null +++ b/doc/guide/learners/python/set_hyperparams.rst @@ -0,0 +1,61 @@ +The learners are set during initialization of the :ref:`DoubleML ` model classes +:py:class:`doubleml.DoubleMLPLR`, :py:class:`doubleml.DoubleMLPLIV`, +:py:class:`doubleml.DoubleMLIRM` and :py:class:`doubleml.DoubleMLIIVM`. +Lets simulate some data and consider the partially linear regression model. +We need to specify learners for the nuisance functions :math:`g_0(X) = E[Y|X]` and :math:`m_0(X) = E[D|X]`, +for example :py:class:`sklearn.ensemble.RandomForestRegressor`. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + import doubleml as dml + from doubleml.plm.datasets import make_plr_CCDDHNR2018 + from sklearn.ensemble import RandomForestRegressor + + np.random.seed(1234) + ml_l = RandomForestRegressor() + ml_m = RandomForestRegressor() + data = make_plr_CCDDHNR2018(alpha=0.5, return_type='DataFrame') + obj_dml_data = dml.DoubleMLData(data, 'y', 'd') + dml_plr_obj = dml.DoubleMLPLR(obj_dml_data, ml_l, ml_m) + dml_plr_obj.fit().summary + +Without further specification of the hyperparameters, default values are used. To set hyperparameters: + +* We can also use pre-parametrized learners, like ``RandomForestRegressor(n_estimators=10)``. +* Alternatively, hyperparameters can also be set after initialization via the method + ``set_ml_nuisance_params(learner, treat_var, params)`` + + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + np.random.seed(1234) + dml_plr_obj = dml.DoubleMLPLR(obj_dml_data, + RandomForestRegressor(n_estimators=10), + RandomForestRegressor()) + print(dml_plr_obj.fit().summary) + + np.random.seed(1234) + dml_plr_obj = dml.DoubleMLPLR(obj_dml_data, + RandomForestRegressor(), + RandomForestRegressor()) + dml_plr_obj.set_ml_nuisance_params('ml_l', 'd', {'n_estimators': 10}); + print(dml_plr_obj.fit().summary) + +Setting treatment-variable-specific or fold-specific hyperparameters: + +* In the multiple-treatment case, the method ``set_ml_nuisance_params(learner, treat_var, params)`` can be used to set + different hyperparameters for different treatment variables. +* The method ``set_ml_nuisance_params(learner, treat_var, params)`` accepts dicts and lists for ``params``. + A dict should be provided if for each fold the same hyperparameters should be used. + Fold-specific parameters are supported. To do so, provide a nested list as ``params``, where the outer list is of + length ``n_rep`` and the inner list of length ``n_folds``. \ No newline at end of file diff --git a/doc/guide/learners/python/tune_hyperparams.rst b/doc/guide/learners/python/tune_hyperparams.rst new file mode 100644 index 00000000..980ce90c --- /dev/null +++ b/doc/guide/learners/python/tune_hyperparams.rst @@ -0,0 +1,59 @@ +Parameter tuning of learners for the nuisance functions of :ref:`DoubleML ` models can be done via +the ``tune_ml_models()`` method. +To illustrate the parameter tuning, we generate data from a sparse partially linear regression model. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + import doubleml as dml + import numpy as np + + np.random.seed(3141) + n_obs = 200 + n_vars = 200 + theta = 3 + X = np.random.normal(size=(n_obs, n_vars)) + d = np.dot(X[:, :3], np.array([5, 5, 5])) + np.random.standard_normal(size=(n_obs,)) + y = theta * d + np.dot(X[:, :3], np.array([5, 5, 5])) + np.random.standard_normal(size=(n_obs,)) + dml_data = dml.DoubleMLData.from_arrays(X, y, d) + +The hyperparameter-tuning is performed using `Optuna `_ as backend. Here, we illustrate +the tuning via defining a search space for the nuisance function learners over ``100`` trials. The most important input +argument is the hyperparameter space via a dictionary of functions. This search space will internally transformed into a +suitable ``objective(trial)`` function for `Optuna `_. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + :okwarning: + + import doubleml as dml + from sklearn.linear_model import Lasso + import optuna + + ml_l = Lasso() + ml_m = Lasso() + dml_plr_obj = dml.DoubleMLPLR(dml_data, ml_l, ml_m) + + def ml_l_params(trial): + return {'alpha': trial.suggest_float('alpha', 0.05, 1.0)} + + def ml_m_params(trial): + return {'alpha': trial.suggest_float('alpha', 0.05, 1.0)} + + param_space = {'ml_l': ml_l_params, 'ml_m': ml_m_params} + optuna_settings = {'n_trials': 100, 'verbosity': optuna.logging.WARNING} + + dml_plr_obj.tune_ml_models(ml_param_space=param_space, optuna_settings=optuna_settings) + + print(dml_plr_obj.params) + print(dml_plr_obj.fit().summary) + +A more detailed description of hyperparameter-tuning possibilities can be found in the :ref:`Example Gallery `. \ No newline at end of file diff --git a/doc/guide/learners/python/tune_hyperparams_old.rst b/doc/guide/learners/python/tune_hyperparams_old.rst new file mode 100644 index 00000000..79547666 --- /dev/null +++ b/doc/guide/learners/python/tune_hyperparams_old.rst @@ -0,0 +1,80 @@ +Parameter tuning of learners for the nuisance functions of :ref:`DoubleML ` models can be done via +the ``tune()`` method. +To illustrate the parameter tuning, we generate data from a sparse partially linear regression model. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + import doubleml as dml + import numpy as np + + np.random.seed(3141) + n_obs = 200 + n_vars = 200 + theta = 3 + X = np.random.normal(size=(n_obs, n_vars)) + d = np.dot(X[:, :3], np.array([5, 5, 5])) + np.random.standard_normal(size=(n_obs,)) + y = theta * d + np.dot(X[:, :3], np.array([5, 5, 5])) + np.random.standard_normal(size=(n_obs,)) + dml_data = dml.DoubleMLData.from_arrays(X, y, d) + +The hyperparameter-tuning is performed using either an exhaustive search over specified parameter values +implemented in :class:`sklearn.model_selection.GridSearchCV` or via a randomized search implemented in +:class:`sklearn.model_selection.RandomizedSearchCV`. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + :okwarning: + + import doubleml as dml + from sklearn.linear_model import Lasso + + ml_l = Lasso() + ml_m = Lasso() + dml_plr_obj = dml.DoubleMLPLR(dml_data, ml_l, ml_m) + par_grids = {'ml_l': {'alpha': np.arange(0.05, 1., 0.1)}, + 'ml_m': {'alpha': np.arange(0.05, 1., 0.1)}} + dml_plr_obj.tune(par_grids, search_mode='grid_search'); + print(dml_plr_obj.params) + print(dml_plr_obj.fit().summary) + + np.random.seed(1234) + par_grids = {'ml_l': {'alpha': np.arange(0.05, 1., 0.01)}, + 'ml_m': {'alpha': np.arange(0.05, 1., 0.01)}} + dml_plr_obj.tune(par_grids, search_mode='randomized_search', n_iter_randomized_search=20); + print(dml_plr_obj.params) + print(dml_plr_obj.fit().summary) + +Hyperparameter tuning can also be done with more sophisticated methods, like for example an iterative fitting along +a regularization path implemented in :py:class:`sklearn.linear_model.LassoCV`. +In this case the tuning should be done externally and the parameters can then be set via the +``set_ml_nuisance_params()`` method. + +.. tab-set:: + + .. tab-item:: Python + :sync: py + + .. ipython:: python + + import doubleml as dml + from sklearn.linear_model import LassoCV + + np.random.seed(1234) + ml_l_tune = LassoCV().fit(dml_data.x, dml_data.y) + ml_m_tune = LassoCV().fit(dml_data.x, dml_data.d) + + ml_l = Lasso() + ml_m = Lasso() + dml_plr_obj = dml.DoubleMLPLR(dml_data, ml_l, ml_m) + dml_plr_obj.set_ml_nuisance_params('ml_l', 'd', {'alpha': ml_l_tune.alpha_}); + dml_plr_obj.set_ml_nuisance_params('ml_m', 'd', {'alpha': ml_m_tune.alpha_}); + print(dml_plr_obj.params) + print(dml_plr_obj.fit().summary) diff --git a/doc/guide/learners/r/learners_overview.inc b/doc/guide/learners/r/learners_overview.inc new file mode 100644 index 00000000..b98eee20 --- /dev/null +++ b/doc/guide/learners/r/learners_overview.inc @@ -0,0 +1,39 @@ +.. _r_learner_req: + +Minimum requirements for learners +################################# + +.. include:: /guide/learners/r/minimum_req.rst + + +.. _r_set_params: + +Specifying learners and set hyperparameters +########################################### + +.. include:: /guide/learners/r/set_hyperparams.rst + + +.. _r_pipelines: + +Using pipelines to construct learners +##################################### + +.. include:: /guide/learners/r/pipelines.rst + + +.. _r_tune_params: + +Hyperparameter tuning +##################### + +.. include:: /guide/learners/r/tune_hyperparams.rst + + +.. _r_tune_and_pipelines: + +Hyperparameter tuning with pipelines +#################################### + +.. include:: /guide/learners/r/tune_and_pipelines.rst + diff --git a/doc/guide/learners/r/minimum_req.rst b/doc/guide/learners/r/minimum_req.rst new file mode 100644 index 00000000..7f3abdf0 --- /dev/null +++ b/doc/guide/learners/r/minimum_req.rst @@ -0,0 +1,23 @@ +The minimum requirement for a learner to be used for nuisance models in the :ref:`DoubleML ` package is + +* The implementation as a learner for regression or classification in the `mlr3 `_ package + or its extension packages `mlr3learners `_ and + `mlr3extralearners `_ . A guide on how to add a learner is provided in the + `chapter on extending learners in the mlr3 book `_ . +* The `mlr3 `_ package makes sure that the learners satisfy some core functionalities. + To specify a specific learner in :ref:`DoubleML ` users can pass objects of the class + `Learner `_. A fast way to construct these objects is to use the + `mlr3 `_ function `lrn() `_. + An introduction to learners in `mlr3 `_ is provided in the `chapter on learners of the mlr3 book `_. +* It is also possible to pass learners that have been constructed from a pipeline with the `mlr3pipelines `_ + package. +* The models `DoubleML::DoubleMLIRM `_ and + `DoubleML::DoubleMLIIVM `_ require classifiers. + Users can also specify classifiers in the `DoubleML::DoubleMLPLR `_ + in cases with binary treatment variables. +* Hyperparameters of learners can either be set at instantiation in `mlr3 `_ or after + instantiation using the ``set_ml_nuisance_params()`` method. + + +An interactive list of provided learners in the `mlr3 `_ and extension packages can be found on the +`website of the mlr3extralearners package `_. diff --git a/doc/guide/learners/r/pipelines.rst b/doc/guide/learners/r/pipelines.rst new file mode 100644 index 00000000..de18b6ab --- /dev/null +++ b/doc/guide/learners/r/pipelines.rst @@ -0,0 +1,59 @@ +Users can also specify learners that have been constructed from a pipeline using the `mlr3pipelines `_ +package. In general, pipelines can be used to perform data preprocessing, feature selection, combine learners and even +to perform hyperparameter tuning. In the following, we provide two examples on how to construct a single learner and how +to stack different learners via a pipeline. For a more detailed introduction to `mlr3pipelines `_, +we refer to the `Pipelines Chapter in the mlr3book `_. Moreover, a +notebook on how to use `mlr3pipelines `_ in combination with :ref:`DoubleML ` +is available in the example gallery. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + library(DoubleML) + library(mlr3) + library(mlr3learners) + library(mlr3pipelines) + library(data.table) + + set.seed(3141) + # Define random forest learner in a pipeline + single_learner_pipeline = po("learner", lrn("regr.ranger", num.trees = 10)) + + # Use pipeline to create a new instance of a learner + ml_g = as_learner(single_learner_pipeline) + ml_m = as_learner(single_learner_pipeline) + + data = make_plr_CCDDHNR2018(alpha=0.5, return_type='data.table') + obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") + + n_rep = 2 + n_folds = 3 + dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_g, ml_m, n_rep=n_rep, n_folds=n_folds) + dml_plr_obj$learner + dml_plr_obj$fit() + dml_plr_obj$summary() + + set.seed(3141) + # Define ensemble learner in a pipeline + ensemble_learner_pipeline = gunion(list( + po("learner", lrn("regr.cv_glmnet", s = "lambda.min")), + po("learner", lrn("regr.ranger")), + po("learner", lrn("regr.rpart", cp = 0.01)))) %>>% + po("regravg", 3) + + # Use pipeline to create a new instance of a learner + ml_g = as_learner(ensemble_learner_pipeline) + ml_m = as_learner(ensemble_learner_pipeline) + + obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") + + n_rep = 2 + n_folds = 3 + dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_g, ml_m, n_rep=n_rep, n_folds=n_folds) + dml_plr_obj$learner + dml_plr_obj$fit() + dml_plr_obj$summary() diff --git a/doc/guide/learners/r/set_hyperparams.rst b/doc/guide/learners/r/set_hyperparams.rst new file mode 100644 index 00000000..266b8372 --- /dev/null +++ b/doc/guide/learners/r/set_hyperparams.rst @@ -0,0 +1,130 @@ +The learners are set during initialization of the :ref:`DoubleML ` model classes +`DoubleML::DoubleMLPLR `_, +`DoubleML::DoubleMLPLIV `_ , +`DoubleML::DoubleMLIRM `_ +and `DoubleML::DoubleMLIIVM `_. +Lets simulate some data and consider the partially linear regression model. +We need to specify learners for the nuisance functions :math:`g_0(X) = E[Y|X]` and :math:`m_0(X) = E[D|X]`, +for example `LearnerRegrRanger `_ +(``lrn("regr.ranger")``) for regression with random forests based on the `ranger `_ +package for R. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + library(DoubleML) + library(mlr3) + library(mlr3learners) + library(data.table) + lgr::get_logger("mlr3")$set_threshold("warn") + + # set up a mlr3 learner + learner = lrn("regr.ranger") + ml_l = learner$clone() + ml_m = learner$clone() + set.seed(3141) + data = make_plr_CCDDHNR2018(alpha=0.5, return_type='data.table') + obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") + dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m) + dml_plr_obj$fit() + dml_plr_obj$summary() + +Without further specification of the hyperparameters, default values are used. To set hyperparameters: + +* We can also use pre-parametrized learners ``lrn("regr.ranger", num.trees=10)``. +* Alternatively, hyperparameters can be set after initialization via the method + ``set_ml_nuisance_params(learner, treat_var, params, set_fold_specific)``. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + set.seed(3141) + ml_l = lrn("regr.ranger", num.trees=10) + ml_m = lrn("regr.ranger") + obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") + dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m) + dml_plr_obj$fit() + dml_plr_obj$summary() + + set.seed(3141) + ml_l = lrn("regr.ranger") + dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l , ml_m) + dml_plr_obj$set_ml_nuisance_params("ml_l", "d", list("num.trees"=10)) + dml_plr_obj$fit() + dml_plr_obj$summary() + +Setting treatment-variable-specific or fold-specific hyperparameters: + +* In the multiple-treatment case, the method ``set_ml_nuisance_params(learner, treat_var, params, set_fold_specific)`` + can be used to set different hyperparameters for different treatment variables. +* The method ``set_ml_nuisance_params(learner, treat_var, params, set_fold_specific)`` accepts lists for ``params``. + The structure of the list depends on whether the same parameters should be provided for all folds or separate values + are passed for specific folds. +* Global parameter passing: The values in ``params`` are used for estimation on all folds. + The named list in the argument ``params`` should have entries with names corresponding to + the parameters of the learners. It is required that option ``set_fold_specific`` is set to ``FALSE`` (default). +* Fold-specific parameter passing: ``params`` is a nested list. The outer list needs to be of length ``n_rep`` and the inner + list of length ``n_folds``. The innermost list must have named entries that correspond to the parameters of the learner. + It is required that option ``set_fold_specific`` is set to ``TRUE``. Moreover, fold-specific + parameter passing is only supported, if all parameters are set fold-specific. +* External setting of parameters will override previously set parameters. To assert the choice of parameters, access the + fields ``$learner`` and ``$params``. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + set.seed(3141) + ml_l = lrn("regr.ranger") + ml_m = lrn("regr.ranger") + obj_dml_data = DoubleMLData$new(data, y_col="y", d_cols="d") + + n_rep = 2 + n_folds = 3 + dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m, n_rep=n_rep, n_folds=n_folds) + + # Set globally + params = list("num.trees"=10) + dml_plr_obj$set_ml_nuisance_params("ml_l", "d", params=params) + dml_plr_obj$set_ml_nuisance_params("ml_m", "d", params=params) + dml_plr_obj$learner + dml_plr_obj$params + dml_plr_obj$fit() + dml_plr_obj$summary() + + +The following example illustrates how to set parameters for each fold. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + learner = lrn("regr.ranger") + ml_l = learner$clone() + ml_m = learner$clone() + dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m, n_rep=n_rep, n_folds=n_folds) + + # Set values for each fold + params_exact = rep(list(rep(list(params), n_folds)), n_rep) + dml_plr_obj$set_ml_nuisance_params("ml_l", "d", params=params_exact, + set_fold_specific=TRUE) + dml_plr_obj$set_ml_nuisance_params("ml_m", "d", params=params_exact, + set_fold_specific=TRUE) + dml_plr_obj$learner + dml_plr_obj$params + dml_plr_obj$fit() + dml_plr_obj$summary() diff --git a/doc/guide/learners/r/tune_and_pipelines.rst b/doc/guide/learners/r/tune_and_pipelines.rst new file mode 100644 index 00000000..3b735f4e --- /dev/null +++ b/doc/guide/learners/r/tune_and_pipelines.rst @@ -0,0 +1,61 @@ +As an alternative to the previously presented tuning approach, it is possible to base the parameter tuning on a pipeline +as provided by the `mlr3pipelines `_ package. The basic idea of this approach is to +define a learner via a pipeline and then perform the tuning via the ``tune()``. We will shortly repeat the lasso example +from above. In general, the pipeline-based approach can be used to find optimal values not only for the parameters of +one or multiple learners, but also for other parameters, which are, for example, involved in the data preprocessing. We +refer to more details provided in the `Pipelines Chapter in the mlr3book `_. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + library(DoubleML) + library(mlr3) + library(mlr3tuning) + library(mlr3pipelines) + lgr::get_logger("mlr3")$set_threshold("warn") + lgr::get_logger("bbotk")$set_threshold("warn") + + set.seed(3141) + n_obs = 200 + n_vars = 200 + theta = 3 + X = matrix(stats::rnorm(n_obs * n_vars), nrow = n_obs, ncol = n_vars) + d = X[, 1:3, drop = FALSE] %*% c(5, 5, 5) + stats::rnorm(n_obs) + y = theta * d + X[, 1:3, drop = FALSE] %*% c(5, 5, 5) + stats::rnorm(n_obs) + dml_data = double_ml_data_from_matrix(X = X, y = y, d = d) + + # Define learner in a pipeline + set.seed(1234) + lasso_pipe = po("learner", + learner = lrn("regr.glmnet")) + ml_g = as_learner(lasso_pipe) + ml_m = as_learner(lasso_pipe) + + # Instantiate a DoubleML object + dml_plr_obj = DoubleMLPLR$new(dml_data, ml_g, ml_m) + + # Parameter grid for lambda + par_grids = ps(regr.glmnet.lambda = p_dbl(lower = 0.05, upper = 0.1)) + + tune_settings = list(terminator = trm("evals", n_evals = 100), + algorithm = tnr("grid_search", resolution = 10), + rsmp_tune = rsmp("cv", folds = 5), + measure = list("ml_g" = msr("regr.mse"), + "ml_m" = msr("regr.mse"))) + dml_plr_obj$tune(param_set = list("ml_g" = par_grids, + "ml_m" = par_grids), + tune_settings=tune_settings, + tune_on_fold=TRUE) + dml_plr_obj$fit() + dml_plr_obj$summary() + +References +++++++++++ + +* Lang, M., Binder, M., Richter, J., Schratz, P., Pfisterer, F., Coors, S., Au, Q., Casalicchio, G., Kotthoff, L., Bischl, B. (2019), mlr3: A modern object-oriented machine learing framework in R. Journal of Open Source Software, `doi:10.21105/joss.01903 `_. + +* Becker, M., Binder, M., Bischl, B., Lang, M., Pfisterer, F., Reich, N.G., Richter, J., Schratz, P., Sonabend, R. (2020), mlr3 book, available at `https://mlr3book.mlr-org.com `_. diff --git a/doc/guide/learners/r/tune_hyperparams.rst b/doc/guide/learners/r/tune_hyperparams.rst new file mode 100644 index 00000000..8567794c --- /dev/null +++ b/doc/guide/learners/r/tune_hyperparams.rst @@ -0,0 +1,177 @@ +Parameter tuning of learners for the nuisance functions of :ref:`DoubleML ` models can be done via the ``tune()`` method. +The ``tune()`` method passes various options and parameters to the tuning interface provided by the +`mlr3tuning `_ package. The `mlr3 book `_ provides a +`step-by-step introduction to parameter tuning `_. + +To illustrate the parameter tuning, we generate data from a sparse partially linear regression model. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + library(DoubleML) + library(mlr3) + library(data.table) + + set.seed(3141) + n_obs = 200 + n_vars = 200 + theta = 3 + X = matrix(stats::rnorm(n_obs * n_vars), nrow = n_obs, ncol = n_vars) + d = X[, 1:3, drop = FALSE] %*% c(5, 5, 5) + stats::rnorm(n_obs) + y = theta * d + X[, 1:3, drop = FALSE] %*% c(5, 5, 5) + stats::rnorm(n_obs) + dml_data = double_ml_data_from_matrix(X = X, y = y, d = d) + +The hyperparameter-tuning is performed according to options passed through a named list ``tune_settings``. +The entries in the list specify options during parameter tuning with `mlr3tuning `_: + +* ``terminator`` is a `Terminator object `_ passed to + `mlr3tuning `_ that manages the budget to solve the tuning problem. +* ``algorithm`` is an object of class + `Tuner `_ and specifies the tuning algorithm. + Alternatively, ``algorithm`` can be a ``character()`` that is used as an argument in the wrapper + `mlr3tuning `_ call + `tnr(algorithm) `_. + `The corresponding chapter in the mlr3book `_ illustrates + how the `Tuner `_ class supports grid search, random search, + generalized simulated annealing and non-linear optimization. +* ``rsmp_tune`` is an object of class `mlr3 resampling `_ + that specifies the resampling method for evaluation, for example `rsmp("cv", folds = 5)` implements 5-fold cross-validation. + `rsmp("holdout", ratio = 0.8)` implements an evaluation based on a hold-out sample that contains 20 percent of the observations. + By default, 5-fold cross-validation is performed. +* ``measure`` is a named list containing the measures used for tuning of the nuisance components. + The names of the entries must match the learner names (see method ``learner_names()``). The entries in the list must either be + objects of class `Measure `_ or keys passed to `msr() `_. + If ``measure`` is not provided by the user, default measures are used, i.e., mean squared error for regression models + and classification error for binary outcomes. + +In the following example, we tune the penalty parameter :math:`\lambda` (``lambda``) for lasso with the R package +`glmnet `_. To tune the value of ``lambda``, a grid search is performed over a grid of values that range from 0.05 +to 0.1 at a resolution of 10. Using a resolution of 10 splits the grid of values in 10 equally spaced values ranging from a minimum of 0.05 +to a maximum of 0.1. To evaluate the predictive performance in both nuisance parts, the cross-validated mean squared error is used. + +Setting the option ``tune_on_folds=FALSE``, the tuning is performed on the whole sample. Hence, the cross-validated errors +are obtained from a random split of the whole sample into 5 folds. As a result, one set of ``lambda`` values are obtained +which are later used in the fitting stage for all folds. + +Alternatively, setting the option ``tune_on_folds=TRUE`` would assign the tuning resampling scheme ``rsmp_tune`` to each fold. +For example, if we set ``n_folds=2`` at initialization of the ``DoubleMLPLR`` object and use a 5-fold cross-validated error +for tuning, each of the two folds would be split up into 5 subfolds and the error would be evaluated on these subfolds. + + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + library(DoubleML) + library(mlr3) + library(data.table) + library(mlr3learners) + library(mlr3tuning) + library(paradox) + lgr::get_logger("mlr3")$set_threshold("warn") + lgr::get_logger("bbotk")$set_threshold("warn") + + set.seed(1234) + ml_l = lrn("regr.glmnet") + ml_m = lrn("regr.glmnet") + dml_plr_obj = DoubleMLPLR$new(dml_data, ml_l, ml_m) + + par_grids = list( + "ml_l" = ps(lambda = p_dbl(lower = 0.05, upper = 0.1)), + "ml_m" = ps(lambda = p_dbl(lower = 0.05, upper = 0.1))) + + tune_settings = list(terminator = trm("evals", n_evals = 100), + algorithm = tnr("grid_search", resolution = 10), + rsmp_tune = rsmp("cv", folds = 5), + measure = list("ml_l" = msr("regr.mse"), + "ml_m" = msr("regr.mse"))) + dml_plr_obj$tune(param_set=par_grids, tune_settings=tune_settings, tune_on_fold=TRUE) + dml_plr_obj$params + + dml_plr_obj$fit() + dml_plr_obj$summary() + + +Hyperparameter tuning can also be done with more sophisticated methods, for example by using built-in tuning +paths of learners. For example, the learner `regr.cv_glmnet `_ +performs an internal cross-validated choice of the parameter ``lambda``. +Alternatively, the powerful functionalities of the `mlr3tuning `_ package can be used for +external parameter tuning of the nuisance parts. The optimally chosen parameters can then be passed to the +:ref:`DoubleML ` models using the ``set_ml_nuisance_params()`` method. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + library(DoubleML) + library(mlr3) + library(data.table) + library(mlr3learners) + library(mlr3tuning) + lgr::get_logger("mlr3")$set_threshold("warn") + lgr::get_logger("bbotk")$set_threshold("warn") + + set.seed(1234) + ml_l = lrn("regr.cv_glmnet", s="lambda.min") + ml_m = lrn("regr.cv_glmnet", s="lambda.min") + dml_plr_obj = DoubleMLPLR$new(dml_data, ml_l, ml_m) + + dml_plr_obj$fit() + dml_plr_obj$summary() + + +The following code chunk illustrates another example for global parameter tuning with random forests +as provided by the `ranger `_ package. In this example, we use random search to find optimal +parameters ``mtry`` and ``max.depth`` of a random forest. Evaluation is based on 3-fold cross-validation. + +.. tab-set:: + + .. tab-item:: R + :sync: r + + .. jupyter-execute:: + + library(DoubleML) + library(mlr3) + library(mlr3learners) + library(data.table) + library(mlr3tuning) + library(paradox) + lgr::get_logger("mlr3")$set_threshold("warn") + lgr::get_logger("bbotk")$set_threshold("warn") + + # set up a mlr3 learner + learner = lrn("regr.ranger") + ml_l = learner$clone() + ml_m = learner$clone() + + set.seed(3141) + obj_dml_data = make_plr_CCDDHNR2018(alpha=0.5) + dml_plr_obj = DoubleMLPLR$new(obj_dml_data, ml_l, ml_m) + + # set up a list of parameter grids + param_grid = list("ml_l" = ps(mtry = p_int(lower = 2 , upper = 20), + max.depth = p_int(lower = 2, upper = 5)), + "ml_m" = ps(mtry = p_int(lower = 2 , upper = 20), + max.depth = p_int(lower = 2, upper = 5))) + + tune_settings = list(terminator = mlr3tuning::trm("evals", n_evals = 20), + algorithm = tnr("random_search"), + rsmp_tune = rsmp("cv", folds = 3), + measure = list("ml_l" = msr("regr.mse"), + "ml_m" = msr("regr.mse"))) + dml_plr_obj$tune(param_set=param_grid, tune_settings=tune_settings, tune_on_folds=FALSE) + dml_plr_obj$params + + dml_plr_obj$fit() + dml_plr_obj$summary() \ No newline at end of file diff --git a/doc/guide/resampling.rst b/doc/guide/resampling.rst index 3670c28c..61c8e43a 100644 --- a/doc/guide/resampling.rst +++ b/doc/guide/resampling.rst @@ -203,6 +203,7 @@ The aggregation of the estimates of the causal parameter and its standard errors .. math:: \tilde{\theta}_{0} = \text{Median}\big((\tilde{\theta}_{0,m})_{m \in [M]}\big). + The estimate of the causal parameter :math:`\tilde{\theta}_{0}` is stored in the ``coef`` attribute and the asymptotic standard error :math:`\hat{\sigma}/\sqrt{N}` in ``se``. @@ -356,7 +357,7 @@ Note that cross-fitting performs well empirically and is recommended to remove b .. note:: The flag ``apply_cross_fitting`` is deprecated for the python package. To avoid cross-fitting, please use the option - to set :ref:`external predictions `. + to set :ref:`external predictions `. .. tab-item:: R :sync: r @@ -407,7 +408,7 @@ justification, see also :ref:`bias_overfitting`. .. note:: The flag ``apply_cross_fitting`` is deprecated for the python package. To avoid cross-fitting, please use the option - to set :ref:`external predictions `. Additionally, the number of folds ``n_folds`` is expected to be at least ``2``. + to set :ref:`external predictions `. Additionally, the number of folds ``n_folds`` is expected to be at least ``2``. .. tab-item:: R :sync: r diff --git a/doc/index.rst b/doc/index.rst index 6586a827..742a9cf7 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -263,7 +263,7 @@ Acknowledgements ---------------- Funding by the Deutsche Forschungsgemeinschaft (DFG, German Research -Foundation) is acknowledged – Project Number 431701914. +Foundation) is acknowledged – Project Number 431701914 and Grant GRK 2805/1. References ---------- diff --git a/doc/release/release.rst b/doc/release/release.rst index 101e2be5..ccfa60f4 100644 --- a/doc/release/release.rst +++ b/doc/release/release.rst @@ -7,6 +7,26 @@ Release Notes .. tab-item:: Python + .. dropdown:: DoubleML 0.11.1 + :class-title: sd-bg-primary sd-font-weight-bold + :open: + + - **Release highlight:** Hyperparameter tuning with `Optuna` (by `Jan Teichert-Kluge `_) + + - Implementation of ``tune_ml_models()`` method for DoubleML classes + `Py #368 `_ + - Extended User Guide and Example Gallery + `Docs #260 `_ + + - Maintenance package + `Py #372 `_ + `Py #374 `_ + `Py #377 `_ + + - Maintenance documentation + `Docs #261 `_ + `Docs #262 `_ + .. dropdown:: DoubleML 0.11.0 :class-title: sd-bg-primary sd-font-weight-bold :open: