diff --git a/lessons/StatisticalConsiderations/L2_Early_Stopping-zh.ipynb b/lessons/StatisticalConsiderations/L2_Early_Stopping-zh.ipynb
new file mode 100644
index 00000000..f5e1142f
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Early_Stopping-zh.ipynb
@@ -0,0 +1,208 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# 提前结束实验\n",
+ "\n",
+ "如果你在数据收集完之前窥视实验结果，并且因为检验展现出统计显著性而提前结束实验，那么 I 型错误率可能会显著上升，即认为实验已产生效果，但实际上没有。在此 notebook 中，你将重复视频中提出的断言：实验基于一个传统统计学检验，在实验运行到一半时窥视一次将使 I 型错误率从 5% 上升到约 8.6%。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import scipy.stats as stats"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "下面的模拟函数使用伯努利/二项式成功模型作为结果指标，并以历史基准为衡量依据。也就是说，每个观测值都好比投掷硬币一次，成功概率为“p”。如果基准值“p”的成功次数不太正常，则声明统计显著性结果。我们将实验分成多个模块，每个模块完成后都检查实验状态。目标输出是在任何检验中都具有统计显著性的试验所占的比例，以及在每个模块之后都具有统计显著性的试验所占的比例。\n",
+ "\n",
+ "填写 `peeking_sim()` 函数主要包含三个步骤。\n",
+ "\n",
+ "1.模拟数据\n",
+ " - 计算每个模块的试验次数。为简单起见，假设每个模块的试验次数都一样：每个模块最后的试验次数可能比对应的函数参数要大。\n",
+ " - 使用每个模块观察到的成功次数创建一个数据矩阵：行数等于模拟次数，列数等于模块数量。调用 numpy 的[`random.binomial`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.binomial.html) 函数一次即可完成此操作\n",
+ " \n",
+ "2.计算每个峰值处的 z 分数\n",
+ " - 对于每行，使用 numpy 的 [`cumsum`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cumsum.html) 函数计算每个模块之后的累积成功次数。结果应该是维度与数据相同的矩阵，但是每列都累积地将到此点的行值相加。\n",
+ " - 在每个模块之后计算成功次数的预期均值和标准差。注意，分布将基于[二项分布](https://en.wikipedia.org/wiki/Binomial_distribution)，并且以原始计数居中，而不是成功次数的比例。为了便于计数，有必要根据每个模块之后的试验次数累积之和创建一个向量。\n",
+ " - 使用累积计数、预期计数和标准差计算每个峰值的z 分数\n",
+ " \n",
+ "3.汇总检验结果\n",
+ " - 使用假设的 I 型错误率计算临界 z 值。根据此临界值标记哪些 z 分数具有统计显著性，哪些没有。\n",
+ " - 在任何检验中都具有显著性的试验所占的比例将等于至少具有一个标记值的行所占的比例。在每个模块都具有显著性的试验所占的比例将等于每列被标记值的平均数量，它是一个一维数组。将这两个值返回为函数的输出。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def peeking_sim(alpha = .05, p = .5, n_trials = 1000, n_blocks = 2, n_sims = 10000):\n",
+ " \"\"\"\n",
+ " This function simulates the rate of Type I errors made if an early\n",
+ " stopping decision is made based on a significant result when peeking ahead.\n",
+ " \n",
+ " Input parameters:\n",
+ " alpha: Supposed Type I error rate\n",
+ " p: Probability of individual trial success\n",
+ " n_trials: Number of trials in a full experiment\n",
+ " n_blocks: Number of times data is looked at (including end)\n",
+ " n_sims: Number of simulated experiments run\n",
+ " \n",
+ " Return:\n",
+ " p_sig_any: Proportion of simulations significant at any check point, \n",
+ " p_sig_each: Proportion of simulations significant at each check point\n",
+ " \"\"\"\n",
+ " \n",
+ " # generate data\n",
+ " trials_per_block = \n",
+ " data = \n",
+ " \n",
+ " # standardize data\n",
+ " data_cumsum = \n",
+ " block_sizes = \n",
+ " block_means = \n",
+ " block_sds = \n",
+ " data_zscores =\n",
+ " \n",
+ " # test outcomes\n",
+ " z_crit = \n",
+ " sig_flags = \n",
+ " p_sig_any = \n",
+ " p_sig_each = \n",
+ " \n",
+ " return (p_sig_any, p_sig_each)\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "根据提供的默认参数运行函数应该返回一个结果元组，其中在两个模块中任何具有统计显著性的检验结果的概率应该约为 8.6%，在每个模块检查点具有统计显著性的检验结果概率约为 5%。增加试验次数和模拟次数可以获得更准确的估算值。并且增加峰值后，总体 I 型错误率应该会增加。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "peeking_sim()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 提前窥视多次比较法\n",
+ "\n",
+ "避免多次检查并作出糟糕的提前结束决策的最保险方法是不要这么做。规划好实验并检查了所有分配流程后，应该让实验一直运行完毕，并在最终结束时评估结果。这并不是说不能提前结束，但是需要额外的规划工作。\n",
+ "\n",
+ "解决多次窥视的一种方式是调整单个检验的显著水平，使总体错误率保持理想水平。但是采用之前演示的帮费罗尼或 Šidák 校正法肯定过于保守了，因为我们知道峰值之间的检验结果存在联系。如果我们在中间位置看到某些模拟检验的 z 分数高于阈值，那么与在中间点不具统计显著性的其他模拟检验相比，这些检验在结束时更有可能高于该阈值。获得更好的显著性阈值的一种方式是采用模拟法。在执行了上述第 1 步和第 2 步后，我们希望得出使期望的模拟检验所占比例具有统计显著性的显著水平：\n",
+ "\n",
+ "1. 模拟数据（同上）\n",
+ "2. 在每个峰值计算 z 分数（同上）\n",
+ "3. 达到所需的单个检验错误率\n",
+ " - 如果在任何峰值都超过临界值，则一次实验被视为具有统计显著性。从每行获取最大 z 分数，作为零假设被错误拒绝的最坏情形。\n",
+ " - 算出会拒绝期望的总体 I 型错误率的 z 分数阈值。\n",
+ " - 将 z 分数转换为对等的单个检验错误率。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def peeking_correction(alpha = .05, p = .5, n_trials = 1000, n_blocks = 2, n_sims = 10000):\n",
+ " \"\"\"\n",
+ " This function uses simulations to estimate the individual error rate necessary\n",
+ " to limit the Type I error rate, if an early stopping decision is made based on\n",
+ " a significant result when peeking ahead.\n",
+ " \n",
+ " Input parameters:\n",
+ " alpha: Desired overall Type I error rate\n",
+ " p: Probability of individual trial success\n",
+ " n_trials: Number of trials in a full experiment\n",
+ " n_blocks: Number of times data is looked at (including end)\n",
+ " n_sims: Number of simulated experiments run\n",
+ " \n",
+ " Return:\n",
+ " alpha_ind: Individual error rate required to achieve overall error rate\n",
+ " \"\"\"\n",
+ " \n",
+ " # generate data\n",
+ " # You can copy the code from the previous function.\n",
+ " \n",
+ " \n",
+ " # standardize data\n",
+ " # You can copy the code from the previous function.\n",
+ " \n",
+ " \n",
+ " # find necessary individual error rate\n",
+ " max_zscores = \n",
+ " z_crit_ind = \n",
+ " alpha_ind = \n",
+ " \n",
+ " return alpha_ind"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "使用默认参数运行函数应该会达到所需的单个错误率 （约 0.029）。注意，该错误率比帮费罗尼校正和 Šidák 校正生成的错误率 .025 或 .0253 高一些。通过更多的模拟和试验获得更准确的估算值，并尝试不同数量的模块，看看会如何更改所需的单个错误率。结果应该与[这篇文章](https://www.evanmiller.org/how-not-to-run-an-ab-test.html)中间的表格中列出的数字大致相同；注意，窥视 $n$ 次表示将实验拆分为 $n + 1$ 个模块。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "peeking_correction()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "```python\n",
+ "\n",
+ "```"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Early_Stopping.ipynb b/lessons/StatisticalConsiderations/L2_Early_Stopping.ipynb
new file mode 100644
index 00000000..ba692bf2
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Early_Stopping.ipynb
@@ -0,0 +1,204 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Early Stopping\n",
+ "\n",
+ "If you peek at the results of an experiment before data collection is complete, and choose to stop early because the test is showing statistical significance, you run the risk of a significant increase in your Type I error rate: believing that your experiment had an effect, when in fact it did not. In this notebook, you'll duplicate the assertions made in the video: namely that for an experiment based off of a single traditional statistical test, doing a single peek halfway through the run-time will increase a base Type I error rate from 5% to about 8.6%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import scipy.stats as stats"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The simulation function below uses a bernoulli / binomial success model for the outcome metric, measuring against a historic baseline. That is, each observation is a single coin flip with success probability \"p\". If we see a number of successes that is unusual for our baseline value of \"p\", then we declare a statistically significant result. We will divide the experiment length into multiple 'blocks', checking the status of the experiment after each block is complete. Our outputs of interest are the proportion of trials that are statistically significant in _any_ test, and the proportion of trials that are statistically significant after _each_ individual block.\n",
+ "\n",
+ "There are three main steps to filling out the `peeking_sim()` function.\n",
+ "\n",
+ "1. Simulate some data\n",
+ " - Compute the number of trials per block. For simplicity, just round up any fractions so that each block has the same number of trials: we might end up with slightly more trials per block than the corresponding function parameter.\n",
+ " - Generate a data matrix with the number of successes observed in each block: the number of rows should be the number of simulations and the number of columns the number of blocks. You can do this with a single call to numpy's [`random.binomial`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.binomial.html) function.\n",
+ "2. Compute z-scores at each 'peek'\n",
+ " - For each row, compute the cumulative number of successes after each 'block' of the experiment using numpy's [`cumsum`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cumsum.html) function. The result should be a matrix with the same dimensions as the data, but each column cumulatively adds up the values in each row up to that point.\n",
+ " - Compute the expected mean and standard deviation for the number of successes after each 'block' of the experiment. Remember that this will be based on the [binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution) and is centered on the raw counts, rather than proportion of successes. It'll be useful to create a vector with the cumulative sum of trials after each block to facilitate these calculations.\n",
+ " - Use the cumulative counts, the expected counts, and the standard deviations, to compute the z-scores for each peek at the experiment.\n",
+ "3. Aggregate test outcomes\n",
+ " - Compute a critical z-value using the supposed Type I error rate. Use this critical value to flag which of the z-scores would be counted as statistically significant, and which would not.\n",
+ " - The proportion of trials that are significant at _any_ test will be the proportion of rows that have at least one flagged value. The proportion of trials that are significant at _each_ block will be the mean number of flagged values in each column; this will be a 1-d array. Return both of these values as the output of the function."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def peeking_sim(alpha = .05, p = .5, n_trials = 1000, n_blocks = 2, n_sims = 10000):\n",
+ " \"\"\"\n",
+ " This function simulates the rate of Type I errors made if an early\n",
+ " stopping decision is made based on a significant result when peeking ahead.\n",
+ " \n",
+ " Input parameters:\n",
+ " alpha: Supposed Type I error rate\n",
+ " p: Probability of individual trial success\n",
+ " n_trials: Number of trials in a full experiment\n",
+ " n_blocks: Number of times data is looked at (including end)\n",
+ " n_sims: Number of simulated experiments run\n",
+ " \n",
+ " Return:\n",
+ " p_sig_any: Proportion of simulations significant at any check point, \n",
+ " p_sig_each: Proportion of simulations significant at each check point\n",
+ " \"\"\"\n",
+ " \n",
+ " # generate data\n",
+ " trials_per_block = \n",
+ " data = \n",
+ " \n",
+ " # standardize data\n",
+ " data_cumsum = \n",
+ " block_sizes = \n",
+ " block_means = \n",
+ " block_sds = \n",
+ " data_zscores =\n",
+ " \n",
+ " # test outcomes\n",
+ " z_crit = \n",
+ " sig_flags = \n",
+ " p_sig_any = \n",
+ " p_sig_each = \n",
+ " \n",
+ " return (p_sig_any, p_sig_each)\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Running the function on the default parameters as given should return a tuple of results where the probability of any significant test outcome across the two blocks is around 8.6% and the probability of a significant test outcome at each individual block checkpoint is around 5%. Increase the number of trials and number of simulations to get more accurate estimates. You should also see how the overall Type I error rate increases with additional peeks!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "peeking_sim()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## A Multiple Comparisons Approach to Early Peeking\n",
+ "\n",
+ "The safest way we could deal with performing multiple checks and making poor early stopping decisions is to simply not do it. Once an experiment has been planned and all assignment procedures checked, you should just let the experiment run to completion and just assess the results at the very end. That's not to say that you can't perform early stopping, but it does require additional planning.\n",
+ "\n",
+ "One way in which you could solve for multiple peeking is to adjust the significance level of individual tests so that the overall error rate is at its desired level. But applying the Bonferroni or Šidák corrections as shown earlier in the lesson will definitely be too conservative, since we know that there is a correlation in test results between peeks. If we see some simulated run with z-score above the threshold at the halfway point, it's more likely to be above that threshold at the end point, compared to some other simulated run that is not statistically significant at the halfway point. One way in which we can obtain a better significance threshold is through the power of simulation. After performing the same steps 1 and 2 above, we want to find a significance level that would call our desired proportion of simulated tests as statistically significant:\n",
+ "\n",
+ "1. Simulate some data (as above)\n",
+ "2. Compute z-scores at each 'peek' (as above)\n",
+ "3. Obtain required individual-test error rate\n",
+ " - A run is considered statistically significant if it exceeds the critical bounds at _any_ peek. Obtain the maximum z-score from each row as a worst-case scenario for a null run to be falsely rejected.\n",
+ " - Find the z-score threshold that would reject our desired overall Type I error rate.\n",
+ " - Convert that z-score into an equivalent individual-test error rate."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def peeking_correction(alpha = .05, p = .5, n_trials = 1000, n_blocks = 2, n_sims = 10000):\n",
+ " \"\"\"\n",
+ " This function uses simulations to estimate the individual error rate necessary\n",
+ " to limit the Type I error rate, if an early stopping decision is made based on\n",
+ " a significant result when peeking ahead.\n",
+ " \n",
+ " Input parameters:\n",
+ " alpha: Desired overall Type I error rate\n",
+ " p: Probability of individual trial success\n",
+ " n_trials: Number of trials in a full experiment\n",
+ " n_blocks: Number of times data is looked at (including end)\n",
+ " n_sims: Number of simulated experiments run\n",
+ " \n",
+ " Return:\n",
+ " alpha_ind: Individual error rate required to achieve overall error rate\n",
+ " \"\"\"\n",
+ " \n",
+ " # generate data\n",
+ " # You can copy the code from the previous function.\n",
+ " \n",
+ " \n",
+ " # standardize data\n",
+ " # You can copy the code from the previous function.\n",
+ " \n",
+ " \n",
+ " # find necessary individual error rate\n",
+ " max_zscores = \n",
+ " z_crit_ind = \n",
+ " alpha_ind = \n",
+ " \n",
+ " return alpha_ind"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Running the function on the default parameters should give a required individual error rate of about .029. Note how this is somewhat higher than the .025 or .0253 that would have been generated from the Bonferroni and Šidák corrections, respectively. Test with a higher number of simulations and trials to get more accurate estimates, and try out different numbers of blocks to see how it changes the individual error rate needed. The results should approximately match up with the numbers given in the table in the middle of [this article](https://www.evanmiller.org/how-not-to-run-an-ab-test.html); note that peeking $n$ times means splitting the experiment into $n + 1$ blocks."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "peeking_correction()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Experiment_Size-zh.ipynb b/lessons/StatisticalConsiderations/L2_Experiment_Size-zh.ipynb
new file mode 100644
index 00000000..b45e2adf
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Experiment_Size-zh.ipynb
@@ -0,0 +1,274 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# 实验大小\n",
+ "\n",
+ "我们可以根据期望的实际显著性界限规划实验。我们需要知道，需要多少观察值，才能让我们的实验运行结果变得有效，然后再来判断实验需要运行多久，以及是否可行。\n",
+ "\n",
+ "我们再看看视频中的示例，其中基准点击率为 10%，我们希望操控后点击率上升到 12%。每个小组需要多少观察值，才能检测到此变化？其中功效为 $1-\\beta = .80$（即有 80% 的时间检测到 2% 的绝对增加量），I 型错误率为 $\\alpha = .05$。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import packages\n",
+ "import numpy as np\n",
+ "import scipy.stats as stats\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 方法 1：试错法\n",
+ "\n",
+ "一种方法是试错法。每个样本量都存在一定的功效水平；测试多个样本量将逐渐使我们接近获得期望的功效水平所需的最低样本量。这并不是很高效的方法，但是能够直观地说明实验大小的确定方式。\n",
+ "\n",
+ "请按以下步骤填写 `power()` 函数：\n",
+ "\n",
+ "1.在零假设下，应该有一个临界值使 I 型错误率达到期望的 α 水平。\n",
+ " - `se_null`：计算两组在零假设下的比例差异标准差。`p_null` 会提供基础概率。注意，差异分布的方差等于单个分布的方差之和，每个小组有 `n` 个观察值。\n",
+ " - `null_dist`：为了便于重复利用，它应该是一个 [SciPy 规范对象](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html)。分别使用“loc”和“scale”参数指定正态分布的中心和标准差。\n",
+ " - `p_crit`：计算会导致我们拒绝零假设的分布临界值。`null_dist` 对象的某个方法会帮助我们获得此值（传入期望的错误率 `alpha` 的某个函数）。\n",
+ "\n",
+ "2.功效是指在零假设下超过之前获得的临界值的分布比例。\n",
+ " - `se_alt`：现在朝着另一个方向计算值。它将等于期望的可检测差异下的差异标准差。注意，单个分布的方差现在将不同：一个的成功概率为 `p_null`，另一个的成功概率为 `p_alt`。\n",
+ " - `alt_dist`：它将是如上所示的 SciPy 规范对象。注意这个对象里的“loc”参数。根据 `power` 函数的设置，它要求 `p_alt` 大于 `p_null`，使差异为正。\n",
+ " - `beta`：Beta 是 II 型错误的概率，或者特定非零状态未能拒绝零假设的概率。这表示你应该利用 `alt_dist` 和 `p_crit`。\n",
+ "\n",
+ "我们已经完成函数的第二部分，这部分会可视化零假设的差异和期望的可检测差异的分布情况。请在后面的单元格中运行该函数，并观察可视化结果，然后用一些断言语句测试代码。如果你想查看答案，请转到下一页。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def power(p_null, p_alt, n, alpha = .05, plot = True):\n",
+ " \"\"\"\n",
+ " Compute the power of detecting the difference in two populations with \n",
+ " different proportion parameters, given a desired alpha rate.\n",
+ " \n",
+ " Input parameters:\n",
+ " p_null: base success rate under null hypothesis\n",
+ " p_alt : desired success rate to be detected, must be larger than\n",
+ " p_null\n",
+ " n : number of observations made in each group\n",
+ " alpha : Type-I error rate\n",
+ " plot : boolean for whether or not a plot of distributions will be\n",
+ " created\n",
+ " \n",
+ " Output value:\n",
+ " power : Power to detect the desired difference, under the null.\n",
+ " \"\"\"\n",
+ " \n",
+ " # Compute the power\n",
+ " se_null = \n",
+ " null_dist = \n",
+ " p_crit = \n",
+ " \n",
+ " se_alt = \n",
+ " alt_dist = \n",
+ " beta = \n",
+ " \n",
+ " if plot:\n",
+ " # Compute distribution heights\n",
+ " low_bound = null_dist.ppf(.01)\n",
+ " high_bound = alt_dist.ppf(.99)\n",
+ " x = np.linspace(low_bound, high_bound, 201)\n",
+ " y_null = null_dist.pdf(x)\n",
+ " y_alt = alt_dist.pdf(x)\n",
+ "\n",
+ " # Plot the distributions\n",
+ " plt.plot(x, y_null)\n",
+ " plt.plot(x, y_alt)\n",
+ " plt.vlines(p_crit, 0, np.amax([null_dist.pdf(p_crit), alt_dist.pdf(p_crit)]),\n",
+ " linestyles = '--')\n",
+ " plt.fill_between(x, y_null, 0, where = (x >= p_crit), alpha = .5)\n",
+ " plt.fill_between(x, y_alt , 0, where = (x <= p_crit), alpha = .5)\n",
+ " \n",
+ " plt.legend(['null','alt'])\n",
+ " plt.xlabel('difference')\n",
+ " plt.ylabel('density')\n",
+ " plt.show()\n",
+ " \n",
+ " # return power\n",
+ " return (1 - beta)\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "power(.1, .12, 1000)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "assert np.isclose(power(.1, .12, 1000, plot = False), 0.4412, atol = 1e-4)\n",
+ "assert np.isclose(power(.1, .12, 3000, plot = False), 0.8157, atol = 1e-4)\n",
+ "assert np.isclose(power(.1, .12, 5000, plot = False), 0.9474, atol = 1e-4)\n",
+ "print('You should see this message if all the assertions passed!')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 方法 2：分析方法\n",
+ "\n",
+ "你已经通过试错法大概了解了功效，现在我们可以通过封闭解计算最小实验大小了。要注意的关键一点是，如果 $\\alpha$ 和 $\\beta$ 都小于 5，那么确定统计显著性的临界值将在零假设点击率和期望的备择点击率之间。所以，$p_0$ 和 $p_1$ 之间的差异可以再划分为从 $p_0$ 到临界值 $p^*$ 的距离和从 $p^*$ 到 $p_1$ 的距离。\n",
+ "\n",
+ "
\n",
+ "\n",
+ "这些细分差异可以用标准误差和 z 分数表示：\n",
+ "\n",
+ "$$p^* - p_0 = z_{1-\\alpha} SE_{0},$$\n",
+ "$$p_1 - p^* = -z_{\\beta} SE_{1};$$\n",
+ "\n",
+ "$$p_1 - p_0 = z_{1-\\alpha} SE_{0} - z_{\\beta} SE_{1}$$\n",
+ "\n",
+ "标准误差的计算方法是分布的标准差除以每组的样本数的平方根：\n",
+ "\n",
+ "$$SE_{0} = \\frac{s_{0}}{\\sqrt{n}},$$\n",
+ "$$SE_{1} = \\frac{s_{1}}{\\sqrt{n}}$$\n",
+ "\n",
+ "代替这些值并求解 $n$ 将得出一个公式，使我们能够计算在期望的功效水平检测到指定差异的最小样本量：\n",
+ "\n",
+ "$$n = \\lceil \\big(\\frac{z_{\\alpha} s_{0} - z_{\\beta} s_{1}}{p_1 - p_0}\\big)^2 \\rceil$$\n",
+ "\n",
+ "其中 $\\lceil ...\\rceil$ 表示顶函数，并且将小数值四舍五入到下个更大的整数。请在下面实现必要的函数变量，然后在后续单元格中测试这些变量。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def experiment_size(p_null, p_alt, alpha = .05, beta = .20):\n",
+ " \"\"\"\n",
+ " Compute the minimum number of samples needed to achieve a desired power\n",
+ " level for a given effect size.\n",
+ " \n",
+ " Input parameters:\n",
+ " p_null: base success rate under null hypothesis\n",
+ " p_alt : desired success rate to be detected\n",
+ " alpha : Type-I error rate\n",
+ " beta : Type-II error rate\n",
+ " \n",
+ " Output value:\n",
+ " n : Number of samples required for each group to obtain desired power\n",
+ " \"\"\"\n",
+ " \n",
+ " # Get necessary z-scores and standard deviations (@ 1 obs per group)\n",
+ " z_null = \n",
+ " z_alt = \n",
+ " sd_null = \n",
+ " sd_alt = \n",
+ " \n",
+ " # Compute and return minimum sample size\n",
+ " n = \n",
+ " return np.ceil(n)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "experiment_size(.1, .12)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "assert np.isclose(experiment_size(.1, .12), 2863)\n",
+ "print('You should see this message if the assertion passed!')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 解析注意事项\n",
+ "\n",
+ "上述示例是一个单尾检验，备择假设的值比零假设的要大。如果备择假设的比例大于零假设的比例，那么第一部分的功效计算方法将不可行，例如检测到比例参数为 0.88，而零假设的为 0.9。你可能需要重新编写代码来处理这种情形。第二种方法应该不会出现这种问题，我们直接计算了样本量。\n",
+ "\n",
+ "如果你需要执行双尾检验，则要注意两大事项。首先，“alpha”参数需要考虑到拒绝区域分成了两个部分。其次，你应该根据最糟糕情形完成计算，即具有最高可变性的备择情形。由于在二项分布中，当 $p = .5$ 时方差最高，并且当 $p$ 接近 0 或 1 时方差会降低，所以在计算必要样本量时，应该选择非常接近 0.5 的备择值作为参考。\n",
+ "\n",
+ "注意，上述方法仅考虑了统计显著性，没有考虑实际显著性。需要注意的一点是，如果实验效果的真实大小与期望的实际显著性水平一样，那么均值高于或低于实际显著性界限的概率是 0.5。这甚至没有考虑到置信区间对界限的影响。从某种程度上来说，确定实验大小可以检查你能否通过运行实验获得想要的结果，而不是检查你是否将获得所需的结果。\n",
+ "\n",
+ "## 其他方法\n",
+ "\n",
+ "还有一些工具和 Python 软件包可以帮助你确定样本量，所以你不需要自己处理每种情形。这个[样本量计算器](http://www.evanmiller.org/ab-testing/sample-size.html)适用于比例，并且提供的结果与上述方法的一样。（但是注意，该计算器针对的是双尾检验。）Python 软件包“statsmodels”在其 [`power` 模块](https://www.statsmodels.org/stable/stats.html#power-and-sample-size-calculations)中提供了多种计算功效和样本量的函数。与之前演示的方法不同，零假设和备择假设之间的差异会参数化为效应量（小组均值之间的标准化差异除以标准差）。所以，这些函数不仅仅可用于比例检验。如果我们想完成和之前一样的检验，[`proportion_effectsize`](http://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportion_effectsize.html) 函数会计算 [Cohen's h](https://en.wikipedia.org/wiki/Cohen%27s_h)，用于衡量效应量。因此，statsmodel 函数的输出应该与上述预期结果不同。这并不是大问题，因为在大多数情形下，你并不会根据具体的观察值数量停止实验，只是根据该值做出一般设计决策。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# example of using statsmodels for sample size calculation\n",
+ "from statsmodels.stats.power import NormalIndPower\n",
+ "from statsmodels.stats.proportion import proportion_effectsize\n",
+ "\n",
+ "# leave out the \"nobs\" parameter to solve for it\n",
+ "NormalIndPower().solve_power(effect_size = proportion_effectsize(.12, .1), alpha = .05, power = 0.8,\n",
+ " alternative = 'larger')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "```python\n",
+ "\n",
+ "```"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Experiment_Size.ipynb b/lessons/StatisticalConsiderations/L2_Experiment_Size.ipynb
new file mode 100644
index 00000000..dc5c1147
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Experiment_Size.ipynb
@@ -0,0 +1,276 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Experiment Size\n",
+ "\n",
+ "We can use the knowledge of our desired practical significance boundary to plan out our experiment. By knowing how many observations we need in order to detect our desired effect to our desired level of reliability, we can see how long we would need to run our experiment and whether or not it is feasible.\n",
+ "\n",
+ "Let's use the example from the video, where we have a baseline click-through rate of 10% and want to see a manipulation increase this baseline to 12%. How many observations would we need in each group in order to detect this change with power $1-\\beta = .80$ (i.e. detect the 2% absolute increase 80% of the time), at a Type I error rate of $\\alpha = .05$?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import packages\n",
+ "import numpy as np\n",
+ "import scipy.stats as stats\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Method 1: Trial and Error\n",
+ "\n",
+ "One way we could solve this is through trial and error. Every sample size will have a level of power associated with it; testing multiple sample sizes will gradually allow us to narrow down the minimum sample size required to obtain our desired power level. This isn't a particularly efficient method, but it can provide an intuition for how experiment sizing works.\n",
+ "\n",
+ "Fill in the `power()` function below following these steps:\n",
+ "\n",
+ "1. Under the null hypothesis, we should have a critical value for which the Type I error rate is at our desired alpha level.\n",
+ " - `se_null`: Compute the standard deviation for the difference in proportions under the null hypothesis for our two groups. The base probability is given by `p_null`. Remember that the variance of the difference distribution is the sum of the variances for the individual distributions, and that _each_ group is assigned `n` observations.\n",
+ " - `null_dist`: To assist in re-use, this should be a [scipy norm object](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html). Specify the center and standard deviation of the normal distribution using the \"loc\" and \"scale\" arguments, respectively.\n",
+ " - `p_crit`: Compute the critical value of the distribution that would cause us to reject the null hypothesis. One of the methods of the `null_dist` object will help you obtain this value (passing in some function of our desired error rate `alpha`).\n",
+ "2. The power is the proportion of the distribution under the alternative hypothesis that is past that previously-obtained critical value.\n",
+ " - `se_alt`: Now it's time to make computations in the other direction. This will be standard deviation of differences under the desired detectable difference. Note that the individual distributions will have different variances now: one with `p_null` probability of success, and the other with `p_alt` probability of success.\n",
+ " - `alt_dist`: This will be a scipy norm object like above. Be careful of the \"loc\" argument in this one. The way the `power` function is set up, it expects `p_alt` to be greater than `p_null`, for a positive difference.\n",
+ " - `beta`: Beta is the probability of a Type-II error, or the probability of failing to reject the null for a particular non-null state. That means you should make use of `alt_dist` and `p_crit` here!\n",
+ "\n",
+ "The second half of the function has already been completed for you, which creates a visualization of the distribution of differences for the null case and for the desired detectable difference. Use the cells that follow to run the function and observe the visualizations, and to test your code against a few assertion statements. Check the following page if you need help coming up with the solution."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def power(p_null, p_alt, n, alpha = .05, plot = True):\n",
+ " \"\"\"\n",
+ " Compute the power of detecting the difference in two populations with \n",
+ " different proportion parameters, given a desired alpha rate.\n",
+ " \n",
+ " Input parameters:\n",
+ " p_null: base success rate under null hypothesis\n",
+ " p_alt : desired success rate to be detected, must be larger than\n",
+ " p_null\n",
+ " n : number of observations made in each group\n",
+ " alpha : Type-I error rate\n",
+ " plot : boolean for whether or not a plot of distributions will be\n",
+ " created\n",
+ " \n",
+ " Output value:\n",
+ " power : Power to detect the desired difference, under the null.\n",
+ " \"\"\"\n",
+ " \n",
+ " # Compute the power\n",
+ " se_null = \n",
+ " null_dist = \n",
+ " p_crit = \n",
+ " \n",
+ " se_alt = \n",
+ " alt_dist = \n",
+ " beta = \n",
+ " \n",
+ " if plot:\n",
+ " # Compute distribution heights\n",
+ " low_bound = null_dist.ppf(.01)\n",
+ " high_bound = alt_dist.ppf(.99)\n",
+ " x = np.linspace(low_bound, high_bound, 201)\n",
+ " y_null = null_dist.pdf(x)\n",
+ " y_alt = alt_dist.pdf(x)\n",
+ "\n",
+ " # Plot the distributions\n",
+ " plt.plot(x, y_null)\n",
+ " plt.plot(x, y_alt)\n",
+ " plt.vlines(p_crit, 0, np.amax([null_dist.pdf(p_crit), alt_dist.pdf(p_crit)]),\n",
+ " linestyles = '--')\n",
+ " plt.fill_between(x, y_null, 0, where = (x >= p_crit), alpha = .5)\n",
+ " plt.fill_between(x, y_alt , 0, where = (x <= p_crit), alpha = .5)\n",
+ " \n",
+ " plt.legend(['null','alt'])\n",
+ " plt.xlabel('difference')\n",
+ " plt.ylabel('density')\n",
+ " plt.show()\n",
+ " \n",
+ " # return power\n",
+ " return (1 - beta)\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "power(.1, .12, 1000)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "assert np.isclose(power(.1, .12, 1000, plot = False), 0.4412, atol = 1e-4)\n",
+ "assert np.isclose(power(.1, .12, 3000, plot = False), 0.8157, atol = 1e-4)\n",
+ "assert np.isclose(power(.1, .12, 5000, plot = False), 0.9474, atol = 1e-4)\n",
+ "print('You should see this message if all the assertions passed!')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Method 2: Analytic Solution\n",
+ "\n",
+ "Now that we've got some intuition for power by using trial and error, we can now approach a closed-form solution for computing a minimum experiment size. The key point to notice is that, for an $\\alpha$ and $\\beta$ both < .5, the critical value for determining statistical significance will fall between our null click-through rate and our alternative, desired click-through rate. So, the difference between $p_0$ and $p_1$ can be subdivided into the distance from $p_0$ to the critical value $p^*$ and the distance from $p^*$ to $p_1$.\n",
+ "\n",
+ "\n",
+ "\n",
+ "Those subdivisions can be expressed in terms of the standard error and the z-scores:\n",
+ "\n",
+ "$$p^* - p_0 = z_{1-\\alpha} SE_{0},$$\n",
+ "$$p_1 - p^* = -z_{\\beta} SE_{1};$$\n",
+ "\n",
+ "$$p_1 - p_0 = z_{1-\\alpha} SE_{0} - z_{\\beta} SE_{1}$$\n",
+ "\n",
+ "In turn, the standard errors can be expressed in terms of the standard deviations of the distributions, divided by the square root of the number of samples in each group:\n",
+ "\n",
+ "$$SE_{0} = \\frac{s_{0}}{\\sqrt{n}},$$\n",
+ "$$SE_{1} = \\frac{s_{1}}{\\sqrt{n}}$$\n",
+ "\n",
+ "Substituting these values in and solving for $n$ will give us a formula for computing a minimum sample size to detect a specified difference, at the desired level of power:\n",
+ "\n",
+ "$$n = \\lceil \\big(\\frac{z_{\\alpha} s_{0} - z_{\\beta} s_{1}}{p_1 - p_0}\\big)^2 \\rceil$$\n",
+ "\n",
+ "where $\\lceil ... \\rceil$ represents the ceiling function, rounding up decimal values to the next-higher integer. Implement the necessary variables in the function below, and test them with the cells that follow."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def experiment_size(p_null, p_alt, alpha = .05, beta = .20):\n",
+ " \"\"\"\n",
+ " Compute the minimum number of samples needed to achieve a desired power\n",
+ " level for a given effect size.\n",
+ " \n",
+ " Input parameters:\n",
+ " p_null: base success rate under null hypothesis\n",
+ " p_alt : desired success rate to be detected\n",
+ " alpha : Type-I error rate\n",
+ " beta : Type-II error rate\n",
+ " \n",
+ " Output value:\n",
+ " n : Number of samples required for each group to obtain desired power\n",
+ " \"\"\"\n",
+ " \n",
+ " # Get necessary z-scores and standard deviations (@ 1 obs per group)\n",
+ " z_null = \n",
+ " z_alt = \n",
+ " sd_null = \n",
+ " sd_alt = \n",
+ " \n",
+ " # Compute and return minimum sample size\n",
+ " n = \n",
+ " return np.ceil(n)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "experiment_size(.1, .12)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "assert np.isclose(experiment_size(.1, .12), 2863)\n",
+ "print('You should see this message if the assertion passed!')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Notes on Interpretation\n",
+ "\n",
+ "The example explored above is a one-tailed test, with the alternative value greater than the null. The power computations performed in the first part will _not_ work if the alternative proportion is less than the null, e.g. detecting a proportion parameter of 0.88 against a null of 0.9. You might want to try to rewrite the code to handle that case! The same issue should not show up for the second approach, where we directly compute the sample size.\n",
+ "\n",
+ "If you find that you need to do a two-tailed test, you should pay attention to two main things. First of all, the \"alpha\" parameter needs to account for the fact that the rejection region is divided into two areas. Secondly, you should perform the computation based on the worst-case scenario, the alternative case with the highest variability. Since, for the binomial, variance is highest when $p = .5$, decreasing as $p$ approaches 0 or 1, you should choose the alternative value that is closest to .5 as your reference when computing the necessary sample size.\n",
+ "\n",
+ "Note as well that the above methods only perform sizing for _statistical significance_, and do not take into account _practical significance_. One thing to realize is that if the true size of the experimental effect is the same as the desired practical significance level, then it's a coin flip whether the mean will be above or below the practical significance bound. This also doesn't even consider how a confidence interval might interact with that bound. In a way, experiment sizing is a way of checking on whether or not you'll be able to get what you _want_ from running an experiment, rather than checking if you'll get what you _need_."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Alternative Approaches\n",
+ "\n",
+ "There are also tools and Python packages that can also help with sample sizing decisions, so you don't need to solve for every case on your own. The sample size calculator [here](http://www.evanmiller.org/ab-testing/sample-size.html) is applicable for proportions, and provides the same results as the methods explored above. (Note that the calculator assumes a two-tailed test, however.) Python package \"statsmodels\" has a number of functions in its [`power` module](https://www.statsmodels.org/stable/stats.html#power-and-sample-size-calculations) that perform power and sample size calculations. Unlike previously shown methods, differences between null and alternative are parameterized as an effect size (standardized difference between group means divided by the standard deviation). Thus, we can use these functions for more than just tests of proportions. If we want to do the same tests as before, the [`proportion_effectsize`](http://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportion_effectsize.html) function computes [Cohen's h](https://en.wikipedia.org/wiki/Cohen%27s_h) as a measure of effect size. As a result, the output of the statsmodel functions will be different from the result expected above. This shouldn't be a major concern since in most cases, you're not going to be stopping based on an exact number of observations. You'll just use the value to make general design decisions."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# example of using statsmodels for sample size calculation\n",
+ "from statsmodels.stats.power import NormalIndPower\n",
+ "from statsmodels.stats.proportion import proportion_effectsize\n",
+ "\n",
+ "# leave out the \"nobs\" parameter to solve for it\n",
+ "NormalIndPower().solve_power(effect_size = proportion_effectsize(.12, .1), alpha = .05, power = 0.8,\n",
+ " alternative = 'larger')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_1-zh.ipynb b/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_1-zh.ipynb
new file mode 100644
index 00000000..54247d30
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_1-zh.ipynb
@@ -0,0 +1,267 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# 非参数检验（第一部分）\n",
+ "\n",
+ "到目前为止，你在设计和分析实验时，对正态分布的均值采用的都是标准假设检验。但是，有时候不能只依赖于标准检验。原因可能是不确定指标分布的真实变化性、缺少满足正态要求的数据，或者想对缺少标准检验的统计量进行推理。这时候就有必要了解一些**非参数检验**，不仅仅是为了处理上述情况，而且可以作为实验结果的辅助检查工具。非参数检验的主要优势是，它们不依赖于对底层总体做出很多假设，与标准检验相比，非参数检验适用于更广泛的情形。在此 notebook 中，你将学习两种非参数检验方法，这两种方法会对数据重新抽样，并对分布和差异做出推理。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 自助法\n",
+ "\n",
+ "自助法会使用实际收集的数据生成假设会收集的新样本，并估算抽样分布情况。在标准自助法中，自助样本是指从原始数据中抽出放回地抽取数据点，直到获得和原始数据一样多的数据点。本质上，我们将原始数据当做总体：不对原始总体分布做出假设，我们最多只能使用原始数据作为总体模型。\n",
+ "\n",
+ "获取很多自助样本使我们能够针对原始数据的各种统计量估算抽样分布。例如，假设我们想对数据集（有 5000 个数据）中的第 90 百分位数值创建 95% 的置信区间。（也许我们查看的是网站加载时间，想要减少最糟糕的情况。）通过自助法，可以轻松地估算结果。首先，我们进行自助抽样（即从原始数据中抽出放回地抽取 5000 个数据点），记录第 90 百分位数值，然后重复这一过程很多次，例如 100 000 次。根据这么多自助的第 90 百分位估算值，我们算出估算值的中心 95% 区域对应的值（截取首尾 2.5% 的区域），并获得 95% 的置信区间。请根据以下步骤在下面的单元格中实现这一过程：\n",
+ "\n",
+ "- 初始化一些有用的变量，将数据点的数量存储在 `n_points` 中，并用 `sample_qs` 创建一个自助分位数值空列表。\n",
+ "- 为每个试验创建一个循环：\n",
+ " - 首先从数据中抽出放回地抽取数据，生成自助样本。（可以使用[`random.choice`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.choice.html)。）\n",
+ " - 然后，计算样本的第 `q` 分位数，并将其添加到 `sample_qs` 列表中。如果你使用的是 NumPy v0.15 或更高版本，可以使用 [`quantile`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.quantile.html) 函数直接获取 `q` 的分为数，在 v0.14 或更低版本中，你需要将 `q` 变成百分位数并使用 [`percentile`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.percentile.html)。\n",
+ "- 收集自助的分位数后，找到捕获分位数中心 `c` 比例的上下限，从而得出估算的置信区间。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def quantile_ci(data, q, c = .95, n_trials = 1000):\n",
+ " \"\"\"\n",
+ " Compute a confidence interval for a quantile of a dataset using a bootstrap\n",
+ " method.\n",
+ " \n",
+ " Input parameters:\n",
+ " data: data in form of 1-D array-like (e.g. numpy array or Pandas series)\n",
+ " q: quantile to be estimated, must be between 0 and 1\n",
+ " c: confidence interval width\n",
+ " n_trials: number of bootstrap samples to perform\n",
+ " \n",
+ " Output value:\n",
+ " ci: Tuple indicating lower and upper bounds of bootstrapped\n",
+ " confidence interval\n",
+ " \"\"\"\n",
+ " \n",
+ " # initialize storage of bootstrapped sample quantiles\n",
+ " n_points = # number of data points\n",
+ " sample_qs = # storage of sampled quantiles\n",
+ " \n",
+ " # For each trial...\n",
+ " for _ in range(n_trials):\n",
+ " # draw a random sample from the data with replacement...\n",
+ " sample = \n",
+ " \n",
+ " # compute the desired quantile...\n",
+ " sample_q = \n",
+ " \n",
+ " # and add the value to the list of sampled quantiles\n",
+ " \n",
+ " \n",
+ " # Compute the confidence interval bounds\n",
+ " lower_limit = \n",
+ " upper_limit = \n",
+ " \n",
+ " return (lower_limit, upper_limit)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = pd.read_csv('data/bootstrapping_data.csv')\n",
+ "data.head(10)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data visualization\n",
+ "plt.hist(data['time'], bins = np.arange(0, data['time'].max()+400, 400));"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "lims = quantile_ci(data['time'], 0.9)\n",
+ "print(lims)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 自助法注意事项\n",
+ "\n",
+ "与真实世界相比，通过自助法获得的置信区间更加乐观。这是因为我们没有关于真实世界的参数化模型，对于真实世界还有一些不了解的情况。思考下尝试了解最大值分布的极端情形：置信区间永远无法包含大于最大观察值的值，并且下限小于最大观察值是不合理的。但是，很有可能有一些未观察到的值大于我们观察到的值，尤其是像示例中所演示的偏斜数据。\n",
+ "\n",
+ "但是这并不能否定自助法的优势。自助流程很简单直白。由于不对数据分布做出假设，所以适用于任何情形。结果应该与标准检验的不相上下。但是需要投入计算精力，并且输出取决于输入数据。例如，对于上述示例中第 90 百分位上的 95% 置信区间，推断的区间只能捕获原始生成分布中约 83% 的第 90 百分位数值。但是使用更复杂的二项假设对观察到的数据编制索引，只能使结果提高一个百分点，达到 84%。这两种方法都取决于生成的具体数据：不同的 5000 个数据点将生成不同的区间，准确率也不同。\n",
+ "\n",
+ "百分位置信区间的二项式方法参考文档：[1](https://www-users.york.ac.uk/~mb55/intro/cicent.htm)、[2](https://stats.stackexchange.com/questions/99829/how-to-obtain-a-confidence-interval-for-a-percentile)\n",
+ "\n",
+ "## 置换检验\n",
+ "\n",
+ "置换检验是一种重新抽样检验，用于比较两组或多组之间的结果变量的值。在置换检验中，我们对组标签进行重新抽样。原理是，在零假设下，所有组的结果分布应该一样，无论是对照组还是实验组。所以，我们可以将所有数据值当做一个大的小组，并模拟零假设。将标签随机地分配给数据点（同时保持原始小组成员比例）可以得出零假设的一个模拟结果。\n",
+ "\n",
+ "剩下的步骤与标准假设检验中使用的抽样方法类似，但是我们尚未指定要从中抽样的参考分布，我们直接从收集的数据中抽样。将标签随机分配给所有数据并多次记录结果统计量后，我们将实际观察到的统计量与模拟统计量进行比较。我们看看有多少模拟统计值和实际观察到的统计值一样极端或更极端，并算出 p 值，然后得出结论。\n",
+ "\n",
+ "请在下面的单元格中实现置换检验，检验与对照组相比，实验组的次数第 90 百分位是否统计显著性地更小：\n",
+ "\n",
+ "- 初始化一个空列表，用于将样本分位数的差异存储为 `sample_diffs`。\n",
+ "- 为每个试验创建一个循环：\n",
+ " - 首先通过随机重排数据点标签，生成一个置换样本。（可以使用[`random.permutation`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.permutation.html)。）\n",
+ " - 然后，根据置换的标签计算分配给每组的数据点的第 `q` 分位数。将分位数差异附加到 `sample_diffs` 列表中。\n",
+ "- 收集置换样本的分位数差异后，计算实际数据观察到的差异。然后，看看有多少置换样本差异小于或大于观察到的差异，并计算 p 值，具体取决于期望的备择假设。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def quantile_permtest(x, y, q, alternative = 'less', n_trials = 10_000):\n",
+ " \"\"\"\n",
+ " Compute a confidence interval for a quantile of a dataset using a bootstrap\n",
+ " method.\n",
+ " \n",
+ " Input parameters:\n",
+ " x: 1-D array-like of data for independent / grouping feature as 0s and 1s\n",
+ " y: 1-D array-like of data for dependent / output feature\n",
+ " q: quantile to be estimated, must be between 0 and 1\n",
+ " alternative: type of test to perform, {'less', 'greater'}\n",
+ " n_trials: number of permutation trials to perform\n",
+ " \n",
+ " Output value:\n",
+ " p: estimated p-value of test\n",
+ " \"\"\"\n",
+ " \n",
+ " \n",
+ " # initialize storage of bootstrapped sample quantiles\n",
+ " sample_diffs = \n",
+ " \n",
+ " # For each trial...\n",
+ " for _ in range(n_trials):\n",
+ " # randomly permute the grouping labels\n",
+ " \n",
+ " \n",
+ " # compute the difference in quantiles\n",
+ " \n",
+ " \n",
+ " # and add the value to the list of sampled differences\n",
+ " \n",
+ " \n",
+ " # compute observed statistic\n",
+ " \n",
+ " \n",
+ " # compute a p-value\n",
+ " if alternative == 'less':\n",
+ " hits = \n",
+ " elif alternative == 'greater':\n",
+ " hits = \n",
+ " \n",
+ " return (hits / n_trials)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = pd.read_csv('data/permutation_data.csv')\n",
+ "data.head(10)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data visualization\n",
+ "bin_borders = np.arange(0, data['time'].max()+400, 400)\n",
+ "plt.hist(data[data['condition'] == 0]['time'], alpha = 0.5, bins = bin_borders)\n",
+ "plt.hist(data[data['condition'] == 1]['time'], alpha = 0.5, bins = bin_borders)\n",
+ "plt.legend(labels = ['control', 'experiment']);"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Just how different are the two distributions' 90th percentiles?\n",
+ "print(np.percentile(data[data['condition'] == 0]['time'], 90),\n",
+ " np.percentile(data[data['condition'] == 1]['time'], 90))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "quantile_permtest(data['time'], data['condition'], 0.9,\n",
+ " alternative = 'less')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "```python\n",
+ "\n",
+ "```"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_1.ipynb b/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_1.ipynb
new file mode 100644
index 00000000..f698c5a5
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_1.ipynb
@@ -0,0 +1,270 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Non-Parametric Tests Part I\n",
+ "\n",
+ "Up until now, you've been using standard hypothesis tests on means of normal distributions to design and analyze experiments. However, it's possible that you will encounter scenarios where you can't rely on only standard tests. This might be due to uncertainty about the true variability of a metric's distribution, a lack of data to assume normality, or wanting to do inference on a statistic that lacks a standard test. It's useful to know about some **non-parametric tests**, not just as a workaround for cases like this, but also as a second check on your experimental results. The main benefit of non-parametric tests is that they don't rely on many assumptions of the underlying population, and so can be used in a wider range of circumstances compared to standard tests. In this notebook, you'll cover two non-parametric approaches that use resampling of the data to make inferences about distributions and differences."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Bootstrapping\n",
+ "\n",
+ "Bootstrapping is used to estimate sampling distributions by using the actually collected data to generate new samples that could have been hypothetically collected. In a standard bootstrap, a bootstrapped sample means drawing points from the original data _with replacement_ until we get as many points as there were in the original data. Essentially, we're treating the original data as the population: without making assumptions about the original population distribution, using the original data as a model of the population is the best that we can do.\n",
+ "\n",
+ "Taking a lot of bootstrapped samples allows us to estimate the sampling distribution for various statistics on our original data. For example, let's say that we wanted to create a 95% confidence interval for the 90th percentile from a dataset of 5000 data points. (Perhaps we're looking at website load times and want to reduce the worst cases.) Bootstrapping makes this easy to estimate. First of all, we take a bootstrap sample (i.e., draw 5000 points with replacement from the original data), record the 90th percentile, and then repeat this a large number of times, let's say 100 000. From this bunch of bootstrapped 90th percentile estimates, we form our confidence interval by finding the values that capture the central 95% of the estimates (cutting off 2.5% on each tail). Implement this operation in the cells below, using the following steps:\n",
+ "\n",
+ "- Initialize some useful variables by storing the number of data points in `n_points` and setting up an empty list for the bootstrapped quantile values in `sample_qs`.\n",
+ "- Create a loop for each trial:\n",
+ " - First generate a bootstrap sample by sampling from our data with replacement. ([`random.choice`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.choice.html) will be useful here.)\n",
+ " - Then, compute the `q`th quantile of the sample and add it to the `sample_qs` list. If you're using NumPy v0.15 or later, you can use the [`quantile`](https://numpy.org/doc/stable/reference/generated/numpy.quantile.html) function to get the quantile directly with `q`; on v0.14 or earlier, you'll need to put `q` in terms of a percentile and use [`percentile`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.permutation.html) instead.\n",
+ "- After gathering the bootstrapped quantiles, find the limits that capture the central `c` proportion of quantiles to form the estimated confidence interval."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def quantile_ci(data, q, c = .95, n_trials = 1000):\n",
+ " \"\"\"\n",
+ " Compute a confidence interval for a quantile of a dataset using a bootstrap\n",
+ " method.\n",
+ " \n",
+ " Input parameters:\n",
+ " data: data in form of 1-D array-like (e.g. numpy array or Pandas series)\n",
+ " q: quantile to be estimated, must be between 0 and 1\n",
+ " c: confidence interval width\n",
+ " n_trials: number of bootstrap samples to perform\n",
+ " \n",
+ " Output value:\n",
+ " ci: Tuple indicating lower and upper bounds of bootstrapped\n",
+ " confidence interval\n",
+ " \"\"\"\n",
+ " \n",
+ " # initialize storage of bootstrapped sample quantiles\n",
+ " n_points = # number of data points\n",
+ " sample_qs = # storage of sampled quantiles\n",
+ " \n",
+ " # For each trial...\n",
+ " for _ in range(n_trials):\n",
+ " # draw a random sample from the data with replacement...\n",
+ " sample = \n",
+ " \n",
+ " # compute the desired quantile...\n",
+ " sample_q = \n",
+ " \n",
+ " # and add the value to the list of sampled quantiles\n",
+ " \n",
+ " \n",
+ " # Compute the confidence interval bounds\n",
+ " lower_limit = \n",
+ " upper_limit = \n",
+ " \n",
+ " return (lower_limit, upper_limit)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = pd.read_csv('data/bootstrapping_data.csv')\n",
+ "data.head(10)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data visualization\n",
+ "plt.hist(data['time'], bins = np.arange(0, data['time'].max()+400, 400));"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "lims = quantile_ci(data['time'], 0.9)\n",
+ "print(lims)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Bootstrapping Notes\n",
+ "\n",
+ "Confidence intervals coming from the bootstrap procedure will be optimistic compared to the true state of the world. This is because there will be things that we don't know about the real world that we can't account for, due to not having a parametric model of the world's state. Consider the extreme case of trying to understand the distribution of the maximum value: our confidence interval would never be able to include any value greater than the largest observed value and it makes no sense to have any lower bound below the maximum observation. Intuitively, however, there's a pretty clear possibility for there to be unobserved values that are larger than the one we've observed, especially for skewed data like shown in the example.\n",
+ "\n",
+ "This doesn't override the bootstrap method's advantages, however. The bootstrap procedure is fairly simple and straightforward. Since you don't make assumptions about the distribution of data, it can be applicable for any case you encounter. The results should also be fairly comparable to standard tests. But it does take computational effort, and its output does depend on the data put in. For reference, for the 95% CI on the 90th percentile example explored above, the inferred interval would only capture about 83% of 90th percentiles from the original generating distribution. But a more intricate procedure using a binomial assumption to index on the observed data only does about one percentage point better (84%). And both of these depend on the specific data generated: a different set of 5000 points will produce different intervals, with different accuracies.\n",
+ "\n",
+ "Binomial solution for percentile CIs reference: [1](https://www-users.york.ac.uk/~mb55/intro/cicent.htm), [2](https://stats.stackexchange.com/questions/99829/how-to-obtain-a-confidence-interval-for-a-percentile)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Permutation Tests\n",
+ "\n",
+ "The permutation test is a resampling-type test used to compare the values on an outcome variable between two or more groups. In the case of the permutation test, resampling is done on the group labels. The idea here is that, under the null hypothesis, the outcome distribution should be the same for all groups, whether control or experimental. Thus, we can emulate the null by taking all of the data values as a single large group. Applying labels randomly to the data points (while maintaining the original group membership ratios) gives us one simulated outcome from the null.\n",
+ "\n",
+ "The rest is similar to the sampling approach used in a standard hypothesis test, except that we haven't specified a reference distribution to sample from – we're sampling directly from the data we've collected. After applying the labels randomly to all the data and recording the outcome statistic many times, we compare our actual, observed statistic against the simulated statistics. A p-value is obtained by seeing how many simulated statistic values are as or more extreme than the one actually observed, and a conclusion is then drawn.\n",
+ "\n",
+ "Try implementing a permutation test in the cells below to test if the 90th percentile of times is statistically significantly smaller for the experimental group, as compared to the control group:\n",
+ "\n",
+ "- Initialize an empty list to store the difference in sample quantiles as `sample_diffs`.\n",
+ "- Create a loop for each trial:\n",
+ " - First generate a permutation sample by randomly shuffling the data point labels. ([`random.permutation`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.permutation.html) will be useful here.)\n",
+ " - Then, compute the `q`th quantile of the data points that have been assigned to each group based on the permuted labels. Append the difference in quantiles to the `sample_diffs` list.\n",
+ "- After gathering the quantile differences for permuted samples, compute the observed difference for the actual data. Then, compute a p-value from the number of permuted sample differences that are less than or greater than the observed difference, depending on the desired alternative hypothesis."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def quantile_permtest(x, y, q, alternative = 'less', n_trials = 10_000):\n",
+ " \"\"\"\n",
+ " Compute a confidence interval for a quantile of a dataset using a bootstrap\n",
+ " method.\n",
+ " \n",
+ " Input parameters:\n",
+ " x: 1-D array-like of data for independent / grouping feature as 0s and 1s\n",
+ " y: 1-D array-like of data for dependent / output feature\n",
+ " q: quantile to be estimated, must be between 0 and 1\n",
+ " alternative: type of test to perform, {'less', 'greater'}\n",
+ " n_trials: number of permutation trials to perform\n",
+ " \n",
+ " Output value:\n",
+ " p: estimated p-value of test\n",
+ " \"\"\"\n",
+ " \n",
+ " \n",
+ " # initialize storage of bootstrapped sample quantiles\n",
+ " sample_diffs = \n",
+ " \n",
+ " # For each trial...\n",
+ " for _ in range(n_trials):\n",
+ " # randomly permute the grouping labels\n",
+ " \n",
+ " \n",
+ " # compute the difference in quantiles\n",
+ " \n",
+ " \n",
+ " # and add the value to the list of sampled differences\n",
+ " \n",
+ " \n",
+ " # compute observed statistic\n",
+ " \n",
+ " \n",
+ " # compute a p-value\n",
+ " if alternative == 'less':\n",
+ " hits = \n",
+ " elif alternative == 'greater':\n",
+ " hits = \n",
+ " \n",
+ " return (hits / n_trials)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = pd.read_csv('data/permutation_data.csv')\n",
+ "data.head(10)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data visualization\n",
+ "bin_borders = np.arange(0, data['time'].max()+400, 400)\n",
+ "plt.hist(data[data['condition'] == 0]['time'], alpha = 0.5, bins = bin_borders)\n",
+ "plt.hist(data[data['condition'] == 1]['time'], alpha = 0.5, bins = bin_borders)\n",
+ "plt.legend(labels = ['control', 'experiment']);"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Just how different are the two distributions' 90th percentiles?\n",
+ "print(np.percentile(data[data['condition'] == 0]['time'], 90),\n",
+ " np.percentile(data[data['condition'] == 1]['time'], 90))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "quantile_permtest(data['time'], data['condition'], 0.9,\n",
+ " alternative = 'less')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_2-zh.ipynb b/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_2-zh.ipynb
new file mode 100644
index 00000000..5c279ad8
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_2-zh.ipynb
@@ -0,0 +1,259 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# 非参数检验（第二部分）\n",
+ "\n",
+ "到目前为止，你在设计和分析实验时，对正态分布的均值采用的都是标准假设检验。但是，有时候不能只依赖于标准检验。原因可能是不确定指标分布的真实变化性、缺少满足正态要求的数据，或者想对缺少标准检验的统计量进行推理。这时候就有必要了解一些非参数检验，不仅仅是为了处理上述情况，而且可以作为实验结果的辅助检查工具。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import scipy.stats as stats\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 秩和检验（曼-惠特尼检验）\n",
+ "\n",
+ "秩和检验与之前的两个方法差别很大。不需要重新抽样，仅对存在的数据进行检验。秩和检验又称为曼-惠特尼 U 检验，它并不检验特定的统计量，例如均值或中值。而是检验分布：假设我们随机地从每个小组的总体中抽取一个值。零假设表示更大的值来自第一个小组的概率与来自第二个小组的概率一样；备择假设表示这两个概率不相同，可以表示为单尾或双尾检验。\n",
+ "\n",
+ "为了检验这种假设，我们应该查看收集的数据，并看看在多少情形下一个小组的值大于另一个小组的值。即对于第一个小组中的每个数据点，我们都数数第二个小组里有多少值小于它。（如果两个值相等，则表示打成平手，使计数器加 0.5。）第一组获胜的数量用值 $U$ 表示。\n",
+ "\n",
+ "如果样本量足够大的话，则 $U$ 大致呈正态分布。如果第一组有 $n_1$ 个数据点，第二组有 $n_2$ 个数据点，那么总共有 $n_1 n_2$ 次比对和同等数量的获胜点。在零假设下，获胜次数在两组之间的分布应该是均匀的，所以预期的获胜次数为 $\\mu_U = \\frac{n_1 n_2}{2}$。获胜次数的可变性可以用以下方程表示（假设没有平局或只有少数几个平局）：\n",
+ "\n",
+ "$$ \n",
+ "\\sigma_U = \\sqrt{\\frac{n_1n_2(n_1+n_2+1)}{12}}\n",
+ "$$\n",
+ "\n",
+ "$\\mu_U$ 和 $\\sigma_U$ 值然后可以用来计算标准正态 z 分数，进而生成 p 值。请在以下单元格中实现秩和检验方法。\n",
+ "\n",
+ "- 提示：计算 z 分数后，可以使用 scipy stats 的[`norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) 类获取 p 值。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def ranked_sum(x, y, alternative = 'two-sided'):\n",
+ " \"\"\"\n",
+ " Return a p-value for a ranked-sum test, assuming no ties.\n",
+ " \n",
+ " Input parameters:\n",
+ " x: 1-D array-like of data for first group\n",
+ " y: 1-D array-like of data for second group\n",
+ " alternative: type of test to perform, {'two-sided', less', 'greater'}\n",
+ " \n",
+ " Output value:\n",
+ " p: estimated p-value of test\n",
+ " \"\"\"\n",
+ " \n",
+ " # compute U\n",
+ " u = \n",
+ " \n",
+ " # compute a z-score\n",
+ " n_1 = \n",
+ " n_2 = \n",
+ " mean_u = # expected value for U statistic\n",
+ " sd_u = # expected standard deviation for U statistic\n",
+ " z = # U value z-score\n",
+ " \n",
+ " # compute a p-value\n",
+ " if alternative == 'two-sided':\n",
+ " p = \n",
+ " if alternative == 'less':\n",
+ " p = \n",
+ " elif alternative == 'greater':\n",
+ " p = \n",
+ " \n",
+ " return p"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = pd.read_csv('data/permutation_data.csv')\n",
+ "data.head(10)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data visualization\n",
+ "bin_borders = np.arange(0, data['time'].max()+400, 400)\n",
+ "plt.hist(data[data['condition'] == 0]['time'], alpha = 0.5, bins = bin_borders)\n",
+ "plt.hist(data[data['condition'] == 1]['time'], alpha = 0.5, bins = bin_borders)\n",
+ "plt.legend(labels = ['control', 'experiment']);"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ranked_sum(data[data['condition'] == 0]['time'],\n",
+ " data[data['condition'] == 1]['time'],\n",
+ " alternative = 'greater')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 秩和检验注意事项\n",
+ "\n",
+ "如果样本量更小，可以进行置换检验。在检查某个值的所有可能小组标签分配的获胜分布后，可以计算 p 值并判断实际观察到的 $U$ 有多不寻常。\n",
+ "\n",
+ "此外，scipy stats 软件包中已经有一个函数 [`mannwhitneyu`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html) 能够执行曼惠特尼 U 检验。此函数考虑的因素比上述实现要多，包括对平局的标准差进行校正，以及连续校正（因为我们通过连续分布逼近离散值分布）。此外，他们采取的方法计算更高效，基于的是秩和（因而得名秩和检验），而不是上面解释的比对方法。\n",
+ "\n",
+ "参考资料：[维基百科文章](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "stats.mannwhitneyu(data[data['condition'] == 0]['time'],\n",
+ " data[data['condition'] == 1]['time'],\n",
+ " alternative = 'greater')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 符号检验\n",
+ "\n",
+ "符号检验也仅使用收集的数据计算检验结果。只需两组之间有可比较的配对值，并且一组的值一般比另一组的要大。\n",
+ "\n",
+ "在符号检验中，我们不关心两组之间的差异有多大，只关心哪组的值更大。所以 0.21 与 0.22 比较和 0.21 与 0.31 比较都表示第二组的值更大。这就导致符号检验是一个不太可靠的检验，虽然可以广泛应用。如果可以从中抽样的观察值太少，并且无法对底层的分布特征做出很好的假设，那么这种检验最有用。例如，可以使用符号检验进一步检查日常汇总的点击率。\n",
+ "\n",
+ "特定组的获胜次数可以用二项分布表示。在零假设下，任何一组有更大值的概率是一样的（如果是平局，则不比较）：二项分布的成功参数是 $p = 0.5$。请在下面的函数中实现符号检验。\n",
+ "\n",
+ "- 提示：计算比对和获胜次数后，可以使用 scipy stats 的 [`binom`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binom.html) 类获取 p 值。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def sign_test(x, y, alternative = 'two-sided'):\n",
+ " \"\"\"\n",
+ " Return a p-value for a ranked-sum test, assuming no ties.\n",
+ " \n",
+ " Input parameters:\n",
+ " x: 1-D array-like of data for first group\n",
+ " y: 1-D array-like of data for second group\n",
+ " alternative: type of test to perform, {'two-sided', less', 'greater'}\n",
+ " \n",
+ " Output value:\n",
+ " p: estimated p-value of test\n",
+ " \"\"\"\n",
+ " \n",
+ " # compute parameters\n",
+ " n = # number of matchups\n",
+ " k = # number of victories for first group\n",
+ "\n",
+ " # compute a p-value\n",
+ " if alternative == 'two-sided':\n",
+ " p = \n",
+ " if alternative == 'less':\n",
+ " p = \n",
+ " elif alternative == 'greater':\n",
+ " p = \n",
+ " \n",
+ " return p"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = pd.read_csv('data/signtest_data.csv')\n",
+ "data.head()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data visualization\n",
+ "plt.plot(data['day'], data['control'])\n",
+ "plt.plot(data['day'], data['exp'])\n",
+ "plt.legend()\n",
+ "\n",
+ "plt.xlabel('Day of Experiment')\n",
+ "plt.ylabel('Success rate');"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sign_test(data['control'], data['exp'], 'less')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "```python\n",
+ "\n",
+ "```"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_2.ipynb b/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_2.ipynb
new file mode 100644
index 00000000..dcccbe0a
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Non-Parametric_Tests_Part_2.ipynb
@@ -0,0 +1,258 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Non-Parametric Tests Part II\n",
+ "\n",
+ "Up until now, you've been using standard hypothesis tests on means of normal distributions to design and analyze experiments. However, it's possible that you might encounter scenarios where you can't rely on only standard tests. This might be due to uncertainty about the true variability of a metric's distribution, a lack of data to assume normality, or wanting to do inference on a statistic that lacks a standard test. It's useful to know about some **non-parametric tests** not just as a workaround for cases like this, but also as a second check on your experimental results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import scipy.stats as stats\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Rank-Sum Test (Mann-Whitney)\n",
+ "\n",
+ "The rank-sum test is fairly different from the two previous approaches. There's no resamplng involved; the test is performed only on the data present. The rank-sum test, also known as the Mann-Whitney U test, is not a test of any particular statistic, like the mean or median. Instead, it's a test of distributions: let's say we draw one value at random from the populations behind each group. The null hypothesis says that there's an equal chance that the larger value is from the first group as the second group; the alternative hypothesis says that there's an unequal chance, which can be specified as one- or two-tailed.\n",
+ "\n",
+ "In order to test this hypothesis, we should look at the data we've collected and see in how many cases values from one group win compared to values in the second. That is, for each data point in the first group, we count how many values in the second group that are smaller than it. (If both values are equal, we count that as a tie, worth +0.5 to the tally.) This number of wins for the first group gives us a value $U$.\n",
+ "\n",
+ "It turns out that $U$ is approximately normally-distributed, given a large enough sample size. If we have $n_1$ data points in the first group and $n_2$ points in the second, then we have a total of $n_1 n_2$ matchups and an equivalent number of victory points to hand out. Under the null hypothesis, we should expect the number of wins to be evenly distributed between groups, and so the expected wins are $\\mu_U = \\frac{n_1 n_2}{2}$. The variability in the number of wins can be found to be the following equation (assuming no or few ties):\n",
+ "\n",
+ "$$ \n",
+ "\\sigma_U = \\sqrt{\\frac{n_1n_2(n_1+n_2+1)}{12}}\n",
+ "$$\n",
+ "\n",
+ "These $\\mu_U$ and $\\sigma_U$ values can then be used to compute a standard normal z-score, which generates a p-value. Implement this method of performing the rank-sum test in the cells below!\n",
+ "\n",
+ "- HINT: scipy stats' [`norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) class can be used to obtain p-values after computing a z-score."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def ranked_sum(x, y, alternative = 'two-sided'):\n",
+ " \"\"\"\n",
+ " Return a p-value for a ranked-sum test, assuming no ties.\n",
+ " \n",
+ " Input parameters:\n",
+ " x: 1-D array-like of data for first group\n",
+ " y: 1-D array-like of data for second group\n",
+ " alternative: type of test to perform, {'two-sided', less', 'greater'}\n",
+ " \n",
+ " Output value:\n",
+ " p: estimated p-value of test\n",
+ " \"\"\"\n",
+ " \n",
+ " # compute U\n",
+ " u = \n",
+ " \n",
+ " # compute a z-score\n",
+ " n_1 = \n",
+ " n_2 = \n",
+ " mean_u = # expected value for U statistic\n",
+ " sd_u = # expected standard deviation for U statistic\n",
+ " z = # U value z-score\n",
+ " \n",
+ " # compute a p-value\n",
+ " if alternative == 'two-sided':\n",
+ " p = \n",
+ " if alternative == 'less':\n",
+ " p = \n",
+ " elif alternative == 'greater':\n",
+ " p = \n",
+ " \n",
+ " return p"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = pd.read_csv('data/permutation_data.csv')\n",
+ "data.head(10)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data visualization\n",
+ "bin_borders = np.arange(0, data['time'].max()+400, 400)\n",
+ "plt.hist(data[data['condition'] == 0]['time'], alpha = 0.5, bins = bin_borders)\n",
+ "plt.hist(data[data['condition'] == 1]['time'], alpha = 0.5, bins = bin_borders)\n",
+ "plt.legend(labels = ['control', 'experiment']);"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ranked_sum(data[data['condition'] == 0]['time'],\n",
+ " data[data['condition'] == 1]['time'],\n",
+ " alternative = 'greater')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Rank-Sum Test Notes\n",
+ "\n",
+ "For smaller sample sizes, something like the permutation test can be performed. After exhaustively checking the distribution of victories for every possible assignment of group labels to value, a p-value can be computed for how unusual the actually-observed $U$ was.\n",
+ "\n",
+ "Also, there already exists a function in the scipy stats package [`mannwhitneyu`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html) that performs the Mann Whitney U test. This function considers more factors than the implementation above, including a correction on the standard deviation for ties and a continuity correction (since we're approximating a discretely-valued distribution with a continuous one). In addition, the approach they take is computationally more efficient, based on the sum of value ranks (hence the rank-sum test name) rather than the matchups explanation provided above.\n",
+ "\n",
+ "Reference: [Wikipedia](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "stats.mannwhitneyu(data[data['condition'] == 0]['time'],\n",
+ " data[data['condition'] == 1]['time'],\n",
+ " alternative = 'greater')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Sign Test\n",
+ "\n",
+ "The sign test also only uses the collected data to compute a test result. It only requires that there be paired values between two groups to compare, and tests whether one group's values tend to be higher than the other's.\n",
+ "\n",
+ "In the sign test, we don't care how large differences are between groups, only which group takes a larger value. So comparisons of 0.21 vs. 0.22 and 0.21 vs. 0.31 are both counted equally as a point in favor of the second group. This makes the sign test a fairly weak test, though also a test that can be applied fairly broadly. It's most useful when we have very few observations to draw from and can't make a good assumption of underlying distribution characteristics. For example, you might use a sign test as an additional check on click rates that have been aggregated on a daily basis.\n",
+ "\n",
+ "The count of victories for a particular group can be modeled with the binomial distribution. Under the null hypothesis, it is equally likely that either group has a larger value (in the case of a tie, we ignore the comparison): the binomial distribution's success parameter is $p = 0.5$. Implement the sign test in the function below!\n",
+ "\n",
+ "- HINT: scipy stats' [`binom`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binom.html) class can be used to obtain p-values after computing the number of matchups and victories."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def sign_test(x, y, alternative = 'two-sided'):\n",
+ " \"\"\"\n",
+ " Return a p-value for a ranked-sum test, assuming no ties.\n",
+ " \n",
+ " Input parameters:\n",
+ " x: 1-D array-like of data for first group\n",
+ " y: 1-D array-like of data for second group\n",
+ " alternative: type of test to perform, {'two-sided', less', 'greater'}\n",
+ " \n",
+ " Output value:\n",
+ " p: estimated p-value of test\n",
+ " \"\"\"\n",
+ " \n",
+ " # compute parameters\n",
+ " n = # number of matchups\n",
+ " k = # number of victories for first group\n",
+ "\n",
+ " # compute a p-value\n",
+ " if alternative == 'two-sided':\n",
+ " p = \n",
+ " if alternative == 'less':\n",
+ " p = \n",
+ " elif alternative == 'greater':\n",
+ " p = \n",
+ " \n",
+ " return p"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = pd.read_csv('data/signtest_data.csv')\n",
+ "data.head()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data visualization\n",
+ "plt.plot(data['day'], data['control'])\n",
+ "plt.plot(data['day'], data['exp'])\n",
+ "plt.legend()\n",
+ "\n",
+ "plt.xlabel('Day of Experiment')\n",
+ "plt.ylabel('Success rate');"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sign_test(data['control'], data['exp'], 'less')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Statistical_Significance-zh.ipynb b/lessons/StatisticalConsiderations/L2_Statistical_Significance-zh.ipynb
new file mode 100644
index 00000000..1483a331
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Statistical_Significance-zh.ipynb
@@ -0,0 +1,118 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# 练习：统计显著性\n",
+ "\n",
+ "假设我们为一项网络实验收集了数据。在实验中，我们检验了产品信息页的布局变化，看看这种变化是否会影响到点击按钮并转到下载页的用户所占的比例。该实验以 Cookie 为分组依据，我们记录了用户的两项数据：他们访问的是哪个网页版本，以及在数据记录阶段是否访问了下载页。（在此示例中，我们没有跟踪任何其他因素，例如页面查看次数，或从访问网页到下载产品的时间间隔，这些因素可能值得进一步研究。）\n",
+ "\n",
+ "在此 notebook 中，你的目标是对这两个记录指标执行统计学检验，看看两组之间是否存在统计学差异。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import packages\n",
+ "\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import scipy.stats as stats\n",
+ "from statsmodels.stats import proportion as proptests\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import data\n",
+ "\n",
+ "data = pd.read_csv('data/statistical_significance_data.csv')\n",
+ "data.head(10)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "在数据集的“condition”列中，0 表示对照组，1 表示实验组。对于“click”列，0 表示没有点击，1 表示点击了。\n",
+ "\n",
+ "## 检查不变指标\n",
+ "\n",
+ "首先，我们应该检查分配给每组的用户人数是否差不多。务必要检查作为前提条件的不变指标，这样可以保证根据评估指标做出的推断有扎实的依据。如果我们发现两组在不变指标方面不平衡，则需要仔细观察访问者的划分方式，看看能否找出任何偏差原因。如果发现不变指标存在统计学显著性差异，则可能需要修改随机分配流程，并重新收集数据。\n",
+ "\n",
+ "在这种情形下，我们需要对分配给某个条件的访问者比例进行双边假设检验。选择对照组还是实验组并不重要：你会获得相同的结果。你可以选择使用任何一种方法，我们将在下面主要介绍两种方法。\n",
+ "\n",
+ "如果你想采用模拟方法，你可以模拟分配到每组的访问者人数，假设按照 50/50 的比例划分。重复这一过程很多次（在此示例中，重复 200 000 次应该会达到很好的速度可变性平衡），然后看看有多少模拟情形完全偏离了 50/50 的分配结果。注意，由于我们完成的是双边检验，因此极端情况还包括 50/50 相反情形的值。（例如，由于 0.48 及更低的模拟结果被视为比 0.48 的实际观测值更极端，所以 0.52 及更高的模拟结果也一样。）我们可以根据被标记模拟结果的比例获得 p 值，并用p 值评估观察到的比例。我们希望看到更大的 p 值，表明拒绝零假设的证据不足。\n",
+ "\n",
+ "如果你想采取分析方法，可以使用精确二项分布计算检验的 p 值。但是，更常见的方法是使用正态分布逼近法。由于样本量很大，并且存在中心极限定理，因此这种逼近法是可行的。要获得精确的 p 值，还应该进行连续校正， \n",
+ "在计算曲线下面积之前，使总数加上或减去 0.5。（例如，如果对照组的分配比例是 415 / 850，那么正态逼近计算的面积向左为 $(415 + 0.5) / 850 = 0.489$，向右为 $(435 - 0.5) / 850 = 0.511$。）\n",
+ "\n",
+ "完成下面的 workspace 后，你可以查看下个页面的解答内容。你还可以尝试多种方法，看看它们是否会取得相似的结果。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# your work here: feel free to create additional code cells as needed!\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 检查评估指标\n",
+ "\n",
+ "检查完不变指标后，我们可以继续对评估指标进行假设检验，即点击率。在此示例中，我们希望看到实验组的点击率比对照组的高很多，这是一个单尾检验。\n",
+ "\n",
+ "对于该指标来说，模拟方法与不变指标的区别不大。你需要将总体点击率作为共同比例，并从中为每个小组抽取模拟值。此外，需要模拟更多次，因为该检验的方差更高。\n",
+ "\n",
+ "有几种分析方法可以采用，但是你很可能会再次利用正态逼近法。除了合并点击率之外，还需要计算合并标准差，以便计算 z 分数。虽然也可以采用连续校正，但比模拟通常暗示的 p 值更保守。在不连续校正的情况下计算 z 分数和 p 值应该更接近模拟结果，但是更加确信两组之间存在统计学差异。\n",
+ "\n",
+ "与上个问题一样，你可以在 workspace 后面找到练习答案。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# your work here: feel free to create additional code cells as needed!"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/L2_Statistical_Significance.ipynb b/lessons/StatisticalConsiderations/L2_Statistical_Significance.ipynb
new file mode 100644
index 00000000..dbbcf92f
--- /dev/null
+++ b/lessons/StatisticalConsiderations/L2_Statistical_Significance.ipynb
@@ -0,0 +1,138 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Practice: Statistical Significance\n",
+ "\n",
+ "Let's say that we've collected data for a web-based experiment. In the experiment, we're testing the change in layout of a product information page to see if this affects the proportion of people who click on a button to go to the download page. This experiment has been designed to have a cookie-based diversion, and we record two things from each user: which page version they received, and whether or not they accessed the download page during the data recording period. (We aren't keeping track of any other factors in this example, such as number of pageviews, or time between accessing the page and making the download, that might be of further interest.)\n",
+ "\n",
+ "Your objective in this notebook is to perform a statistical test on both recorded metrics to see if there is a statistical difference between the two groups.\n",
+ "\n",
+ "*Note: if you need a refresher on the Statistical Significance topic, please review Udacity free course on [Intro to Statistics](https://www.udacity.com/course/intro-to-statistics--st101).*"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import packages\n",
+ "\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import scipy.stats as stats\n",
+ "from statsmodels.stats import proportion as proptests\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import data\n",
+ "\n",
+ "data = pd.read_csv('data/statistical_significance_data.csv')\n",
+ "data.head(10)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In the dataset, the 'condition' column takes a 0 for the control group, and 1 for the experimental group. The 'click' column takes a values of 0 for no click, and 1 for a click.\n",
+ "\n",
+ "## Checking the Invariant Metric\n",
+ "\n",
+ "First of all, we should check that the number of visitors assigned to each group is similar. It's important to check the invariant metrics as a prerequisite so that our inferences on the evaluation metrics are founded on solid ground. If we find that the two groups are imbalanced on the invariant metric, then this will require us to look carefully at how the visitors were split so that any sources of bias are accounted for. It's possible that a statistically significant difference in an invariant metric will require us to revise random assignment procedures and re-do data collection.\n",
+ "\n",
+ "In this case, we want to do a two-sided hypothesis test on the proportion of visitors assigned to one of our conditions. Choosing the control or the experimental condition doesn't matter: you'll get the same result either way. Feel free to use whatever method you'd like: we'll highlight two main avenues below.\n",
+ "\n",
+ "If you want to take a simulation-based approach, you can simulate the number of visitors that would be assigned to each group for the number of total observations, assuming that we have an expected 50/50 split. Do this many times (200 000 repetitions should provide a good speed-variability balance in this case) and then see in how many simulated cases we get as extreme or more extreme a deviation from 50/50 that we actually observed. Don't forget that, since we have a two-sided test, an extreme case also includes values on the opposite side of 50/50. (e.g. Since simulated outcomes of .48 and lower are considered as being more extreme than an actual observation of 0.48, so too will simulated outcomes of .52 and higher.) The proportion of flagged simulation outcomes gives us a p-value on which to assess our observed proportion. We hope to see a larger p-value, insufficient evidence to reject the null hypothesis.\n",
+ "\n",
+ "If you want to take an analytic approach, you could use the exact binomial distribution to compute a p-value for the test. The more usual approach, however, is to use the normal distribution approximation. Recall that this is possible thanks to our large sample size and the central limit theorem. To get a precise p-value, you should also perform a \n",
+ "continuity correction, either adding or subtracting 0.5 to the total count before computing the area underneath the curve. (e.g. If we had 415 / 850 assigned to the control group, then the normal approximation would take the area to the left of $(415 + 0.5) / 850 = 0.489$ and to the right of $(435 - 0.5) / 850 = 0.511$.)\n",
+ "\n",
+ "You can check your results by completing the following the workspace and the solution on the following page. You could also try using multiple approaches and seeing if they come up with similar outcomes!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Analytic approach: your work here: feel free to create additional code cells as needed!\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Simulation approach: your work here: feel free to create additional code cells as needed!\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Checking the Evaluation Metric\n",
+ "\n",
+ "After performing our checks on the invariant metric, we can move on to performing a hypothesis test on the evaluation metric: the click-through rate. In this case, we want to see that the experimental group has a significantly larger click-through rate than the control group, a one-tailed test.\n",
+ "\n",
+ "The simulation approach for this metric isn't too different from the approach for the invariant metric. You'll need the overall click-through rate as the common proportion to draw simulated values from for each group. You may also want to perform more simulations since there's higher variance for this test.\n",
+ "\n",
+ "There are a few analytic approaches possible here, but you'll probably make use of the normal approximation again in these cases. In addition to the pooled click-through rate, you'll need a pooled standard deviation in order to compute a z-score. While there is a continuity correction possible in this case as well, it's much more conservative than the p-value that a simulation will usually imply. Computing the z-score and resulting p-value without a continuity correction should be closer to the simulation's outcomes, though slightly more optimistic about there being a statistical difference between groups.\n",
+ "\n",
+ "As with the previous question, you'll find a quiz and solution following the workspace for you to check your results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Analytic approach: your work here: feel free to create additional code cells as needed!\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Simulation approach: your work here: feel free to create additional code cells as needed!\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/data/bootstrapping_data.csv b/lessons/StatisticalConsiderations/data/bootstrapping_data.csv
new file mode 100644
index 00000000..d85b7883
--- /dev/null
+++ b/lessons/StatisticalConsiderations/data/bootstrapping_data.csv
@@ -0,0 +1,5001 @@
+time
+8152
+2082
+3049
+3317
+813
+1442
+3815
+2113
+738
+2499
+4240
+1060
+2085
+1365
+3063
+1929
+875
+1862
+1087
+2679
+485
+662
+1674
+1631
+929
+744
+1254
+1818
+883
+2429
+1777
+1022
+799
+7623
+3155
+2212
+647
+1293
+1955
+2323
+4125
+4073
+350
+5000
+2685
+6550
+2676
+1078
+1818
+4360
+2978
+1670
+2893
+2516
+1352
+1781
+2013
+484
+5060
+1138
+1965
+6714
+5553
+3660
+1618
+2405
+2731
+573
+1419
+197
+939
+4614
+1493
+1426
+2043
+1390
+891
+1543
+2725
+1589
+1075
+2179
+3777
+4123
+2069
+2600
+1494
+9763
+6538
+5759
+9116
+1866
+1400
+2035
+577
+1221
+3403
+2333
+1209
+1191
+2931
+4722
+5686
+915
+3451
+10667
+5381
+2633
+1702
+794
+2827
+955
+1196
+1782
+1755
+3071
+1545
+1834
+1102
+1303
+659
+2147
+2235
+4039
+1643
+2303
+1220
+1380
+689
+5069
+1768
+1343
+6695
+1310
+1022
+1582
+2990
+2856
+2205
+4172
+315
+431
+2574
+945
+546
+640
+390
+5813
+854
+1601
+2441
+1274
+1871
+2264
+1497
+746
+713
+1588
+7328
+2174
+6280
+297
+3251
+3013
+371
+3847
+928
+1168
+709
+1897
+6613
+822
+2351
+1215
+1503
+2233
+2532
+4605
+2029
+1650
+5267
+4886
+939
+1628
+2606
+1858
+132
+4129
+4715
+988
+2278
+2587
+1015
+1659
+1861
+1492
+3269
+2698
+3798
+6642
+3767
+3983
+4076
+1848
+1643
+2382
+1492
+462
+4528
+1361
+235
+6027
+761
+5593
+2907
+2019
+2715
+865
+1528
+1797
+940
+960
+2689
+1359
+2109
+4174
+1106
+333
+1007
+1480
+2257
+4174
+507
+5044
+1516
+1915
+6933
+1104
+850
+1537
+1328
+5483
+437
+4962
+2727
+3398
+870
+1443
+1041
+603
+334
+3882
+774
+2284
+1352
+1203
+1006
+1446
+1855
+1584
+1331
+550
+3696
+1442
+1802
+1114
+834
+2387
+932
+6424
+1757
+1008
+5557
+2483
+1190
+3413
+683
+4258
+210
+2420
+4099
+3966
+5379
+937
+6343
+2823
+1492
+5622
+544
+675
+2353
+3719
+1138
+1313
+528
+1243
+6823
+381
+2150
+2708
+1260
+4558
+800
+1393
+2687
+2681
+1177
+2881
+3194
+1802
+783
+12654
+2632
+534
+1696
+6403
+2731
+647
+1773
+3988
+1277
+1630
+1215
+778
+2433
+6825
+6511
+1854
+3496
+2607
+1060
+1793
+5300
+5040
+1290
+2161
+3212
+1685
+2598
+5795
+5167
+5195
+835
+5475
+2247
+2170
+963
+3542
+7658
+1919
+3696
+2729
+5454
+1096
+1477
+1089
+1698
+1335
+4907
+5891
+1050
+2337
+2948
+1195
+1039
+2305
+2702
+769
+1930
+4299
+3170
+1054
+839
+6486
+2812
+2186
+2867
+4523
+742
+1687
+900
+825
+546
+1153
+7086
+741
+1593
+1315
+2045
+969
+2138
+3140
+5300
+1234
+3976
+9008
+1613
+2998
+816
+2182
+3112
+907
+799
+1961
+7061
+1315
+1271
+547
+5283
+1610
+2308
+1657
+2953
+1618
+1212
+6677
+4914
+7374
+2096
+616
+1141
+4482
+3469
+824
+1501
+3034
+3893
+5571
+1139
+2905
+825
+5220
+4172
+4998
+5479
+2594
+1900
+4756
+1888
+1937
+925
+2925
+693
+4258
+3282
+2687
+7795
+253
+2359
+1912
+2649
+2689
+3626
+1015
+2277
+3576
+1537
+3258
+1170
+556
+209
+908
+2803
+2001
+1694
+1052
+1305
+344
+4725
+2539
+2358
+1951
+1711
+2430
+5135
+2998
+5453
+4617
+1692
+3336
+2647
+1920
+2802
+1124
+7374
+1606
+202
+702
+3992
+1549
+1653
+2288
+3858
+3552
+6059
+2222
+4112
+7736
+2962
+1152
+1648
+6493
+2563
+1151
+2873
+1965
+4443
+810
+4901
+3603
+105
+995
+1108
+2228
+1692
+1544
+6609
+1915
+3504
+4564
+2234
+1226
+7474
+1378
+8749
+4298
+1120
+10763
+461
+1639
+5292
+1028
+2820
+1417
+884
+2328
+5569
+579
+1637
+3317
+2048
+387
+1738
+1146
+2371
+4935
+5754
+1727
+1195
+3782
+1137
+1520
+5256
+3904
+3429
+2965
+1713
+2370
+2324
+7599
+1616
+1118
+3169
+1128
+7189
+4152
+3156
+984
+1763
+135
+3659
+6161
+6035
+3557
+1951
+1269
+393
+3043
+8552
+12033
+2353
+1281
+2870
+1897
+976
+5463
+4148
+1759
+1078
+1742
+1955
+5377
+1260
+835
+1601
+1333
+1358
+5647
+2039
+2578
+878
+1629
+3663
+1354
+4606
+5774
+6911
+733
+343
+1004
+3097
+1722
+4505
+4176
+8546
+4765
+335
+1043
+2756
+1517
+750
+929
+991
+4565
+200
+3701
+5677
+1234
+5492
+1589
+6996
+2593
+5719
+2290
+3248
+3672
+4961
+794
+417
+954
+2218
+5051
+2686
+774
+14003
+841
+4424
+739
+233
+1250
+904
+2042
+1629
+11436
+2778
+2625
+4375
+2267
+4106
+2105
+1503
+1294
+2709
+2117
+3223
+960
+2344
+617
+7843
+416
+1756
+5150
+1224
+3134
+2565
+1164
+2571
+2385
+495
+200
+764
+513
+978
+1044
+705
+639
+2033
+1784
+8570
+3429
+205
+2398
+4539
+1999
+286
+6367
+6494
+341
+3637
+1981
+573
+1297
+2811
+4607
+721
+1088
+2517
+1395
+4429
+2360
+1122
+1058
+639
+2848
+3437
+1567
+3870
+1396
+10254
+931
+1672
+893
+1338
+1068
+2420
+4072
+2035
+1281
+3315
+1203
+1834
+5977
+2259
+1818
+1656
+1935
+5766
+6319
+3765
+2099
+878
+9464
+4054
+2541
+4445
+4862
+1404
+2667
+3786
+2052
+867
+610
+1370
+729
+3165
+310
+3459
+1734
+4048
+1888
+4467
+3756
+2052
+365
+640
+1126
+2570
+11281
+8066
+1594
+6019
+1549
+3466
+2982
+4276
+2474
+1393
+3939
+4804
+729
+471
+3081
+4826
+2647
+2112
+1479
+876
+4636
+511
+4919
+5807
+2510
+2225
+1502
+1200
+3018
+2103
+2326
+1782
+2247
+6623
+1166
+4021
+12142
+4340
+1984
+4438
+8680
+2302
+2608
+962
+2044
+843
+5763
+1326
+1783
+2428
+2271
+3599
+4320
+408
+1275
+1254
+1078
+2770
+1208
+2202
+1640
+3512
+3870
+4087
+770
+705
+3580
+464
+4312
+841
+5861
+3337
+2071
+6056
+8479
+2817
+4149
+247
+1707
+1184
+3738
+472
+790
+1289
+4673
+812
+6523
+2304
+408
+3826
+1576
+12001
+2430
+8083
+5087
+1425
+2381
+308
+2985
+2279
+685
+6899
+5596
+1323
+1591
+567
+2361
+4097
+3235
+1584
+1635
+5498
+2300
+845
+2737
+688
+842
+2110
+805
+1244
+3575
+7687
+6510
+1264
+6576
+8550
+3576
+1713
+6509
+312
+2196
+3282
+2762
+385
+2066
+4707
+6704
+208
+2820
+10755
+421
+9710
+1436
+2818
+2242
+1693
+1962
+1662
+9360
+8582
+562
+1612
+566
+5680
+1078
+5284
+1582
+1127
+2269
+1030
+3975
+1307
+499
+1575
+1765
+1812
+2630
+2108
+1551
+6169
+364
+1633
+533
+5053
+2018
+3575
+721
+1870
+1068
+1457
+944
+778
+8468
+4198
+2068
+950
+1788
+1603
+6153
+5064
+1703
+1510
+353
+5061
+1525
+6562
+2338
+1707
+1762
+843
+4749
+1662
+1615
+8118
+1234
+3507
+894
+4712
+275
+4835
+916
+995
+200
+2464
+2152
+1162
+969
+1018
+6861
+5752
+936
+773
+3772
+702
+1276
+183
+551
+2193
+5102
+1312
+465
+3258
+1829
+432
+3048
+3727
+5877
+3123
+2939
+1397
+1063
+2114
+1395
+1984
+7465
+854
+4118
+1774
+5102
+1507
+3603
+3639
+1870
+4902
+1228
+330
+4412
+444
+2044
+7436
+480
+4827
+781
+2738
+2514
+2003
+293
+660
+3062
+3427
+3051
+5387
+676
+748
+1661
+1671
+3643
+1552
+1817
+2518
+1816
+4346
+1716
+6773
+2889
+1839
+630
+920
+2181
+1961
+1089
+1346
+3207
+613
+6321
+6835
+2821
+6542
+5349
+5289
+2350
+1014
+1643
+2179
+4807
+827
+2223
+1216
+7708
+3257
+3029
+389
+2077
+1190
+2080
+2103
+1621
+2627
+1450
+2716
+1233
+2528
+7580
+6211
+1812
+883
+4131
+2456
+2872
+1629
+139
+4192
+2436
+4694
+1059
+9682
+8466
+1771
+2581
+2666
+2818
+1213
+1122
+1442
+2716
+3393
+3955
+481
+3751
+5542
+5505
+1313
+754
+8997
+902
+634
+712
+2626
+681
+5264
+1204
+3911
+2155
+9315
+3346
+4912
+778
+3361
+1198
+2778
+3979
+3847
+2179
+1546
+5411
+1630
+3490
+1213
+971
+4750
+2063
+1858
+1056
+1181
+4819
+927
+5051
+5907
+1516
+2684
+7342
+9083
+859
+1488
+2004
+5263
+1509
+5137
+803
+3494
+1658
+1312
+4109
+2749
+3128
+2635
+5201
+1593
+438
+1323
+3081
+4343
+936
+4245
+1683
+1624
+1443
+1812
+2581
+3248
+6351
+1856
+1153
+2226
+5764
+3561
+6978
+2359
+292
+2217
+2487
+684
+1240
+4017
+2150
+764
+676
+840
+3897
+3404
+873
+4408
+1810
+1733
+2715
+2874
+2223
+5884
+1959
+2014
+1220
+2481
+3005
+2596
+1415
+1720
+2002
+343
+2820
+1567
+5229
+613
+698
+9558
+869
+968
+3380
+2047
+272
+2977
+6011
+4508
+3571
+1713
+2494
+631
+1025
+6158
+210
+4971
+2700
+9538
+1564
+1277
+943
+1883
+3504
+5605
+6246
+3291
+1761
+4661
+1839
+1561
+1911
+1327
+2548
+4216
+288
+517
+2534
+875
+5142
+3603
+2531
+1346
+2014
+2834
+4128
+1539
+3784
+2943
+1947
+4579
+2131
+3896
+1840
+1080
+2020
+7125
+2150
+9856
+2230
+2511
+1558
+2289
+1754
+2236
+10355
+432
+1743
+3477
+1459
+240
+2204
+5627
+6976
+732
+569
+3524
+527
+2630
+1454
+1026
+7451
+2069
+865
+2561
+1460
+290
+3137
+645
+8434
+1408
+4814
+2356
+4254
+1384
+6482
+1083
+4138
+2301
+802
+2560
+2765
+660
+3707
+3913
+2401
+1451
+401
+885
+741
+6722
+799
+494
+1340
+671
+8406
+653
+4177
+7343
+3185
+437
+2559
+5174
+2115
+10415
+1510
+9032
+1728
+2150
+1692
+896
+7825
+2168
+63
+1331
+4252
+523
+840
+3648
+822
+776
+590
+2909
+1973
+1936
+714
+3564
+2272
+764
+1543
+636
+8746
+1550
+922
+232
+1067
+2241
+1255
+4658
+1058
+2307
+1373
+1282
+1270
+2013
+1726
+3097
+4246
+145
+1436
+3863
+965
+5165
+3274
+4639
+1506
+2777
+1947
+3924
+2443
+9726
+7495
+1045
+6935
+1359
+1245
+2121
+3252
+902
+2671
+1273
+466
+717
+7742
+3555
+2473
+6511
+1678
+5481
+2517
+966
+1750
+1681
+953
+3431
+4063
+2008
+2441
+6317
+1206
+1343
+1497
+935
+1378
+6114
+2148
+5954
+587
+1687
+2296
+3801
+3427
+455
+1107
+3102
+626
+2083
+1319
+786
+6392
+2645
+1687
+4970
+1707
+2149
+6434
+1035
+2157
+1381
+567
+2544
+995
+6559
+3816
+1985
+3369
+1083
+1798
+2392
+4483
+3209
+2588
+572
+4298
+6082
+6023
+637
+1908
+3740
+1610
+758
+272
+1949
+2837
+9104
+2069
+674
+68
+2378
+2274
+1098
+4052
+6677
+1200
+2745
+4001
+2648
+266
+6487
+1813
+1828
+1238
+695
+1499
+2541
+3162
+1999
+2529
+1436
+6567
+1477
+2814
+2672
+2599
+1443
+1711
+1197
+291
+1551
+1481
+6906
+3581
+579
+270
+2211
+1071
+5848
+2533
+1834
+987
+2227
+4975
+2321
+5182
+768
+1879
+6082
+3830
+1759
+4789
+4056
+1381
+1147
+7277
+481
+2362
+652
+1113
+6598
+4769
+1004
+1196
+2017
+445
+1819
+2710
+2649
+2332
+3313
+4342
+163
+8224
+4170
+4086
+1661
+4620
+1513
+6374
+3689
+1296
+4718
+2159
+1826
+2588
+4337
+1314
+2132
+907
+505
+1315
+555
+4936
+330
+2115
+882
+1338
+7184
+5849
+2290
+5246
+182
+1494
+752
+1145
+4329
+5166
+4064
+4453
+480
+7532
+1809
+2142
+1485
+516
+1825
+1038
+2822
+2613
+1497
+907
+4631
+1224
+3673
+2723
+1200
+1268
+4815
+4643
+4316
+8122
+313
+1327
+1704
+1929
+2075
+1953
+875
+3202
+1554
+3301
+1605
+433
+2857
+1805
+866
+2953
+2209
+2103
+3739
+481
+3472
+1574
+1870
+1049
+9267
+2373
+723
+4050
+1853
+642
+103
+2822
+4112
+509
+418
+483
+4006
+2795
+1913
+1267
+779
+4349
+1610
+1031
+532
+5625
+1791
+3469
+1340
+1474
+2152
+1635
+2574
+789
+1203
+5368
+2744
+1276
+284
+1455
+164
+1688
+1117
+3295
+1764
+655
+3943
+2844
+650
+1787
+1699
+942
+3909
+361
+5909
+589
+1675
+1434
+734
+784
+1162
+1416
+918
+1012
+3614
+2519
+1866
+3550
+6344
+6682
+866
+5403
+655
+1115
+2964
+7843
+656
+1826
+1808
+2926
+1445
+1350
+2984
+5624
+3360
+2527
+1913
+95
+2226
+543
+2238
+544
+533
+1130
+1448
+1455
+1342
+1988
+3291
+414
+3152
+4774
+741
+5326
+695
+2884
+4978
+907
+2661
+957
+3081
+1745
+1381
+1860
+1312
+6450
+3001
+1642
+1727
+720
+3383
+3184
+2673
+586
+934
+1932
+2472
+1139
+748
+502
+1509
+4031
+4754
+364
+1539
+1654
+103
+2392
+1748
+1766
+2060
+828
+1573
+4236
+2664
+1746
+7080
+2967
+2887
+1249
+2127
+8165
+1528
+357
+1373
+2513
+1455
+1632
+1459
+1568
+2672
+2563
+3408
+2054
+1255
+1888
+5796
+3325
+6681
+5618
+3208
+2368
+1002
+1694
+2242
+1142
+2698
+2425
+1854
+3290
+496
+3953
+4444
+1381
+1817
+2197
+502
+9787
+598
+1515
+1617
+2460
+4341
+3768
+2952
+2102
+1132
+1554
+8224
+2420
+3719
+1850
+1277
+869
+1578
+5497
+3273
+892
+694
+1414
+901
+1871
+1839
+6658
+3853
+7054
+1433
+627
+9496
+1248
+6737
+2525
+6069
+2669
+1869
+1958
+5050
+813
+2273
+4630
+8339
+521
+1376
+2888
+436
+6397
+723
+501
+764
+1447
+837
+4752
+95
+3776
+1052
+366
+1935
+996
+2200
+2369
+2452
+5014
+4211
+1166
+6190
+220
+1685
+990
+4112
+6915
+4207
+348
+1373
+5837
+978
+2318
+918
+2108
+666
+6901
+6650
+1260
+2552
+2750
+652
+3685
+5289
+4074
+1308
+2839
+4341
+3335
+4701
+1981
+6332
+3667
+1021
+2265
+986
+4828
+3372
+2498
+4463
+941
+1018
+1968
+1785
+1046
+174
+5058
+1019
+1089
+5731
+341
+2603
+3929
+8922
+982
+2310
+5997
+2548
+903
+3427
+2272
+2766
+2117
+1281
+3770
+1626
+2508
+1544
+1227
+7464
+1219
+5913
+1972
+1367
+3689
+2313
+7882
+1392
+1100
+1328
+1157
+5583
+2151
+1563
+6152
+373
+6136
+988
+2061
+2858
+4910
+7862
+2602
+2381
+6224
+1345
+2074
+1670
+1257
+1992
+5661
+1114
+4960
+953
+3488
+1249
+2128
+948
+1725
+3643
+988
+6012
+6893
+2032
+1458
+1399
+4045
+776
+1016
+699
+6951
+5837
+2397
+2798
+2160
+3400
+2002
+226
+2035
+2082
+5361
+1282
+4217
+1472
+768
+9136
+5746
+2086
+2397
+1975
+1692
+1547
+1820
+2043
+2176
+815
+1379
+381
+2452
+3626
+2869
+2688
+8456
+1680
+964
+1645
+4049
+1002
+7982
+1845
+1904
+515
+847
+1362
+475
+3270
+2474
+2946
+1866
+1428
+4614
+6180
+1582
+7069
+1831
+278
+935
+762
+5073
+7078
+1055
+1646
+3776
+828
+2225
+4147
+263
+1529
+12728
+6733
+623
+2019
+821
+1133
+4509
+1595
+798
+1175
+3716
+683
+1343
+1335
+6146
+2257
+4998
+702
+2029
+2842
+1291
+1455
+2536
+465
+892
+4390
+3579
+1091
+1760
+3953
+3292
+1489
+1686
+4899
+551
+5065
+4637
+602
+2418
+3391
+4700
+2795
+408
+918
+3177
+2332
+2519
+6508
+9345
+1757
+2799
+3414
+1653
+3632
+4169
+855
+1228
+3349
+3204
+785
+1006
+778
+1822
+5932
+948
+1757
+1532
+4637
+3160
+1043
+3211
+4835
+930
+2249
+1255
+744
+2500
+1102
+3038
+907
+1467
+5621
+3300
+4140
+1008
+2364
+3991
+2968
+3826
+1863
+2179
+2323
+1927
+1400
+1978
+1780
+1399
+6824
+8282
+2751
+1466
+1452
+4656
+7116
+2791
+688
+896
+1768
+2685
+2043
+523
+2289
+350
+7068
+1687
+563
+2860
+2936
+2161
+401
+987
+3010
+1112
+2872
+1710
+1431
+3967
+4530
+2560
+605
+1791
+2853
+945
+1330
+2069
+4420
+2092
+4497
+4424
+6086
+1526
+3852
+5651
+2476
+4285
+397
+2382
+4745
+1136
+9009
+909
+3909
+4514
+2624
+101
+950
+1365
+3552
+2623
+1937
+2034
+1280
+4094
+1833
+1808
+5432
+3041
+3454
+2888
+1205
+676
+1648
+3565
+1226
+3210
+1246
+2427
+3109
+2870
+4839
+867
+1325
+3592
+1610
+9899
+1122
+2509
+4258
+3176
+3394
+645
+2288
+4543
+1158
+2141
+14297
+2222
+699
+838
+469
+1475
+854
+2065
+1170
+245
+5344
+5283
+2004
+1135
+7980
+1838
+629
+2261
+6292
+2797
+840
+4107
+1831
+2147
+1797
+1073
+3600
+3397
+1385
+3514
+998
+624
+2995
+742
+7912
+4333
+2603
+6234
+1928
+480
+3530
+7969
+791
+1210
+621
+10777
+525
+4543
+2610
+2139
+865
+3948
+2336
+3390
+348
+1690
+791
+948
+6284
+3505
+3290
+1715
+7887
+1328
+1711
+1243
+3028
+1940
+474
+1362
+1747
+1580
+3881
+4857
+5619
+3970
+2830
+781
+1134
+3998
+2438
+1583
+223
+4598
+938
+182
+2386
+5199
+4135
+2405
+2422
+9968
+700
+1498
+5065
+3108
+2889
+7216
+2720
+453
+306
+1957
+2649
+803
+3447
+3212
+120
+848
+3232
+3041
+990
+6808
+474
+1104
+5714
+774
+1281
+2487
+1356
+12053
+6248
+4034
+1115
+1543
+6538
+1056
+4114
+3258
+1649
+876
+6889
+3095
+1895
+2391
+3774
+3186
+1862
+8345
+958
+10173
+919
+3989
+3048
+2395
+329
+1345
+2067
+631
+4890
+491
+5022
+2384
+3482
+2595
+1653
+2515
+4782
+113
+3387
+2348
+2242
+1220
+384
+392
+4535
+6139
+2602
+4511
+2150
+887
+5743
+1126
+5642
+3698
+1076
+2740
+2289
+1255
+2124
+685
+12170
+5448
+830
+319
+412
+625
+1669
+3085
+2097
+1854
+1633
+2407
+1181
+1358
+6075
+1017
+3113
+3688
+5008
+5923
+1076
+2980
+7204
+911
+676
+845
+3780
+2632
+1505
+1906
+6048
+1374
+3094
+2351
+2667
+1220
+2098
+6482
+2323
+2121
+7935
+6331
+2607
+1246
+5670
+1833
+3745
+4174
+134
+811
+2212
+1171
+1011
+1438
+316
+2073
+1605
+721
+6838
+1535
+1120
+3783
+1148
+8265
+1543
+501
+3028
+2093
+901
+5434
+4557
+5476
+4466
+4791
+840
+486
+2594
+2765
+1834
+1541
+882
+1204
+1070
+2474
+3243
+863
+2401
+523
+1147
+5722
+1161
+521
+1241
+1792
+1066
+1871
+6711
+3486
+1668
+1339
+249
+5571
+639
+1813
+573
+9264
+2550
+920
+906
+3953
+4990
+1829
+1019
+852
+1986
+1482
+913
+2884
+7308
+2022
+8239
+1368
+1025
+2508
+1761
+1152
+4680
+7363
+5163
+2061
+5326
+598
+5177
+6505
+2374
+8591
+1639
+1196
+2818
+1987
+156
+3925
+3492
+627
+1944
+539
+5503
+1124
+1328
+1587
+974
+5674
+1702
+1418
+5870
+1402
+2167
+1158
+6192
+1161
+1435
+3669
+3066
+785
+582
+2082
+1229
+2280
+4458
+1018
+1708
+593
+1441
+918
+2038
+2177
+530
+1835
+675
+9947
+488
+2381
+1543
+4175
+1973
+1834
+3154
+999
+1805
+3209
+2555
+1687
+5826
+6702
+1159
+8770
+1134
+4979
+5016
+2802
+3864
+2505
+978
+679
+2326
+1838
+945
+1377
+439
+1197
+4703
+483
+1214
+167
+1688
+1015
+1501
+3282
+2162
+6013
+5142
+2772
+2581
+1505
+286
+1448
+1197
+3291
+4136
+791
+3815
+1880
+5531
+1492
+1779
+2617
+5817
+6358
+3891
+3234
+1870
+1006
+2451
+4826
+665
+1632
+7863
+8659
+1160
+3223
+783
+1414
+3341
+2159
+8055
+2958
+3682
+3073
+1561
+4384
+1211
+7689
+2106
+659
+5514
+2103
+603
+6055
+948
+642
+874
+2358
+985
+2090
+6962
+5313
+3173
+918
+2340
+1299
+1639
+5184
+1687
+759
+2556
+3309
+2066
+970
+4948
+1976
+446
+2747
+1483
+3565
+450
+434
+2044
+1078
+3781
+3475
+2254
+740
+914
+2867
+5244
+4080
+143
+2084
+871
+3691
+439
+2126
+864
+2200
+936
+1716
+816
+368
+1934
+1376
+3281
+1297
+514
+845
+4033
+1633
+2569
+1271
+635
+5517
+3419
+2102
+4320
+1821
+3519
+945
+1819
+5758
+1407
+2427
+2536
+1291
+898
+5647
+3051
+550
+11591
+1741
+4503
+1334
+3010
+2350
+6641
+831
+3098
+1533
+664
+9930
+2230
+1446
+651
+1149
+969
+3613
+771
+2650
+1305
+1580
+864
+1292
+1762
+12486
+1465
+1165
+5210
+2041
+1547
+2003
+4083
+1565
+750
+1124
+684
+4777
+720
+3973
+647
+4804
+683
+5837
+7530
+246
+1133
+3910
+2185
+5372
+628
+4177
+3767
+1530
+1053
+1223
+633
+1144
+1136
+1523
+1310
+1926
+1092
+991
+1716
+507
+3793
+3894
+820
+3446
+2010
+2710
+1607
+1230
+1554
+1142
+910
+2535
+547
+916
+1382
+4513
+472
+1809
+1854
+952
+609
+2670
+1317
+1120
+2620
+1140
+592
+1674
+1222
+649
+683
+6836
+2778
+339
+3624
+2464
+1780
+4644
+2486
+778
+10245
+2441
+8303
+799
+763
+429
+1331
+2795
+1615
+1311
+1686
+3539
+7224
+3887
+2750
+3968
+1117
+4561
+1337
+1585
+1299
+213
+2914
+1547
+438
+2421
+1699
+1534
+2193
+10673
+3552
+1251
+2285
+7231
+3369
+2960
+1391
+1195
+1092
+802
+1446
+4897
+6130
+5479
+3684
+1968
+4640
+408
+1909
+974
+1650
+2004
+1911
+774
+450
+1516
+593
+2633
+2153
+6615
+5814
+998
+4563
+155
+4049
+797
+6461
+1534
+1166
+7722
+663
+2257
+1010
+2657
+1344
+1234
+3757
+884
+1016
+4003
+1673
+1203
+8846
+779
+383
+3554
+2851
+989
+904
+2857
+1877
+7512
+347
+1381
+1001
+4482
+2312
+2693
+1074
+2236
+4744
+876
+636
+4202
+6696
+4130
+7935
+1306
+1275
+898
+314
+847
+2730
+369
+1398
+2798
+2932
+2221
+2494
+4128
+893
+1427
+3825
+2337
+785
+2242
+3973
+5361
+1202
+1833
+7057
+4317
+4500
+1971
+1011
+4433
+4666
+615
+3899
+2354
+6924
+1774
+3601
+8886
+2326
+5305
+5195
+1951
+1643
+638
+5354
+837
+1159
+522
+4347
+3429
+424
+3640
+622
+5697
+1887
+4635
+1081
+2708
+1862
+3131
+2940
+1521
+1023
+1001
+1176
+1887
+1000
+1584
+2315
+2812
+1544
+3294
+854
+757
+1746
+1487
+868
+2540
+807
+996
+1557
+1305
+4677
+898
+692
+1141
+1376
+4546
+7008
+163
+3300
+6267
+2256
+1238
+2680
+815
+2005
+2264
+2233
+3887
+1314
+4950
+1969
+4776
+1374
+7728
+1591
+1460
+4641
+909
+2878
+3078
+2191
+609
+1812
+830
+1114
+2743
+795
+2766
+2026
+1013
+3448
+2000
+1460
+10424
+2140
+3062
+539
+3221
+3791
+825
+1731
+198
+2335
+2691
+600
+1586
+954
+4140
+4367
+277
+709
+7655
+395
+2275
+2969
+2020
+3568
+2774
+1730
+621
+1211
+1498
+4050
+4963
+3548
+299
+1503
+2297
+2510
+2645
+2681
+3783
+6026
+3838
+2058
+2086
+1958
+2091
+1468
+1924
+153
+1977
+2720
+2762
+3572
+1438
+1113
+7578
+2132
+4098
+1542
+6588
+3200
+8616
+5500
+2230
+3189
+1790
+2094
+1623
+877
+1153
+4495
+1710
+1730
+3541
+1544
+795
+1592
+770
+2700
+1560
+11001
+2143
+639
+6359
+3280
+4405
+980
+998
+4008
+4796
+1879
+2480
+2989
+816
+1338
+2031
+1284
+2670
+607
+2154
+725
+2398
+1600
+2367
+4124
+2370
+5812
+1641
+729
+1008
+1640
+658
+5894
+745
+974
+5068
+2440
+938
+1608
+7309
+3678
+6756
+1670
+1077
+722
+11319
+5782
+2084
+2197
+2462
+2920
+747
+864
+2643
+6332
+2246
+1745
+3158
+1334
+1011
+2556
+1193
+7828
+1940
+4497
+8565
+2364
+1679
+1945
+1354
+3006
+403
+510
+3810
+1123
+3753
+1840
+8356
+825
+1616
+1795
+1071
+1753
+4775
+2671
+3636
+1605
+852
+2695
+4124
+4812
+1131
+3235
+576
+4854
+2189
+1197
+4200
+468
+1080
+888
+979
+5271
+5975
+586
+1751
+492
+982
+6858
+984
+1410
+3569
+4131
+1255
+120
+6946
+3160
+3845
+3071
+4570
+1055
+1814
+545
+1050
+4312
+1607
+888
+3514
+5820
+159
+4024
+459
+456
+2302
+1400
+1621
+4835
+3280
+2133
+2537
+1443
+3113
+4949
+7481
+2225
+877
+2692
+3188
+5135
+4895
+1708
+906
+949
+9547
+1532
+408
+6841
+1935
+9078
+2456
+1710
+373
+1963
+7821
+1240
+4705
+271
+1004
+2298
+895
+2234
+691
+3189
+1829
+6026
+4126
+2646
+3228
+3382
+1952
+5800
+1860
+2170
+1817
+1934
+303
+6731
+9834
+1777
+555
+1198
+1536
+5268
+1621
+1088
+951
+1245
+9105
+2380
+5542
+2930
+604
+5903
+1695
+3953
+1810
+5485
+5346
+2560
+4696
+511
+1512
+823
+6950
+2910
+6054
+2280
+6780
+2104
+6729
+254
+1608
+3607
+1834
+2191
+758
+5832
+1425
+3158
+3767
+1651
+3211
+1257
+1981
+4335
+1997
+1688
+2098
+1084
+9950
+600
+3198
+2875
+1936
+1052
+6688
+2804
+104
+7604
+5459
+3669
+1104
+615
+3095
+765
+2443
+703
+784
+6279
+1476
+1003
+1644
+919
+1577
+1259
+4134
+439
+4132
+757
+218
+1621
+4848
+1318
+1592
+6480
+2065
+3072
+1543
+6255
+2275
+4009
+4377
+817
+5210
+1990
+163
+571
+2098
+3155
+1437
+2043
+2309
+5602
+2605
+1668
+2184
+3852
+11536
+1411
+3830
+488
+1459
+2941
+1259
+1272
+207
+2904
+1540
+2817
+4824
+356
+1929
+5041
+1645
+2454
+4811
+717
+2261
+2527
+3408
+1630
+1756
+1366
+2805
+633
+3578
+1284
+1490
+2516
+1166
+568
+3023
+7754
+877
+1926
+759
+1526
+932
+3209
+2228
+1846
+271
+2255
+3008
+2709
+3216
+2501
+3978
+1855
+3594
+4148
+2106
+3660
+1914
+8667
+2495
+890
+6935
+1063
+3761
+517
+798
+1318
+1907
+948
+1890
+3231
+337
+2575
+1003
+2667
+3173
+1054
+130
+1860
+6341
+2475
+581
+3609
+5687
+1370
+1756
+3297
+3962
+4783
+193
+879
+601
+527
+190
+298
+2525
+3301
+552
+2619
+1601
+1362
+427
+1360
+3391
+6695
+1785
+2441
+1090
+1720
+1544
+513
+1450
+1586
+397
+2155
+1255
+628
+2304
+1528
+5380
+625
+4279
+8227
+1221
+2471
+2191
+7262
+1404
+2076
+1034
+10597
+798
+4013
+7523
+913
+1229
+4817
+3018
+2346
+1169
+8131
+1923
+1642
+4363
+411
+2231
+1588
+5004
+274
+4253
+2223
+1142
+2299
+2326
+2201
+3407
+3024
+522
+1548
+2600
+1128
+1586
+3355
+2714
+1243
+3812
+213
+4086
+2736
+6462
+2758
+3879
+4502
+1590
+692
+6549
+61
+5463
+296
+2216
+3400
+1083
+2165
+2549
+5104
+11581
+835
+1461
+1641
+756
+6718
+1479
+1988
+1425
+3234
+526
+5225
+944
+5255
+1262
+3775
+8015
+296
+1528
+1129
+7033
+1290
+5675
+1052
+662
+8345
+1024
+3052
+424
+1694
+2244
+3511
+533
+5007
+2308
+563
+1279
+3012
+4070
+4040
+950
+3426
+2231
+1987
+1314
+3529
+1284
+2495
+236
+2627
+2673
+269
+739
+3688
+10759
+2983
+6827
+3287
+621
+1212
+1701
+2581
+2908
+4302
+1064
+4474
+1274
+5590
+7446
+2519
+1920
+1506
+936
+2132
+1006
+2256
+2673
+1192
+6505
+1447
+940
+710
+333
+2479
+226
+4490
+1973
+3197
+11433
+3049
+1356
+3353
+1438
+1985
+1584
+853
+300
+2078
+1041
+6366
+7037
+2894
+998
+1281
+1955
+1955
+3237
+1552
+2217
+1001
+6211
+637
+2615
+835
+2146
+3401
+1127
+4647
+1780
+2581
+1751
+1393
+4697
+1752
+1129
+5145
+994
+2551
+1518
+4776
+1095
+2498
+1318
+1170
+1230
+454
+1740
+2897
+3317
+5184
+2304
+5447
+1909
+1794
+1425
+2551
+2103
+4717
+2166
+1603
+2071
+2781
+4586
+1784
+6008
+1395
+344
+4473
+6683
+2591
+477
+253
+900
+2132
+3453
+2288
+463
+1768
+1543
+1555
+7338
+2845
+3594
+1741
+1559
+6068
+3928
+3318
+2864
+1193
+5414
+1079
+792
+878
+8359
+648
+1129
+2080
+895
+1713
+1092
+5361
+3646
+1180
+2007
+2508
+3489
+2543
+1748
+2175
+1316
+6110
+2909
+3396
+1025
+5661
+2923
+1165
+982
+7126
+1163
+3973
+1694
+5175
+816
+619
+1609
+3341
+3254
+3822
+600
+667
+1543
+1157
+5803
+626
+1161
+2963
+1375
+4884
+740
+430
+1619
+2209
+4595
+5489
+1979
+3341
+1378
+1378
+4281
+1269
+943
+1527
+698
+2627
+1216
+6714
+4458
+1660
+6583
+1282
+4629
+8640
+3493
+1897
+994
+10654
+1168
+2195
+1955
+7826
+1814
+1092
+2822
+668
+6959
+4241
+1654
+3006
+1206
+1639
+2631
+5495
+2597
+670
+3226
+1884
+2284
+1731
+2435
+1131
+2058
+743
+1087
+1991
+688
+899
+3842
+4772
+751
+2197
+1695
+1300
+2825
+2945
+1751
+951
+2743
+5611
+1164
+4682
+3112
+2129
+2036
+1300
+2887
+1737
+1209
+118
+2751
+3863
+865
+759
+1146
+5985
+6447
+2215
+319
+3914
+2689
+1963
+7651
+5910
+260
+6574
+686
+1683
+957
+1570
+942
+910
+1612
+1179
+2199
+2294
+948
+3202
+1075
+2093
+1616
+1862
+5485
+4956
+634
+2055
+1072
+1319
+971
+3798
+789
+524
+1292
+3355
+5380
+2520
+4088
+1262
+659
+1638
+2069
+3114
+4495
+2988
+1094
+4777
+3114
+1122
+899
+1876
+1027
+8328
+4469
+4519
+8529
+1653
+5026
+1480
+3556
+3926
+2309
+1992
+1063
+1629
+2781
+1717
+2273
+2148
+5971
+1080
+1152
+1833
+1413
+5393
+5822
+5833
+437
+813
+1051
+3416
+1494
+839
+518
+1617
+1015
+3280
+1874
+1423
+2231
+367
+867
+1380
+702
+8459
+533
+6646
+2776
+543
+6371
+1015
+2607
+6747
+1097
+3375
+2771
+932
+1714
+4368
+6813
+1719
+1913
+1202
+3439
+1614
+8372
+1349
+1149
+1389
+408
+3886
+7102
+3848
+871
+1157
+5278
+2265
+1825
+1014
+3208
+2800
+719
+2517
+2976
+9853
+2667
+3282
+1781
+8692
+11205
+2319
+2907
+1284
+2411
+510
+2834
+8250
+2871
+1421
+340
+3580
+3107
+2870
+1156
+644
+1835
+5302
+352
+1343
+1900
+262
+2302
+1370
+537
+1433
+1504
+2364
+3619
+983
+4552
+2225
+1531
+1413
+2242
+2610
+2230
+3228
+5138
+3538
+5834
+3366
+1251
+878
+4694
+2797
+5994
+419
+6432
+752
+4758
+2244
+2938
+2707
+1929
+1111
+1826
+4310
+3597
+6769
+1746
+1175
+2424
+5462
+514
+7844
+964
+7634
+2803
+5212
+2077
+2712
+1842
+1493
+954
+1665
+1536
+879
+2221
+359
+3113
+2404
+6064
+568
+4595
+1998
+2183
+1061
+6661
+5034
+3216
+3543
+714
+4122
+1512
+2405
+5448
+1539
+428
+835
+276
+2408
+3555
+3667
+916
+2900
+835
+1516
+1392
+1376
+1321
+2534
+791
+1164
+805
+2604
+645
+1886
+297
+3395
+8899
+1753
+3945
+854
+749
+1892
+5666
+1327
+7067
+1330
+2550
+3595
+1958
+640
+2260
+929
+469
+9382
+1741
+2734
+596
+4293
+1777
+2493
+3397
+2022
+462
+2072
+4107
+988
+9444
+954
+319
+1043
+5041
+4142
+2259
+1990
+293
+1495
+5023
+428
+2278
+597
+8538
+2662
+1047
+1746
+2695
+5162
+2603
+1486
+808
+2189
+1240
+1418
+3003
+680
+6985
+2609
+2754
+1646
+1364
+7658
+3545
+2547
+2133
+996
+1164
+7632
+1239
+1216
+7263
+3032
+3568
+2311
+1996
+176
+1424
+1747
+887
+1965
+1801
+1161
+1102
+1204
+1452
+983
+4526
+698
+1465
+282
+341
+1470
+2577
+776
+1929
+7180
+2478
+3571
+946
+4121
+2294
+1181
+667
+3022
+2988
+1019
+2228
+5977
+627
+1707
+2149
+1644
+1154
+2369
+6318
+3323
+446
+1083
+1750
+6236
+1558
+1257
+3042
+2741
+97
+2752
+1340
+616
+4950
+1499
+7940
+1531
+2727
+4265
+1203
+1293
+1026
+837
+8011
+1021
+996
+7357
+2874
+1375
+546
+2448
+1998
+6005
+532
+4454
+702
+926
+5629
+1597
+1517
+3363
+375
+333
+4996
+1031
+3353
+6361
+1688
+941
+2822
+541
+3021
+1877
+2414
+1138
+1232
+1998
+2383
+1072
+1096
+2407
+5199
+606
+1377
+955
+2692
+3672
+2209
+1365
+6275
+8260
+374
+1713
+3298
+8003
+822
+2279
+683
+1475
+348
+2719
+1509
+4899
+3663
+1937
+3552
+2258
+3953
+3415
+613
+3565
+1161
+1801
+2323
+810
+533
+1062
+5483
+2651
+2453
+1097
+404
+1811
+7277
+716
+1791
+2200
+2485
+2688
+4860
+1840
+3471
+3489
+1599
+1483
+5744
+1377
+3676
+2796
+1088
+2830
+1122
+1820
+8776
+5243
+2385
+4264
+971
+657
+9318
+1797
+7395
+4421
+5440
+550
+7630
+8296
+565
+676
+2574
+2703
+2854
+1413
+1693
+1126
+3472
+1724
+645
+8698
+6773
+464
+2038
+2207
+8245
+4133
+1686
+2288
+8900
+3343
+1550
+287
+2780
+5738
+7775
+4142
+1739
+4110
+5434
+1263
+1876
+1931
+838
+2184
+1229
+5636
+4833
+782
+2054
+1219
+6627
+2867
+1539
+1565
+10256
+1873
+2225
+511
+1504
+1570
+1921
+1598
+1388
+553
+5309
+622
+1930
+647
+5158
+1726
+1760
+3370
+9201
+1066
+1319
+307
+1991
+4326
+2477
+1455
+3108
+6682
+4284
+488
+4823
+3664
+2688
+2920
+1539
+7110
+488
+3552
+1853
+1106
+1747
+6428
+5264
+2804
+2669
+1010
+2012
+2175
+2000
+704
+3329
+5566
+3534
+6948
+226
+3672
+749
+1759
+1478
+5306
+1477
+2006
+9422
+1164
+1015
+4442
+4645
+1930
+1965
+573
+7604
+839
+684
+5476
+3502
+1763
+1266
+7567
+1415
+2260
+3535
+5979
+3642
+2078
+1201
+2894
+1893
+480
+2026
+809
+3375
+3760
+3697
+1837
+1536
+1387
+837
+196
+2046
+6252
+496
+2089
+10369
+1814
+2498
+3159
+1929
+5420
+984
+2811
+1497
+2027
+4895
+1115
+1982
+1825
+2486
+999
+2153
+1124
+2275
+2309
+588
+1681
+6384
+480
+1478
+815
+2619
+1929
+3409
+5445
+7252
+3386
+2185
+326
+3297
+959
+650
+8137
+1397
+1648
+5020
+1129
+923
+1911
+1703
+1102
+1372
+5660
+1619
+6244
+280
+1669
+3495
+2208
+1300
+3667
+6658
+71
+926
+1681
+628
+5226
+7232
+1081
+4726
+1761
+1909
+1540
+4263
+3512
+3493
+1681
+4897
+3887
+793
+1190
+2130
+2908
+282
+572
+2465
+1941
+3438
+3810
+2555
+7546
+3998
+1784
+2520
+2030
+3428
+431
+2739
+5785
+632
+1851
+1999
+717
+632
+3711
+2485
+1482
+1740
+3194
+3614
+1912
+2010
+510
+560
+4141
+2796
+604
+897
+625
+3022
+3132
+2082
+5101
+1704
+1505
+2720
+3837
+311
+1399
+3075
+5639
+1209
+2112
+1034
+518
+1945
+1597
+2201
+8105
+1749
+4639
+1511
+1519
+1893
+1843
+1187
+815
+8134
+3040
+1733
+740
+3948
+277
+3414
+5328
+2579
+4336
+2343
+311
+2682
+497
+1572
+1761
+2121
+5948
+1331
+734
+1255
+1707
+141
+2309
+2824
+2722
+2014
+1844
+817
+2543
+2415
+1630
+830
+9948
+535
+5501
+1143
+1567
+470
+3398
+1740
+4936
+4399
+3849
+3115
+1210
+1166
+1384
+363
+817
+1360
+1367
+685
+2450
+3938
+3474
+2642
+4013
+545
+2375
+1438
+3526
+1492
+4049
+3599
+222
+7579
+3617
+7578
+8036
+341
+3698
+8524
+150
+2338
+2366
+9323
+1263
+929
+1349
+1207
+1787
+2344
+6180
+2215
+863
+1487
+8369
+6286
+485
+2743
+1258
+4714
+3671
+2577
+1290
+1734
+2144
+4218
+789
+2964
+932
+947
+1289
+708
+2073
+1433
+3486
+1626
+3696
+391
+2486
+4461
+2936
+933
+915
+5195
+3505
+1794
+1233
+1595
+1826
+4341
+1172
+5495
+1266
+1598
+1867
+8877
+2279
+6342
+1279
+3558
+10583
+2957
+4739
+1145
+1876
+3015
+1909
+8371
+3117
+1062
+3061
+3810
+1247
+3542
+1568
+1942
+917
+2165
+1987
+1908
+2380
+1388
+2449
+1512
+3005
+1096
+6220
+699
+808
+1359
+938
+1561
+651
+1130
+13536
+1686
diff --git a/lessons/StatisticalConsiderations/data/permutation_data.csv b/lessons/StatisticalConsiderations/data/permutation_data.csv
new file mode 100644
index 00000000..98012c67
--- /dev/null
+++ b/lessons/StatisticalConsiderations/data/permutation_data.csv
@@ -0,0 +1,5001 @@
+condition,time
+0,5940
+0,666
+1,571
+1,779
+0,1928
+0,3791
+0,736
+1,8482
+0,4594
+0,1171
+1,3061
+0,1178
+1,5210
+0,1498
+0,8305
+1,1731
+1,485
+1,2455
+1,3144
+1,7865
+0,8132
+1,1302
+0,3144
+0,3689
+1,1784
+0,2567
+1,2543
+1,3508
+0,1047
+1,3793
+1,515
+1,1035
+0,4367
+1,2664
+0,1464
+0,5470
+1,1520
+1,1752
+0,3406
+1,1801
+0,2033
+0,2703
+0,1097
+1,1320
+0,2406
+1,6467
+0,3864
+1,1191
+1,2983
+1,1134
+1,2828
+1,1564
+0,454
+0,5455
+1,6835
+1,279
+0,1711
+0,550
+0,574
+0,6380
+1,8493
+1,1376
+0,1795
+1,1711
+1,3361
+1,7463
+0,4762
+1,1271
+1,2487
+0,1022
+0,777
+1,1625
+1,1438
+0,2057
+1,3809
+1,1044
+1,6726
+0,3495
+1,3115
+1,2015
+0,1185
+1,3153
+1,2817
+1,843
+1,3790
+1,2182
+1,3760
+0,1077
+0,2032
+0,3323
+0,9198
+0,5657
+1,1778
+0,1235
+1,384
+0,2352
+1,512
+0,2096
+0,292
+0,464
+1,443
+1,1950
+1,1641
+0,1312
+0,8859
+1,1229
+0,7485
+1,1675
+0,5136
+0,2131
+0,1877
+1,2500
+1,1309
+0,3148
+0,1918
+1,1806
+1,4313
+1,630
+0,1095
+0,3775
+0,2116
+1,833
+1,377
+1,391
+1,5198
+0,556
+0,8516
+0,1159
+0,7695
+1,2016
+1,1903
+1,1111
+0,493
+1,2335
+1,872
+1,804
+1,2721
+1,1403
+0,816
+1,241
+1,1569
+0,2243
+1,1278
+0,1117
+0,1845
+1,2119
+0,1852
+0,832
+0,5574
+1,2190
+1,1899
+0,2999
+0,1641
+1,3310
+1,107
+0,6273
+1,7772
+0,2069
+1,1399
+1,1950
+0,1431
+1,2396
+0,1335
+1,5038
+0,1257
+1,1545
+1,4052
+0,1975
+0,699
+0,914
+0,5509
+0,1392
+1,2802
+1,3106
+0,1209
+1,1816
+1,8746
+1,3992
+1,3629
+0,1581
+1,7541
+0,2270
+1,790
+0,1119
+0,1383
+1,2397
+0,6950
+0,2086
+0,3623
+0,1739
+1,1149
+0,762
+1,1315
+1,1265
+1,4040
+1,1107
+0,351
+0,3178
+1,1392
+0,1833
+1,3004
+0,1343
+0,1329
+0,1712
+0,1876
+1,1624
+0,5971
+1,3858
+0,1643
+0,5841
+1,1062
+0,9238
+1,612
+1,2420
+1,886
+1,3117
+0,4046
+0,2475
+1,1441
+0,5385
+0,1638
+1,1840
+0,1062
+1,5738
+1,110
+1,1409
+1,2516
+0,2967
+0,805
+0,1108
+1,1198
+1,2430
+1,2132
+1,1068
+1,2111
+0,1387
+1,2235
+1,1550
+1,1516
+1,1138
+1,1615
+1,4059
+1,1445
+0,814
+0,3074
+0,7805
+1,3688
+0,7211
+0,3974
+0,8871
+1,956
+1,735
+0,1289
+0,4723
+0,3343
+0,2029
+0,2029
+1,5428
+1,1782
+0,858
+0,3061
+1,2972
+1,2607
+0,1699
+0,422
+1,1413
+1,2503
+1,2434
+1,2876
+0,1625
+1,860
+0,6081
+0,1997
+0,631
+0,6022
+1,5242
+1,1416
+1,4888
+1,1733
+1,1331
+1,8529
+0,1714
+1,966
+1,965
+0,2699
+0,1457
+1,723
+0,1803
+0,6866
+0,4812
+1,386
+0,273
+1,1813
+1,2878
+1,5750
+0,7479
+1,306
+1,1638
+1,6218
+0,416
+1,1068
+0,2473
+1,1322
+0,4678
+0,1350
+1,1449
+1,1502
+1,964
+1,4878
+1,2470
+0,9441
+0,3208
+0,1551
+1,6290
+1,750
+0,995
+1,3647
+0,9059
+0,1322
+1,1612
+1,3179
+0,1085
+1,5275
+0,1987
+0,1344
+1,1738
+0,1279
+1,8392
+0,4891
+0,2684
+1,5073
+1,1187
+0,7083
+0,944
+0,2299
+1,601
+0,1353
+0,1226
+1,2445
+1,4641
+0,6492
+1,2213
+1,1131
+0,2288
+1,6254
+0,695
+0,544
+0,2870
+1,1560
+1,4556
+1,1635
+0,1880
+0,6465
+1,1525
+1,6087
+1,362
+1,3486
+0,3962
+1,5877
+0,2397
+0,1724
+0,8797
+0,832
+0,2987
+1,346
+1,7770
+0,1780
+1,1884
+1,8242
+1,729
+0,3170
+0,4490
+1,4692
+1,4368
+1,1040
+1,2569
+0,1337
+1,416
+0,4733
+0,1103
+0,795
+1,7152
+1,2908
+0,766
+1,1804
+1,1077
+1,1809
+1,196
+1,4153
+1,2770
+0,1674
+1,3266
+1,1083
+0,1432
+0,5213
+1,1882
+1,2162
+1,1120
+1,1218
+0,6717
+1,5453
+1,979
+1,1563
+1,1252
+0,3290
+0,1132
+0,176
+1,2544
+1,3706
+0,1192
+0,985
+0,1460
+1,893
+0,2065
+0,810
+1,2040
+0,89
+1,1686
+0,686
+0,5190
+1,211
+1,4019
+0,1803
+0,6571
+1,886
+0,991
+0,2543
+0,3001
+1,445
+1,1620
+1,693
+1,1054
+0,4430
+1,2399
+1,1775
+1,449
+0,664
+0,1152
+0,1918
+0,2073
+1,676
+1,2650
+1,505
+0,1928
+0,777
+1,5966
+1,1024
+1,659
+0,4725
+0,1861
+0,1184
+1,2139
+1,4216
+1,2167
+0,8124
+1,3916
+1,6223
+1,419
+0,226
+1,10835
+0,1569
+0,1217
+0,1097
+1,6656
+1,3078
+0,1590
+0,2391
+1,1165
+0,856
+1,1158
+1,1289
+0,5754
+0,11727
+0,831
+1,2060
+0,1624
+0,1080
+0,72
+0,1122
+0,1212
+0,5918
+0,2118
+0,6939
+1,1665
+0,1259
+0,7794
+1,701
+1,473
+1,5330
+1,1721
+1,346
+1,293
+0,2365
+0,1095
+0,1690
+1,3399
+1,6883
+0,1224
+1,284
+1,3568
+1,4775
+1,2174
+0,3227
+0,684
+0,822
+0,4129
+0,2306
+0,2073
+1,1847
+1,948
+0,10157
+1,3020
+0,370
+0,6980
+0,4051
+0,1787
+0,1103
+1,2038
+0,643
+0,2031
+1,1849
+1,3482
+1,3933
+1,2584
+0,1973
+1,2183
+0,1659
+1,11091
+0,283
+1,1723
+0,9195
+0,725
+1,2574
+1,1364
+0,1314
+0,1430
+1,4658
+0,825
+0,2030
+0,1820
+0,1488
+1,2167
+0,1611
+1,3582
+1,2737
+0,602
+0,666
+1,4140
+0,4764
+0,385
+1,1371
+1,2370
+0,887
+1,6108
+0,4685
+0,2785
+1,2764
+1,6024
+0,827
+1,4284
+0,2213
+0,960
+0,3587
+0,1896
+1,2331
+0,1232
+0,6041
+1,1482
+0,775
+1,1793
+0,2070
+0,545
+1,1913
+1,168
+0,2053
+1,3769
+0,1105
+1,5632
+0,5927
+1,825
+1,1785
+0,2833
+1,1199
+0,3039
+0,3098
+1,519
+1,2280
+0,7166
+0,2695
+0,1123
+0,2443
+1,937
+0,8024
+0,840
+0,3640
+1,811
+1,5494
+0,1898
+0,1104
+1,4721
+0,1454
+0,5372
+1,3655
+1,1770
+0,2617
+0,2474
+1,1647
+1,2103
+1,6755
+1,3221
+1,3034
+1,1122
+1,1915
+1,811
+1,2916
+1,382
+0,8408
+1,1995
+1,3818
+0,2369
+1,526
+1,2180
+0,4759
+0,350
+1,3836
+1,1134
+1,1955
+1,2513
+0,5351
+0,4067
+1,1639
+0,2586
+0,1220
+1,1491
+1,5659
+0,13095
+0,3369
+0,1201
+0,617
+0,2169
+0,859
+1,2386
+1,7761
+1,1764
+0,1442
+0,214
+1,793
+0,2888
+1,5138
+1,3695
+0,268
+0,1518
+1,2690
+0,1638
+0,3060
+0,496
+0,1923
+0,70
+0,1475
+1,432
+0,2384
+0,1998
+0,1318
+1,1239
+1,2336
+1,3373
+1,1181
+0,1695
+1,1378
+1,7095
+1,2113
+1,3900
+0,1542
+0,1826
+0,3147
+0,1780
+1,7517
+1,1290
+1,2875
+1,1050
+1,543
+1,3469
+0,6032
+0,2386
+0,9193
+1,849
+1,5700
+0,1821
+0,1437
+1,4261
+1,13985
+1,2417
+1,643
+1,2853
+0,5403
+0,1365
+0,4235
+0,3183
+1,302
+0,6272
+1,8976
+1,831
+0,1859
+0,2348
+0,1648
+0,745
+1,992
+0,5615
+0,2114
+1,5037
+1,346
+1,478
+1,9509
+0,2023
+0,2930
+1,2818
+1,2010
+0,1840
+1,698
+0,6210
+0,1996
+0,2413
+0,1163
+0,3203
+0,3659
+0,1049
+1,1688
+1,2387
+1,2226
+1,2172
+1,1463
+1,5648
+0,1319
+0,2519
+0,1050
+0,4236
+0,3563
+1,3995
+1,423
+1,994
+1,1925
+1,1857
+1,1003
+0,1429
+0,7253
+1,6809
+1,4054
+1,2997
+1,1131
+1,1102
+1,1235
+0,1328
+0,5040
+0,90
+1,898
+0,2684
+0,1498
+1,7694
+0,2503
+0,2736
+1,1337
+0,6919
+1,3453
+0,2628
+1,937
+0,4163
+0,1122
+1,4052
+1,4569
+1,2126
+0,5726
+0,1784
+0,3989
+1,1389
+1,1517
+0,849
+0,2227
+1,5417
+1,5134
+0,2509
+0,2670
+0,1835
+0,5412
+1,5477
+1,2502
+1,1419
+1,2294
+0,2777
+0,4416
+0,2252
+1,8541
+1,350
+0,174
+1,3024
+0,2091
+0,3323
+1,304
+1,978
+0,1991
+0,6738
+0,1850
+1,3011
+1,1476
+1,2673
+0,827
+0,2167
+1,1596
+1,1865
+1,699
+0,5008
+1,4593
+0,4329
+1,2069
+0,7486
+0,3432
+1,3179
+0,6926
+1,7520
+1,2854
+1,411
+1,2467
+1,714
+0,3952
+1,987
+1,5425
+1,6176
+1,140
+0,6560
+0,1896
+1,2280
+0,3977
+1,1001
+1,2500
+0,9532
+0,4274
+0,764
+1,2698
+1,651
+0,4090
+1,2213
+0,1214
+1,524
+0,3533
+0,3705
+1,2459
+0,3304
+0,2100
+1,1335
+0,1765
+0,2061
+0,2866
+1,804
+0,1341
+0,1701
+0,685
+0,1521
+0,4628
+0,2277
+1,374
+1,492
+0,2746
+0,8448
+1,4734
+1,1115
+0,2876
+1,474
+0,1919
+1,1086
+1,5752
+1,1512
+1,1189
+0,2518
+1,2876
+0,5695
+0,500
+1,2154
+0,480
+1,5387
+1,432
+1,635
+1,3345
+0,1576
+1,2700
+0,705
+1,1159
+1,1520
+0,5677
+0,310
+0,820
+1,2071
+0,2120
+0,3764
+1,2466
+0,2365
+0,7824
+0,3906
+0,1231
+0,3534
+0,2131
+1,1281
+1,1149
+1,4062
+1,3099
+0,5432
+1,1752
+0,890
+0,2715
+0,10286
+0,967
+0,1280
+1,5299
+0,1167
+1,3647
+0,1330
+0,1925
+0,1189
+0,9016
+1,3119
+1,145
+0,1187
+1,3650
+0,1634
+1,2311
+0,767
+0,2747
+0,1660
+0,1266
+0,4576
+0,1337
+1,1809
+1,797
+1,981
+0,3303
+1,1289
+0,917
+1,1099
+1,2142
+0,3338
+0,5660
+1,2361
+0,826
+1,387
+1,3652
+1,137
+1,5275
+1,1233
+1,2139
+0,883
+1,9260
+1,1824
+1,2271
+1,2338
+1,146
+1,5306
+1,603
+1,681
+0,2289
+0,1792
+0,1180
+1,1147
+1,2735
+0,2394
+0,884
+0,3277
+0,1762
+1,1808
+0,2747
+1,3283
+1,1257
+0,2000
+1,2706
+0,9771
+1,4016
+1,195
+1,2390
+0,3483
+0,1136
+1,8406
+0,1513
+0,968
+0,3068
+1,1510
+1,652
+1,2940
+1,1877
+1,3048
+1,2218
+0,1440
+1,5959
+0,1427
+1,5202
+0,2277
+1,993
+1,750
+1,2438
+0,1637
+0,1008
+1,4697
+1,1423
+0,894
+0,494
+0,6946
+0,2106
+1,1674
+1,249
+1,532
+1,2109
+1,2285
+1,2579
+1,2887
+0,4631
+1,2460
+1,1590
+0,2659
+1,1006
+0,2217
+1,1363
+1,375
+1,4376
+0,1323
+0,4596
+1,7548
+1,3583
+0,1906
+0,7461
+1,1245
+1,1082
+0,2298
+0,865
+1,298
+1,4794
+0,630
+1,1859
+1,2540
+1,2548
+1,1554
+0,1981
+1,2922
+0,855
+1,5589
+1,4330
+0,3980
+0,3183
+1,1462
+0,2648
+0,2182
+0,2687
+1,6920
+0,856
+0,571
+1,832
+1,923
+0,2176
+1,2039
+1,246
+0,479
+0,1367
+0,2362
+1,4237
+0,481
+1,1585
+0,2339
+0,2959
+1,1188
+1,9257
+0,734
+0,2493
+0,2956
+1,1700
+0,1553
+1,1296
+1,2527
+0,1418
+0,663
+0,1104
+1,1675
+0,1112
+1,4202
+1,2269
+1,603
+1,2369
+0,1087
+0,642
+0,2462
+0,8804
+1,1184
+1,1676
+0,3111
+0,1544
+0,2537
+0,6766
+1,1836
+0,1476
+1,1511
+0,3943
+0,1233
+0,8193
+1,2506
+1,183
+1,1599
+0,4945
+0,5544
+1,979
+0,2262
+1,4228
+0,3501
+1,6594
+1,1920
+0,2177
+0,375
+0,802
+1,1260
+0,2385
+1,5404
+1,6998
+1,2806
+1,3365
+0,3814
+0,774
+1,1573
+0,2453
+1,2476
+0,3594
+1,1485
+1,1326
+1,1212
+0,1261
+0,3374
+1,2701
+1,6498
+0,4331
+1,1258
+0,2875
+1,4154
+1,3079
+0,2282
+0,3090
+0,8322
+1,2541
+0,2352
+0,2650
+1,851
+0,887
+0,1240
+0,1071
+1,2667
+1,2453
+0,1012
+0,7674
+0,2587
+1,3847
+1,4360
+1,1862
+1,1423
+1,3297
+0,1088
+0,3066
+1,544
+0,1054
+0,1169
+1,2231
+1,7613
+1,3929
+0,590
+1,757
+1,527
+0,1594
+0,1122
+0,883
+0,2036
+1,1188
+0,4157
+0,1098
+0,2967
+1,2044
+1,675
+1,4003
+0,1028
+0,384
+1,1487
+0,795
+1,1826
+0,2743
+0,3835
+0,1138
+0,1659
+1,2084
+1,6061
+0,2217
+1,1627
+0,9205
+0,942
+0,1981
+1,3146
+0,3953
+1,1989
+0,1249
+1,2135
+0,4622
+0,1463
+0,3941
+1,556
+1,2901
+1,1494
+1,3017
+1,623
+0,3219
+1,3134
+1,6903
+0,6620
+0,6831
+0,1859
+1,2776
+0,2106
+0,5743
+0,6046
+1,9773
+0,983
+0,5333
+1,6146
+1,463
+0,3636
+1,1510
+0,3782
+1,519
+1,2545
+0,3204
+0,736
+0,5439
+1,545
+1,1467
+0,335
+1,3109
+0,6456
+0,1478
+1,1794
+1,1841
+0,1437
+0,5076
+1,4955
+0,1670
+0,4846
+0,1553
+1,686
+0,1184
+1,5821
+0,2036
+0,504
+0,2096
+1,1146
+1,2769
+1,1576
+1,3451
+1,1475
+1,476
+0,1843
+0,689
+0,800
+1,4711
+0,1823
+1,3886
+0,1598
+1,1643
+1,505
+0,2596
+0,1687
+1,852
+1,1575
+0,277
+1,801
+1,2174
+0,1259
+1,2425
+1,1710
+1,5995
+0,779
+0,286
+1,1502
+1,1569
+0,1067
+1,1155
+1,1804
+1,953
+1,561
+1,2309
+0,501
+0,2047
+0,4093
+0,2211
+0,1186
+0,6776
+1,65
+1,1288
+0,1151
+0,3170
+1,1044
+1,3228
+1,2363
+1,2541
+0,1355
+1,3554
+0,1786
+1,592
+0,812
+0,1131
+0,5165
+1,5208
+0,1572
+1,5382
+0,6024
+1,1426
+0,4205
+0,481
+0,3672
+1,2504
+1,1784
+0,2529
+1,2961
+0,3562
+0,529
+0,3408
+0,5036
+1,795
+0,2951
+0,2021
+0,2138
+1,1204
+1,2301
+0,234
+1,518
+0,2761
+0,6439
+0,548
+0,7810
+1,1059
+0,1990
+1,5401
+1,3428
+0,1039
+0,6643
+0,2864
+1,269
+1,1184
+1,2517
+0,911
+0,2179
+1,1573
+1,1584
+0,2827
+1,2633
+1,5002
+1,440
+0,2106
+1,3844
+0,3748
+0,1463
+0,1032
+1,7599
+0,1424
+1,4680
+0,3640
+1,345
+0,1331
+1,4333
+1,1035
+1,2129
+1,3155
+1,3079
+0,7496
+1,1311
+1,1934
+1,3647
+1,686
+0,3132
+1,6470
+1,712
+1,964
+1,2788
+0,886
+0,2842
+0,472
+0,2369
+0,2900
+1,2982
+1,2089
+0,7939
+0,682
+1,2783
+0,3909
+0,6219
+0,480
+0,4839
+1,2178
+0,789
+1,1338
+0,2955
+1,131
+0,5691
+1,1149
+1,842
+0,1208
+0,3822
+0,696
+1,847
+1,907
+1,8102
+1,735
+0,459
+0,8333
+0,7238
+0,579
+1,5871
+0,1025
+1,1678
+1,2617
+0,4547
+0,1533
+0,3729
+1,1005
+1,1569
+0,3770
+0,556
+0,4457
+0,1361
+0,2743
+1,2384
+0,2578
+1,1458
+0,1123
+1,3511
+0,928
+0,893
+0,3112
+0,3015
+0,4377
+0,6481
+1,11920
+0,2770
+0,653
+0,1477
+0,695
+0,5392
+1,213
+1,2564
+0,2566
+0,1310
+1,2355
+1,443
+0,186
+1,1038
+1,1234
+0,4802
+1,486
+1,822
+0,297
+1,369
+1,2383
+0,1289
+1,558
+1,4196
+1,2651
+0,4304
+1,475
+1,3566
+0,3036
+1,2048
+0,2022
+1,1687
+1,1481
+0,1086
+0,1759
+0,314
+0,1336
+0,1651
+1,314
+0,691
+0,5371
+1,5708
+0,1131
+1,4296
+1,5849
+1,3933
+0,2562
+1,2455
+0,6810
+1,1321
+0,4509
+1,1070
+0,7202
+0,1943
+0,3947
+0,2206
+1,3593
+1,2213
+1,1339
+1,2675
+0,2192
+1,7304
+0,1033
+0,1966
+1,1124
+0,2329
+1,7715
+0,4474
+1,1661
+1,2754
+0,1855
+1,561
+1,492
+1,789
+1,5290
+1,6383
+0,2439
+1,1616
+0,4504
+0,1871
+0,4888
+0,887
+0,5412
+0,3146
+0,2024
+0,616
+1,533
+0,3416
+1,2244
+1,1113
+1,5058
+0,3009
+1,2236
+0,1574
+1,9814
+1,3008
+1,436
+0,3308
+0,758
+1,2034
+1,4595
+1,516
+1,3636
+0,321
+1,7131
+0,482
+1,1515
+1,1331
+1,986
+1,1380
+1,5846
+1,1004
+1,8108
+0,3940
+0,529
+0,3542
+1,1013
+0,1179
+1,2580
+0,1014
+0,2921
+0,7385
+1,2141
+0,1364
+1,2220
+1,1461
+0,317
+1,4166
+0,350
+0,1597
+0,2163
+1,1285
+1,1266
+0,4853
+1,3847
+1,1038
+1,233
+0,272
+1,812
+1,933
+0,4189
+0,1728
+1,1684
+0,3933
+0,513
+1,1104
+0,4809
+0,4432
+1,3175
+0,3963
+1,2589
+0,3427
+0,247
+1,1501
+1,2248
+1,852
+0,2941
+0,1563
+1,3217
+1,3724
+0,2808
+1,757
+1,1455
+1,4344
+1,9905
+0,4731
+1,3280
+1,2042
+1,1479
+1,3833
+0,1488
+0,1742
+1,939
+1,467
+0,1402
+0,807
+0,222
+1,1033
+0,3517
+1,379
+0,1886
+0,5245
+1,9321
+0,2430
+1,2905
+0,1458
+1,3531
+0,1150
+1,6864
+0,1926
+1,710
+0,3897
+1,6143
+0,693
+1,2392
+0,5478
+1,1216
+0,4097
+1,12281
+0,3183
+1,1418
+1,5422
+1,2472
+1,1281
+1,2112
+1,6407
+1,1060
+1,1520
+0,3559
+0,2088
+1,3678
+0,1077
+1,985
+0,9316
+1,4612
+1,4961
+1,1767
+0,1275
+1,5016
+0,1053
+0,1706
+0,3889
+0,1000
+1,667
+1,9161
+1,2341
+0,4750
+1,2658
+0,4101
+0,1879
+1,1455
+0,1705
+0,2487
+0,811
+0,2186
+0,6285
+1,1119
+1,667
+1,5492
+1,2504
+0,7340
+1,1024
+1,1612
+0,2352
+1,836
+0,2863
+1,5003
+0,1022
+1,2742
+0,1792
+0,1895
+0,510
+0,886
+0,1066
+0,919
+1,2261
+0,2744
+0,918
+1,1948
+1,2747
+0,2565
+0,3368
+1,8260
+1,4159
+0,678
+0,536
+0,6999
+1,8302
+1,3496
+0,1326
+0,1008
+0,1008
+0,1509
+0,323
+1,1497
+1,869
+0,3533
+0,152
+0,5314
+1,1162
+1,3774
+0,2976
+0,815
+0,2733
+1,193
+1,806
+0,1123
+0,817
+1,4135
+0,1194
+1,1359
+1,9340
+0,1850
+0,1094
+1,453
+1,894
+0,5331
+1,3619
+0,8116
+0,775
+1,912
+0,1140
+0,5790
+0,1603
+1,4773
+0,2398
+1,499
+0,2548
+0,3266
+0,3410
+1,3355
+1,9886
+0,4057
+0,3056
+0,2705
+0,1133
+0,2237
+0,4389
+1,761
+1,7978
+1,5317
+1,4976
+0,3808
+1,2387
+0,5988
+1,2626
+0,5473
+0,4809
+0,1597
+1,1127
+0,3296
+1,615
+0,7551
+1,3727
+1,1556
+0,5671
+1,650
+0,549
+1,3040
+1,5769
+0,233
+0,6334
+1,1672
+0,1861
+0,282
+0,2976
+0,1486
+1,1263
+0,508
+1,1298
+1,1668
+1,1501
+1,540
+0,903
+0,6998
+0,1479
+1,236
+1,336
+1,2911
+1,544
+0,3459
+1,3736
+1,6478
+0,910
+0,3587
+0,1643
+1,1394
+0,1758
+0,789
+1,1205
+0,4364
+1,1805
+0,1626
+0,923
+0,4579
+1,2152
+1,1545
+1,629
+1,1894
+1,3478
+1,1636
+1,1151
+0,5184
+0,1259
+0,2523
+1,2164
+1,2134
+0,2494
+0,1095
+0,452
+0,1874
+1,1231
+1,1097
+1,1317
+1,550
+0,530
+1,1898
+1,241
+0,1841
+0,609
+0,6959
+1,2473
+0,1427
+0,1134
+0,884
+0,2007
+0,1413
+1,7973
+1,5023
+0,1128
+0,1823
+0,446
+0,3437
+1,386
+0,5830
+0,2937
+0,1531
+0,3955
+1,1215
+1,1167
+1,1197
+0,1980
+1,2797
+1,2178
+1,1184
+0,7241
+0,261
+1,2296
+0,4957
+0,2177
+1,1552
+0,335
+1,854
+0,1359
+1,370
+0,1618
+0,1496
+0,7724
+1,3898
+0,494
+0,3134
+0,2573
+0,1206
+0,2595
+1,1254
+1,2024
+0,1291
+1,1285
+1,3092
+0,1861
+0,2995
+0,1723
+1,2970
+1,1153
+0,551
+1,239
+1,1097
+0,1103
+1,411
+0,4721
+1,3154
+1,2955
+0,8203
+0,1679
+0,8270
+1,3921
+0,574
+0,796
+1,6644
+0,5610
+1,1124
+1,2965
+1,904
+0,1108
+1,2142
+0,1755
+1,3239
+1,1879
+0,1469
+0,1867
+1,699
+0,1305
+0,5639
+1,2574
+0,4109
+1,2656
+0,2049
+0,1257
+1,1462
+1,250
+1,1092
+0,5513
+0,4559
+1,2377
+1,1250
+0,966
+0,1237
+1,11010
+1,1547
+1,1681
+1,1849
+1,1083
+1,1502
+1,2709
+0,4245
+1,7056
+1,4866
+1,1347
+1,905
+0,8712
+0,3859
+0,880
+1,1711
+0,4863
+0,3895
+0,271
+0,673
+0,3318
+1,556
+1,5978
+1,1552
+1,1989
+0,1033
+0,5257
+1,4312
+1,2533
+1,390
+1,2334
+0,930
+1,1503
+0,724
+1,7177
+0,4423
+1,691
+0,2204
+1,1434
+1,6706
+0,1791
+0,5298
+0,3123
+1,1115
+0,4953
+1,2375
+1,2143
+1,906
+0,2157
+0,7263
+0,3711
+1,1878
+1,2978
+1,1217
+0,3229
+0,5112
+1,7571
+1,3819
+1,1805
+0,3997
+0,4470
+1,2187
+0,1237
+1,1020
+0,953
+1,935
+0,918
+1,811
+1,2635
+0,1600
+0,1213
+0,2080
+1,1334
+0,1664
+0,1735
+0,1475
+0,1923
+0,2764
+1,3198
+0,2627
+0,1831
+0,4017
+1,580
+0,517
+0,1418
+0,1624
+0,857
+1,951
+0,924
+0,2712
+1,637
+1,1680
+1,1390
+0,4996
+0,1178
+1,7820
+1,2549
+1,3248
+0,1573
+0,1569
+1,1410
+1,8071
+0,990
+0,3806
+0,12073
+1,9101
+1,1542
+0,693
+1,811
+1,3963
+1,2365
+1,6075
+1,4691
+1,3124
+1,1690
+0,1782
+1,209
+1,1919
+1,1497
+1,3021
+1,2049
+0,723
+0,3905
+0,1778
+1,4682
+0,2983
+0,336
+0,4144
+1,1462
+1,1156
+1,3109
+0,3313
+1,4316
+0,7726
+0,1633
+1,4375
+0,1364
+0,1116
+0,833
+0,4482
+1,753
+1,2608
+0,1736
+1,2461
+1,154
+1,935
+1,189
+1,5309
+0,1642
+0,7775
+0,4894
+1,3816
+0,3835
+1,2051
+0,738
+0,346
+0,2143
+1,3456
+0,1536
+0,2505
+0,655
+1,5246
+0,466
+0,5420
+1,3624
+1,856
+0,1743
+0,2750
+0,3844
+0,2287
+1,444
+0,4979
+0,1439
+1,917
+1,381
+0,623
+0,1644
+0,3338
+0,1589
+1,4143
+0,2302
+0,7416
+0,2808
+0,1538
+1,2510
+0,1287
+0,2119
+1,1103
+1,2843
+0,964
+1,676
+0,2864
+1,5643
+1,2503
+1,276
+1,5039
+0,1206
+1,727
+1,1596
+0,2212
+1,387
+1,1579
+0,2708
+1,2259
+0,4083
+0,1690
+1,1416
+0,685
+0,8542
+0,2143
+0,4708
+1,2094
+1,10097
+0,2126
+0,3506
+1,4387
+0,2874
+1,4017
+0,1562
+1,614
+1,2036
+1,1067
+1,203
+1,7029
+1,2930
+1,3338
+0,1904
+1,361
+1,785
+0,2025
+0,3079
+0,1018
+0,1377
+1,180
+1,3637
+0,1446
+0,871
+1,1314
+0,6274
+1,1730
+1,3873
+0,2214
+0,2992
+0,1614
+1,587
+1,3622
+0,3685
+0,2198
+1,5361
+1,2461
+1,3470
+1,452
+1,3166
+0,1645
+0,1681
+1,1201
+1,1495
+0,819
+0,1881
+1,3168
+1,743
+1,3457
+1,1125
+0,6521
+0,577
+1,1711
+1,521
+1,2987
+1,1168
+1,1824
+1,361
+1,2625
+1,425
+0,1582
+1,6820
+0,668
+0,2003
+1,1002
+1,879
+0,1008
+1,1264
+1,6233
+0,1448
+1,238
+0,2940
+0,4943
+0,559
+1,3975
+1,1287
+1,3056
+0,2760
+1,1571
+0,961
+1,2219
+0,5176
+1,1291
+0,1680
+1,567
+1,1482
+1,2817
+0,1763
+1,495
+0,5388
+0,6343
+0,2244
+1,7048
+0,1411
+0,4576
+1,8348
+1,1830
+1,1694
+0,2399
+1,1113
+1,2922
+1,4418
+0,135
+0,780
+0,906
+0,1885
+1,1516
+0,5490
+0,2670
+1,1508
+0,1200
+1,1999
+1,3916
+0,1328
+1,1670
+0,2038
+0,2577
+0,1475
+0,2871
+1,611
+1,3482
+0,1707
+1,1929
+0,1799
+1,3339
+0,651
+1,2965
+1,7970
+0,1880
+1,2898
+1,267
+1,3170
+1,7790
+0,3939
+0,1171
+1,1277
+1,1700
+0,1227
+1,1410
+0,9018
+0,4730
+1,1206
+1,1211
+1,1575
+0,2921
+0,1141
+0,877
+1,4151
+0,386
+0,3048
+0,7041
+0,513
+1,1907
+1,202
+1,5734
+0,503
+0,8396
+0,1366
+0,789
+0,338
+1,1207
+1,3348
+0,1907
+0,3570
+1,1122
+1,2382
+0,6092
+0,391
+1,2269
+1,5454
+1,1858
+1,633
+0,1054
+1,1222
+0,6290
+1,2776
+0,1713
+1,1832
+0,1067
+0,4612
+1,2108
+1,5946
+1,2422
+0,5475
+1,1532
+1,2339
+1,3789
+0,1507
+0,793
+1,183
+1,6193
+0,940
+0,2030
+0,2463
+1,5009
+1,654
+1,2008
+0,1118
+1,2669
+0,2092
+1,480
+0,261
+0,372
+1,1364
+0,417
+1,5234
+1,304
+0,6602
+1,1331
+0,986
+1,2887
+1,1057
+1,1996
+0,1824
+0,1785
+1,1976
+1,3654
+0,1886
+1,1946
+0,3435
+1,1384
+1,488
+1,879
+1,2141
+1,471
+1,2117
+0,14301
+0,1814
+0,1768
+0,294
+1,3727
+0,2040
+0,1517
+1,1570
+1,1435
+0,1888
+1,1521
+1,5872
+0,1723
+1,3810
+0,2445
+0,1587
+0,1592
+0,3638
+0,6282
+1,3362
+0,1229
+1,5998
+0,2415
+1,1271
+0,346
+0,2844
+0,707
+1,3476
+1,2153
+0,2475
+0,3012
+1,573
+1,927
+1,7032
+0,1147
+0,1684
+1,844
+0,7570
+0,3321
+0,5007
+0,655
+0,2697
+0,4452
+1,1909
+0,4148
+0,4426
+0,5091
+1,2690
+0,297
+1,1336
+0,1018
+1,1242
+0,691
+1,590
+0,9186
+1,4850
+0,1380
+1,1176
+1,3234
+0,1463
+0,6309
+1,4339
+1,2573
+1,3303
+1,962
+0,2874
+0,5525
+1,2416
+0,2213
+0,1412
+0,1995
+1,2263
+1,1798
+0,2993
+1,321
+0,2350
+1,4304
+0,6870
+0,7864
+0,2345
+0,1710
+1,1392
+1,3820
+1,5694
+1,3192
+0,2581
+0,1580
+1,1109
+1,2735
+1,3109
+0,917
+1,6526
+1,3779
+1,1476
+0,590
+0,1183
+0,2174
+0,721
+0,1351
+0,1226
+0,110
+0,2821
+0,4013
+0,1813
+0,4494
+0,6218
+1,2911
+0,2511
+1,380
+0,1094
+0,2111
+1,1069
+1,1924
+0,2017
+1,2013
+0,4842
+0,5594
+0,5899
+0,2304
+1,1011
+1,2521
+0,1722
+0,1407
+0,1864
+1,1432
+0,773
+1,1693
+0,2540
+0,4626
+0,6311
+0,1772
+0,2004
+1,931
+0,3825
+1,6006
+1,7779
+0,3532
+0,1118
+1,1860
+1,5732
+1,1123
+0,1024
+0,3100
+1,1499
+1,1611
+0,2159
+0,2176
+0,5779
+0,660
+1,568
+0,1048
+0,1819
+1,3913
+0,7358
+0,2869
+1,5736
+1,1456
+1,4707
+0,528
+0,433
+1,500
+1,5744
+1,4581
+0,1913
+1,951
+0,4989
+1,1022
+1,939
+1,6027
+1,1781
+1,1239
+1,1357
+0,2107
+1,1724
+0,1679
+1,665
+1,1319
+1,649
+1,2380
+1,734
+1,1048
+1,987
+0,3409
+0,1339
+1,3092
+0,450
+0,1579
+0,789
+1,909
+0,53
+1,2182
+1,1731
+0,1480
+0,1229
+1,1314
+1,875
+0,1197
+1,2285
+1,952
+1,6291
+1,1675
+0,2670
+1,2986
+0,1931
+1,438
+1,1042
+0,2028
+0,2786
+1,1089
+0,393
+0,4974
+0,2668
+1,5806
+1,6121
+0,4885
+1,1301
+1,616
+0,1999
+1,3678
+1,674
+1,1227
+0,2212
+0,1236
+0,2418
+1,5117
+1,3815
+1,1524
+1,114
+1,2592
+1,1835
+1,3079
+0,4059
+1,3794
+0,705
+1,400
+0,5093
+0,993
+1,2055
+0,1382
+0,1381
+1,3470
+1,871
+1,1882
+0,989
+1,209
+1,3579
+0,881
+0,1802
+1,1525
+1,4235
+0,1706
+1,9126
+0,2787
+0,2560
+1,851
+1,5818
+0,1880
+1,6185
+0,5797
+1,616
+1,3701
+1,608
+1,3151
+1,5571
+1,16372
+1,4941
+1,542
+0,3215
+0,2919
+0,3509
+1,796
+1,366
+1,2049
+1,6000
+0,4085
+1,1001
+0,810
+0,1546
+0,2272
+1,2480
+1,1065
+0,1560
+1,1756
+0,4396
+0,2055
+1,1407
+1,1095
+0,820
+0,2483
+1,996
+1,2403
+0,2092
+0,4765
+1,4262
+0,2054
+1,3610
+0,1161
+1,1211
+1,2194
+1,1821
+0,720
+1,1481
+1,5691
+0,1226
+1,1301
+0,1333
+0,533
+1,1036
+0,331
+0,1400
+1,888
+0,2230
+1,2153
+1,582
+1,4701
+1,2242
+1,2037
+0,3438
+1,3741
+1,4645
+0,6660
+1,1388
+0,4843
+1,2790
+0,6067
+0,896
+1,6310
+0,1395
+1,4355
+0,5586
+0,548
+0,8173
+1,1435
+0,2093
+0,569
+1,1908
+1,1778
+1,6605
+0,5033
+1,1398
+1,783
+0,4842
+0,2118
+1,1668
+0,8575
+1,3242
+0,2632
+1,1312
+0,2774
+1,2285
+0,1235
+0,712
+0,1491
+1,3242
+1,525
+0,2045
+0,3088
+1,6371
+1,3107
+0,2059
+1,1794
+0,2479
+0,2297
+0,6719
+0,6655
+1,4162
+1,7840
+0,5141
+0,4237
+1,2201
+0,347
+1,962
+0,2076
+0,194
+0,1365
+0,1843
+0,2869
+0,762
+1,3475
+0,2799
+0,2580
+0,658
+0,1094
+1,6715
+0,3599
+0,1737
+1,6171
+1,2292
+1,2310
+1,5472
+0,1983
+1,2573
+0,8844
+0,1527
+0,1423
+1,943
+0,1918
+0,2292
+0,1990
+1,4420
+1,693
+1,682
+1,1970
+0,2003
+1,2821
+1,1631
+1,754
+0,1559
+1,1952
+0,669
+1,6106
+0,2852
+1,1924
+1,1380
+1,613
+0,3308
+0,1383
+0,5794
+0,424
+1,11266
+0,1545
+1,1312
+0,1135
+1,2881
+1,752
+1,993
+1,2125
+0,3905
+0,1777
+0,1377
+1,1949
+0,5709
+1,784
+1,2096
+0,8129
+1,710
+1,3400
+1,384
+0,2209
+1,8151
+0,2396
+0,1105
+1,2057
+0,1621
+0,1146
+1,1407
+0,2265
+1,2527
+1,4714
+0,1777
+0,640
+0,5189
+1,2079
+0,759
+0,2397
+1,2391
+0,1388
+1,1477
+0,6252
+1,2178
+1,5235
+0,1464
+1,2884
+0,2535
+1,2573
+0,5928
+1,1907
+0,398
+1,213
+1,12487
+1,1569
+0,1706
+1,1426
+0,1783
+0,2209
+0,1024
+1,1825
+1,834
+1,1458
+1,2526
+0,2022
+0,4401
+1,570
+0,2536
+1,6576
+1,1178
+1,1001
+1,1776
+0,1928
+0,369
+1,1490
+1,2903
+0,2600
+1,621
+0,764
+1,521
+1,235
+0,3185
+0,2915
+0,1336
+1,1103
+1,873
+1,1876
+1,292
+1,189
+0,5058
+0,2693
+0,2138
+0,442
+1,1764
+0,854
+1,744
+0,8595
+0,2342
+0,2117
+0,1672
+0,3018
+1,388
+0,1878
+1,3514
+0,1436
+1,3620
+0,1625
+0,2449
+1,1627
+0,1618
+1,1150
+1,2382
+0,5884
+1,1509
+0,952
+0,2132
+1,9126
+1,784
+0,952
+0,4588
+0,2623
+0,1258
+0,2408
+0,965
+1,1419
+1,1261
+0,6264
+1,1011
+1,1627
+0,1777
+0,5844
+1,1110
+0,636
+0,1422
+1,1379
+0,1704
+0,4481
+1,6987
+0,116
+0,2917
+1,3752
+0,2045
+0,1258
+1,1747
+0,1059
+0,4888
+0,1641
+0,3680
+0,2268
+1,1424
+1,3751
+0,3574
+1,341
+1,1435
+1,4171
+0,1163
+0,4513
+1,1747
+0,4016
+0,4522
+0,1116
+0,5698
+0,5679
+1,1747
+0,971
+0,4500
+1,450
+0,1733
+0,2411
+1,651
+0,1976
+1,798
+0,3509
+1,1954
+1,5617
+1,1657
+1,2205
+0,2502
+1,1772
+1,1119
+0,4129
+1,2519
+0,6414
+0,1882
+0,1947
+1,1873
+0,409
+0,373
+1,5180
+0,4989
+0,2263
+0,5357
+0,1818
+1,7719
+0,2659
+1,768
+1,1897
+0,215
+1,1660
+1,1068
+0,1206
+1,1599
+1,3605
+0,963
+1,2334
+0,162
+1,4589
+0,554
+1,1930
+0,4062
+0,7948
+1,2051
+1,709
+0,3128
+1,1196
+0,1127
+0,599
+1,1763
+1,3637
+1,3622
+1,757
+1,3148
+0,5745
+0,3287
+0,4333
+1,566
+0,5735
+1,4801
+0,594
+0,4462
+0,621
+0,2509
+1,1306
+0,777
+1,1038
+1,2804
+0,3338
+0,7169
+0,1230
+0,604
+0,354
+0,3653
+1,3236
+1,4113
+1,1993
+1,404
+1,3738
+0,1603
+0,1647
+1,2492
+1,806
+1,2414
+1,1034
+1,2015
+1,964
+1,2822
+0,1448
+1,1683
+0,1470
+0,282
+1,1993
+1,653
+1,999
+0,1090
+0,146
+1,1261
+0,4584
+0,5834
+0,1641
+1,2806
+0,2891
+1,1983
+0,9055
+0,536
+0,1750
+1,260
+0,3035
+1,2232
+0,526
+0,1263
+1,1263
+1,2549
+1,3463
+1,7148
+0,1562
+1,2919
+0,3913
+1,825
+1,2468
+0,3418
+0,1374
+1,1853
+1,5490
+0,2701
+1,1771
+1,332
+0,3285
+1,2195
+1,3271
+0,5358
+0,1103
+0,2732
+0,1122
+0,821
+0,2670
+1,4502
+1,786
+1,1668
+1,1564
+1,357
+1,1226
+0,1141
+1,2200
+0,6666
+0,9092
+1,5977
+0,1295
+1,1428
+1,1640
+0,1128
+0,9046
+0,1250
+0,3848
+0,2164
+0,2950
+0,1415
+0,610
+0,1901
+0,3575
+1,1372
+1,1371
+1,141
+0,2489
+0,1721
+1,5934
+0,7806
+0,7351
+0,1083
+1,2268
+0,2859
+1,1477
+1,2840
+0,3575
+1,4401
+1,1499
+1,1221
+1,959
+1,1470
+1,521
+0,2422
+1,485
+0,1336
+0,4435
+0,9490
+1,2565
+0,4518
+1,1801
+0,1428
+1,6758
+0,1367
+1,766
+1,3192
+0,1904
+1,1556
+0,1091
+0,969
+1,9180
+0,1410
+1,2262
+1,4701
+0,3868
+1,8152
+1,1127
+1,5304
+1,1330
+0,4737
+0,5941
+1,1557
+0,2680
+1,1989
+1,236
+0,1806
+0,1383
+0,712
+0,2905
+0,1869
+0,1545
+0,924
+1,646
+0,6740
+1,2333
+1,5839
+1,1827
+0,1760
+1,2467
+1,4823
+0,682
+0,3080
+0,1053
+0,4021
+0,1770
+0,2701
+1,3595
+0,2338
+1,1616
+1,807
+1,547
+1,6285
+0,5580
+0,857
+1,1585
+1,902
+1,1683
+0,4081
+0,1726
+1,7896
+0,3031
+1,407
+0,1824
+1,4788
+0,4432
+0,2543
+0,2732
+1,399
+1,4552
+0,897
+0,8804
+1,6048
+0,1251
+0,2650
+0,2114
+0,1922
+1,2053
+1,3586
+1,3997
+0,5955
+1,4920
+0,2825
+1,2704
+0,1488
+0,2910
+1,2454
+0,1488
+1,271
+1,7422
+0,1493
+1,6743
+0,3651
+0,1474
+0,3015
+1,1698
+0,1981
+0,1877
+0,2477
+1,419
+1,718
+0,6328
+1,3566
+1,694
+1,7331
+0,2096
+0,5293
+0,1597
+1,5965
+0,5320
+0,4590
+1,763
+0,1548
+0,817
+1,2770
+0,2950
+1,1393
+0,1428
+0,3768
+1,1775
+1,2649
+0,142
+1,1270
+0,6547
+1,2339
+0,7589
+0,1001
+0,218
+0,5661
+1,2474
+0,6488
+0,1656
+0,3716
+1,869
+0,1758
+0,8598
+1,5474
+1,302
+0,2030
+0,7535
+0,546
+1,1315
+0,2432
+0,2369
+0,3464
+1,2538
+1,416
+0,1447
+1,705
+0,1172
+0,2749
+0,1087
+1,1766
+0,1461
+0,1166
+1,3337
+0,733
+0,2021
+0,1401
+1,1819
+1,890
+0,1528
+1,481
+0,2966
+0,897
+1,3558
+1,447
+0,13226
+1,1131
+0,4903
+1,855
+0,1798
+1,2140
+1,2662
+0,1913
+0,1134
+1,3316
+1,626
+0,1191
+1,2693
+0,885
+1,1471
+1,1884
+0,6966
+1,3138
+1,4335
+0,3927
+1,8567
+0,1044
+0,1138
+0,3115
+0,2428
+1,1758
+0,2030
+1,1356
+1,4477
+1,7822
+1,2082
+0,189
+1,927
+0,1323
+1,1287
+1,338
+1,1415
+1,453
+1,1786
+0,2509
+0,4020
+0,622
+0,56
+1,2847
+1,1129
+1,3039
+0,1041
+1,1805
+1,918
+0,2613
+0,849
+1,1749
+0,2956
+0,4880
+0,1280
+0,4881
+0,2335
+0,1132
+0,2582
+1,2302
+1,758
+1,4693
+0,639
+1,259
+0,1760
+1,1585
+0,3531
+0,6809
+1,1075
+0,3862
+0,124
+0,746
+0,6324
+0,3777
+0,4744
+0,7356
+1,7286
+0,6535
+1,1367
+1,5120
+0,1781
+0,393
+0,6453
+1,5940
+1,1188
+0,5600
+1,4484
+1,1720
+1,3937
+1,3105
+1,805
+1,2829
+0,1670
+1,954
+1,3482
+0,583
+0,1262
+0,1697
+1,1873
+0,770
+0,1407
+0,2096
+1,904
+1,4435
+1,2706
+0,5346
+0,1417
+0,2166
+0,2969
+0,1581
+0,1760
+0,6968
+0,1615
+1,944
+0,1574
+1,2568
+1,2607
+1,3733
+1,1411
+1,9783
+1,2811
+0,1142
+0,1695
+0,375
+1,1839
+1,3728
+0,8045
+1,3066
+0,4684
+0,3305
+0,3197
+1,3413
+1,5138
+0,7212
+1,1104
+0,2663
+0,1595
+0,1641
+0,2789
+1,3966
+0,2696
+0,2582
+0,1130
+1,3356
+0,1958
+0,1397
+1,975
+0,608
+0,756
+1,2109
+1,1405
+1,4329
+0,5205
+0,1961
+1,2519
+0,4215
+0,108
+0,2896
+1,2297
+1,792
+0,871
+0,4161
+1,4845
+0,870
+1,2052
+0,2282
+1,894
+0,855
+1,1747
+0,750
+1,311
+0,312
+1,440
+0,2380
+1,1908
+1,1914
+0,7091
+1,3390
+1,1018
+0,8048
+1,5123
+0,2480
+0,1513
+0,4183
+1,2920
+0,2067
+1,4175
+1,1757
+0,6133
+1,749
+1,5719
+0,6799
+1,7564
+1,3030
+1,2637
+0,1016
+0,1920
+0,4319
+0,1521
+1,2913
+1,2820
+0,3929
+0,867
+1,3146
+0,6157
+1,7419
+1,554
+0,409
+1,1028
+1,613
+0,578
+1,774
+1,2036
+1,2346
+1,671
+1,6458
+0,2069
+1,335
+0,2151
+1,1083
+1,1171
+1,895
+1,558
+1,1626
+0,8215
+1,1866
+0,5030
+1,8283
+1,5842
+0,1737
+1,928
+0,3297
+1,790
+1,1023
+1,690
+0,1851
+1,5321
+0,2677
+1,853
+0,2951
+0,1084
+0,889
+1,995
+0,1318
+0,6942
+0,689
+0,1774
+1,1018
+1,2936
+1,1260
+1,1353
+1,973
+0,2364
+0,2954
+1,2013
+0,463
+1,3267
+0,7149
+1,892
+0,1315
+1,2872
+0,3310
+0,3059
+1,1731
+1,5929
+1,1014
+1,5231
+1,1635
+0,401
+0,3741
+1,3729
+1,1091
+1,3455
+1,262
+1,1847
+0,10122
+1,5993
+1,1176
+1,4351
+0,1604
+0,2756
+0,4576
+1,2623
+1,2148
+0,3249
+0,2943
+0,1749
+0,2754
+0,677
+1,1673
+1,4741
+1,1912
+1,1964
+1,3831
+1,1645
+0,1390
+1,986
+1,218
+0,7067
+1,1333
+1,5246
+0,3381
+0,6928
+1,5459
+1,1211
+1,1816
+1,1260
+0,3377
+0,5809
+1,6644
+1,2077
+0,5462
+1,761
+0,4575
+0,6990
+1,7927
+1,169
+0,1321
+1,4614
+1,988
+1,832
+0,2662
+1,1008
+0,982
+0,741
+0,760
+1,2434
+0,1001
+0,2905
+1,1154
+0,5368
+0,3562
+1,1060
+1,3782
+0,4322
+0,4128
+0,3124
+0,1940
+0,5032
+1,669
+1,712
+1,5384
+1,2489
+1,2203
+0,6316
+0,2276
+1,4822
+0,763
+0,10779
+1,1435
+1,1401
+0,3546
+0,1513
+0,3003
+0,4144
+1,2341
+0,1742
+0,1699
+0,961
+0,1366
+1,3218
+0,1758
+0,1944
+0,6899
+1,4763
+0,2213
+0,783
+0,516
+1,2419
+0,741
+0,1747
+1,1549
+1,3586
+0,1664
+0,713
+0,2041
+1,2575
+0,8585
+1,2458
+0,4728
+1,1053
+0,739
+1,420
+1,6144
+0,639
+0,3492
+0,2884
+1,1858
+0,8579
+0,1205
+0,6306
+1,1789
+0,2326
+1,471
+1,1617
+1,498
+0,1398
+0,729
+1,504
+1,7184
+1,6585
+0,1531
+0,3535
+0,565
+1,150
+0,9206
+0,2806
+0,3626
+1,3459
+0,699
+0,1714
+1,3548
+0,1420
+1,1128
+0,4011
+0,3135
+0,3939
+1,1249
+1,1489
+1,5075
+0,1611
+0,605
+0,1032
+0,1073
+0,1080
+0,1336
+0,1844
+1,2933
+0,5122
+1,5383
+1,822
+1,1591
+1,2241
+0,7588
+0,1264
+0,2903
+1,445
+0,495
+0,1720
+1,3610
+1,859
+1,2237
+1,2127
+0,4598
+0,986
+0,622
+1,1326
+1,6660
+1,680
+0,3159
+1,2380
+1,820
+0,670
+1,539
+0,4711
+0,3509
+1,1423
+0,4294
+1,2613
+0,3896
+1,1938
+0,2181
+0,1065
+1,998
+0,5655
+1,5046
+0,773
+0,6868
+0,1755
+1,1824
+0,4361
+1,1512
+1,1334
+0,1052
+0,1253
+1,1809
+1,6005
+0,1816
+0,447
+0,6244
+0,2856
+1,1422
+0,3718
+0,1026
+1,1378
+1,1572
+1,1280
+1,6175
+0,797
+0,6267
+1,765
+0,4105
+1,3205
+0,3521
+0,5594
+0,4640
+0,6476
+1,2650
+0,3913
+1,1666
+1,1332
+0,1811
+0,5999
+1,1767
+1,1591
+1,1172
+1,6175
+1,1879
+1,1178
+1,761
+1,4188
+0,2343
+0,1288
+0,2822
+0,1129
+0,1635
+0,1132
+0,2074
+0,4394
+0,1786
+0,3161
+1,3807
+0,1190
+0,2442
+1,1245
+0,1488
+1,2161
+0,5518
+1,1811
+0,2117
+0,2922
+1,467
+1,2175
+1,1162
+1,202
+1,1837
+0,1144
+1,613
+1,858
+1,2497
+0,3857
+1,1570
+0,1544
+1,4701
+0,3220
+0,2315
+0,7701
+1,1584
+1,322
+0,3522
+1,5460
+0,1006
+0,920
+1,1038
+0,5117
+1,4600
+0,2429
+0,1465
+1,3368
+1,2727
+1,3024
+1,7255
+1,2302
+1,298
+0,2670
+1,323
+1,4950
+1,5367
+1,14325
+0,2599
+0,5551
+0,2652
+0,361
+0,1489
+0,498
+1,8997
+0,2691
+0,1286
+0,7869
+1,4092
+1,194
+0,803
+1,1957
+1,9846
+1,1202
+0,892
+0,9257
+1,3638
+0,2880
+0,4307
+0,2532
+1,985
+0,1705
+0,911
+0,551
+1,471
+0,2624
+0,2200
+0,5016
+1,6399
+0,465
+0,1729
+0,2524
+1,984
+1,3459
+1,1229
+1,2016
+1,6075
+1,1904
+1,1809
+0,1829
+0,1545
+0,1633
+1,2400
+1,1494
+0,4831
+1,998
+0,1454
+0,807
+0,1994
+1,2933
+0,1234
+0,1623
+1,2132
+1,4612
+0,2132
+1,3408
+1,1375
+1,2635
+0,6313
+1,9761
+1,2250
+0,5094
+1,2340
+0,2585
+0,2802
+1,954
+0,1217
+1,1225
+1,4459
+0,2308
+1,826
+1,1464
+0,5811
+0,10467
+0,4748
+0,1649
+1,5488
+0,2489
+0,2193
+0,5644
+1,2341
+1,2330
+0,1222
+0,348
+1,2649
+0,3016
+1,3336
+1,1608
+0,612
+0,81
+1,5798
+1,2162
+0,4830
+1,308
+1,478
+0,1833
+1,5935
+0,3200
+0,2264
+0,1018
+1,534
+1,170
+0,5474
+0,6976
+1,1538
+1,2095
+1,2398
+1,281
+1,2798
+0,2170
+0,7772
+1,4541
+1,550
+1,6180
+1,1875
+1,2796
+0,139
+0,1005
+0,2926
+1,1311
+0,2107
+0,2296
+1,6254
+1,3245
+0,4041
+0,1315
+1,2375
+1,5088
+1,2276
+1,2168
+1,5740
+1,598
+1,4921
+0,239
+0,480
+0,1991
+1,3114
+1,5121
+0,670
+0,1672
+1,1793
+0,3510
+0,698
+1,3636
+0,1342
+0,4176
+0,3567
+1,6580
+1,1677
+0,132
+1,1789
+0,741
+1,1937
+0,1480
+0,621
+1,1300
+0,3950
+0,2826
+1,3148
+0,5397
+1,1313
+1,1349
+1,5964
+0,2185
+1,629
+1,3473
+0,486
+0,6370
+0,4034
+1,4762
+1,586
+0,2924
+0,4968
+0,2738
+1,2279
+1,4353
+1,683
+1,2726
+0,864
+1,3625
+0,370
+0,7766
+0,968
+1,595
+1,2428
+1,766
+0,1086
+1,580
+1,593
+1,1406
+0,2372
+0,2105
+0,1000
+1,6267
+0,1666
+1,5490
+0,1063
+0,2118
+1,2239
+1,282
+1,1075
+0,2294
+1,6147
+1,1164
+0,1192
+1,5491
+1,724
+1,2775
+1,4423
+0,526
+1,3615
+0,2895
+0,2709
+0,1690
+1,2857
+1,4181
+1,1113
+1,3717
+0,771
+0,8166
+0,2290
+1,1587
+1,1233
+0,412
+0,1589
+0,689
+1,793
+0,812
+1,1477
+0,1218
+1,885
+0,1561
+1,1108
+1,1545
+1,800
+0,2696
+0,424
+0,2544
+0,1072
+1,179
+0,947
+1,1591
+0,2466
+0,2538
+0,579
+1,1756
+1,3479
+0,3984
+0,821
+1,5188
+0,859
+0,3954
+1,2883
+1,1692
+1,542
+1,638
+0,704
+1,245
+1,3092
+1,1805
+1,6345
+0,2099
+0,355
+1,2394
+0,1834
+1,1451
+1,1212
+1,1654
+0,1778
+1,3408
+0,951
+1,753
+1,1960
+0,1234
+0,1057
+0,2736
+1,896
+1,1324
+1,1466
+1,2025
+0,864
+1,1406
+0,1518
+0,618
+1,4974
+1,2201
+0,442
+1,2201
+0,2137
+0,1294
+1,4325
+0,2363
+0,3505
+0,1822
+0,859
+1,3462
+1,6625
+0,196
+0,1241
+1,1555
+0,721
+1,3478
+0,1100
+0,795
+0,2498
+0,1999
+0,1289
+0,3465
+1,1189
+0,4180
+0,5247
+0,2832
+1,3974
+0,3034
+1,4937
+0,2823
+0,1344
+0,1246
+1,4774
+1,2763
+0,10625
+0,720
+0,2840
+0,2612
+0,3621
+0,4363
+1,817
+0,3949
+0,540
+0,1864
+1,701
+0,3254
+0,2682
+1,4456
+0,6097
+1,1099
+0,5660
+0,6944
+1,1416
+0,4419
+0,2104
+1,1650
+0,2609
+0,1837
+0,4591
+0,1703
+1,3298
+1,355
+0,2092
+0,4137
+0,3033
+1,1097
+1,3089
+0,4611
+0,3476
+0,9278
+0,3901
+1,1687
+0,4065
+1,8796
+1,1161
+0,1018
+1,228
+0,1681
+0,2167
+0,2101
+0,1265
+1,1994
+0,7617
+0,920
+1,5123
+1,1252
+0,276
+1,10073
+0,1936
+0,803
+1,3933
+1,2197
+1,1290
+1,3122
+1,4068
+1,6361
+0,743
+1,1402
+1,3539
+0,1602
+1,2689
+0,1212
+0,528
+1,2005
+0,1960
+0,6441
+0,2972
+0,1407
+1,582
+0,4566
+1,7776
+1,365
+1,3268
+0,6780
+0,616
+0,2386
+1,4401
+1,974
+1,320
+1,5424
+1,486
+0,7159
+1,3970
+0,3113
+1,3548
+1,2901
+1,1759
+0,1768
+1,1161
+1,1495
+1,2388
+1,5818
+0,2220
+1,1307
+0,3480
+1,3389
+1,820
+1,906
+1,1080
+1,1708
+0,1125
+0,1642
+0,2370
+0,2366
+1,937
+1,1735
+1,4887
+1,822
+0,2201
+1,4697
+1,8518
+0,878
+0,1067
+0,1055
+1,2496
+1,2003
+0,5628
+1,3002
+0,1166
+1,1716
+1,992
+1,2181
+1,2227
+1,1314
+1,2072
+1,4618
+1,9415
+1,8039
+1,1322
+1,3270
+1,648
+1,1262
+0,9851
+1,529
+0,490
+0,499
+1,4862
+1,4301
+1,2532
+1,2855
+1,7373
+0,3066
+0,2243
+1,3510
+0,1160
+0,3732
+0,2172
+0,2921
+1,2246
+1,406
+1,463
+1,1341
+1,999
+0,1774
+0,1815
+0,1138
+1,4505
+0,4232
+0,601
+0,1044
+0,183
+1,3335
+1,3075
+0,645
+0,9627
+1,1532
+0,3225
+1,3784
+1,1501
+1,4754
+0,9078
+1,1027
+1,1352
+0,349
+0,1226
+0,1621
+1,8660
+0,2429
+1,4129
+1,4125
+1,7158
+1,1210
+0,7678
+0,542
+0,3316
+1,6445
+1,1284
+0,2000
+0,4796
+0,1689
+0,771
+1,1332
+1,3120
+1,1445
+1,2328
+0,1476
+0,1623
+0,624
+1,2336
+0,3777
+1,192
+0,1688
+1,3173
+0,2979
+0,5819
+0,1690
+0,2004
+1,7247
+0,1374
+0,2579
+1,803
+1,7231
+0,2680
+1,1335
+0,2401
+1,400
+0,761
+1,3182
+1,1408
+0,4668
+0,7910
+0,3101
+0,5779
+0,4449
+0,5300
+1,7658
+0,1800
+1,1364
+1,2448
+1,1466
+1,3008
+1,5376
+1,1088
+1,1340
+1,1927
+1,893
+1,977
+0,4246
+0,3048
+0,3992
+0,6292
+1,1388
+0,1882
+0,3668
+1,522
+1,1013
+0,2521
+1,3609
+0,1087
+0,9149
+1,781
+0,5656
+0,2393
+1,7587
+1,4083
+0,1856
+0,7913
+1,2608
+0,1132
+1,1774
+0,850
+1,982
+0,315
+0,3812
+1,2382
+1,1596
+0,6933
+0,173
+1,848
+1,203
+0,2875
+0,2036
+1,363
+0,3451
+0,1312
+1,83
+0,6930
+0,270
+1,4880
+0,521
+0,2725
+1,1890
+0,2734
+0,2292
+1,3961
+0,6228
+0,953
+1,1305
+0,746
+0,940
+1,4069
+1,2301
+1,146
+0,2632
+0,6413
+1,1810
+0,3280
+1,10251
+0,2412
+0,4342
+0,2957
+0,9002
+0,1995
+1,919
+1,1654
+1,798
+1,682
+1,3962
+0,2834
+1,6867
+0,746
+0,3144
+0,572
+1,2222
+0,2974
+1,1795
+1,4590
+0,7218
+0,2478
+1,2768
+1,900
+0,5199
+1,1662
+1,5447
+0,1639
+1,663
+0,3743
+0,1137
+1,569
+0,868
+1,885
+0,2873
+0,1357
+1,1478
+1,1937
+0,2380
+0,1061
+0,325
+0,1112
+0,1154
+1,246
+1,5350
+0,6478
+1,4356
+1,2502
+0,224
+0,3956
+1,3511
+1,4855
+1,2056
+1,695
+1,1867
+0,626
+0,1993
+0,8101
+1,384
+1,2162
+1,1129
+0,3158
+1,9598
+0,1673
+1,4096
+0,1849
+0,2299
+0,1569
+0,1773
+1,1248
+0,742
+0,2594
+1,814
+0,302
+0,5648
+1,1098
+0,1914
+0,3887
+1,5249
+1,1751
+0,4541
+1,1861
+0,612
+0,497
+1,1110
+0,1139
+0,5847
+0,4059
+1,1057
+1,2459
+0,1584
+1,1144
+0,2144
+1,965
+0,1324
+0,3399
+0,5143
+1,4743
+1,2627
+1,998
+0,3869
+0,2664
+1,1362
+1,2151
+1,1263
+1,1416
+0,1142
+0,5769
+0,3114
+0,1224
+1,1128
+0,1236
+1,1089
+0,1535
+1,1512
+1,2617
+1,8065
+0,6150
+1,492
+1,1007
+1,8100
+0,1474
+1,1423
+1,1853
+0,2079
+0,413
+0,764
+1,1984
+1,3049
+1,1105
+0,3630
+1,7143
+1,1041
+0,2049
+1,1522
+1,776
+0,2506
+0,5393
+0,1902
+0,679
+1,7832
+0,641
+1,299
+0,2408
+0,472
+1,864
+0,574
+0,4978
+0,4469
+1,868
+0,3564
+0,934
+0,1717
+0,1356
+0,2020
+0,664
+1,465
+0,1444
+1,374
+1,1868
+0,1203
+0,7244
+0,1134
+0,2748
+1,3791
+0,3078
+1,1192
+0,606
+0,2736
+0,833
+1,2883
+1,2212
+1,359
+1,1510
+0,4310
+0,875
+0,1834
+1,2257
+1,1345
+1,3949
+1,991
+1,2443
+1,2161
+1,4415
+0,8172
+0,1449
+1,6051
+0,4135
+1,3694
+1,290
+1,2114
+1,1342
+0,511
+0,1597
+0,3450
+1,736
+1,1699
+0,1123
+1,166
+0,3118
+1,7432
+1,2441
+1,5157
+0,927
+1,1172
+0,1979
+0,667
+1,1325
+0,4642
+1,1163
+1,2229
+1,579
+1,7360
+0,2215
+0,820
+0,4999
+1,923
+0,3127
+1,6025
+1,2103
+1,1313
+1,837
+0,656
+1,4072
+0,2949
+0,2409
+1,4796
+0,8472
+0,1454
+1,6733
+0,4059
+1,1294
+1,1461
+0,10460
+0,2806
+0,727
+0,1365
+0,1211
+0,1342
+1,2600
+0,3427
+1,1829
+0,6670
+1,3169
+1,2258
+0,2897
+0,644
+1,3578
+1,101
+0,1026
+0,225
+1,6698
+0,4186
+1,8538
+1,514
+0,784
+0,351
+0,1557
+0,2244
+1,3675
+1,2119
+0,9840
+0,942
+1,509
+1,1433
+0,1606
+0,1155
+1,5708
+0,1785
+1,1034
+1,1130
+0,5843
+1,1934
+1,1183
+1,1746
+0,1441
+1,507
+0,3790
+1,1423
+1,2996
+1,2742
+1,2448
+0,1581
+1,4652
+1,2273
+0,1340
+1,1942
+1,4773
+0,1635
+0,5474
+1,1623
+0,1319
+0,980
+0,1458
+1,757
+0,1832
+0,8852
+0,1741
+1,1596
+1,452
+0,2630
+1,5096
+1,1080
+0,2213
+1,2396
+0,714
+0,669
+1,1830
+1,6012
+1,728
+1,5857
+1,1129
+1,6733
+1,733
+1,538
+0,3840
+0,2392
+0,619
+1,1561
+0,4341
+1,4318
+0,2576
+0,727
+0,3197
+1,919
+0,4534
+0,3953
+1,1920
+0,1436
+1,5609
+1,269
+1,1755
+0,5402
+1,610
+1,4245
+0,158
+0,2187
+0,1066
+0,3534
+1,2885
+1,3194
+0,3019
+0,2519
+1,528
+0,5355
+1,1027
+1,539
+0,4180
+0,665
+1,5367
+1,3306
+1,997
+1,1212
+0,2637
+0,3692
+1,5139
+0,1346
+1,4605
+0,2024
+0,2656
+0,850
+0,902
+0,579
+0,2382
+0,897
+1,1768
+0,5329
+0,6201
+1,579
+0,9005
+0,566
+1,340
+1,408
+1,731
+1,902
+1,654
+1,1679
+0,298
+1,1408
+1,3438
+0,1332
+1,6224
+0,816
+0,904
+0,8363
+0,841
+1,579
+0,1827
+0,9554
+0,1544
+0,6404
+0,6998
+1,2080
+0,678
+1,3240
+0,2683
+1,2120
+0,1843
+1,3611
+0,6048
+1,1954
+0,1004
+1,6483
+0,242
+1,2099
+1,1129
+1,949
+0,2864
+1,1124
+0,322
+0,1279
+1,1297
+1,1410
+1,2804
+1,1919
+1,719
+1,3179
+0,1275
+1,5011
+0,1705
+1,1329
+1,1660
+1,2757
+0,369
+1,1050
+0,1169
+1,2703
+1,8165
+0,3753
+0,2808
+0,4114
+0,4425
+0,5788
+1,1258
+0,2929
+0,2887
+1,121
+0,3046
+0,1306
+0,3990
+0,3268
+0,785
+1,906
+1,1411
+0,588
+0,451
+0,260
+1,2310
+0,8420
+1,862
+1,1746
+1,194
+0,5563
+0,1806
+0,2622
+1,1099
+1,2184
+1,2521
+1,2390
+1,654
+1,3611
+1,3750
+0,1781
+1,2194
+0,1221
+1,1745
+1,784
+1,930
+0,1377
+0,2631
+0,471
+0,2534
+1,7076
+0,2945
+1,4252
+0,1903
+1,570
+0,1619
+1,1140
+0,2313
+1,2038
+1,1526
+0,7880
+1,970
+1,2336
+1,1079
+1,7931
+0,835
+1,4381
+1,3738
+1,1099
+1,955
+0,7808
+0,5389
+0,1053
+1,3955
diff --git a/lessons/StatisticalConsiderations/data/signtest_data.csv b/lessons/StatisticalConsiderations/data/signtest_data.csv
new file mode 100644
index 00000000..bd5298b0
--- /dev/null
+++ b/lessons/StatisticalConsiderations/data/signtest_data.csv
@@ -0,0 +1,15 @@
+day,control,exp
+1,0.09610,0.09715
+2,0.07357,0.06954
+3,0.10030,0.08782
+4,0.07225,0.06936
+5,0.06173,0.09893
+6,0.07736,0.08050
+7,0.07121,0.11161
+8,0.08207,0.10802
+9,0.10602,0.08793
+10,0.10000,0.12043
+11,0.08100,0.09236
+12,0.080357,0.10494
+13,0.081081,0.09392
+14,0.068323,0.11607
\ No newline at end of file
diff --git a/lessons/StatisticalConsiderations/data/statistical_significance_data.csv b/lessons/StatisticalConsiderations/data/statistical_significance_data.csv
new file mode 100644
index 00000000..35e2de51
--- /dev/null
+++ b/lessons/StatisticalConsiderations/data/statistical_significance_data.csv
@@ -0,0 +1,1000 @@
+condition,click
+1,0
+0,0
+0,0
+1,1
+1,0
+1,0
+0,0
+1,1
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+1,1
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,0
+1,0
+1,0
+0,1
+0,0
+0,0
+0,0
+0,1
+1,0
+1,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,1
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,1
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+1,0
+1,1
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,1
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,0
+1,0
+1,0
+0,0
+0,0
+1,1
+1,1
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+0,0
+0,0
+1,1
+1,0
+0,0
+1,0
+0,0
+1,1
+1,0
+0,0
+1,0
+0,0
+1,0
+0,1
+1,1
+1,0
+1,0
+1,1
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,1
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,1
+1,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,0
+0,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+0,1
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,1
+1,0
+0,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,1
+0,0
+1,1
+1,0
+0,1
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+0,0
+1,1
+0,0
+1,0
+1,0
+1,0
+0,0
+1,1
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,1
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,1
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,1
+1,0
+1,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,0
+0,0
+1,1
+1,0
+1,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,1
+1,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+0,0
+0,0
+1,1
+0,0
+1,0
+1,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,1
+1,1
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,0
+1,0
+1,0
+1,1
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+0,0
+0,1
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+0,1
+1,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,1
+1,0
+1,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+0,0
+0,0
+1,1
+0,1
+1,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,1
+1,0
+0,0
+1,1
+0,0
+1,0
+0,0
+1,0
+0,0
+1,1
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,1
+1,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,1
+1,0
+1,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,1
+1,0
+0,0
+0,0
+0,0
+1,0
+1,1
+1,0
+0,0
+1,0
+0,0
+0,1
+0,0
+0,0
+1,1
+1,1
+0,0
+1,0
+1,0
+0,1
+0,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,1
+0,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+0,1
+0,1
+0,0
+1,1
+1,0
+1,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+1,1
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,1
+1,0
+1,0
+1,1
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+1,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,1
+1,0
+0,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,1
+1,0
+1,1
+0,0
+1,0
+1,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,1
+0,0
+1,0
+0,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,1
+1,0
+1,0
+1,0
+0,0
+1,0
+1,1
+0,0
+1,0
+1,0
+0,1
+1,0
+1,1
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,1
+1,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,1
+1,0
+0,0
+0,0
+0,0
+0,1
+1,0
+0,0
+1,0
+0,1
+1,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,1
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+1,0
+1,1
+0,1
+0,0
+0,0
+0,0
+0,0
+0,0
+0,1
+1,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,1
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,1
+0,1
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+1,0
+1,1
+0,0
+0,0
+0,0
+1,0
+1,1
+0,0
+1,0
+1,0
+1,0
+1,0
+0,1
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,0
+1,0
+1,0
+1,1
+1,0
+1,1
+0,0
+1,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,1
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+1,0
+1,0
+1,0
+1,1
+1,0
+1,0
+0,0
+0,0
+0,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,0
+1,1
+1,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+1,1
+1,0
+0,0
+1,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+1,1
+0,0
+1,0
+1,0
+1,0
+0,0
+1,0
+0,0
+0,0
+0,0
+1,0
+0,0
+1,1
+0,0
+1,0
+0,0
+1,0
+1,1
+0,0
+0,0
+0,0
+1,1
+1,0
+0,0
+1,0
+1,1
+1,1
+0,0
+1,0
+0,0
+0,0
+0,1
+1,0
+1,0
+1,0
+0,0
+0,0
+0,1
+0,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,1
+0,0
+1,0
+1,0
+1,0
+1,1
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+1,0
+1,0
+0,0
+0,0
+0,0
+0,0
+1,0
+1,0
+1,0
+1,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+1,0
+1,0
+0,0
+0,0
+0,0
+1,0
+0,0
+0,1
+0,0
+1,0
+0,0
+1,0
+0,0
+1,0
+0,0
+0,0
diff --git a/lessons/StatisticalConsiderations/images/ExpSize_Power.png b/lessons/StatisticalConsiderations/images/ExpSize_Power.png
new file mode 100644
index 00000000..7dddd5b9
Binary files /dev/null and b/lessons/StatisticalConsiderations/images/ExpSize_Power.png differ
diff --git a/lessons/StatisticalConsiderations/solutions/L2_Early_Stopping_Solution.ipynb b/lessons/StatisticalConsiderations/solutions/L2_Early_Stopping_Solution.ipynb
new file mode 100644
index 00000000..3580fef8
--- /dev/null
+++ b/lessons/StatisticalConsiderations/solutions/L2_Early_Stopping_Solution.ipynb
@@ -0,0 +1,229 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Early Stopping\n",
+ "\n",
+ "If you peek at the results of an experiment before data collection is complete, and choose to stop early because the test is showing statistical significance, you run the risk of a significant increase in your Type I error rate: believing that your experiment had an effect, when in fact it did not. In this notebook, you'll duplicate the assertions made in the video: namely that for an experiment based off of a single traditional statistical test, doing a single peek halfway through the run-time will increase a base Type I error rate from 5% to about 8.6%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import scipy.stats as stats"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The simulation function below uses a bernoulli / binomial success model for the outcome metric, measuring against a historic baseline. That is, each observation is a single coin flip with success probability \"p\". If we see a number of successes that is unusual for our baseline value of \"p\", then we declare a statistically significant result. We will divide the experiment length into multiple 'blocks', checking the status of the experiment after each block is complete. Our outputs of interest are the proportion of trials that are statistically significant in _any_ test, and the proportion of trials that are statistically significant after _each_ individual block.\n",
+ "\n",
+ "There are three main steps to filling out the `peeking_sim()` function.\n",
+ "\n",
+ "1. Simulate some data\n",
+ " - Compute the number of trials per block. For simplicity, just round up any fractions so that each block has the same number of trials: we might end up with slightly more trials per block than the corresponding function parameter.\n",
+ " - Generate a data matrix with the number of successes observed in each block: the number of rows should be the number of simulations and the number of columns the number of blocks. You can do this with a single call to numpy's [`random.binomial`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.binomial.html) function.\n",
+ "2. Compute z-scores at each 'peek'\n",
+ " - For each row, compute the cumulative number of successes after each 'block' of the experiment using numpy's [`cumsum`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cumsum.html) function. The result should be a matrix with the same dimensions as the data, but each column cumulatively adds up the values in each row up to that point.\n",
+ " - Compute the expected mean and standard deviation for the number of successes after each 'block' of the experiment. Remember that this will be based on the [binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution) and is centered on the raw counts, rather than proportion of successes. It'll be useful to create a vector with the cumulative sum of trials after each block to facilitate these calculations.\n",
+ " - Use the cumulative counts, the expected counts, and the standard deviations, to compute the z-scores for each peek at the experiment.\n",
+ "3. Aggregate test outcomes\n",
+ " - Compute a critical z-value using the supposed Type I error rate. Use this critical value to flag which of the z-scores would be counted as statistically significant, and which would not.\n",
+ " - The proportion of trials that are significant at _any_ test will be the proportion of rows that have at least one flagged value. The proportion of trials that are significant at _each_ block will be the mean number of flagged values in each column; this will be a 1-d array. Return both of these values as the output of the function."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def peeking_sim(alpha = .05, p = .5, n_trials = 1000, n_blocks = 2, n_sims = 10000):\n",
+ " \"\"\"\n",
+ " This function simulates the rate of Type I errors made if an early\n",
+ " stopping decision is made based on a significant result when peeking ahead.\n",
+ " \n",
+ " Input parameters:\n",
+ " alpha: Supposed Type I error rate\n",
+ " p: Probability of individual trial success\n",
+ " n_trials: Number of trials in a full experiment\n",
+ " n_blocks: Number of times data is looked at (including end)\n",
+ " n_sims: Number of simulated experiments run\n",
+ " \n",
+ " Return:\n",
+ " p_sig_any: Proportion of simulations significant at any check point, \n",
+ " p_sig_each: Proportion of simulations significant at each check point\n",
+ " \"\"\"\n",
+ " \n",
+ " # generate data\n",
+ " trials_per_block = np.ceil(n_trials / n_blocks).astype(int)\n",
+ " data = np.random.binomial(trials_per_block, p, [n_sims, n_blocks])\n",
+ " \n",
+ " # standardize data\n",
+ " data_cumsum = np.cumsum(data, axis = 1)\n",
+ " block_sizes = trials_per_block * np.arange(1, n_blocks+1, 1)\n",
+ " block_means = block_sizes * p\n",
+ " block_sds = np.sqrt(block_sizes * p * (1-p))\n",
+ " data_zscores = (data_cumsum - block_means) / block_sds\n",
+ " \n",
+ " # test outcomes\n",
+ " z_crit = stats.norm.ppf(1-alpha/2)\n",
+ " sig_flags = np.abs(data_zscores) > z_crit\n",
+ " p_sig_any = (sig_flags.sum(axis = 1) > 0).mean()\n",
+ " p_sig_each = sig_flags.mean(axis = 0)\n",
+ " \n",
+ " return (p_sig_any, p_sig_each)\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Running the function on the default parameters as given should return a tuple of results where the probability of any significant test outcome across the two blocks is around 8.6% and the probability of a significant test outcome at each individual block checkpoint is around 5%. Increase the number of trials and number of simulations to get more accurate estimates. You should also see how the overall Type I error rate increases with additional peeks!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(0.084169999999999995, array([ 0.0494, 0.0516]))"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "peeking_sim(n_trials = 10_000, n_sims = 100_000)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## A Multiple Comparisons Approach to Early Peeking\n",
+ "\n",
+ "The safest way we could deal with performing multiple checks and making poor early stopping decisions is to simply not do it. Once an experiment has been planned and all assignment procedures checked, you should just let the experiment run to completion and just assess the results at the very end. That's not to say that you can't perform early stopping, but it does require additional planning.\n",
+ "\n",
+ "One way in which you could solve for multiple peeking is to adjust the significance level of individual tests so that the overall error rate is at its desired level. But applying the Bonferroni or Šidák corrections as shown earlier in the lesson will definitely be too conservative, since we know that there is a correlation in test results between peeks. If we see some simulated run with z-score above the threshold at the halfway point, it's more likely to be above that threshold at the end point, compared to some other simulated run that is not statistically significant at the halfway point. One way in which we can obtain a better significance threshold is through the power of simulation. After performing the same steps 1 and 2 above, we want to find a significance level that would call our desired proportion of simulated tests as statistically significant:\n",
+ "\n",
+ "1. Simulate some data (as above)\n",
+ "2. Compute z-scores at each 'peek' (as above)\n",
+ "3. Obtain required individual-test error rate\n",
+ " - A run is considered statistically significant if it exceeds the critical bounds at _any_ peek. Obtain the maximum z-score from each row as a worst-case scenario for a null run to be falsely rejected.\n",
+ " - Find the z-score threshold that would reject our desired overall Type I error rate.\n",
+ " - Convert that z-score into an equivalent individual-test error rate."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def peeking_correction(alpha = .05, p = .5, n_trials = 1000, n_blocks = 2, n_sims = 10000):\n",
+ " \"\"\"\n",
+ " This function uses simulations to estimate the individual error rate necessary\n",
+ " to limit the Type I error rate, if an early stopping decision is made based on\n",
+ " a significant result when peeking ahead.\n",
+ " \n",
+ " Input parameters:\n",
+ " alpha: Desired overall Type I error rate\n",
+ " p: Probability of individual trial success\n",
+ " n_trials: Number of trials in a full experiment\n",
+ " n_blocks: Number of times data is looked at (including end)\n",
+ " n_sims: Number of simulated experiments run\n",
+ " \n",
+ " Return:\n",
+ " alpha_ind: Individual error rate required to achieve overall error rate\n",
+ " \"\"\"\n",
+ " \n",
+ " # generate data\n",
+ " trials_per_block = np.ceil(n_trials / n_blocks).astype(int)\n",
+ " data = np.random.binomial(trials_per_block, p, [n_sims, n_blocks])\n",
+ " \n",
+ " # standardize data\n",
+ " data_cumsum = np.cumsum(data, axis = 1)\n",
+ " block_sizes = trials_per_block * np.arange(1, n_blocks+1, 1)\n",
+ " block_means = block_sizes * p\n",
+ " block_sds = np.sqrt(block_sizes * p * (1-p))\n",
+ " data_zscores = (data_cumsum - block_means) / block_sds\n",
+ " \n",
+ " # find necessary individual error rate\n",
+ " max_zscores = np.abs(data_zscores).max(axis = 1)\n",
+ " z_crit_ind = np.percentile(max_zscores, 100 * (1 - alpha))\n",
+ " alpha_ind = 2 * (1 - stats.norm.cdf(z_crit_ind))\n",
+ " \n",
+ " return alpha_ind"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Running the function on the default parameters should give a required individual error rate of about .029. Note how this is somewhat higher than the .025 or .0253 that would have been generated from the Bonferroni and Šidák corrections, respectively. Test with a higher number of simulations and trials to get more accurate estimates, and try out different numbers of blocks to see how it changes the individual error rate needed. The results should approximately match up with the numbers given in the table in the middle of [this article](https://www.evanmiller.org/how-not-to-run-an-ab-test.html); note that peeking $n$ times means splitting the experiment into $n + 1$ blocks."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.029414310138636379"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "peeking_correction(n_trials = 10_000, n_sims = 100_000)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/solutions/L2_Experiment_Size_Solution.ipynb b/lessons/StatisticalConsiderations/solutions/L2_Experiment_Size_Solution.ipynb
new file mode 100644
index 00000000..4340e0e5
--- /dev/null
+++ b/lessons/StatisticalConsiderations/solutions/L2_Experiment_Size_Solution.ipynb
@@ -0,0 +1,338 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Experiment Size\n",
+ "\n",
+ "We can use the knowledge of our desired practical significance boundary to plan out our experiment. By knowing how many observations we need in order to detect our desired effect to our desired level of reliability, we can see how long we would need to run our experiment and whether or not it is feasible.\n",
+ "\n",
+ "Let's use the example from the video, where we have a baseline click-through rate of 10% and want to see a manipulation increase this baseline to 12%. How many observations would we need in each group in order to detect this change with power $1-\\beta = .80$ (i.e. detect the 2% absolute increase 80% of the time), at a Type I error rate of $\\alpha = .05$?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import packages\n",
+ "import numpy as np\n",
+ "import scipy.stats as stats\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Method 1: Trial and Error\n",
+ "\n",
+ "One way we could solve this is through trial and error. Every sample size will have a level of power associated with it; testing multiple sample sizes will gradually allow us to narrow down the minimum sample size required to obtain our desired power level. This isn't a particularly efficient method, but it can provide an intuition for how experiment sizing works.\n",
+ "\n",
+ "Fill in the `power()` function below following these steps:\n",
+ "\n",
+ "1. Under the null hypothesis, we should have a critical value for which the Type I error rate is at our desired alpha level.\n",
+ " - `se_null`: Compute the standard deviation for the difference in proportions under the null hypothesis for our two groups. The base probability is given by `p_null`. Remember that the variance of the difference distribution is the sum of the variances for the individual distributions, and that _each_ group is assigned `n` observations.\n",
+ " - `null_dist`: To assist in re-use, this should be a [scipy norm object](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html). Specify the center and standard deviation of the normal distribution using the \"loc\" and \"scale\" arguments, respectively.\n",
+ " - `p_crit`: Compute the critical value of the distribution that would cause us to reject the null hypothesis. One of the methods of the `null_dist` object will help you obtain this value (passing in some function of our desired error rate `alpha`).\n",
+ "2. The power is the proportion of the distribution under the alternative hypothesis that is past that previously-obtained critical value.\n",
+ " - `se_alt`: Now it's time to make computations in the other direction. This will be standard deviation of differences under the desired detectable difference. Note that the individual distributions will have different variances now: one with `p_null` probability of success, and the other with `p_alt` probability of success.\n",
+ " - `alt_dist`: This will be a scipy norm object like above. Be careful of the \"loc\" argument in this one. The way the `power` function is set up, it expects `p_alt` to be greater than `p_null`, for a positive difference.\n",
+ " - `beta`: Beta is the probability of a Type-II error, or the probability of failing to reject the null for a particular non-null state. That means you should make use of `alt_dist` and `p_crit` here!\n",
+ "\n",
+ "The second half of the function has already been completed for you, which creates a visualization of the distribution of differences for the null case and for the desired detectable difference. Use the cells that follow to run the function and observe the visualizations, and to test your code against a few assertion statements. Check the video above the workspace if you need help coming up with the solution."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def power(p_null, p_alt, n, alpha = .05, plot = True):\n",
+ " \"\"\"\n",
+ " Compute the power of detecting the difference in two populations with \n",
+ " different proportion parameters, given a desired alpha rate.\n",
+ " \n",
+ " Input parameters:\n",
+ " p_null: base success rate under null hypothesis\n",
+ " p_alt : desired success rate to be detected, must be larger than\n",
+ " p_null\n",
+ " n : number of observations made in each group\n",
+ " alpha : Type-I error rate\n",
+ " plot : boolean for whether or not a plot of distributions will be\n",
+ " created\n",
+ " \n",
+ " Output value:\n",
+ " power : Power to detect the desired difference, under the null.\n",
+ " \"\"\"\n",
+ " \n",
+ " # Compute the power\n",
+ " se_null = np.sqrt((p_null * (1-p_null) + p_null * (1-p_null)) / n)\n",
+ " null_dist = stats.norm(loc = 0, scale = se_null)\n",
+ " p_crit = null_dist.ppf(1 - alpha)\n",
+ " \n",
+ " se_alt = np.sqrt((p_null * (1-p_null) + p_alt * (1-p_alt) ) / n)\n",
+ " alt_dist = stats.norm(loc = p_alt - p_null, scale = se_alt)\n",
+ " beta = alt_dist.cdf(p_crit)\n",
+ " \n",
+ " if plot:\n",
+ " # Compute distribution heights\n",
+ " low_bound = null_dist.ppf(.01)\n",
+ " high_bound = alt_dist.ppf(.99)\n",
+ " x = np.linspace(low_bound, high_bound, 201)\n",
+ " y_null = null_dist.pdf(x)\n",
+ " y_alt = alt_dist.pdf(x)\n",
+ "\n",
+ " # Plot the distributions\n",
+ " plt.plot(x, y_null)\n",
+ " plt.plot(x, y_alt)\n",
+ " plt.vlines(p_crit, 0, np.amax([null_dist.pdf(p_crit), alt_dist.pdf(p_crit)]),\n",
+ " linestyles = '--')\n",
+ " plt.fill_between(x, y_null, 0, where = (x >= p_crit), alpha = .5)\n",
+ " plt.fill_between(x, y_alt , 0, where = (x <= p_crit), alpha = .5)\n",
+ " \n",
+ " plt.legend(['null','alt'])\n",
+ " plt.xlabel('difference')\n",
+ " plt.ylabel('density')\n",
+ " plt.show()\n",
+ " \n",
+ " # return power\n",
+ " return (1 - beta)\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd4VFX6wPHvmfQe0kM6EDqhd1REUFBQFLuArrrYXdfyW3V1LbuurBXUVQEbgqIIVhRZRJEWSijSIbQUSiohgdSZOb8/7qCU9MzMvZk5n+eZZyZT7n0zhHnn3HvO+wopJYqiKIr7MukdgKIoiqIvlQgURVHcnEoEiqIobk4lAkVRFDenEoGiKIqbU4lAURTFzalEoCiK4uZUIlAURXFzKhEoiqK4OU+9A2iMiIgImZycrHcYiqIorcrGjRsLpZSRDT2vVSSC5ORkMjIy9A5DURSlVRFCZDXmeerQkKIoiptTiUBRFMXNqUSgKIri5hx2jkAI4QusAHxs+1kgpXxGCJECfAaEAZuASVLKakfFoSiKclpNTQ25ublUVlbqHYpd+fr6Eh8fj5eXV7Ne78iTxVXACCnlSSGEF7BKCLEYeBh4XUr5mRDiXeAO4B0HxqEoigJAbm4uQUFBJCcnI4TQOxy7kFJSVFREbm4uKSkpzdqGww4NSc1J249etosERgALbPfPBsY7KgZFUZQzVVZWEh4e7jJJAEAIQXh4eItGOQ49RyCE8BBCbAHygaXAfqBESmm2PSUXiKvjtVOEEBlCiIyCggJHhqkoihtxpSRwWkt/J4euI5BSWoBeQohQ4CugS21Pq+O1M4GZAP369VP9NN1MdlE5v+WWkHO8nMpqC14eJmJD/ejWNphO0UGYTK73n1lR9OKUBWVSyhIhxHJgEBAqhPC0jQrigSPOiEExvvyySj5bn8OXm3I5VFRe5/PCA7wZ17MtEwcl0iEqyIkRKorzLV++nFdeeYVFixbx0UcfkZGRwVtvvWXXfThy1lAkUGNLAn7ASOA/wC/AtWgzh24FvnFUDErrUFZZw5s/72P2mkNUma0MaR/O7cNS6JcURlK4PwE+nlTWWDhcUsGW7BJ+3p3Pp+uymZ1+iLFpbXliTGfahvrp/WsoSqvlyBFBLDBbCOGBdi5ivpRykRBiJ/CZEOJfwGbgfQfGoBjc8j35PL5wG3lllVzdO44HRqSSEhFw3vN8vTxoHxlI+8hAJvSNp/hUNe+tPMAHqw+ybFcefxvdmcmDk1zy+K/iWg4dOsSYMWMYNmwYa9asIS4ujm+++YYxY8bwyiuv0K9fPwoLC+nXrx+HDh1ySkwOSwRSyq1A71ruPwAMcNR+ldbBYpW8vGQP7/66n07RQbwzsQ+9E9s0+vVhAd783+jO3DQgkae+3s4z3+5gzf5CXrmuJ0G+zZtLrbiX577bwc4jpXbdZte2wTwzrluDz8vMzGTevHnMmjWL66+/noULF9o1jqZSK4sVpyuvNnPXnI28++t+bhqQyDf3D21SEjhTQpg/H/2pP09d0YVlu/K57t10jpRU2DliRbGvlJQUevXqBUDfvn2d9s2/Lq2i+qjiOsoqa7jtww1szj7Oc1d249YhyS3ephCCOy9oR6eYIO6du4lr31nDvCmDSAo//xCTopzWmG/ujuLj4/P7bQ8PDyoqKvD09MRqtQI4feWzGhEoTnOyyszkD9bzW04J/725j12SwJkuSI1k3pRBVNRYuH5GOllFp+y6fUVxpOTkZDZu3AjAggULGni2falEoDhFldnCXXMy2Jp7gv/e0ocxPWIdsp/ucSHMmzKIKrOVSe+vJ7/MtWrK/M5SA0d/g63zYc1bsGoarJ8Fe5dAqZqR3Ro9+uijvPPOOwwZMoTCwkKn7ltIafy1Wv369ZOqMU3rJaXk4fm/8dXmw7x6XU8m9I13+D43Zx/nlvfW0S4ygAV3D8HXy8Ph+3Q4qwUy/we/zYPMpVBT91oLwtpBt6uh90TttgLArl276NKltnWtrV9tv5sQYqOUsl9Dr1XnCBSH+2D1Ib7afJiHR3V0ShIA6J3Yhjdu7M2dH2fw5FfbePW6nq13aqnVCtsXwC8vwPFD4B0IER0hNAkCIsEnEIQJzNVQWQJlR6D4AKx8DVa9Dl2vghFPQ3h7vX8TxaBUIlAcKn1/Ef/+YRejukZz/8UdnLrvkV2jeWhkKtN+yiQtLoTbhjavMqOu8nbCt/fD4Y0QGA1dx0N4KphqGeF4eGtJISQe4gdAVRkc3kj5b1/jte1rvC78Kwx/HDx9zn+t4tbUOQLFYY6eqOD+TzeRFO7Pa9f31KU+0IMjUhnZJZp/fr+LtQeKnL7/ZpMS1r4DMy6A/N3QeSz0uQ0iO9eeBGrjEwTthjNxZRLL8tvAqtdg5nAo2OPAwJXWSCUCxSEsVskDn26mssbCzEl9dVvkZTIJXruhJ0nh/jw4bzMl5a2gB1JNBSy4HX58HNqkQP87ILo7NPPQVnG1J1N3J8HNX8CpApg1AnZ9Z+egldZMJQLFId5beYCMrOP8c3x33QvDBft68caNvSk+Vc3T3+zQNZYGlRfDR1fAjq8gZTh0uwa8/O2z7Y6XwpRftVHF55O0EYeioBKB4gB788p49X97uaxbNFf3rrXdhNN1jwvhL5ek8t1vR1i01aDTK8uLYfaV2rTQbuMhcVCzRwF1ComD2xZB5yu0EceKV+y7faVVUolAsasai5WH528hyNeTF67uYaiZOvcMb0/PhFCe+no7+aUGW19wOgkU7NJGARGd7LbpOdOeY86cOX/c4eUH138MaTfAz/+E9P/abV9K8yUnJ1NYWEhJSQlvv/22U/etEoFiV//9ZR/bD5fywtU9iAg01uwUTw8Tr17Xk4pqC098uQ3DrKGpPAEfn5EE7DzvP6FtNAkJCWffafKAq97WppYueRI2qCLARqESgdKqHSg4ydu/7OeqXm0Z3T1G73Bq1SEqkEcv7cSy3fn8b2ee3uGAxQxf/AnydkDXqx2y+Ovzb5fy+eefn/+Ahydc8x50HA3fP6ydl1CcYvz48fTt25du3boxc+bMsx57/PHH2b9/P7169eKxxx5zSjxqHYFiF1JKnvl2Bz6eJv5+hbFXbt42NJkFG3N5/rudXJAagb+3jv8NljwJ+5dB6miHLfh6Z+5C8AnihhtuOP9BT2+4brY2Ivn6XghrD7FpDonDcBY/Dse22XebMT1gzNQGn/bBBx8QFhZGRUUF/fv3Z8KECb8/NnXqVLZv386WLVvsG1s91IhAsYsftx9jZWYhj1zakaggX73DqZeXh4l/ju/O4ZIK3vp5n36BbHgf1s+AuH7Qtpd+cXj5wvVzwDcUPrsFTjm3zo07euONN+jZsyeDBg0iJyeHzMxMXeNRIwKlxcqrzTy/aCddYoOZOChJ73AaZUBKGNf0iWPWygNM6BtP+8hA5waQvRZ+eEz7Bt5+hHP3XZugaLjxE/hwDMy/FSZ/DR4u3uCnEd/cHWH58uX89NNPpKen4+/vz/Dhw51edvpcakSgtNibP+/j6IlK/nlVNzw9Ws+f1BNjuuDr5cGz3+5w7onjiuPagjHfYOgyTqsTZARxfeDKNyFrFSx7Xu9oXNaJEydo06YN/v7+7N69m7Vr1571eFBQEGVlZU6NySB/gUprlVNczvsrD3JNnzj6JYfpHU6TRAb58PCojqzMLGT53gLn7FRK+PYBKDsKnceBp8EOo6VdD33/BGvehAO/6h2NSxo9ejRms5m0tDSefvppBg0adNbj4eHhDB06lO7du6uTxUrr8Or/9iAEPHaZ/ea9O9MtA5P4aM0hpv6wmwtTI/FwdD2kjR9q5R3aXQzBbR27L5sF706FuPPah9ftshfg0Cr46m64ZzX4t64Eb3Q+Pj4sXrz4vPvPbFf56aefOjEiNSJQWmD74RN8veUItw9LITbET+9wmsXb08T/XdaZPXllLNyU69idFezVZqq0SdGqgzpJRFgoERERjX+BdwBMeE+rS/TdX7RRjOLSVCJQmkVKyYuLd9HG34t7hrfuOveX94ihV0Ior/5vDxXVFsfsxGqBb+7VFnJ1vsL+pSPq8dEXi/joo4+a9qK2vWDEU7DrW/jtM4fEpRiHSgRKs6zILGT1viIeGJFKsE6VRe1FCMGTl3chr7SKD1YfdMxONrwHuRu0GULezp2h1KxEADDkQUgYBEuegJNOOofiBIZZUW5HLf2dVCJQmsxqlUxdvJvEMP9WM120IQNSwhjVNZp3lu+3f6nq41nw0zPaquGobvbdtiOZTDBuOlSd1JKBC/D19aWoqMilkoGUkqKiInx9mz/xwGEni4UQCcDHQAxgBWZKKacLIZ4F/gyc/orxpJTyB0fFodjfkh3H2HW0lNdv6Im3p+t8l3jk0o6MnraS91Ye5FF7nfyWEr57UGs3mXqZUw8J2UVUZ7jgEfh1qlakLnWU3hG1SHx8PLm5uRQUuM4IB7QEFx/f/Dawjpw1ZAYekVJuEkIEARuFEEttj70upVT1b1shq1Uy7adM2kUGcGVPY5SYtpfOMcFckRbLh6sPcvuwFMICvFu+0W1fwIHl0OFS8A1p+fb0cMHDsONLWPQw3JuutcNspby8vEhJaYUtSx3MYV/npJRHpZSbbLfLgF2Aa31yuKHF24+xJ6+Mv1yS6vipljp46JJUymsszFp5oOUbqyzVagkFtYW2TZi+aTSePjDuDTiRDSte0jsaxQGcMq4XQiQDvYF1trvuF0JsFUJ8IIRo44wYlJazWiXTl+2lQ1QgY9OcMwfe2VKjgxiX1pbZaw5RdLKqZRtb8ZI2BbPDSF0PCf0wexo//NDCo69Jg6HXLZD+NhTtt09gimE4PBEIIQKBhcBDUspS4B2gPdALOAq8WsfrpgghMoQQGa52PK+1+n7bUfbmneRBFx0NnPbgJalar+UVLRgVFOzVWkHGpDlt4Vhd/P188fe3Q7vLS57RVkIvebLl21IMxaGJQAjhhZYEPpFSfgkgpcyTUlqklFZgFlDryhop5UwpZT8pZb/IyEhHhqk0gtUqeWNZJqlRgVzRI1bvcByqQ1QgV/WKY3Z6M0cFUsLix8DkCSkX2T2+pnr74wX2aXQSFA0XPQZ7f4TMpQ0/X2k1HJYIhNaj8H1gl5TytTPuP/NT5Gpgu6NiUOxn6a48MvNPcv+IDi49GjjtvovbU2W28tGaQ01/8Z4ftBPEScO0Vbo6m7/oJ+bPn2+fjQ28R6uY+uPjYLbzNFtFN44cEQwFJgEjhBBbbJfLgZeEENuEEFuBi4G/OjAGxQ6klLy9fD+JYf4uPxo4rUNUEJd2jWb2mkOUVdY0/oWWGvjf0+AfoVXzdDWe3jD6RSjaBxtm6R2NYieOnDW0SkoppJRpUspetssPUspJUsoetvuvlFIedVQMin2k7y/it5wS7rqoXasqM91S9w7vQGmlmU/XZTf+RZs+huL92iEho5SXtreOl2krpFe8DBUlekej2IGL/qUq9vT28v1EBvkwoU/zF6y0Rj0TQhnaIZz3Vh2ksqYRNYiqyuCXFyAkAcI7OD5APY18TuursHqa3pEodqASgVKvrbklrNpXyJ3DUvD18tA7HKe7b3gHCsqqWLCxEZVJ17wJ5UVaienWtoK4qWLToMf12syo0iN6R6O0kEoESr3e/mU/wb6e3OIiNYWaanD7cHomhDJjxX7MFmvdTyw7BqvfgMjOuk8XPdfy+e+yfPly+294xFMgrbD8RftvW3EqlQiUOh0qPMWSnceYPDiZQB/37GEkhOCei9qRU1zB0p15dT9xxStgqTLEdFGnaZME/e+EzXMhf7fe0SgtoBKBUqcPVx/Ey2Ri8hD3HA2cNqprDIlh/ry3qo4S1SXZWuexmDTwM95C+VdmzOWVVxxU2uuCR7Wy2r+84JjtK06hEoFSqxPlNczPyOXKXm2JCjJYX10n8zAJbh+azMas42zKPn7+E3611d9JGuLcwBpp0bJVLFq0yDEbDwiHQfdoDWyObXPMPhSHU4lAqdWn67OpqLFwxzBVqRHgun4JBPl68v65o4Ki/bDlU4jtBT7B+gSnt0H3gk8ILJ+qdyRKM6lEoJyn2mzlozUHGdYhgi6xbvrhdo4AH09uHpjI4m1HySku/+OB5VO19pOJg/QLTm9+oTD4Xti9CI5u1TsapRlUIlDO88O2o+SVVqnRwDluG5KMSQhmny47kb9L6zfQto/T208azsC7tX4LalTQKqlEoJxFSsl7qw7QPjKAizqqYn9nig3x44q0WD7bkKOVnVjxMnh4Q8JAvUOrl5+vD35+fg7eSSgMug/2fA9Htjh2X4rdqUSgnGX9wWK2Hy7ljmHtMLlBcbmmumNYCierzPy4fCVs/1JrOOPl4A/ZFlr88XQWL17s+B0NUqOC1kolAuUs7606SBt/L67po5rJ1SYtPpQBKWH4b3gTafKE+P56h2QcviEw+AHYuxiObNY7GqUJVCJQfnew8BQ/7cpj0qAktywn0Vj39/biMvNyDob0N0SZ6Yb8c/r7/POf/3TOzgbeBb6halTQyqhEoPxu9ppDeJoEEwe79wKyhgzLm4tVmPhP+Xi9Q2mUZas3sGzZMufszDcYhtyvNa9Ro4JWQyUCBYDyajMLN+ZyeY9Yt19AVq/SI5g2z2V/YH+WnIhnR6l6r84zYIq2pmLV63pHojSSSgQKAN9sOUJZlZlJblpcrtFWvwHSQlzHPviarMzJDtc7IuPxDdFqEO38Fgoz9Y5GaQSVCBSklMxJz6JzTBB9k4xXK8cwTuZrNYWiuhEcFMTVbY/z9dE2nKhR51POM+he8PRR/QpaCZUIFDbnlLDzaCkTByUhXL2Ofkuk/xfMVZA4GIBJCUVUWk18cdjYyTO8TQjh4U4euQRGQu9J8NvncOKwc/etNJlKBApz12YR6OPJ+N5qymidKkpg/Syt34B/GABdgyvpH3qKj7PDsUqd46vHwhn/YeHChc7f8ZAHtH4F6W85f99Kk6hE4OaKT1WzaOtRru4d57Y9Bxpl02yoOXXeKuLJiYVkV/jwa2GQToEZWJsk6HEdbPwIThXpHY1SD5UI3NwXGTlUm61MVCeJ62ap0VoyhiZBUMxZD10WXUqUTw0fG/ik8RNT/8sTTzyhz86HPQQ15bB+hj77VxpFJQI3ZrVKPlmXzYCUMDrFqG+0ddrxFZQdrXUVsbdJclN8McsLgzh0yluH4BqWvmkb6enp+uw8qgt0ugLWzYCqMn1iUBqkEoEbW5FZQHZxuRoN1EdKWD0d/MMhrH2tT7k5vggPAXNzjDsq0NUFD0NliXaISDEklQjc2Ny12UQEejO6W0zDT3ZXh1ZC3nZtNFDHjKpoXzOjo08w/3AbKixq1tV54vtB8gXa4TVLjd7RKLVwWCIQQiQIIX4RQuwSQuwQQvzFdn+YEGKpECLTdm3suXcuKvd4OT/vzuOG/gl4e6rvA3Va/QZ4+UN093qfNjmxkFKzJ98cVX/OtRryAJQe1g6zKYbjyE8AM/CIlLILMAi4TwjRFXgcWCalTAWW2X5WnGze+mwAbhqQqHMkBlawF/Yt1RrPmOqfUdU/tJzOgRXMzg5HGmwqaXxsFPHx8foG0WEURHSCNW9guDdIcVwikFIelVJust0uA3YBccBVwGzb02YDraNylwupNlv5fEMOIzpHE9/GX+9wjCv9LS0BtO3T4FOFgFsTi9hV5kdGibHe07nTn2fu3Ln6BmEyacXojm2Dg7/qG4tyHqccExBCJAO9gXVAtJTyKGjJAohyRgzKH37ccYzCk9VMHKRGA3U6VQi/zdMOCXk37oP9qtjjBHuamZ0d4eDgWqke10NAFKxRC8yMxuGJQAgRCCwEHpJSljbhdVOEEBlCiIyCggLHBeiG5q7NIjHMnwtTVSvKOq2fBZbqJjWe8feUXBt3nCV5wRRUGWdx3kPPvsZDDz2kdxjg5atVJt23FPJ26h2NcgaHJgIhhBdaEvhESvml7e48IUSs7fFYIL+210opZ0op+0kp+0VGqg8se9lzrIz1B4uZOChRtaKsS02FtgAqrL02bbQJbkkopkaamG+g+kNbdu5lyxaD9BHufwd4+ml1mxTDcOSsIQG8D+ySUr52xkPfArfabt8KfOOoGJTzzV2bhbeniev6JugdinFt/RwqjkP8gCa/tH1AFUPCyvg0JxyLOid6Pv8w6D0Rts2HsmN6R6PYOHJEMBSYBIwQQmyxXS4HpgKjhBCZwCjbz4oTnKwy89Xmw4xNi6VNgDFXwerOatWmjAZGQ2jzzqFMSijicKU3vxSo1dq1GnSPtp5g/Uy9I1FsHHYgU0q5Cqjr2MMljtqvUrevNx/mZJVZrSSuz76lULwfOo+rcwFZQ0ZGafWH5uaEMzJKlVU4T3h76DIWNrwPFzzSKvo+uzq1kshNSCmZuzaLbm2D6Z0Qqnc4xrV6OvgEaeWmm8nLBDfFF/NrYRDZ5fqPvDqmJNKxY0e9wzjbkAe1shObP9E7EgWVCNzGxqzj7D5WxiTVfKZuR3+DrNUQ1w9MLes6dlN8ESYBn+SE2Sm45pv5nyeZOdNgh2ESBmglvdPfAqtF72jcnkoEbmLO2iyCfD25sldbvUMxrjVvgoc3xPZs8aZifM2Miixl/uEwKlX9odoNvh9KsmDXd3pH4vZUInADhSerWLztGBP6xOPvbZz57YZy4jDs+BJi0sDT1y6bnJhYxPEaTxbnhdhle8015W//ZsqUKbrGUKvOV0CbFNXBzABUInAD8zNyqLao5jP1WjdDmzEU189umxwSdpJ2/lXM0bk89d6D2ezdu1fXGGpl8oDB90HuBshep3c0bk0lAhdnsUo+XZfN4HbhdIgK1DscY6oqg4z3IbIT+NnvRLpJwM0JRWwqCWBHqX1GGS6n183gG6pGBTpTicDF/bo3n9zjFUwarEYDddo8F6pPNmsBWUOubXscH5NVNa2pi3cA9Lsddi+C4oN6R+O2VCJwcXPSs4gK8mFU12i9QzEmi1n7NhocD8H2P5Ee6m3hytgSvjkaSplZ/Xer1YApIDxg3bt6R+K21F+mC8spLmf53gJuHJCIl4f6p67V7u/gRC4kNL64XFNNTCii3OLBV0f0qT/Uq2tHevXqpcu+GyU4FnpcC5vmQEWJ3tG4JfXp4MI+WZeNSQhuGqDqCtVKSq2chF8bCE912G56hlSQFlzOHJ2a1kx79mGmTZvm/B03xaB7oeaU6musE5UIXFSV2cL8jBxGdokiNsRP73CMKWcdHNmkzRQSjv2vMDGhiMxTvqw/rsop1Co2DVIu1GZvqb7GTqcSgYtavO0YxaeqmTQoWe9QjGvNm1pJ5JgeDt/VuNgSgj3NukwlnfiXfzBx4kSn77fJBj8AZUdUX2MdqETgouaszSIlIoAh7dVslVoVH4Dd30PbXtpqYgfz8/ijaU2+k5vW5B7NJzc316n7bJYOIyGio3byXvU1dqpGJQIhxFghHDx2Vuxm55FSNmYd55aBqvlMndLf1g4HNaIfsb383rQmV//6Q4ZkMmkLzE7XfFKcprEf7jcCmUKIl4QQXRwZkNJyc9dl4eulms/UqbwYNs+BqC5apVEnaR9QxdCwMublhqmmNXVJu0HrCqf6GjtVoxKBlHIiWvP5/cCHQoh0W09h1XnDYMoqa/h682HGpbUlxN9L73CMaeOHYK50yAKyhkxUTWvq5+UH/e+EvYuhcJ/e0biNRh/usTWeXwh8BsQCVwObhBAPOCg2pRm+2nyY8mqLWklcF3M1rH0X2iRDYJTTdz8yqpRonxqnnjQe3KcHgwcPdtr+Wqz/neDhA2tVX2Nnaew5giuFEF8BPwNewAAp5RigJ/CoA+NTmkBKyZz0LHrGh5AWr5rP1Gr7AjiVr8toALSmNTfGF7OiMIgsJzWtefHx+3jxxRedsi+7CIyCtOthyzw4VaR3NG6hsSOCa4HXpZRpUsqXpZT5AFLKcuB2h0WnNMn6g8Vk5p/kFlVltHZSwpo3ICBSK3+sk9NNaz41QNMawxp8H5grIOMDvSNxC41NBEellCvOvEMI8R8AKeUyu0elNMuctVmE+HkxLk01n6nVgV8gfxfE9292P2J7cHbTmgl3/Y0JEyY4fD92FdVFm066fiaYq/SOxuU1NhGMquW+MfYMRGmZgrIqluw4xrV94/HzblmbRZe15i3wDoSornpH8nvTmh+c0LSm6PgJiopa4SGWwfdrh/G2faF3JC6v3kQghLhHCLEN6CyE2HrG5SCw1TkhKo3x2fpsaiySWwYm6h2KMeXvgv3LtHUDJv27tJ1uWqPKU9ej3XCI6gbp/1ULzBysoRHBp8A44Bvb9elLX9uUUsUAzBYrn6zL5oLUCNpFquYztVrzFpi8oG1vvSMBVNOaRhFCO1eQv1M7rKc4TEOJQEopDwH3AWVnXBBCqDNdBrF0Zx7HSiuZPDhZ71CMqSwPtn0OMd21eeoGcV3ccXxV05r69bgWAqPVAjMHa8yIAGAjkGG73njGz4oBfJyeRVyoHyM6O39efKuwYZbWgCbOcT0HmiPEy8I4W9Oa0hrHVXC5ZGh/LrnkEodt36E8fWDAn7XDenk79Y7GZdX71yelHGu7TpFStrNdn760q++1QogPhBD5QojtZ9z3rBDisBBii+1yuX1+DfeVmVdG+oEiJg5KwkPVFTpfdTlseA/CO4C/8Qaxk5zQtObpv9zB008/7bDtO1y/O7QqsWqBmcM0dkHZUCFEgO32RCHEa0KIhs5KfgSMruX+16WUvWyXH5oWrnKuOWuz8PY0cUN/VVeoVls+gYrjui0ga0iarWnN3Bx9mta0Cv5hWpP7rfPhZL7e0bikxo5H3wHKhRA9gf8DsoA59b3Atu6guGXhKfUpq6xh4cZcxqbFEhbgnFWqrYrVYutHHAch8XpHU6fTTWvWOahpzZjJf2HMmFY+23vQvVrDmvWz9I7EJTU2EZillBK4CpgupZwONLdq1v22KagfCCHqHA/bitplCCEyCgoKmrkr1/bV5sOcqraok8R12b0Ijh/SfQFZQ043rXHUSeOKyioqKiocsm2niegAncZoh/lqWvnvYkCNTQRlQogngInA90IID7SaQ031DtAe6AUcBV6t64lSyplSyn57d4J6AAAgAElEQVRSyn6RkZHN2JVrk1LycXoWafEh9EpQdYXOIyWsnq71I47oqHc09TrdtObHvBCnN61pVQbfBxXF8Ns8vSNxOY1NBDcAVcAdUspjQBzwclN3JqXMk1JapJRWYBZgzAO3rUD6gSL25Z9kkqorVLvstXB4o1P6EdvDLQnFmKVQTWvqkzQUYntpTYWsVr2jcSmN7UdwTEr5mpRype3nbCnlx03dmRAi9owfrwa21/VcpX5z0rMI9fdiXE9VV6hWa6aDlz/EpOkdSaOcblrzqWpaUzchtLITRZmwb6ne0biUxs4aukYIkSmEOCGEKBVClAkhSht4zTwgHegkhMgVQtwBvCSE2CaE2ApcDPy1xb+BGzp6ooL/7czjhv4J+HqpukLnKdgLexZrq4g9Wk9znkmJRRyp9ObngmC7bnfsJcMYO3asXbepm27jtZP/a97UOxKX0tgDki8B46SUuxq7YSnlTbXc/X5jX6/U7dN12VilZOJAdVioVumny0k4rx+xPYyM1JrWzM0JY1RUvd+zmuTRuyZCgrEW0zWbhxcMvAuW/kPrbRzbU++IXEJjD57mNSUJKI5TbbYyb30OIzpFkRDmr3c4xnMyXzuZGN0NvB0zHdNRPG1Na34tDHZa05pWqc+tWhXZdLXAzF4amwgyhBCfCyFush0mukYIcY1DI1NqtXj7UQpPVqlWlHVZN0Obb27QBWQNuSm+CA8h+TjbflNJh19/N8OHD7fb9nTnFwq9J8H2hVB6RO9oXEJjE0EwUA5cyh8VSF3koGPrIaXk/VUHaRcZwIWpakrteapPaXWFIlINWU6iMWJ8zVwefYL5h8M4aTb+bCfdDLobpFVL/EqLNXbW0J9quagWlU62Kfs4W3NP8KehKZhUXaHzbZ4LlSda7WjgtNuTCigze7DgsOPqD7V6bZKhyzjY+CFUlekdTavX2FlDHYUQy04XkBNCpAkhnnJsaMq5Plh1iGBfTyb0idM7FOOxmLWZJMHxhi4n0Ri9QyvoHXKKD7MisKqppHUb8hct8Wd8qHckrV5jx56zgCeAGgAp5VbgRkcFpZwv93g5i7cf5aaBifh7q9Wn59n1DZzI0cpJuIDbkwrJqvDh54LmVnJxA/F9IeVC7aSx6mvcIo1NBP5SyvXn3Ge2dzBK3eakZyGEUHWFaiMlrHwV/CMMX06isUZHnyDWt5oPsyJavK3rx47k+uuvt0NUBjTsYTh5TJWdaKHGJoJCIUR7QAIIIa5FqxWkOMGpKjPz1mczunsMcaHG6bBlGJlLIW8HJAwwdHG5pvAyweSEIlYXB7G7rGWtLO+dfC333nuvnSIzmHbDtbITq6dr1WaVZmlsIrgPmIHWxP4w8BBwt8OiUs7y5aZcSivN3D40Re9QjEdKWPEy+IZqjc5dyE3xxfiarC0eFZRXVFJeXm6nqAxGCLjgYSg+ADu/0TuaVqveRCCEeFgI8TAwHvgBeAF4F/gSmOD48BSrVfLh6kP0TAilT6KqMnqerDWQu147N2ByrXIbod4Wrml7nK+OhlJU3fzf7fJbH+Lyy124GWDncRCeCqteQ3X3aZ6GRgRBtks/4B6gDRCKNhro6tjQFIBfMws4UHiK24cmI1zksIddrXxVW2XaSorLNdWfkgqptpr4VDW4r5vJBEP/Ase2wb5lekfTKjXUs/g5KeVzQATQR0r5qJTyEaAv0Lrn6LUSH6w6SHSwD5f3iG34ye7myBatqXlc31ZVXK4pUgOruDC8jI+zw6myqi8CdUq7QStGt+p1vSNplRp7jiARqD7j52og2e7RKGfZeaSUlZmFTB6cjJeHWmV6nlWvgaevVmXUhf05uYCCai++OaIODdbJ01srUZ21CnLOneCoNKSxny5zgPVCiGeFEM8A64DZjgtLAZixYj8B3h5MVM1nzlewF3Z+q1UY9WzZrBqjGxZ+kq5BFcw4FKkWmNWnz2TwC9MmDyhN0tgSEy8AfwKOAyXAn6SULzoyMHeXU1zOoq1HuXlgIiF+rnnYo0VWvw4mT60DmYsTAu5KKWD/KV+WNaNXwW3XjeW2226zf2BG4xOotbPM/J/WnU5ptEYfb5BSbpJSTrddNjsyKAXeX3UQk4Dbh6kpo+cpPghb50NsGni7RynuK6JLiPOtZsbBphcbdJtEADBgijaV+Fc1KmgKdeDZgI6fqubzDTlc1SuO2BC1gOw8K1/VrhMG6RuHE3matHMFGSUBZBxvWvIrLC6hsLDQQZEZjG+wdq5g72JtMoHSKCoRGNDH6VlU1FiYcmE7vUMxnuKDsOVTrTOVj3vV4bk+rphQLzMzDjVtVHDt3Y9z7bXXOigqAxo4BXxD4NeX9I6k1VCJwGAqqi3MTj/EJZ2j6BjtXh90jbLyFe2guRuNBk7z95RMTixiaX4I+0766B2OcfmGwKB7Yc/3cHSr3tG0CioRGMyCjTkUn6rmrova6x2K8RQfhC3z3HI0cNqtiYX4mKzMauKowO0MvBt8QmCFGhU0hkoEBmK2WJm18iC9E0Ppn6yakpzHjUcDp4V7W7g+rpivjoSSV6nKkdfJL1TrYrbrOzi2Xe9oDE8lAgNZvP0Y2cXl3H1Re1VO4lxqNPC7O5MLMUvBB3YoUe3SBt4N3kFqVNAIKhEYhNUqeevnfbSPDGBUl2i9wzGeFWo0cFqSfzXjYkuYkxNOcSOK0d0zcQL33HOPEyIzGP8wGHiXVpVUjQrqpRKBQSzZcYw9eWU8eEmq6kd8ruKDWuMRNRr43f3t8qmwmHi/EaOCG64cxQ033OCEqAxoyP3auYKf/6V3JIbmsEQghPhACJF/us+x7b4wIcRSIUSm7VodCEcbDUxflkm7iADGprXVOxzjWf4iCJMaDZwhNbCKy2NOMDsrgpIGRgU5R/LIyclxUmQG49cGhj6orSvIXqd3NIblyBHBR8Doc+57HFgmpUwFltl+dntLd+Wx+1gZ94/ogIcaDZzt2HZtFXHbPmo0cI4H2uVz0uLR4LmCSQ89w6RJk5wUlQENugcComDZ86pfQR0clgiklCuA4nPuvoo/itXNRmt449aklLyxLJPkcH+u7KlGA+f5+XmtqFyiGg2cq3NQJWOiS/gwO4ITNeoob528A+DCx7TKpPtVv4LaOPuvJ1pKeRTAdh1V1xOFEFOEEBlCiIyCggKnBehsP+/OZ8eRUu67uAOeqtT02bLSYe8SrRexlyq1UZv72+VTZvbgIzWDqH59b4PQRDUqqINhP3mklDOllP2klP0iI11z8YyU2rmBhDA/xveO0zscY5ESlv5DOxzkBhVGm6tbcCWjok7wflYEZWbD/nfWn6c3DH8Cjv6mehvXwtl/OXlCiFgA23W+k/dvKMv3FrA19wT3X9xBNZ45194ftV7EiUNctvuYvTzYLp9SsycfZ6t2lvVKuwEiO2sziCxmvaMxFGd/+nwL3Gq7fSvgtqlZSsn0nzKJC/Xj6t6q6+dZrBb46VnwD3fZXsT21COkghGRpcw6FMnJWkYFj/z5Fh555BEdIjMYkweMeAqKMrXpyMrvHDl9dB6QDnQSQuQKIe4ApgKjhBCZwCjbz27p5935bMkp4b6LO+DtqUYDZ9k6Hwp2Q9Iw7T+v0qCH2udRUuPJ+4fOP1cwbtQFjBs3ToeoDKjzWK3H9S8vQPUpvaMxDEfOGrpJShkrpfSSUsZLKd+XUhZJKS+RUqbars+dVeQWrFbJy0v2kBzuz3X91GjgLDWV8Mu/IChWG8YrjZIWUsHoqBPMOhR53mrjPfuz2LNnj06RGYwQcOkLUHYUVr+hdzSGob6K6uC7rUfYfayMv47qqM4NnCv9LTiRCykXaf9plUZ7JPUY5RYT7xw8ezLeXU+8yF133aVTVAaUNBi6jofV0+HEYb2jMQT1KeRkNRYrry3dS5fYYMapVcRnKz2qdR+L6AhtkvWOptVJDazi6rbHmZ0dztFKdYK9XqOeA2nRppMqKhE42+cbcsgqKuexyzqqmkLnWvYcWKqh3Qi9I2m1HuqQh5Twxv46l+gooH3RGHQvbP0MDm/SOxrdqUTgRBXVFt5Ylkm/pDZc3En9Rz1L7kZtJkdcP62WvNIsCX413JxQzPzDYew/pbqY1euCRyAgEpY86faLzFQicKJZKw+QX1bF38Z0Vv0GziQl/Pg38A6ExMF6R9Pq3d8uHz+Tlal7Y/QOxdh8g+Hiv0N2utsvMlMtjpwkv7SSd3/dz5juMfRPDtM7HGPZtgByN0Cny8FTfYttqUgfM/e0y+flzFjWFgfw1AO3Q1QnvcMypj6TYf0sbRV7x9Hg5at3RLpQIwIneW3pXmosVh4fo6ZEnqX6FCx9WpsuGt1D72hcxh1JhcT6VvOvPbGMGNafkSNH6h2SMZk8YPS/oSQL1rypdzS6UYnACXYfK2V+Rg6TByeTFB6gdzjGsmqaNqe7/Qg1XdSOfD0kj6UeY3upP9N/PsSWLVv0Dsm42g3XppOufAWOH9I5GH2oROAEL3y/iyBfLx4Y0UHvUIylMBNWvw5RXSEkQe9oXM742BJ6BJfz1hYzDz78qN7hGNtl/wbhAYv/pnckulCJwMF+2pnHysxCHrwklVB/b73DMQ4pYdHD2n++9pfoHY1LMgl4qtNRLL4hnGg7QO9wjC0kDi5+Qit2uPsHvaNxOpUIHKiyxsJzi3aQGhXI5MFJeodjLFvnw6EV2gpib3W4zFEGhp0iIH8rJ9oOJLuoXO9wjG3g3RDZRRsVVLvXe6USgQO9++t+cooreO6qbqqUxJkqjsOSJyA4DmJ76R2Ny2uzfwlCWnl+0Q69QzE2Dy8Y+xqcyIbl/9Y7GqdSn04OklNczjvL9zM2LZYh7VX3qLP87x9aMki9VJ0gdgLP6jJCctP5aVc+v+x26xYgDUsaonUzS/+vW604VonAQZ5ftBMPk+DvV3TROxRj2f8LbP4Y4vpDYLTe0biFf//fvbx29zjC/L155tsdVJkteodkbKOe1/42v30ALDV6R+MUKhE4wNKdeSzdmccDI1KJDVG9dn9XdRK+vV9rOJM8TO9o3MaQfmlcMGwoF3aMILu4nLd/2a93SMbmGwJXvAp522H1NL2jcQqVCOysrLKGp7/eTqfoIO4YlqJ3OMby07Na2d+OY1T7SSdak7GV9DVrSAoPoFN0EP/9ZR9788r0DsvYOl8B3a6GX1+C/N16R+NwKhHY2ctL9pBXVsnUCT1U57EzHVoNG2Zp3aFCVDMeZ3rypbd5+qmnALiwYwTeniYeX7gVq9W9C601aMxLWv2rr+5y+UNE6pPKjjZmFTNnbRa3DUmmd2IbvcMxjspS+Ppu8GsDKRfqHY1b8/f2ZFiHCDZll/DJuiy9wzG2wCgYNw2OboEVr+gdjUOpRGAnVWYLjy/cRtsQPx69VBX4OsuPf4OSHOh0BXioRXV66xwTRGKYP1MX7+ZISYXe4Rhb16sg7UZY8bJWKt1FqURgJ9N+yiQz/yT/Gt+dAB9V1PV3O76GLZ9q5aXVISFDEEIwonMUNRbJYwt+U4eIGjLmPxAUA19NcdmFZioR2EHGoWJm/LqfG/sncHFn1XDmd6VH4LsHIagtJA3VOxrlDCF+XgzrEMHqfUXMVYeI6ucXCuPfhqJ92kJIF6QSQQudrDLz8PzfiGvjx1Nju+odjnFYLfDlFKipgM5jtXK/ii6mPfMwr7z22nn3d48LJjncnxe+38XBwlM6RNaKtBsOQx+CjR/B9i91Dsb+VCJooRe+30XO8XJeva4XgeqQ0B9+/Q8cWgkdRoK/asSjp17dOtKr1/mlPIQQXNI5GpMQ/PXzLZgtVh2ia0VGPAXxA+C7v0DxAb2jsSuVCFrgfzuOMW99NlMubMeAFPVh97t9y7T519E9ICZN72jc3k8r17Psp59qfSzQ15OLOkayJaeEN37e5+TIWhkPL7j2fa0syoLbwVytd0R2o0siEEIcEkJsE0JsEUJk6BFDS+UeL+fRL36jR1wID4/qqHc4xlF6BBbeqTUFT71U72gU4F9vfsCL/667iFqnmCC6xAbx5rJM1uwrdGJkrVBoIlz1NhzZ7FLnC/QcEVwspewlpeynYwzNUmOx8sC8zVglvHVzb3w81fFvAMxVMP9WqD6pTbtTq4dbjeEdowgL8ObBzzZTUFaldzjG1mUsDHkQNrwHm+boHY1dqENDzfDKkj1szi5h6oQeqvXkaacbzeSu15rQ+4frHZHSBN6eJkZ3j6GkvIa/fr5FTSltyCXPaCeQv3/YJdYX6JUIJPA/IcRGIcQUnWJolqU785ix4gC3DExkbFpbvcMxjnUzYMtcSBwCkZ31jkZphohAHy7qGMmqfYVMX5apdzjG5uEJ136orS/4fCKU5ekdUYvolQiGSin7AGOA+4QQ59UdEEJMEUJkCCEyCgoKnB9hLfbmlfHQZ5tJiw/haTVV9A/7f4ElT0J4KiRfoHc0Sgt0axtMl9ggpi/L5Mftx/QOx9j8w+DGT6GyBD67qVUvNhNS6jsEFEI8C5yUUtZZzKNfv34yI0Pfc8ol5dVc9d/VnKqy8N0DQ1V56dPydsIHl4KnL/SaCJ4+ekeknGOPSKUmqhtLDzfuXJbZYuXLzYcpKa/h6/uG0ikmyMERtnK7v4fPbtEqll7/saHWzAghNjbmPKzTRwRCiAAhRNDp28ClwHZnx9EUZtvJ4SMlFcyY1EclgdNOHIa512i3u1+rkoBBdWqfRKdOja9/5elh4vIesXiYBHfO3kBJuetMk3SIzlfA6Bdh9yJY+g+9o2kWPQ4NRQOrhBC/AeuB76WUP+oQR6NIKXn2ux2szCzkn1d1p2+SWi8AQEUJzJ0AFcVaEvAN0TsipQ7fLV3Jou++a9JrAn08ubxHDEdOVPLnjzOorFFdzeo16B4YeDekvwXpb+sdTZM5PRFIKQ9IKXvaLt2klC84O4ameHv5fuauzeaui9px44BEvcMxhupy+OxmKNwDXa9WLScN7tVZnzDt9deb/LrYED8u7RrNhkPHeXi+mknUoMv+DV3GaesLNs/VO5omUdNH67FgYy4vL9nD+F5t+dtlaiYMADWV2vHQrDVaDaE2yXpHpDhQx+ggLugQwQ/bjvHCD7v0DsfYTB4w4X1oP0Lrd7zjK70jajSVCOrwy+58Hl+4laEdwnnp2p6YTELvkPRnroYvboUDP0OnMRClZk65g96JofRKCOX9VQd5Z7nqd1wvTx+4Ya5Wk2jhn2HvEr0jahSVCGqxYm8Bd83dSOfYIN6d2Fe1nAQtCSy4Hfb+CKmXqRpCbkQIwQWpEXSMDuQ/P+7m/VUH9Q7J2LwD4Jb5EN1NGz3v/l7viBqkPuHOsWZ/IX/+OIN2EQHMvWMgQb6qTIJ2TuAm2P0dtB8JbXvrHZHiZCYhuKxrDB2iAvnnop3MWat6GNTLNwQmfwOxPWH+ZMOXrlaJ4AxrDxRxx0cZJIb588mdAwn1V20VqSzVpojuWwYdx0B8qysN5fbmTHuOD2fPbvF2TCbB6G4xtIsI4OmvtzNXJYP6+YXC5K9th4nugM2f6B1RnVQisFm6M4/JH6wnro0fn/x5IOGBak48ZXkweyzkrIMuV2rfbpRWJ6FtNAkJCXbZlodJMKZHDCkRATz19XbeXr4PvRelGppPEExcACkXwjf3wq8va3W5DEYlAmDhxlzunruRLjFBzL9rMFFBvnqHpL+8nTDrYsjfBd2ugaguekekNNPn3y5l/vzP7bY9T5OJK3rE0ik6iJd+3MPUxbtVMqiPdwDc/AWk3QC//EtrbGMx6x3VWdy6pZaUkhkrDjB18W6GdghnxqR+qssYwL6ftOOawgN63qwV1lJarXfmLkR6BzH+qZl226aHSXBZt2h8PE3MWHGA/LIqpk7ooUqy18XTG66eASHxsPJVOJGjTTU1SPc+tx0RVJktPPrFVqYu3s3YtFg+uK2/SgJSwqrX4ZPrwDsQek9USUCpkxCC4Z0iGdQujK82H+bmWesoOql6GdRJCLjkHzDuDTi4Uhtx5+3QOyrATRNB4ckqbpm1joWbcnloZCpv3qSay1B5Qlst/NOzENEJet0CPsF6R6UYnBCCgSnhjOkew285JVz51mp2HyvVOyxj63sr/OkHbXHmeyNh2wK9I3K/RLD2QBGXT1/JtsMnePOm3jw0siNCuPlisdwMmHGhtkag/UjtxLCHmjGlNF7H6CAm9I3nREUNV721mvkZOXqHZGwJA+CuX7X1OAvvgK/vhaqTuoXjNonAapW89XMmN89aS6CPJ1/dO5RxPd28sYylBn55Ed6/FCqOa+cD4vtpQ1hFaaKYYF9u7J9AVJAP/7dgK4/M30J5tbFOihpKUAzc9j1c+Bhs+VT7MnZYn25nuvcjaIyW9iPILirnsQW/se5gMVf1assLV/dQ5wOObYdv79eacEd3gw6jtJ4CisspDOtHTUxPPtta4pT9WaVk/cFi1h8sJjHcn9eu76mq9jbk0Cr4cgqUHYUhD8DwJ8Cr5eXuDduPwJmsVsnH6YcYPX0FO4+U8tK1aUy7oZd7J4Hqclj6DMy8CAr2Qtfx0HmcSgIuLCIslIiICKftzyQEg9qFc02fOErKa7j2nXRe+H6nKmVdn+RhcM8abYLG6unwzlDthLKTuPSI4IkvtzJvfQ4Xdoxk6jU9aBvqxg1lpISd38DSp6EkWzs22e5iu3zrUIzto82VWEKTKE0Y5vR9V5utrNpXyLbDJ0gI8+O5K7sxorMqW16vA8u1tQbHD2m9Pi79FwTHNmtTjR0RuHQi2JR9nMy8Mq7vl+DeJ4RzM7SewjnrIDBKOyEcqnoruIvhTy6w+zqCpsopLmf53gKKT1VzSeco/jGuK0nhAbrFY3jV5bB6Gqya9sfK5GZobCJw6WMkfRLb0Cexjd5h6OfIZlg+VZsN5B2o1QqK6QHCpY8IKgaUEObPzQMS2ZJTwsp9hYx49VduHpDIAyM6EBWsDkuex9sfLn4S+v8ZAiMdvjuXTgRuSUqtaczq6ZC5RDv0k3wBxPVTPYUVXXmYBH2T2tA5Joh1B4v5ZF0W8zNymDw4iTsvaEe0Sgjnc0ISAJUIXEdNJez6VuuZevQ38PK3JYC+6kSwYigBPp6M6BxFn8RQ1h0s5r1VB/lw9SGu6RPHlAvb0SEqSO8Q3Y5KBK2ZlHBkkzYHedsX2upg/witcUx0d/BQvRQU4wr19+aybjEMahfOpqzjfLnpMPMzchmYEsbEQUlc1i1GNYVyEpc+WeyyCvbC7kWw9XMo2A0mL4hI1WYChSapBWHKWcrjhlET25sP1h7RO5R6lVeb2XmklB1HSimpqCHEz4vLe8QwLq0tA9uF46HaxTaZOlnsSixmbcXhnu9h1yIotvWNDY7Tvv1HdVGHf5Q6+fv5UuPvr3cYDfL39qRfchh9k9qQXVzOrmNlLNx4mHnrc4gI9GZsWlsu6RJF/+QwfL3cvDaYnakRgRFZLXBsq7ba8OBKyFoN1Se12T6hSRDeAcJTwVcVhVMa9vaaEiwhiZg7jdI7lCarsVg5VHiKvXknOVR0CrNV4uNpYlC7cC7sGMmwDhGkRgViUqOFWqkRQWthtULJIW2q55HNcHgzHN0M1ae0x/0jtA/+0CQIS1Hf/JUmm7/oJ9s6gtaXCLw8TKRGB5EaHUSNxUru8Qqyik6x7fAJft1bAECgjye9E0Ppk9iG3omhdI8LIUJ1GGwSXRKBEGI0MB3wAN6TUk7VIw6nqjqprRQs3AuFmbZr221zhfYck6e24CuiIwTHa4u+fNQMCkUBLSmkRASQEqEtRCutqCH3eAVHSyvYfayMVfsKf+8CGRbgTZeYIDrFBNMxOpDEcH8Sw/yJDfFT5xpq4fREIITwAP4LjAJygQ1CiG+llDudHUuLSQmVJVBeDOVFf1zKjkHpYTiR+8el6swa7QL82mjNraO7Q0A4BMZAQCSY1LFPRWmMYD8vuvp50bWtdoi02mwlr7SSgpNVFJ2sZn/BKdYfKqbG8sfhbw+TIC7Uj8QwfxLC/IgK8iUyyOePS6B27W7nIPQYEQwA9kkpDwAIIT4DrgLsnwhqKrSl2pZqsFRpZZfNVbafbRdz9TmPV2rf3qtPQXWZdl11UjtGX30Sqk5B1QmoKIby4yDrKKTl5a8dw/cO1A7t+ASDbwj4h2tJQE3tVBS78vY0kRDmT0LYHyfGrVJSVmnmREUNpRU1v1/vyy9jc/ZxTlXX/v/Xx9NEkK8nwb5ehPh5EXz64utJoI8nvl4e+Hp54Odlws/b4/eftfs88PE04ekh8PIw4WESeJm0nz1NAk+PM26bTHh5CN1L4OiRCOKAM7tW5AIDHbKnJU9CxgfNf73JUzsm7+kNHj7ah7fJS2vaEpoEkV20xtRe/trF+/R1gGrsohiI9iHj5+1e33JPC/DxJCak9nNrFqukvNrMqSoLp6rMnKwyc6raTFWNlUqzhaoaK8fLq8krqzzrPrPVvpNsTEKr2moSApNJu+0hBCaT4O1b+jC0g2Orx+qRCGpLfee9q0KIKcAUgMTEZhZI6zoeIjtrH8oe3lqJBQ8v7UPd03bf6Q94T58/nuMdoH2TV9/aFRewPP0hvUNwOVarpNJsobLGSkWNhcoaCxXVFqrMFiqqrVSZLdRYJBarxGy12m5r12aLlkjM1jNuWyRWKbFIiZRagrJKidUqiQ52/IlvPRJBLpBwxs/xwHkrXaSUM4GZoE0fbdae2l2kXRRFUezIZBL4e3vi7yIDfz3Wb28AUoUQKUIIb+BG4Fsd4lAURVHQYUQgpTQLIe4HlqBNH/1ASrnD2XEoiqIoGl3WEUgpfwB+0GPfiqIoytlUaT9FURQ3pxKBoiiKm1OJQFEUxc2pRKAoiuLmVCJQFEVxc62iH4EQogDI0jsOB4oACvUOwuDUe1Q/9f40zB3foyQpZWRDT2oVicDVCSEyGtM8wp2p96h+6v1pmHqP6qYODSmKorg5lQgURVHcnEoExqmH3R0AAAYQSURBVDBT7wBaAfUe1U+9Pw1T71Ed1DkCRVEUN6dGBIqiKG5OJQInEUKECSGWCiEybddt6njerbbnZAohbrXd5y+E+F4IsVsIsUMIMdW50TuOEGK0EGKPEGKfEOLxWh73EUJ8bnt8nRAi+YzHnrDdv0cIcZkz43am5r5HQohRQoiNQohttusRzo7dWVryd2R7PFEIcVII8aizYjYUKaW6OOECvAQ8brv9OPCfWp4TBhywXbex3W4D+AMX257jDawExuj9O9nhPfEA9gPtbL/Xb0DXc55zL/Cu7faNwOe2211tz/cBUmzb8dD7dzLYe9QbaGu73R04rPfvY7T36IzHFwJfAI/q/fvocVEjAue5Cphtuz0bGF/Lcy4Dlkopi6WUx4GlwGgpZbmU8hcAKWU1sAmts1trNwDYJ6U8YPu9PkN7n8505vu2ALhEaJ2+rwI+k1JWSSkPAvts23M1zX6PpJSbpZSnu//tAHyFEI7ve+h8Lfk7QggxHu1Ll9v2RVGJwHmipZRHAWzXUbU8Jw7IOePnXNt9vxNChALjgGUOitOZGvx9z3yOlNIMnADCG/laV9CS9+hME4DNUsoqB8Wpp2a/R0KIAOBvwHNOiNOwdGlM46qEED8BMbU89PfGbqKW+36f1iWE8ATmAW9IKQ80PULDqff3beA5jXmtK2jJe6Q9KEQ34D/ApXaMy0ha8h49B7wupTxpGyC4JZUI7EhKObKux4QQeUKIWCnlUSFELJBfy9NygeFn/BwPLD/j55lAppRymh3CNYJcIOGMn+OBI3U8J9eWCEOA4ka+1hW05D1CCBEPfAVMllLud3y4umjJezQQuFYI8RIQCliFEJVSyrccH7ZxqENDzvMtcKvt9q3AN7U8ZwlwqRCijW1W0aW2+xBC/Avtj/chJ8TqLBuAVCFEihDCG+0k3rfnPOfM9+1a4Gepnd37FrjRNhskBUgF1jspbmdq9ntkO4z4PfCElHK10yJ2vma/R1LKC6SUyVLKZGAa8G93SwKAmjXkrAvaMdtlQKbtOsx2fz/gvTOedzvaic99wJ9s98WjDWN3AVtslzv1/p3s9L5cDuxFm/Xxd9t9zwNX2m77os3m2If2Qd/ujNf+3fa6PbjALCp7v0fAU8CpM/5mtgBRev8+RnqPztnGs7jprCG1slhRFMXNqUNDiqIobk4lAkVRFDenEoGiKIqbU4lAURTFzalEoCiK4uZUIlDchhDiWSHEo0KI54UQI233XWCr6LpFCOEnhHjZ9vPLeserKM6iVhYrbkdK+Y8zfrwFeEVK+SGAEOIuIFI2siaPEMJTarVrFKXVUusIFJcmhPg7MBmt4FgBsBGtJPMitJICL6EVIFsDBAFXANuAF4GfgXeBRNvmHpJSrhZCPAu0BZKB/2/vjl2jCMIwDv9eQUTkYm9hKaggCSGFGCFgsBeENFoJoqD+HbYiKYTUQkAQxCYpg2kSPCSVrSBYmCaiOQzIazFzcBx3yOFds/s+1eze7Oxy3O7HzM59cwjcB55T0oOcAdZtv5K0QvmT0mE950fgnm1LWgJeAOeA38At4HhUO9P+TiKGpUcQjSVpkZJuYIHyW+9SHsYA2N6QtAy8t/2mHvPT9nwtv6YkJPsg6SIl3cflevgisGy7J+khcGR7qaZ53pW0XestAFcpuW92gRuS9oBNYM32vqQ5oAc8GNWOS5rtiJlJIIgmuwm8tX0MIGk4/8y/rAJXBrJSzknq1PI7271avg1ck3S3bp+n5D46AfZsf63n/0TpRRwB32zvA9j+UT8f104CQcxUAkE03f+MfZ4Crg888AGogeHX4C7gqe2toXorlGGfvj+Ue05jrmtkOxGzlllD0WQ7wJ06G6hDWdBnEtvAk/6GpPkx9baAx5JO13qX6oIn43wGLtT3BEjq1NTIk7YTMRXpEURj2e5K2qRk3fxCWet5Es+AdUkHlHtlB3g0ot4GZcinW5c//M7opUj713UiaQ14Keks5f3A6qTtRExLZg1FRLRchoYiIlougSAiouUSCCIiWi6BICKi5RIIIiJaLoEgIqLlEggiIlougSAiouX+AjRc09ZQ9e7EAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "0.44122379261151545"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "power(.1, .12, 1000)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "You should see this message if all the assertions passed!\n"
+ ]
+ }
+ ],
+ "source": [
+ "assert np.isclose(power(.1, .12, 1000, plot = False), 0.4412, atol = 1e-4)\n",
+ "assert np.isclose(power(.1, .12, 3000, plot = False), 0.8157, atol = 1e-4)\n",
+ "assert np.isclose(power(.1, .12, 5000, plot = False), 0.9474, atol = 1e-4)\n",
+ "print('You should see this message if all the assertions passed!')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Method 2: Analytic Solution\n",
+ "\n",
+ "Now that we've got some intuition for power by using trial and error, we can now approach a closed-form solution for computing a minimum experiment size. The key point to notice is that, for an $\\alpha$ and $\\beta$ both < .5, the critical value for determining statistical significance will fall between our null click-through rate and our alternative, desired click-through rate. So, the difference between $p_0$ and $p_1$ can be subdivided into the distance from $p_0$ to the critical value $p^*$ and the distance from $p^*$ to $p_1$.\n",
+ "\n",
+ "
\n",
+ "\n",
+ "Those subdivisions can be expressed in terms of the standard error and the z-scores:\n",
+ "\n",
+ "$$p^* - p_0 = z_{1-\\alpha} SE_{0},$$\n",
+ "$$p_1 - p^* = -z_{\\beta} SE_{1};$$\n",
+ "\n",
+ "$$p_1 - p_0 = z_{1-\\alpha} SE_{0} - z_{\\beta} SE_{1}$$\n",
+ "\n",
+ "In turn, the standard errors can be expressed in terms of the standard deviations of the distributions, divided by the square root of the number of samples in each group:\n",
+ "\n",
+ "$$SE_{0} = \\frac{s_{0}}{\\sqrt{n}},$$\n",
+ "$$SE_{1} = \\frac{s_{1}}{\\sqrt{n}}$$\n",
+ "\n",
+ "Substituting these values in and solving for $n$ will give us a formula for computing a minimum sample size to detect a specified difference, at the desired level of power:\n",
+ "\n",
+ "$$n = \\lceil \\big(\\frac{z_{\\alpha} s_{0} - z_{\\beta} s_{1}}{p_1 - p_0}\\big)^2 \\rceil$$\n",
+ "\n",
+ "where $\\lceil ... \\rceil$ represents the ceiling function, rounding up decimal values to the next-higher integer. Implement the necessary variables in the function below, and test them with the cells that follow."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def experiment_size(p_null, p_alt, alpha = .05, beta = .20):\n",
+ " \"\"\"\n",
+ " Compute the minimum number of samples needed to achieve a desired power\n",
+ " level for a given effect size.\n",
+ " \n",
+ " Input parameters:\n",
+ " p_null: base success rate under null hypothesis\n",
+ " p_alt : desired success rate to be detected\n",
+ " alpha : Type-I error rate\n",
+ " beta : Type-II error rate\n",
+ " \n",
+ " Output value:\n",
+ " n : Number of samples required for each group to obtain desired power\n",
+ " \"\"\"\n",
+ " \n",
+ " # Get necessary z-scores and standard deviations (@ 1 obs per group)\n",
+ " z_null = stats.norm.ppf(1 - alpha)\n",
+ " z_alt = stats.norm.ppf(beta)\n",
+ " sd_null = np.sqrt(p_null * (1-p_null) + p_null * (1-p_null))\n",
+ " sd_alt = np.sqrt(p_null * (1-p_null) + p_alt * (1-p_alt) )\n",
+ " \n",
+ " # Compute and return minimum sample size\n",
+ " p_diff = p_alt - p_null\n",
+ " n = ((z_null*sd_null - z_alt*sd_alt) / p_diff) ** 2\n",
+ " return np.ceil(n)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "2863.0"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "experiment_size(.1, .12)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "You should see this message if the assertion passed!\n"
+ ]
+ }
+ ],
+ "source": [
+ "assert np.isclose(experiment_size(.1, .12), 2863)\n",
+ "print('You should see this message if the assertion passed!')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Notes on Interpretation\n",
+ "\n",
+ "The example explored above is a one-tailed test, with the alternative value greater than the null. The power computations performed in the first part will _not_ work if the alternative proportion is greater than the null, e.g. detecting a proportion parameter of 0.88 against a null of 0.9. You might want to try to rewrite the code to handle that case! The same issue should not show up for the second approach, where we directly compute the sample size.\n",
+ "\n",
+ "If you find that you need to do a two-tailed test, you should pay attention to two main things. First of all, the \"alpha\" parameter needs to account for the fact that the rejection region is divided into two areas. Secondly, you should perform the computation based on the worst-case scenario, the alternative case with the highest variability. Since, for the binomial, variance is highest when $p = .5$, decreasing as $p$ approaches 0 or 1, you should choose the alternative value that is closest to .5 as your reference when computing the necessary sample size.\n",
+ "\n",
+ "Note as well that the above methods only perform sizing for _statistical significance_, and do not take into account _practical significance_. One thing that should be realized is that if the true size of the experimental effect is the same as the desired practical significance level, then it's a coin flip whether the mean will be above or below the practical significance bound. This also doesn't even consider how a confidence interval might interact with that bound. In a way, experiment sizing is a way of checking about whether or not you'll be able to get what you _want_ from running an experiment, rather than checking if you'll get what you _need_."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Alternative Approaches\n",
+ "\n",
+ "There are also tools and Python packages that can also help with sample sizing decisions, so you don't need to solve for every case on your own. The sample size calculator [here](http://www.evanmiller.org/ab-testing/sample-size.html) is applicable for proportions, and provides the same results as the methods explored above. (Note that the calculator assumes a two-tailed test, however.) Python package \"statsmodels\" has a number of functions in its [`power` module](https://www.statsmodels.org/stable/stats.html#power-and-sample-size-calculations) that perform power and sample size calculations. Unlike previously shown methods, differences between null and alternative are parameterized as an effect size (standardized difference between group means divided by the standard deviation). Thus, we can use these functions for more than just tests of proportions. If we want to do the same tests as before, the [`proportion_effectsize`](http://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportion_effectsize.html) function computes [Cohen's h](https://en.wikipedia.org/wiki/Cohen%27s_h) as a measure of effect size. As a result, the output of the statsmodel functions will be different from the result expected above. This shouldn't be a major concern since in most cases, you're not going to be stopping based on an exact number of observations. You'll just use the value to make general design decisions."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "3020.515856462414"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# example of using statsmodels for sample size calculation\n",
+ "from statsmodels.stats.power import NormalIndPower\n",
+ "from statsmodels.stats.proportion import proportion_effectsize\n",
+ "\n",
+ "# leave out the \"nobs\" parameter to solve for it\n",
+ "NormalIndPower().solve_power(effect_size = proportion_effectsize(.12, .1), alpha = .05, power = 0.8,\n",
+ " alternative = 'larger')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lessons/StatisticalConsiderations/solutions/L2_Non-Parametric_Tests_Part_1_Solution.ipynb b/lessons/StatisticalConsiderations/solutions/L2_Non-Parametric_Tests_Part_1_Solution.ipynb
new file mode 100644
index 00000000..7114b85e
--- /dev/null
+++ b/lessons/StatisticalConsiderations/solutions/L2_Non-Parametric_Tests_Part_1_Solution.ipynb
@@ -0,0 +1,515 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Non-Parametric Tests Part I\n",
+ "\n",
+ "Up until now, you've been using standard hypothesis tests on means of normal distributions to design and analyze experiments. However, it's possible that you will encounter scenarios where you can't rely on only standard tests. This might be due to uncertainty about the true variability of a metric's distribution, a lack of data to assume normality, or wanting to do inference on a statistic that lacks a standard test. It's useful to know about some **non-parametric tests** not just as a workaround for cases like this, but also as a second check on your experimental results. The main benefit of a non-parametric test is that they don't rely on many assumptions of the underlying population, and so can be used in a wider range of circumstances compared to standard tests. In this notebook, you'll cover two non-parametric approaches that use resampling of the data to make inferences about distributions and differences."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "% matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Bootstrapping\n",
+ "\n",
+ "Bootstrapping is used to estimate sampling distributions by using the actually collected data to generate new samples that could have been hypothetically collected. In a standard bootstrap, a bootstrapped sample means drawing points from the original data _with replacement_ until we get as many points as there were in the original data. Essentially, we're treating the original data as the population: without making assumptions about the original population distribution, using the original data as a model of the population is the best that we can do.\n",
+ "\n",
+ "Taking a lot of bootstrapped samples allows us to estimate the sampling distribution for various statistics on our original data. For example, let's say that we wanted to create a 95% confidence interval for the 90th percentile from a dataset of 5000 data points. (Perhaps we're looking at website load times and want to reduce the worst cases.) Bootstrapping makes this easy to estimate. First of all, we take a bootstrap sample (i.e. draw 5000 points with replacement from the original data) and record the 90th percentile and repeat this a large number of times, let's say 100 000. From this bunch of bootstrapped 90th percentile estimates, we form our confidence interval by finding the values that capture the central 95% of the estimates (cutting off 2.5% on each tail). Implement this operation in the cells below, using the following steps:\n",
+ "\n",
+ "- Initialize some useful variables by storing the number of data points in `n_points` and setting up an empty list for the bootstrapped quantile values in `sample_qs`.\n",
+ "- Create a loop for each trial where:\n",
+ " - First generate a bootstrap sample by sampling from our data with replacement. ([`random.choice`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.choice.html) will be useful here.)\n",
+ " - Then, compute the `q`th quantile of the sample and add it to the `sample_qs` list. If you're using numpy v0.15 or later, you can use the [`quantile`](https://numpy.org/doc/stable/reference/generated/numpy.quantile.html) function to get the quantile directly with `q`; on v0.14 or earlier, you'll need to put `q` in terms of a percentile and use [`percentile`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.permutation.html) instead.\n",
+ "- After gathering the bootstrapped quantiles, find the limits that capture the central `c` proportion of quantiles to form the estimated confidence interval."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def quantile_ci(data, q, c = .95, n_trials = 1000):\n",
+ " \"\"\"\n",
+ " Compute a confidence interval for a quantile of a dataset using a bootstrap\n",
+ " method.\n",
+ " \n",
+ " Input parameters:\n",
+ " data: data in form of 1-D array-like (e.g. numpy array or Pandas series)\n",
+ " q: quantile to be estimated, must be between 0 and 1\n",
+ " c: confidence interval width\n",
+ " n_trials: number of bootstrap samples to perform\n",
+ " \n",
+ " Output value:\n",
+ " ci: Tuple indicating lower and upper bounds of bootstrapped\n",
+ " confidence interval\n",
+ " \"\"\"\n",
+ " \n",
+ " # initialize storage of bootstrapped sample quantiles\n",
+ " n_points = data.shape[0]\n",
+ " sample_qs = []\n",
+ " \n",
+ " # For each trial...\n",
+ " for _ in range(n_trials):\n",
+ " # draw a random sample from the data with replacement...\n",
+ " sample = np.random.choice(data, n_points, replace = True)\n",
+ " \n",
+ " # compute the desired quantile...\n",
+ " sample_q = np.percentile(sample, 100 * q)\n",
+ " \n",
+ " # and add the value to the list of sampled quantiles\n",
+ " sample_qs.append(sample_q)\n",
+ " \n",
+ " # Compute the confidence interval bounds\n",
+ " lower_limit = np.percentile(sample_qs, (1 - c)/2 * 100)\n",
+ " upper_limit = np.percentile(sample_qs, (1 + c)/2 * 100)\n",
+ " \n",
+ " return (lower_limit, upper_limit)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " time | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " 8152 | \n",
+ "
\n",
+ " \n",
+ " | 1 | \n",
+ " 2082 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " 3049 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " 3317 | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " 813 | \n",
+ "
\n",
+ " \n",
+ " | 5 | \n",
+ " 1442 | \n",
+ "
\n",
+ " \n",
+ " | 6 | \n",
+ " 3815 | \n",
+ "
\n",
+ " \n",
+ " | 7 | \n",
+ " 2113 | \n",
+ "
\n",
+ " \n",
+ " | 8 | \n",
+ " 738 | \n",
+ "
\n",
+ " \n",
+ " | 9 | \n",
+ " 2499 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " condition | \n",
+ " time | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " 0 | \n",
+ " 5940 | \n",
+ "
\n",
+ " \n",
+ " | 1 | \n",
+ " 0 | \n",
+ " 666 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " 1 | \n",
+ " 571 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " 1 | \n",
+ " 779 | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " 0 | \n",
+ " 1928 | \n",
+ "
\n",
+ " \n",
+ " | 5 | \n",
+ " 0 | \n",
+ " 3791 | \n",
+ "
\n",
+ " \n",
+ " | 6 | \n",
+ " 0 | \n",
+ " 736 | \n",
+ "
\n",
+ " \n",
+ " | 7 | \n",
+ " 1 | \n",
+ " 8482 | \n",
+ "
\n",
+ " \n",
+ " | 8 | \n",
+ " 0 | \n",
+ " 4594 | \n",
+ "
\n",
+ " \n",
+ " | 9 | \n",
+ " 0 | \n",
+ " 1171 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " condition | \n",
+ " time | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " 0 | \n",
+ " 5940 | \n",
+ "
\n",
+ " \n",
+ " | 1 | \n",
+ " 0 | \n",
+ " 666 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " 1 | \n",
+ " 571 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " 1 | \n",
+ " 779 | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " 0 | \n",
+ " 1928 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " day | \n",
+ " control | \n",
+ " exp | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " 1 | \n",
+ " 0.09610 | \n",
+ " 0.09715 | \n",
+ "
\n",
+ " \n",
+ " | 1 | \n",
+ " 2 | \n",
+ " 0.07357 | \n",
+ " 0.06954 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " 3 | \n",
+ " 0.10030 | \n",
+ " 0.08782 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " 4 | \n",
+ " 0.07225 | \n",
+ " 0.06936 | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " 5 | \n",
+ " 0.06173 | \n",
+ " 0.09893 | \n",
+ "
\n",
+ " \n",
+ " | 5 | \n",
+ " 6 | \n",
+ " 0.07736 | \n",
+ " 0.08050 | \n",
+ "
\n",
+ " \n",
+ " | 6 | \n",
+ " 7 | \n",
+ " 0.07121 | \n",
+ " 0.11161 | \n",
+ "
\n",
+ " \n",
+ " | 7 | \n",
+ " 8 | \n",
+ " 0.08207 | \n",
+ " 0.10802 | \n",
+ "
\n",
+ " \n",
+ " | 8 | \n",
+ " 9 | \n",
+ " 0.10602 | \n",
+ " 0.08793 | \n",
+ "
\n",
+ " \n",
+ " | 9 | \n",
+ " 10 | \n",
+ " 0.10000 | \n",
+ " 0.12043 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " condition | \n",
+ " click | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " 1 | \n",
+ " 0 | \n",
+ "
\n",
+ " \n",
+ " | 1 | \n",
+ " 0 | \n",
+ " 0 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " 0 | \n",
+ " 0 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " 1 | \n",
+ " 1 | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " 1 | \n",
+ " 0 | \n",
+ "
\n",
+ " \n",
+ " | 5 | \n",
+ " 1 | \n",
+ " 0 | \n",
+ "
\n",
+ " \n",
+ " | 6 | \n",
+ " 0 | \n",
+ " 0 | \n",
+ "
\n",
+ " \n",
+ " | 7 | \n",
+ " 1 | \n",
+ " 1 | \n",
+ "
\n",
+ " \n",
+ " | 8 | \n",
+ " 0 | \n",
+ " 0 | \n",
+ "
\n",
+ " \n",
+ " | 9 | \n",
+ " 1 | \n",
+ " 0 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "