From 95bd0fd44901fd295f3f8925ac36da2f4058f582 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 10:17:14 -0700 Subject: [PATCH 01/44] Draft notebook for model validation quickstart --- .../quickstart_model_validation.ipynb | 276 ++++++++++++++++++ 1 file changed, 276 insertions(+) create mode 100644 notebooks/quickstart/quickstart_model_validation.ipynb diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb new file mode 100644 index 000000000..210fed677 --- /dev/null +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -0,0 +1,276 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "f75e73d7", + "metadata": {}, + "source": [ + "# Quickstart for model validation" + ] + }, + { + "cell_type": "markdown", + "id": "about-intro-747d2ae3-4cbf-4bf5-9e74-d84b4a3f6241", + "metadata": {}, + "source": [ + "## About ValidMind\n", + "\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "\n", + "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + ] + }, + { + "cell_type": "markdown", + "id": "about-begin-94cdfeb8-2a9a-470d-8d7b-788f58c2a2e1", + "metadata": {}, + "source": [ + "### Before you begin\n", + "\n", + "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", + "\n", + "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)." + ] + }, + { + "cell_type": "markdown", + "id": "about-signup-b3848272-3ee8-41b6-9bc2-5fabf68703af", + "metadata": {}, + "source": [ + "### New to ValidMind?\n", + "\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", + "\n", + "
For access to all features available in this notebook, create a free ValidMind account.\n", + "

\n", + "Signing up is FREE — Register with ValidMind
" + ] + }, + { + "cell_type": "markdown", + "id": "about-concepts-9cb03e6d-d057-4be3-8188-ce3705d56cb3", + "metadata": {}, + "source": [ + "### Key concepts\n", + "\n", + "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", + "\n", + "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "\n", + "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", + "\n", + "**Metrics**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.\n", + "\n", + "**Custom metrics**: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.\n", + "\n", + "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n", + "\n", + " - **model**: A single model that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n", + " - **dataset**: Single dataset that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n", + " - **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.\n", + " - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/run_tests_that_require_multiple_datasets.html))\n", + "\n", + "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.\n", + "\n", + "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.\n", + "\n", + "**Test suites**: Collections of tests designed to run together to automate and generate model documentation end-to-end for specific use-cases.\n", + "\n", + "Example: the [`classifier_full_suite`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html#ClassifierFullSuite) test suite runs tests from the [`tabular_dataset`](https://docs.validmind.ai/validmind/validmind/test_suites/tabular_datasets.html) and [`classifier`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html) test suites to fully document the data and model sections for binary classification model use-cases." + ] + }, + { + "cell_type": "markdown", + "id": "install-library-d030a2d2-38ae-4739-b2b9-9eaaabf9772f", + "metadata": {}, + "source": [ + "## Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-python-0d4ab69a-3025-42dc-b356-b82d1e7875e4", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "id": "install-initialize-63aa8322-784e-4db0-bad7-0cb9934c7385", + "metadata": {}, + "source": [ + "## Initialize the ValidMind Library\n", + "\n", + "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook." + ] + }, + { + "cell_type": "markdown", + "id": "install-snippet-5035301c-b262-4ade-9b98-3a1d6a466980", + "metadata": {}, + "source": [ + "### Get your code snippet\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + " For example, to register a model for use with this notebook, select:" + ] + }, + { + "cell_type": "markdown", + "id": "install-variables-c21f78d2-e547-462e-8dff-151da5f4080f", + "metadata": {}, + "source": [ + " - Documentation template: `Binary classification`\n", + " - Use case: `Marketing/Sales - Attrition/Churn Management`" + ] + }, + { + "cell_type": "markdown", + "id": "install-credentials-d723fa5a-0f7a-421b-a1ff-8b88684b2209", + "metadata": {}, + "source": [ + " You can fill in other options according to your preference.\n", + " \n", + "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "\n", + "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-init-c2877372-5824-4dfe-ae33-d5242ee6715b", + "metadata": {}, + "outputs": [], + "source": [ + "# Load your model identifier credentials from an `.env` file\n", + "\n", + "%load_ext dotenv\n", + "%dotenv .env\n", + "\n", + "# Or replace with your code snippet\n", + "\n", + "import validmind as vm\n", + "\n", + "vm.init(\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "next-steps-493d40db-f4d7-4516-8753-41c5e19eb2b9", + "metadata": {}, + "source": [ + "## Next steps\n", + "\n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation." + ] + }, + { + "cell_type": "markdown", + "id": "next-docs-4432e8f7-f07b-4f63-8387-57398be50a50", + "metadata": {}, + "source": [ + "### Work with your model documentation\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "\n", + "2. In the left sidebar that appears for your model, click **Documentation**.\n", + "\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + ] + }, + { + "cell_type": "markdown", + "id": "next-resources-e2951762-c812-4621-bae1-a077d21f04c7", + "metadata": {}, + "source": [ + "### Discover more learning resources\n", + "\n", + "We offer many interactive notebooks to help you document models:\n", + "\n", + "- [Run tests & test suites](https://docs.validmind.ai/guide/testing-overview.html)\n", + "- [Code samples](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", + "\n", + "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind." + ] + }, + { + "cell_type": "markdown", + "id": "upgrade-vm-fecfaf5b-97dc-4b22-9949-42e7e850689a", + "metadata": {}, + "source": [ + "## Upgrade ValidMind\n", + "\n", + "
After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.
\n", + "\n", + "Retrieve the information for the currently installed version of ValidMind:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "upgrade-show-c0a446ff-f26f-4ad0-839a-e92927711798", + "metadata": { + "vscode": { + "languageId": "plaintext" + } + }, + "outputs": [], + "source": [ + "%pip show validmind" + ] + }, + { + "cell_type": "markdown", + "id": "upgrade-version-098b75d8-4380-4cc0-ac06-da363a3cf5a0", + "metadata": {}, + "source": [ + "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n", + "\n", + "```bash\n", + "%pip install --upgrade validmind\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "upgrade-restart-aebde628-1c17-4f70-88c8-d4561e803abe", + "metadata": {}, + "source": [ + "You may need to restart your kernel after running the upgrade package for changes to be applied." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From e8a46ebbbad94760706636f098fe44b820a25252 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 10:48:52 -0700 Subject: [PATCH 02/44] Validator intro --- .../quickstart_model_validation.ipynb | 188 ++++++++++++++---- 1 file changed, 148 insertions(+), 40 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 210fed677..5feb81ee1 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -5,7 +5,31 @@ "id": "f75e73d7", "metadata": {}, "source": [ - "# Quickstart for model validation" + "# Quickstart for model validation\n", + "\n", + "Learn the basics of using ValidMind to validate models as part of a model validation workflow. Set up the ValidMind Library in your environment, and generate a draft of a validation report using ValidMind tests for a binary classification model.\n", + "\n", + "To validate a model with the ValidMind Library, we'll:\n", + "\n", + "1. Independently verify data quality tests performed on datasets by model development\n", + "2. Split the datasets and initialize them for use with ValidMind\n", + "3. Initialize a model object for use with testing\n", + "4. Run model evaluation tests with the ValidMind Library, which will send the results of those tests to the ValidMind Platform" + ] + }, + { + "cell_type": "markdown", + "id": "a83709ec", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "Model validation aims to independently assess the compliance of *champion models* created by model developers with regulatory guidance by conducting thorough testing and analysis, potentially including the use of challenger models to benchmark performance. Assessments, presented in the form of a validation report, typically include *model findings* and recommendations to address those issues.\n", + "\n", + "A *binary classification model* is a type of predictive model used in churn analysis to identify customers who are likely to leave a service or subscription by analyzing various behavioral, transactional, and demographic factors.\n", + "\n", + "- This model helps businesses take proactive measures to retain at-risk customers by offering personalized incentives, improving customer service, or adjusting pricing strategies.\n", + "- Effective validation of a churn prediction model ensures that businesses can accurately identify potential churners, optimize retention efforts, and enhance overall customer satisfaction while minimizing revenue loss." ] }, { @@ -15,9 +39,9 @@ "source": [ "## About ValidMind\n", "\n", - "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.\n", "\n", - "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + "You use the ValidMind Library to automate comparison and other validation tests, and then use the ValidMind Platform to submit compliance assessments of champion models via comprehensive validation reports. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model developers." ] }, { @@ -53,11 +77,11 @@ "source": [ "### Key concepts\n", "\n", - "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", + "**Validation report**: A comprehensive and structured assessment of a model’s development and performance, focusing on verifying its integrity, appropriateness, and alignment with its intended use. It includes analyses of model assumptions, data quality, performance metrics, outcomes of testing procedures, and risk considerations. The validation report supports transparency, regulatory compliance, and informed decision-making by documenting the validator’s independent review and conclusions.\n", "\n", - "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "**Validation report template**: Serves as a standardized framework for conducting and documenting model validation activities. It outlines the required sections, recommended analyses, and expected validation tests, ensuring consistency and completeness across validation reports. The template helps guide validators through a systematic review process while promoting comparability and traceability of validation outcomes.\n", "\n", - "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", + "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets.\n", "\n", "**Metrics**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.\n", "\n", @@ -72,11 +96,68 @@ "\n", "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.\n", "\n", - "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.\n", + "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures." + ] + }, + { + "cell_type": "markdown", + "id": "38f18a6b", + "metadata": {}, + "source": [ + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "f7625c26", + "metadata": {}, + "source": [ + "### Register a sample model\n", + "\n", + "In a usual model lifecycle, a champion model will have been independently registered in your model inventory and submitted to you for validation by your model development team as part of the effective challenge process. (**Learn more:** [Submit for approval](https://docs.validmind.ai/guide/model-documentation/submit-for-approval.html))\n", + "\n", + "For this series of notebooks, we'll have you register a dummy model in the ValidMind Platform inventory and assign yourself as the validator to familiarize you with the ValidMind interface and circumvent the need for an existing model:\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + " For example, to register a model for use with this notebook, select:\n", + "\n", + " - Documentation template: `Binary classification`\n", + " - Use case: `Marketing/Sales - Attrition/Churn Management`\n", + "\n", + " You can fill in other options according to your preference." + ] + }, + { + "cell_type": "markdown", + "id": "2c93f237", + "metadata": {}, + "source": [ + "#### Assign validator credentials\n", + "\n", + "In order to log tests as a validator instead of as a developer, on the model details page that appears after you've successfully registered your sample model:\n", + "\n", + "1. Remove yourself as a model owner: \n", + "\n", + " - Click on the **OWNERS** tile.\n", + " - Click the **x** next to your name to remove yourself from that model's role.\n", + " - Click **Save** to apply your changes to that role.\n", "\n", - "**Test suites**: Collections of tests designed to run together to automate and generate model documentation end-to-end for specific use-cases.\n", + "2. Remove yourself as a developer: \n", "\n", - "Example: the [`classifier_full_suite`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html#ClassifierFullSuite) test suite runs tests from the [`tabular_dataset`](https://docs.validmind.ai/validmind/validmind/test_suites/tabular_datasets.html) and [`classifier`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html) test suites to fully document the data and model sections for binary classification model use-cases." + " - Click on the **DEVELOPERS** tile.\n", + " - Click the **x** next to your name to remove yourself from that model's role.\n", + " - Click **Save** to apply your changes to that role.\n", + "\n", + "3. Add yourself as a validator: \n", + "\n", + " - Click on the **VALIDATORS** tile.\n", + " - Select your name from the drop-down menu.\n", + " - Click **Save** to apply your changes to that role." ] }, { @@ -84,7 +165,7 @@ "id": "install-library-d030a2d2-38ae-4739-b2b9-9eaaabf9772f", "metadata": {}, "source": [ - "## Install the ValidMind Library\n", + "### Install the ValidMind Library\n", "\n", "
Recommended Python versions\n", "

\n", @@ -108,7 +189,7 @@ "id": "install-initialize-63aa8322-784e-4db0-bad7-0cb9934c7385", "metadata": {}, "source": [ - "## Initialize the ValidMind Library\n", + "### Initialize the ValidMind Library\n", "\n", "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook." ] @@ -118,34 +199,13 @@ "id": "install-snippet-5035301c-b262-4ade-9b98-3a1d6a466980", "metadata": {}, "source": [ - "### Get your code snippet\n", + "#### Get your code snippet\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", - "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this notebook.\n", "\n", - "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", - "\n", - " For example, to register a model for use with this notebook, select:" - ] - }, - { - "cell_type": "markdown", - "id": "install-variables-c21f78d2-e547-462e-8dff-151da5f4080f", - "metadata": {}, - "source": [ - " - Documentation template: `Binary classification`\n", - " - Use case: `Marketing/Sales - Attrition/Churn Management`" - ] - }, - { - "cell_type": "markdown", - "id": "install-credentials-d723fa5a-0f7a-421b-a1ff-8b88684b2209", - "metadata": {}, - "source": [ - " You can fill in other options according to your preference.\n", - " \n", - "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", "\n", "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] @@ -174,6 +234,49 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "4cea824e", + "metadata": {}, + "source": [ + "### Initialize the Python environment\n", + "\n", + "Then, let's import the necessary libraries and set up your Python environment for data analysis:\n", + "\n", + "- Import **Extreme Gradient Boosting** (XGBoost) with an alias so that we can reference its functions in later calls. XGBoost is a powerful machine learning library designed for speed and performance, especially in handling structured or tabular data.\n", + "- Enable **`matplotlib`**, a plotting library used for visualizing data. Ensures that any plots you generate will render inline in our notebook output rather than opening in a separate window." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "de14ed5a", + "metadata": {}, + "outputs": [], + "source": [ + "import xgboost as xgb\n", + "\n", + "%matplotlib inline" + ] + }, + { + "cell_type": "markdown", + "id": "cd977c5a", + "metadata": {}, + "source": [ + "## In summary\n", + "\n", + "In this notebook, you learned how to:\n", + "\n", + "- [x] Register a model within the ValidMind Platform\n", + "- [x] Install and initialize the ValidMind Library\n", + "- [x] Preview the validation report template for your model\n", + "- [x] Import a sample dataset*\n", + "- [x] Initialize ValidMind datasets and model objects*\n", + "- [x] Assign model predictions to your ValidMind model objects*\n", + "- [x] Run a full suite of documentation tests*" + ] + }, { "cell_type": "markdown", "id": "next-steps-493d40db-f4d7-4516-8753-41c5e19eb2b9", @@ -181,7 +284,7 @@ "source": [ "## Next steps\n", "\n", - "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation." + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your validation report." ] }, { @@ -189,13 +292,13 @@ "id": "next-docs-4432e8f7-f07b-4f63-8387-57398be50a50", "metadata": {}, "source": [ - "### Work with your model documentation\n", + "### Work with your validation report\n", "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation**.\n", + "2. In the left sidebar that appears for your model, click **Validation Report**.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + "- add a validation test as evidence goes here" ] }, { @@ -205,7 +308,12 @@ "source": [ "### Discover more learning resources\n", "\n", - "We offer many interactive notebooks to help you document models:\n", + "For a more in-depth introduction to using the ValidMind Library for validation, check out our introductory validation series and the accompanying interactive training:\n", + "\n", + "- **[ValidMind for model validation](https://docs.validmind.ai/developer/validmind-library.html#for-model-validation)**\n", + "- **[Validator Fundamentals](https://docs.validmind.ai/training/validator-fundamentals/validator-fundamentals-register.html)**\n", + "\n", + "We offer many interactive notebooks to help you validate models:\n", "\n", "- [Run tests & test suites](https://docs.validmind.ai/guide/testing-overview.html)\n", "- [Code samples](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", From a1e174693366e8eb0b07d16239d23ae99c436586 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 11:33:02 -0700 Subject: [PATCH 03/44] Headings --- .../quickstart_model_validation.ipynb | 228 +++++++++++++++++- 1 file changed, 227 insertions(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 5feb81ee1..8bf6bf197 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -259,6 +259,232 @@ "%matplotlib inline" ] }, + { + "cell_type": "markdown", + "id": "8f62809c", + "metadata": {}, + "source": [ + "## Getting to know ValidMind" + ] + }, + { + "cell_type": "markdown", + "id": "393b94c8", + "metadata": {}, + "source": [ + "### Preview the validation report template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for model validation. A template predefines sections for your validation report and provides a general outline to follow, making the validation process much easier.\n", + "\n", + "You will attach evidence to this template in the form of risk assessment notes, findings, and test results later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6bddf4bb", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, + { + "cell_type": "markdown", + "id": "ba022e64", + "metadata": {}, + "source": [ + "### View validation report in the ValidMind Platform\n", + "\n", + "Next, let's head to the ValidMind Platform to see the template in action:\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this notebook.\n", + "\n", + "3. Click on the **Validation Report** for your model and note:\n", + "\n", + " - [x] The risk assessment compliance summary at the top of the report (screenshot below)\n", + " - [x] How the structure of the validation report reflects the previewed template\n", + "\n", + " \"Screenshot\n", + "

" + ] + }, + { + "cell_type": "markdown", + "id": "9ed985b3", + "metadata": {}, + "source": [ + "## Verifying data quality adjustments" + ] + }, + { + "cell_type": "markdown", + "id": "cf2da2be", + "metadata": {}, + "source": [ + "### Load the sample dataset\n", + "\n", + "Let's first import the public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle, which was used to develop the dummy champion model.\n", + "\n", + "We'll use this dataset to review steps that should have been conducted during the initial development and documentation of the model to ensure that the model was built correctly. By independently performing steps taken by the model development team, we can confirm whether the model was built using appropriate and properly processed data.\n", + "\n", + "In our below example, note that:\n", + "\n", + "- The target column, `Exited` has a value of `1` when a customer has churned and `0` otherwise.\n", + "- The ValidMind Library provides a wrapper to automatically load the dataset as a [Pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "684abe24", + "metadata": {}, + "outputs": [], + "source": [ + "from validmind.datasets.classification import customer_churn\n", + "\n", + "print(\n", + " f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{customer_churn.target_column}' \\n\\t• Class labels: {customer_churn.class_labels}\"\n", + ")\n", + "\n", + "raw_df = customer_churn.load_data()\n", + "raw_df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "6596e866", + "metadata": {}, + "source": [ + "### Preprocess the raw dataset\n", + "\n", + "Let's say that thanks to the documentation submitted by the model development team ([Learn more ...](https://docs.validmind.ai/notebooks/quickstart_model_documentation.html)), we know that the sample dataset was first preprocessed before being used to train the champion model.\n", + "\n", + "During model validation, we use the same data processing logic and training procedure to confirm that the model's results can be reproduced independently, so let's also start by preprocessing our imported dataset to verify that preprocessing was done correctly. This involves splitting the data and separating the features (inputs) from the targets (outputs)." + ] + }, + { + "cell_type": "markdown", + "id": "6dacb398", + "metadata": {}, + "source": [ + "#### Split the dataset\n", + "\n", + "Splitting our dataset helps assess how well the model generalizes to unseen data.\n", + "\n", + "Use [`preprocess()`](https://docs.validmind.ai/validmind/validmind/datasets/classification/customer_churn.html#preprocess) to split our dataset into three subsets:\n", + "\n", + "1. **train_df** — Used to train the model.\n", + "2. **validation_df** — Used to evaluate the model's performance during training.\n", + "3. **test_df** — Used later on to asses the model's performance on new, unseen data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b34f9bfc", + "metadata": {}, + "outputs": [], + "source": [ + "train_df, validation_df, test_df = customer_churn.preprocess(raw_df)" + ] + }, + { + "cell_type": "markdown", + "id": "b0d2fb15", + "metadata": {}, + "source": [ + "#### Separate features and targets\n", + "\n", + "To train the model, we need to provide it with:\n", + "\n", + "1. **Inputs** — Features such as customer age, usage, etc.\n", + "2. **Outputs (Expected answers/labels)** — in our case, we would like to know whether the customer churned or not.\n", + "\n", + "Here, we'll use `x_train` and `x_val` to hold the input data (features), and `y_train` and `y_val` to hold the answers (the target we want to predict):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2d089b26", + "metadata": {}, + "outputs": [], + "source": [ + "x_train = train_df.drop(customer_churn.target_column, axis=1)\n", + "y_train = train_df[customer_churn.target_column]\n", + "x_val = validation_df.drop(customer_churn.target_column, axis=1)\n", + "y_val = validation_df[customer_churn.target_column]" + ] + }, + { + "cell_type": "markdown", + "id": "b5ed7410", + "metadata": {}, + "source": [ + "### Run data quality tests" + ] + }, + { + "cell_type": "markdown", + "id": "d51a436e", + "metadata": {}, + "source": [ + "## Import the champion model\n", + "\n", + "With our raw dataset preprocessed, let's go ahead and import the champion model submitted by the model development team in the format of a `.pkl` file: **[xgboost_model_champion.pkl](xgboost_model_champion.pkl)**" + ] + }, + { + "cell_type": "markdown", + "id": "1d1aa3d6", + "metadata": {}, + "source": [ + "### Initialize a model object" + ] + }, + { + "cell_type": "markdown", + "id": "983367c9", + "metadata": {}, + "source": [ + "### Assign predictions" + ] + }, + { + "cell_type": "markdown", + "id": "701e38e7", + "metadata": {}, + "source": [ + "## Running model evaluation tests" + ] + }, + { + "cell_type": "markdown", + "id": "8ebc45fb", + "metadata": {}, + "source": [ + "### Run model performance tests" + ] + }, + { + "cell_type": "markdown", + "id": "746860d8", + "metadata": {}, + "source": [ + "### Run diagnostic tests" + ] + }, + { + "cell_type": "markdown", + "id": "4e64ea12", + "metadata": {}, + "source": [ + "### Run feature importance tests" + ] + }, { "cell_type": "markdown", "id": "cd977c5a", @@ -298,7 +524,7 @@ "\n", "2. In the left sidebar that appears for your model, click **Validation Report**.\n", "\n", - "- add a validation test as evidence goes here" + "From here, you can link test results as evidence, add model findings, assess compliance, and submit your validation report for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html)" ] }, { From bc75cc50481df912575649ae7d23af3aeba94cf0 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 11:39:59 -0700 Subject: [PATCH 04/44] Champion model export for validation quickstart --- .gitignore | 2 ++ .../quickstart/xgboost_model_champion.pkl | Bin 0 -> 123851 bytes 2 files changed, 2 insertions(+) create mode 100644 notebooks/quickstart/xgboost_model_champion.pkl diff --git a/.gitignore b/.gitignore index 9ab39fd08..bc3a104d3 100644 --- a/.gitignore +++ b/.gitignore @@ -197,6 +197,8 @@ lending_club_loan_data_*.csv !notebooks/code_samples/model_validation/xgb_model_champion.pkl # Sample logistic regression model for validation series — do not remove! !notebooks/tutorials/model_validation/lr_model_champion.pkl +# Sample XGBoost model for validation quickstart — do not remove! +!notebooks/quickstart/xgboost_model_champion.pkl notebooks/llm/datasets/*.jsonl diff --git a/notebooks/quickstart/xgboost_model_champion.pkl b/notebooks/quickstart/xgboost_model_champion.pkl new file mode 100644 index 0000000000000000000000000000000000000000..e6fdef9b043c4592dd3a1c0f95fa0a02e0b465df GIT binary patch literal 123851 zcmeEv2Urxz+Vy}U5=1a(KtT{uQ4|bJS5?p8U>0-65k?p#Nir*902Pdwb9PmX7*?4c z*RbXs)>YS>v#vS*?>kHl=z{lt_qty9dH(V_RC=qry1K$S-92q$!D*@jGJ{JkIv})H zR8(wS&Dg%-8g)!$dTNnQDp3)xj*Zp!)@owXr>0jMkv=rtJ=HccND~{UjZnu$#l)t! zOtp&Y)khN?r%lkLrxxv{ja0`BY#1IDss)?i^i-<|^?;xdO>|rsm|9~5s!z~hK}(E5 z0SMZJ#zgfGijE4`1`mX25knKSk)c5`>NpJ~SS4s;dPT)*<1p9K&`@X+l#7eeXo4a% zabZy*5VZ(ZM?|P0U>~843Y&&-O*Euf#m4uFRYydJ z;~|O~LU>G5b@stg;o12?y$0fuz|0|cTDT@b6Ap==8Ae8h7@8cT2@O(*M~7j1tT7U< zj_4I)XkBcuI$Q&%8=KQQ>t5iRy)&TOuMVGAb@AQWF#$6&V{Bqt-^oVR~usH8EI$qp=-@LNp25UVBdpHP+U|Hv?(kKFGx;eSd1n%EGir?pBU=5_0sj?C*fOqeg~ zw~da83f9EN8YYLSv}K5$!%C<7>T>Z7N8eTrX_0v>D#w`nFulJ@5GgFUzzP*_Y@@g*M zBTYsH>eTb$NBHXkYBDqTsry$ezV@1|%uGgKY>qh=rsX7OBhSe+wItWZo|9`T`j_1} z$HvqrcDQ55Mu%(Tf^bdGu`tz==Z#+7R1U6BitY{6!mOvNAIbL;G0R%3;IDh1jt5nQV%C z)vS{}Xia}jhtDA_CUdi#Rgj1dt6jt-2)o%#j55IqUjYM8WcM)GB_+I zDpETr*R(SAN{Q_K8zy;(VR;Hdl4ELvg-^YD-ah#(JozndLrwO@joV^SL>`NJE?#6h zo^73)-oD;mzO{Y)Ynb|`m|?33>Ky}g2nc43Iy9$&xvo7f|AyV)R9mhk05gr$#guKH zvvyb;_6~JSs3!M1_t*2ttD8A#HP!pYf?PgC8*5-v#_c317wa*V$!Y2rG|6O992s9! zd|Y&VoMAV{Ws;AE9HMSA|B|np&c!mDZ^h<}r2?k0^!dh;&Bli2lgl|;e)-7UOb5+r zU_P~d*<5~Rb_RM4%f=UD+x|(5{64GFqT{=t2Nh}X$zWKX3+wNrm0nb zA#&F0IH0G!;^Q>2CL_CEAd~W-L+0M5F>7rwF_3;!%N)FF*00x~uD6d*Js+RC4eEJ$ zn_6L_)wr|gx-_D8O<}jY91-k-*eV#+ITw92I~m zp-9X4h*rH@$3%t12gl{C%T@vESVfFN9h1Wb?V7|ggAQz=iRhKKwN2Q60P)FtzDw>nkHX4>QJ_bykgVH6>&M5%W$8_v} zGh)#@JWBoFmNL_t4*OYT_o>!x+rR?~X$F0e9?PHklGsGLK3z#+f+KYl<9JYcS2 z;$iA!TZ0nMzHJcO)g{EVn>LYL?!MMn6Go84M>q7W`%UUy;2M=pa};{EEi9c?Hq)od ze9880ktFfWaFX&WmJ)VRKJN8;+So#^FL(Tj;6HJewDqBzKJ;w~QsikVzh>9X$gRXo z`TCjDsdB?p!EXB&;y!q=-ns2Wl6-%x{HIM>dinDul11Cfx}%nnBjji743w2kzJ&q^H$Q@iBEMO&bA`9J`)A;+$_@c*&ZRx(L>T6 zSt{R9s~hP(I9bp+RuMYUiKN`k(el=>c1UYy4%1D!vQJu3>{s2NCr1k7pKE#lTJt*s z(8(_0ram;WWBZ(Z1DVL%s(3D|SWRdINJ21d{`Kl&fy3l&Wy?3$a82*Ht}g9O^)xXm z_W$1F=WnhJ(V9*c2!HRm?E6tMn#kP!fd@7N;B7byfF-~RU;`)uC<<@@6#oPc{E~pu zfUv04Ix-Y5aITrD_!#z+BH)0zqQ&-7;+S+>`mkxDc=*&|ak%#q zvG7$7IxTD`4VhmbS}zklV{b6`N()8pjc2UD+Se@N;eIwQeJSSA*a_$v95ZY7TJ-Hb zgmuo^i3#jMryFeOU>`#!#+y8R!it^rVzu^0Q~jwuv|Y$fGVw(sJ%2w%nCmB_nf1FF z%B4E4H25jZmX+~8Nj-ZzvC?i`cs<|C=e=kN$|HLol?2RY}J_3h1qLdLh^a5Dm@3BtpO>fj?h5xfKKV(^Y~zS9}|4EqWD2K&Pr zfaj0rjOU5xR|2OG_;`M00C+wX0C*lw#=Oetqk}YTZ&iT1F^=u?G-PJy*Mztipbnre z00$`!_J#mIfM6^~jlKwJNk0I%nJ`Me)N1EbuBo6ta6fqs8JrA6p-N@X&V2tG7Z z)gius2mwqJp|4ac*$#RI^WO}IB$+ra+fE#kT?WhHW{bXQvxTssd?)A<*lfiRsAtWf zD8AXMMlX4$+U?UYz{NQ#pv1;*ira~@faQk^(b#jx4J@eG?S)Q87a z7NRm`%|zM%YO^hp-IKuC;ATB^W3Gt}Ej0(=WW@=Jen9|FLm%TfxzS;M?mTOVBhX=a z?l6tpVj8c9=VP6@f$6#97)FP6pu_sGj{k|{1JXLX8&N>~z2v|q7H%|z!L4Mi8rVRt zeN5Te>$lKP(@<)nk4f`pk_AN&@jqlAW^jso_XT1`; zESadVxg)0sw^UPv{HhekclT2?3@;;|J6)fSu6LFE+@Z8+@z|FgnUf^8S#yzmJlj*8 zvHH1~=rV?`cxge#%O)#ooBu4fva2b!PCU<2W94FCLocysBxQIUujO_u^J6ipygxuR zn^A`?uHH@*3pug#OD2jg9XqhC*ZYc%MC(Q8ejmlgJzOcIvB6I*$?eti z2$pH?eg^BMvGgNT9Y4Vw{of<-_-xqOp7d<9@oZz6?F`C^R%thx`pN^gd}L>OJn=l8 z3+HNoNW}Um)MvU{2FZX=#3D@G-3IT_fKrFYQ#rac=dOt%`ztqi3=aqJ)>e+rHTU)A z2ee0GX}GBvlmHJrqjGNGv}U3#0uLNYQ)V`T=E@OO2b|)|U^SsXOj-A?xP2Orm`B zTw3#dwm#@xDPi>6`ue7}ZH24P#>%(bR3JO7rVFEw9@JMW(Vt9sy-~iQ?>%yD&I{r; zzPPkAL69}^Z0Z!1k6;iKy@-A>X}`XGg-ZS-5&GgE$kV=JjZy0=vS$su8G zv8M9lgS684qZWeJyS~(U)>>(7`N>kLz+R+`{(-Q**ing?H5VpaDj^LX>Ly?7aZA{1 zTQ+Nzhoj)Akcf4ZN_aHsg8V^IK^oaCMV@GpsayNDh_rO}34SsDS^6UIo3CrUA5dqQ z@kstU0~=rlz;zziTwKdJLHoCL29!ZERmDgmTp{cRK*D6u7U0(~=H;m^jClrq0k#C= z=pf7h3V=VrpfA900%!(k0ce$N`M>?kyE$JS5~vs$94MX|@mib~Ac!MRSyR{bGG?*M ziEUaRA-ip3!JS z77Ufq1Ci=qJU5WI_%l52T+Fbhc#8a%E?-*9C+fvMeGzscgG?y{$qXbqb>8;TJ6(;@99q;A5bV}gx z?BT*2i9;HJ)B%CR8^Ngt4t2!0)+s2-5nQj7fH2};>6Y?fg8=?&z`xQjmB9{C#RwPh z5jgD-?ndAgM0gnC2|l8R5vUIkI4x`O4*rUc!Q}$`6Z`Q$)-g~P+wOvs7JP$#0Uz6F z&@SNDH0I^0U5t5tU?%_!+QpbALR<+j=@$q$1>mW*1hhuG|Fz@&{8gP3D1u(26sBS# z_|S9_G@=Ma#ZdqlCnD%W)yl0$krWv6$K}{#83bw}#hXMKk^-d{l?4>f?5-*`y{LbQ zH~o+lU^yfO0i|w7pkCoR$53{2)P5udsHxgy4M0OLQEVxtlBawpqbr2uWxENRh&^ONmDA>{QeDtkz0VhOmhjB#ia+r^4 zSU=ClV`3N``h1ZxBpCl-I@W{bxZ~w{{D1260iBOd!an}Cjvp{Ty%eZy73-@k>9AFy z9V1f~ZC;XX%d8{*?wBdQdC*KTYswwPXbX{<4d~0F9={P23rC4{YK{|6FKo`H)%vJt z`QB38-!@(0Jt9ENC^mq68d8Psb~q|}Z9c;0Z>b_yTY8!np3_6zw|^`>GNdN!KG&Jf zO5UufP^W;{%-vp`7&4Sy&+01v@h)8)Tj(8~U{+OJTBkQno-~|1vS=haCTyf*UteGa zIyPYqOMD>nFCG?MG9HQv^ACt+o;6`DYOW?D`)(kohi#$($L+=P*C&YgVgy#^~v?g?u-dnNmAp*?xJ zWjifjstdg_KY^ssXjZv=Z=ydek?kk%ij{urz}h5viR`1CEiXKrjlWo&tT>&_>ensD zJK6gC=&Aaqr=S0=p6aaobo{*Ze5NGqujZ+@zGg?pOh_UPExX8v&%Y^6*VWZ`ebbjz zH*Z2#TplWn{{0$Ba#axNQPy791+P3b`ZQ*xsE?`Lt9>zvSVkewm^#c7B79xjSCI z@o+cky2k*LUPPvM>JliZn#r}=8puyJUvvJQe&*V)dyE$agig!Ui04aCr-C?a~`-$ ztxg{k?l;m1vn|f+-gb==x}KXUke7kFH9;CF@zyVXV>*wOTOGP9ghNN)r%J=cLS11JFg z02Safy#?{WT*qk(@%Dg@JncvF`DMsgmvvUyvM!=;%1P1c^*zzbeFzz`@CjS`dOsVn zc`9psWB{Fg!-+ao=tNd3omi2?J~Xa9WmOZpvjHpAtU;$I^l{)gGT@qwt{EIn3qCMC z1A-oru?B^98~lb=0}OtV)!hZxyhEo+$*@!COBXH^>Py->{jBp1Z@uhyN4rvHaBfive zwILl*#|V6cYr0SQ6rQEo?o%DE+fuBb9VdyfA}0@87*NpeSU|O;8feE6V439@P_Sn-ox#i%N0 z#T{?Qh?nm^5F5-stne;0Lri|2CZ4ahN?dyDAl*66fmTmjPhVebMs_%_61$}zr|AP~ ziEF7Ny|6q&JiYrTx-q1>Xj9sOHMG00s4jZa*J;h!)28uaX~h+>!HD7Fpc7|Bm(FcO z*Eo@ChR>np)00K(3#HhS*f=pMbsAfH)Ll%fJV106d(lbzrjju+YMLbz+3lJR%sI0i z*>(D7u?g`LpWfI(L+!2TXbTnFaacjmyxvDQPEHhMPuGh@wm)ID%K5R454W*td+v*6 z%f6)kOSe<=!z;;N%}p|E$zn2ZbU3wNoi3Uk@6HsTHZb$sJ}my^Mp3nN7%P}mj&&~Q z!GM&=+_}$Z?3P%S>5EZ*K+npSI&zC}3;;aPgHu(nDWVgIK_(oD;3 zWM!dF@<|b6b!~1^sxN7+(Dk_ah}2wHjW{0g6-xY+s4uzRO-HVmBFa-A zgv%~nr7~}Ag+CuY(A!%~lfR4}pl|BmML)*-CuwQBYx>Tg+6mikg$tv??g;C8P9xoS zw-H*EpD9htysocTdABsBw|yxezqGQ@u6&xXDgBY3*PV7kjc!%tTQ*OTG*@oPCslEm1l4W9-L|UaY1Z&(=aF4*f1NVqLb)%6$nY5!Z zULNdC`q7vM8@lObCnTImFu(e|4)PlT^3;sRI#rPWnQp}NAI0%*&Q})(ia#+K(|ah{ ziDTW^${};fm{avddou;`+)r82L2K!7ODB@yxE;2_qXWaYVH%nLz7KspGm$>9wj!qrEM&4LjwG$bErb2sHUiDMeu55;nnY=t{bWM& zdO=l3M*Thvq*kpH4dv(B6{nS2FQk=kP;9F_LtTT&?>QatV8i=(Wx+?_G(!BR`p*T*B3z9?&4h3>;@{GJyo0}@ z>u|Z?l(7(mkq+jm>u_p9o=Mvo<9L3CcMrigymtsbwj0~(4)6eY0>1jZ4)XKWbjITZ zfX!#R4%2@W$NTxKx-C$2f~il9fYU+E1X08DI+6(53ZBKGE<#dZh=X>*cNvisp!N!= z?pg=a5e8BKKI%SvE+<0&h|q_q1(6i6*APBfGfI`w+)oK(*8k6yB?9C$Mzvl$9CaF! z0>d%z-AK^H7)DZ{xCi>J!cLIWI}LSW`j1Eo^4Dd2K;z!Wfdd@F=;PjqS_cO@I&6Wv8sD7wr-GBE-YMMb4@ zV*TIjXy1&>%x2YHaxXqsEHtyN=v{I?c|80g8BwVTdpJbR4otaB%dlP4dg)5?(aWCI zx)dbV4q70(mOsdrMNSu|{?bn@(4r8Tn7%-?yu6(rxU!BpZ|_ZeG~dGxEUL>Ycur#* zdaNQddlVrXb`Rs7{Jr$lH+x>!QMb~5h;$*P15&FJ|TS? z9z#5~#Y-hpYY68Z-pK3UKCL(ZbE;0*TV3kmJV|&_p}KHONY*v$tJC`)@$>8devY*3 z&=h@ao#DEHpK9oX>!b-Ik{0R9+7A-alzmC(0goi#)z1X0CHBHNR*BFtUb@Xm1%%a? zZ1t5~+Dmm7ZEi)@M*DJi}HP7<*T@Zbo@wm7=*{_v$m`gZEDm zR?eR%BrJt@C^M;iTAv=ewI62+o-G`7RRdq@o(&F^##^3~XLy=R722&5>LixY?RvRL zsyt}2F!FwD)2R4IJ*VsVZPy=DZM-*7Z58B^{M8mVroE z7Np~YpV|PtU0;vW9tisa8UPvrOlr$mH-j_cXMrW*v7k*whfGn6^lb*bLm@aI3pZsY#3Dnnd8rJX@ zZSw0)UY@)SzCoj&4x^D%YLIhFCSkh#eMolQo`chgu8#{Db?_~>r!xjP&@QX)(CrK4 zG*!4xOR2ZfvX6(+U)}c8;ft?vyC22py`Psl>o^?7NUnzB6hI)6K_H1ja1z9c7!oN2 zk}d=%OGthYNRANrL=ItV#J^H!#la5oe^qJapbP@1A%d$lM+hUFjBo}YQP~I=@DWvw za04IVZUo*SL3kSRaegk?eh&b)yCwkJhi$40@X1vkRL8&mg;*~80I+Wgz*93lm4h%S4_OCp_oMi{ z|G%o{I^vTv3eBhRha_hE7{|9S1Kbuh!!*^YwcYToN<%nS`GHdsRC6L+J4{zPQ<*;* z*IV_Qexb6=P8mKe6R9x|#su0Jp01TlS;r|!fVW>!Wr>}#fQkoS29%sL5h)4UAth0@ z?Q=tEztd2TQ<4CuxTpB!Pqnb<h9$0`P?e7atzx z69X3*oFM3OhcS+e6WZbAL5CBF+oF$oIHAztLWFU&;f{xMm%}>I@jPB0)3DCmz%+DN zhT9tweuO`K0_3Cqu7ymAPlY_vmqVxX$8^uT1=)^eJ5^y?k!eb{6So2^-`=9 zx>$6%rexi#%oTe+yu@m5vKMQRj;ywAb7qsYkvyu^iFx-nBdKK@vf}3lv+0v=u(A6+ zX=3JTmcFtXIrZyta^jMOIAY8x`t0O#mhsCovVL70xtbxP4W1LSvQ#B;)yGqGU5CPA zWzSw@!l@ym+0R7m)+K|TX>o?NGQZ6Bbl%2B_c=~=wGPn_rNfzduL&Y;--tZD=FjXC z_VdpFUfR%?AMn3@*5-X}f*Lqo&LihGye|^aL{rbXYSEbatG7fWGnZO2V*= zu2NaM)$;ak5o0vMF=|ez|Z@@T*;^G zBFU`pGHIqRL<&5(LSDX1P2KfjkBLQv`_i0#)pf7SzY=B+bJgun8LMlU`A(Wm#tXt| zFFyy1Ls?U;2MP_#S_{K|KjYV=zlXfa?Iu#$9_x7L{k*_R~W^Do-Su%WowpFGnK60w};-C z$x{*` z>^`%Q);zV4Cf)3bY1~o8$tg5omMvX0_BEN`@Co%)NUYt!g2Ij|JISZa`NBev0o<1R zv|d*9i3c{bM9HYGpq*n!CdkpU!VmdmAC)+aTFXRaT3J?LxoSeTm#0T0}6^cLqeV8~YyNUSz9fh{y#%NF#wxCqZ zHa5qm8{$c@!kkSss14Ps?K4p!7*qvpjO-4)Ub+8O0=<%5SMH4wZ34U;641x`(N{vB zF&X%<98!pXxiN}y*>J+3uL3)@Gw zeyF6dD!fPR`SVb5RPRTkWn4TvP;`%&I6aazUlBnA=ANSt!z+?`JMGAnc8O%?vj|+v6wh&d%W1Kz-%%-D_jg%7Qzx+KcN?_;@P>XX>5tJ zlFs<~gE(M#Z)*Kp4_4;z7TT)MALLQ7N@R)K6f$PZdoough(Wi`GxLBnwys8TrX13M zHR`ujZ2ye1hN}*+)Sn0OPWw*0-RGOVBiB~f?ybEvIcto3X|z&z@Z&S-dg=1=TFvK^ zUPFpY>l@UPuiP+8O7445U!`^#zo&b;5$j2C>{`g`uw=NxeE< zkxtE6Gi z!i0r-VN|sR!k|tTlFy)tjSgjKq{-Dobi0F`g<(VL>3Sp+;oP4Wgi&6}x`~@tO1|r= z3auvjORY9^5Z1Q~*PV%#OEZrY5<)7qk;=;@DZKkcoqdrP!lJP!{8Cd(%OC80DbEprw zEWY3Nn=BijSHkB5vj0Lnl0W;$Cg2LrS7%&NaaG2ZI8Ww}8$0BcHTvZtY*GLat^#lc zeD?J`h#TIPGnUDh{e#_SY#-vdvZ?@00L=g`0IdL~Usz$CKZ@hsov&(W%cjh)N)Ofu zV?I)2Iw{tPCft6)EaRI9PmU+jA%zD~I(Qcy^{Ff^`0xpRGjJuGsjN}jdh}$pee*W7 zSZV-`9;Tpk?*2iGH8@G-DYvM72JXc&<~FD$o!;WTuzqJE#?et>Oek#`(T`g0BXAlZuwxN8B@jqF5I8;f z6yeh(FO`9;9Fd>eC=GcCxR?If+jIOm3{EY?zpFNI`XOA5z^RBp#*c6_!X11>H6sja z1j5yg@B|-G1Hn7`D=Gz-5$sp&OYA@FGwdf@K%NQ(r!3@^Gy0z?6-dYSV|zdQY9GW6 zZ_ydcU}%n58lerXk8UBn%KAr)Y~skCZotDaN_=}%rY zLOXmrFTl*f!4L-fADKSnuZs8p$3+Ae3LNOTK%kEUpD!jj=y@Il4SC$hNrAw89>zR$ zx$DSI$+5#ctQQ>~hnK~AcwPMXxJaQrrsG7!@DJzs0MAEFVPAi1#}ApGwrZ;^5#2@6 z;lwJ1uKy)*-{v^+=EJLs%_-yA(0iexd!MCZs7xuw6-pL6o@u1W%s5TXZ+XW0Ry;ta zhdg9?4m) zOY};syDYtjg7rBxjJ2Bkp6z}g&Jt6)(=nUPMaO$f>DJ)2oyy3#LG5XP=Xp`qW+HvRb2XLEI!wLNCX+vk|4PPY%%|O~ z+b{>~=d{c0S1jvlOO~?x0{t}W4{@{O7%}xxaW*13idv`66^D0O&o+zq+4TY|#4&Z@ zt&#IxsEyQ)clLMEWc5wI-~Df&rKof+yLw5sCvAj=8&*jka|RO27B7UJkz1r`=Dxa` zEJL~&?j|&`ZbKRtmE`BvJQn_V+E4iT-b!6~hvoWWOUp|Q+fB*(SoosQr|U}+ctVmF zs#8ixett-*)7_p_v|b~1FEPSz^_)kN`eQrk*g3VH#@-XsOD0Qu8+8|cS>-9YIWH7W z*^LpFx-FN+q^}W{*gn$7W}K2wPW_E|oNgcu4s{h)jvJycIn67p$ckk`n{^>l)d5*q zu9F5yt;c=PO`3L3J~ec_6gRfLP-EUyssG9rSr@FB6#bNCed;R;eZ#K{1I8WF#W?R1 zl3OU`p`#q7Ro7B_(D0vs{oMDA(2D! zhJ+1C+Fvx4v0MZ6(Sc1PfE+*o%$TNtu<89cgW-4h!@DnEoup!_5+3yJq?Ppf&~hwb zZ4)xEKcz_zXF_XbbkWeuG&y7sNpemSJ{54Hi+4?;mph*~O6Z%MZ+NXKO) zV`&)r=;*wIsbs{p)^tuiB~7s(Vn{1j>=3!J1$Wu+>Ui(vrF&Gv;qv8b&A+RAkjfx9 zwLltyK-z&ot%I;N0y#YbHBVl;hwB}DSct%<5{Uft&j0mki!;=Pa4`a>1fr@DI6e3j z!D-@w;2rlB9isxf-1G_i1JBg5up_%y91-UWq0xk3_>$6%ocG#_@eJWi@zm0}~gw zP-QVH<6Ef8CNvWB1FCc;s`+bg;ufl`z0*+Uf3=0?uQT`nM2&$15q;$HI0(7JIG?B( zMsQ~d0bG&L&+X8jm*aV;8SohBBhX>|ONV7K|I1_XGGE&NZE-#T^U)31ci+PC!{nzG zRLY``_KJPY_bF;z+o;e7N^H2gvZ7jmE1ep+UXhftLj2RE8$Fu|KYg(@7cZ;b#oj%X z)J8o!vMZDshOSmhT90 zMW@N4t3#Y<`+GgsF}NE0V4Xsno!&wn4g}I6H==0ls#Ub&tlz}3K`zX`e+TxytcU33 zP!N6(<4PBo*-jsX6GfHVGg#e^Q`nOolf({fJF~v4&al%hhLM#Wu85Y?hp{?-Eog~3 z(QNv)4A#8W7IJyUd$P@;AGM7dK(5?<%;dIKV(?&pdQDT1ysf&3omy#47ws8J3uitM z-9~Sw{ZBR}Go?0k=l%U;t=W0rdEbfq`+l?E*tFMKI<%5T4jCb@DYquS??^(NKN@ZsgPbJZy3@?n4X-W zcZf>U^*!24Xn3NL-m5|l`DCS|G`rGL`J8PR^j+pIl@>d_mA_C#l14`|1evO-?n&Fi z!r%(mrCaSv3tO*mluvg|)7#$F`F)x)OGr{j%a`3Mr8`~PdPjKd1gY!PvqIg68M;gR zdP;%28VIUcL*$kA9g_M5dkKXre$XB0yh}HuNwSo7HBr8+ghHwVP!K4S=-hFY1A_3CODl@CAJK>l$Mn1_c3m3V=TVSKcOoW`GueR)9c2TVCIf;`46L zS9!E&c47W>RGM!J3uG>btzmKA~FV`j$g59ZY4>?(K8s0e#t4eFGlF_XfPt)Y* zBgpFWm4qX1>#6ykXK>as>eJ>3we`xxFgkJ~d^@fC;t5^2AdKFQvL>%)q|o65)(Fck z)EAy~OU#1{nAKIbGDjHXH6EE|l_`KKiQY`I9HA`2$p};{h{{G(0Uzhc z5uQfW03T7)2)qG+@Ivqo|BBMV_6-;TR@&F2hvfc{YBwGn&CU;IK?51 z=Y#FXc4J$yeb}ZtfVyad4#GYFPzs;_ipH2nAzcJ00gV9xfTjREt(Jh+fHr`3hRS}R zKfK?+s*u{FYQZNflxC|JR4@=UC?&3NOlkQ2IQZoZ#v7*}O;AF=BGI4_XZs;+BpRyA z?W-!w?v$w(@9U1X0S#WMm5p{9)DhZpqM>qYRY6%B%3c9%gJZ%|AWk%tFa``Nt5)k) zjXjpjRGZ?K8U8;f_>MW236S48#=n2}qY{n$l~wKmkE#k64OB2(VZq58_FJBg69d5) z7YG_0mcfa`9onG7@;HIeM~90L+H%L!u`Ghec|O*IX&n6cn2%`~$HnZ2a(uw&qsXw& zzrEuJ%}<}TS6cUXR9H+aq&Qh+hd6NjGjT#l3B|$w+gN;Jf>^MDExWhuFjd5<#FT)3 ziqzdZ#h1q})6<9PPEs7Etv>anoi?;2DXyLA^96)Wo?DQ0S@oQ**fot+s5hS$Z1$EZoi~zc zBZg4@;<;>F$p&KSo447v#w|oM!BSBu{hqkBZn~%l)6jD7ON+6sB3RukYf0+XUDRsH za57G(p~c(Gp^5wStk>ykG-;`VmF;wroSlD^hSb|dM)=t>Te}KmM8Zyz>|B?OTaZEL zzTH4iubsw{?xu@{f8R@^cU7TXuTNxi4_u(ty?xljk}JfwAGa`vPJfDPyA`2nDN&-` zgF7tn;R~`MaUB`ee>l0g$4d;?He;5hf2F=v=JQVfPP(psJ$}Id_L)v-*0sy~7SD!y5efRX2R-Do58_Z19O-fE7by;1#^O6p_tmuP{?zbBysHP~T{zV@9 zjXuBGZ~vH0(!wJ%q-H69$Wv0gcgi~onwRfZ)^5R~Z-8*&NHblU`jlVr z1}2rb-dP$oW3#T-6q)W!KRcn~oKC`mU+eDpW8-t(`|*y_+@Z&W9>?12Ha^H1RsSe` z82atkv^;C%QvemC6_4bv7_nJMpZTtaI~VRkxN9Kg#T)fVaq;FnQd_(k|CwThI8xLq zfUl}V$gc^2LGag1C;w_B6;eV`xxZxc?@bP^B+(|kD zIs>`_x&wj$YCteRle?}TG0ZPtzREL{$qtsF{Yo^UYi!<7&$WB0zc!4vwf=>M!dK8~ z;dAuS_yKgx>Uz|7VG^BezJZo+vx&AI-JHhHNTZE@Ng`oCAE)KUmL>ZyT%-}+H|gZ& zG1S6*Cz(+EDy?+^ew`Uvj;?roiF&twNEeK~H|> z$l(VC$@&A==;X&kh3)+*&m^7h9_L~7$@C`pm_kN31dTGp541UKh<9*!L0v}vMr*qV zP@k~oL_aHumL774ChvBm=GwC)*~^8h`(8!cAKCG%mzRPK9S-9ySL1OCA~?~*>4@Nj z4hbg$RTl!MDS{I}oU#ZclL(x?2wNj?DkD&(A#hqF?2W)Fj^Ij`KZ}F+w*F6F@cX|$ zlXHUl5x8_9sv6-2KEmCI>fj?#ts-h0;SD|lmlgynLsgQJZC&l z4}d42Ccq0|P`}{+MfE}&1&CiRS&*g#GzJ6!n(}-IH|JsWA>PvH8~*1mh#TH%1iu}i z1E3S23!odI2cRdQ7a#=CJ9k|_VwhjQuPW+LRJ{20KlGJ~M47TK_(kD)AY2QjNM(($ z+`o#?{KNrZ2iLRltBq<@yL2Js%V~C+{LgK4C&Xb)8tMt8hX3I>pxDu-_>GAGn-8-^ zxE2BCK|>5*6>I?c`|6ic)~7Oh1%}!e6i=z#DO0KdZg5<<7M$Wy8GOAk7sbOb4kE?F z|5*$vo)S(ODIT^#B{uBHz#hwBJfLRj(uTG*hBm|XQ=Rsd|5GSl{>mIDpc%j%U4949ckf?t5!@$#6Dj+e`h{q--F<1Rm2UKh731OYq_+Vce; zBn+>0_LOe@!EY4$D5I#<_}dCit<*Brc{wUzp6q|Zx?6EQx&#}3l!!% z)+%buKg-6Ht|)5SmSxjheGm&TdnBG-s}hG@d?b!3d{a@~Yb1OxUPs+SZn9n_&BTsb zQM7ZPB`o1(H`-x&0&|$-#r6+VvgqJoam%6$tXwC#*s1X)_RcR&w2K|3aA|*xUg$7@ z%$VMpESTy{(r%`bT{i?~bE_v^)18UUhDB4w<-6px;taVoxgS&acuLmnXhRmB3#Nq{ z`qQ`Xw~&RmRsl{3d#l z;g%o#GP{@5%kFlUO5PqMe_Xqk(EGZL5Zr4&>G0;XklAXXJfv}~(6iL+tlt`RlJ7aU zRqxQr&#&^)kJ5||O5tiikkmfCt@LOxFr=IaaVX=Z<;RZ%GJ>=s&Y?0Y+*rplCP6ACS#2>KDnV(?)F}x-M%flmcHcdH;M>^bks6C_^*w1xv$cE1+-i#Yz#= z-mY(spO$ExSoI_d1_W+RhR;~QV?@3+@jU;tQwHN_(F5QkdzgkVQmW<^T=M1C6kzvi z5#|RJ@tuJGy)oaKRDqT{+?r5a1vE;CH+=SDp-lF7Z%z4&3LHRu(r}V(4rw@9(B~5i z<2dnfGNI!(*|F?&9C&DlWza|4FYO@7;4mKtEU!O1HM<;!bC<&j{j*f;-; zQu`7fNHAUsJ4?3!?SUeZ)JoKD~P9 z7TH@Vlv+pqPM?d{Xf3a?WK8{y;@Dw*6={DgrESM;XYb60iXqD~#lGjWXvfaQY50Jj z$%wVlWNK}ZB#rxxo?7TBS{EoF7JhS5QSIXh@%q3ta`e~rR(!iLM6go(e;^cxyGL)yN}Qx`f=CRs(g z3wPh1@;g2IoW4QzcJQA)Md=Fu87#T?sw^C@wMF+Z%SN7Z-i4I&o-JQsmZ^)4Yb%&P zvCvH^d0LvtHtQRlZ7z*(p_NvwyC_`GY^0kvdVs#xJ2#!Sodv0J&RQ7R<5!{Vi;}vN zN9qYqB}VS}uxzvL?hY%V$gHb=t~bs67M+cf1~%CtDAX@?_aE(+500)N^$1E9(rRti zO=oNT0{ZpS-M!|JwY2tFskhl1-5)iE3d~(5=zBbrht%1xTXAu>VA+16>1zF>I4<7a z{LdXfXJoiGJd!`d#e?AriK`W^sJQxB;g$#ffAo-97O-3vpOwGsn(fNn* zuCc+9g~KUr{%91HCD~H52M?&d`FvrG*%NZ1!ei=HVJmWcYx_bZV>hae8yrSjg=qp!Q#w$Y;sh{G{6sWt zI7&Eg+;I4R?1PMx?#yPfcuODS%F>B4)$J5LzT}24z3~kPKR2!PfZOBHKln)lzTp`D zW1sSghX340j0TnpKRZ%w$atuPzENG-(GoWtWWC>O!^xky;(+1I6@mtbgD1Dc0fn#u z;G{x^igBE9=x{=z<9>E5`!Cw%m5#ZXhsWZ_<36VUt&R_vd>AYC$$##6<@xbdxl-0) zkm9vw92=FeUa>xXo1*BVrBpT4pLMh=N?rCJSFCGRRSa2tQe5-&GOe(0F{^daO_32; zln%A=6}{YR;+`9?%7?OmD_QZrdTA)N%1K8dJ7;?K<4j3iBhQLNu{Cxjx_QhdRDlZMTe+PjxJ=JJUYI z?}Ps^(x_pOE@OSVzCq`gQoj|QjqSNM?wqT z@Jrd{PQs2=m3K}bAQkzkqWo#Ri@x;q$Bm*D&2*bzY$N+_s2T>fZb~ff>V)|T1%>oQ z&2_&0skG5|lQ8-~Kgqrx$x3|qRPVk&ME-)u&Ocg3+L}&H(Y%Vn%a?5XH>;T=8vlpeF zL+0=<{AYd045`_F;`rGj6Sd`${Fx{o23I&-@k~qY-|jXu(W;((HTmxC~F z|EBw{5UvPt1UMVZRDrN7z{E%){1-NAv^D)FG`#-**5{|6FXQy3gBm;1t63S;x8O5c zAmu615%_@Dp1DF|#n;eM8C^05d9#dEUbm389MX=s*Mp5N+tEoA2zfbgC2_o(h$+-> zNF`cpfv0e@Lr;v8v^5FT{<0G-yX2!#be+2)jqW*3Q+oeG!{;Q@sM*cnrKChm7p!g3 z`=*YceO}DccR2KIuA|`)K;Wi?z~O-48yDX^k!^9tg&Q7jk_a3w2s{f!eoPXF4+7~q zf-_1SMhIXF2H=oFAY0>%?cZUPmBGF;Kk=_HNF4Tfo_K!zeDFN5?RGfqz_&O0rWaK( zjt=s$orZQ9i+DFiG$|0iUr*V_DOGf`ajH;rRJ~l_~ln;}bgp>uZxJ zW9#9S!aoybBBoLJ!8a`V_)`d6@(qj;?#(Le@09T+U-e5h_|K4l@tFrs$0c70=O*?9 zn}R!Kxa4Eq2)s9H_}~>R`S`Mk>aCRy)9Fq~Z{c$cmwb%>T}yucj1UI^zvl=+!(iY( z4h$aVK2OWe$sRa3DL9+sc03;^6Ur|vm)nL<8k}4hMu!9Nn>s!K@?m~>?*ED7)#k_f zd=>A`hAWchHCM#M5Jkds7ZwuguL!UEfbLrr1o~U8h`alaOrB}U+SFAlx?k+TG-K5? z{YE8nsB9s+rEg6|l+PmO-Q^Q`d9aqEoFEx)^%q_kWhUod=+ z?q|h(!F**&((*U86sdBM|586wS{6}F_aJJnwCZ)V)O6Epc}m1Bsafz5og0H6O3zUX z4wEL!XFZ)J-0-L-pEv51PIE^s-1@z-R6Zh5NG?xHYNJG9wT*j4tHRC}OaC^mb5 zuu0KTm)`gfN!{;~T(@$TZorAog40jyrNr`fekXc&mHggzk*D0AEq4r>qFYkGq2yY_ zO}_HYRnyh|N8Ph*@}JD&Jj?L&N7h(`NAhQlcwk)Na23N<&~X0}e53$)Qwg_!16#!H z48pk0m*!g=gbj=ld;?oF+BieJ3cwX$V2j{a2YmIurLk-yutPmd01ROGPc6Xr2dMaQ zzqil3F<mpFivOe zFYF`i7fw>}{6Bj$0H+Y7;rW%pFggfxhHhZN#vJ2eiY&QDs2;G zs&}!s&;~bK^p!Af_-3m*^-HEId}TX)BS7iCQ-+&u0KL_U%3u|H=uuATkZ54ls+((c zBBq6^v=lN`NB6W)yiSxU-a~zLsf=&7s`rQ2C_ce4C$E1HP~zM$(Fu+Vo2{}ulsPiI z>Hm7O&7Ym~{RIai4odDr&@fQZ=Yti)IB0V_wCCXI7{|$vK01_pSO*S#Oydg$1pngD z4jr!(%W%i@c>KHhd_d;I^05!UjpGN(kM}oGyt?P57_y>{BDO{?Ma-dgtfKC`B6?UB ztv+|JqW1QlVv_S6IwNyAU9!(kZ0hQ+kiUo_Gk&Q>^aHwz7WcO+>`6aSnEsZH@4SvY zJp?~%8o81UE>=yn={cXR8*owKUZ?}x9rzYLp!!0LzdT)BHnB9F-QlWOy?G^8Yg9pc zw8utrXJaigs=YIFNgGTvth%z59(7pvn#EZ}sVemPR(J8f^=h$c!2+yzgJ;aCk2zV~ z;V^xZ=p#m5O(FVUHnO!gK4M&%bQ-LANET20iB@|(U36NLPL{TM%gVjApbr0!yzc;u zTIt?j1r!mmqku?J5KurA3$V#dl7&UF_b&E^y;nerSP%iflHk8p)SGbC)*aZyG<;EwD8Vf~xek z&Z@MAjo7no3p4!b6!EV+Vaz9kU}TOS%BP&MMiq?;tBZv$7Z=qU!SZcdNDjOsvRk!X zRrA9TW=Dl=aUSD|xXP_XOIe1R*>sAn`PoiY&7p)eGP{IS+xRwHI&KHMt=c>`?DAaJ zg#V=Wec4Mg8kUBt&GJC)dzWUz4mmNGl+~nZl|7m2)%r^F7S}+LM;uuL zx522)q2RQ-w{NlwU)abq+fEeM-W?=utm7munRAedJ@}YS?vWuj4~t|t#XWXLbxsUV z+Q<5Ki(_RYV$u#+ev#T`7^*%!oxskoZYM5!SWz@vZ6(>PRj?hJj%6xZ?h=phSi$VI z5p;L)j~XWk1)v9L&0&OKP6PT!2yjWDVxUz)>jKRxM(~dj0*TYiClj|k!9PBN0j8eB ze>I=1jnhyatv}C^0;GYE<6+Ljo4}Vqw|1BK5doKz>)-oNn@jIVK)}93(y)^xN2QVu z{<<86pE``=RED_c(zEQm!CO(a9^sG% zf&*&Bqo++X(YvY&Y+gGdCzdqZyG>2m`-LXc@W18K*2;_V3n4kTlU}=^7(b8!fT4*~4Q9WQCF?ATYF^Uom?UZvhVg zaW!+>T0dc}IV&Z!;jcDd)y6rKJbLH2Nt$kcoA~Py_z=hmbZdQy9~1oJ2b*yIf)v`Z z@@4P>CS^gPq&OuB+7j(>K^uE_mx41U|iVWrsHdKX=K!CZ^Fn_X55X_5%x#JQ^ z9%AnJ9%S%R|7J3nJLOyHCT~aLiMfO8D5{5&dGs=ml@BJQ3Ga!&v{!Fx?i5Dpnw1=o z4?xWwtOZu?|Kk3;%$6A z8)#y3;^;Bd)h1~ixGpG%+(OTnJFY;aen)VXUThbYKvs}-A?!FD5%qqq2vv9!pYmy^VUnytjN!3T6RY+_NXa;2fx5) zs+w|>3k$g9@_8t|T5IlYyH?zZ@wRC9l4WSp_P^1z=Pl9v+FNO3{U**`r@%fy?VYx3 zZ@BkfuOiaO+A-qCodX%G<0Yh7J9?&#{GecNZm%e|xVl&>8Tm?GYdvWS!XR73xUCER`zPBHfvAyNY6F3ujcu zmtm4EZZhHb%1Oaty_t9FvErjqhT=V+$?}(OkOw7$TgOVEIR1`QX|FlaD=#Q$GeG|8i3 z(8RUY9+U5C43aNT)=c8-7NptD?_9JQ^fF^_T*l`)E)mO-QpvE%mU`G~StE3;lMI*7 zx{a61UGb(e%P=ZqjF+aa$776x(XEawwlL|6U0!JH0`5^Z8vFNG;0fb;!8uUl8|m1- z!BNcE4@TK9qi{pd6}bG|mWZj;8l7Ky0#_?L9h-VxK&u`HLH^$%v=#DVyxt+C{c;t8 zGF~u*0H`DYx(5IZF8~Hy0BT^tAOe8>0bn2oFx7%GUSNfk84?Dhd5r_Fa{kr1ZrT9( z%u&PR03^T=x)X7sj!-wfY?h>Hm@IKCYLCfxl?KUIoxq)-20={%FM>J*_2Bs5A+#Ox zWvJfp835Su5{v_^u25dTwHV(ZtdxA7OnGW^X=Uc}o3M31Y(^PSIR1(<8Ft=rF zDb8+W5DM#j9rxdV3>Q|H!}oeJoPltjb4r@aRW4;9IPR;A-Mc(M>N*o~V7YkgTc!zK zcVIo&%H{%h{N-%*1L=gn`a_flQ>+HKZ#DY z985|tLl$j|Vf1AdPW5?%uAVE6GL*xS-x!IV7brtBtta8CPs-x4=ZE9Yw~cVwzBAA@ z@`0^pCx+q3s9KuiYnIOfH@BeA@5ZAO509as>loX#sfU+;7>F0oKgP~FHWRyYossiC zLJY{Y@2uaC_AgJs^-hLkvvinhlcAubP2m1cp{$==K0$lmJ^T?1&4n*TR9g0C+U$;{lI|lNMEp3vkxLmAKzC zud5{afP?^R*FAL8Q3G0H5*_Q04G##k0mcjGfytpUH90~6{WnJD|#geu%aWw7c(VG=4+ zu2?b__I7~z+koQBuL9$SbAbs}R7#fxlxnmF%W|#j zc4)hT5PBJX99}6r&8{V6xkhkp=ROnGHC`kX+Ie_TXl`Y$*&}9n0Mf?#VU0aF`z!IfT1-DH9u;UB-tqf(XBE zkM@{fz*fz!py$^b@ZIWTZrv`1t1{s=etx#6ps2BncRAk^@2ORh+hx0i^F4YHMGr2; zH@!NOGg%tWjj4Qy^O^MtMTC#TYc_}Rm6yEd#;&vEEMEn3-lIIwZp8)8Y;FfUIbuC- zy=)}!-!5G!z0jXqU8pX%Y116MG^HPkyn7HEY&gWNTJMOv_^COAcdhx#?{{#urVYn0 z$Dc&u##hnDa}T(7(OLM>AO{?sbRHdBFbVga$@Hq_ff4{DuWj>O!IMaF@AAF z`MEC)GGF5&_D@x*9gN>f9bdUKrU%c8t=ncHC*vS-X6H`GvFs7G4=yH-KXePZy;>*l zYg7|;Iv>nVt}Tdd_oS#xD0{Fm*Ndu(Z}E|$BSOWv%R5!|?6#ojSNB!B5E)IR|HZ}) zeZsWNY9WsQ&(d`AcyYjai#ft0XS$7JIdwHDlRxOIq2s(Ng~w zWG|`DvGUsk!Y}>~f7BeXhNAu86HIFgyj=HV z0{y%kF7XduPU1kP>HS?ENz<&dCaxs`5EdXQKu&;==ssCR4KRrVG6Dp|h2X0_(MZ^h zz=Ob(=Kooj)?F`;u7#esRltGeE1)ks$|BL)5B07RjSZbkF(YQ*JcWIuDyJnk1{@jA3N~6*4qzG(u+8N_j zak2AgKB6aX%WS_05-8)&edE z>jrZFu1?V?RK{a6drH)^=67|Vo=HB1KjEI*-^77>hWGjk!Z8DitwBorKr;n72=aZa zo?$l!_GNoR!irp>W($_X62W+&Q9w);O=WuMU7AO>%| zaU4&ra#?6GV)K(BjN{o4sOS=v?NB!!D_9wTohw zLpye9#})FAQK?cyhilBmJqzUCRoaNwO;54Ai~eV~wEwR-RcYG!evng9BS0Q(4YwjL z7|v3;;=x#Ie!)iEJb4v~|B6SEu!cLqIf(z4`4cTmZkJX!A5Qh+Ln?akbU3;%VRW+b zSadNMUTSItPjmN03n#dsZvX|Int2e8$V%7&-tx+O2T@#G`N}a?} z^Ze2J6HU?WSEtbDjwNyZsfJh{oPyoI+(J|Cf|-@!4iUuk6}-^e76hjPa|Hl%0r++f zk#Z;?kbobZGyHZ0I%2J`>83C$A#$ zU-2jshMQUg&OzK?=1;UNxm{Y{{5ci;no7f$?8vTG*9gC=PS!$d)wA-5WfMhVGH>SU zeHtf{c&&O4sQl6&KJEa&!U`atUkq^X)d}8w!+DhCS6Km0rTg$+IrD6Ayn?g~r17Lk zpq}A#kN>86&Yx2Or=fg_c)#LRx%p1+xh>kEsSEjN2M0d@xLEFEK<80@26+oy=nej@ zoa)hhXCX`d5eHk86owxBB6LfBi;nl5jE%ZB!dphn3U7P1 zkKcT!3AQgejO%lFKT3(HjVGP);vC#(V*bJa?&9P?ZiZbT@9*0LO&;A=*M7h09d&_m zD*Lnr#$UYG-n%RHeD0>IGBSe_R*Q`U{#dupHU=jn)4+o_SStiqI4uv zN9xJ+Jh+j4*QA$<^)g|vCe)N~o^V3lqEZVnZo(DTcf?b9+=+>7H@9V}BSlVpWR{Zh z&|nK2b$^}s=#mRtIVM6{Y=hEz^(v#fH8Y!K6EEso?~md_Fv8AXXuV)IsSFM1XFhO3 z&?W|2RSwe%U;@GP{bRpTs-8(&p3Fyk4qKY_bRhbDag6RMV`){hnEyUu7C#!OPMD=ED%+c0v#Q0yhHU!G#V0E{LW_ z=Xk&GKW$JwmK4gEzt9utn~Sg)vig@79<;Z_#v!GJTjs^N`e7%9Ok;!#-?@nH8*JyQ zhqV_jm14MwH#ijj+8TwOjmNk5SmHk6^@MZThWPM|8l2nu*LZoqx#$V`_SZVYYUpmj z8~v%X=i*Ivo(9BNIl-Q z@P1`-oK>bq%3~@<TB^V0X4$&2;NrQ(CsZAN$O_sjr2xO7dRYPa2vnL+D0L(e+~` zX6cpUlK-~Bs_l-G*!pIF`z9u0CT+xCF@Ez&DZX8_IA_l=R@jxncG#&FBOA{aV|G+y zR=wHCw90zLZW-QQ^sF+9S?*L{%vxHN*|pAyc8Nc0Z$ubef4|T=g00kPPyMzMPOeev zpzMhMD_cpzFg(=E5D-^qAE{y_ae8M6wCAfz@_e;7H$C=OTv|uHw$%--4G%>Ld_Qt^ z8Cn~7fZY*09vu_ApuJgF+0<7(kmAE*y!z!T%$<%xu?u~0*=r`WE7v99mT7Jq zar3jqKzjh7egWUwWEc@<7~XGUd0gJh3;T>3$VWeTKt2+f zjFulAfh`^8V#8)V1iu>dxW}A5@0i>Q^GJqA9?aw>PcF$BnR#$oz1s7&j<3NnL?>qY z3`et*dh?C9T5?g!!6?GG4%#e#MJx52c!*bl?e|wJ%{<6OZZzV9G-G)o_KD9X-%@>c zsp1F4tNo{xMb%Dx5lh^QlG?`{lP`)-kc7}TY@knPDc<>IT9&<+&-+{B*vK|9(w9NL zD#uU5*szbMeW%nNga%h$Ew#C|OYQJ+5Hs(_E_IO&UaX&wLY%qL5}Ax%kyhL_8kx6s zVYsC)*!1#s*zy_v(z*9pQh)o3%&BH&rFnI-nQF65Sc}j{;-(_WY|%BGWDzb&ve8Z1 ze$qvG|8{*<8v-Z#M)!zfqw7}^tyxwYbL776F8opB_gdfo?C5gRngUBHM+5p<3S5Gk zyF3|+R^5NK3*WaYo}2^BRGy4Qd%Pn_*Dx01RwDq-;F0@0B>cke*iLHoKUV3Q=W zX507%+BEDq@`xCWmYVyZ%Y8ONKJ;p5cU+>|M_hMSKdelAhSpdQK}ze>R9zMb>KViN$Uis_{8CWEp2$o}ZuN|BknT8o22c#QPPi zf%tEQG9u{M)DL;+`9KHK^MMQbXd3Zq#DIE!)}=RC4{Q1rwXcsm)e`Qt9V%QOpT&(E z{0`4+SxLBRyO8t9x{l|W*2Zo7W3+M1d+zPb^}_M^SZophf-~`Rz@4+5(30lSD9mXl zE`RFQ|Cx#L`PCp#1`yqU&zp8JfeJ^dNS)lLx7 z-8$fQu~{hojtNTkyN{y#)Wl0xjpNGS9E-v)oW+jy=i-uMZg9rqd*Y3?D&XatuHwVN zM!0C_*_>^sH$tb9RfWn8V$u19cIa~CPTZ?}INIFnKi2$DdzT@k*?qju)LXkw?0Ppw zzK?5*3Q4hQW1Ev=wW58QWraP(5uK_dyXelU@j?5U!X+n4;SH~dUF#NSuX{wWtxl$k zE3PkB4PW+Ix}9x_nsjAY`+zs9`=1W03z<1cTgwKr>%Ggcg(9{i!-hTCvQ^xr-Cfef zRqei@kqLh@oky6d?bFPefT%I-G!Hx0uh2d5-t9)>0MrM09xtix+M_p{>gp{uiEYW) z9&W=V*;t5;zhtx0>-CJogfA-NyIE|d)=J6Htv1_cp&?WA+*YySz1i%Z0diH_Nn@nq zs}HeNO+QPc4j8Hic|28>iJK{zj_bwFACN6^o6@9Z_1cTgPZVaSEK`c-?APg@f zo1T}tB+i$BAwUHG`rc>sy0p3Uay$XW^}c~C7)?Z5BYLALr-mcc8KX6E{m|B>U9h}k zBXlCTCbGXuPeckYlVMwPqV1|bN3SYQ!jsN!$6JvN8la402RbF9g{T{zQtTZLzj+H) zHdATx*Y}xnxH?#K(p>+rah0NY;4=}mtb75z zsyi-M(!c7ZmCTDH{#$#XnUNX+fOi3YcIJ!zW)8YA?Z&_#VP*79z=xifx+G3c zz!G4Be|_&WdR^LH`EpbN_@ZX-vvjgn=26uf@C^|7-C976QX?TPz%Ax2e9t4dxr1*P zLmI5Dfi=hhPD$M`CVR2JCtR28ND)wOQ7r6ij7c6acgi>A-@;eo6x9iDC7f9E<`43e zC=~&}v-{TE1(=07@aRgr7JZr%%i5SkCJ_pLx0!~-{``M=Fm;9=Uq^l)Tot6vp*34NSd5bpN7EevftQD{G* zfzW4b2xk@Y1TSu8EVP=vS1{eNlHdJq7k>Yitox~W06ksM1oN%>3Qafd!wE|-@eAt= z;wN_9!LQyno-^T2;T3nfVRxrE!N1r+&J~@-wxM^hrIf*sI=z$|+T$&6cHk~|v+e?Z z@x5nw`uxqBG5vJi?uC-UcmcS&`c~#GUUsuOV*ub`Y9Bi+(^G26u6%lG}xy#jEybh;<GjI`{NzvebN1)?a;+|g!ZZW(Z>2s^Oyy< z2by*22_{N<@^Y+ifMrW**aH>Q(L+e(C%k0q99_xO4YC&-UL1=|)kB$*9UHMj8~cib z$C!#I#WZ$@Pl`BV(i3Le(vL&w^m0LKG(Kewnm)oGO*&AQoLM7ilGk4Fpc$!N=**Coa9#S952%^6B0flDt+lUY|?LlP_1W zhHr=A8-x>()jRYt1s(v8DXi6nU8po`bS*<=@R;&;WR6Swm@3bX*dE|A$qn)=8mG&? z_c7IPi)jOa9fd}s0ym8cZNS`N>Zdo5+F=@>r%M}1kFA5U=o5OVJ{x8&+>S1RyH*Vn zG7s#*#kU)w^;x@wo2Oo*=f{c){Z1bc+McS4TNEB7^q!oCR)6sj+C(^`{e5bon+Y5E z<>?m$i>gid?O}WPk5RF_)8t)zrFZWIqdhCQaJSBQiGt_XE}p=@xMYl{9-cr8^P6l$ z%>vs8$s;A=`4Qj1C|3#JSS264FHv%R#;S^*xx;#Hd(WO|G*w!CdazH%(E4hhGfSBY zV-K?TRTm|L4OLLU((BR-@0;v}qCWDw+nb1IN{v^=pD!gXu>Z^!j*3^8iN20H`P5}T zjb6*{+@%%|WL0L?6n?ID@t7%h@bAY4O#M5}po%lYA2*Tj_OKOKWgnE#rCZF@p>L#! zVpBvf<8jR8q0QK{FC?ktpj778gAS5))A6kH!Q0ZDQ3u61N8JR%pSAYJLtl^96HogV z*r5tEpx+L`={0r;loIh%<@%NVA@Q1d6XI%YjyCT<-eXb4M)FlAfI(zcg0EKJkgyxg z_p2_go?g4=fkw99hkB+?B3G56L-kML8plRp*P17>L79HIQno3sW&9aMbbp3g@9c<+ z)#!msm8cGRvE$XZxTx$St8V%lZ8&!cmngLs;z6)e?=|eJBx7`l>}Z+UiOY#4=YN)7 z7@c0Jvx_wTXI)ymyzHb$2x+@qCx%<7xsAl7suF8a}&@l z0N>68s?CuGi8h<2O563$F+z}*=BMo2AFoo|u&3a6M%U!UyY0@x9Iu`HoSvIddb~4s@afMNTAPd~27ke4%m2;Uq#VKC zrK_SDi^kza>*QQe>i zBTWjND}K4#pV{j2kTIWjL$XWNP0IiO!(}XKM=M}1LllPQ8#ti&i64vA(-hgL5iE)#U-3~F@j3AwrqO|>va#aCBDbIx8z>!T;KnZ@d}k0(ZB z`^GDluYp`1qZeuY-7pt%ewmu|j* z_`l*aBwU3+$76_Jvj+=t3yi~j+b#3wG6DQxdyL8Y7NSatBU82>VW7Mj6|C?f^Av@) zH`H*Lg@c|epUjx0sF$w!J+Y#EsEn#o&993rDs)qLr)&5QKRSo^K!;Rr_Apf*e|8>J zsdBO5ihoO$=Feq-uK+Irq5OpM8E}ECfJ@`Pp^a$XyQgv7uDeD+_3v3jc$Ubl9UP`9kVP%PY!Kk?}a7hbO! zj^DHqtt@sAJvq1scX)h`|G1(t-`!*!*D2DKf79v?fB4X5?0@JTej&TSnLe$H=B#Xj zyHD{%7uL(sbNb2aDe0HzISJ!Xk4elfAZ|QCpQ?C0bor%f5BpEqB@a49N={ zu}$ZE63uGZNS~)zusv=zQw^J6P8wP|LG@`|hJ+8CK%FwXi+kRzW;&Gon+@4ki&->z z9qY2gTU@!Pomj`?6dN>Xtkh#wecuMJ{KZ4pyydIkEn?4(eZV9wl*C2tt}!E%Y~=Q( zeAqH(B2#qiQdqva~P_W~qM_dY4v`O&g>{OMR<>b&B`FV;p0*G)6uLE7eq=bLnkh2 zPQo78YKNvZ+KB3nvSp`^`Uq**aD_VxeYKP+w?7oM^GriW>%T`X=`5t>7Pjow<|x`X z7%Q%rpe<8xpx9l&c?Up~qb7=kY=Q6-m z{;{7NIh4nQ-w;)**-wtBQsuSTVVeEqPH*lA$ABtT?s0jkIesdcn<48p6pC~WpZQjm zDxad^IFWFhFI_jnz9`?SQgSKTuc*@exeVwM=!_uL&F!agl#_r9x(5XOl)q>q$Uo=+ z-wJeO=;+|m^W@G0X+JFV2L3iKbK}q|AvicjXyqUY-ES`Ej5jYAn%-ZDCkw79AtXWQ zwsr-IwBL)LmOU*r>aY}d=Mb(rbqU)2CKw-Ibx7#wI|5I>T?{wv8Yk4ZKZhrKyP}7R zrRXwq9i1rL8O^Xoqg!R4A#rnEK{@g!Iy-hH zwm*CoU2OgsrDkTMsL)wxb<}5EwRL0OY4|yAn~|kpftKJy>Vv$=`8h(B^pCo>`qQ|K zO=_2M-o6*zH%e2ZRKCTN>$9EiwPvJpg{9-4GJWs7-5`ycCsS9LT|{gmy=13*Fe!6d z1WPS%bQTSb8cWJ%z1iipQ`u7&-NomM`s#`|Zu?#|=+1hbFcrI$JLTKbE}ZSL_$brA z^cHEBu}9kW^XBq%?r(Li^+$0Kn5fV%v`)Y{biahv&p6;@Fd~5`Q^Pnw)q;?)hIwfH zniUBoPV+`clR^8B#Qo##Q10}+{IraEnSu@KeQOiiaae((h9;paJ0i(dG=i#Sk3?N2ut_u+$v>b5c|qu`$)B%j{2-Q`jmC^K9YLcw>euRezeF%9Fy-@mF#=a(_1omm@PFw zyh3jBPPd84pEdsuBZk~Un*uCEMHeS0g#eX6{6<<=w`NTn2GwQ;{@g-OuUFHT zK;BCxAg|{+^V^Pd-$O}JiMen^8T+E!GH`MDsgr1Zr>o%4E%fYpY3vdqq#U_2!wEIj zATBj_8iPl||IpO*#QA7mXcmovfa(&zW}O&ub!)`5@unnBGe&|i6?*=BHE0QVR}6Ek z&{c6L(U_#G!ul`hs>=1L^P#K4H&tM5n7*#6UlY;_fCdAB#sHz3Ew}p*4M!`WN2BTM za{6>w3E|p}M#44a-)R201H#qxvS_l%30J)XQPdH0;fgRFy^?MS@siV!=iC5Lqil{;Cu!w_c zf2GE;%THHePTX0`Du?WrKDK`-evCcMmV2Bf`gXr1kE!}eoRzqkowwUdQVl6BcK&>X z-T25wEIE9))M#cFn;PRTeR$D~dGq0w%5m}#RrcFa()wdjz8f-6OCD#01Af;?SNsefO5RUzxy>pelh20a-HgpSP35|JCjRIY<5Ye_Bnw+S?OVT-63u zIr|tj&Im*HrB)#Gx!u{Vg*US2tHq(pbq#U-!^Z5^LA^)ThT7J8Q=#&U9NOP zi;72qpItO;EHZc?uoitDK^O$ptdfabMIY>BQUmSjxD#8L+M>wrn{!ghF=F?P)c-3k ztz}+T#4{wPx#{_-6#<0^080V@WeI>=4fu8(Kq)750oo0~V47+H3KKxb1Nz=hmM3`t z^ljvSo8g0QquZ)HA8Q8PRb$PF3uTA$YQ_)5{m0mW#OYREL8l^N-ME4H|LY1YlBYU> zZZ?njHM<|fxqe0Hjm+1odh(%!P+&4s2X7?F8%KEC$dgW}5JF`nT80zJ3?2CuKPX)# z*W)PsDvK&z%~v0MNSG*H7{>${nrFa%N>u48Y;n$;OmZSvl?~T`d{pTwQ;IDtNTsXa zy3-p9oe>0XE(L@ko_1UaL+6EbD%AQz7&M(6nP zn`#Tg1`p%Y?W=IzOSeMHMpZ^>i8eUoRt2;uFdXfUazqJ6%h2rNhfz{W3OYFZKDuwb zj}I)K%K1k=!o!~$2Oc+c&+R{K4C$g9A{W`3#e zrRv(bSlce|_}e*}aR{)@2pTG!2!9mcAujU+|o5zw3iE zZpRhYx}|4IC#nj6b5gn|~C)^84;*Jim;z?!dV!(SUx=1sDBg&PC3l;ak)Yk+40% zS9}Z7NStm}81Z`$fIg~4P@4dLPvuSEOTZ8y0**kS=lxwSZ8p6e&JQ{DUxenyMxsFf zUC3_A8B~7D4b-hi0(#ci6m2d48l4^%$m|#!L`pA1XUvo+W5h%7gJ6rmp}1E6@pwX) ze%N%cE*{O99*5bsU9sDW6f}F*E_9^GM#zsYnLgJXCqJ2h{d)M|dR_J))gW7R#cMX>TeCrHPr@<3Q)sK@#R>gFa&A8@4|F*IHLw2`CoDy- z0ifIvh716GtM4f{v?h4~(5(T$&j1>3NL)KD?1>9-&;q(Vpo$ib#05BM0YeXfGk`W& zJ{%JsPiP~c7C+4~$+1v&-9FjG59Nh2Lb;$Ux}5^x5hG!Df*J%h3A_mE5Y!_eJnrk= z0Z3Q@$NWyA?UpZR^#cyc_aS@}cD4E(DuWeud=K(Ck$B-nsOIas;0MFVKD>ogRT<#OdUEWVZnxll0)VEtQ`>rYb*} z74d(XvYBs5;$Wp8c~7Cdymo$o#k~RlE5oRt14D-^N&ubCm;eM^2tx;?0uL_;#OLO) z5QfeQ$SnpW0Qta$G{^@+^W~0*>*WUI&0Q9H3>^PkgmySRoEqBf*M(k0AJ6s^t}M7M z^cl89XqxUKbYG~%i(KyufrpmjrKRd}aqSNX&CH+Rz*ui|HpX43zGou$*0Th<%pb;0 z!-{h)Y-ZvO9xr*z4UvNDN;CBMt`S;SZ-8Jtsy$!Ic_tUz*MVymdy%WxDH)ypycgH# zRRlY}%j6mun{p>w7;xoR+X(h98-&8C`|$zp#t+=rA$8f)Tr??)9lAMYCBRu2s zWqc~62wo!ZhoY@Nq6=F~;!DahIOT{DCk~v9i&nnJ87*CeW_lIly|>Nb?~IM%*DGT9 zI$hu3Xekn01Af;@Lnse@tjSfm7>dhV*l4xB%!!&}l{lpie>f!Q_=@9+MhN64rdpjkslU z#cAd>Nu1`_7R04Q1jq-FjmiX$1XbY};<^x2BXA=qkU2~^FRim)4(y3cA{U`)M;da4 zrv{^oEq9d7Ba!lwBAS9pWt8K(B zn-imn)68pzOYn^bDwCKZx!I z6=2=?s{hWYeV7y_pz2@+a6zhcVGa_MF7P?3bd~$+_$U)gT!$~SfznmjV;Ss#s@!u# zr6>`q*(FsOv#?D-1wI3mZh%W>9Q=}$C+Ft~3R>yv=WuXCfg=H+ z8WnmhbP6i8gFqPka2@~%k~&kMbwdN>!<@yCRoP#?eY6fRdeFPxk{i(41b zN;sBsk+W)J!iCPchF9#lA{;&O0A)QjL281zW28NgBMUgGm?-=g}GrTbiy7 z>*YYX^ZkgQR!c9}aYKeRXCp_4v24m`88WT87+KBO%P!44$J`uQ4;z$yf&EO?q|_Qg zRofp$5zjJo;j=}$VUzMR(S7fUXq7k+;@Q+dFq1k#U5husriCqVq0}kZ(CP{#pfuA8 zxSS$Em+1#Rv{rd>8-3rX$*smgl~YpyG!_8-005L0K+OP9TY#w+`Zx}qha+itk^t?an(t72~UEDAxTw*HW8{|E!(x$&J@D+T6a2GSj zI^b7&PV7n*q;mP8PQXQ|f`ng<@}$hbLll0YGEk849yYgvgmWl(EF*uKIWka?u-gXg zuJNl1Qa@LL8wMSV-ZZ#jbe;(Ol(P`8MrenlVK@$un|DB-+yWf|1oBZogmWJQ;qMlD z!}M^IpHr~(<;zbf`w^#m=L;*933nS#QN zZs3Ttb$q!(7kJCNBF=JNjFzQ06B7Qe%+1(phFL(xXnj+q+SYB>E z=e%gXv!&Xz`f}D~V8t{KtdPdKjAI_|*(6PP`c4cgTZ6q6uuwdEf1|YH!5rTWH&!uc zBNwa76gq(Fus&j}(Jj`Y{!sSvmLO?$qspkb^)j`c%~l8IZj>ld?mKS{?WdM)l_W%>*tvn!!yLUh6BXu6Q4^_7Uxuk%XhJT?)fv38w81| zf|-UDdPoD-X0UDiZma5yieo+*-xB{SyhWN;g6usx;I$;Bj$y2aUtvqEZzf*I*rzJ- z!Ay#@_VYcobGq2g-j+4&x>&M1$?NX!AH{p>fA{nK-!jtr11Hk`7EnJYf(!pJCnCpu z#f3=NmO#UOv_{8~gsT#0b`2x$S93rl{MFZQNI1{;tkhVMbVL9aLi4_rxPAmmn*MLQ zv`zJLu=?zU+rDVqk^89bbz3y_Ll@>)a7px`aUB%Nv_$D+PGQ;h_N2rb!QDGpVY5PY zuxk-J6fkNE+8urkNiH4H_8z6t3Fpb!VbEmAh*sQgjg}rPj1HA_fG~DnZG&3v{em3o z&4n;C;AtqhR8YgZ$>`COdMJKDA{rRm9L-nMWV#Kb$ue{+?Fw@5-2!bW;SZ50e&bQ> zX$Vs!zeQ*(=EeEyhmd=is}#^B0l7`+uW&vyaxMVyEWm%`fONh~-#piUS@z)1_3CItXj1Hj`#9~m&&?f}|=`EX2loT1GO34l%j-7q0ACD6+;N!nN3 zl7#IDpxjm9(IPIC)0sf`E|&PAOqz9m#Qo|kKqO4&q5k<^7S07i(l7x}pm{$_T3EhqRp_XYw#=KWcYO&L<#-z%FB*>`x0mIbUTw#< zaPq|u&z=`{Kg-0@(AIcI!#lY7el^!-;R3E!$TF1mSj|*<6v3 zhr$jq6VD&H2hDrZ0?)420YyZSeRxg!pqn8Uc+%jfoJBW#9QtY`zFX`#esJa)_6Zt- zW-SXq569id6YQE|!+~))Q)PqJnco*SoFB}MKT!`|x0hkAlL3mJ+5!iaios>-SfX2P zV(`%UP4JGEy>S!m zAN;MVc6ct^xzc*cy_#C?*E~voiMuQ`G9nQ)V??s|&W6jmQn>CgiA4y>EJd6?dPo9H9?gvR%PbaDe zR*Lp{d-#a>K6(#Y8U2}kEQ zV$MjecY@h5*tFf3jK}Rx_uCPRWxE;4s{cEWOWtycJgEPt6H&* ziX0S2-&)IV7|@FGAJ;`%)7f6EI67D|o@>IieRiFFZM#}^a*M6h_u+e{(uF(Hl1|md zBiW0k;xL8ErezAMF~a6K~O*- zoM1cUq#1RN%DFPsdx_98jZ%qJ`fVKoc_$m`P5>zE{A)qAAgZSaSP@dx) z5|0Qt0)fDfKuJ)aQX%3GBxpp?grFJ36SsxdZAJWT2-*>JAn2rx??U`t3Az*XBb|~f$<9evY#|cOtk6f}UJrPIx809J+Y_3j`-NsAWl{r8 z{JAqP*!t$9QDk->+~>($JdK|M@gUfUzl*+nsE>Bul%v)=rsc%``Tyx%&TAYwBqXPM z(^|qK4aiOV3`iL09RMD60F0FY@Yn+={R9;N04fM5u7xRa0iYTHFt!4G@3<1?_5u3F zmoOd!=o@FkI{*Nv9ssOq06?b!*lA%;T!4cX(AfZ0v;Z{)aMr?wxByozsuLFgV?6+< zF+dG1K&1g-EiPS;3uC|9TGSyfpsp5V&R(-t*O#~exfTp@0W6>)`3I8EIClt1hIrIR;*Qgag}ignU16Cn3$QVbM4>t3 zV9lBq@?5|dQaEyEzzuP*!bHQF-ieUHGaRhBUI?S@Siu*Iap}dQcbm zEg&2ZX#sYD;14zTDs36K?UqmHbrCt-RuSd!h33 zdc4iU>fG++m)NxVZmwnvHU9iQ3BTANo{xrynlRKvIVbv_~%)T4= zGR=(0?><}cr~kUo_sTBG9gAPe1%(X9C;Fe{A}Yzyl#p&H*2xSfg*)S|Ra)UVd<3@| zb{2`#4DkK{gsy(7&Dk$&%Hi?#@lLrJ-tee3_B%cm+w>}nH>*lw?@Lyk*%mL}`bb~y zxMc)x^yV>Iwd)kOde~kr)LqGy>6w8WPPF8c2KV8{-ulA(nSJJpM)+fO!WwSTmJ#?; zN;!Pj?=(&*Pu6`kS8%qCUgE#6SmKub8C-Tz8Qj#P3@&LoAGbdI60I1*;^L+*C@!NN zj_neUPA^}L&y|S8?R$A~HvNuZQ@_u6V$x@HKFS-1#WQGK+wSO@k0)9fbQ_%+(GN1Q*oBx7qq|Y8t&Mw7F?UopSk|hPR@OJORka4U--+MA^1tfVO+7st?;aE zBk)`+XI!LEQ#AeEEP9iF&$==LPtO9IkZF<@6iGBaTkoThVT+pb>@QcPLp~o+u~}n$=bbyHE?X3-Eh}9` zRg*3<$-6?;4%>SpuSb=ndmB1Qaq9(j`__WgeMhV0qbd8?(^ zUm3}Ez8$XW&+kyzw13HpsUIY>qXzOZy_{5w?B9zr`ABKXr?=9~s8)=h*+q85i8W$W zYBOo6!#pwC(LyR`+l#&0XtnC#*=~~0=POdX{Q@)Q#&ap5-axUU!k?8oU11g!DJwoz zWUvlBE;9Si50=b3_Y=Q}4OuJS@lwd=1hJIIf3|u4qh9pF_`M*7c7B-Vs7wR;r#ayA zV0OS%fms3*2jd$U*T6WY1VJeR%`}IWd+L4FOY%8t^AH~N&ksbAI2bE<5_~o7p*;rp zKX86T0Hi6;pVE?eKLRB|eS$!OM*l13OPg2k^g?E36_$@Xc@R8I+hD{ttqI;|tHa}VxC5$_U7aO~xlx_~#cgAb> z{Hd*whrPNe0k<{n1bz@~yDJerzE%g9c-0ttkY{+TvkjYds3O{W`Z?Mbq#-Y8MRsdE zyL|<=>Dc`!#>oO%yo3?ZKLlMd#pu&CgidcT#3hF`LcHVre`NYi5wvyl8rC-nA@@92 zIiL#zC<~;_3n~wQt_=XD2!L)5po|Q;PnFBCA0gHlN{i6Os?w650;h3)+dci#hR^B!K0SFlC`x zl1N_rL(_n2jx}*Qah;c1{;y5F{^28Z zVCcLQpb(4+pi@IfhIkMNQ-O3UG#$c_hvuU$%?B>zg=67C0ilWoY$z2t4)Rf##^-h+ z4FKn&<;|V`+x)Zx>lwB|+ZLqI%jp}GH4)AQ3BvZvGx!;$&kO56tmDRgoQ5x?+VG*1 zYYWTIm*pnaZ_nA38H2@Bj_Bl!VR&VHDDN(t#jUp9E+p(Y$tn5_;wqSR!WT_8b0&u( z(CjWNF%#>E*>aA!#o}o6u+lTmH25=ba;F(L7oEdJuH6t8bv=unKamYn?;j@{gv-(P z7H_#id#~XREzjZB=~1}PeRsUf>KV$uVS%Qxwb1-g$>`ShIVe4O9=^ZfEZ4o&bZ({X zRn9E41XpqKVl@4|EehS2D0r9I!GGxT2>q3G9!2`Au;sI*Xx)=e_|dU5Xf{6$w|@8x z&o8kb?cTZoeSTLOC9mv*US3&=u4WBIFYaXFAsM|<;$N?^ck+b)SKXDs#niq3DZN^x zM2M8Mr?Q3A%)RH_nbe@{*=ZA^g$j`vk*$)Y6teGIcA}X(vc0^v5VG$tZ?=~g+2Vhm zF=vd(|M&5F{eJKH^zA%nyXT&BpL5UmIdks4IQ3LE=hJ;OdS$SRo8!L5Vbn>*<-E;CQ_ zTd)7cH{^1e#*?GimMdQ>CI{#;ftG(N!wys{*!&bR<&z#OE!Zcle!8<#XIo!3v#XGw zTV9X3o^^q7GB8v&lDy^X>2zimkEzOU7%^Khd8~u?!=;IQz}&%XyDEjEWkn14^u1P0 zY72d~SyQC3kU;6+ZmY3aV2USUQv#LMLZ^HZ z21-_`Ha8OsW+-vLZBwwwF&LXw?}J@s!?FIVK}ea} zpS2HNO3VFIKJAsSSqRJoaxJtf29*KONfy)x0FyERR0#l+H~=PLKt;>odo2QB%7DLX z2{aFQm#R*4j{EuRGu?=J&9El1MWA!W?nu8CiJe zfe+l&mOHuFNk%v((3G=Z_)HwQRTqyOmy6RS2N3TcjWT{>Qq6Lh-Hnbb?e_p(fJ`G{Yz{n*+lYv*(PyvfTb^ zr0)yt-a8Y`@~wdli+yp8?TB{DPvWVqo6`#a?k38>fcu}(>?4sA@ z{2Q0oY~Wc#*5{&ysC>u%%&w_3TDYr z{$R8}d&ky;5d_p$HoW1?)(R?72+u!bXKmcc->!XL>G`rFQ`}#Q6g~P=9)5}a z^O2HoX>uExB>VF^r(BiYET;0;G9;pQI?h3TyHq(>|~UyKVTio!yvv{gyCJLk6L`mo_ORw>B~F z{xD+`Iz5y(zn!KK=+@y=>Kstm358^1q#rL{y^6^+E>{fH*~+I^ozC~#X3Ux#k~5d8 zm&vDHFQ*sh-?gbR@cZux-zZ7@0{F3Jw`b-27)}E2#cy02F)0i8Ohmw`(>{A}i^R7h=s*xa04oe# z2vi#yA^(5d(Q8mC{|-cEh2zk%E!|Pu9??iNE{By*{u3>l5rA@U#-Y^2Fx)c_dR)LJ z+S#Cs>Fsd+ThFoHTU%^9;}!CFUj%8m@i1>x|4}f=Y{{_xxPJTM zH@ItsHW$DGs{j?cr=Ywxr_ji&olr*M6}0|9KZqAiSOK2OVZrD{*wAVTTDUYBBCwuC z^hbYsOSDwi0F_O@jt zJph3E1AsOFv?&bmUKkbtN;7~11Ax{5jMb=&@0*cy0IY)3m2lwq6<7XYXaRsb11j== zxR?MdHLQsbfZ+$QQNxz_02qb<;MV{cl5|*7Q`MEkxv2qWD**U608CgwGc{o70$}yM z1?})}*bi`ff&PU)g?@zogMNelG9jozpn35Uk-xATU?`J#8vdnn3^q0B#u6!N7pL5Km=DgE+V$frEH$2YCTBorb}O1qkrL!7-?# zaTSjR=@3WL!Pk}{{lD(0!dGHnL7)G4M=Pz&?h2F?i(Di@N7hKBn>R>$ZSugkqsNJ- z8J?5qt=}i{+L6kY9b!=OyEMFib8qgjOd@tWJ53xky|#G%tTKsb{y|iJ`%msSx4w8+ z*c3dliW_d!cM3PVGkGvT(G-Vgg^I1mwh%W%Q5@sA1f@DG6=x3&7RSumFD_f>h!ebv zxeX3FC_SSP&-pM9Wu9DuHroPG=3ne+W4K zkFzJl=7l_V@8*tfOtr`LecaK*+81%>H;HI*LNmPCZ30$IDn>`9*XLZ~b;W~5ePZ%I z=i#_EKkVfi!UDzl!Sd$mI&9}flSRy3Z^i60X3PRvDtjWKzi7hNp1eVmQiZvRA)npv zslwZ@HghX5f^jZSV;!5FVYc{Z$ZIUzEq9$YT3KshRiz-=UhXz?7%!?eg_(A`pVB1K zf_WG1%5RW|ibg-Fu1Gi@%C@_4mybL?Nz~KBTuMf~6nx0o3bs}#arkNJMj<}$60TwyJ94fkF*Xv=&U^+Nt4vYSFNu03;V;x0aE z&KQNG*#c(wthwYX#|eu4R`%=&(J4N#uRz}0>Jf9OuAXB5LCu#a|6TT$@Ne8B=*s6+;e_Qjql?K*y>O#7x%Ye&f5Eeu7bY0ue#)-RE%$50g1d>EUybrzEp zy9MbT0J~CvHqGvZf{)H)r!JR3m^$=gCxdQ!H$}4AO_14O%Z#!h=JhdfBo z@;TNo=!zM#Rn%~NDpFYQMY~5oqvhG{wU(mk4LhRa>&CJvK{@RG17JJ;V~+O5*Q^lX z1k&$XHG_%(V4|iIKBx>#$N*3s01ycPs1a2qP$yv008lFc80!FF)c~+K0AO(dprZj` zd1zZ7pl*Q5tPW5^0JT0q9RW6KfNBD4)vzPJ9qov3SO+jnpxw|`N|vBK#sn(s0MwMo zaPDxvn%9}BTl{hlkviRxq&X8bCU7IrSPAO%79_4E!Dp5Nr2P|)_Qy9_yAWf6 zfc&Xmg$A!g^YJEZn7qymepP^vJHj*~6^N%ZYm{8GjA)zgTWFniS_E#wyk=`VZsLiwLD zja1I2K|$ys03`us0S63~Dh>_PfdpadARm+i2l-)e0Vu;IqDqD^1+4?pK&J8FL)m}K zQ3a`lqjLQloXO6a$oa``^Gc72W%-iu)e69B#e6&F*cH7-dl6AENU9i|7 z-rv=M3uemjvT-43v8*aOHT5?%{r7p|`M-Edg1pSd1@XSPZ^3>1(s&d8Ee7Mz(HW>v zHyro*>#cZ>do~w-YmKDyrBZQD^@&{i!kXA}avvm%vBa}lnBWdO5guVQ1KXW_fg45+ zKvU|FhZyc<;93F?ap3#GT<)H9Ts9{Z7q@tbpIX^S+I7{z%UZaL)87Q)@M~ST5IzG- zmS>|qlMK*JhX-hvPbT6=+{I?6bI`A;>yfMhhqgF7pqX28XlMMS-ws;+ustpT1y@U+ zi`LW+Qdk^5BfKu@&7L~z&R2Jrd*|EfGDC9|ib0+2k*?jDf{l1Ev*0Ah$8}nYTHD-| zXAZXzWjowMmiMbEqiuG{C!RDDwUw@7MV&9Ot%Kuu;Syi|LxabPg(r`*HxBwJa!Ob5 z#@Y9ov<+98Issew(wJSSf444*spk&y>z^=+8RPxf%~|@2F(bCK+i^M1*3DoSHZD*$ zwcpIo?evPRQ!-hRGQ?0B*i#=#=O0rT3=j)l!cMYP;(9ZR{#*GGyJw3Yv`=Ff3`pTy zmW^WTbH<94+)d$b2X|jeIW+8*!~v~Pz1W}lwF$n-SYbLPdH&ApnG!1<8_~ zr3yP`co=)3RE9nbOhP5U@5PNxUa+e--@x{7UtxoePqoYbOqurT*VuHx1af_~P6kw# zk^t?=9tZ${5{ZgznojV*r~#Bs8xt7-1{(kvDd7KO*)SaFkkDq;mLwiPS#)J<_;w@@ zz+Mdp;sb!q0-V%vCO*JL4PdzdS2f&-4{%ol*zbSx5;z$C(0|Zp&{xnuMg-6o(0;Hi zRE!*k7m>eMvnO#j1hxdK7r?3WIjVg)=Wp)nhkVqLyc%{+WWrECt=$vZhrpLWO3Qy45&3#(l(&6q^C@P&kR2)?v-FDF>UHBev+gw zRwlp3D}7^7{<`tpjB)$KwY=8hHmPqV#{RFwH#c6wS9MR~&S9;^`5&&MoX{*hu5Tun z*uDljkvNg-o<5zEyQQIZ5fG#SY>c`!l%Ntp?x$(Ra`^uRPpi?dlTZviHvN zo+I`6gUj65x$J7ioC%|NmsugK%=D~0KkNzH!MPQi``ZR(E4?9XnUR!#;c}KSbU(_M z6=d>83SM+2zdCD^5v1sBdbi-gq7r3Yk63<{^gO#XexJhHt&|;Pna9t(CS>IAYN6iF z;fizZgnZxEcjdF*EaQ8=&18;e?&AlXUa5FeAT4-i_&aNGq!};UaGcfea+~j%dxGhf zdWNmhcA`8hCrPxzaXy=SBb)D2Y{hKbevQdV8lteD*_1h-G>5N$=+oy){#AAdQ0(t> zR5`%L(0u;0avK9G00@|2Bk4oA_VY86=Jps;{n!lXM_Rs*i?gPy8ef$Po}C zcLL1{In6`jniF`?F!&_gO6_}U$A6DZ`=nBvz>7VV-JCrzLdxd0(nTAO??fea2az)n zu)CGjP_wyLu(eql?p=K>DqK5|UCT5>^$KLT&6Qv@ZcJCy`w@$tAF#&p4d<}ec4nf2 zfE;90Ocwt>ci2AjGIrQG1i2J$Lq57~K2HT(V~ifw8FyVf9wm58fG`R#0-E}_9PNd# zad@u@q+hj;4k`hFsSyC`0f4C%0DKex>H?rsD@@-2;HiMW%i$}ol!Mv#f&S z9JrNCweMcDpT7P!Qy*Qzi{Hlm&e;f5d*l6nZ_-HGynH>Zl2@D@q~;~#VYM5M1*kYV zs8-vHq*!}$@Vg%92$O@J zZk}YTEhp}Ab0e-2pN~&W(2;D+yvdnuG~||{Y;jfY1{QT^@PKs(C9Ad_<@$-=VDGb= z@u9~;F1fT6$F3U2UB2=R?_VfHll3p#dAefh1loK6O4sZCAsada?R79 z;$!dQ(1zqCXsYKCls+2eBux&S7u->ZT!imTC%~9=wlBdfkb$t>uWezgv!u zkBy?8@RRsqs~^S>QNf))XBAI+IqrGipsLb%Bkhk*xIe5wb;| zoB8|X3AJ#&EX5PqX8y{~{(Fs&cH?(-G2mzJS0Z=4-F%gz^?aJOEz>G|J6aJEBM+$R zq10b)&%99REA7%`tggcmw)>t%a@)9gK1i>zFjZGyc+7MQ8ko14HHh;Ii@BdG`r&3ml z*zAlZSo+&@Ty3E%np`Is8$W-F8#Ne)o>ea(XCq*z?jDXVZkvc!cwNC&?PJmM*;mkN z3m%DV=AinURzn^%EqWiCANUMpaA@m{9pJO=U&+8n7rHe=Zb|W2w?#4vegSiofW36# ziYl$k`XZHYeqkN>+72mSU-~Ru^h+#Sa6A_~#Uvvyz0TTc-y_rB`5N0q6Fy!JdR$N; z08DZK?FkbmIZ!PCG){XG{$92T>Ii_{ z(lty}9jDp7tq!*YU)zx~Dke(fNEl2RZvw*P1T2mJUY{z?H`yst?Xe&`sD0y$^DwE_ z#}lg_RA<`(WaqPbYQi@J z0+^a$F9u9acGkrz8hm^*Q60N-nU+ zb0De;iDY}?WVC3x6*en*hh7{h z$MdXOp|#tKuvz2fxJuGYGD%o*vCSo40Q3u6Z=IW@t=ITzt@= z*qErGkWt~WA%i~^(9z^Fj2;;k6c!m07pFNL9nCwwszKtQ@R0bCT9xTG*F5QC1nKc1 zv4g|oKb3B-X@_A<=+N-6_=r*AnzR4B;ijPx(IK&8ydz@4CPp-!;_ z!(+n-`b31qXU@%Z>6+=B_DgtdY)ous|4e5;^y`BZ>-;Vc{m_19_=*cgj2|lO=lo)S zMn6BaZ|qIKJo7{%WT>LFg)UI6D~b7dM<4<%uFNxsa(;-Xr9KI8Jd%hMv{HVM%K0Is zk*Yiq;B+3OXbbdwlp29%<{_(fzSe1IYX5X%s?#wi2PCZ{P1MdzV>C&c;-6w@o=*p) z=lrREzKl5z?gXhzBWOXA!BgqY^82`>&1p;?ou?U@losA+Cz2bqg6?y=qq{a1W&)iVJ! zw>W3Y&Ao7~iWe>1(MXlFq;2$};VVhZ^1If(;NlnO4GZ6`idk{KSA;6h8vBG1pbo_w zFW2#hx{9~HoE@h+;2LJCJ}5}>(lgaYz^N9mNnTk8np?a$yLvlSsU?O3>#7d8^4Bt> zkD2e=PFB^m@-26`p9_>)Waz#2B@s5AxJ#@QBJ8ppW;%ZX#EiW(R+VQ(%{x|N(&S?K zABTt)LWI1T<)NlfyIj)dovK~O2AK(a8j+YA6LUMNDt!`m;x3%eIo$($PAi~)EdJ<) zrV_0@*ZOHA2V+R`ewAlGg9%To5;8LE<25Kfp6XqS;{4D9!)XFfaR?C{?$7P(wS)+@ z985Ex)g?m14EwVS{vtw`KSG&XLJ(X&B(%CpgwFkK`^n0P&`Vr&NOq41eZso-#s)-) zITrbTuoV%atIpn@cZvvuO!Ur(rV=5ve4)f*7ZLhD+56#cIuRoLxs=<5L>MxP4G=A)-kLGM2GClevMz^&y}IE`_` zhGn&AKm^&ufOR3z!1(9fqYXWYka)N}aUQf{^vDwrFGHTFf#)_zhZ14PnzND8cp`+Z vujamGArS_Aczg5^w7tj7Mz^XO5~0JPt0v*wh~R6v@@}i=;UnWlhBo;>nP`G5 literal 0 HcmV?d00001 From f535c269bc687c6b37f8253c6b567e0c5a36d5fb Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 12:04:05 -0700 Subject: [PATCH 05/44] Verify data quality WIP --- .../quickstart_model_validation.ipynb | 126 +++++++++++++++++- 1 file changed, 122 insertions(+), 4 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 8bf6bf197..dea74a389 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -12,9 +12,8 @@ "To validate a model with the ValidMind Library, we'll:\n", "\n", "1. Independently verify data quality tests performed on datasets by model development\n", - "2. Split the datasets and initialize them for use with ValidMind\n", - "3. Initialize a model object for use with testing\n", - "4. Run model evaluation tests with the ValidMind Library, which will send the results of those tests to the ValidMind Platform" + "2. Import a champion model for evaluation\n", + "3. Run model evaluation tests with the ValidMind Library, which will send the results of those tests to the ValidMind Platform" ] }, { @@ -326,7 +325,7 @@ "source": [ "### Load the sample dataset\n", "\n", - "Let's first import the public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle, which was used to develop the dummy champion model.\n", + "First, let's import the public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle, which was used to develop the dummy champion model.\n", "\n", "We'll use this dataset to review steps that should have been conducted during the initial development and documentation of the model to ensure that the model was built correctly. By independently performing steps taken by the model development team, we can confirm whether the model was built using appropriate and properly processed data.\n", "\n", @@ -427,6 +426,125 @@ "### Run data quality tests" ] }, + { + "cell_type": "markdown", + "id": "09b76a17", + "metadata": {}, + "source": [ + "#### Identify qualitative tests\n", + "\n", + "Next, let's isolate some qualitative tests to help us assess the quality of our datasets. \n", + "\n", + "Use the [`vm.tests.list_tests()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) in combination with [`vm.tests.list_tags()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tags) and [`vm.tests.list_tasks()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tasks) to find which prebuilt tests are relevant for data quality assessment:\n", + "\n", + "- **`tasks`** represent the kind of modeling task associated with a test. Here we'll focus on `classification` tasks.\n", + "- **`tags`** are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the `data_quality` tag." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0ef2e2b0", + "metadata": {}, + "outputs": [], + "source": [ + "# Get the list of available task types\n", + "sorted(vm.tests.list_tasks())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "035cd165", + "metadata": {}, + "outputs": [], + "source": [ + "# Get the list of available tags\n", + "sorted(vm.tests.list_tags())" + ] + }, + { + "cell_type": "markdown", + "id": "43b90c02", + "metadata": {}, + "source": [ + "You can pass `tags` and `tasks` as parameters to the `vm.tests.list_tests()` function to filter the tests based on the tags and task types.\n", + "\n", + "For example, to find tests related to tabular data quality for classification models, you can call `list_tests()` like this:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9394b58e", + "metadata": {}, + "outputs": [], + "source": [ + "vm.tests.list_tests(task=\"classification\", tags=[\"tabular_data\", \"data_quality\"])" + ] + }, + { + "cell_type": "markdown", + "id": "4163b766", + "metadata": {}, + "source": [ + "#### Initialize the ValidMind datasets\n", + "\n", + "Before you can run tests with your preprocessed datasets, you must first initialize a ValidMind `Dataset` object using the [`init_dataset`](https://docs.validmind.ai/validmind/validmind.html#init_dataset) function from the ValidMind (`vm`) module. **This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind,** but you only need to do it once per dataset.\n", + "\n", + "For this example, we'll pass in the following arguments:\n", + "\n", + "- **`dataset`** — The raw dataset that you want to provide as input to tests.\n", + "- **`input_id`** — A unique identifier that allows tracking what inputs are used when running each individual test.\n", + "- **`target_column`** — A required argument if tests require access to true values. This is the name of the target column in the dataset.\n", + "- **`class_labels`** — An optional value to map predicted classes to class labels." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "003f18f7", + "metadata": {}, + "outputs": [], + "source": [ + "# Initialize the raw dataset\n", + "vm_raw_dataset = vm.init_dataset(\n", + " dataset=raw_df,\n", + " input_id=\"raw_dataset\",\n", + " target_column=customer_churn.target_column,\n", + " class_labels=customer_churn.class_labels,\n", + ")\n", + "\n", + "# Initialize the training dataset\n", + "vm_train_ds = vm.init_dataset(\n", + " dataset=train_df,\n", + " input_id=\"train_dataset\",\n", + " target_column=customer_churn.target_column,\n", + ")\n", + "\n", + "# Initialize the testing dataset\n", + "vm_test_ds = vm.init_dataset(\n", + " dataset=test_df,\n", + " input_id=\"test_dataset\",\n", + " target_column=customer_churn.target_column\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "a28b5c30", + "metadata": {}, + "source": [ + "#### Run tabular data tests\n", + "\n", + "Now that we know how to initialize ValidMind `dataset` objects, we're ready to run some tests!\n", + "\n", + "You run individual tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module. For the examples below, we'll pass in the following arguments:\n", + "\n", + "- **`test_id`** — The ID of the test to run, as seen in the `ID` column when you run `list_tests`. \n", + "- **`params`** — A dictionary of parameters for the test. These will override any `default_params` set in the test definition. " + ] + }, { "cell_type": "markdown", "id": "d51a436e", From 06923fdbebfad19d6f4d50ea9e33e98759b43d2c Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 12:28:29 -0700 Subject: [PATCH 06/44] Save point --- .../quickstart_model_validation.ipynb | 61 ++++++++++--------- 1 file changed, 32 insertions(+), 29 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index dea74a389..2b482dd72 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -423,7 +423,9 @@ "id": "b5ed7410", "metadata": {}, "source": [ - "### Run data quality tests" + "### Run data quality tests\n", + "\n", + "With everything ready to go, let's explore some of ValidMind's available tests to help us assess the quality of our datasets. Using ValidMind’s repository of tests streamlines your validation testing, and helps you ensure that your models are being validated appropriately." ] }, { @@ -433,9 +435,7 @@ "source": [ "#### Identify qualitative tests\n", "\n", - "Next, let's isolate some qualitative tests to help us assess the quality of our datasets. \n", - "\n", - "Use the [`vm.tests.list_tests()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) in combination with [`vm.tests.list_tags()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tags) and [`vm.tests.list_tasks()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tasks) to find which prebuilt tests are relevant for data quality assessment:\n", + "We want to narrow down the tests we want to run from the selection provided by ValidMind, so we'll use the [`vm.tests.list_tasks_and_tags()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tasks_and_tags) to list which `tags` are associated with each `task` type:\n", "\n", "- **`tasks`** represent the kind of modeling task associated with a test. Here we'll focus on `classification` tasks.\n", "- **`tags`** are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the `data_quality` tag." @@ -444,43 +444,31 @@ { "cell_type": "code", "execution_count": null, - "id": "0ef2e2b0", - "metadata": {}, - "outputs": [], - "source": [ - "# Get the list of available task types\n", - "sorted(vm.tests.list_tasks())" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "035cd165", + "id": "85bc2f85", "metadata": {}, "outputs": [], "source": [ - "# Get the list of available tags\n", - "sorted(vm.tests.list_tags())" + "vm.tests.list_tasks_and_tags()" ] }, { "cell_type": "markdown", - "id": "43b90c02", + "id": "39f050b1", "metadata": {}, "source": [ - "You can pass `tags` and `tasks` as parameters to the `vm.tests.list_tests()` function to filter the tests based on the tags and task types.\n", - "\n", - "For example, to find tests related to tabular data quality for classification models, you can call `list_tests()` like this:" + "Then we'll call [the `vm.tests.list_tests()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) to list all the data quality tests for classification:" ] }, { "cell_type": "code", "execution_count": null, - "id": "9394b58e", + "id": "c626b19d", "metadata": {}, "outputs": [], "source": [ - "vm.tests.list_tests(task=\"classification\", tags=[\"tabular_data\", \"data_quality\"])" + "vm.tests.list_tests(\n", + " tags=[\"data_quality\"], task=\"classification\"\n", + ")" ] }, { @@ -535,14 +523,29 @@ "id": "a28b5c30", "metadata": {}, "source": [ - "#### Run tabular data tests\n", + "#### Run and log an individual data quality test\n", "\n", - "Now that we know how to initialize ValidMind `dataset` objects, we're ready to run some tests!\n", + "Next, we'll use our previously initialized raw dataset (`vm_raw_dataset`) as input to run an individual test, then log the result to the ValidMind Platform.\n", "\n", - "You run individual tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module. For the examples below, we'll pass in the following arguments:\n", + "- You run validation tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module.\n", + "- Every test result returned by the `run_test()` function has a [`.log()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) that can be used to send the test results to the ValidMind Platform.\n", "\n", - "- **`test_id`** — The ID of the test to run, as seen in the `ID` column when you run `list_tests`. \n", - "- **`params`** — A dictionary of parameters for the test. These will override any `default_params` set in the test definition. " + "Here, we'll use the [`HighPearsonCorrelation` test](https://docs.validmind.ai/tests/data_validation/HighPearsonCorrelation.html) as an example:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dcb9b017", + "metadata": {}, + "outputs": [], + "source": [ + "vm.tests.run_test(\n", + " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n", + " inputs={\n", + " \"dataset\": vm_raw_dataset\n", + " }\n", + ").log()" ] }, { From a6438b00b6d289297e467d677f6611f17f5629a3 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 12:40:48 -0700 Subject: [PATCH 07/44] Save point --- notebooks/quickstart/quickstart_model_validation.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 2b482dd72..036dd0b13 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -525,7 +525,7 @@ "source": [ "#### Run and log an individual data quality test\n", "\n", - "Next, we'll use our previously initialized raw dataset (`vm_raw_dataset`) as input to run an individual test, then log the result to the ValidMind Platform.\n", + "Next, we'll use our previously initialized training dataset (`vm_train_ds`) as input to run an individual test, then log the result to the ValidMind Platform.\n", "\n", "- You run validation tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module.\n", "- Every test result returned by the `run_test()` function has a [`.log()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) that can be used to send the test results to the ValidMind Platform.\n", @@ -543,7 +543,7 @@ "vm.tests.run_test(\n", " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n", " inputs={\n", - " \"dataset\": vm_raw_dataset\n", + " \"dataset\": vm_train_ds\n", " }\n", ").log()" ] From 248217853f1560478411a4f8c9f6db54a70fe364 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 12:42:34 -0700 Subject: [PATCH 08/44] Save point --- notebooks/quickstart/quickstart_model_validation.ipynb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 036dd0b13..2a57e3261 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -115,7 +115,7 @@ "\n", "In a usual model lifecycle, a champion model will have been independently registered in your model inventory and submitted to you for validation by your model development team as part of the effective challenge process. (**Learn more:** [Submit for approval](https://docs.validmind.ai/guide/model-documentation/submit-for-approval.html))\n", "\n", - "For this series of notebooks, we'll have you register a dummy model in the ValidMind Platform inventory and assign yourself as the validator to familiarize you with the ValidMind interface and circumvent the need for an existing model:\n", + "For this notebook, we'll have you register a dummy model in the ValidMind Platform inventory and assign yourself as the validator to familiarize you with the ValidMind interface and circumvent the need for an existing model:\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", @@ -525,7 +525,7 @@ "source": [ "#### Run and log an individual data quality test\n", "\n", - "Next, we'll use our previously initialized training dataset (`vm_train_ds`) as input to run an individual test, then log the result to the ValidMind Platform.\n", + "Next, we'll use our previously initialized raw dataset (`vm_raw_dataset`) as input to run an individual test, then log the result to the ValidMind Platform.\n", "\n", "- You run validation tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module.\n", "- Every test result returned by the `run_test()` function has a [`.log()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) that can be used to send the test results to the ValidMind Platform.\n", @@ -543,7 +543,7 @@ "vm.tests.run_test(\n", " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n", " inputs={\n", - " \"dataset\": vm_train_ds\n", + " \"dataset\": vm_raw_dataset\n", " }\n", ").log()" ] From 9863af1f493e6bc4d00389e747fe6f5737e1327d Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 12:58:46 -0700 Subject: [PATCH 09/44] Edit --- .../tutorials/model_validation/2-start_validation_process.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/tutorials/model_validation/2-start_validation_process.ipynb b/notebooks/tutorials/model_validation/2-start_validation_process.ipynb index 93812fc21..747a5dd1e 100644 --- a/notebooks/tutorials/model_validation/2-start_validation_process.ipynb +++ b/notebooks/tutorials/model_validation/2-start_validation_process.ipynb @@ -564,7 +564,7 @@ "\n", "## Documenting test results\n", "\n", - "Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it. As we learned above, every test result returned by the `run_test()` function has a `.log()` method that can be used to send the test results to the ValidMind Platform.\n", + "Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it. Every test result returned by the `run_test()` function has a `.log()` method that can be used to send the test results to the ValidMind Platform.\n", "\n", "When logging validation test results to the platform, you'll need to manually add those results to the desired section of the validation report. To demonstrate how to add test results to your validation report, we'll log our data quality tests and insert the results via the ValidMind Platform." ] From d81a04978d88e6334ec7df8cb66461b3b805a193 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 13:00:01 -0700 Subject: [PATCH 10/44] Editing test example --- notebooks/quickstart/quickstart_model_validation.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 2a57e3261..ecb5ff885 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -530,7 +530,7 @@ "- You run validation tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module.\n", "- Every test result returned by the `run_test()` function has a [`.log()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) that can be used to send the test results to the ValidMind Platform.\n", "\n", - "Here, we'll use the [`HighPearsonCorrelation` test](https://docs.validmind.ai/tests/data_validation/HighPearsonCorrelation.html) as an example:" + "Here, we'll use the [`ClassImbalance` test](https://docs.validmind.ai/tests/data_validation/ClassImbalance.html) as an example:" ] }, { @@ -541,7 +541,7 @@ "outputs": [], "source": [ "vm.tests.run_test(\n", - " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n", + " test_id=\"validmind.data_validation.ClassImbalance\",\n", " inputs={\n", " \"dataset\": vm_raw_dataset\n", " }\n", From 8214acea7b773214a97e79026befc206a526f59f Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 15:50:36 -0700 Subject: [PATCH 11/44] Data comparison tests --- .../quickstart_model_validation.ipynb | 61 ++++++++++++++++++- 1 file changed, 60 insertions(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index ecb5ff885..e290a05f3 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -510,6 +510,13 @@ " target_column=customer_churn.target_column,\n", ")\n", "\n", + "# Initialize the validation dataset\n", + "vm_validation_ds = vm.init_dataset(\n", + " dataset=validation_df,\n", + " input_id=\"validation_dataset\",\n", + " target_column=customer_churn.target_column,\n", + ")\n", + "\n", "# Initialize the testing dataset\n", "vm_test_ds = vm.init_dataset(\n", " dataset=test_df,\n", @@ -523,7 +530,7 @@ "id": "a28b5c30", "metadata": {}, "source": [ - "#### Run and log an individual data quality test\n", + "#### Run an individual data quality test\n", "\n", "Next, we'll use our previously initialized raw dataset (`vm_raw_dataset`) as input to run an individual test, then log the result to the ValidMind Platform.\n", "\n", @@ -548,6 +555,58 @@ ").log()" ] }, + { + "cell_type": "markdown", + "id": "95d37a54", + "metadata": {}, + "source": [ + "#### Run data comparison tests\n", + "\n", + "We can also use ValidMind to perform comparison tests between our datasets, again logging the results to the ValidMind Platform.\n", + "\n", + "Below, we'll perform two sets of comparison tests with a mix of our datasets and the same class imbalance test:\n", + "\n", + "1. **Training (`vm_train_ds`) vs. validation (`vm_validation_ds`) datasets** — These two sets should have a similar mix of classes; we want to check if the class balance is the same in both sets.\n", + "2. **Training (`vm_train_ds`) vs. testing (`vm_test_ds`) datasets** — These two sets should have similar class distributions; if the test set has a very different mix our model won't be fairly tested.\n", + "\n", + "When running individual tests, **you can use a custom `result_id` to tag the individual result with a unique identifier:**\n", + "\n", + "- This `result_id` can be appended to `test_id` with a `:` separator.\n", + "- The `train_vs_validation` result identifier will correspond to the training vs. validation comparison test, and the `train_vs_test` result identifier will correspond to the training vs. testing comparison test." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3b7e94b5", + "metadata": {}, + "outputs": [], + "source": [ + "# Comparison between training and validation datasets\n", + "vm.tests.run_test(\n", + " test_id=\"validmind.data_validation.ClassImbalance:train_vs_validation\",\n", + " inputs={\n", + " \"dataset\": [vm_train_ds,vm_validation_ds]\n", + " }\n", + ").log()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "66fab745", + "metadata": {}, + "outputs": [], + "source": [ + "# Comparison between training and testing datasets\n", + "vm.tests.run_test(\n", + " test_id=\"validmind.data_validation.ClassImbalance:train_vs_test\",\n", + " inputs={\n", + " \"dataset\": [vm_train_ds,vm_test_ds]\n", + " }\n", + ").log()" + ] + }, { "cell_type": "markdown", "id": "d51a436e", From 9eda2cb359f9927edca2edcac2bc910902c1e1b0 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 16:01:49 -0700 Subject: [PATCH 12/44] Data comparison tests edit --- .../quickstart_model_validation.ipynb | 68 ++++++++++++------- 1 file changed, 42 insertions(+), 26 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index e290a05f3..65bc69ff1 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -562,49 +562,65 @@ "source": [ "#### Run data comparison tests\n", "\n", - "We can also use ValidMind to perform comparison tests between our datasets, again logging the results to the ValidMind Platform.\n", + "We can also use ValidMind to perform comparison tests between our datasets, again logging the results to the ValidMind Platform. Below, we'll perform two sets of comparison tests with a mix of our datasets and the same class imbalance test:\n", "\n", - "Below, we'll perform two sets of comparison tests with a mix of our datasets and the same class imbalance test:\n", - "\n", - "1. **Training (`vm_train_ds`) vs. validation (`vm_validation_ds`) datasets** — These two sets should have a similar mix of classes; we want to check if the class balance is the same in both sets.\n", - "2. **Training (`vm_train_ds`) vs. testing (`vm_test_ds`) datasets** — These two sets should have similar class distributions; if the test set has a very different mix our model won't be fairly tested.\n", - "\n", - "When running individual tests, **you can use a custom `result_id` to tag the individual result with a unique identifier:**\n", - "\n", - "- This `result_id` can be appended to `test_id` with a `:` separator.\n", - "- The `train_vs_validation` result identifier will correspond to the training vs. validation comparison test, and the `train_vs_test` result identifier will correspond to the training vs. testing comparison test." + "- When running individual tests, you can use a custom **`result_id`** to tag the individual result with a unique identifier, appended to the `test_id` with a `:` separator.\n", + "- We can specify all the tests we'd ike to run in a dictionary called `test_config`, and we'll pass in an **`input_grid`** of individual test inputs to compare. In this case, we'll input our two datasets for comparison. Note here that the `input_grid` expects the `input_id` of the dataset as the value rather than the variable name we specified." ] }, { "cell_type": "code", "execution_count": null, - "id": "3b7e94b5", + "id": "d53edde7", "metadata": {}, "outputs": [], "source": [ - "# Comparison between training and validation datasets\n", - "vm.tests.run_test(\n", - " test_id=\"validmind.data_validation.ClassImbalance:train_vs_validation\",\n", - " inputs={\n", - " \"dataset\": [vm_train_ds,vm_validation_ds]\n", - " }\n", - ").log()" + "# Individual test config with inputs specified\n", + "test_config = {\n", + " # Comparison between training and testing datasets to check if class balance is the same in both sets\n", + " \"validmind.data_validation.ClassImbalance:train_vs_validation\": {\n", + " \"input_grid\": {\"dataset\": [\"train_dataset\", \"validation_dataset\"]}\n", + " },\n", + " # Comparison between training and testing datasets to confirm that both sets have similar class distributions\n", + " \"validmind.data_validation.ClassImbalance:train_vs_test\": {\n", + " \"input_grid\": {\"dataset\": [\"train_dataset\", \"test_dataset\"]},\n", + " },\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "16132a57", + "metadata": {}, + "source": [ + "Then batch run and log our tests in `test_config`:" ] }, { "cell_type": "code", "execution_count": null, - "id": "66fab745", + "id": "3b7e94b5", "metadata": {}, "outputs": [], "source": [ - "# Comparison between training and testing datasets\n", - "vm.tests.run_test(\n", - " test_id=\"validmind.data_validation.ClassImbalance:train_vs_test\",\n", - " inputs={\n", - " \"dataset\": [vm_train_ds,vm_test_ds]\n", - " }\n", - ").log()" + "for t in test_config:\n", + " print(t)\n", + " try:\n", + " # Check if test has input_grid\n", + " if 'input_grid' in test_config[t]:\n", + " # For tests with input_grid, pass the input_grid configuration\n", + " if 'params' in test_config[t]:\n", + " vm.tests.run_test(t, input_grid=test_config[t]['input_grid'], params=test_config[t]['params']).log()\n", + " else:\n", + " vm.tests.run_test(t, input_grid=test_config[t]['input_grid']).log()\n", + " else:\n", + " # Original logic for regular inputs\n", + " if 'params' in test_config[t]:\n", + " vm.tests.run_test(t, inputs=test_config[t]['inputs'], params=test_config[t]['params']).log()\n", + " else:\n", + " vm.tests.run_test(t, inputs=test_config[t]['inputs']).log()\n", + " except Exception as e:\n", + " print(f\"Error running test {t}: {str(e)}\")" ] }, { From b75ae8b186a18f44c4c4e1312d3275ccfaeee84f Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 16:10:02 -0700 Subject: [PATCH 13/44] Import champion wip --- .../quickstart_model_documentation.ipynb | 2 +- .../quickstart_model_validation.ipynb | 59 ++++++++++++++++++- 2 files changed, 58 insertions(+), 3 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_documentation.ipynb b/notebooks/quickstart/quickstart_model_documentation.ipynb index c42413519..dc3f375be 100644 --- a/notebooks/quickstart/quickstart_model_documentation.ipynb +++ b/notebooks/quickstart/quickstart_model_documentation.ipynb @@ -617,7 +617,7 @@ "\n", "### Assign predictions\n", "\n", - "Once the model has been registered you can assign model predictions to the training and testing datasets.\n", + "Once the model has been registered, you can assign model predictions to the training and testing datasets.\n", "\n", "- The [`assign_predictions()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#assign_predictions) from the `Dataset` object can link existing predictions to any number of models.\n", "- This method links the model's class prediction values and probabilities to our `vm_train_ds` and `vm_test_ds` datasets.\n", diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 65bc69ff1..d78349cf2 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -633,12 +633,44 @@ "With our raw dataset preprocessed, let's go ahead and import the champion model submitted by the model development team in the format of a `.pkl` file: **[xgboost_model_champion.pkl](xgboost_model_champion.pkl)**" ] }, + { + "cell_type": "code", + "execution_count": null, + "id": "e0457832", + "metadata": {}, + "outputs": [], + "source": [ + "# Import the champion model\n", + "import pickle as pkl\n", + "\n", + "with open(\"xgboost_model_champion.pkl\", \"rb\") as f:\n", + " xgboost = pkl.load(f)" + ] + }, { "cell_type": "markdown", "id": "1d1aa3d6", "metadata": {}, "source": [ - "### Initialize a model object" + "### Initialize a model object\n", + "\n", + "In addition to the initialized datasets, you'll also need to initialize a ValidMind model object (`vm_model`) that can be passed to other functions for analysis and tests on the data for our champion model.\n", + "\n", + "You simply initialize this model object with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f53fb3d3", + "metadata": {}, + "outputs": [], + "source": [ + "# Initialize the champion XGBoost model\n", + "vm_xgboost = vm.init_model(\n", + " xgboost,\n", + " input_id=\"xgboost_champion\",\n", + ")" ] }, { @@ -646,7 +678,30 @@ "id": "983367c9", "metadata": {}, "source": [ - "### Assign predictions" + "### Assign predictions\n", + "\n", + "Once the model has been registered, you can assign model predictions to the training and testing datasets.\n", + "\n", + "- The [`assign_predictions()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#assign_predictions) from the `Dataset` object can link existing predictions to any number of models.\n", + "- This method links the model's class prediction values and probabilities to our `vm_train_ds` and `vm_test_ds` datasets.\n", + "\n", + "If no prediction values are passed, the method will compute predictions automatically:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ae94d53e", + "metadata": {}, + "outputs": [], + "source": [ + "vm_train_ds.assign_predictions(\n", + " model=vm_xgboost,\n", + ")\n", + "\n", + "vm_test_ds.assign_predictions(\n", + " model=vm_xgboost,\n", + ")" ] }, { From 8d0549a5e1a790b7b8f49145c94d841c107a9722 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 16:14:56 -0700 Subject: [PATCH 14/44] Modified validator credentials for app scorecard --- .../validate_application_scorecard.ipynb | 10 ++++++++-- notebooks/quickstart/quickstart_model_validation.ipynb | 7 ++++--- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb b/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb index 2ac83e369..3b1ea6b31 100644 --- a/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb +++ b/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb @@ -202,13 +202,19 @@ "\n", "In order to log tests as a validator instead of as a developer, on the model details page that appears after you've successfully registered your sample model:\n", "\n", - "1. Remove yourself as a developer: \n", + "1. Remove yourself as a model owner: \n", + "\n", + " - Click on the **OWNERS** tile.\n", + " - Click the **x** next to your name to remove yourself from that model's role.\n", + " - Click **Save** to apply your changes to that role.\n", + "\n", + "2. Remove yourself as a developer: \n", "\n", " - Click on the **DEVELOPERS** tile.\n", " - Click the **x** next to your name to remove yourself from that model's role.\n", " - Click **Save** to apply your changes to that role.\n", "\n", - "2. Add yourself as a validator: \n", + "3. Add yourself as a validator: \n", "\n", " - Click on the **VALIDATORS** tile.\n", " - Select your name from the drop-down menu.\n", diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index d78349cf2..51f4b6622 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -11,9 +11,10 @@ "\n", "To validate a model with the ValidMind Library, we'll:\n", "\n", - "1. Independently verify data quality tests performed on datasets by model development\n", - "2. Import a champion model for evaluation\n", - "3. Run model evaluation tests with the ValidMind Library, which will send the results of those tests to the ValidMind Platform" + "1. Import a sample dataset and preprocess it, then split the datasets and initialize them for use with ValidMind\n", + "2. Independently verify data quality tests performed on datasets by model development\n", + "3. Import a champion model for evaluation\n", + "4. Run model evaluation tests with the ValidMind Library, which will send the results of those tests to the ValidMind Platform" ] }, { From 388ae2038a60b3bad8b40eec55995bff0af9b16e Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 16:21:48 -0700 Subject: [PATCH 15/44] Performance tests WIP --- .../quickstart_model_validation.ipynb | 88 +++++++++++++++++-- 1 file changed, 79 insertions(+), 9 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 51f4b6622..a2ac33af6 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -316,7 +316,7 @@ "id": "9ed985b3", "metadata": {}, "source": [ - "## Verifying data quality adjustments" + "## Importing the sample dataset" ] }, { @@ -421,10 +421,10 @@ }, { "cell_type": "markdown", - "id": "b5ed7410", + "id": "d43c9c28", "metadata": {}, "source": [ - "### Run data quality tests\n", + "## Running data quality tests\n", "\n", "With everything ready to go, let's explore some of ValidMind's available tests to help us assess the quality of our datasets. Using ValidMind’s repository of tests streamlines your validation testing, and helps you ensure that your models are being validated appropriately." ] @@ -434,7 +434,7 @@ "id": "09b76a17", "metadata": {}, "source": [ - "#### Identify qualitative tests\n", + "### Identify qualitative tests\n", "\n", "We want to narrow down the tests we want to run from the selection provided by ValidMind, so we'll use the [`vm.tests.list_tasks_and_tags()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tasks_and_tags) to list which `tags` are associated with each `task` type:\n", "\n", @@ -477,7 +477,7 @@ "id": "4163b766", "metadata": {}, "source": [ - "#### Initialize the ValidMind datasets\n", + "### Initialize the ValidMind datasets\n", "\n", "Before you can run tests with your preprocessed datasets, you must first initialize a ValidMind `Dataset` object using the [`init_dataset`](https://docs.validmind.ai/validmind/validmind.html#init_dataset) function from the ValidMind (`vm`) module. **This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind,** but you only need to do it once per dataset.\n", "\n", @@ -531,7 +531,7 @@ "id": "a28b5c30", "metadata": {}, "source": [ - "#### Run an individual data quality test\n", + "### Run an individual data quality test\n", "\n", "Next, we'll use our previously initialized raw dataset (`vm_raw_dataset`) as input to run an individual test, then log the result to the ValidMind Platform.\n", "\n", @@ -561,7 +561,7 @@ "id": "95d37a54", "metadata": {}, "source": [ - "#### Run data comparison tests\n", + "### Run data comparison tests\n", "\n", "We can also use ValidMind to perform comparison tests between our datasets, again logging the results to the ValidMind Platform. Below, we'll perform two sets of comparison tests with a mix of our datasets and the same class imbalance test:\n", "\n", @@ -710,7 +710,9 @@ "id": "701e38e7", "metadata": {}, "source": [ - "## Running model evaluation tests" + "## Running model evaluation tests\n", + "\n", + "With everything ready for us, let's run the rest of our validation tests. We'll focus on testing around model performance of the champion model going forward as we've already verified the data quality of the datasets used to train the champion model." ] }, { @@ -718,7 +720,75 @@ "id": "8ebc45fb", "metadata": {}, "source": [ - "### Run model performance tests" + "### Run model performance tests\n", + "\n", + "First, let's run some performance tests. Use [`vm.tests.list_tests()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) to identify all the model performance tests for classification:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "202792e8", + "metadata": {}, + "outputs": [], + "source": [ + "vm.tests.list_tests(tags=[\"model_performance\"], task=\"classification\")" + ] + }, + { + "cell_type": "markdown", + "id": "0b49a782", + "metadata": {}, + "source": [ + "We'll isolate the specific tests we want to run in `mpt`:\n", + "\n", + "- [`ClassifierPerformance`](https://docs.validmind.ai/tests/model_validation/sklearn/ClassifierPerformance.html)\n", + "- [`ConfusionMatrix`](https://docs.validmind.ai/tests/model_validation/sklearn/ConfusionMatrix.html)\n", + "- [`MinimumAccuracy`](https://docs.validmind.ai/tests/model_validation/sklearn/MinimumAccuracy.html)\n", + "- [`MinimumF1Score`](https://docs.validmind.ai/tests/model_validation/sklearn/MinimumF1Score.html)\n", + "- [`ROCCurve`](https://docs.validmind.ai/tests/model_validation/sklearn/ROCCurve.html)\n", + "\n", + "As we learned above, you can use a custom `result_id` to tag the individual result with a unique identifier by appending this `result_id` to the `test_id` with a `:` separator. We'll append an identifier for our champion model here:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9fc18843", + "metadata": {}, + "outputs": [], + "source": [ + "mpt = [\n", + " \"validmind.model_validation.sklearn.ClassifierPerformance:xgboost_champion\",\n", + " \"validmind.model_validation.sklearn.ConfusionMatrix:xgboost_champion\",\n", + " \"validmind.model_validation.sklearn.MinimumAccuracy:xgboost_champion\",\n", + " \"validmind.model_validation.sklearn.MinimumF1Score:xgboost_champion\",\n", + " \"validmind.model_validation.sklearn.ROCCurve:xgboost_champion\"\n", + "]" + ] + }, + { + "cell_type": "markdown", + "id": "78becfa1", + "metadata": {}, + "source": [ + "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "55d10d9c", + "metadata": {}, + "outputs": [], + "source": [ + "for test in mpt:\n", + " vm.tests.run_test(\n", + " test,\n", + " inputs={\n", + " \"dataset\": vm_test_ds, \"model\" : vm_log_model,\n", + " },\n", + " ).log()" ] }, { From 4e3ad3bd90559f8d0ad7bb00d1a9ee3906ccdd1b Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 16:25:58 -0700 Subject: [PATCH 16/44] Performance tests edit --- .../quickstart/quickstart_model_validation.ipynb | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index a2ac33af6..1c340d1ef 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -556,6 +556,16 @@ ").log()" ] }, + { + "cell_type": "markdown", + "id": "0cf47f2b", + "metadata": {}, + "source": [ + "
Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for some test IDs. \n", + "

\n", + "That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform. You'll continue to see this message throughout this notebook as we run and log more tests.
" + ] + }, { "cell_type": "markdown", "id": "95d37a54", @@ -786,7 +796,7 @@ " vm.tests.run_test(\n", " test,\n", " inputs={\n", - " \"dataset\": vm_test_ds, \"model\" : vm_log_model,\n", + " \"dataset\": vm_test_ds, \"model\" : vm_xgboost,\n", " },\n", " ).log()" ] From 83e47432c20a85b713c408ac460620ca4530304b Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 16:26:58 -0700 Subject: [PATCH 17/44] Performance tests edit2 --- notebooks/quickstart/quickstart_model_validation.ipynb | 4 ---- 1 file changed, 4 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 1c340d1ef..730eadbb6 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -754,8 +754,6 @@ "\n", "- [`ClassifierPerformance`](https://docs.validmind.ai/tests/model_validation/sklearn/ClassifierPerformance.html)\n", "- [`ConfusionMatrix`](https://docs.validmind.ai/tests/model_validation/sklearn/ConfusionMatrix.html)\n", - "- [`MinimumAccuracy`](https://docs.validmind.ai/tests/model_validation/sklearn/MinimumAccuracy.html)\n", - "- [`MinimumF1Score`](https://docs.validmind.ai/tests/model_validation/sklearn/MinimumF1Score.html)\n", "- [`ROCCurve`](https://docs.validmind.ai/tests/model_validation/sklearn/ROCCurve.html)\n", "\n", "As we learned above, you can use a custom `result_id` to tag the individual result with a unique identifier by appending this `result_id` to the `test_id` with a `:` separator. We'll append an identifier for our champion model here:" @@ -771,8 +769,6 @@ "mpt = [\n", " \"validmind.model_validation.sklearn.ClassifierPerformance:xgboost_champion\",\n", " \"validmind.model_validation.sklearn.ConfusionMatrix:xgboost_champion\",\n", - " \"validmind.model_validation.sklearn.MinimumAccuracy:xgboost_champion\",\n", - " \"validmind.model_validation.sklearn.MinimumF1Score:xgboost_champion\",\n", " \"validmind.model_validation.sklearn.ROCCurve:xgboost_champion\"\n", "]" ] From f94f06d35af8e809c3b4d3f2adec1a734e0e18fd Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 16:32:42 -0700 Subject: [PATCH 18/44] Diagnostic test WIP --- .../quickstart_model_validation.ipynb | 66 ++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 730eadbb6..d8143044f 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -802,7 +802,71 @@ "id": "746860d8", "metadata": {}, "source": [ - "### Run diagnostic tests" + "### Run diagnostic tests\n", + "\n", + "Next we want to inspect the robustness and stability for our champion model. Use `list_tests()` to identify all the model diagnosis tests for classification:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c9b3caa4", + "metadata": {}, + "outputs": [], + "source": [ + "vm.tests.list_tests(tags=[\"model_diagnosis\"], task=\"classification\")" + ] + }, + { + "cell_type": "markdown", + "id": "ecefecac", + "metadata": {}, + "source": [ + "Let's see if the model suffers from any *overfit* potentials and also where there are potential sub-segments of issues with the [`OverfitDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/OverfitDiagnosis.html).\n", + "\n", + "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "82f824f2", + "metadata": {}, + "outputs": [], + "source": [ + "vm.tests.run_test(\n", + " test_id=\"validmind.model_validation.sklearn.OverfitDiagnosis:xgboost_champion\",\n", + " input_grid={\n", + " \"datasets\": [[vm_train_ds,vm_test_ds]],\n", + " \"model\" : vm_xgboost\n", + " }\n", + ").log()" + ] + }, + { + "cell_type": "markdown", + "id": "c7f7988e", + "metadata": {}, + "source": [ + "Let's also conduct *robustness* and *stability* testing with the [`RobustnessDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/RobustnessDiagnosis.html).\n", + "\n", + "Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8f2e53ee", + "metadata": {}, + "outputs": [], + "source": [ + "vm.tests.run_test(\n", + " test_id=\"validmind.model_validation.sklearn.RobustnessDiagnosis:xgboost_champion\",\n", + " input_grid={\n", + " \"datasets\": [[vm_train_ds,vm_test_ds]],\n", + " \"model\" : vm_xgboost\n", + " },\n", + ").log()" ] }, { From f6ab1bd49700a24175652cfa98231a3ce241ea9e Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 16:44:09 -0700 Subject: [PATCH 19/44] Feature importance tests WIP --- .../quickstart_model_validation.ipynb | 44 ++++++++++++++++--- 1 file changed, 39 insertions(+), 5 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index d8143044f..8857d287d 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -824,7 +824,10 @@ "source": [ "Let's see if the model suffers from any *overfit* potentials and also where there are potential sub-segments of issues with the [`OverfitDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/OverfitDiagnosis.html).\n", "\n", - "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data." + "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", + "\n", + "- Since the training dataset (`vm_train_ds`) was used to fit the model, we use this set to establish a baseline performance for how well the model performs on data it has already seen.\n", + "- The testing dataset (`vm_test_ds`) was never seen during training, and here simulates real-world generalization, or how well the model performs on new, unseen data. " ] }, { @@ -838,7 +841,7 @@ " test_id=\"validmind.model_validation.sklearn.OverfitDiagnosis:xgboost_champion\",\n", " input_grid={\n", " \"datasets\": [[vm_train_ds,vm_test_ds]],\n", - " \"model\" : vm_xgboost\n", + " \"model\" : [vm_xgboost]\n", " }\n", ").log()" ] @@ -850,7 +853,7 @@ "source": [ "Let's also conduct *robustness* and *stability* testing with the [`RobustnessDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/RobustnessDiagnosis.html).\n", "\n", - "Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets." + "Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets. Again, we'll use both the training and testing dataset to evaluate baseline performance and to simulate real-word generalization." ] }, { @@ -864,7 +867,7 @@ " test_id=\"validmind.model_validation.sklearn.RobustnessDiagnosis:xgboost_champion\",\n", " input_grid={\n", " \"datasets\": [[vm_train_ds,vm_test_ds]],\n", - " \"model\" : vm_xgboost\n", + " \"model\" : [vm_xgboost]\n", " },\n", ").log()" ] @@ -874,7 +877,38 @@ "id": "4e64ea12", "metadata": {}, "source": [ - "### Run feature importance tests" + "### Run feature importance tests\n", + "\n", + "We also want to verify the relative influence of different input features on our model's predictions. Use `list_tests()` to identify all the feature importance tests for classification:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9c8c26e6", + "metadata": {}, + "outputs": [], + "source": [ + "# Store the feature importance tests\n", + "FI = vm.tests.list_tests(tags=[\"feature_importance\"], task=\"classification\",pretty=False)\n", + "FI" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "054c0d83", + "metadata": {}, + "outputs": [], + "source": [ + "# Run and log our feature importance tests for both models for the testing dataset\n", + "for test in FI:\n", + " vm.tests.run_test(\n", + " \"\".join((test,':xgboost_champion')),\n", + " input_grid={\n", + " \"dataset\": [vm_test_ds], \"model\" : [vm_xgboost]\n", + " },\n", + " ).log()" ] }, { From 2868b51311db488cdf38b3a5f8a58bc6800af110 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 17:09:37 -0700 Subject: [PATCH 20/44] Editing... --- .../quickstart_model_validation.ipynb | 42 ++++++++++++------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 8857d287d..06cce1b7d 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -572,6 +572,7 @@ "metadata": {}, "source": [ "### Run data comparison tests\n", + "\n", "\n", "We can also use ValidMind to perform comparison tests between our datasets, again logging the results to the ValidMind Platform. Below, we'll perform two sets of comparison tests with a mix of our datasets and the same class imbalance test:\n", "\n", @@ -879,7 +880,7 @@ "source": [ "### Run feature importance tests\n", "\n", - "We also want to verify the relative influence of different input features on our model's predictions. Use `list_tests()` to identify all the feature importance tests for classification:" + "We also want to verify the relative influence of different input features on our model's predictions. Use `list_tests()` to identify all the feature importance tests for classification and store them in `FI`:" ] }, { @@ -894,6 +895,14 @@ "FI" ] }, + { + "cell_type": "markdown", + "id": "31326bc8", + "metadata": {}, + "source": [ + "We'll only use our testing dataset (`vm_test_ds`) here, to provide a realistic, unseen sample that mimic future or production data, as the training dataset has already influenced our model during learning:" + ] + }, { "cell_type": "code", "execution_count": null, @@ -901,7 +910,7 @@ "metadata": {}, "outputs": [], "source": [ - "# Run and log our feature importance tests for both models for the testing dataset\n", + "# Run and log our feature importance tests with the testing dataset\n", "for test in FI:\n", " vm.tests.run_test(\n", " \"\".join((test,':xgboost_champion')),\n", @@ -923,10 +932,19 @@ "- [x] Register a model within the ValidMind Platform\n", "- [x] Install and initialize the ValidMind Library\n", "- [x] Preview the validation report template for your model\n", - "- [x] Import a sample dataset*\n", - "- [x] Initialize ValidMind datasets and model objects*\n", - "- [x] Assign model predictions to your ValidMind model objects*\n", - "- [x] Run a full suite of documentation tests*" + "- [x] Import a sample dataset and champion model\n", + "- [x] Initialize ValidMind datasets and model objects\n", + "- [x] Assign model predictions to your ValidMind model objects\n", + "- [x] Identify and run various validation tests\n", + "\n", + "In a usual model validation workflow, you would wrap up your validation testing by verifying that all the tests provided by the model development team were run and reported accurately, and perhaps even propose a challenger model, comparing the performance of the challenger with the running champion.\n", + "\n", + "
With ValidMind, you can easily:\n", + "
    \n", + "
  • Specify all the tests you'd like to independently rerun, just like you did in the step Run data comparision tests
    • \n", + "
  • Evaluate the performance of a challenger model against the champion, just like you did in the steps under Running model evaluation tests
    • \n", + "
\n", + "
" ] }, { @@ -989,11 +1007,7 @@ "cell_type": "code", "execution_count": null, "id": "upgrade-show-c0a446ff-f26f-4ad0-839a-e92927711798", - "metadata": { - "vscode": { - "languageId": "plaintext" - } - }, + "metadata": {}, "outputs": [], "source": [ "%pip show validmind" @@ -1022,13 +1036,13 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "ValidMind Library", "language": "python", - "name": "python3" + "name": "validmind" }, "language_info": { "name": "python", - "version": "3.10" + "version": "3.10.13" } }, "nbformat": 4, From 15f12bfc641ed5995ca027e56ae06a0e412327e5 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 17:13:01 -0700 Subject: [PATCH 21/44] More editing --- notebooks/quickstart/quickstart_model_validation.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 06cce1b7d..1374a142c 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -941,8 +941,8 @@ "\n", "
With ValidMind, you can easily:\n", "
    \n", - "
  • Specify all the tests you'd like to independently rerun, just like you did in the step Run data comparision tests
    • \n", - "
  • Evaluate the performance of a challenger model against the champion, just like you did in the steps under Running model evaluation tests
    • \n", + "
  • Specify all the tests you'd like to independently rerun, just like you did in the step Run data comparision tests
    • \n", + "
  • Evaluate the performance of a challenger model against the champion, just like you did in the steps under Running model evaluation tests
    • \n", "
\n", "
" ] From b65ab59cbd0969b63cd786140203f472f234e943 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 21 May 2025 17:14:13 -0700 Subject: [PATCH 22/44] ToC --- .../quickstart_model_validation.ipynb | 222 +++++++++++++----- 1 file changed, 167 insertions(+), 55 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 1374a142c..cc3ff79dc 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "f75e73d7", + "id": "926aad7c", "metadata": {}, "source": [ "# Quickstart for model validation\n", @@ -19,9 +19,67 @@ }, { "cell_type": "markdown", - "id": "a83709ec", + "id": "dc56a98f", + "metadata": {}, + "source": [ + "::: {.content-hidden when-format=\"html\"}\n", + "## Contents \n", + "- [Introduction](#toc1_) \n", + "- [About ValidMind](#toc2_) \n", + " - [Before you begin](#toc2_1_) \n", + " - [New to ValidMind?](#toc2_2_) \n", + " - [Key concepts](#toc2_3_) \n", + "- [Setting up](#toc3_) \n", + " - [Register a sample model](#toc3_1_) \n", + " - [Assign validator credentials](#toc3_1_1_) \n", + " - [Install the ValidMind Library](#toc3_2_) \n", + " - [Initialize the ValidMind Library](#toc3_3_) \n", + " - [Get your code snippet](#toc3_3_1_) \n", + " - [Initialize the Python environment](#toc3_4_) \n", + "- [Getting to know ValidMind](#toc4_) \n", + " - [Preview the validation report template](#toc4_1_) \n", + " - [View validation report in the ValidMind Platform](#toc4_2_) \n", + "- [Importing the sample dataset](#toc5_) \n", + " - [Load the sample dataset](#toc5_1_) \n", + " - [Preprocess the raw dataset](#toc5_2_) \n", + " - [Split the dataset](#toc5_2_1_) \n", + " - [Separate features and targets](#toc5_2_2_) \n", + "- [Running data quality tests](#toc6_) \n", + " - [Identify qualitative tests](#toc6_1_) \n", + " - [Initialize the ValidMind datasets](#toc6_2_) \n", + " - [Run an individual data quality test](#toc6_3_) \n", + " - [Run data comparison tests](#toc6_4_) \n", + "- [Import the champion model](#toc7_) \n", + " - [Initialize a model object](#toc7_1_) \n", + " - [Assign predictions](#toc7_2_) \n", + "- [Running model evaluation tests](#toc8_) \n", + " - [Run model performance tests](#toc8_1_) \n", + " - [Run diagnostic tests](#toc8_2_) \n", + " - [Run feature importance tests](#toc8_3_) \n", + "- [In summary](#toc9_) \n", + "- [Next steps](#toc10_) \n", + " - [Work with your validation report](#toc10_1_) \n", + " - [Discover more learning resources](#toc10_2_) \n", + "- [Upgrade ValidMind](#toc11_) \n", + "\n", + ":::\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "e3f7cf22", "metadata": {}, "source": [ + "\n", + "\n", "## Introduction\n", "\n", "Model validation aims to independently assess the compliance of *champion models* created by model developers with regulatory guidance by conducting thorough testing and analysis, potentially including the use of challenger models to benchmark performance. Assessments, presented in the form of a validation report, typically include *model findings* and recommendations to address those issues.\n", @@ -34,9 +92,11 @@ }, { "cell_type": "markdown", - "id": "about-intro-747d2ae3-4cbf-4bf5-9e74-d84b4a3f6241", + "id": "e5065e3c", "metadata": {}, "source": [ + "\n", + "\n", "## About ValidMind\n", "\n", "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.\n", @@ -46,9 +106,11 @@ }, { "cell_type": "markdown", - "id": "about-begin-94cdfeb8-2a9a-470d-8d7b-788f58c2a2e1", + "id": "541fadfb", "metadata": {}, "source": [ + "\n", + "\n", "### Before you begin\n", "\n", "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", @@ -58,9 +120,11 @@ }, { "cell_type": "markdown", - "id": "about-signup-b3848272-3ee8-41b6-9bc2-5fabf68703af", + "id": "a196a66c", "metadata": {}, "source": [ + "\n", + "\n", "### New to ValidMind?\n", "\n", "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", @@ -72,9 +136,11 @@ }, { "cell_type": "markdown", - "id": "about-concepts-9cb03e6d-d057-4be3-8188-ce3705d56cb3", + "id": "29878854", "metadata": {}, "source": [ + "\n", + "\n", "### Key concepts\n", "\n", "**Validation report**: A comprehensive and structured assessment of a model’s development and performance, focusing on verifying its integrity, appropriateness, and alignment with its intended use. It includes analyses of model assumptions, data quality, performance metrics, outcomes of testing procedures, and risk considerations. The validation report supports transparency, regulatory compliance, and informed decision-making by documenting the validator’s independent review and conclusions.\n", @@ -101,17 +167,21 @@ }, { "cell_type": "markdown", - "id": "38f18a6b", + "id": "58b32cff", "metadata": {}, "source": [ + "\n", + "\n", "## Setting up" ] }, { "cell_type": "markdown", - "id": "f7625c26", + "id": "e9be1dd1", "metadata": {}, "source": [ + "\n", + "\n", "### Register a sample model\n", "\n", "In a usual model lifecycle, a champion model will have been independently registered in your model inventory and submitted to you for validation by your model development team as part of the effective challenge process. (**Learn more:** [Submit for approval](https://docs.validmind.ai/guide/model-documentation/submit-for-approval.html))\n", @@ -134,9 +204,11 @@ }, { "cell_type": "markdown", - "id": "2c93f237", + "id": "f7815e5f", "metadata": {}, "source": [ + "\n", + "\n", "#### Assign validator credentials\n", "\n", "In order to log tests as a validator instead of as a developer, on the model details page that appears after you've successfully registered your sample model:\n", @@ -162,9 +234,10 @@ }, { "cell_type": "markdown", - "id": "install-library-d030a2d2-38ae-4739-b2b9-9eaaabf9772f", "metadata": {}, "source": [ + "\n", + "\n", "### Install the ValidMind Library\n", "\n", "
Recommended Python versions\n", @@ -177,7 +250,7 @@ { "cell_type": "code", "execution_count": null, - "id": "install-python-0d4ab69a-3025-42dc-b356-b82d1e7875e4", + "id": "64eb485c", "metadata": {}, "outputs": [], "source": [ @@ -186,9 +259,11 @@ }, { "cell_type": "markdown", - "id": "install-initialize-63aa8322-784e-4db0-bad7-0cb9934c7385", + "id": "330c8f40", "metadata": {}, "source": [ + "\n", + "\n", "### Initialize the ValidMind Library\n", "\n", "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook." @@ -196,9 +271,10 @@ }, { "cell_type": "markdown", - "id": "install-snippet-5035301c-b262-4ade-9b98-3a1d6a466980", "metadata": {}, "source": [ + "\n", + "\n", "#### Get your code snippet\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", @@ -213,7 +289,7 @@ { "cell_type": "code", "execution_count": null, - "id": "install-init-c2877372-5824-4dfe-ae33-d5242ee6715b", + "id": "9c6ce354", "metadata": {}, "outputs": [], "source": [ @@ -236,9 +312,10 @@ }, { "cell_type": "markdown", - "id": "4cea824e", "metadata": {}, "source": [ + "\n", + "\n", "### Initialize the Python environment\n", "\n", "Then, let's import the necessary libraries and set up your Python environment for data analysis:\n", @@ -250,7 +327,7 @@ { "cell_type": "code", "execution_count": null, - "id": "de14ed5a", + "id": "1e53065d", "metadata": {}, "outputs": [], "source": [ @@ -261,17 +338,20 @@ }, { "cell_type": "markdown", - "id": "8f62809c", + "id": "ecc3d3a1", "metadata": {}, "source": [ + "\n", + "\n", "## Getting to know ValidMind" ] }, { "cell_type": "markdown", - "id": "393b94c8", "metadata": {}, "source": [ + "\n", + "\n", "### Preview the validation report template\n", "\n", "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for model validation. A template predefines sections for your validation report and provides a general outline to follow, making the validation process much easier.\n", @@ -282,7 +362,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6bddf4bb", + "id": "be445598", "metadata": {}, "outputs": [], "source": [ @@ -291,9 +371,11 @@ }, { "cell_type": "markdown", - "id": "ba022e64", + "id": "43966e46", "metadata": {}, "source": [ + "\n", + "\n", "### View validation report in the ValidMind Platform\n", "\n", "Next, let's head to the ValidMind Platform to see the template in action:\n", @@ -313,17 +395,20 @@ }, { "cell_type": "markdown", - "id": "9ed985b3", + "id": "5cd4890e", "metadata": {}, "source": [ + "\n", + "\n", "## Importing the sample dataset" ] }, { "cell_type": "markdown", - "id": "cf2da2be", "metadata": {}, "source": [ + "\n", + "\n", "### Load the sample dataset\n", "\n", "First, let's import the public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle, which was used to develop the dummy champion model.\n", @@ -339,7 +424,7 @@ { "cell_type": "code", "execution_count": null, - "id": "684abe24", + "id": "73076ee3", "metadata": {}, "outputs": [], "source": [ @@ -355,9 +440,11 @@ }, { "cell_type": "markdown", - "id": "6596e866", + "id": "6c1e567a", "metadata": {}, "source": [ + "\n", + "\n", "### Preprocess the raw dataset\n", "\n", "Let's say that thanks to the documentation submitted by the model development team ([Learn more ...](https://docs.validmind.ai/notebooks/quickstart_model_documentation.html)), we know that the sample dataset was first preprocessed before being used to train the champion model.\n", @@ -367,9 +454,10 @@ }, { "cell_type": "markdown", - "id": "6dacb398", "metadata": {}, "source": [ + "\n", + "\n", "#### Split the dataset\n", "\n", "Splitting our dataset helps assess how well the model generalizes to unseen data.\n", @@ -384,7 +472,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b34f9bfc", + "id": "ee8cfaaf", "metadata": {}, "outputs": [], "source": [ @@ -393,9 +481,10 @@ }, { "cell_type": "markdown", - "id": "b0d2fb15", "metadata": {}, "source": [ + "\n", + "\n", "#### Separate features and targets\n", "\n", "To train the model, we need to provide it with:\n", @@ -409,7 +498,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2d089b26", + "id": "6fe65be5", "metadata": {}, "outputs": [], "source": [ @@ -421,9 +510,11 @@ }, { "cell_type": "markdown", - "id": "d43c9c28", + "id": "a467c9ce", "metadata": {}, "source": [ + "\n", + "\n", "## Running data quality tests\n", "\n", "With everything ready to go, let's explore some of ValidMind's available tests to help us assess the quality of our datasets. Using ValidMind’s repository of tests streamlines your validation testing, and helps you ensure that your models are being validated appropriately." @@ -431,9 +522,10 @@ }, { "cell_type": "markdown", - "id": "09b76a17", "metadata": {}, "source": [ + "\n", + "\n", "### Identify qualitative tests\n", "\n", "We want to narrow down the tests we want to run from the selection provided by ValidMind, so we'll use the [`vm.tests.list_tasks_and_tags()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tasks_and_tags) to list which `tags` are associated with each `task` type:\n", @@ -463,7 +555,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c626b19d", + "id": "31b31a51", "metadata": {}, "outputs": [], "source": [ @@ -474,9 +566,10 @@ }, { "cell_type": "markdown", - "id": "4163b766", "metadata": {}, "source": [ + "\n", + "\n", "### Initialize the ValidMind datasets\n", "\n", "Before you can run tests with your preprocessed datasets, you must first initialize a ValidMind `Dataset` object using the [`init_dataset`](https://docs.validmind.ai/validmind/validmind.html#init_dataset) function from the ValidMind (`vm`) module. **This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind,** but you only need to do it once per dataset.\n", @@ -492,7 +585,7 @@ { "cell_type": "code", "execution_count": null, - "id": "003f18f7", + "id": "ba677dd7", "metadata": {}, "outputs": [], "source": [ @@ -528,9 +621,10 @@ }, { "cell_type": "markdown", - "id": "a28b5c30", "metadata": {}, "source": [ + "\n", + "\n", "### Run an individual data quality test\n", "\n", "Next, we'll use our previously initialized raw dataset (`vm_raw_dataset`) as input to run an individual test, then log the result to the ValidMind Platform.\n", @@ -558,7 +652,7 @@ }, { "cell_type": "markdown", - "id": "0cf47f2b", + "id": "cc4829be", "metadata": {}, "source": [ "
Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for some test IDs. \n", @@ -568,9 +662,10 @@ }, { "cell_type": "markdown", - "id": "95d37a54", "metadata": {}, "source": [ + "\n", + "\n", "### Run data comparison tests\n", "\n", "\n", @@ -611,7 +706,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3b7e94b5", + "id": "1b97e404", "metadata": {}, "outputs": [], "source": [ @@ -637,9 +732,10 @@ }, { "cell_type": "markdown", - "id": "d51a436e", "metadata": {}, "source": [ + "\n", + "\n", "## Import the champion model\n", "\n", "With our raw dataset preprocessed, let's go ahead and import the champion model submitted by the model development team in the format of a `.pkl` file: **[xgboost_model_champion.pkl](xgboost_model_champion.pkl)**" @@ -648,7 +744,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e0457832", + "id": "7f18188e", "metadata": {}, "outputs": [], "source": [ @@ -661,9 +757,10 @@ }, { "cell_type": "markdown", - "id": "1d1aa3d6", "metadata": {}, "source": [ + "\n", + "\n", "### Initialize a model object\n", "\n", "In addition to the initialized datasets, you'll also need to initialize a ValidMind model object (`vm_model`) that can be passed to other functions for analysis and tests on the data for our champion model.\n", @@ -674,7 +771,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f53fb3d3", + "id": "0a799cf2", "metadata": {}, "outputs": [], "source": [ @@ -687,9 +784,10 @@ }, { "cell_type": "markdown", - "id": "983367c9", "metadata": {}, "source": [ + "\n", + "\n", "### Assign predictions\n", "\n", "Once the model has been registered, you can assign model predictions to the training and testing datasets.\n", @@ -703,7 +801,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ae94d53e", + "id": "71dd8e7b", "metadata": {}, "outputs": [], "source": [ @@ -718,9 +816,11 @@ }, { "cell_type": "markdown", - "id": "701e38e7", + "id": "629964ef", "metadata": {}, "source": [ + "\n", + "\n", "## Running model evaluation tests\n", "\n", "With everything ready for us, let's run the rest of our validation tests. We'll focus on testing around model performance of the champion model going forward as we've already verified the data quality of the datasets used to train the champion model." @@ -728,9 +828,10 @@ }, { "cell_type": "markdown", - "id": "8ebc45fb", "metadata": {}, "source": [ + "\n", + "\n", "### Run model performance tests\n", "\n", "First, let's run some performance tests. Use [`vm.tests.list_tests()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) to identify all the model performance tests for classification:" @@ -785,7 +886,7 @@ { "cell_type": "code", "execution_count": null, - "id": "55d10d9c", + "id": "6866b21c", "metadata": {}, "outputs": [], "source": [ @@ -800,9 +901,10 @@ }, { "cell_type": "markdown", - "id": "746860d8", "metadata": {}, "source": [ + "\n", + "\n", "### Run diagnostic tests\n", "\n", "Next we want to inspect the robustness and stability for our champion model. Use `list_tests()` to identify all the model diagnosis tests for classification:" @@ -860,7 +962,7 @@ { "cell_type": "code", "execution_count": null, - "id": "8f2e53ee", + "id": "b2676197", "metadata": {}, "outputs": [], "source": [ @@ -875,9 +977,10 @@ }, { "cell_type": "markdown", - "id": "4e64ea12", "metadata": {}, "source": [ + "\n", + "\n", "### Run feature importance tests\n", "\n", "We also want to verify the relative influence of different input features on our model's predictions. Use `list_tests()` to identify all the feature importance tests for classification and store them in `FI`:" @@ -906,7 +1009,7 @@ { "cell_type": "code", "execution_count": null, - "id": "054c0d83", + "id": "5a49f550", "metadata": {}, "outputs": [], "source": [ @@ -922,9 +1025,11 @@ }, { "cell_type": "markdown", - "id": "cd977c5a", + "id": "19a0caa4", "metadata": {}, "source": [ + "\n", + "\n", "## In summary\n", "\n", "In this notebook, you learned how to:\n", @@ -949,9 +1054,11 @@ }, { "cell_type": "markdown", - "id": "next-steps-493d40db-f4d7-4516-8753-41c5e19eb2b9", + "id": "ed4bc468", "metadata": {}, "source": [ + "\n", + "\n", "## Next steps\n", "\n", "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your validation report." @@ -959,9 +1066,11 @@ }, { "cell_type": "markdown", - "id": "next-docs-4432e8f7-f07b-4f63-8387-57398be50a50", + "id": "3836cbc2", "metadata": {}, "source": [ + "\n", + "\n", "### Work with your validation report\n", "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", @@ -973,9 +1082,11 @@ }, { "cell_type": "markdown", - "id": "next-resources-e2951762-c812-4621-bae1-a077d21f04c7", + "id": "8bdf85fe", "metadata": {}, "source": [ + "\n", + "\n", "### Discover more learning resources\n", "\n", "For a more in-depth introduction to using the ValidMind Library for validation, check out our introductory validation series and the accompanying interactive training:\n", @@ -993,9 +1104,10 @@ }, { "cell_type": "markdown", - "id": "upgrade-vm-fecfaf5b-97dc-4b22-9949-42e7e850689a", "metadata": {}, "source": [ + "\n", + "\n", "## Upgrade ValidMind\n", "\n", "
After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.
\n", From 47a66ad4c3a96865b3201c501d778c37fa2e26b2 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 22 May 2025 08:55:02 -0700 Subject: [PATCH 23/44] Editing --- .../quickstart_model_validation.ipynb | 30 ++++++++++++++----- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index cc3ff79dc..c81ba8093 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -271,6 +271,7 @@ }, { "cell_type": "markdown", + "id": "3f04449a", "metadata": {}, "source": [ "\n", @@ -312,6 +313,7 @@ }, { "cell_type": "markdown", + "id": "b07758a3", "metadata": {}, "source": [ "\n", @@ -348,6 +350,7 @@ }, { "cell_type": "markdown", + "id": "ce709365", "metadata": {}, "source": [ "\n", @@ -405,6 +408,7 @@ }, { "cell_type": "markdown", + "id": "6aeb700e", "metadata": {}, "source": [ "\n", @@ -454,6 +458,7 @@ }, { "cell_type": "markdown", + "id": "4b98b04a", "metadata": {}, "source": [ "\n", @@ -481,6 +486,7 @@ }, { "cell_type": "markdown", + "id": "f6549607", "metadata": {}, "source": [ "\n", @@ -522,6 +528,7 @@ }, { "cell_type": "markdown", + "id": "da09a71d", "metadata": {}, "source": [ "\n", @@ -566,6 +573,7 @@ }, { "cell_type": "markdown", + "id": "d433c949", "metadata": {}, "source": [ "\n", @@ -621,6 +629,7 @@ }, { "cell_type": "markdown", + "id": "45ff3201", "metadata": {}, "source": [ "\n", @@ -662,6 +671,7 @@ }, { "cell_type": "markdown", + "id": "43411ece", "metadata": {}, "source": [ "\n", @@ -732,6 +742,7 @@ }, { "cell_type": "markdown", + "id": "bd8d4805", "metadata": {}, "source": [ "\n", @@ -757,6 +768,7 @@ }, { "cell_type": "markdown", + "id": "9f858689", "metadata": {}, "source": [ "\n", @@ -784,6 +796,7 @@ }, { "cell_type": "markdown", + "id": "20ef72d4", "metadata": {}, "source": [ "\n", @@ -828,6 +841,7 @@ }, { "cell_type": "markdown", + "id": "b3b40873", "metadata": {}, "source": [ "\n", @@ -852,13 +866,7 @@ "id": "0b49a782", "metadata": {}, "source": [ - "We'll isolate the specific tests we want to run in `mpt`:\n", - "\n", - "- [`ClassifierPerformance`](https://docs.validmind.ai/tests/model_validation/sklearn/ClassifierPerformance.html)\n", - "- [`ConfusionMatrix`](https://docs.validmind.ai/tests/model_validation/sklearn/ConfusionMatrix.html)\n", - "- [`ROCCurve`](https://docs.validmind.ai/tests/model_validation/sklearn/ROCCurve.html)\n", - "\n", - "As we learned above, you can use a custom `result_id` to tag the individual result with a unique identifier by appending this `result_id` to the `test_id` with a `:` separator. We'll append an identifier for our champion model here:" + "We'll isolate the specific tests we want to run in `mpt`, and append an identifier for our champion model here to the `result_id` with a `:` separator like we did above in another test:" ] }, { @@ -880,7 +888,10 @@ "id": "78becfa1", "metadata": {}, "source": [ - "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:" + "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:\n", + "\n", + "- Here, the testing set serves as a proxy for real-world data, as it was untouched during training or tuning, giving us a more authentic estimate of performance.\n", + "- The testing set acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint." ] }, { @@ -901,6 +912,7 @@ }, { "cell_type": "markdown", + "id": "2aae3040", "metadata": {}, "source": [ "\n", @@ -977,6 +989,7 @@ }, { "cell_type": "markdown", + "id": "0be88c10", "metadata": {}, "source": [ "\n", @@ -1104,6 +1117,7 @@ }, { "cell_type": "markdown", + "id": "383cd893", "metadata": {}, "source": [ "\n", From 1a80c533fd8d4880a5bc4db5a9f3c5cb5573cf51 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 22 May 2025 08:58:20 -0700 Subject: [PATCH 24/44] More context to validation series --- .../3-developing_challenger_model.ipynb | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb b/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb index ba6958f28..5066b924f 100644 --- a/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb +++ b/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb @@ -584,7 +584,10 @@ "\n", "#### Evaluate performance of the champion model\n", "\n", - "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:" + "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:\n", + "\n", + "- Here, the testing set serves as a proxy for real-world data, as it was untouched during training or tuning, giving us a more authentic estimate of performance.\n", + "- The testing set acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint." ] }, { @@ -745,7 +748,10 @@ "source": [ "Let's see if models suffer from any *overfit* potentials and also where there are potential sub-segments of issues with the [`OverfitDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/OverfitDiagnosis.html). \n", "\n", - "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data." + "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", + "\n", + "- Since the training dataset (`vm_train_ds`) was used to fit the model, we use this set to establish a baseline performance for how well the model performs on data it has already seen.\n", + "- The testing dataset (`vm_test_ds`) was never seen during training, and here simulates real-world generalization, or how well the model performs on new, unseen data. " ] }, { @@ -769,7 +775,7 @@ "source": [ "Let's also conduct *robustness* and *stability* testing of the two models with the [`RobustnessDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/RobustnessDiagnosis.html).\n", "\n", - "Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets." + "Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets. Again, we'll use both the training and testing dataset to evaluate baseline performance and to simulate real-word generalization." ] }, { @@ -811,6 +817,13 @@ "FI" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We'll only use our testing dataset (`vm_test_ds`) here, to provide a realistic, unseen sample that mimic future or production data, as the training dataset has already influenced our model during learning:" + ] + }, { "cell_type": "code", "execution_count": null, From e3967b6a7be5582c72f1475a253121a05967e657 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 22 May 2025 09:38:29 -0700 Subject: [PATCH 25/44] Added extended Next steps to model development series --- .../quickstart_model_documentation.ipynb | 2 +- .../quickstart/quickstart_model_validation.ipynb | 6 +++--- .../4-finalize_testing_documentation.ipynb | 14 +++++++++++++- .../4-finalize_validation_reporting.ipynb | 10 +++++++--- 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_documentation.ipynb b/notebooks/quickstart/quickstart_model_documentation.ipynb index dc3f375be..67b3f1816 100644 --- a/notebooks/quickstart/quickstart_model_documentation.ipynb +++ b/notebooks/quickstart/quickstart_model_documentation.ipynb @@ -755,7 +755,7 @@ "\n", "2. In the left sidebar that appears for your model, click **Documentation**.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + " What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" ] }, { diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index c81ba8093..078f6aa99 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -49,7 +49,7 @@ " - [Initialize the ValidMind datasets](#toc6_2_) \n", " - [Run an individual data quality test](#toc6_3_) \n", " - [Run data comparison tests](#toc6_4_) \n", - "- [Import the champion model](#toc7_) \n", + "- [Importing the champion model](#toc7_) \n", " - [Initialize a model object](#toc7_1_) \n", " - [Assign predictions](#toc7_2_) \n", "- [Running model evaluation tests](#toc8_) \n", @@ -747,7 +747,7 @@ "source": [ "\n", "\n", - "## Import the champion model\n", + "## Importing the champion model\n", "\n", "With our raw dataset preprocessed, let's go ahead and import the champion model submitted by the model development team in the format of a `.pkl` file: **[xgboost_model_champion.pkl](xgboost_model_champion.pkl)**" ] @@ -1090,7 +1090,7 @@ "\n", "2. In the left sidebar that appears for your model, click **Validation Report**.\n", "\n", - "From here, you can link test results as evidence, add model findings, assess compliance, and submit your validation report for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html)" + " From here, you can link test results as evidence, add model findings, assess compliance, and submit your validation report for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html)" ] }, { diff --git a/notebooks/tutorials/model_development/4-finalize_testing_documentation.ipynb b/notebooks/tutorials/model_development/4-finalize_testing_documentation.ipynb index e38978aec..58e401692 100644 --- a/notebooks/tutorials/model_development/4-finalize_testing_documentation.ipynb +++ b/notebooks/tutorials/model_development/4-finalize_testing_documentation.ipynb @@ -896,7 +896,19 @@ "\n", "### Work with your model documentation\n", "\n", - "Now that you've logged all your test results and generated a draft for your model documentation, head to the ValidMind Platform to make qualitative edits, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. **Learn more:** [Working with model documentation](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)" + "Now that you've logged all your test results and generated a draft for your model documentation, head to the ValidMind Platform to wrap up your model documentation. Continue to work on your model documentation by:\n", + "\n", + "- **Run and log more tests:** Use the skills you learned in this series of notebooks to run and log more individual tests, including custom tests, then insert them into your documentation as supplementary evidence. (Learn more: [`validmind.tests`](https://docs.validmind.ai/validmind/validmind/tests.html))\n", + "\n", + "- **Inserting additional test results:** Add **Test-Driven Blocks** under any relevant section of your model documentation. (Learn more: [Work with test results](https://docs.validmind.ai/guide/model-documentation/work-with-test-results.html))\n", + "\n", + "- **Making qualitative edits to your test descriptions:** Click on the description of any inserted test results to review and edit the ValidMind-generated test descriptions for quality and accuracy. (Learn more: [Working with model documentation](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html#add-or-edit-documentation))\n", + "\n", + "- **View guidelines:** In any section of your model documentation, click **​ValidMind Insights** in the top right corner to reveal the Documentation Guidelines for each section to help guide the contents of your model documentation. (Learn more: [View documentation guidelines](https://docs.validmind.ai/guide/model-documentation/view-documentation-guidelines.html))\n", + "\n", + "- **Collaborate with other stakeholders:** Use the ValidMind Platform's real-time collaborative features to work seamlessly together with the rest of your organization, including model validators. Review suggested changes in your content blocks, work with versioned history, and use comments to discuss specific portions of your model documentation. (Learn more: [Collaborate with others](https://docs.validmind.ai/guide/model-documentation/collaborate-with-others.html))\n", + "\n", + "When your model documentation is complete and ready for review, submit it for approval from the same ValidMind Platform where you made your edits and collaborated with the rest of your organization, ensuring transparency and a thorough model development history. (Learn more: [Submit for approval](https://docs.validmind.ai/guide/model-documentation/submit-for-approval.html))" ] }, { diff --git a/notebooks/tutorials/model_validation/4-finalize_validation_reporting.ipynb b/notebooks/tutorials/model_validation/4-finalize_validation_reporting.ipynb index 886f9e906..ffdc4522a 100644 --- a/notebooks/tutorials/model_validation/4-finalize_validation_reporting.ipynb +++ b/notebooks/tutorials/model_validation/4-finalize_validation_reporting.ipynb @@ -1144,13 +1144,17 @@ "\n", "- **Inserting additional test results:** Click **Link Evidence to Report** under any section of 2. Validation in your validation report. (Learn more: [Link evidence to reports](https://docs.validmind.ai/guide/model-validation/assess-compliance.html#link-evidence-to-reports))\n", "\n", - "- **Making qualitative edits to your test descriptions:** Expand any linked evidence under Validator Evidence and click **See evidence details** to review and edit the ValidMind-generated test descriptions for quality and accuracy.\n", + "- **Making qualitative edits to your test descriptions:** Expand any linked evidence under Validator Evidence and click **See evidence details** to review and edit the ValidMind-generated test descriptions for quality and accuracy. (Learn more: [Preparing validation reports](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html#get-started))\n", "\n", "- **Adding more findings:** Click **Link Finding to Report** in any validation report section, then click **+ Create New Finding**. (Learn more: [Add and manage model findings](https://docs.validmind.ai/guide/model-validation/add-manage-model-findings.html))\n", "\n", - "- **Adding risk assessment notes:** Click under **Risk Assessment Notes** in any validation report section to access the text editor and content editing toolbar, including an option to generate a draft with AI. Edit your ValidMind-generated test descriptions (Learn more: [Work with content blocks](https://docs.validmind.ai/guide/model-documentation/work-with-content-blocks.html#content-editing-toolbar))\n", + "- **Adding risk assessment notes:** Click under **Risk Assessment Notes** in any validation report section to access the text editor and content editing toolbar, including an option to generate a draft with AI. Once generated, edit your ValidMind-generated test descriptions to adhere to your organization's requirements. (Learn more: [Work with content blocks](https://docs.validmind.ai/guide/model-documentation/work-with-content-blocks.html#content-editing-toolbar))\n", "\n", - "- **Assessing compliance:** Under the Guideline for any validation report section, click **ASSESSMENT** and select the compliance status from the drop-down menu. (Learn more: [Provide compliance assessments](https://docs.validmind.ai/guide/model-validation/assess-compliance.html#provide-compliance-assessments))" + "- **Assessing compliance:** Under the Guideline for any validation report section, click **ASSESSMENT** and select the compliance status from the drop-down menu. (Learn more: [Provide compliance assessments](https://docs.validmind.ai/guide/model-validation/assess-compliance.html#provide-compliance-assessments))\n", + "\n", + "- **Collaborate with other stakeholders:** Use the ValidMind Platform's real-time collaborative features to work seamlessly together with the rest of your organization, including model developers. Propose suggested changes in the model documentation, work with versioned history, and use comments to discuss specific portions of the model documentation. (Learn more: [Collaborate with others](https://docs.validmind.ai/guide/model-documentation/collaborate-with-others.html))\n", + "\n", + "When your validation report is complete and ready for review, submit it for approval from the same ValidMind Platform where you made your edits and collaborated with the rest of your organization, ensuring transparency and a thorough model validation history. (Learn more: [Submit for approval](https://docs.validmind.ai/guide/model-documentation/submit-for-approval.html))" ] }, { From 58c53109f42d957dbcff50de1aa842d060484622 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 22 May 2025 11:58:21 -0700 Subject: [PATCH 26/44] Broken link, cleanup --- notebooks/quickstart/quickstart_model_validation.ipynb | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 078f6aa99..4ade4b0cc 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -451,7 +451,7 @@ "\n", "### Preprocess the raw dataset\n", "\n", - "Let's say that thanks to the documentation submitted by the model development team ([Learn more ...](https://docs.validmind.ai/notebooks/quickstart_model_documentation.html)), we know that the sample dataset was first preprocessed before being used to train the champion model.\n", + "Let's say that thanks to the documentation submitted by the model development team ([Learn more ...](https://docs.validmind.ai/notebooks/quickstart/quickstart_model_documentation.html)), we know that the sample dataset was first preprocessed before being used to train the champion model.\n", "\n", "During model validation, we use the same data processing logic and training procedure to confirm that the model's results can be reproduced independently, so let's also start by preprocessing our imported dataset to verify that preprocessing was done correctly. This involves splitting the data and separating the features (inputs) from the targets (outputs)." ] @@ -937,9 +937,7 @@ "id": "ecefecac", "metadata": {}, "source": [ - "Let's see if the model suffers from any *overfit* potentials and also where there are potential sub-segments of issues with the [`OverfitDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/OverfitDiagnosis.html).\n", - "\n", - "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", + "Let's see if the model suffers from any *overfit* potentials and also where there are potential sub-segments of issues. Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", "\n", "- Since the training dataset (`vm_train_ds`) was used to fit the model, we use this set to establish a baseline performance for how well the model performs on data it has already seen.\n", "- The testing dataset (`vm_test_ds`) was never seen during training, and here simulates real-world generalization, or how well the model performs on new, unseen data. " @@ -966,9 +964,9 @@ "id": "c7f7988e", "metadata": {}, "source": [ - "Let's also conduct *robustness* and *stability* testing with the [`RobustnessDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/RobustnessDiagnosis.html).\n", + "Let's also conduct *robustness* and *stability* testing. Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.\n", "\n", - "Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets. Again, we'll use both the training and testing dataset to evaluate baseline performance and to simulate real-word generalization." + "Again, we'll use both the training and testing dataset to evaluate baseline performance and to simulate real-word generalization." ] }, { From 60d6c3516987880a47262840ca1467cacbfb6d70 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 22 May 2025 12:11:18 -0700 Subject: [PATCH 27/44] Tweak --- notebooks/quickstart/quickstart_model_validation.ipynb | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 4ade4b0cc..88d114f9f 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -451,7 +451,7 @@ "\n", "### Preprocess the raw dataset\n", "\n", - "Let's say that thanks to the documentation submitted by the model development team ([Learn more ...](https://docs.validmind.ai/notebooks/quickstart/quickstart_model_documentation.html)), we know that the sample dataset was first preprocessed before being used to train the champion model.\n", + "Let's say that thanks to the documentation submitted by the model development team ([Learn more ...](quickstart_model_documentation.ipynb)), we know that the sample dataset was first preprocessed before being used to train the champion model.\n", "\n", "During model validation, we use the same data processing logic and training procedure to confirm that the model's results can be reproduced independently, so let's also start by preprocessing our imported dataset to verify that preprocessing was done correctly. This involves splitting the data and separating the features (inputs) from the targets (outputs)." ] @@ -937,7 +937,9 @@ "id": "ecefecac", "metadata": {}, "source": [ - "Let's see if the model suffers from any *overfit* potentials and also where there are potential sub-segments of issues. Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", + "Let's see if the model suffers from any *overfit* potentials and also where there are potential sub-segments of issues. \n", + "\n", + "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", "\n", "- Since the training dataset (`vm_train_ds`) was used to fit the model, we use this set to establish a baseline performance for how well the model performs on data it has already seen.\n", "- The testing dataset (`vm_test_ds`) was never seen during training, and here simulates real-world generalization, or how well the model performs on new, unseen data. " @@ -966,7 +968,7 @@ "source": [ "Let's also conduct *robustness* and *stability* testing. Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.\n", "\n", - "Again, we'll use both the training and testing dataset to evaluate baseline performance and to simulate real-word generalization." + "Again, we'll use both the training and testing dataset to evaluate baseline performance and to simulate real-word generalization:" ] }, { From 7e5e83de3db70b729a07b67ac048bd83c3f6ee61 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 09:50:38 -0700 Subject: [PATCH 28/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Juan <117463657+juanmleng@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 1 - 1 file changed, 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 88d114f9f..7fd1ddf28 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -333,7 +333,6 @@ "metadata": {}, "outputs": [], "source": [ - "import xgboost as xgb\n", "\n", "%matplotlib inline" ] From 62dd32e63e8507ea84c9e49d7d6a8553531db32b Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 09:52:33 -0700 Subject: [PATCH 29/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Juan <117463657+juanmleng@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 -- 1 file changed, 2 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 7fd1ddf28..518cf0ad9 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -509,8 +509,6 @@ "source": [ "x_train = train_df.drop(customer_churn.target_column, axis=1)\n", "y_train = train_df[customer_churn.target_column]\n", - "x_val = validation_df.drop(customer_churn.target_column, axis=1)\n", - "y_val = validation_df[customer_churn.target_column]" ] }, { From e456d9712f81b3da527b7798f67a033825c00d9b Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 09:53:06 -0700 Subject: [PATCH 30/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Juan <117463657+juanmleng@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 518cf0ad9..9a51e8ede 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -497,7 +497,7 @@ "1. **Inputs** — Features such as customer age, usage, etc.\n", "2. **Outputs (Expected answers/labels)** — in our case, we would like to know whether the customer churned or not.\n", "\n", - "Here, we'll use `x_train` and `x_val` to hold the input data (features), and `y_train` and `y_val` to hold the answers (the target we want to predict):" + "Here, we'll use `x_train` to hold the input features, and `y_train` to hold the target variable — the values we want the model to predict." ] }, { From 31c07d442265badc4497c09df50cda464617f798 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 09:53:44 -0700 Subject: [PATCH 31/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Juan <117463657+juanmleng@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 9a51e8ede..6eeccc64a 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -757,10 +757,9 @@ "outputs": [], "source": [ "# Import the champion model\n", - "import pickle as pkl\n", + "import joblib\n", "\n", - "with open(\"xgboost_model_champion.pkl\", \"rb\") as f:\n", - " xgboost = pkl.load(f)" + "xgboost = joblib.load("xgboost_model_champion.pkl")" ] }, { From f6749711c7471f8403971845d325039c37df694a Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 09:54:19 -0700 Subject: [PATCH 32/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Juan <117463657+juanmleng@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 6eeccc64a..c9662807a 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -1026,8 +1026,8 @@ "for test in FI:\n", " vm.tests.run_test(\n", " \"\".join((test,':xgboost_champion')),\n", - " input_grid={\n", - " \"dataset\": [vm_test_ds], \"model\" : [vm_xgboost]\n", + " inputs={\n", + " \"dataset\": vm_test_ds, \"model\": vm_xgboost\n", " },\n", " ).log()" ] From 5c2586a727ac6ad19656a5e4feb88779030a46f2 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:01:12 -0700 Subject: [PATCH 33/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Lois Ansah <133300328+LoiAnsah@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index c9662807a..9a1d82f34 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -964,7 +964,7 @@ "source": [ "Let's also conduct *robustness* and *stability* testing. Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.\n", "\n", - "Again, we'll use both the training and testing dataset to evaluate baseline performance and to simulate real-word generalization:" + "We'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:" ] }, { From e614fb5329855c96a0a8b038fcebf4f013f8f0e5 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:01:51 -0700 Subject: [PATCH 34/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Lois Ansah <133300328+LoiAnsah@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 9a1d82f34..289b759e4 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -962,7 +962,7 @@ "id": "c7f7988e", "metadata": {}, "source": [ - "Let's also conduct *robustness* and *stability* testing. Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.\n", + "Let's also conduct *robustness* and *stability* tests.\n- Robustness evaluates the model’s ability to maintain consistent performance under varying input conditions.\n- Stability assesses whether the model produces consistent outputs across different data subsets or over time." "\n", "We'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:" ] From 3107dd4bcd4b4c83c2bf49d961091fba900e0fdf Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:03:19 -0700 Subject: [PATCH 35/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Lois Ansah <133300328+LoiAnsah@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 289b759e4..de3f5ca90 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -933,7 +933,7 @@ "id": "ecefecac", "metadata": {}, "source": [ - "Let's see if the model suffers from any *overfit* potentials and also where there are potential sub-segments of issues. \n", + "Let’s now assess the model for potential signs of overfitting and identify any sub-segments where performance may inconsistent.\n", "\n", "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", "\n", From fbb4aee5581667650b4f85ac92f3ce94e03ef6f5 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:03:47 -0700 Subject: [PATCH 36/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Lois Ansah <133300328+LoiAnsah@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index de3f5ca90..a9bc3cae8 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -915,7 +915,7 @@ "\n", "### Run diagnostic tests\n", "\n", - "Next we want to inspect the robustness and stability for our champion model. Use `list_tests()` to identify all the model diagnosis tests for classification:" + "Next, we want to inspect the robustness and stability of our champion model. Use `list_tests()` to list all available diagnosis tests applicable to classification tasks:" ] }, { From 339d696945027b249be74bd74a1ad6aa213a68aa Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:04:04 -0700 Subject: [PATCH 37/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Lois Ansah <133300328+LoiAnsah@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index a9bc3cae8..4b6e7a1f1 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -887,7 +887,7 @@ "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:\n", "\n", "- Here, the testing set serves as a proxy for real-world data, as it was untouched during training or tuning, giving us a more authentic estimate of performance.\n", - "- The testing set acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint." + "- The test set also acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint." ] }, { From b39795622d86c084e3edbbe82bb5d821ac075ba2 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:04:26 -0700 Subject: [PATCH 38/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Lois Ansah <133300328+LoiAnsah@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 4b6e7a1f1..a91e25336 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -886,7 +886,7 @@ "source": [ "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:\n", "\n", - "- Here, the testing set serves as a proxy for real-world data, as it was untouched during training or tuning, giving us a more authentic estimate of performance.\n", + "- The test set serves as a proxy for real-world data, providing an unbiased estimate of model performance since it was not used during training or tuning.\n", "- The test set also acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint." ] }, From 4a8e33fbad34628b07bcb50c0d0897bbffe1f3d4 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:07:54 -0700 Subject: [PATCH 39/44] Update notebooks/quickstart/quickstart_model_validation.ipynb Co-authored-by: Lois Ansah <133300328+LoiAnsah@users.noreply.github.com> --- notebooks/quickstart/quickstart_model_validation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index a91e25336..5f9d8e75f 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -832,7 +832,7 @@ "\n", "## Running model evaluation tests\n", "\n", - "With everything ready for us, let's run the rest of our validation tests. We'll focus on testing around model performance of the champion model going forward as we've already verified the data quality of the datasets used to train the champion model." + "With our setup complete, let's run the rest of our validation tests. Since we have already verified the data quality of the dataset used to train our champion model, we will now focus on evaluating the model's performance." ] }, { From f5b4a856c12574bca27b535918d83bf62ca06507 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:28:50 -0700 Subject: [PATCH 40/44] Fixing JSON errors from suggestions --- notebooks/quickstart/quickstart_model_validation.ipynb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 5f9d8e75f..22d07161f 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -508,7 +508,7 @@ "outputs": [], "source": [ "x_train = train_df.drop(customer_churn.target_column, axis=1)\n", - "y_train = train_df[customer_churn.target_column]\n", + "y_train = train_df[customer_churn.target_column]\n" ] }, { @@ -759,7 +759,7 @@ "# Import the champion model\n", "import joblib\n", "\n", - "xgboost = joblib.load("xgboost_model_champion.pkl")" + "xgboost = joblib.load(\"xgboost_model_champion.pkl\")" ] }, { @@ -962,7 +962,7 @@ "id": "c7f7988e", "metadata": {}, "source": [ - "Let's also conduct *robustness* and *stability* tests.\n- Robustness evaluates the model’s ability to maintain consistent performance under varying input conditions.\n- Stability assesses whether the model produces consistent outputs across different data subsets or over time." + "Let's also conduct *robustness* and *stability* tests.\n- Robustness evaluates the model’s ability to maintain consistent performance under varying input conditions.\n- Stability assesses whether the model produces consistent outputs across different data subsets or over time.\n", "\n", "We'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:" ] From e5d98af44b448d876fe91f8e26f9d8b336e9e444 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:46:08 -0700 Subject: [PATCH 41/44] Fixing context around Juan's suggestions --- .../quickstart/quickstart_model_validation.ipynb | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 22d07161f..370332446 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -320,10 +320,9 @@ "\n", "### Initialize the Python environment\n", "\n", - "Then, let's import the necessary libraries and set up your Python environment for data analysis:\n", + "Then, let's import the necessary libraries and set up your Python environment for data analysis by enabling **`matplotlib`**, a plotting library used for visualizing data.\n", "\n", - "- Import **Extreme Gradient Boosting** (XGBoost) with an alias so that we can reference its functions in later calls. XGBoost is a powerful machine learning library designed for speed and performance, especially in handling structured or tabular data.\n", - "- Enable **`matplotlib`**, a plotting library used for visualizing data. Ensures that any plots you generate will render inline in our notebook output rather than opening in a separate window." + "This ensures that any plots you generate will render inline in our notebook output rather than opening in a separate window:" ] }, { @@ -497,7 +496,7 @@ "1. **Inputs** — Features such as customer age, usage, etc.\n", "2. **Outputs (Expected answers/labels)** — in our case, we would like to know whether the customer churned or not.\n", "\n", - "Here, we'll use `x_train` to hold the input features, and `y_train` to hold the target variable — the values we want the model to predict." + "Here, we'll use `x_train` to hold the input features, and `y_train` to hold the target variable — the values we want the model to predict:" ] }, { @@ -508,7 +507,7 @@ "outputs": [], "source": [ "x_train = train_df.drop(customer_churn.target_column, axis=1)\n", - "y_train = train_df[customer_churn.target_column]\n" + "y_train = train_df[customer_churn.target_column]" ] }, { @@ -962,7 +961,9 @@ "id": "c7f7988e", "metadata": {}, "source": [ - "Let's also conduct *robustness* and *stability* tests.\n- Robustness evaluates the model’s ability to maintain consistent performance under varying input conditions.\n- Stability assesses whether the model produces consistent outputs across different data subsets or over time.\n", + "Let's also conduct *robustness* and *stability* tests.\n", + "- Robustness evaluates the model’s ability to maintain consistent performance under varying input conditions.\n", + "- Stability assesses whether the model produces consistent outputs across different data subsets or over time.\n", "\n", "We'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:" ] From 88396e94b9094a641a8de86d6b9280f524a29cd4 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 10:55:09 -0700 Subject: [PATCH 42/44] Removing unneeded validation dataset initialization --- notebooks/quickstart/quickstart_model_validation.ipynb | 7 ------- 1 file changed, 7 deletions(-) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 370332446..30afd1b80 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -608,13 +608,6 @@ " target_column=customer_churn.target_column,\n", ")\n", "\n", - "# Initialize the validation dataset\n", - "vm_validation_ds = vm.init_dataset(\n", - " dataset=validation_df,\n", - " input_id=\"validation_dataset\",\n", - " target_column=customer_churn.target_column,\n", - ")\n", - "\n", "# Initialize the testing dataset\n", "vm_test_ds = vm.init_dataset(\n", " dataset=test_df,\n", From 6248385107e6eb651dfbcc96df3b072bc5839d8e Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 11:05:55 -0700 Subject: [PATCH 43/44] Proofreading Ama's suggestions --- .../validate_application_scorecard.ipynb | 4 ++-- .../quickstart/quickstart_model_validation.ipynb | 5 +++-- .../3-developing_challenger_model.ipynb | 16 ++++++++-------- 3 files changed, 13 insertions(+), 12 deletions(-) diff --git a/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb b/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb index 3b1ea6b31..6dd025294 100644 --- a/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb +++ b/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb @@ -1364,9 +1364,9 @@ "\n", "## Run diagnostic tests\n", "\n", - "Next we want to inspect the robustness and stability testing comparison between our champion and challenger model.\n", + "Next, we want to inspect the robustness and stability testing comparison between our champion and challenger model.\n", "\n", - "Use `list_tests()` to identify all the model diagnosis tests for classification:" + "Use `list_tests()` to list all available diagnosis tests applicable to classification tasks:" ] }, { diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 30afd1b80..45a25f4e5 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -925,7 +925,7 @@ "id": "ecefecac", "metadata": {}, "source": [ - "Let’s now assess the model for potential signs of overfitting and identify any sub-segments where performance may inconsistent.\n", + "Let’s now assess the model for potential signs of *overfitting* and identify any sub-segments where performance may inconsistent.\n", "\n", "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", "\n", @@ -955,10 +955,11 @@ "metadata": {}, "source": [ "Let's also conduct *robustness* and *stability* tests.\n", + "\n", "- Robustness evaluates the model’s ability to maintain consistent performance under varying input conditions.\n", "- Stability assesses whether the model produces consistent outputs across different data subsets or over time.\n", "\n", - "We'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:" + "Again, we'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:" ] }, { diff --git a/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb b/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb index 5066b924f..83061a827 100644 --- a/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb +++ b/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb @@ -520,7 +520,7 @@ "\n", "## Running model evaluation tests\n", "\n", - "With everything ready for us, let's run the rest of our validation tests. We'll focus on comprehensive testing around model performance of both the champion and challenger models going forward as we've already verified the data quality of the datasets used to train the champion model." + "With our setup complete, let's run the rest of our validation tests. Since we have already verified the data quality of the dataset used to train our champion model, we will now focus on comprehensive performance evaluations of both the champion and challenger models." ] }, { @@ -586,8 +586,8 @@ "\n", "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:\n", "\n", - "- Here, the testing set serves as a proxy for real-world data, as it was untouched during training or tuning, giving us a more authentic estimate of performance.\n", - "- The testing set acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint." + "- The test set serves as a proxy for real-world data, providing an unbiased estimate of model performance since it was not used during training or tuning.\n", + "- The test set also acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint." ] }, { @@ -728,9 +728,9 @@ "\n", "### Run diagnostic tests\n", "\n", - "Next we want to inspect the robustness and stability testing comparison between our champion and challenger model.\n", + "Next, we want to inspect the robustness and stability testing comparison between our champion and challenger model.\n", "\n", - "Use `list_tests()` to identify all the model diagnosis tests for classification:" + "Use `list_tests()` to list all available diagnosis tests applicable to classification tasks:" ] }, { @@ -746,7 +746,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's see if models suffer from any *overfit* potentials and also where there are potential sub-segments of issues with the [`OverfitDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/OverfitDiagnosis.html). \n", + "Let’s now assess the models for potential signs of *overfitting* and identify any sub-segments where performance may inconsistent with the [`OverfitDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/OverfitDiagnosis.html).\n", "\n", "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n", "\n", @@ -773,9 +773,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's also conduct *robustness* and *stability* testing of the two models with the [`RobustnessDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/RobustnessDiagnosis.html).\n", + "Let's also conduct *robustness* and *stability* testing of the two models with the [`RobustnessDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/RobustnessDiagnosis.html). Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.\n", "\n", - "Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets. Again, we'll use both the training and testing dataset to evaluate baseline performance and to simulate real-word generalization." + "Again, we'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:" ] }, { From 4bf526694553d94f76d8aeaf4307311a1981904f Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 27 May 2025 12:50:11 -0700 Subject: [PATCH 44/44] Readding the validation dataset lol oops --- notebooks/quickstart/quickstart_model_validation.ipynb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index 45a25f4e5..4e233f963 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -608,6 +608,13 @@ " target_column=customer_churn.target_column,\n", ")\n", "\n", + "# Initialize the validation dataset\n", + "vm_validation_ds = vm.init_dataset(\n", + " dataset=validation_df,\n", + " input_id=\"validation_dataset\",\n", + " target_column=customer_churn.target_column,\n", + ")\n", + "\n", "# Initialize the testing dataset\n", "vm_test_ds = vm.init_dataset(\n", " dataset=test_df,\n",