For access to all features available in this notebook, create a free ValidMind account.\n",
+ "
\n", + "Signing up is FREE — Register with ValidMind
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "31c5cde0",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Key concepts\n",
+ "\n",
+ "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n",
+ "\n",
+ "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n",
+ "\n",
+ "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n",
+ "\n",
+ "**Metrics**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.\n",
+ "\n",
+ "**Custom metrics**: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.\n",
+ "\n",
+ "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n",
+ "\n",
+ " - **model**: A single model that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n",
+ " - **dataset**: Single dataset that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n",
+ " - **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.\n",
+ " - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/run_tests_that_require_multiple_datasets.html))\n",
+ "\n",
+ "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.\n",
+ "\n",
+ "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.\n",
+ "\n",
+ "**Test suites**: Collections of tests designed to run together to automate and generate model documentation end-to-end for specific use-cases.\n",
+ "\n",
+ "Example: the [`classifier_full_suite`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html#ClassifierFullSuite) test suite runs tests from the [`tabular_dataset`](https://docs.validmind.ai/validmind/validmind/test_suites/tabular_datasets.html) and [`classifier`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html) test suites to fully document the data and model sections for binary classification model use-cases."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1c06378f",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Initializing the ValidMind Library\n",
+ "\n",
+ "The ValidMind Library provides a rich collection of documentation tools and test suites, from documenting descriptions of datasets to validation and testing of models using a variety of open-source testing frameworks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Install the ValidMind Library\n",
+ "\n",
+ "\n", + "Signing up is FREE — Register with ValidMind
Recommended Python versions\n",
+ "
\n", + "Python 3.8 <= x <= 3.11
\n",
+ "\n",
+ "To install the library:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8883bbc3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%pip install -q validmind"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "780b6b39",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Initialize the ValidMind Library\n",
+ "\n",
+ "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Get your code snippet\n",
+ "\n",
+ "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
+ "\n",
+ "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n",
+ "\n",
+ "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
+ "\n",
+ " For example, to register a model for use with this notebook, select:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d00f6f07",
+ "metadata": {},
+ "source": [
+ " - Documentation template: `Binary classification`\n",
+ " - Use case: `Marketing/Sales - Attrition/Churn Management`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "install-credentials-50e67128-2eb5-470a-aeaf-1c692fd3f847",
+ "metadata": {},
+ "source": [
+ " You can fill in other options according to your preference.\n",
+ " \n",
+ "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
+ "\n",
+ "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5f22e91d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Load your model identifier credentials from an `.env` file\n",
+ "\n",
+ "%load_ext dotenv\n",
+ "%dotenv .env\n",
+ "\n",
+ "# Or replace with your code snippet\n",
+ "\n",
+ "import validmind as vm\n",
+ "\n",
+ "vm.init(\n",
+ " # api_host=\"...\",\n",
+ " # api_key=\"...\",\n",
+ " # api_secret=\"...\",\n",
+ " # model=\"...\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c3186121",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Getting to know ValidMind"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Preview the documentation template\n",
+ "\n",
+ "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n",
+ "\n",
+ "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "32ab4cac",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm.preview_template()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Explore available tests\n",
+ "\n",
+ "Next, let's explore the list of all available tests in the ValidMind Library with [the `vm.tests.list_tests()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) — we'll learn how to run tests shortly. \n",
+ "\n",
+ "You can see that the documentation template for this model has references to some of the **test `ID`s used to run tests listed below:**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "acf76128",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm.tests.list_tests()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Upgrade ValidMind\n",
+ "\n",
+ "\n", + "Python 3.8 <= x <= 3.11
After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.
\n",
+ "\n",
+ "Retrieve the information for the currently installed version of ValidMind:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "95bede03",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%pip show validmind"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "540efef8",
+ "metadata": {},
+ "source": [
+ "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
+ "\n",
+ "```bash\n",
+ "%pip install --upgrade validmind\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6a7bf101",
+ "metadata": {},
+ "source": [
+ "You may need to restart your kernel after running the upgrade package for changes to be applied."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "207875f2",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## In summary\n",
+ "\n",
+ "In this first notebook, you learned how to:\n",
+ "\n",
+ "- [ ] Register a model within the ValidMind Platform\n",
+ "- [ ] Install and initialize the ValidMind Library\n",
+ "- [ ] Preview the documentation template for your model\n",
+ "- [ ] Explore the available tests offered by the ValidMind Library"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "29781eb4",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Next steps"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Start the model development process\n",
+ "\n",
+ "Now that the ValidMind Library is connected to your model in the ValidMind Library with the correct template applied, we can go ahead and start the model development process: **[102 Start the model development process](102-start_development_process.ipynb)**"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "ValidMind Library",
+ "language": "python",
+ "name": "validmind"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.10.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/site/notebooks/EXECUTED/model_development/102-start_development_process.ipynb b/site/notebooks/EXECUTED/model_development/102-start_development_process.ipynb
new file mode 100644
index 0000000000..4cdfa0247e
--- /dev/null
+++ b/site/notebooks/EXECUTED/model_development/102-start_development_process.ipynb
@@ -0,0 +1,950 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# ValidMind for model development — 102 Start the model development process\n",
+ "\n",
+ "Learn how to use ValidMind for your end-to-end model documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.\n",
+ "\n",
+ "You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a model is being built appropriately. \n",
+ "\n",
+ "**For a full list of out-of-the-box tests,** refer to our [Test descriptions](https://docs.validmind.ai/developer/model-testing/test-descriptions.html) or try the interactive [Test sandbox](https://docs.validmind.ai/developer/model-testing/test-sandbox.html)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "::: {.content-hidden when-format=\"html\"}\n",
+ "## Contents \n",
+ "- [Prerequisites](#toc1_) \n",
+ "- [Setting up](#toc2_) \n",
+ " - [Initialize the ValidMind Library](#toc2_1_) \n",
+ " - [Import sample dataset](#toc2_2_) \n",
+ " - [Identify qualitative tests](#toc2_3_) \n",
+ " - [Initialize the ValidMind datasets](#toc2_4_) \n",
+ "- [Running tests](#toc3_) \n",
+ " - [Run tabular data tests](#toc3_1_) \n",
+ " - [Utilize test output](#toc3_2_) \n",
+ "- [Documenting results](#toc4_) \n",
+ " - [Run and log multiple tests](#toc4_1_) \n",
+ " - [Run and log an individual test](#toc4_2_) \n",
+ " - [Add individual test results to model documentation](#toc4_2_1_) \n",
+ "- [Model testing](#toc5_) \n",
+ " - [Train simple logistic regression model](#toc5_1_) \n",
+ " - [Initialize model evaluation objects](#toc5_2_) \n",
+ " - [Assign predictions](#toc5_3_) \n",
+ " - [Run the model evaluation tests](#toc5_4_) \n",
+ "- [In summary](#toc6_) \n",
+ "- [Next steps](#toc7_) \n",
+ " - [Integrate custom tests](#toc7_1_) \n",
+ "\n",
+ ":::\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Prerequisites\n",
+ "\n",
+ "In order to log test results or evidence to your model documentation with this notebook, you'll need to first have:\n",
+ "\n",
+ "- [ ] Registered a model within the ValidMind Platform with a predefined documentation template\n",
+ "- [ ] Installed the ValidMind Library in your local environment, allowing you to access all its features\n",
+ "\n",
+ "Need help with the above steps?\n",
+ "
\n", + "Refer to the first notebook in this series: 101 Set up ValidMind
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Initialize the ValidMind Library\n",
+ "\n",
+ "First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
+ "\n",
+ "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
+ "\n",
+ "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n",
+ "\n",
+ "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
+ "\n",
+ "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Make sure the ValidMind Library is installed\n",
+ "\n",
+ "%pip install -q validmind\n",
+ "\n",
+ "# Load your model identifier credentials from an `.env` file\n",
+ "\n",
+ "%load_ext dotenv\n",
+ "%dotenv .env\n",
+ "\n",
+ "# Or replace with your code snippet\n",
+ "\n",
+ "import validmind as vm\n",
+ "\n",
+ "vm.init(\n",
+ " # api_host=\"...\",\n",
+ " # api_key=\"...\",\n",
+ " # api_secret=\"...\",\n",
+ " # model=\"...\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Import sample dataset\n",
+ "\n",
+ "Then, let's import the public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle. \n",
+ "\n",
+ "In our below example, note that: \n",
+ "\n",
+ "- The target column, `Exited` has a value of `1` when a customer has churned and `0` otherwise.\n",
+ "- The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from validmind.datasets.classification import customer_churn as demo_dataset\n",
+ "\n",
+ "print(\n",
+ " f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
+ ")\n",
+ "\n",
+ "raw_df = demo_dataset.load_data()\n",
+ "raw_df.head()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Identify qualitative tests\n",
+ "\n",
+ "Next, let's say we want to do some data quality assessments by running a few individual tests.\n",
+ "\n",
+ "Use the [`vm.tests.list_tests()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) introduced by the first notebook in this series in combination with [`vm.tests.list_tags()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tags) and [`vm.tests.list_tasks()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tasks) to find which prebuilt tests are relevant for data quality assessment:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Get the list of available tags\n",
+ "sorted(vm.tests.list_tags())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Get the list of available task types\n",
+ "sorted(vm.tests.list_tasks())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You can pass `tags` and `tasks` as parameters to the `vm.tests.list_tests()` function to filter the tests based on the tags and task types.\n",
+ "\n",
+ "For example, to find tests related to tabular data quality for classification models, you can call `list_tests()` like this:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm.tests.list_tests(task=\"classification\", tags=[\"tabular_data\", \"data_quality\"])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Initialize the ValidMind datasets\n",
+ "\n",
+ "With the individual tests we want to run identified, the next step is to connect your data with a ValidMind `Dataset` object. **This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind,** but you only need to do it once per dataset.\n",
+ "\n",
+ "Initialize a ValidMind dataset object using the [`init_dataset` function](https://docs.validmind.ai/validmind/validmind.html#init_dataset) from the ValidMind (`vm`) module. For this example, we'll pass in the following arguments:\n",
+ "\n",
+ "- **`dataset`** — The raw dataset that you want to provide as input to tests.\n",
+ "- **`input_id`** — A unique identifier that allows tracking what inputs are used when running each individual test.\n",
+ "- **`target_column`** — A required argument if tests require access to true values. This is the name of the target column in the dataset.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test\n",
+ "vm_raw_dataset = vm.init_dataset(\n",
+ " dataset=raw_df,\n",
+ " input_id=\"raw_dataset\",\n",
+ " target_column=\"Exited\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Running tests\n",
+ "\n",
+ "Now that we know how to initialize a ValidMind `dataset` object, we're ready to run some tests!\n",
+ "\n",
+ "You run individual tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module. For the examples below, we'll pass in the following arguments:\n",
+ "\n",
+ "- **`test_id`** — The ID of the test to run, as seen in the `ID` column when you run `list_tests`. \n",
+ "- **`params`** — A dictionary of parameters for the test. These will override any `default_params` set in the test definition. \n",
+ "\n",
+ "\n", + "Refer to the first notebook in this series: 101 Set up ValidMind
Want to learn more about ValidMind tests?\n",
+ "
\n", + "Refer to our notebook that includes code samples and usage of key functions: Explore tests
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Run tabular data tests\n",
+ "\n",
+ "The inputs expected by a test can also be found in the test definition — let's take [`validmind.data_validation.DescriptiveStatistics`](https://docs.validmind.ai/tests/data_validation/DescriptiveStatistics.html) as an example.\n",
+ "\n",
+ "Note that the output of the [`describe_test()` function](https://docs.validmind.ai/validmind/validmind/tests.html#describe_test) below shows that this test expects a `dataset` as input:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm.tests.describe_test(\"validmind.data_validation.DescriptiveStatistics\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, let's run a few tests to assess the quality of the dataset:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.DescriptiveStatistics\",\n",
+ " inputs={\"dataset\": vm_raw_dataset},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "result2 = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.ClassImbalance\",\n",
+ " inputs={\"dataset\": vm_raw_dataset},\n",
+ " params={\"min_percent_threshold\": 30},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The output above shows that [the class imbalance test](https://docs.validmind.ai/tests/data_validation/ClassImbalance.html) did not pass according to the value we set for `min_percent_threshold`.\n",
+ "\n",
+ "To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "raw_copy_df = raw_df.sample(frac=1) # Create a copy of the raw dataset\n",
+ "\n",
+ "# Create a balanced dataset with the same number of exited and not exited customers\n",
+ "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
+ "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
+ "\n",
+ "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
+ "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.\n",
+ "\n",
+ "As this is technically a different dataset, **remember to first initialize a new ValidMind `Dataset` object** to pass in as input as required by `run_test()`:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
+ "vm_balanced_raw_dataset = vm.init_dataset(\n",
+ " dataset=balanced_raw_df,\n",
+ " input_id=\"balanced_raw_dataset\",\n",
+ " target_column=\"Exited\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Pass the initialized `balanced_raw_dataset` as input into the test run\n",
+ "result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.ClassImbalance\",\n",
+ " inputs={\"dataset\": vm_balanced_raw_dataset},\n",
+ " params={\"min_percent_threshold\": 30},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ "### Utilize test output\n",
+ "\n",
+ "You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.\n",
+ "\n",
+ "First, we'll run [`validmind.data_validation.HighPearsonCorrelation`](https://docs.validmind.ai/tests/data_validation/HighPearsonCorrelation.html) with the `balanced_raw_dataset` we initialized previously as input as is for comparison with later runs:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "corr_result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
+ " params={\"max_threshold\": 0.3},\n",
+ " inputs={\"dataset\": vm_balanced_raw_dataset},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "`corr_result` is an object of type `TestResult`. We can inspect the result object to see what the test has produced:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "print(type(corr_result))\n",
+ "print(\"Result ID: \", corr_result.result_id)\n",
+ "print(\"Params: \", corr_result.params)\n",
+ "print(\"Passed: \", corr_result.passed)\n",
+ "print(\"Tables: \", corr_result.tables)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's remove the highly correlated features and create a new VM `dataset` object.\n",
+ "\n",
+ "We'll begin by checking out the table in the result and extracting a list of features that failed the test:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "# Extract table from `corr_result.tables`\n",
+ "features_df = corr_result.tables[0].data\n",
+ "features_df"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "# Extract list of features that failed the test\n",
+ "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
+ "high_correlation_features"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, extract the feature names from the list of strings (example: `(Age, Exited)` > `Age`):"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
+ "high_correlation_features"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, it's time to re-initialize the dataset with the highly correlated features removed.\n",
+ "\n",
+ "**Note the use of a different `input_id`.** This allows tracking the inputs used when running each individual test."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "# Remove the highly correlated features from the dataset\n",
+ "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
+ "\n",
+ "# Re-initialize the dataset object\n",
+ "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
+ " dataset=balanced_raw_no_age_df,\n",
+ " input_id=\"raw_dataset_preprocessed\",\n",
+ " target_column=\"Exited\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Re-running the test with the reduced feature set should pass the test:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "corr_result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
+ " params={\"max_threshold\": 0.3},\n",
+ " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You can also plot the correlation matrix to visualize the new correlation between features:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "corr_result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.PearsonCorrelationMatrix\",\n",
+ " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Documenting results\n",
+ "\n",
+ "We've now done some analysis on two different datasets, and we should be able to document why certain things were done to the raw data with testing to support it.\n",
+ "\n",
+ "Every test result returned by the `run_test()` function has a [`.log()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) that can be used to send the test results to the ValidMind Platform:\n",
+ "\n",
+ "- When using `run_documentation_tests()`, documentation sections will be automatically populated with the results of all tests registered in the documentation template.\n",
+ "- When logging individual test results to the platform, you'll need to manually add those results to the desired section of the model documentation.\n",
+ "\n",
+ "To demonstrate how to add test results to your model documentation, we'll populate the entire **Data Preparation** section of the documentation using the clean `vm_raw_dataset_preprocessed` dataset as input, and then document an additional individual result for the highly correlated dataset `vm_balanced_raw_dataset`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ "### Run and log multiple tests\n",
+ "\n",
+ "[`run_documentation_tests()`](https://docs.validmind.ai/validmind/validmind.html#run_documentation_tests) allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized `vm_raw_dataset_preprocessed` as input — this will populate the entire **Data Preparation** section for every test that is part of the documentation template.\n",
+ "\n",
+ "For this example, we'll pass in the following arguments:\n",
+ "\n",
+ "- **`inputs`:** Any inputs to be passed to the tests.\n",
+ "- **`config`:** A dictionary `\n", + "Refer to our notebook that includes code samples and usage of key functions: Explore tests
Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID. \n",
+ "
\n", + "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Add individual test results to model documentation\n",
+ "\n",
+ "With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation ([Need more help?](https://docs.validmind.ai/developer/model-documentation/work-with-test-results.html)):\n",
+ "\n",
+ "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n",
+ "\n",
+ "2. In the left sidebar that appears for your model, click **Documentation**.\n",
+ "\n",
+ "3. Locate the Data Preparation section and click on **2.3 Correlations and Interactions** to expand that section.\n",
+ "\n",
+ "4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a **+** button appears, indicating that you can insert a new block.\n",
+ "\n",
+ " \n", + "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.
![\"Screenshot](\"add-content-block.gif\")
\n",
+ " \n", + "\n", + "5. Click **+** and then select **Test-Driven Block**:\n", + "\n", + " - In the search bar, type in `HighPearsonCorrelation`.\n", + " - Select `HighPearsonCorrelation:balanced_raw_dataset` as the test.\n", + "\n", + " A preview of the test gets shown:\n", + "\n", + "
![\"Screenshot](\"selecting-high-pearson-correlation-test.png\")
\n",
+ " \n", + "\n", + "6. Finally, click **Insert 1 Test Result to Document** to add the test result to the documentation.\n", + "\n", + " Confirm that the individual results for the high correlation test has been correctly inserted into section **2.3 Correlations and Interactions** of the documentation.\n", + "\n", + "7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:\n", + "\n", + "
![\"Screenshot](\"high-pearson-correlation-block.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Model testing\n",
+ "\n",
+ "So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Train simple logistic regression model\n",
+ "\n",
+ "Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the `LogisticRegression` class from the `sklearn.linear_model`.\n",
+ "\n",
+ "To start, let's grab the first few rows from the `balanced_raw_no_age_df` dataset with the highly correlated features removed we initialized earlier:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "balanced_raw_no_age_df.head()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Before training the model, we need to encode the categorical features in the dataset:\n",
+ "\n",
+ "- Use the `OneHotEncoder` class from the `sklearn.preprocessing` module to encode the categorical features.\n",
+ "- The categorical features in the dataset are `Geography` and `Gender`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "balanced_raw_no_age_df = pd.get_dummies(\n",
+ " balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
+ ")\n",
+ "balanced_raw_no_age_df.head()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Using `GridSearchCV`, we'll find the best-performing hyperparameters or settings and save them:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.linear_model import LogisticRegression\n",
+ "from sklearn.model_selection import train_test_split\n",
+ "\n",
+ "# Split the input and target variables\n",
+ "X = balanced_raw_no_age_df.drop(\"Exited\", axis=1)\n",
+ "y = balanced_raw_no_age_df[\"Exited\"]\n",
+ "X_train, X_test, y_train, y_test = train_test_split(\n",
+ " X,\n",
+ " y,\n",
+ " test_size=0.2,\n",
+ " random_state=42,\n",
+ ")\n",
+ "\n",
+ "# Logistic Regression grid params\n",
+ "log_reg_params = {\n",
+ " \"penalty\": [\"l1\", \"l2\"],\n",
+ " \"C\": [0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
+ " \"solver\": [\"liblinear\"],\n",
+ "}\n",
+ "\n",
+ "# Grid search for Logistic Regression\n",
+ "from sklearn.model_selection import GridSearchCV\n",
+ "\n",
+ "grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)\n",
+ "grid_log_reg.fit(X_train, y_train)\n",
+ "\n",
+ "# Logistic Regression best estimator\n",
+ "log_reg = grid_log_reg.best_estimator_"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Initialize model evaluation objects\n",
+ "\n",
+ "The last step for evaluating the model's performance is to initialize the ValidMind `Dataset` and `Model` objects in preparation for assigning model predictions to each dataset.\n",
+ "\n",
+ "Use the `init_dataset` and [`init_model`](https://docs.validmind.ai/validmind/validmind.html#init_model) functions to initialize these objects:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train_df = X_train\n",
+ "train_df[\"Exited\"] = y_train\n",
+ "test_df = X_test\n",
+ "test_df[\"Exited\"] = y_test\n",
+ "\n",
+ "vm_train_ds = vm.init_dataset(\n",
+ " input_id=\"train_dataset_final\",\n",
+ " dataset=train_df,\n",
+ " target_column=\"Exited\",\n",
+ ")\n",
+ "\n",
+ "vm_test_ds = vm.init_dataset(\n",
+ " input_id=\"test_dataset_final\",\n",
+ " dataset=test_df,\n",
+ " target_column=\"Exited\",\n",
+ ")\n",
+ "\n",
+ "# Register the model\n",
+ "vm_model = vm.init_model(log_reg, input_id=\"log_reg_model_v1\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Assign predictions\n",
+ "\n",
+ "Once the model has been registered you can assign model predictions to the training and test datasets. The [`assign_predictions()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#VMDataset.assign_predictions) from the `Dataset` object can link existing predictions to any number of models.\n",
+ "\n",
+ "If no prediction values are passed, the method will compute predictions automatically:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm_train_ds.assign_predictions(model=vm_model)\n",
+ "vm_test_ds.assign_predictions(model=vm_model)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Run the model evaluation tests\n",
+ "\n",
+ "In this next example, we'll focus on running the tests within the Model Development section of the model documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the model documentation.\n",
+ "\n",
+ "- Note the additional config that is passed to `run_documentation_tests()` — this allows you to override `inputs` or `params` in certain tests.\n",
+ "- In our case, we want to explicitly use the `vm_train_ds` for the [`validmind.model_validation.sklearn.ClassifierPerformance:in_sample` test](https://docs.validmind.ai/tests/model_validation/sklearn/ClassifierPerformance.html), since it's supposed to run on the training dataset and not the test dataset.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "test_config = {\n",
+ " \"validmind.model_validation.sklearn.ClassifierPerformance:in_sample\": {\n",
+ " \"inputs\": {\n",
+ " \"dataset\": vm_train_ds,\n",
+ " \"model\": vm_model,\n",
+ " },\n",
+ " }\n",
+ "}\n",
+ "results = vm.run_documentation_tests(\n",
+ " section=[\"model_development\"],\n",
+ " inputs={\n",
+ " \"dataset\": vm_test_ds, # Any test that requires a single dataset will use vm_test_ds\n",
+ " \"model\": vm_model,\n",
+ " \"datasets\": (\n",
+ " vm_train_ds,\n",
+ " vm_test_ds,\n",
+ " ), # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds\n",
+ " },\n",
+ " config=test_config,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## In summary\n",
+ "\n",
+ "In this second notebook, you learned how to:\n",
+ "\n",
+ "- [ ] Import a sample dataset\n",
+ "- [ ] Identify which tests you might want to run with ValidMind\n",
+ "- [ ] Initialize ValidMind datasets\n",
+ "- [ ] Run individual tests\n",
+ "- [ ] Utilize the output from tests you've run\n",
+ "- [ ] Log test results from sets of or individual tests as evidence to the ValidMind Platform\n",
+ "- [ ] Add supplementary individual test results to your documentation\n",
+ "- [ ] Assign model predictions to your ValidMind datasets\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Next steps"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Integrate custom tests\n",
+ "\n",
+ "Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your model documentation, let's learn how to incorporate your own custom tests into ValidMind: **[103 Integrate custom tests](103-integrate_custom_tests.ipynb)**"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "ValidMind Library",
+ "language": "python",
+ "name": "validmind"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.10.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/site/notebooks/EXECUTED/model_development/103-integrate_custom_tests.ipynb b/site/notebooks/EXECUTED/model_development/103-integrate_custom_tests.ipynb
new file mode 100644
index 0000000000..bcde58c62c
--- /dev/null
+++ b/site/notebooks/EXECUTED/model_development/103-integrate_custom_tests.ipynb
@@ -0,0 +1,986 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# ValidMind for model development — 103 Integrate custom tests\n",
+ "\n",
+ "Learn how to use ValidMind for your end-to-end model documentation process with our series of four introductory notebooks. In this third notebook, supplement ValidMind tests with your own and include them as additional evidence in your documentation.\n",
+ "\n",
+ "This notebook assumes that you already have a repository of custom made tests considered critical to include in your documentation. A custom test is any function that takes a set of inputs and parameters as arguments and returns one or more outputs:\n",
+ "\n",
+ "- The function can be as simple or as complex as you need it to be — it can use external libraries, make API calls, or do anything else that you can do in Python.\n",
+ "- The only requirement is that the function signature and return values can be \"understood\" and handled by the ValidMind Library. As such, custom tests offer added flexibility by extending the default tests provided by ValidMind, enabling you to document any type of model or use case.\n",
+ "\n",
+ "**For a more in-depth introduction to custom tests,** refer to our [Implement custom tests](../../code_samples/custom_tests/implement_custom_tests.ipynb) notebook."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "::: {.content-hidden when-format=\"html\"}\n",
+ "## Contents \n",
+ "- [Prerequisites](#toc1_) \n",
+ "- [Setting up](#toc2_) \n",
+ " - [Initialize the ValidMind Library](#toc2_1_) \n",
+ " - [Import sample dataset](#toc2_2_) \n",
+ " - [Remove highly correlated features](#toc2_2_1_) \n",
+ " - [Train the model](#toc2_3_) \n",
+ " - [Initialize the ValidMind objects](#toc2_3_1_) \n",
+ " - [Assign predictions](#toc2_3_2_) \n",
+ "- [Implementing a custom inline test](#toc3_) \n",
+ " - [Create a confusion matrix plot](#toc3_1_) \n",
+ " - [Add parameters to custom tests](#toc3_2_) \n",
+ " - [Pass parameters to custom tests](#toc3_3_) \n",
+ " - [Log the confusion matrix results](#toc3_4_) \n",
+ "- [Using external test providers](#toc4_) \n",
+ " - [Create custom tests folder](#toc4_1_) \n",
+ " - [Save an inline test](#toc4_2_) \n",
+ " - [Register a local test provider](#toc4_3_) \n",
+ " - [Initialize a local test provider](#toc4_3_1_) \n",
+ " - [Run test provider tests](#toc4_3_2_) \n",
+ "- [Add test results to documentation](#toc5_) \n",
+ "- [In summary](#toc6_) \n",
+ "- [Next steps](#toc7_) \n",
+ " - [Finalize testing and documentation](#toc7_1_) \n",
+ "\n",
+ ":::\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Prerequisites\n",
+ "\n",
+ "In order to log test results or evidence to your model documentation with this notebook, you'll need to first have:\n",
+ "\n",
+ "- [ ] Registered a model within the ValidMind Platform with a predefined documentation template\n",
+ "- [ ] Installed the ValidMind Library in your local environment, allowing you to access all its features\n",
+ "- [ ] Learned how to import and initialize datasets for use with ValidMind\n",
+ "- [ ] Understood the basics of how to run and log tests with ValidMind\n",
+ "- [ ] Inserted a test-driven block for the results of your `HighPearsonCorrelation:balanced_raw_dataset` test into your model's documentation\n",
+ "\n",
+ "Need help with the above steps?\n",
+ "
\n", + "Refer to the first two notebooks in this series:\n", + "\n", + "
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Setting up\n",
+ "\n",
+ "This section should be quite familiar to you — as we performed the same actions in the previous notebook, **[102 Start the model development process](102-start_development_process.ipynb)**."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Initialize the ValidMind Library\n",
+ "\n",
+ "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
+ "\n",
+ "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
+ "\n",
+ "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n",
+ "\n",
+ "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
+ "\n",
+ "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Make sure the ValidMind Library is installed\n",
+ "\n",
+ "%pip install -q validmind\n",
+ "\n",
+ "# Load your model identifier credentials from an `.env` file\n",
+ "\n",
+ "%load_ext dotenv\n",
+ "%dotenv .env\n",
+ "\n",
+ "# Or replace with your code snippet\n",
+ "\n",
+ "import validmind as vm\n",
+ "\n",
+ "vm.init(\n",
+ " # api_host=\"...\",\n",
+ " # api_key=\"...\",\n",
+ " # api_secret=\"...\",\n",
+ " # model=\"...\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Import sample dataset\n",
+ "\n",
+ "Next, we'll import the same public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle we used in the last notebook so that we have something to work with:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from validmind.datasets.classification import customer_churn as demo_dataset\n",
+ "\n",
+ "print(\n",
+ " f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
+ ")\n",
+ "\n",
+ "raw_df = demo_dataset.load_data()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll apply a simple rebalancing technique to the dataset before continuing:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "raw_copy_df = raw_df.sample(frac=1) # Create a copy of the raw dataset\n",
+ "\n",
+ "# Create a balanced dataset with the same number of exited and not exited customers\n",
+ "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
+ "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
+ "\n",
+ "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
+ "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Remove highly correlated features\n",
+ "\n",
+ "Let's also quickly remove highly correlated features from the dataset using the output from a ValidMind test.\n",
+ "\n",
+ "As you learned previously, before we can run tests you'll need to initialize a ValidMind dataset object:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
+ "vm_balanced_raw_dataset = vm.init_dataset(\n",
+ " dataset=balanced_raw_df,\n",
+ " input_id=\"balanced_raw_dataset\",\n",
+ " target_column=\"Exited\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object\n",
+ "corr_result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
+ " params={\"max_threshold\": 0.3},\n",
+ " inputs={\"dataset\": vm_balanced_raw_dataset},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# From result object, extract table from `corr_result.tables`\n",
+ "features_df = corr_result.tables[0].data\n",
+ "features_df"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Extract list of features that failed the test\n",
+ "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
+ "high_correlation_features"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Extract feature names from the list of strings\n",
+ "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
+ "high_correlation_features"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can then re-initialize the dataset with a different `input_id` and the highly correlated features removed and re-run the test for confirmation:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Remove the highly correlated features from the dataset\n",
+ "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
+ "\n",
+ "# Re-initialize the dataset object\n",
+ "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
+ " dataset=balanced_raw_no_age_df,\n",
+ " input_id=\"raw_dataset_preprocessed\",\n",
+ " target_column=\"Exited\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Re-run the test with the reduced feature set\n",
+ "corr_result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
+ " params={\"max_threshold\": 0.3},\n",
+ " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Train the model\n",
+ "\n",
+ "We'll then use ValidMind tests to train a simple logistic regression model on our prepared dataset:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# First encode the categorical features in our dataset with the highly correlated features removed\n",
+ "balanced_raw_no_age_df = pd.get_dummies(\n",
+ " balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
+ ")\n",
+ "balanced_raw_no_age_df.head()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.linear_model import LogisticRegression\n",
+ "from sklearn.model_selection import train_test_split\n",
+ "\n",
+ "# Split the input and target variables\n",
+ "X = balanced_raw_no_age_df.drop(\"Exited\", axis=1)\n",
+ "y = balanced_raw_no_age_df[\"Exited\"]\n",
+ "X_train, X_test, y_train, y_test = train_test_split(\n",
+ " X,\n",
+ " y,\n",
+ " test_size=0.2,\n",
+ " random_state=42,\n",
+ ")\n",
+ "\n",
+ "# Logistic Regression grid params\n",
+ "log_reg_params = {\n",
+ " \"penalty\": [\"l1\", \"l2\"],\n",
+ " \"C\": [0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
+ " \"solver\": [\"liblinear\"],\n",
+ "}\n",
+ "\n",
+ "# Grid search for Logistic Regression\n",
+ "from sklearn.model_selection import GridSearchCV\n",
+ "\n",
+ "grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)\n",
+ "grid_log_reg.fit(X_train, y_train)\n",
+ "\n",
+ "# Logistic Regression best estimator\n",
+ "log_reg = grid_log_reg.best_estimator_"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Initialize the ValidMind objects\n",
+ "\n",
+ "Let's initialize the ValidMind `Dataset` and `Model` objects in preparation for assigning model predictions to each dataset:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train_df = X_train\n",
+ "train_df[\"Exited\"] = y_train\n",
+ "test_df = X_test\n",
+ "test_df[\"Exited\"] = y_test\n",
+ "\n",
+ "# Initialize the datasets into their own dataset objects\n",
+ "vm_train_ds = vm.init_dataset(\n",
+ " input_id=\"train_dataset_final\",\n",
+ " dataset=train_df,\n",
+ " target_column=\"Exited\",\n",
+ ")\n",
+ "\n",
+ "vm_test_ds = vm.init_dataset(\n",
+ " input_id=\"test_dataset_final\",\n",
+ " dataset=test_df,\n",
+ " target_column=\"Exited\",\n",
+ ")\n",
+ "\n",
+ "# Initialize a model object\n",
+ "vm_model = vm.init_model(log_reg, input_id=\"log_reg_model_v1\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Assign predictions\n",
+ "\n",
+ "Once the model is registered, we'll assign predictions to the training and test datasets:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm_train_ds.assign_predictions(model=vm_model)\n",
+ "vm_test_ds.assign_predictions(model=vm_model)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Implementing a custom inline test\n",
+ "\n",
+ "With the set up out of the way, let's implement a custom *inline test* that calculates the confusion matrix for a binary classification model.\n",
+ "\n",
+ "- An inline test refers to a test written and executed within the same environment as the code being tested — in this case, right in this Jupyter Notebook — without requiring a separate test file or framework.\n",
+ "- You'll note that the custom test function is just a regular Python function that can include and require any Python library as you see fit."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Create a confusion matrix plot\n",
+ "\n",
+ "Let's first create a confusion matrix plot using the `confusion_matrix` function from the `sklearn.metrics` module:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "import matplotlib.pyplot as plt\n",
+ "from sklearn import metrics\n",
+ "\n",
+ "# Get the predicted classes\n",
+ "y_pred = log_reg.predict(vm_test_ds.x)\n",
+ "\n",
+ "confusion_matrix = metrics.confusion_matrix(y_test, y_pred)\n",
+ "\n",
+ "cm_display = metrics.ConfusionMatrixDisplay(\n",
+ " confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
+ ")\n",
+ "cm_display.plot()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, create a [`@vm.test` wrapper](https://docs.validmind.ai/validmind/validmind.html#test) that will allow you to create a reusable test. **Note the following changes in the code below:**\n",
+ "\n",
+ "- The function `confusion_matrix` takes two arguments `dataset` and `model`. This is a `VMDataset` and `VMModel` object respectively.\n",
+ " - `VMDataset` objects allow you to access the dataset's true (target) values by accessing the `.y` attribute.\n",
+ " - `VMDataset` objects allow you to access the predictions for a given model by accessing the `.y_pred()` method.\n",
+ "- The function docstring provides a description of what the test does. This will be displayed along with the result in this notebook as well as in the ValidMind Platform.\n",
+ "- The function body calculates the confusion matrix using the `sklearn.metrics.confusion_matrix` function as we just did above.\n",
+ "- The function then returns the `ConfusionMatrixDisplay.figure_` object — this is important as the ValidMind Library expects the output of the custom test to be a plot or a table.\n",
+ "- The `@vm.test` decorator is doing the work of creating a wrapper around the function that will allow it to be run by the ValidMind Library. It also registers the test so it can be found by the ID `my_custom_tests.ConfusionMatrix`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
+ "def confusion_matrix(dataset, model):\n",
+ " \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
+ "\n",
+ " The confusion matrix is a 2x2 table that contains 4 values:\n",
+ "\n",
+ " - True Positive (TP): the number of correct positive predictions\n",
+ " - True Negative (TN): the number of correct negative predictions\n",
+ " - False Positive (FP): the number of incorrect positive predictions\n",
+ " - False Negative (FN): the number of incorrect negative predictions\n",
+ "\n",
+ " The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
+ " \"\"\"\n",
+ " y_true = dataset.y\n",
+ " y_pred = dataset.y_pred(model=model)\n",
+ "\n",
+ " confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
+ "\n",
+ " cm_display = metrics.ConfusionMatrixDisplay(\n",
+ " confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
+ " )\n",
+ " cm_display.plot()\n",
+ "\n",
+ " plt.close() # close the plot to avoid displaying it\n",
+ "\n",
+ " return cm_display.figure_ # return the figure object itself"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You can now run the newly created custom test on both the training and test datasets using the [`run_test()` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test):\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "# Training dataset\n",
+ "result = vm.tests.run_test(\n",
+ " \"my_custom_tests.ConfusionMatrix:training_dataset\",\n",
+ " inputs={\"model\": vm_model, \"dataset\": vm_train_ds},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Test dataset\n",
+ "result = vm.tests.run_test(\n",
+ " \"my_custom_tests.ConfusionMatrix:test_dataset\",\n",
+ " inputs={\"model\": vm_model, \"dataset\": vm_test_ds},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Add parameters to custom tests\n",
+ "\n",
+ "Custom tests can take parameters just like any other function. To demonstrate, let's modify the `confusion_matrix` function to take an additional parameter `normalize` that will allow you to normalize the confusion matrix:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
+ "def confusion_matrix(dataset, model, normalize=False):\n",
+ " \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
+ "\n",
+ " The confusion matrix is a 2x2 table that contains 4 values:\n",
+ "\n",
+ " - True Positive (TP): the number of correct positive predictions\n",
+ " - True Negative (TN): the number of correct negative predictions\n",
+ " - False Positive (FP): the number of incorrect positive predictions\n",
+ " - False Negative (FN): the number of incorrect negative predictions\n",
+ "\n",
+ " The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
+ " \"\"\"\n",
+ " y_true = dataset.y\n",
+ " y_pred = dataset.y_pred(model=model)\n",
+ "\n",
+ " if normalize:\n",
+ " confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize=\"all\")\n",
+ " else:\n",
+ " confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
+ "\n",
+ " cm_display = metrics.ConfusionMatrixDisplay(\n",
+ " confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
+ " )\n",
+ " cm_display.plot()\n",
+ "\n",
+ " plt.close() # close the plot to avoid displaying it\n",
+ "\n",
+ " return cm_display.figure_ # return the figure object itself"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Pass parameters to custom tests\n",
+ "\n",
+ "You can pass parameters to custom tests by providing a dictionary of parameters to the `run_test()` function.\n",
+ "\n",
+ "- The parameters will override any default parameters set in the custom test definition. Note that `dataset` and `model` are still passed as `inputs`.\n",
+ "- Since these are `VMDataset` or `VMModel` inputs, they have a special meaning.\n",
+ "- When declaring a `dataset`, `model`, `datasets` or `models` argument in a custom test function, the ValidMind Library will expect these get passed as `inputs` to `run_test()` or `run_documentation_tests()`.\n",
+ "\n",
+ "Re-running the confusion matrix with `normalize=True` looks like this:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Test dataset with normalize=True\n",
+ "result = vm.tests.run_test(\n",
+ " \"my_custom_tests.ConfusionMatrix:test_dataset_normalized\",\n",
+ " inputs={\"model\": vm_model, \"dataset\": vm_test_ds},\n",
+ " params={\"normalize\": True},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Log the confusion matrix results\n",
+ "\n",
+ "As we learned in **[102 Start the model development process](102-start_development_process.ipynb)** under **Documenting results** > **Run and log an individual tests**, you can log any result to the ValidMind Platform with the [`.log()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) of the result object, allowing you to then add the result to the documentation.\n",
+ "\n",
+ "You can now do the same for the confusion matrix results:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "result.log()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n", + "Refer to the first two notebooks in this series:\n", + "\n", + "
- \n",
+ "
- 101 Set up ValidMind \n", + "
- 102 Start the model development process \n", + "
Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID. \n",
+ "
\n", + "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Using external test providers\n",
+ "\n",
+ "Creating inline custom tests with a function is a great way to customize your model documentation. However, sometimes you may want to reuse the same set of tests across multiple models and share them with others in your organization. In this case, you can create an external custom *test provider* that will allow you to load custom tests from a local folder or a Git repository.\n",
+ "\n",
+ "In this section you will learn how to declare a local filesystem test provider that allows loading tests from a local folder following these high level steps:\n",
+ "\n",
+ "1. Create a folder of custom tests from existing inline tests (tests that exist in your active Jupyter Notebook)\n",
+ "2. Save an inline test to a file\n",
+ "3. Define and register a [`LocalTestProvider`](https://docs.validmind.ai/validmind/validmind/tests.html#LocalTestProvider) that points to that folder\n",
+ "4. Run test provider tests\n",
+ "5. Add the test results to your documentation\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Create custom tests folder\n",
+ "\n",
+ "Let's start by creating a new folder that will contain reusable custom tests from your existing inline tests.\n",
+ "\n",
+ "The following code snippet will create a new `my_tests` directory in the current working directory if it doesn't exist:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "tests_folder = \"my_tests\"\n",
+ "\n",
+ "import os\n",
+ "\n",
+ "# create tests folder\n",
+ "os.makedirs(tests_folder, exist_ok=True)\n",
+ "\n",
+ "# remove existing tests\n",
+ "for f in os.listdir(tests_folder):\n",
+ " # remove files and pycache\n",
+ " if f.endswith(\".py\") or f == \"__pycache__\":\n",
+ " os.system(f\"rm -rf {tests_folder}/{f}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "After running the command above, confirm that the new `my_tests` directory was created successfully:\n",
+ "\n",
+ "\n", + "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.
![\"Screenshot](\"my_tests_directory.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Save an inline test\n",
+ "\n",
+ "The `@vm.test` decorator we used in **Implementing a custom inline test** above to register one-off custom tests also includes a convenience method on the function object that allows you to simply call `Want to learn more about test providers?\n",
+ "
\n", + "An extended introduction to test providers can be found in: Integrate external test providers
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Initialize a local test provider\n",
+ "\n",
+ "For most use cases, using a `LocalTestProvider` that allows you to load custom tests from a designated directory should be sufficient.\n",
+ "\n",
+ "**The most important attribute for a test provider is its `namespace`.** This is a string that will be used to prefix test IDs in model documentation. This allows you to have multiple test providers with tests that can even share the same ID, but are distinguished by their namespace.\n",
+ "\n",
+ "Let's go ahead and load the custom tests from our `my_tests` directory:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from validmind.tests import LocalTestProvider\n",
+ "\n",
+ "# initialize the test provider with the tests folder we created earlier\n",
+ "my_test_provider = LocalTestProvider(tests_folder)\n",
+ "\n",
+ "vm.tests.register_test_provider(\n",
+ " namespace=\"my_test_provider\",\n",
+ " test_provider=my_test_provider,\n",
+ ")\n",
+ "# `my_test_provider.load_test()` will be called for any test ID that starts with `my_test_provider`\n",
+ "# e.g. `my_test_provider.ConfusionMatrix` will look for a function named `ConfusionMatrix` in `my_tests/ConfusionMatrix.py` file"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Run test provider tests\n",
+ "\n",
+ "Now that we've set up the test provider, we can run any test that's located in the tests folder by using the `run_test()` method as with any other test:\n",
+ "\n",
+ "- For tests that reside in a test provider directory, the test ID will be the `namespace` specified when registering the provider, followed by the path to the test file relative to the tests folder.\n",
+ "- For example, the Confusion Matrix test we created earlier will have the test ID `my_test_provider.ConfusionMatrix`. You could organize the tests in subfolders, say `classification` and `regression`, and the test ID for the Confusion Matrix test would then be `my_test_provider.classification.ConfusionMatrix`.\n",
+ "\n",
+ "Let's go ahead and re-run the confusion matrix test by using the test ID `my_test_provider.ConfusionMatrix`. This should load the test from the test provider and run it as before.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "result = vm.tests.run_test(\n",
+ " \"my_test_provider.ConfusionMatrix\",\n",
+ " inputs={\"model\": vm_model, \"dataset\": vm_test_ds},\n",
+ " params={\"normalize\": True},\n",
+ ")\n",
+ "\n",
+ "result.log()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n", + "An extended introduction to test providers can be found in: Integrate external test providers
Again, note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID. \n",
+ "
\n", + "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Add test results to documentation\n",
+ "\n",
+ "With our custom tests run and results logged to the ValidMind Platform, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation ([Need more help?](https://docs.validmind.ai/developer/model-documentation/work-with-test-results.html)):\n",
+ "\n",
+ "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n",
+ "\n",
+ "2. In the left sidebar that appears for your model, click **Documentation**.\n",
+ "\n",
+ "3. Locate the Data Preparation section and click on **3.2 Model Evaluation** to expand that section.\n",
+ "\n",
+ "4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a **+** button appears, indicating that you can insert a new block.\n",
+ "\n",
+ " \n", + "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.
\n",
+ " \n", + "\n", + "5. Click **+** and then select **Test-Driven Block**:\n", + "\n", + " - In the search bar, type in `ConfusionMatrix`.\n", + " - Select the custom `ConfusionMatrix` tests you logged above:\n", + "\n", + "
![\"Screenshot](\"selecting-confusion-matrix-test.png\")
\n",
+ " \n", + "\n", + "6. Finally, click **Insert 2 Test Results to Document** to add the test results to the documentation.\n", + "\n", + " Confirm that the two individual results for the confusion matrix tests have been correctly inserted into section **3.2 Model Evaluation** of the documentation." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "## In summary\n", + "\n", + "In this third notebook, you learned how to:\n", + "\n", + "- [ ] Implement a custom inline test\n", + "- [ ] Run and log your custom inline tests\n", + "- [ ] Use external custom test providers\n", + "- [ ] Run and log tests from your custom test providers" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Next steps" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Finalize testing and documentation\n", + "\n", + "Now that you're proficient at using the ValidMind Library to run and log tests, let's put the last pieces in place to prepare our fully documented sample model for review: **[104 Finalize testing and documentation](104-finalize_testing_documentation.ipynb)**" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "ValidMind Library", + "language": "python", + "name": "validmind" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.13" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/site/notebooks/EXECUTED/model_development/104-finalize_testing_documentation.ipynb b/site/notebooks/EXECUTED/model_development/104-finalize_testing_documentation.ipynb new file mode 100644 index 0000000000..fd17733fcd --- /dev/null +++ b/site/notebooks/EXECUTED/model_development/104-finalize_testing_documentation.ipynb @@ -0,0 +1,966 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# ValidMind for model development — 104 Finalize testing and documentation\n", + "\n", + "Learn how to use ValidMind for your end-to-end model documentation process with our introductory notebook series. In this last notebook, finalize the testing and documentation of your model and have a fully documented sample model ready for review.\n", + "\n", + "We'll first use [`run_documentation_tests()`](https://docs.validmind.ai/validmind/validmind.html#run_documentation_tests) previously covered in **[102 Start the model development process](102-start_development_process.ipynb)** to ensure that your custom test results generated in **[103 Integrate custom tests](103-integrate_custom_tests.ipynb)** are included in your documentation. Then, we'll view and update the configuration for the entire model documentation template to suit your needs.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "::: {.content-hidden when-format=\"html\"}\n", + "## Contents \n", + "- [Prerequisites](#toc1_) \n", + "- [Setting up](#toc2_) \n", + " - [Initialize the ValidMind Library](#toc2_1_) \n", + " - [Import sample dataset](#toc2_2_) \n", + " - [Remove highly correlated features](#toc2_2_1_) \n", + " - [Train the model](#toc2_3_) \n", + " - [Initialize the ValidMind objects](#toc2_3_1_) \n", + " - [Assign predictions](#toc2_3_2_) \n", + " - [Add custom tests](#toc2_4_) \n", + " - [Implement custom inline test](#toc2_4_1_) \n", + " - [Add a local test provider](#toc2_4_2_) \n", + "- [Reconnect to ValidMind](#toc3_) \n", + "- [Include custom test results](#toc4_) \n", + "- [Documentation template configuration](#toc5_) \n", + " - [Update the config](#toc5_1_) \n", + "- [In summary](#toc6_) \n", + "- [Next steps](#toc7_) \n", + " - [Work with your model documentation](#toc7_1_) \n", + " - [Learn more](#toc7_2_) \n", + " - [Use cases](#toc7_2_1_) \n", + " - [More how-to guides and code samples](#toc7_2_2_) \n", + " - [Discover more learning resources](#toc7_2_3_) \n", + "\n", + ":::\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Prerequisites\n", + "\n", + "In order to finalize the testing and documentation for your sample model, you'll need to first have:\n", + "\n", + "- [ ] Registered a model within the ValidMind Platform with a predefined documentation template\n", + "- [ ] Installed the ValidMind Library in your local environment, allowing you to access all its features\n", + "- [ ] Learned how to import and initialize datasets for use with ValidMind\n", + "- [ ] Learned how to run and log default and custom tests with ValidMind, including from external test providers\n", + "- [ ] Inserted test-driven blocks for the results of the following tests into your model's documentation:\n", + " - [ ] `HighPearsonCorrelation:balanced_raw_dataset`\n", + " - [ ] `my_test_provider.ConfusionMatrix`\n", + " - [ ] `my_custom_tests.ConfusionMatrix:test_dataset_normalized`\n", + "\n", + "
Need help with the above steps?\n",
+ "
\n", + "Refer to the first three notebooks in this series:\n", + "\n", + "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Setting up\n",
+ "\n",
+ "This section should be very familiar to you now — as we performed the same actions in the previous two notebooks in this series."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Initialize the ValidMind Library\n",
+ "\n",
+ "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
+ "\n",
+ "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
+ "\n",
+ "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n",
+ "\n",
+ "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
+ "\n",
+ "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Make sure the ValidMind Library is installed\n",
+ "\n",
+ "%pip install -q validmind\n",
+ "\n",
+ "# Load your model identifier credentials from an `.env` file\n",
+ "\n",
+ "%load_ext dotenv\n",
+ "%dotenv .env\n",
+ "\n",
+ "# Or replace with your code snippet\n",
+ "\n",
+ "import validmind as vm\n",
+ "\n",
+ "vm.init(\n",
+ " # api_host=\"...\",\n",
+ " # api_key=\"...\",\n",
+ " # api_secret=\"...\",\n",
+ " # model=\"...\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Import sample dataset\n",
+ "\n",
+ "Next, we'll import the same public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle we used in the last notebook so that we have something to work with:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from validmind.datasets.classification import customer_churn as demo_dataset\n",
+ "\n",
+ "print(\n",
+ " f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
+ ")\n",
+ "\n",
+ "raw_df = demo_dataset.load_data()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll apply a simple rebalancing technique to the dataset before continuing:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "raw_copy_df = raw_df.sample(frac=1) # Create a copy of the raw dataset\n",
+ "\n",
+ "# Create a balanced dataset with the same number of exited and not exited customers\n",
+ "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
+ "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
+ "\n",
+ "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
+ "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Remove highly correlated features\n",
+ "\n",
+ "Let's also quickly remove highly correlated features from the dataset using the output from a ValidMind test.\n",
+ "\n",
+ "As you learned previously, before we can run tests you'll need to initialize a ValidMind dataset object:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
+ "vm_balanced_raw_dataset = vm.init_dataset(\n",
+ " dataset=balanced_raw_df,\n",
+ " input_id=\"balanced_raw_dataset\",\n",
+ " target_column=\"Exited\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object\n",
+ "corr_result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
+ " params={\"max_threshold\": 0.3},\n",
+ " inputs={\"dataset\": vm_balanced_raw_dataset},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# From result object, extract table from `corr_result.tables`\n",
+ "features_df = corr_result.tables[0].data\n",
+ "features_df"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Extract list of features that failed the test\n",
+ "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
+ "high_correlation_features"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Extract feature names from the list of strings\n",
+ "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
+ "high_correlation_features"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can then re-initialize the dataset with a different `input_id` and the highly correlated features removed and re-run the test for confirmation:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Remove the highly correlated features from the dataset\n",
+ "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
+ "\n",
+ "# Re-initialize the dataset object\n",
+ "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
+ " dataset=balanced_raw_no_age_df,\n",
+ " input_id=\"raw_dataset_preprocessed\",\n",
+ " target_column=\"Exited\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Re-run the test with the reduced feature set\n",
+ "corr_result = vm.tests.run_test(\n",
+ " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
+ " params={\"max_threshold\": 0.3},\n",
+ " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Train the model\n",
+ "\n",
+ "We'll then use ValidMind tests to train a simple logistic regression model on our prepared dataset:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# First encode the categorical features in our dataset with the highly correlated features removed\n",
+ "balanced_raw_no_age_df = pd.get_dummies(\n",
+ " balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
+ ")\n",
+ "balanced_raw_no_age_df.head()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.linear_model import LogisticRegression\n",
+ "from sklearn.model_selection import train_test_split\n",
+ "\n",
+ "# Split the input and target variables\n",
+ "X = balanced_raw_no_age_df.drop(\"Exited\", axis=1)\n",
+ "y = balanced_raw_no_age_df[\"Exited\"]\n",
+ "X_train, X_test, y_train, y_test = train_test_split(\n",
+ " X,\n",
+ " y,\n",
+ " test_size=0.2,\n",
+ " random_state=42,\n",
+ ")\n",
+ "\n",
+ "# Logistic Regression grid params\n",
+ "log_reg_params = {\n",
+ " \"penalty\": [\"l1\", \"l2\"],\n",
+ " \"C\": [0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
+ " \"solver\": [\"liblinear\"],\n",
+ "}\n",
+ "\n",
+ "# Grid search for Logistic Regression\n",
+ "from sklearn.model_selection import GridSearchCV\n",
+ "\n",
+ "grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)\n",
+ "grid_log_reg.fit(X_train, y_train)\n",
+ "\n",
+ "# Logistic Regression best estimator\n",
+ "log_reg = grid_log_reg.best_estimator_"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Initialize the ValidMind objects\n",
+ "\n",
+ "Let's initialize the ValidMind `Dataset` and `Model` objects in preparation for assigning model predictions to each dataset:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train_df = X_train\n",
+ "train_df[\"Exited\"] = y_train\n",
+ "test_df = X_test\n",
+ "test_df[\"Exited\"] = y_test\n",
+ "\n",
+ "# Initialize the datasets into their own dataset objects\n",
+ "vm_train_ds = vm.init_dataset(\n",
+ " input_id=\"train_dataset_final\",\n",
+ " dataset=train_df,\n",
+ " target_column=\"Exited\",\n",
+ ")\n",
+ "\n",
+ "vm_test_ds = vm.init_dataset(\n",
+ " input_id=\"test_dataset_final\",\n",
+ " dataset=test_df,\n",
+ " target_column=\"Exited\",\n",
+ ")\n",
+ "\n",
+ "# Initialize a model object\n",
+ "vm_model = vm.init_model(log_reg, input_id=\"log_reg_model_v1\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Assign predictions\n",
+ "\n",
+ "Once the model is registered, we'll assign predictions to the training and test datasets:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm_train_ds.assign_predictions(model=vm_model)\n",
+ "vm_test_ds.assign_predictions(model=vm_model)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Add custom tests\n",
+ "\n",
+ "We'll also add the same custom tests we implemented in the previous notebook so that this session has access to the same custom inline test and local test provider."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Implement custom inline test\n",
+ "\n",
+ "Let's set up a custom inline test that calculates the confusion matrix for a binary classification model:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# First create a confusion matrix plot\n",
+ "import matplotlib.pyplot as plt\n",
+ "from sklearn import metrics\n",
+ "\n",
+ "# Get the predicted classes\n",
+ "y_pred = log_reg.predict(vm_test_ds.x)\n",
+ "\n",
+ "confusion_matrix = metrics.confusion_matrix(y_test, y_pred)\n",
+ "\n",
+ "cm_display = metrics.ConfusionMatrixDisplay(\n",
+ " confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
+ ")\n",
+ "cm_display.plot()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create the reusable ConfusionMatrix inline test with normalized matrix\n",
+ "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
+ "def confusion_matrix(dataset, model, normalize=False):\n",
+ " \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
+ "\n",
+ " The confusion matrix is a 2x2 table that contains 4 values:\n",
+ "\n",
+ " - True Positive (TP): the number of correct positive predictions\n",
+ " - True Negative (TN): the number of correct negative predictions\n",
+ " - False Positive (FP): the number of incorrect positive predictions\n",
+ " - False Negative (FN): the number of incorrect negative predictions\n",
+ "\n",
+ " The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
+ " \"\"\"\n",
+ " y_true = dataset.y\n",
+ " y_pred = dataset.y_pred(model=model)\n",
+ "\n",
+ " if normalize:\n",
+ " confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize=\"all\")\n",
+ " else:\n",
+ " confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
+ "\n",
+ " cm_display = metrics.ConfusionMatrixDisplay(\n",
+ " confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
+ " )\n",
+ " cm_display.plot()\n",
+ "\n",
+ " plt.close() # close the plot to avoid displaying it\n",
+ "\n",
+ " return cm_display.figure_ # return the figure object itself"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Test dataset with normalize=True\n",
+ "result = vm.tests.run_test(\n",
+ " \"my_custom_tests.ConfusionMatrix:test_dataset_normalized\",\n",
+ " inputs={\"model\": vm_model, \"dataset\": vm_test_ds},\n",
+ " params={\"normalize\": True},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Add a local test provider\n",
+ "\n",
+ "Finally, let's save our custom inline test to our local test provider:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create custom tests folder\n",
+ "tests_folder = \"my_tests\"\n",
+ "\n",
+ "import os\n",
+ "\n",
+ "# create tests folder\n",
+ "os.makedirs(tests_folder, exist_ok=True)\n",
+ "\n",
+ "# remove existing tests\n",
+ "for f in os.listdir(tests_folder):\n",
+ " # remove files and pycache\n",
+ " if f.endswith(\".py\") or f == \"__pycache__\":\n",
+ " os.system(f\"rm -rf {tests_folder}/{f}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Save custom inline test to custom tests folder\n",
+ "confusion_matrix.save(\n",
+ " tests_folder,\n",
+ " imports=[\"import matplotlib.pyplot as plt\", \"from sklearn import metrics\"],\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Register local test provider\n",
+ "from validmind.tests import LocalTestProvider\n",
+ "\n",
+ "# initialize the test provider with the tests folder we created earlier\n",
+ "my_test_provider = LocalTestProvider(tests_folder)\n",
+ "\n",
+ "vm.tests.register_test_provider(\n",
+ " namespace=\"my_test_provider\",\n",
+ " test_provider=my_test_provider,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Reconnect to ValidMind\n",
+ "\n",
+ "After you insert test-driven blocks into your model documentation, changes should persist and become available every time you call [`vm.preview_template()`](https://docs.validmind.ai/validmind/validmind.html#preview_template).\n",
+ "\n",
+ "However, you'll need to reload the connection to the ValidMind Platform if you have added test-driven blocks when the connection was already established using [`reload()`](https://docs.validmind.ai/validmind/validmind.html#reload):"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm.reload()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, when you run `preview_template()` again, the three test-driven blocks you added to your documentation in the last two notebooks in should show up in the template in sections **2.3 Correlations and Interactions** and **3.2 Model Evaluation**:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vm.preview_template()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Include custom test results\n",
+ "\n",
+ "Since your custom test IDs are now part of your documentation template, you can now run tests for an entire section and all additional custom tests should be loaded without any issues.\n",
+ "\n",
+ "Let's run all tests in the Model Evaluation section of the documentation. Note that we have been running the sample custom confusion matrix with `normalize=True` to demonstrate the ability to provide custom parameters.\n",
+ "\n",
+ "In the **Run the model evaluation tests** section of **[102 Start the model development process](102-start_development_process.ipynb)**, you learned how to assign inputs to individual tests with [`run_documentation_tests()`](https://docs.validmind.ai/validmind/validmind.html#run_documentation_tests). Assigning parameters is similar, you only need to provide assign a `params` dictionary to a given test ID, `my_test_provider.ConfusionMatrix` in this case.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "test_config = {\n",
+ " \"validmind.model_validation.sklearn.ClassifierPerformance:in_sample\": {\n",
+ " \"inputs\": {\n",
+ " \"dataset\": vm_train_ds,\n",
+ " \"model\": vm_model,\n",
+ " },\n",
+ " },\n",
+ " \"my_test_provider.ConfusionMatrix\": {\n",
+ " \"params\": {\"normalize\": True},\n",
+ " \"inputs\": {\"dataset\": vm_test_ds, \"model\": vm_model},\n",
+ " },\n",
+ "}\n",
+ "results = vm.run_documentation_tests(\n",
+ " section=[\"model_evaluation\"],\n",
+ " inputs={\n",
+ " \"dataset\": vm_test_ds, # Any test that requires a single dataset will use vm_test_ds\n",
+ " \"model\": vm_model,\n",
+ " \"datasets\": (\n",
+ " vm_train_ds,\n",
+ " vm_test_ds,\n",
+ " ), # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds\n",
+ " },\n",
+ " config=test_config,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Documentation template configuration\n",
+ "\n",
+ "Let's call the utility function [`vm.get_test_suite().get_default_config()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestSuite.get_default_config) which will return the **default configuration for the entire documentation template as a dictionary:**\n",
+ "\n",
+ "- This configuration will contain all the test IDs and their default parameters.\n",
+ "- You can then modify this configuration as needed and pass it to `run_documentation_tests()` to run all tests in the documentation template if needed.\n",
+ "- You still have the option to continue running tests for one section at a time; `get_default_config()` simply provides a useful reference for providing default parameters to every test."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "\n",
+ "model_test_suite = vm.get_test_suite()\n",
+ "config = model_test_suite.get_default_config()\n",
+ "print(\"Suite Config: \\n\", json.dumps(config, indent=2))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ "### Update the config\n",
+ "\n",
+ "The default config does not assign any inputs to a test, but you can assign inputs to individual tests as needed depending on the datasets and models you want to pass to individual tests.\n",
+ "\n",
+ "For this particular documentation template (binary classification), the ValidMind Library provides a sample configuration that can be used to populate the entire model documentation using the following inputs as placeholders:\n",
+ "\n",
+ "- A **`raw_dataset`** raw dataset\n",
+ "- A **`train_dataset`** training dataset\n",
+ "- A **`test_dataset`** test dataset\n",
+ "- A trained **`model`** instance\n",
+ "\n",
+ "As part of updating the `config` you will need to ensure the correct `input_id`s are used in the final config passed to `run_documentation_tests()`.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "metadata": {}
+ },
+ "outputs": [],
+ "source": [
+ "from validmind.datasets.classification import customer_churn\n",
+ "from validmind.utils import preview_test_config\n",
+ "\n",
+ "test_config = customer_churn.get_demo_test_config()\n",
+ "preview_test_config(test_config)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Using this sample configuration, let's finish populating model documentation by running all tests for the Model Development section of the documentation.\n",
+ "\n",
+ "Recall that the training and test datasets in our exercise have the following `input_id` values:\n",
+ "\n",
+ "- **`train_dataset_final`** for the training dataset\n",
+ "- **`test_dataset_final`** for the test dataset\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "config = {\n",
+ " \"validmind.model_validation.ModelMetadata\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\"},\n",
+ " },\n",
+ " \"validmind.data_validation.DatasetSplit\": {\n",
+ " \"inputs\": {\"datasets\": [\"train_dataset_final\", \"test_dataset_final\"]},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.PopulationStabilityIndex\": {\n",
+ " \"inputs\": {\n",
+ " \"model\": \"log_reg_model_v1\",\n",
+ " \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
+ " },\n",
+ " \"params\": {\"num_bins\": 10, \"mode\": \"fixed\"},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.ConfusionMatrix\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
+ " },\n",
+ " \"my_test_provider.ConfusionMatrix\": {\n",
+ " \"inputs\": {\"dataset\": \"test_dataset_final\", \"model\": \"log_reg_model_v1\"},\n",
+ " },\n",
+ " \"my_custom_tests.ConfusionMatrix:test_dataset_normalized\": {\n",
+ " \"inputs\": {\"dataset\": \"test_dataset_final\", \"model\": \"log_reg_model_v1\"},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.ClassifierPerformance:in_sample\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"train_dataset_final\"}\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"}\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.PrecisionRecallCurve\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.ROCCurve\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.TrainingTestDegradation\": {\n",
+ " \"inputs\": {\n",
+ " \"model\": \"log_reg_model_v1\",\n",
+ " \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
+ " },\n",
+ " \"params\": {\n",
+ " \"metrics\": [\"accuracy\", \"precision\", \"recall\", \"f1\"],\n",
+ " \"max_threshold\": 0.1,\n",
+ " },\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.MinimumAccuracy\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
+ " \"params\": {\"min_threshold\": 0.7},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.MinimumF1Score\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
+ " \"params\": {\"min_threshold\": 0.5},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.MinimumROCAUCScore\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
+ " \"params\": {\"min_threshold\": 0.5},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.PermutationFeatureImportance\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.SHAPGlobalImportance\": {\n",
+ " \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
+ " \"params\": {\"kernel_explainer_samples\": 10},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.WeakspotsDiagnosis\": {\n",
+ " \"inputs\": {\n",
+ " \"model\": \"log_reg_model_v1\",\n",
+ " \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
+ " },\n",
+ " \"params\": {\n",
+ " \"thresholds\": {\"accuracy\": 0.75, \"precision\": 0.5, \"recall\": 0.5, \"f1\": 0.7}\n",
+ " },\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.OverfitDiagnosis\": {\n",
+ " \"inputs\": {\n",
+ " \"model\": \"log_reg_model_v1\",\n",
+ " \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
+ " },\n",
+ " \"params\": {\"cut_off_percentage\": 4},\n",
+ " },\n",
+ " \"validmind.model_validation.sklearn.RobustnessDiagnosis\": {\n",
+ " \"inputs\": {\n",
+ " \"model\": \"log_reg_model_v1\",\n",
+ " \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
+ " },\n",
+ " \"params\": {\n",
+ " \"scaling_factor_std_dev_list\": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5],\n",
+ " \"accuracy_decay_threshold\": 4,\n",
+ " },\n",
+ " },\n",
+ "}\n",
+ "\n",
+ "\n",
+ "full_suite = vm.run_documentation_tests(\n",
+ " section=\"model_development\",\n",
+ " config=config,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## In summary\n",
+ "\n",
+ "In this final notebook, you learned how to:\n",
+ "\n",
+ "- [ ] Refresh the connection from the ValidMind Library to the ValidMind Platform after you've inserted test-driven blocks to your documentation\n",
+ "- [ ] Include custom test results in your model documentation\n",
+ "- [ ] View and configure the configuration for your model documentation template\n",
+ "\n",
+ "With our ValidMind for model development series of notebooks, you learned how to document a model end-to-end with the ValidMind Library by running through some common scenarios in a typical model development setting:\n",
+ "\n",
+ "- Running out-of-the-box tests\n",
+ "- Documenting your model by adding evidence to model documentation\n",
+ "- Extending the capabilities of the ValidMind Library by implementing custom tests\n",
+ "- Ensuring that the documentation is complete by running all tests in the documentation template"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "## Next steps"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Work with your model documentation\n",
+ "\n",
+ "Now that you've logged all your test results and generated a draft for your model documentation, head to the ValidMind Platform to make qualitative edits, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. **Learn more:** [Working with model documentation](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "### Learn more\n",
+ "\n",
+ "Now that you're familiar with the basics, you can explore the following notebooks to get a deeper understanding on how the ValidMind Library allows you generate model documentation for any use case:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Use cases\n",
+ "\n",
+ "- [Application scorecard demo](../../code_samples/credit_risk/application_scorecard_demo.ipynb)\n",
+ "- [Linear regression documentation demo](../../code_samples/regression/quickstart_regression_full_suite.ipynb)\n",
+ "- [LLM model documentation demo](../../code_samples/nlp_and_llm/foundation_models_integration_demo.ipynb)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### More how-to guides and code samples\n",
+ "\n",
+ "- [Explore available tests in detail](../../how_to/explore_tests.ipynb)\n",
+ "- [In-depth guide for implementing custom tests](../../code_samples/custom_tests/implement_custom_tests.ipynb)\n",
+ "- [In-depth guide to external test providers](../../code_samples/custom_tests/integrate_external_test_providers.ipynb)\n",
+ "- [Configuring dataset features](../../how_to/configure_dataset_features.ipynb)\n",
+ "- [Introduction to unit and composite metrics](../../how_to/run_unit_metrics.ipynb)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "#### Discover more learning resources\n",
+ "\n",
+ "All notebook samples can be found in the following directories of the ValidMind Library GitHub repository:\n",
+ "\n",
+ "- [Code samples](https://github.com/validmind/validmind-library/tree/main/notebooks/code_samples)\n",
+ "- [How-to guides](https://github.com/validmind/validmind-library/tree/main/notebooks/how_to)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "ValidMind Library",
+ "language": "python",
+ "name": "validmind"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.10.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/site/notebooks/EXECUTED/model_development/add-content-block.gif b/site/notebooks/EXECUTED/model_development/add-content-block.gif
new file mode 100644
index 0000000000000000000000000000000000000000..daab9d44d8e85afe0de4c59d7dd03dc94bc1bd98
GIT binary patch
literal 114727
zcmb@tWl$V%*X}#$3\n", + "Refer to the first three notebooks in this series:\n", + "\n", + "
- \n",
+ "
- 101 Set up ValidMind \n", + "
- 102 Start the model development process \n", + "
- 103 Integrate custom tests \n", + "