diff --git a/notebooks/how_to/configure_pii_detection.ipynb b/notebooks/how_to/configure_pii_detection.ipynb deleted file mode 100644 index e9461279b..000000000 --- a/notebooks/how_to/configure_pii_detection.ipynb +++ /dev/null @@ -1,189 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# PII Detection Modes with a Custom Test\n", - "\n", - "This notebook shows how to initialize ValidMind, implement a custom test that emits PII, and observe behavior differences under each `VALIDMIND_PII_DETECTION` mode when running the test with `validmind.tests.run_test`.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Prerequisites\n", - "\n", - "- `validmind` installed with PII extras:\n", - "\n", - "```bash\n", - "%pip install -q validmind[pii-detection]\n", - "```\n", - "\n", - "- A ValidMind model registered. We'll initialize the library using your model snippet.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%pip install -q \"validmind[pii-detection]\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Initialize ValidMind\n", - "\n", - "Initialize using your model code snippet or a `.env` file, as shown in other quickstarts.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Load your model identifier credentials from an `.env` file\n", - "%load_ext dotenv\n", - "%dotenv .env\n", - "\n", - "# Or initialize with your code snippet\n", - "import validmind as vm\n", - "\n", - "vm.init(\n", - " # api_host=\"...\",\n", - " # api_key=\"...\",\n", - " # api_secret=\"...\",\n", - " # model=\"...\",\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Create a custom test that emits PII\n", - "\n", - "We'll create a custom test that returns:\n", - "- A description string containing PII (name, email, phone)\n", - "- A small table containing PII in columns\n", - "\n", - "This mirrors the structure used in other custom test notebooks and will exercise both table and description PII detection paths.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import pandas as pd\n", - "\n", - "from validmind import test\n", - "\n", - "@test(\"my_pii_demo.PIIEmittingTest\")\n", - "def pii_emitting_test():\n", - " \"\"\"A demo test that returns PII\"\"\"\n", - " return pd.DataFrame(\n", - " {\n", - " \"name\": [\"Jane Smith\", \"John Doe\", \"Alice Johnson\"],\n", - " \"email\": [\n", - " \"jane.smith@bank.example\",\n", - " \"john.doe@company.example\",\n", - " \"alice.johnson@service.example\",\n", - " ],\n", - " \"phone\": [\"(212) 555-9876\", \"(415) 555-1234\", \"(646) 555-5678\"],\n", - " }\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Run the test under different PII detection modes\n", - "\n", - "We'll switch `VALIDMIND_PII_DETECTION` across modes and run the same test with `validmind.tests.run_test`. We catch exceptions to observe blocking behavior.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "from validmind.tests import run_test\n", - "\n", - "MODES = [\"disabled\", \"test_results\", \"test_descriptions\", \"all\"]\n", - "\n", - "for mode in MODES:\n", - " print(\"\\n=== Mode:\", mode, \"===\")\n", - " os.environ[\"VALIDMIND_PII_DETECTION\"] = mode\n", - " try:\n", - " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - "\n", - " # check if the description was generated\n", - " if not result._was_description_generated:\n", - " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " else:\n", - " print(\"Description was generated by LLM\")\n", - "\n", - " # Try logging (this triggers PII checks before upload)\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - " except Exception as e:\n", - " print(\"Blocked: Test Result was not logged due to PII\")\n", - " # print(e)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Expected behavior by mode\n", - "\n", - "- disabled: No PII checks.\n", - "- test_results: Description is generated but result is not logged.\n", - "- test_descriptions: Description generation is blocked but result is logged.\n", - "- all: Description generation and logging are both blocked.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Notes\n", - "\n", - "- If you see warnings that Presidio is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", - "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": ".venv", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.11" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/notebooks/how_to/enable_pii_detection.ipynb b/notebooks/how_to/enable_pii_detection.ipynb new file mode 100644 index 000000000..a7e123286 --- /dev/null +++ b/notebooks/how_to/enable_pii_detection.ipynb @@ -0,0 +1,615 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "f5b060b7", + "metadata": {}, + "source": [ + "# Enable PII detection in tests" + ] + }, + { + "cell_type": "markdown", + "id": "403b35b1", + "metadata": {}, + "source": [ + "Learn how to enable and configure Personally Identifiable Information (PII) detection when running tests with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform." + ] + }, + { + "cell_type": "markdown", + "id": "1f0a64f3", + "metadata": {}, + "source": [ + "::: {.content-hidden when-format=\"html\"}\n", + "## Contents \n", + "- [About ValidMind](#toc1__) \n", + " - [Before you begin](#toc1_1__) \n", + " - [New to ValidMind?](#toc1_2__) \n", + " - [Key concepts](#toc1_3__) \n", + "- [Setting up](#toc2__) \n", + " - [Install the ValidMind Library with PII detection](#toc2_1__) \n", + " - [Initialize the ValidMind Library](#toc2_2__) \n", + " - [Get your code snippet](#toc2_2_1__) \n", + "- [Using PII detection](#toc3__) \n", + " - [Create a custom test that outputs PII](#toc3_1__) \n", + " - [Run test under different PII detection modes](#toc3_2__) \n", + " - [disabled](#toc3_2_1__) \n", + " - [test_results](#toc3_2_2__) \n", + " - [test_descriptions](#toc3_2_3__) \n", + " - [all](#toc3_2_4__) \n", + " - [Override detection](#toc3_3__) \n", + " - [Override test result logging](#toc3_3_1__) \n", + " - [Override test descriptions and test result logging](#toc3_3_2__) \n", + " - [Review logged test results](#toc3_4__) \n", + "- [Troubleshooting](#toc4__) \n", + "- [Learn more](#toc5__) \n", + "\n", + ":::\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "68cf8398", + "metadata": {}, + "source": [ + "\n", + "\n", + "## About ValidMind\n", + "\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "\n", + "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + ] + }, + { + "cell_type": "markdown", + "id": "ba6bd554", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Before you begin\n", + "\n", + "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", + "\n", + "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)." + ] + }, + { + "cell_type": "markdown", + "id": "ac231640", + "metadata": {}, + "source": [ + "\n", + "\n", + "### New to ValidMind?\n", + "\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", + "\n", + "
For access to all features available in this notebook, you'll need access to a ValidMind account.\n", + "

\n", + "Register with ValidMind
" + ] + }, + { + "cell_type": "markdown", + "id": "75626c3d", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Key concepts\n", + "\n", + "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", + "\n", + "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "\n", + "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", + "\n", + "**Metrics**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.\n", + "\n", + "**Custom metrics**: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.\n", + "\n", + "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n", + "\n", + " - **model**: A single model that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n", + " - **dataset**: Single dataset that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n", + " - **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.\n", + " - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/run_tests_that_require_multiple_datasets.html))\n", + "\n", + "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.\n", + "\n", + "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.\n", + "\n", + "**Test suites**: Collections of tests designed to run together to automate and generate model documentation end-to-end for specific use-cases.\n", + "\n", + "Example: the [`classifier_full_suite`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html#ClassifierFullSuite) test suite runs tests from the [`tabular_dataset`](https://docs.validmind.ai/validmind/validmind/test_suites/tabular_datasets.html) and [`classifier`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html) test suites to fully document the data and model sections for binary classification model use-cases." + ] + }, + { + "cell_type": "markdown", + "id": "ee06fdea", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "1343e1dd", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Install the ValidMind Library with PII detection\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To use PII detection powered by [Microsoft Presidio](https://microsoft.github.io/presidio/), install the library with the explicit `[pii-detection]` extra specifier:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b830ae91", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q \"validmind[pii-detection]\"" + ] + }, + { + "cell_type": "markdown", + "id": "b5eae826", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Initialize the ValidMind Library\n", + "\n", + "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook." + ] + }, + { + "cell_type": "markdown", + "id": "e6cc7cd2", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Get your code snippet\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "\n", + "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eeda4c8c", + "metadata": {}, + "outputs": [], + "source": [ + "# Load your model identifier credentials from an `.env` file\n", + "\n", + "%load_ext dotenv\n", + "%dotenv .env\n", + "\n", + "# Or replace with your code snippet\n", + "\n", + "import validmind as vm\n", + "\n", + "vm.init(\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "5676cd64", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Using PII detection" + ] + }, + { + "cell_type": "markdown", + "id": "120644e3", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Create a custom test that outputs PII\n", + "\n", + "To demonstrate the feature, we'll need a test that outputs PII. First we'll create a custom test that returns:\n", + "\n", + "- A description string containing PII (name, email, phone)\n", + "- A small table containing PII in columns\n", + "\n", + "This output mirrors the structure used in other custom test notebooks and will exercise both table and description PII detection paths. However, if structured detection is unavailable, the library falls back to token-level text scans when possible." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "04d8c802", + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "\n", + "from validmind import test\n", + "\n", + "@test(\"pii_demo.PIIDetection\")\n", + "def pii_custom_test():\n", + " \"\"\"A custom test that returns demo PII.\n", + " This default test description will display when PII is not sent to the LLM to generate test descriptions based on test result data.\"\"\"\n", + " return pd.DataFrame(\n", + " {\n", + " \"name\": [\"Jane Smith\", \"John Doe\", \"Alice Johnson\"],\n", + " \"email\": [\n", + " \"jane.smith@bank.example\",\n", + " \"john.doe@company.example\",\n", + " \"alice.johnson@service.example\",\n", + " ],\n", + " \"phone\": [\"(212) 555-9876\", \"(415) 555-1234\", \"(646) 555-5678\"],\n", + " }\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "a8ab74cd", + "metadata": {}, + "source": [ + "
Want to learn more about custom tests?\n", + "

\n", + "Check out our extended introduction to custom tests — Implement custom tests
" + ] + }, + { + "cell_type": "markdown", + "id": "7d2110d3", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Run test under different PII detection modes\n", + "\n", + "Next, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test via a function called `run_pii_test()` that catches exceptions to observe blocking behavior when PII is present:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b42288e5", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from validmind.tests import run_test\n", + "\n", + "# Run test and tag result with unique `result_id`\n", + "def run_pii_test(result_id=\"\"):\n", + " try:\n", + " test_name = f\"pii_demo.PIIDetection:{result_id}\"\n", + " result = run_test(test_name)\n", + "\n", + " # Check if the test description was generated by LLM\n", + " if not result._was_description_generated:\n", + " print(\"PII detected: LLM-generated test description skipped\")\n", + " else:\n", + " print(\"No PII detected or detection disabled: Test description generated by LLM\")\n", + "\n", + " # Try logging test results to the ValidMind Platform\n", + " result.log()\n", + " print(\"No PII detected or detection disabled: Test results logged to the ValidMind Platform\")\n", + " except Exception as e:\n", + " print(\"PII detected: Test results not logged to the ValidMind Platform\")" + ] + }, + { + "cell_type": "markdown", + "id": "e3c82205", + "metadata": {}, + "source": [ + "We'll then switch the `VALIDMIND_PII_DETECTION` environment variable across modes in the below examples.\n", + "\n", + "
Note that since we are running a custom test that does not exist in your model's default documentation template, we'll receive output indicating that a test-driven block doesn't currently exist in your model's documentation for that particular test ID.\n", + "

\n", + "That's expected, as when we run custom tests the results logged need to be manually added to your documentation within the ValidMind Platform or added to your documentation template.
" + ] + }, + { + "cell_type": "markdown", + "id": "543b08dc", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### disabled\n", + "\n", + "When detection is set to `disabled`, tests run and generate test descriptions. Logging tests with [`.log()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) will also send test descriptions and test results to the ValidMind Platform as usual:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3078af64", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: disabled ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", + "\n", + "# Run test and tag result with unique ID `disabled`\n", + "run_pii_test(\"disabled\")" + ] + }, + { + "cell_type": "markdown", + "id": "490c397c", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### test_results\n", + "\n", + "When detection is set for `test_results`, tests run and generate test descriptions for review in your environment, but logging tests will not send descriptions or test results to the ValidMind Platform:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12e61a80", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: test_results ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", + "\n", + "# Run test and tag result with unique ID `results_blocked`\n", + "run_pii_test(\"results_blocked\")" + ] + }, + { + "cell_type": "markdown", + "id": "c21de2bb", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### test_descriptions\n", + "\n", + "When detection is set for `test_descriptions`, tests run but will not generate test descriptions, and logging tests will not send descriptions but will send test results to the ValidMind Platform:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "feba6207", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: test_descriptions ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", + "\n", + "# Run test and tag result with unique ID `desc_blocked`\n", + "run_pii_test(\"desc_blocked\")" + ] + }, + { + "cell_type": "markdown", + "id": "0e3c9f24", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### all\n", + "\n", + "When detection is set to `all`, tests run will not generate test descriptions or log test results to the ValidMind Platform." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "af5040b5", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: all ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", + "\n", + "# Run test and tag result with unique ID `all_blocked`\n", + "run_pii_test(\"all_blocked\")" + ] + }, + { + "cell_type": "markdown", + "id": "e30c0ccd", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Override detection\n", + "\n", + "You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", + "\n", + "To demonstrate, let's rerun our custom test with some override scenarios." + ] + }, + { + "cell_type": "markdown", + "id": "6b5b2df9", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Override test result logging\n", + "\n", + "First, let's rerun our custom test with detection set to `all`, which will send the test results but not the test descriptions to the ValidMind Platform:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0387be21", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: all & unsafe=True ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", + "\n", + "# Run test and tag result with unique ID `override_results`\n", + "try:\n", + " result = run_test(\"pii_demo.PIIDetection:override_results\")\n", + "\n", + " # Check if the test description was generated by LLM\n", + " if not result._was_description_generated:\n", + " print(\"PII detected: LLM-generated test description skipped\")\n", + " else:\n", + " print(\"No PII detected or detection disabled: Test description generated by LLM\")\n", + "\n", + " # Try logging test results to the ValidMind Platform\n", + " result.log(unsafe=True)\n", + " print(\"No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform\")\n", + "except Exception as e:\n", + " print(\"PII detected: Test results not logged to the ValidMind Platform\")" + ] + }, + { + "cell_type": "markdown", + "id": "a11072fb", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Override test descriptions and test result logging\n", + "\n", + "To send both the test descriptions and test results via override, set the `VALIDMIND_PII_DETECTION` environment variable to `test_results` while including the override flag:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b40a2670", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: test_results & unsafe=True ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", + "\n", + "# Run test and tag result with unique ID `override_both`\n", + "try:\n", + " result = run_test(\"pii_demo.PIIDetection:override_both\")\n", + "\n", + " # Check if the test description was generated by LLM\n", + " if not result._was_description_generated:\n", + " print(\"PII detected: LLM-generated test description skipped\")\n", + " else:\n", + " print(\"No PII detected, detection disabled, or override set: Test description generated by LLM\")\n", + "\n", + " # Try logging test results to the ValidMind Platform\n", + " result.log(unsafe=True)\n", + " print(\"No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform\")\n", + "except Exception as e:\n", + " print(\"PII detected: Test results not logged to the ValidMind Platform\")" + ] + }, + { + "cell_type": "markdown", + "id": "017ff3ed", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Review logged test results\n", + "\n", + "Now let's take a look at the results that were logged to the ValidMind Platform:\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier.\n", + "\n", + "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "\n", + "3. Click on any section heading to expand that section to add a new test-driven block ([Need more help?](https://docs.validmind.ai/developer/model-documentation/work-with-test-results.html)).\n", + "\n", + "4. Under TEST-DRIVEN in the sidebar, click **Custom**.\n", + "\n", + "5. Confirm that you're able to insert the following logged results:\n", + "\n", + " - `pii_demo.PIIDetection:disabled`\n", + " - `pii_demo.PIIDetection:desc_blocked`\n", + " - `pii_demo.PIIDetection:override_results`\n", + " - `pii_demo.PIIDetection:override_both`" + ] + }, + { + "cell_type": "markdown", + "id": "e60ecaf4", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Troubleshooting\n", + "\n", + "- [x] If you see warnings that Presidio or Presidio analyzer is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", + "- [x] Ensure your environment is restarted after installing new packages if imports fail." + ] + }, + { + "cell_type": "markdown", + "id": "7e0641e1", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Learn more\n", + "\n", + "We offer many interactive notebooks to help you document models:\n", + "\n", + "- [Run tests & test suites](https://docs.validmind.ai/guide/testing-overview.html)\n", + "- [Code samples](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", + "\n", + "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}