From d4b9572349090d296fde1b764b645d2daf3b4125 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 20 Aug 2025 12:42:55 -0700 Subject: [PATCH 01/37] Drafting complete PII notebook --- .../code_sharing/enable_pii_detection.ipynb | 230 ++++++++++++++++++ 1 file changed, 230 insertions(+) create mode 100644 notebooks/code_sharing/enable_pii_detection.ipynb diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb new file mode 100644 index 000000000..13504a2d5 --- /dev/null +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -0,0 +1,230 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "dd37d6f9", + "metadata": {}, + "source": [ + "# Enable PII detection in test results" + ] + }, + { + "cell_type": "markdown", + "id": "about-intro-85b40b47-34c7-4da4-b0e7-c491341a9f20", + "metadata": {}, + "source": [ + "## About ValidMind\n", + "\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "\n", + "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + ] + }, + { + "cell_type": "markdown", + "id": "about-begin-c86ec11e-4e52-4799-bf11-fea595c2671e", + "metadata": {}, + "source": [ + "### Before you begin\n", + "\n", + "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", + "\n", + "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)." + ] + }, + { + "cell_type": "markdown", + "id": "about-signup-db4c0a03-ea37-4e81-b417-2e79eed1bc53", + "metadata": {}, + "source": [ + "### New to ValidMind?\n", + "\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", + "\n", + "
For access to all features available in this notebook, you'll need access to a ValidMind account.\n", + "

\n", + "Register with ValidMind
" + ] + }, + { + "cell_type": "markdown", + "id": "about-concepts-fd7b60a6-60a8-420a-8ec2-287874cae377", + "metadata": {}, + "source": [ + "### Key concepts\n", + "\n", + "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", + "\n", + "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "\n", + "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", + "\n", + "**Metrics**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.\n", + "\n", + "**Custom metrics**: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.\n", + "\n", + "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n", + "\n", + " - **model**: A single model that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n", + " - **dataset**: Single dataset that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n", + " - **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.\n", + " - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/run_tests_that_require_multiple_datasets.html))\n", + "\n", + "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.\n", + "\n", + "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.\n", + "\n", + "**Test suites**: Collections of tests designed to run together to automate and generate model documentation end-to-end for specific use-cases.\n", + "\n", + "Example: the [`classifier_full_suite`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html#ClassifierFullSuite) test suite runs tests from the [`tabular_dataset`](https://docs.validmind.ai/validmind/validmind/test_suites/tabular_datasets.html) and [`classifier`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html) test suites to fully document the data and model sections for binary classification model use-cases." + ] + }, + { + "cell_type": "markdown", + "id": "install-library-f5c5d59c-e356-4f22-a71e-d0c84c269273", + "metadata": {}, + "source": [ + "## Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-python-6511976a-8cb2-423d-bfbf-a6e39cc90e4a", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "id": "install-initialize-e13c0197-f742-453c-9422-ebae53ddcf68", + "metadata": {}, + "source": [ + "## Initialize the ValidMind Library\n", + "\n", + "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook." + ] + }, + { + "cell_type": "markdown", + "id": "install-snippet-dc2c3a8b-a2d8-4632-84f9-bb5c9dd8721c", + "metadata": {}, + "source": [ + "### Get your code snippet\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + " For example, to register a model for use with this notebook, select:" + ] + }, + { + "cell_type": "markdown", + "id": "install-variables-b1d4c97e-05b1-40b5-9928-30f1c5e1f0e4", + "metadata": {}, + "source": [ + " - Documentation template: `{template}`\n", + " - Use case: `{use-case}`" + ] + }, + { + "cell_type": "markdown", + "id": "install-credentials-ff45c932-7622-4030-8217-d52597bda4ca", + "metadata": {}, + "source": [ + " You can fill in other options according to your preference.\n", + " \n", + "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "\n", + "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-init-43c8827f-b023-4324-be38-c58c999d4575", + "metadata": {}, + "outputs": [], + "source": [ + "# Load your model identifier credentials from an `.env` file\n", + "\n", + "%load_ext dotenv\n", + "%dotenv .env\n", + "\n", + "# Or replace with your code snippet\n", + "\n", + "import validmind as vm\n", + "\n", + "vm.init(\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "next-steps-35e81828-fe3d-46ae-aaa1-250f0575595f", + "metadata": {}, + "source": [ + "## Next steps\n", + "\n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation." + ] + }, + { + "cell_type": "markdown", + "id": "next-docs-ae68157f-de90-449f-8674-326faa2db7ea", + "metadata": {}, + "source": [ + "### Work with your model documentation\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "\n", + "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + ] + }, + { + "cell_type": "markdown", + "id": "next-resources-c5b44493-85c8-491b-8eaa-d1bdfd81bc86", + "metadata": {}, + "source": [ + "### Discover more learning resources\n", + "\n", + "We offer many interactive notebooks to help you document models:\n", + "\n", + "- [Run tests & test suites](https://docs.validmind.ai/guide/testing-overview.html)\n", + "- [Code samples](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", + "\n", + "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 5730fd9faff48ad88a421f5b3eb52f77687eae66 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 20 Aug 2025 12:48:43 -0700 Subject: [PATCH 02/37] Editing --- .../code_sharing/enable_pii_detection.ipynb | 31 ++++++------------- 1 file changed, 10 insertions(+), 21 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 13504a2d5..9e827e524 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -8,6 +8,14 @@ "# Enable PII detection in test results" ] }, + { + "cell_type": "markdown", + "id": "d4721b73", + "metadata": {}, + "source": [ + "Learn how to enable Personally Identifiable Information (PII) detection when generating test results with the ValidMind Library, and configure safety overrides for specific test results." + ] + }, { "cell_type": "markdown", "id": "about-intro-85b40b47-34c7-4da4-b0e7-c491341a9f20", @@ -84,7 +92,7 @@ "id": "install-library-f5c5d59c-e356-4f22-a71e-d0c84c269273", "metadata": {}, "source": [ - "## Install the ValidMind Library\n", + "## Install the ValidMind Library with PII detection\n", "\n", "
Recommended Python versions\n", "

\n", @@ -100,7 +108,7 @@ "metadata": {}, "outputs": [], "source": [ - "%pip install -q validmind" + "%pip install -q validmind[pii-detection]" ] }, { @@ -126,25 +134,6 @@ "\n", "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select:" - ] - }, - { - "cell_type": "markdown", - "id": "install-variables-b1d4c97e-05b1-40b5-9928-30f1c5e1f0e4", - "metadata": {}, - "source": [ - " - Documentation template: `{template}`\n", - " - Use case: `{use-case}`" - ] - }, - { - "cell_type": "markdown", - "id": "install-credentials-ff45c932-7622-4030-8217-d52597bda4ca", - "metadata": {}, - "source": [ - " You can fill in other options according to your preference.\n", - " \n", "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", "\n", "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" From 8eed298d5c1f22b9f693366c4c06c09f15767958 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 20 Aug 2025 12:54:06 -0700 Subject: [PATCH 03/37] Save point --- .../code_sharing/enable_pii_detection.ipynb | 40 +++++++++++++++++-- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 9e827e524..7b3b7c599 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -87,18 +87,26 @@ "Example: the [`classifier_full_suite`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html#ClassifierFullSuite) test suite runs tests from the [`tabular_dataset`](https://docs.validmind.ai/validmind/validmind/test_suites/tabular_datasets.html) and [`classifier`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html) test suites to fully document the data and model sections for binary classification model use-cases." ] }, + { + "cell_type": "markdown", + "id": "33f87f48", + "metadata": {}, + "source": [ + "## Setting up" + ] + }, { "cell_type": "markdown", "id": "install-library-f5c5d59c-e356-4f22-a71e-d0c84c269273", "metadata": {}, "source": [ - "## Install the ValidMind Library with PII detection\n", + "### Install the ValidMind Library with PII detection\n", "\n", "
Recommended Python versions\n", "

\n", "Python 3.8 <= x <= 3.11
\n", "\n", - "To install the library:" + "To use PII detection, install the library with the explicit `[pii-detection]` extra specifier:" ] }, { @@ -116,7 +124,7 @@ "id": "install-initialize-e13c0197-f742-453c-9422-ebae53ddcf68", "metadata": {}, "source": [ - "## Initialize the ValidMind Library\n", + "### Initialize the ValidMind Library\n", "\n", "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook." ] @@ -126,7 +134,7 @@ "id": "install-snippet-dc2c3a8b-a2d8-4632-84f9-bb5c9dd8721c", "metadata": {}, "source": [ - "### Get your code snippet\n", + "#### Get your code snippet\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", @@ -163,6 +171,30 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "f3e47f7a", + "metadata": {}, + "source": [ + "## Using PII detection" + ] + }, + { + "cell_type": "markdown", + "id": "a5cb8d86", + "metadata": {}, + "source": [ + "### Create a custom test that outputs PII" + ] + }, + { + "cell_type": "markdown", + "id": "d9f55c76", + "metadata": {}, + "source": [ + "### Run custom test under different PII detection modes" + ] + }, { "cell_type": "markdown", "id": "next-steps-35e81828-fe3d-46ae-aaa1-250f0575595f", From 19037b990437e63702f23677e06ab731abbd2423 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 20 Aug 2025 12:58:20 -0700 Subject: [PATCH 04/37] Save point --- .../code_sharing/enable_pii_detection.ipynb | 49 ++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 7b3b7c599..37a7550e5 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -173,26 +173,63 @@ }, { "cell_type": "markdown", - "id": "f3e47f7a", + "id": "a5cb8d86", "metadata": {}, "source": [ - "## Using PII detection" + "### Create a custom test that outputs PII\n", + "\n", + "To demonstrate the feature, we'll need a test that outputs PII. We'll create a custom test that returns:\n", + "\n", + "- A description string containing PII (name, email, phone)\n", + "- A small table containing PII in columns\n", + "\n", + "This output mirrors the structure used in other custom test notebooks and will exercise both table and description PII detection paths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6452da1f", + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "\n", + "from validmind import test\n", + "\n", + "@test(\"my_pii_demo.PIIEmittingTest\")\n", + "def pii_emitting_test():\n", + " \"\"\"A demo test that returns both a PII-bearing description and a PII-bearing table.\"\"\"\n", + " description = (\n", + " \"Primary contact: John Doe (john.doe@example.com), phone +1-415-555-1234.\"\n", + " )\n", + " table = pd.DataFrame(\n", + " {\n", + " \"name\": [\"Jane Smith\"],\n", + " \"email\": [\"jane.smith@bank.example\"],\n", + " \"phone\": [\"(212) 555-9876\"],\n", + " }\n", + " )\n", + " # Return order: (description, table)\n", + " return description, table" ] }, { "cell_type": "markdown", - "id": "a5cb8d86", + "id": "e210b665", "metadata": {}, "source": [ - "### Create a custom test that outputs PII" + "
Want to learn more about custom tests?\n", + "

\n", + "Check out our extended introduction to custom tests — Implement custom tests
" ] }, { "cell_type": "markdown", - "id": "d9f55c76", + "id": "f3e47f7a", "metadata": {}, "source": [ - "### Run custom test under different PII detection modes" + "## Using PII detection" ] }, { From daf440a6943f1576c2661fc00c8d772a4a9ce7fd Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 20 Aug 2025 13:34:36 -0700 Subject: [PATCH 05/37] Save point --- .../code_sharing/enable_pii_detection.ipynb | 54 ++++++++++++++++++- 1 file changed, 52 insertions(+), 2 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 37a7550e5..60e5897af 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -106,7 +106,7 @@ "

\n", "Python 3.8 <= x <= 3.11
\n", "\n", - "To use PII detection, install the library with the explicit `[pii-detection]` extra specifier:" + "To use PII detection powered by [Microsoft Presidio](https://microsoft.github.io/presidio/), install the library with the explicit `[pii-detection]` extra specifier:" ] }, { @@ -229,7 +229,57 @@ "id": "f3e47f7a", "metadata": {}, "source": [ - "## Using PII detection" + "## Run different PII detection modes\n", + "\n", + "We'll switch `VALIDMIND_PII_DETECTION` across modes and run the same test with `validmind.tests.run_test`. We catch exceptions to observe blocking behavior.\n", + "\n", + "### Expected behavior by mode\n", + "\n", + "- disabled: No PII checks; test runs and logs.\n", + "- test_results: Table PII triggers blocking; description may still proceed.\n", + "- test_descriptions: Description PII triggers blocking; tables may still proceed.\n", + "- all: Both table and description checks are enforced.\n", + "\n", + "## Notes\n", + "\n", + "- If you see warnings that Presidio is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", + "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", + "- To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop.\n", + "\n", + "## Troubleshooting\n", + "\n", + "- If you see warnings like \"Presidio analyzer not available\", install the extras: `pip install validmind[pii-detection]`.\n", + "- If structured detection is unavailable, the library falls back to token-level text scans when possible.\n", + "- Ensure your environment is restarted after installing new packages if imports fail.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "31ba34a5", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from validmind.tests import run_test\n", + "\n", + "MODES = [\"disabled\", \"test_results\", \"test_descriptions\", \"all\"]\n", + "\n", + "for mode in MODES:\n", + " print(\"\\n=== Mode:\", mode, \"===\")\n", + " os.environ[\"VALIDMIND_PII_DETECTION\"] = mode\n", + " try:\n", + " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", + "\n", + " # check if the description was generated\n", + " if not result._was_description_generated:\n", + " print(\"Blocked: Test Description Generation was not run due to PII\")\n", + "\n", + " # Try logging (this triggers PII checks before upload)\n", + " result.log()\n", + " print(\"Run + log succeeded\")\n", + " except Exception as e:\n", + " print(\"Blocked:\", type(e).__name__, str(e)[:200])" ] }, { From 4993f01d0fd5b49b773bbaaa52f4f341b709e9f1 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 20 Aug 2025 13:35:39 -0700 Subject: [PATCH 06/37] Save point --- notebooks/code_sharing/enable_pii_detection.ipynb | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 60e5897af..86ff22c2b 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -171,6 +171,14 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "9296d9c8", + "metadata": {}, + "source": [ + "## Using PII detection" + ] + }, { "cell_type": "markdown", "id": "a5cb8d86", @@ -229,7 +237,7 @@ "id": "f3e47f7a", "metadata": {}, "source": [ - "## Run different PII detection modes\n", + "### Run different PII detection modes\n", "\n", "We'll switch `VALIDMIND_PII_DETECTION` across modes and run the same test with `validmind.tests.run_test`. We catch exceptions to observe blocking behavior.\n", "\n", From 41ca8005a557f0aab260088ab3b1f72c0d9940fd Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 20 Aug 2025 13:46:37 -0700 Subject: [PATCH 07/37] Save point --- .../code_sharing/enable_pii_detection.ipynb | 48 +++++++++++-------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 86ff22c2b..7de0f8245 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -237,28 +237,11 @@ "id": "f3e47f7a", "metadata": {}, "source": [ - "### Run different PII detection modes\n", + "### Run test under different PII detection modes\n", "\n", - "We'll switch `VALIDMIND_PII_DETECTION` across modes and run the same test with `validmind.tests.run_test`. We catch exceptions to observe blocking behavior.\n", + "Let's use [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test.\n", "\n", - "### Expected behavior by mode\n", - "\n", - "- disabled: No PII checks; test runs and logs.\n", - "- test_results: Table PII triggers blocking; description may still proceed.\n", - "- test_descriptions: Description PII triggers blocking; tables may still proceed.\n", - "- all: Both table and description checks are enforced.\n", - "\n", - "## Notes\n", - "\n", - "- If you see warnings that Presidio is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", - "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", - "- To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop.\n", - "\n", - "## Troubleshooting\n", - "\n", - "- If you see warnings like \"Presidio analyzer not available\", install the extras: `pip install validmind[pii-detection]`.\n", - "- If structured detection is unavailable, the library falls back to token-level text scans when possible.\n", - "- Ensure your environment is restarted after installing new packages if imports fail.\n" + "We'll switch `VALIDMIND_PII_DETECTION` across modes and catch exceptions to observe blocking behavior:" ] }, { @@ -290,6 +273,31 @@ " print(\"Blocked:\", type(e).__name__, str(e)[:200])" ] }, + { + "cell_type": "markdown", + "id": "85e31c5a", + "metadata": {}, + "source": [ + "#### Expected behavior by mode\n", + "\n", + "- **disabled:** No PII checks; test runs and logs.\n", + "- **test_results:** Table PII triggers blocking; description may still proceed.\n", + "- **test_descriptions:** Description PII triggers blocking; tables may still proceed.\n", + "- **all:** Both table and description checks are enforced.\n", + "\n", + "## Notes\n", + "\n", + "- If you see warnings that Presidio is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", + "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", + "- To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop.\n", + "\n", + "## Troubleshooting\n", + "\n", + "- If you see warnings like \"Presidio analyzer not available\", install the extras: `pip install validmind[pii-detection]`.\n", + "- If structured detection is unavailable, the library falls back to token-level text scans when possible.\n", + "- Ensure your environment is restarted after installing new packages if imports fail." + ] + }, { "cell_type": "markdown", "id": "next-steps-35e81828-fe3d-46ae-aaa1-250f0575595f", From 5d7a2f2fd98059131d4fb793d1a60f8f8af8dd44 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 20 Aug 2025 13:59:49 -0700 Subject: [PATCH 08/37] Editing --- .../code_sharing/enable_pii_detection.ipynb | 35 ++++++------------- 1 file changed, 11 insertions(+), 24 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 7de0f8245..9ab1fb0c6 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -283,43 +283,30 @@ "- **disabled:** No PII checks; test runs and logs.\n", "- **test_results:** Table PII triggers blocking; description may still proceed.\n", "- **test_descriptions:** Description PII triggers blocking; tables may still proceed.\n", - "- **all:** Both table and description checks are enforced.\n", - "\n", - "## Notes\n", - "\n", - "- If you see warnings that Presidio is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", - "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", - "- To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop.\n", - "\n", - "## Troubleshooting\n", - "\n", - "- If you see warnings like \"Presidio analyzer not available\", install the extras: `pip install validmind[pii-detection]`.\n", - "- If structured detection is unavailable, the library falls back to token-level text scans when possible.\n", - "- Ensure your environment is restarted after installing new packages if imports fail." + "- **all:** Both table and description checks are enforced." ] }, { "cell_type": "markdown", - "id": "next-steps-35e81828-fe3d-46ae-aaa1-250f0575595f", + "id": "f2cdc27f", "metadata": {}, "source": [ - "## Next steps\n", + "### Notes\n", "\n", - "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation." + "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", + "- To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop." ] }, { "cell_type": "markdown", - "id": "next-docs-ae68157f-de90-449f-8674-326faa2db7ea", + "id": "53a1a62d", "metadata": {}, "source": [ - "### Work with your model documentation\n", - "\n", - "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "### Troubleshooting\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", - "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + "- If you see warnings that Presidio or Presidio analyzer is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", + "- If structured detection is unavailable, the library falls back to token-level text scans when possible.\n", + "- Ensure your environment is restarted after installing new packages if imports fail." ] }, { @@ -327,7 +314,7 @@ "id": "next-resources-c5b44493-85c8-491b-8eaa-d1bdfd81bc86", "metadata": {}, "source": [ - "### Discover more learning resources\n", + "## Learn more\n", "\n", "We offer many interactive notebooks to help you document models:\n", "\n", From 652dfff9366a442b96096366fdd087c371515a22 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 08:46:55 -0700 Subject: [PATCH 09/37] Editing --- .../code_sharing/enable_pii_detection.ipynb | 137 ++++++++++++++---- 1 file changed, 111 insertions(+), 26 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 9ab1fb0c6..caf19c805 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -116,7 +116,7 @@ "metadata": {}, "outputs": [], "source": [ - "%pip install -q validmind[pii-detection]" + "%pip install -q \"validmind[pii-detection]\"" ] }, { @@ -239,51 +239,136 @@ "source": [ "### Run test under different PII detection modes\n", "\n", - "Let's use [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test.\n", + "Now, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test.\n", "\n", - "We'll switch `VALIDMIND_PII_DETECTION` across modes and catch exceptions to observe blocking behavior:" + "We'll then switch `VALIDMIND_PII_DETECTION` across modes when we run our custom test and catch exceptions to observe blocking behavior." ] }, { "cell_type": "code", "execution_count": null, - "id": "31ba34a5", + "id": "ef7c2b50", "metadata": {}, "outputs": [], "source": [ "import os\n", - "from validmind.tests import run_test\n", - "\n", - "MODES = [\"disabled\", \"test_results\", \"test_descriptions\", \"all\"]\n", + "from validmind.tests import run_test" + ] + }, + { + "cell_type": "markdown", + "id": "b39d6c1e", + "metadata": {}, + "source": [ + "#### disabled\n", "\n", - "for mode in MODES:\n", - " print(\"\\n=== Mode:\", mode, \"===\")\n", - " os.environ[\"VALIDMIND_PII_DETECTION\"] = mode\n", - " try:\n", - " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", + "No PII checks; test runs and logs:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0c6cbcba", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: disabled ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", + "try:\n", + " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", + " if not result._was_description_generated:\n", + " print(\"Blocked: Test Description Generation was not run due to PII\")\n", + " result.log()\n", + " print(\"Run + log succeeded\")\n", + "except Exception as e:\n", + " print(\"Blocked:\", type(e).__name__, str(e)[:200])" + ] + }, + { + "cell_type": "markdown", + "id": "dcaf5605", + "metadata": {}, + "source": [ + "#### test_results\n", "\n", - " # check if the description was generated\n", - " if not result._was_description_generated:\n", - " print(\"Blocked: Test Description Generation was not run due to PII\")\n", + "Table PII triggers blocking; description may still proceed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "06c3f4f9", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: test_results ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", + "try:\n", + " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", + " if not result._was_description_generated:\n", + " print(\"Blocked: Test Description Generation was not run due to PII\")\n", + " result.log()\n", + " print(\"Run + log succeeded\")\n", + "except Exception as e:\n", + " print(\"Blocked:\", type(e).__name__, str(e)[:200])\n" + ] + }, + { + "cell_type": "markdown", + "id": "808f95fb", + "metadata": {}, + "source": [ + "#### test_descriptions\n", "\n", - " # Try logging (this triggers PII checks before upload)\n", - " result.log()\n", - " print(\"Run + log succeeded\")\n", - " except Exception as e:\n", - " print(\"Blocked:\", type(e).__name__, str(e)[:200])" + "Description PII triggers blocking; tables may still proceed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "46636a1c", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: test_descriptions ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", + "try:\n", + " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", + " if not result._was_description_generated:\n", + " print(\"Blocked: Test Description Generation was not run due to PII\")\n", + " result.log()\n", + " print(\"Run + log succeeded\")\n", + "except Exception as e:\n", + " print(\"Blocked:\", type(e).__name__, str(e)[:200])" ] }, { "cell_type": "markdown", - "id": "85e31c5a", + "id": "6121b341", "metadata": {}, "source": [ - "#### Expected behavior by mode\n", + "#### all\n", "\n", - "- **disabled:** No PII checks; test runs and logs.\n", - "- **test_results:** Table PII triggers blocking; description may still proceed.\n", - "- **test_descriptions:** Description PII triggers blocking; tables may still proceed.\n", - "- **all:** Both table and description checks are enforced." + "Both table and description checks are enforced:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cb627f9a", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: all ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", + "try:\n", + " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", + " if not result._was_description_generated:\n", + " print(\"Blocked: Test Description Generation was not run due to PII\")\n", + " result.log()\n", + " print(\"Run + log succeeded\")\n", + "except Exception as e:\n", + " print(\"Blocked:\", type(e).__name__, str(e)[:200])" ] }, { From aae77af0e92273c28fb247c806f88980d5b4f9aa Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 09:42:29 -0700 Subject: [PATCH 10/37] Editing --- .../code_sharing/enable_pii_detection.ipynb | 85 +++++++++++++------ 1 file changed, 58 insertions(+), 27 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index caf19c805..1d36e2360 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -207,19 +207,18 @@ "\n", "@test(\"my_pii_demo.PIIEmittingTest\")\n", "def pii_emitting_test():\n", - " \"\"\"A demo test that returns both a PII-bearing description and a PII-bearing table.\"\"\"\n", - " description = (\n", - " \"Primary contact: John Doe (john.doe@example.com), phone +1-415-555-1234.\"\n", - " )\n", - " table = pd.DataFrame(\n", + " \"\"\"A demo test that returns PII\"\"\"\n", + " return pd.DataFrame(\n", " {\n", - " \"name\": [\"Jane Smith\"],\n", - " \"email\": [\"jane.smith@bank.example\"],\n", - " \"phone\": [\"(212) 555-9876\"],\n", + " \"name\": [\"Jane Smith\", \"John Doe\", \"Alice Johnson\"],\n", + " \"email\": [\n", + " \"jane.smith@bank.example\",\n", + " \"john.doe@company.example\",\n", + " \"alice.johnson@service.example\",\n", + " ],\n", + " \"phone\": [\"(212) 555-9876\", \"(415) 555-1234\", \"(646) 555-5678\"],\n", " }\n", - " )\n", - " # Return order: (description, table)\n", - " return description, table" + " )" ] }, { @@ -272,16 +271,24 @@ "metadata": {}, "outputs": [], "source": [ + "# Disabled Mode\n", "print(\"\\n=== Mode: disabled ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - " if not result._was_description_generated:\n", + " if not getattr(result, \"_was_description_generated\", False):\n", " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " result.log()\n", - " print(\"Run + log succeeded\")\n", + " else:\n", + " print(\"Description was generated by LLM\")\n", + " try:\n", + " result.log()\n", + " print(\"Logging to API succeeded\")\n", + " except Exception as log_error:\n", + " print(\"Blocked: Test Result was not logged due to PII\")\n", + " print(\"Reason:\", log_error)\n", "except Exception as e:\n", - " print(\"Blocked:\", type(e).__name__, str(e)[:200])" + " print(\"Blocked: Test execution failed due to PII\")\n", + " print(\"Reason:\", e)" ] }, { @@ -301,16 +308,24 @@ "metadata": {}, "outputs": [], "source": [ + "# Test Results Mode\n", "print(\"\\n=== Mode: test_results ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - " if not result._was_description_generated:\n", + " if not getattr(result, \"_was_description_generated\", False):\n", " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " result.log()\n", - " print(\"Run + log succeeded\")\n", + " else:\n", + " print(\"Description was generated by LLM\")\n", + " try:\n", + " result.log()\n", + " print(\"Logging to API succeeded\")\n", + " except Exception as log_error:\n", + " print(\"Blocked: Test Result was not logged due to PII\")\n", + " print(\"Reason:\", log_error)\n", "except Exception as e:\n", - " print(\"Blocked:\", type(e).__name__, str(e)[:200])\n" + " print(\"Blocked: Test execution failed due to PII\")\n", + " print(\"Reason:\", e)" ] }, { @@ -330,16 +345,24 @@ "metadata": {}, "outputs": [], "source": [ + "# Test Descriptions Mode\n", "print(\"\\n=== Mode: test_descriptions ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - " if not result._was_description_generated:\n", + " if not getattr(result, \"_was_description_generated\", False):\n", " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " result.log()\n", - " print(\"Run + log succeeded\")\n", + " else:\n", + " print(\"Description was generated by LLM\")\n", + " try:\n", + " result.log()\n", + " print(\"Logging to API succeeded\")\n", + " except Exception as log_error:\n", + " print(\"Blocked: Test Result was not logged due to PII\")\n", + " print(\"Reason:\", log_error)\n", "except Exception as e:\n", - " print(\"Blocked:\", type(e).__name__, str(e)[:200])" + " print(\"Blocked: Test execution failed due to PII\")\n", + " print(\"Reason:\", e)" ] }, { @@ -359,16 +382,24 @@ "metadata": {}, "outputs": [], "source": [ + "# All Mode\n", "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - " if not result._was_description_generated:\n", + " if not getattr(result, \"_was_description_generated\", False):\n", " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " result.log()\n", - " print(\"Run + log succeeded\")\n", + " else:\n", + " print(\"Description was generated by LLM\")\n", + " try:\n", + " result.log()\n", + " print(\"Logging to API succeeded\")\n", + " except Exception as log_error:\n", + " print(\"Blocked: Test Result was not logged due to PII\")\n", + " print(\"Reason:\", log_error)\n", "except Exception as e:\n", - " print(\"Blocked:\", type(e).__name__, str(e)[:200])" + " print(\"Blocked: Test execution failed due to PII\")\n", + " print(\"Reason:\", e)" ] }, { From 31a02e2232eaff207b38c577939ca265aeb70f79 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 09:51:47 -0700 Subject: [PATCH 11/37] Editing... --- .../code_sharing/enable_pii_detection.ipynb | 92 +++++++++---------- 1 file changed, 42 insertions(+), 50 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 1d36e2360..ee85954f1 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -205,8 +205,8 @@ "\n", "from validmind import test\n", "\n", - "@test(\"my_pii_demo.PIIEmittingTest\")\n", - "def pii_emitting_test():\n", + "@test(\"my_pii_demo.PIICustomTest\")\n", + "def pii_custom_test():\n", " \"\"\"A demo test that returns PII\"\"\"\n", " return pd.DataFrame(\n", " {\n", @@ -271,24 +271,22 @@ "metadata": {}, "outputs": [], "source": [ - "# Disabled Mode\n", - "print(\"\\n=== Mode: disabled ===\")\n", + "print(\"\\n=== Mode:\", mode, \"===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", "try:\n", - " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - " if not getattr(result, \"_was_description_generated\", False):\n", + " result = run_test(\"my_pii_demo.PIICustomTest\")\n", + "\n", + " # check if the description was generated\n", + " if not result._was_description_generated:\n", " print(\"Blocked: Test Description Generation was not run due to PII\")\n", " else:\n", " print(\"Description was generated by LLM\")\n", - " try:\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - " except Exception as log_error:\n", - " print(\"Blocked: Test Result was not logged due to PII\")\n", - " print(\"Reason:\", log_error)\n", + "\n", + " # Try logging (this triggers PII checks before upload)\n", + " result.log()\n", + " print(\"Logging to API succeeded\")\n", "except Exception as e:\n", - " print(\"Blocked: Test execution failed due to PII\")\n", - " print(\"Reason:\", e)" + " print(\"Blocked: Test Result was not logged due to PII\")" ] }, { @@ -308,24 +306,22 @@ "metadata": {}, "outputs": [], "source": [ - "# Test Results Mode\n", - "print(\"\\n=== Mode: test_results ===\")\n", + "print(\"\\n=== Mode:\", mode, \"===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", "try:\n", - " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - " if not getattr(result, \"_was_description_generated\", False):\n", + " result = run_test(\"my_pii_demo.PIICustomTest\")\n", + "\n", + " # check if the description was generated\n", + " if not result._was_description_generated:\n", " print(\"Blocked: Test Description Generation was not run due to PII\")\n", " else:\n", " print(\"Description was generated by LLM\")\n", - " try:\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - " except Exception as log_error:\n", - " print(\"Blocked: Test Result was not logged due to PII\")\n", - " print(\"Reason:\", log_error)\n", + "\n", + " # Try logging (this triggers PII checks before upload)\n", + " result.log()\n", + " print(\"Logging to API succeeded\")\n", "except Exception as e:\n", - " print(\"Blocked: Test execution failed due to PII\")\n", - " print(\"Reason:\", e)" + " print(\"Blocked: Test Result was not logged due to PII\")" ] }, { @@ -345,24 +341,22 @@ "metadata": {}, "outputs": [], "source": [ - "# Test Descriptions Mode\n", - "print(\"\\n=== Mode: test_descriptions ===\")\n", + "print(\"\\n=== Mode:\", mode, \"===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", "try:\n", - " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - " if not getattr(result, \"_was_description_generated\", False):\n", + " result = run_test(\"my_pii_demo.PIICustomTest\")\n", + "\n", + " # check if the description was generated\n", + " if not result._was_description_generated:\n", " print(\"Blocked: Test Description Generation was not run due to PII\")\n", " else:\n", " print(\"Description was generated by LLM\")\n", - " try:\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - " except Exception as log_error:\n", - " print(\"Blocked: Test Result was not logged due to PII\")\n", - " print(\"Reason:\", log_error)\n", + "\n", + " # Try logging (this triggers PII checks before upload)\n", + " result.log()\n", + " print(\"Logging to API succeeded\")\n", "except Exception as e:\n", - " print(\"Blocked: Test execution failed due to PII\")\n", - " print(\"Reason:\", e)" + " print(\"Blocked: Test Result was not logged due to PII\")" ] }, { @@ -382,24 +376,22 @@ "metadata": {}, "outputs": [], "source": [ - "# All Mode\n", - "print(\"\\n=== Mode: all ===\")\n", + "print(\"\\n=== Mode:\", mode, \"===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "try:\n", - " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - " if not getattr(result, \"_was_description_generated\", False):\n", + " result = run_test(\"my_pii_demo.PIICustomTest\")\n", + "\n", + " # check if the description was generated\n", + " if not result._was_description_generated:\n", " print(\"Blocked: Test Description Generation was not run due to PII\")\n", " else:\n", " print(\"Description was generated by LLM\")\n", - " try:\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - " except Exception as log_error:\n", - " print(\"Blocked: Test Result was not logged due to PII\")\n", - " print(\"Reason:\", log_error)\n", + "\n", + " # Try logging (this triggers PII checks before upload)\n", + " result.log()\n", + " print(\"Logging to API succeeded\")\n", "except Exception as e:\n", - " print(\"Blocked: Test execution failed due to PII\")\n", - " print(\"Reason:\", e)" + " print(\"Blocked: Test Result was not logged due to PII\")" ] }, { From c54511a29fc39e0dd36a72cab734741cb8957c49 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 09:55:23 -0700 Subject: [PATCH 12/37] Save point --- .../code_sharing/enable_pii_detection.ipynb | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index ee85954f1..e9f3ae715 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -271,7 +271,7 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"\\n=== Mode:\", mode, \"===\")\n", + "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIICustomTest\")\n", @@ -306,7 +306,7 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"\\n=== Mode:\", mode, \"===\")\n", + "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIICustomTest\")\n", @@ -341,7 +341,7 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"\\n=== Mode:\", mode, \"===\")\n", + "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIICustomTest\")\n", @@ -376,7 +376,7 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"\\n=== Mode:\", mode, \"===\")\n", + "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIICustomTest\")\n", @@ -394,12 +394,20 @@ " print(\"Blocked: Test Result was not logged due to PII\")" ] }, + { + "cell_type": "markdown", + "id": "284e69e5", + "metadata": {}, + "source": [ + "### Overriding detection" + ] + }, { "cell_type": "markdown", "id": "f2cdc27f", "metadata": {}, "source": [ - "### Notes\n", + "### Notes*\n", "\n", "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", "- To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop." From 739523278599bc75f1c2707e30a6441504ed8dfe Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 09:59:14 -0700 Subject: [PATCH 13/37] Save point --- .../code_sharing/enable_pii_detection.ipynb | 36 ++++++++++++------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index e9f3ae715..9d5e14978 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -191,7 +191,7 @@ "- A description string containing PII (name, email, phone)\n", "- A small table containing PII in columns\n", "\n", - "This output mirrors the structure used in other custom test notebooks and will exercise both table and description PII detection paths." + "This output mirrors the structure used in other custom test notebooks and will exercise both table and description PII detection paths. However, if structured detection is unavailable, the library falls back to token-level text scans when possible." ] }, { @@ -271,7 +271,7 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"\\n=== Mode: all ===\")\n", + "print(\"\\n=== Mode: disabled ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIICustomTest\")\n", @@ -306,7 +306,7 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"\\n=== Mode: all ===\")\n", + "print(\"\\n=== Mode: test_results ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIICustomTest\")\n", @@ -341,7 +341,7 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"\\n=== Mode: all ===\")\n", + "print(\"\\n=== Mode: test_descriptions ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", "try:\n", " result = run_test(\"my_pii_demo.PIICustomTest\")\n", @@ -394,23 +394,34 @@ " print(\"Blocked: Test Result was not logged due to PII\")" ] }, + { + "cell_type": "markdown", + "id": "d2c36322", + "metadata": {}, + "source": [ + "### Detection adjustments" + ] + }, { "cell_type": "markdown", "id": "284e69e5", "metadata": {}, "source": [ - "### Overriding detection" + "#### Overriding detection\n", + "\n", + "You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", + "\n", + "For example," ] }, { "cell_type": "markdown", - "id": "f2cdc27f", + "id": "310f69e1", "metadata": {}, "source": [ - "### Notes*\n", + "### Test only a subset\n", "\n", - "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", - "- To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop." + "To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop:" ] }, { @@ -418,11 +429,10 @@ "id": "53a1a62d", "metadata": {}, "source": [ - "### Troubleshooting\n", + "## Troubleshooting\n", "\n", - "- If you see warnings that Presidio or Presidio analyzer is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", - "- If structured detection is unavailable, the library falls back to token-level text scans when possible.\n", - "- Ensure your environment is restarted after installing new packages if imports fail." + "- [x] If you see warnings that Presidio or Presidio analyzer is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", + "- [x] Ensure your environment is restarted after installing new packages if imports fail." ] }, { From a88531855770a0ca9bf3f0a51946777fc8ca0023 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 10:29:05 -0700 Subject: [PATCH 14/37] Save point --- notebooks/code_sharing/enable_pii_detection.ipynb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 9d5e14978..34281e68a 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -261,7 +261,7 @@ "source": [ "#### disabled\n", "\n", - "No PII checks; test runs and logs:" + "When PII checks are disabled, tests run and generate test descriptions. Logging tests with [`.log()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) will also send test results to the ValidMind Platform as usual:" ] }, { @@ -273,14 +273,15 @@ "source": [ "print(\"\\n=== Mode: disabled ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", + "\n", "try:\n", " result = run_test(\"my_pii_demo.PIICustomTest\")\n", "\n", " # check if the description was generated\n", " if not result._was_description_generated:\n", - " print(\"Blocked: Test Description Generation was not run due to PII\")\n", + " print(\"PII detected: LLM-generated test description skipped\")\n", " else:\n", - " print(\"Description was generated by LLM\")\n", + " print(\"No PII detected: Test description generated by LLM\")\n", "\n", " # Try logging (this triggers PII checks before upload)\n", " result.log()\n", From ce5f83d8c85bd714e4b7267556b4138aa53fc6e8 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 10:39:56 -0700 Subject: [PATCH 15/37] ... --- .../code_sharing/enable_pii_detection.ipynb | 44 ++++++++++++++----- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 34281e68a..4fd1be0db 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -205,7 +205,7 @@ "\n", "from validmind import test\n", "\n", - "@test(\"my_pii_demo.PIICustomTest\")\n", + "@test(\"pii_demo.PIICustomTest\")\n", "def pii_custom_test():\n", " \"\"\"A demo test that returns PII\"\"\"\n", " return pd.DataFrame(\n", @@ -238,9 +238,7 @@ "source": [ "### Run test under different PII detection modes\n", "\n", - "Now, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test.\n", - "\n", - "We'll then switch `VALIDMIND_PII_DETECTION` across modes when we run our custom test and catch exceptions to observe blocking behavior." + "Now, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test:" ] }, { @@ -251,7 +249,31 @@ "outputs": [], "source": [ "import os\n", - "from validmind.tests import run_test" + "from validmind.tests import run_test\n", + "\n", + "def run_pii_test():\n", + " try:\n", + " result = run_test(\"pii_demo.PIICustomTest\")\n", + "\n", + " # Check if the test description was generated by LLM\n", + " if not result._was_description_generated:\n", + " print(\"PII detected: LLM-generated test description skipped\")\n", + " else:\n", + " print(\"No PII detected: Test description generated by LLM\")\n", + "\n", + " # Try logging test results to the ValidMind Platform\n", + " result.log()\n", + " print(\"No PII detected: Test results logged to the ValidMind Platform\")\n", + " except Exception as e:\n", + " print(\"PII detected: Test results not logged to the ValidMind Platform\")" + ] + }, + { + "cell_type": "markdown", + "id": "e349b5f5", + "metadata": {}, + "source": [ + "We'll then switch `VALIDMIND_PII_DETECTION` across modes when we run our custom test and catch exceptions to observe blocking behavior." ] }, { @@ -275,7 +297,7 @@ "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", "\n", "try:\n", - " result = run_test(\"my_pii_demo.PIICustomTest\")\n", + " result = run_test(\"pii_demo.PIICustomTest\")\n", "\n", " # check if the description was generated\n", " if not result._was_description_generated:\n", @@ -285,9 +307,9 @@ "\n", " # Try logging (this triggers PII checks before upload)\n", " result.log()\n", - " print(\"Logging to API succeeded\")\n", + " print(\"No PII detected: Test results logged to the ValidMind Platform\")\n", "except Exception as e:\n", - " print(\"Blocked: Test Result was not logged due to PII\")" + " print(\"PII detected: Test results not logged to the ValidMind Platform\")" ] }, { @@ -310,7 +332,7 @@ "print(\"\\n=== Mode: test_results ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", "try:\n", - " result = run_test(\"my_pii_demo.PIICustomTest\")\n", + " result = run_test(\"pii_demo.PIICustomTest\")\n", "\n", " # check if the description was generated\n", " if not result._was_description_generated:\n", @@ -345,7 +367,7 @@ "print(\"\\n=== Mode: test_descriptions ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", "try:\n", - " result = run_test(\"my_pii_demo.PIICustomTest\")\n", + " result = run_test(\"pii_demo.PIICustomTest\")\n", "\n", " # check if the description was generated\n", " if not result._was_description_generated:\n", @@ -380,7 +402,7 @@ "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "try:\n", - " result = run_test(\"my_pii_demo.PIICustomTest\")\n", + " result = run_test(\"pii_demo.PIICustomTest\")\n", "\n", " # check if the description was generated\n", " if not result._was_description_generated:\n", From e0cd4a5fd74f3bce400c7fd7bbba10d52f46267f Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 10:45:45 -0700 Subject: [PATCH 16/37] Editing --- .../code_sharing/enable_pii_detection.ipynb | 67 +++---------------- 1 file changed, 9 insertions(+), 58 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 4fd1be0db..379c327ab 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -13,7 +13,7 @@ "id": "d4721b73", "metadata": {}, "source": [ - "Learn how to enable Personally Identifiable Information (PII) detection when generating test results with the ValidMind Library, and configure safety overrides for specific test results." + "Learn how to enable and configure Personally Identifiable Information (PII) detection when generating test results with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform." ] }, { @@ -205,7 +205,7 @@ "\n", "from validmind import test\n", "\n", - "@test(\"pii_demo.PIICustomTest\")\n", + "@test(\"pii_demo.PIIDetectionTest\")\n", "def pii_custom_test():\n", " \"\"\"A demo test that returns PII\"\"\"\n", " return pd.DataFrame(\n", @@ -238,7 +238,7 @@ "source": [ "### Run test under different PII detection modes\n", "\n", - "Now, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test:" + "Now, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test via a function called `run_pii_test()`:" ] }, { @@ -253,7 +253,7 @@ "\n", "def run_pii_test():\n", " try:\n", - " result = run_test(\"pii_demo.PIICustomTest\")\n", + " result = run_test(\"pii_demo.PIIDetectionTest\")\n", "\n", " # Check if the test description was generated by LLM\n", " if not result._was_description_generated:\n", @@ -296,20 +296,7 @@ "print(\"\\n=== Mode: disabled ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", "\n", - "try:\n", - " result = run_test(\"pii_demo.PIICustomTest\")\n", - "\n", - " # check if the description was generated\n", - " if not result._was_description_generated:\n", - " print(\"PII detected: LLM-generated test description skipped\")\n", - " else:\n", - " print(\"No PII detected: Test description generated by LLM\")\n", - "\n", - " # Try logging (this triggers PII checks before upload)\n", - " result.log()\n", - " print(\"No PII detected: Test results logged to the ValidMind Platform\")\n", - "except Exception as e:\n", - " print(\"PII detected: Test results not logged to the ValidMind Platform\")" + "run_pii_test()" ] }, { @@ -331,20 +318,8 @@ "source": [ "print(\"\\n=== Mode: test_results ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", - "try:\n", - " result = run_test(\"pii_demo.PIICustomTest\")\n", - "\n", - " # check if the description was generated\n", - " if not result._was_description_generated:\n", - " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " else:\n", - " print(\"Description was generated by LLM\")\n", "\n", - " # Try logging (this triggers PII checks before upload)\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - "except Exception as e:\n", - " print(\"Blocked: Test Result was not logged due to PII\")" + "run_pii_test()" ] }, { @@ -366,20 +341,8 @@ "source": [ "print(\"\\n=== Mode: test_descriptions ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", - "try:\n", - " result = run_test(\"pii_demo.PIICustomTest\")\n", "\n", - " # check if the description was generated\n", - " if not result._was_description_generated:\n", - " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " else:\n", - " print(\"Description was generated by LLM\")\n", - "\n", - " # Try logging (this triggers PII checks before upload)\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - "except Exception as e:\n", - " print(\"Blocked: Test Result was not logged due to PII\")" + "run_pii_test()" ] }, { @@ -401,20 +364,8 @@ "source": [ "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", - "try:\n", - " result = run_test(\"pii_demo.PIICustomTest\")\n", - "\n", - " # check if the description was generated\n", - " if not result._was_description_generated:\n", - " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " else:\n", - " print(\"Description was generated by LLM\")\n", - "\n", - " # Try logging (this triggers PII checks before upload)\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - "except Exception as e:\n", - " print(\"Blocked: Test Result was not logged due to PII\")" + "\n", + "run_pii_test()" ] }, { From 527274ccadac609d9e43bc5a702f646b265fb1c2 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 10:50:54 -0700 Subject: [PATCH 17/37] result_id --- notebooks/code_sharing/enable_pii_detection.ipynb | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 379c327ab..1be73aa8d 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -251,19 +251,21 @@ "import os\n", "from validmind.tests import run_test\n", "\n", - "def run_pii_test():\n", + "# Run test and tag result with unique `result_id`\n", + "def run_pii_test(result_id=\"\"):\n", " try:\n", - " result = run_test(\"pii_demo.PIIDetectionTest\")\n", + " test_name = f\"pii_demo.PIIDetectionTest:{result_id}\"\n", + " result = run_test(test_name)\n", "\n", " # Check if the test description was generated by LLM\n", " if not result._was_description_generated:\n", - " print(\"PII detected: LLM-generated test description skipped\")\n", + " print(\"PII detected or detection disabled: LLM-generated test description skipped\")\n", " else:\n", " print(\"No PII detected: Test description generated by LLM\")\n", "\n", " # Try logging test results to the ValidMind Platform\n", " result.log()\n", - " print(\"No PII detected: Test results logged to the ValidMind Platform\")\n", + " print(\"No PII detected or detection disabled:: Test results logged to the ValidMind Platform\")\n", " except Exception as e:\n", " print(\"PII detected: Test results not logged to the ValidMind Platform\")" ] From e4420664fbe2fc1ed4c23927b4717014c43ca401 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 10:57:42 -0700 Subject: [PATCH 18/37] Editing --- .../code_sharing/enable_pii_detection.ipynb | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 1be73aa8d..b4d667c81 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -275,7 +275,7 @@ "id": "e349b5f5", "metadata": {}, "source": [ - "We'll then switch `VALIDMIND_PII_DETECTION` across modes when we run our custom test and catch exceptions to observe blocking behavior." + "We'll then switch `VALIDMIND_PII_DETECTION` across modes when we run our custom test and catch exceptions to observe blocking behavior when PII is present." ] }, { @@ -285,7 +285,7 @@ "source": [ "#### disabled\n", "\n", - "When PII checks are disabled, tests run and generate test descriptions. Logging tests with [`.log()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) will also send test results to the ValidMind Platform as usual:" + "When detection is set to `disabled`, tests run and generate test descriptions. Logging tests with [`.log()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) will also send test results to the ValidMind Platform as usual:" ] }, { @@ -298,7 +298,7 @@ "print(\"\\n=== Mode: disabled ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", "\n", - "run_pii_test()" + "run_pii_test(\"disabled\")" ] }, { @@ -308,7 +308,7 @@ "source": [ "#### test_results\n", "\n", - "Table PII triggers blocking; description may still proceed:" + "When detection is set for `test_results`, tests run and generate test descriptions for review in your environment, but logging tests will not send descriptions or test results to the ValidMind Platform:" ] }, { @@ -321,7 +321,7 @@ "print(\"\\n=== Mode: test_results ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", "\n", - "run_pii_test()" + "run_pii_test(\"test_results\")" ] }, { @@ -331,7 +331,7 @@ "source": [ "#### test_descriptions\n", "\n", - "Description PII triggers blocking; tables may still proceed:" + "When detection is set for `test_descriptions`, tests run but will not generate test descriptions, and logging tests will not send descriptions but will send test results to the ValidMind Platform:" ] }, { @@ -344,7 +344,7 @@ "print(\"\\n=== Mode: test_descriptions ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", "\n", - "run_pii_test()" + "run_pii_test(\"test_descriptions\")" ] }, { @@ -354,7 +354,7 @@ "source": [ "#### all\n", "\n", - "Both table and description checks are enforced:" + "When detection is set to `all`, tests run will not generate test descriptions or log test results to the ValidMind Platform." ] }, { @@ -367,7 +367,7 @@ "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "\n", - "run_pii_test()" + "run_pii_test(\"all\")" ] }, { From 7a9c5b83e0b671c459e647b36f74800e52c79ac5 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:01:14 -0700 Subject: [PATCH 19/37] Save point --- notebooks/code_sharing/enable_pii_detection.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index b4d667c81..27f66c21e 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -265,7 +265,7 @@ "\n", " # Try logging test results to the ValidMind Platform\n", " result.log()\n", - " print(\"No PII detected or detection disabled:: Test results logged to the ValidMind Platform\")\n", + " print(\"No PII detected or detection disabled: Test results logged to the ValidMind Platform\")\n", " except Exception as e:\n", " print(\"PII detected: Test results not logged to the ValidMind Platform\")" ] @@ -285,7 +285,7 @@ "source": [ "#### disabled\n", "\n", - "When detection is set to `disabled`, tests run and generate test descriptions. Logging tests with [`.log()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) will also send test results to the ValidMind Platform as usual:" + "When detection is set to `disabled`, tests run and generate test descriptions. Logging tests with [`.log()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) will also send test descriptions and test results to the ValidMind Platform as usual:" ] }, { From 9fd6c66a1a088b4b76ed93a355d372f9a967e454 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:09:09 -0700 Subject: [PATCH 20/37] Save point --- notebooks/code_sharing/enable_pii_detection.ipynb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 27f66c21e..fbb37e70d 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -259,9 +259,9 @@ "\n", " # Check if the test description was generated by LLM\n", " if not result._was_description_generated:\n", - " print(\"PII detected or detection disabled: LLM-generated test description skipped\")\n", + " print(\"PII detected: LLM-generated test description skipped\")\n", " else:\n", - " print(\"No PII detected: Test description generated by LLM\")\n", + " print(\"No PII detected or detection disabled: Test description generated by LLM\")\n", "\n", " # Try logging test results to the ValidMind Platform\n", " result.log()\n", @@ -275,7 +275,12 @@ "id": "e349b5f5", "metadata": {}, "source": [ - "We'll then switch `VALIDMIND_PII_DETECTION` across modes when we run our custom test and catch exceptions to observe blocking behavior when PII is present." + "We'll then switch `VALIDMIND_PII_DETECTION` across modes when we run our custom test and catch exceptions to observe blocking behavior when PII is present.\n", + "\n", + "\n", + "
Note that since we are running a custom test that does not exist in your model's default documentation template, we'll receive output indicating that a test-driven block doesn't currently exist in your model's documentation for that particular test ID.\n", + "

\n", + "That's expected, as when we run custom tests the results logged need to be manually added to your documentation within the ValidMind Platform or added to your documentation template.
" ] }, { From 3b7a5e269f57217766f29791dca088aae19d932e Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:12:09 -0700 Subject: [PATCH 21/37] Save point --- notebooks/code_sharing/enable_pii_detection.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index fbb37e70d..fbb4b7dd7 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -205,7 +205,7 @@ "\n", "from validmind import test\n", "\n", - "@test(\"pii_demo.PIIDetectionTest\")\n", + "@test(\"pii_demo.PIIDetection\")\n", "def pii_custom_test():\n", " \"\"\"A demo test that returns PII\"\"\"\n", " return pd.DataFrame(\n", @@ -254,7 +254,7 @@ "# Run test and tag result with unique `result_id`\n", "def run_pii_test(result_id=\"\"):\n", " try:\n", - " test_name = f\"pii_demo.PIIDetectionTest:{result_id}\"\n", + " test_name = f\"pii_demo.PIIDetection:{result_id}\"\n", " result = run_test(test_name)\n", "\n", " # Check if the test description was generated by LLM\n", From d093f3145b4465f0547c46144aa6226c00ecdd9f Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:16:14 -0700 Subject: [PATCH 22/37] Edit --- notebooks/code_sharing/enable_pii_detection.ipynb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index fbb4b7dd7..0034ffd43 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -207,7 +207,8 @@ "\n", "@test(\"pii_demo.PIIDetection\")\n", "def pii_custom_test():\n", - " \"\"\"A demo test that returns PII\"\"\"\n", + " \"\"\"A custom test that returns demo PII.\n", + " This default test description will display when PII is not sent to the LLM to generate test descriptions based on test result data.\"\"\"\n", " return pd.DataFrame(\n", " {\n", " \"name\": [\"Jane Smith\", \"John Doe\", \"Alice Johnson\"],\n", From 011fc709d725eb976b3d4b6b02aa55c803bf1abb Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:28:52 -0700 Subject: [PATCH 23/37] Edit... --- notebooks/code_sharing/enable_pii_detection.ipynb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 0034ffd43..59614c25f 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -325,7 +325,7 @@ "outputs": [], "source": [ "print(\"\\n=== Mode: test_results ===\")\n", - "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"results_blocked\"\n", "\n", "run_pii_test(\"test_results\")" ] @@ -348,7 +348,7 @@ "outputs": [], "source": [ "print(\"\\n=== Mode: test_descriptions ===\")\n", - "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"descriptions_blocked\"\n", "\n", "run_pii_test(\"test_descriptions\")" ] @@ -371,7 +371,7 @@ "outputs": [], "source": [ "print(\"\\n=== Mode: all ===\")\n", - "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"results_descriptions_blocked\"\n", "\n", "run_pii_test(\"all\")" ] From 56d60c2222fbee950a6abf94f29e4bcb0514e7d4 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:36:54 -0700 Subject: [PATCH 24/37] ... --- .../code_sharing/enable_pii_detection.ipynb | 23 +++++++++++-------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 59614c25f..3e3d19cd5 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -186,7 +186,7 @@ "source": [ "### Create a custom test that outputs PII\n", "\n", - "To demonstrate the feature, we'll need a test that outputs PII. We'll create a custom test that returns:\n", + "To demonstrate the feature, we'll need a test that outputs PII. First we'll create a custom test that returns:\n", "\n", "- A description string containing PII (name, email, phone)\n", "- A small table containing PII in columns\n", @@ -239,7 +239,7 @@ "source": [ "### Run test under different PII detection modes\n", "\n", - "Now, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test via a function called `run_pii_test()`:" + "Next, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test via a function called `run_pii_test()` that catches exceptions to observe blocking behavior when PII is present:" ] }, { @@ -276,8 +276,7 @@ "id": "e349b5f5", "metadata": {}, "source": [ - "We'll then switch `VALIDMIND_PII_DETECTION` across modes when we run our custom test and catch exceptions to observe blocking behavior when PII is present.\n", - "\n", + "We'll then switch `VALIDMIND_PII_DETECTION` across modes in the below examples.\n", "\n", "
Note that since we are running a custom test that does not exist in your model's default documentation template, we'll receive output indicating that a test-driven block doesn't currently exist in your model's documentation for that particular test ID.\n", "

\n", @@ -304,6 +303,7 @@ "print(\"\\n=== Mode: disabled ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"disabled\"\n", "\n", + "# Run test and tag result with unique ID `disabled`\n", "run_pii_test(\"disabled\")" ] }, @@ -325,9 +325,10 @@ "outputs": [], "source": [ "print(\"\\n=== Mode: test_results ===\")\n", - "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"results_blocked\"\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", "\n", - "run_pii_test(\"test_results\")" + "# Run test and tag result with unique ID `results_blocked`\n", + "run_pii_test(\"results_blocked\")" ] }, { @@ -348,9 +349,10 @@ "outputs": [], "source": [ "print(\"\\n=== Mode: test_descriptions ===\")\n", - "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"descriptions_blocked\"\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_descriptions\"\n", "\n", - "run_pii_test(\"test_descriptions\")" + "# Run test and tag result with unique ID `desc_blocked`\n", + "run_pii_test(\"desc_blocked\")" ] }, { @@ -371,9 +373,10 @@ "outputs": [], "source": [ "print(\"\\n=== Mode: all ===\")\n", - "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"results_descriptions_blocked\"\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "\n", - "run_pii_test(\"all\")" + "# Run test and tag result with unique ID `results_desc_blocked`\n", + "run_pii_test(\"results_desc_blocked\")" ] }, { From 78486876cd934119998a6140371eb92c5268409f Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:44:04 -0700 Subject: [PATCH 25/37] . --- notebooks/code_sharing/enable_pii_detection.ipynb | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 3e3d19cd5..840976914 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -375,8 +375,8 @@ "print(\"\\n=== Mode: all ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "\n", - "# Run test and tag result with unique ID `results_desc_blocked`\n", - "run_pii_test(\"results_desc_blocked\")" + "# Run test and tag result with unique ID `all_blocked`\n", + "run_pii_test(\"all_blocked\")" ] }, { @@ -404,11 +404,19 @@ "id": "310f69e1", "metadata": {}, "source": [ - "### Test only a subset\n", + "#### Test only a subset\n", "\n", "To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop:" ] }, + { + "cell_type": "markdown", + "id": "9ab973f2", + "metadata": {}, + "source": [ + "### Review logged test results" + ] + }, { "cell_type": "markdown", "id": "53a1a62d", From 39d3a0643708308222b8ec6a0fe10966d08d0509 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:48:18 -0700 Subject: [PATCH 26/37] override --- .../code_sharing/enable_pii_detection.ipynb | 29 ++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 840976914..4ae55a833 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -396,7 +396,34 @@ "\n", "You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", "\n", - "For example," + "For example, let's rerun our custom test with detection set to `all`, then override the blocking:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c8dac20b", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"\\n=== Mode: all & unsafe=True ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", + "\n", + "# Run test and tag result with unique ID `unsafe_true`\n", + "try:\n", + " result = run_test(\"pii_demo.PIIDetection:unsafe_true\")\n", + "\n", + " # Check if the test description was generated by LLM\n", + " if not result._was_description_generated:\n", + " print(\"PII detected: LLM-generated test description skipped\")\n", + " else:\n", + " print(\"No PII detected, detection disabled, or override set: Test description generated by LLM\")\n", + "\n", + " # Try logging test results to the ValidMind Platform\n", + " result.log()\n", + " print(\"No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform\")\n", + "except Exception as e:\n", + " print(\"PII detected: Test results not logged to the ValidMind Platform\")" ] }, { From 6c7cab2e1bc17232412d517130931ceeece68779 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 11:58:01 -0700 Subject: [PATCH 27/37] Ugh --- notebooks/code_sharing/enable_pii_detection.ipynb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 4ae55a833..4ed04d33d 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -276,7 +276,7 @@ "id": "e349b5f5", "metadata": {}, "source": [ - "We'll then switch `VALIDMIND_PII_DETECTION` across modes in the below examples.\n", + "We'll then switch the `VALIDMIND_PII_DETECTION` environment variable across modes in the below examples.\n", "\n", "
Note that since we are running a custom test that does not exist in your model's default documentation template, we'll receive output indicating that a test-driven block doesn't currently exist in your model's documentation for that particular test ID.\n", "

\n", @@ -411,7 +411,7 @@ "\n", "# Run test and tag result with unique ID `unsafe_true`\n", "try:\n", - " result = run_test(\"pii_demo.PIIDetection:unsafe_true\")\n", + " result = run_test(\"pii_demo.PIIDetection:all_unsafe_true\")\n", "\n", " # Check if the test description was generated by LLM\n", " if not result._was_description_generated:\n", @@ -420,7 +420,7 @@ " print(\"No PII detected, detection disabled, or override set: Test description generated by LLM\")\n", "\n", " # Try logging test results to the ValidMind Platform\n", - " result.log()\n", + " result.log(unsafe=True)\n", " print(\"No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform\")\n", "except Exception as e:\n", " print(\"PII detected: Test results not logged to the ValidMind Platform\")" From 0ae145d2eafe7b1b554e6393957db54cd9dcd7f7 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 12:08:40 -0700 Subject: [PATCH 28/37] Save point --- .../code_sharing/enable_pii_detection.ipynb | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 4ed04d33d..c9b8d6ab4 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -409,9 +409,9 @@ "print(\"\\n=== Mode: all & unsafe=True ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "\n", - "# Run test and tag result with unique ID `unsafe_true`\n", + "# Run test and tag result with unique ID `override`\n", "try:\n", - " result = run_test(\"pii_demo.PIIDetection:all_unsafe_true\")\n", + " result = run_test(\"pii_demo.PIIDetection:override\")\n", "\n", " # Check if the test description was generated by LLM\n", " if not result._was_description_generated:\n", @@ -431,7 +431,7 @@ "id": "310f69e1", "metadata": {}, "source": [ - "#### Test only a subset\n", + "#### Test only a subset*\n", "\n", "To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop:" ] @@ -441,7 +441,23 @@ "id": "9ab973f2", "metadata": {}, "source": [ - "### Review logged test results" + "### Review logged test results\n", + "\n", + "Now let's take a look at the results that were logged to the ValidMind Platform:\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "\n", + "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "\n", + "3. Click on any section heading to expand that section to add a new test-driven block ([Need more help?](https://docs.validmind.ai/developer/model-documentation/work-with-test-results.html)).\n", + "\n", + "4. Under TEST-DRIVEN in the sidebar, click **Custom**.\n", + "\n", + "5. Confirm that you're able to insert the following logged results:\n", + "\n", + " - `pii_demo.PIIDetection:disabled`\n", + " - `pii_demo.PIIDetection:desc_blocked`\n", + " - `pii_demo.PIIDetection:override`" ] }, { From 6d5105ab93aa84dcfaae9bcfe96421b003e17f4e Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 13:56:10 -0700 Subject: [PATCH 29/37] Finishing up? --- .../code_sharing/enable_pii_detection.ipynb | 52 +++++++++++++------ 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index c9b8d6ab4..30435bd57 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -379,24 +379,16 @@ "run_pii_test(\"all_blocked\")" ] }, - { - "cell_type": "markdown", - "id": "d2c36322", - "metadata": {}, - "source": [ - "### Detection adjustments" - ] - }, { "cell_type": "markdown", "id": "284e69e5", "metadata": {}, "source": [ - "#### Overriding detection\n", + "### Override detection\n", "\n", "You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", "\n", - "For example, let's rerun our custom test with detection set to `all`, then override the blocking:" + "To demonstrate, let's rerun our custom test with some override scenarios. First, let's rerun our custom test with detection set to `all`, which will send the test results but not the test descriptions to the ValidMind Platform:" ] }, { @@ -409,15 +401,15 @@ "print(\"\\n=== Mode: all & unsafe=True ===\")\n", "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"all\"\n", "\n", - "# Run test and tag result with unique ID `override`\n", + "# Run test and tag result with unique ID `override_results`\n", "try:\n", - " result = run_test(\"pii_demo.PIIDetection:override\")\n", + " result = run_test(\"pii_demo.PIIDetection:override_results\")\n", "\n", " # Check if the test description was generated by LLM\n", " if not result._was_description_generated:\n", " print(\"PII detected: LLM-generated test description skipped\")\n", " else:\n", - " print(\"No PII detected, detection disabled, or override set: Test description generated by LLM\")\n", + " print(\"No PII detected or detection disabled: Test description generated by LLM\")\n", "\n", " # Try logging test results to the ValidMind Platform\n", " result.log(unsafe=True)\n", @@ -428,12 +420,37 @@ }, { "cell_type": "markdown", - "id": "310f69e1", + "id": "b2b4c7ab", + "metadata": {}, + "source": [ + "To send both the test descriptions and test results via override, set the `VALIDMIND_PII_DETECTION` environment variable to `test_results` while including the override flag:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3aeef63b", "metadata": {}, + "outputs": [], "source": [ - "#### Test only a subset*\n", + "print(\"\\n=== Mode: test_results & unsafe=True ===\")\n", + "os.environ[\"VALIDMIND_PII_DETECTION\"] = \"test_results\"\n", + "\n", + "# Run test and tag result with unique ID `override_both`\n", + "try:\n", + " result = run_test(\"pii_demo.PIIDetection:override_both\")\n", "\n", - "To test only a subset (tables or descriptions), adjust the test to emit only that type and re-run the mode loop:" + " # Check if the test description was generated by LLM\n", + " if not result._was_description_generated:\n", + " print(\"PII detected: LLM-generated test description skipped\")\n", + " else:\n", + " print(\"No PII detected or detection disabled: Test description generated by LLM\")\n", + "\n", + " # Try logging test results to the ValidMind Platform\n", + " result.log(unsafe=True)\n", + " print(\"No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform\")\n", + "except Exception as e:\n", + " print(\"PII detected: Test results not logged to the ValidMind Platform\")" ] }, { @@ -457,7 +474,8 @@ "\n", " - `pii_demo.PIIDetection:disabled`\n", " - `pii_demo.PIIDetection:desc_blocked`\n", - " - `pii_demo.PIIDetection:override`" + " - `pii_demo.PIIDetection:override_results`\n", + " - `pii_demo.PIIDetection:override_both`" ] }, { From 688ac945541c2a5e8f11ce6b680d06d5aa3330e0 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 13:57:04 -0700 Subject: [PATCH 30/37] hdgkj --- notebooks/code_sharing/enable_pii_detection.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 30435bd57..7b32c6f8e 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -444,7 +444,7 @@ " if not result._was_description_generated:\n", " print(\"PII detected: LLM-generated test description skipped\")\n", " else:\n", - " print(\"No PII detected or detection disabled: Test description generated by LLM\")\n", + " print(\"No PII detected, detection disabled, or override set: Test description generated by LLM\")\n", "\n", " # Try logging test results to the ValidMind Platform\n", " result.log(unsafe=True)\n", From c20853b7ffe2e08de5928c4dbf7b5e7f3d7959d5 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 14:01:37 -0700 Subject: [PATCH 31/37] Proofreading --- notebooks/code_sharing/enable_pii_detection.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/code_sharing/enable_pii_detection.ipynb index 7b32c6f8e..56ac520e6 100644 --- a/notebooks/code_sharing/enable_pii_detection.ipynb +++ b/notebooks/code_sharing/enable_pii_detection.ipynb @@ -462,7 +462,7 @@ "\n", "Now let's take a look at the results that were logged to the ValidMind Platform:\n", "\n", - "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier.\n", "\n", "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", "\n", From 28f5e7abebb5b00a857776ba1e1774cb85dee3a5 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 14:22:57 -0700 Subject: [PATCH 32/37] Moving file --- .../how_to/configure_pii_detection.ipynb | 189 ------------------ .../enable_pii_detection.ipynb | 0 2 files changed, 189 deletions(-) delete mode 100644 notebooks/how_to/configure_pii_detection.ipynb rename notebooks/{code_sharing => how_to}/enable_pii_detection.ipynb (100%) diff --git a/notebooks/how_to/configure_pii_detection.ipynb b/notebooks/how_to/configure_pii_detection.ipynb deleted file mode 100644 index e9461279b..000000000 --- a/notebooks/how_to/configure_pii_detection.ipynb +++ /dev/null @@ -1,189 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# PII Detection Modes with a Custom Test\n", - "\n", - "This notebook shows how to initialize ValidMind, implement a custom test that emits PII, and observe behavior differences under each `VALIDMIND_PII_DETECTION` mode when running the test with `validmind.tests.run_test`.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Prerequisites\n", - "\n", - "- `validmind` installed with PII extras:\n", - "\n", - "```bash\n", - "%pip install -q validmind[pii-detection]\n", - "```\n", - "\n", - "- A ValidMind model registered. We'll initialize the library using your model snippet.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%pip install -q \"validmind[pii-detection]\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Initialize ValidMind\n", - "\n", - "Initialize using your model code snippet or a `.env` file, as shown in other quickstarts.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Load your model identifier credentials from an `.env` file\n", - "%load_ext dotenv\n", - "%dotenv .env\n", - "\n", - "# Or initialize with your code snippet\n", - "import validmind as vm\n", - "\n", - "vm.init(\n", - " # api_host=\"...\",\n", - " # api_key=\"...\",\n", - " # api_secret=\"...\",\n", - " # model=\"...\",\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Create a custom test that emits PII\n", - "\n", - "We'll create a custom test that returns:\n", - "- A description string containing PII (name, email, phone)\n", - "- A small table containing PII in columns\n", - "\n", - "This mirrors the structure used in other custom test notebooks and will exercise both table and description PII detection paths.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import pandas as pd\n", - "\n", - "from validmind import test\n", - "\n", - "@test(\"my_pii_demo.PIIEmittingTest\")\n", - "def pii_emitting_test():\n", - " \"\"\"A demo test that returns PII\"\"\"\n", - " return pd.DataFrame(\n", - " {\n", - " \"name\": [\"Jane Smith\", \"John Doe\", \"Alice Johnson\"],\n", - " \"email\": [\n", - " \"jane.smith@bank.example\",\n", - " \"john.doe@company.example\",\n", - " \"alice.johnson@service.example\",\n", - " ],\n", - " \"phone\": [\"(212) 555-9876\", \"(415) 555-1234\", \"(646) 555-5678\"],\n", - " }\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Run the test under different PII detection modes\n", - "\n", - "We'll switch `VALIDMIND_PII_DETECTION` across modes and run the same test with `validmind.tests.run_test`. We catch exceptions to observe blocking behavior.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "from validmind.tests import run_test\n", - "\n", - "MODES = [\"disabled\", \"test_results\", \"test_descriptions\", \"all\"]\n", - "\n", - "for mode in MODES:\n", - " print(\"\\n=== Mode:\", mode, \"===\")\n", - " os.environ[\"VALIDMIND_PII_DETECTION\"] = mode\n", - " try:\n", - " result = run_test(\"my_pii_demo.PIIEmittingTest\")\n", - "\n", - " # check if the description was generated\n", - " if not result._was_description_generated:\n", - " print(\"Blocked: Test Description Generation was not run due to PII\")\n", - " else:\n", - " print(\"Description was generated by LLM\")\n", - "\n", - " # Try logging (this triggers PII checks before upload)\n", - " result.log()\n", - " print(\"Logging to API succeeded\")\n", - " except Exception as e:\n", - " print(\"Blocked: Test Result was not logged due to PII\")\n", - " # print(e)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Expected behavior by mode\n", - "\n", - "- disabled: No PII checks.\n", - "- test_results: Description is generated but result is not logged.\n", - "- test_descriptions: Description generation is blocked but result is logged.\n", - "- all: Description generation and logging are both blocked.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Notes\n", - "\n", - "- If you see warnings that Presidio is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", - "- You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": ".venv", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.11" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/notebooks/code_sharing/enable_pii_detection.ipynb b/notebooks/how_to/enable_pii_detection.ipynb similarity index 100% rename from notebooks/code_sharing/enable_pii_detection.ipynb rename to notebooks/how_to/enable_pii_detection.ipynb From 150304144cc5b636162c9d7f183a14861d769013 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 14:24:32 -0700 Subject: [PATCH 33/37] Edit --- notebooks/how_to/enable_pii_detection.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/how_to/enable_pii_detection.ipynb b/notebooks/how_to/enable_pii_detection.ipynb index 56ac520e6..7e97a93a5 100644 --- a/notebooks/how_to/enable_pii_detection.ipynb +++ b/notebooks/how_to/enable_pii_detection.ipynb @@ -5,7 +5,7 @@ "id": "dd37d6f9", "metadata": {}, "source": [ - "# Enable PII detection in test results" + "# Enable PII detection in tests" ] }, { From b5c7bae32bf93d35eaeab7a2e52fe528797748af Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 14:24:50 -0700 Subject: [PATCH 34/37] UGH --- notebooks/how_to/enable_pii_detection.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/how_to/enable_pii_detection.ipynb b/notebooks/how_to/enable_pii_detection.ipynb index 7e97a93a5..08db8cd6a 100644 --- a/notebooks/how_to/enable_pii_detection.ipynb +++ b/notebooks/how_to/enable_pii_detection.ipynb @@ -13,7 +13,7 @@ "id": "d4721b73", "metadata": {}, "source": [ - "Learn how to enable and configure Personally Identifiable Information (PII) detection when generating test results with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform." + "Learn how to enable and configure Personally Identifiable Information (PII) detection when running test results with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform." ] }, { From 1c467f0984019a0536afd796bdf89716714accd5 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 14:25:00 -0700 Subject: [PATCH 35/37] sdjfksd --- notebooks/how_to/enable_pii_detection.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/how_to/enable_pii_detection.ipynb b/notebooks/how_to/enable_pii_detection.ipynb index 08db8cd6a..208f37d21 100644 --- a/notebooks/how_to/enable_pii_detection.ipynb +++ b/notebooks/how_to/enable_pii_detection.ipynb @@ -13,7 +13,7 @@ "id": "d4721b73", "metadata": {}, "source": [ - "Learn how to enable and configure Personally Identifiable Information (PII) detection when running test results with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform." + "Learn how to enable and configure Personally Identifiable Information (PII) detection when running tests with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform." ] }, { From b80f9479077a17b651a77b7254e8bd77cea70ea5 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 14:26:27 -0700 Subject: [PATCH 36/37] ToC --- notebooks/how_to/enable_pii_detection.ipynb | 133 +++++++++++++++----- 1 file changed, 99 insertions(+), 34 deletions(-) diff --git a/notebooks/how_to/enable_pii_detection.ipynb b/notebooks/how_to/enable_pii_detection.ipynb index 208f37d21..910cb0e19 100644 --- a/notebooks/how_to/enable_pii_detection.ipynb +++ b/notebooks/how_to/enable_pii_detection.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "dd37d6f9", + "id": "766eb526", "metadata": {}, "source": [ "# Enable PII detection in tests" @@ -10,7 +10,7 @@ }, { "cell_type": "markdown", - "id": "d4721b73", + "id": "fec7c460", "metadata": {}, "source": [ "Learn how to enable and configure Personally Identifiable Information (PII) detection when running tests with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform." @@ -18,9 +18,49 @@ }, { "cell_type": "markdown", - "id": "about-intro-85b40b47-34c7-4da4-b0e7-c491341a9f20", + "id": "fb872491", + "metadata": {}, + "source": [ + "::: {.content-hidden when-format=\"html\"}\n", + "## Contents \n", + "- [About ValidMind](#toc1__) \n", + " - [Before you begin](#toc1_1__) \n", + " - [New to ValidMind?](#toc1_2__) \n", + " - [Key concepts](#toc1_3__) \n", + "- [Setting up](#toc2__) \n", + " - [Install the ValidMind Library with PII detection](#toc2_1__) \n", + " - [Initialize the ValidMind Library](#toc2_2__) \n", + " - [Get your code snippet](#toc2_2_1__) \n", + "- [Using PII detection](#toc3__) \n", + " - [Create a custom test that outputs PII](#toc3_1__) \n", + " - [Run test under different PII detection modes](#toc3_2__) \n", + " - [disabled](#toc3_2_1__) \n", + " - [test_results](#toc3_2_2__) \n", + " - [test_descriptions](#toc3_2_3__) \n", + " - [all](#toc3_2_4__) \n", + " - [Override detection](#toc3_3__) \n", + " - [Review logged test results](#toc3_4__) \n", + "- [Troubleshooting](#toc4__) \n", + "- [Learn more](#toc5__) \n", + "\n", + ":::\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "3df66161", "metadata": {}, "source": [ + "\n", + "\n", "## About ValidMind\n", "\n", "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", @@ -30,9 +70,11 @@ }, { "cell_type": "markdown", - "id": "about-begin-c86ec11e-4e52-4799-bf11-fea595c2671e", + "id": "3231a54b", "metadata": {}, "source": [ + "\n", + "\n", "### Before you begin\n", "\n", "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", @@ -42,9 +84,11 @@ }, { "cell_type": "markdown", - "id": "about-signup-db4c0a03-ea37-4e81-b417-2e79eed1bc53", + "id": "035c6116", "metadata": {}, "source": [ + "\n", + "\n", "### New to ValidMind?\n", "\n", "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", @@ -56,9 +100,11 @@ }, { "cell_type": "markdown", - "id": "about-concepts-fd7b60a6-60a8-420a-8ec2-287874cae377", + "id": "2325efaf", "metadata": {}, "source": [ + "\n", + "\n", "### Key concepts\n", "\n", "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", @@ -89,17 +135,20 @@ }, { "cell_type": "markdown", - "id": "33f87f48", + "id": "279d6958", "metadata": {}, "source": [ + "\n", + "\n", "## Setting up" ] }, { "cell_type": "markdown", - "id": "install-library-f5c5d59c-e356-4f22-a71e-d0c84c269273", "metadata": {}, "source": [ + "\n", + "\n", "### Install the ValidMind Library with PII detection\n", "\n", "
Recommended Python versions\n", @@ -112,7 +161,7 @@ { "cell_type": "code", "execution_count": null, - "id": "install-python-6511976a-8cb2-423d-bfbf-a6e39cc90e4a", + "id": "b830ae91", "metadata": {}, "outputs": [], "source": [ @@ -121,9 +170,11 @@ }, { "cell_type": "markdown", - "id": "install-initialize-e13c0197-f742-453c-9422-ebae53ddcf68", + "id": "1d15779d", "metadata": {}, "source": [ + "\n", + "\n", "### Initialize the ValidMind Library\n", "\n", "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook." @@ -131,9 +182,10 @@ }, { "cell_type": "markdown", - "id": "install-snippet-dc2c3a8b-a2d8-4632-84f9-bb5c9dd8721c", "metadata": {}, "source": [ + "\n", + "\n", "#### Get your code snippet\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", @@ -150,7 +202,7 @@ { "cell_type": "code", "execution_count": null, - "id": "install-init-43c8827f-b023-4324-be38-c58c999d4575", + "id": "eeda4c8c", "metadata": {}, "outputs": [], "source": [ @@ -173,17 +225,20 @@ }, { "cell_type": "markdown", - "id": "9296d9c8", + "id": "4acf7d3c", "metadata": {}, "source": [ + "\n", + "\n", "## Using PII detection" ] }, { "cell_type": "markdown", - "id": "a5cb8d86", "metadata": {}, "source": [ + "\n", + "\n", "### Create a custom test that outputs PII\n", "\n", "To demonstrate the feature, we'll need a test that outputs PII. First we'll create a custom test that returns:\n", @@ -197,7 +252,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6452da1f", + "id": "04d8c802", "metadata": {}, "outputs": [], "source": [ @@ -224,7 +279,7 @@ }, { "cell_type": "markdown", - "id": "e210b665", + "id": "57a011d9", "metadata": {}, "source": [ "
Want to learn more about custom tests?\n", @@ -234,9 +289,10 @@ }, { "cell_type": "markdown", - "id": "f3e47f7a", "metadata": {}, "source": [ + "\n", + "\n", "### Run test under different PII detection modes\n", "\n", "Next, let's import [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module to run our custom test via a function called `run_pii_test()` that catches exceptions to observe blocking behavior when PII is present:" @@ -245,7 +301,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ef7c2b50", + "id": "b42288e5", "metadata": {}, "outputs": [], "source": [ @@ -273,7 +329,7 @@ }, { "cell_type": "markdown", - "id": "e349b5f5", + "id": "5bf99fc0", "metadata": {}, "source": [ "We'll then switch the `VALIDMIND_PII_DETECTION` environment variable across modes in the below examples.\n", @@ -285,9 +341,10 @@ }, { "cell_type": "markdown", - "id": "b39d6c1e", "metadata": {}, "source": [ + "\n", + "\n", "#### disabled\n", "\n", "When detection is set to `disabled`, tests run and generate test descriptions. Logging tests with [`.log()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) will also send test descriptions and test results to the ValidMind Platform as usual:" @@ -296,7 +353,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0c6cbcba", + "id": "3078af64", "metadata": {}, "outputs": [], "source": [ @@ -309,9 +366,10 @@ }, { "cell_type": "markdown", - "id": "dcaf5605", "metadata": {}, "source": [ + "\n", + "\n", "#### test_results\n", "\n", "When detection is set for `test_results`, tests run and generate test descriptions for review in your environment, but logging tests will not send descriptions or test results to the ValidMind Platform:" @@ -320,7 +378,7 @@ { "cell_type": "code", "execution_count": null, - "id": "06c3f4f9", + "id": "12e61a80", "metadata": {}, "outputs": [], "source": [ @@ -333,9 +391,10 @@ }, { "cell_type": "markdown", - "id": "808f95fb", "metadata": {}, "source": [ + "\n", + "\n", "#### test_descriptions\n", "\n", "When detection is set for `test_descriptions`, tests run but will not generate test descriptions, and logging tests will not send descriptions but will send test results to the ValidMind Platform:" @@ -344,7 +403,7 @@ { "cell_type": "code", "execution_count": null, - "id": "46636a1c", + "id": "feba6207", "metadata": {}, "outputs": [], "source": [ @@ -357,9 +416,10 @@ }, { "cell_type": "markdown", - "id": "6121b341", "metadata": {}, "source": [ + "\n", + "\n", "#### all\n", "\n", "When detection is set to `all`, tests run will not generate test descriptions or log test results to the ValidMind Platform." @@ -368,7 +428,7 @@ { "cell_type": "code", "execution_count": null, - "id": "cb627f9a", + "id": "af5040b5", "metadata": {}, "outputs": [], "source": [ @@ -381,9 +441,10 @@ }, { "cell_type": "markdown", - "id": "284e69e5", "metadata": {}, "source": [ + "\n", + "\n", "### Override detection\n", "\n", "You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", @@ -394,7 +455,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c8dac20b", + "id": "0387be21", "metadata": {}, "outputs": [], "source": [ @@ -420,7 +481,6 @@ }, { "cell_type": "markdown", - "id": "b2b4c7ab", "metadata": {}, "source": [ "To send both the test descriptions and test results via override, set the `VALIDMIND_PII_DETECTION` environment variable to `test_results` while including the override flag:" @@ -429,7 +489,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3aeef63b", + "id": "b40a2670", "metadata": {}, "outputs": [], "source": [ @@ -455,9 +515,11 @@ }, { "cell_type": "markdown", - "id": "9ab973f2", + "id": "7704a4c5", "metadata": {}, "source": [ + "\n", + "\n", "### Review logged test results\n", "\n", "Now let's take a look at the results that were logged to the ValidMind Platform:\n", @@ -480,9 +542,11 @@ }, { "cell_type": "markdown", - "id": "53a1a62d", + "id": "ac2922dd", "metadata": {}, "source": [ + "\n", + "\n", "## Troubleshooting\n", "\n", "- [x] If you see warnings that Presidio or Presidio analyzer is unavailable, ensure you installed extras: `validmind[pii-detection]`.\n", @@ -491,9 +555,10 @@ }, { "cell_type": "markdown", - "id": "next-resources-c5b44493-85c8-491b-8eaa-d1bdfd81bc86", "metadata": {}, "source": [ + "\n", + "\n", "## Learn more\n", "\n", "We offer many interactive notebooks to help you document models:\n", From f2cf7492ffaf9cae51d9b399bb5f3794c2f559af Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Thu, 21 Aug 2025 14:29:29 -0700 Subject: [PATCH 37/37] Blah --- notebooks/how_to/enable_pii_detection.ipynb | 59 +++++++++++++++------ 1 file changed, 44 insertions(+), 15 deletions(-) diff --git a/notebooks/how_to/enable_pii_detection.ipynb b/notebooks/how_to/enable_pii_detection.ipynb index 910cb0e19..a7e123286 100644 --- a/notebooks/how_to/enable_pii_detection.ipynb +++ b/notebooks/how_to/enable_pii_detection.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "766eb526", + "id": "f5b060b7", "metadata": {}, "source": [ "# Enable PII detection in tests" @@ -10,7 +10,7 @@ }, { "cell_type": "markdown", - "id": "fec7c460", + "id": "403b35b1", "metadata": {}, "source": [ "Learn how to enable and configure Personally Identifiable Information (PII) detection when running tests with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform." @@ -18,7 +18,7 @@ }, { "cell_type": "markdown", - "id": "fb872491", + "id": "1f0a64f3", "metadata": {}, "source": [ "::: {.content-hidden when-format=\"html\"}\n", @@ -39,6 +39,8 @@ " - [test_descriptions](#toc3_2_3__) \n", " - [all](#toc3_2_4__) \n", " - [Override detection](#toc3_3__) \n", + " - [Override test result logging](#toc3_3_1__) \n", + " - [Override test descriptions and test result logging](#toc3_3_2__) \n", " - [Review logged test results](#toc3_4__) \n", "- [Troubleshooting](#toc4__) \n", "- [Learn more](#toc5__) \n", @@ -56,7 +58,7 @@ }, { "cell_type": "markdown", - "id": "3df66161", + "id": "68cf8398", "metadata": {}, "source": [ "\n", @@ -70,7 +72,7 @@ }, { "cell_type": "markdown", - "id": "3231a54b", + "id": "ba6bd554", "metadata": {}, "source": [ "\n", @@ -84,7 +86,7 @@ }, { "cell_type": "markdown", - "id": "035c6116", + "id": "ac231640", "metadata": {}, "source": [ "\n", @@ -100,7 +102,7 @@ }, { "cell_type": "markdown", - "id": "2325efaf", + "id": "75626c3d", "metadata": {}, "source": [ "\n", @@ -135,7 +137,7 @@ }, { "cell_type": "markdown", - "id": "279d6958", + "id": "ee06fdea", "metadata": {}, "source": [ "\n", @@ -145,6 +147,7 @@ }, { "cell_type": "markdown", + "id": "1343e1dd", "metadata": {}, "source": [ "\n", @@ -170,7 +173,7 @@ }, { "cell_type": "markdown", - "id": "1d15779d", + "id": "b5eae826", "metadata": {}, "source": [ "\n", @@ -182,6 +185,7 @@ }, { "cell_type": "markdown", + "id": "e6cc7cd2", "metadata": {}, "source": [ "\n", @@ -225,7 +229,7 @@ }, { "cell_type": "markdown", - "id": "4acf7d3c", + "id": "5676cd64", "metadata": {}, "source": [ "\n", @@ -235,6 +239,7 @@ }, { "cell_type": "markdown", + "id": "120644e3", "metadata": {}, "source": [ "\n", @@ -279,7 +284,7 @@ }, { "cell_type": "markdown", - "id": "57a011d9", + "id": "a8ab74cd", "metadata": {}, "source": [ "
Want to learn more about custom tests?\n", @@ -289,6 +294,7 @@ }, { "cell_type": "markdown", + "id": "7d2110d3", "metadata": {}, "source": [ "\n", @@ -329,7 +335,7 @@ }, { "cell_type": "markdown", - "id": "5bf99fc0", + "id": "e3c82205", "metadata": {}, "source": [ "We'll then switch the `VALIDMIND_PII_DETECTION` environment variable across modes in the below examples.\n", @@ -341,6 +347,7 @@ }, { "cell_type": "markdown", + "id": "543b08dc", "metadata": {}, "source": [ "\n", @@ -366,6 +373,7 @@ }, { "cell_type": "markdown", + "id": "490c397c", "metadata": {}, "source": [ "\n", @@ -391,6 +399,7 @@ }, { "cell_type": "markdown", + "id": "c21de2bb", "metadata": {}, "source": [ "\n", @@ -416,6 +425,7 @@ }, { "cell_type": "markdown", + "id": "0e3c9f24", "metadata": {}, "source": [ "\n", @@ -441,6 +451,7 @@ }, { "cell_type": "markdown", + "id": "e30c0ccd", "metadata": {}, "source": [ "\n", @@ -449,7 +460,19 @@ "\n", "You can override blocking by passing `unsafe=True` to `result.log(unsafe=True)`, but this is not recommended outside controlled workflows.\n", "\n", - "To demonstrate, let's rerun our custom test with some override scenarios. First, let's rerun our custom test with detection set to `all`, which will send the test results but not the test descriptions to the ValidMind Platform:" + "To demonstrate, let's rerun our custom test with some override scenarios." + ] + }, + { + "cell_type": "markdown", + "id": "6b5b2df9", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Override test result logging\n", + "\n", + "First, let's rerun our custom test with detection set to `all`, which will send the test results but not the test descriptions to the ValidMind Platform:" ] }, { @@ -481,8 +504,13 @@ }, { "cell_type": "markdown", + "id": "a11072fb", "metadata": {}, "source": [ + "\n", + "\n", + "#### Override test descriptions and test result logging\n", + "\n", "To send both the test descriptions and test results via override, set the `VALIDMIND_PII_DETECTION` environment variable to `test_results` while including the override flag:" ] }, @@ -515,7 +543,7 @@ }, { "cell_type": "markdown", - "id": "7704a4c5", + "id": "017ff3ed", "metadata": {}, "source": [ "\n", @@ -542,7 +570,7 @@ }, { "cell_type": "markdown", - "id": "ac2922dd", + "id": "e60ecaf4", "metadata": {}, "source": [ "\n", @@ -555,6 +583,7 @@ }, { "cell_type": "markdown", + "id": "7e0641e1", "metadata": {}, "source": [ "\n",