[SC-10275] Add Satisfactory vs. Needs Attention Flags to Ongoing Monitoring Tests

notebooks/how_to/log_metrics_over_time.ipynb

@@ -39,7 +39,9 @@
     "  - [Log unit metrics over time](#toc6_2_)    \n",
     "  - [Pass thresholds](#toc6_3_)    \n",
     "  - [Log multiple metrics with custom thresholds](#toc6_4_)    \n",
-    "\n",
+    "  - [Add acceptable performance flag](#toc6_5_)\n",
+    "- [Next steps](#toc7_)\n",
+    "- [Upgrade ValidMind](#toc8_)\n",
     ":::\n",
     "<!-- jn-toc-notebook-config\n",
     "\tnumbering=false\n",
@@ -197,7 +199,6 @@
     "    # api_key=\"...\",\n",
     "    # api_secret=\"...\",\n",
     "    # model=\"...\",\n",
-    "    monitoring = True\n",
     ")"
    ]
   },
@@ -216,7 +217,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -300,7 +301,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -494,7 +495,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -542,7 +543,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -565,7 +566,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -694,13 +695,204 @@
     "![Recall Score](../images/log_metric_recall.png)\n",
     "![F1 Score](../images/log_metric_f1.png)"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='toc6_5_'></a>\n",
+    "\n",
+    "### Add acceptable performance flag\n",
+    "\n",
+    "The `passed` parameter in the `log_metric()` function allows you to explicitly mark whether a specific metric value should be considered \"Satisfactory\" or \"Requires Attention\":\n",
+    "   - When `passed=True`: A green \"Satisfactory\" badge appears on the chart, indicating the metric value meets your acceptance criteria.\n",
+    "   - When `passed=False`: A yellow \"Requires Attention\" badge appears, highlighting potential concerns that may require investigation."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the example below, the `passed=True` parameter adds a green \"Satisfactory\" badge to the GINI Score metric visualization, instantly indicating that the 0.75 value meets acceptable performance standards by being above the `medium_risk` threshold of 0.6:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "log_metric(\n",
+    "    key=\"GINI Score\",\n",
+    "    value=0.75,\n",
+    "    recorded_at=datetime(2025, 6, 7),\n",
+    "    thresholds = {\n",
+    "        \"high_risk\": 0.5,\n",
+    "        \"medium_risk\": 0.6,\n",
+    "        \"low_risk\": 0.8,\n",
+    "    },\n",
+    "    passed=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![GINI Score](../images/log_metric_satisfactory.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this example, the `passed=False` parameter adds a yellow \"Requires Attention\" badge to the GINI Score metric visualization, immediately highlighting that the value of 0.5 fails to meet acceptable performance standards by not exceeding the `medium_risk` threshold of 0.6:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "log_metric(\n",
+    "    key=\"GINI Score\",\n",
+    "    value=0.5,\n",
+    "    recorded_at=datetime(2025, 6, 9),\n",
+    "    thresholds = {\n",
+    "        \"high_risk\": 0.5,\n",
+    "        \"medium_risk\": 0.6,\n",
+    "        \"low_risk\": 0.8,\n",
+    "    },\n",
+    "    passed=False\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![GINI Score](../images/log_metric_attention.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here, a custom function `passed_fn` determines the badge status automatically, displaying a green \"Satisfactory\" badge for the 0.65 GINI Score because it exceeds the `medium_risk` threshold of 0.6, enabling programmatic evaluation of metric performance based on predefined business rules:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gini = 0.65\n",
+    "\n",
+    "thresholds = {\n",
+    "    \"high_risk\": 0.5,\n",
+    "    \"medium_risk\": 0.6,\n",
+    "    \"low_risk\": 0.8,\n",
+    "}\n",
+    "\n",
+    "def passed_fn(value):\n",
+    "    return value > thresholds[\"medium_risk\"]\n",
+    "\n",
+    "log_metric(\n",
+    "    key=\"GINI Score\",\n",
+    "    value=gini, \n",
+    "    recorded_at=datetime(2025, 6, 10),\n",
+    "    thresholds=thresholds,\n",
+    "    passed=passed_fn(gini)\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![GINI Score](../images/log_metric_satisfactory_2.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='toc7_'></a>\n",
+    "\n",
+    "## Next steps\n",
+    "\n",
+    "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation.\n",
+    "\n",
+    "<a id='toc9_1_'></a>\n",
+    "\n",
+    "### Work with your model documentation\n",
+    "\n",
+    "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n",
+    "\n",
+    "2. Click and expand the **Model Development** section.\n",
+    "\n",
+    "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n",
+    "\n",
+    "<a id='toc9_2_'></a>\n",
+    "\n",
+    "### Discover more learning resources\n",
+    "\n",
+    "We offer many interactive notebooks to help you document models:\n",
+    "\n",
+    "- [Run tests & test suites](https://docs.validmind.ai/developer/model-testing/testing-overview.html)\n",
+    "- [Code samples](https://docs.validmind.ai/developer/samples-jupyter-notebooks.html)\n",
+    "\n",
+    "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='toc8_'></a>\n",
+    "\n",
+    "## Upgrade ValidMind\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n",
+    "\n",
+    "Retrieve the information for the currently installed version of ValidMind:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip show validmind"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
+    "\n",
+    "```bash\n",
+    "%pip install --upgrade validmind\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You may need to restart your kernel after running the upgrade package for changes to be applied."
+   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "ValidMind Library",
+   "display_name": "validmind-eEL8LtKG-py3.10",
    "language": "python",
-   "name": "validmind"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -712,7 +904,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.13"
+   "version": "3.10.15"
   }
  },
  "nbformat": 4,

pyproject.toml

@@ -10,7 +10,7 @@ description = "ValidMind Library"
 license = "Commercial License"
 name = "validmind"
 readme = "README.pypi.md"
-version = "2.8.24"
+version = "2.8.25"
 
 [tool.poetry.dependencies]
 aiohttp = {extras = ["speedups"], version = "*"}

validmind/__version__.py

@@ -1 +1 @@
-__version__ = "2.8.24"
+__version__ = "2.8.25"

validmind/api_client.py

@@ -448,6 +448,7 @@ async def alog_metric(
     params: Optional[Dict[str, Any]] = None,
     recorded_at: Optional[str] = None,
     thresholds: Optional[Dict[str, Any]] = None,
+    passed: Optional[bool] = None,
 ):
     """See log_metric for details."""
     if not key or not isinstance(key, str):
@@ -476,6 +477,7 @@ async def alog_metric(
                     "params": params or {},
                     "recorded_at": recorded_at,
                     "thresholds": thresholds or {},
+                    "passed": passed if passed is not None else None,
                 },
                 cls=NumpyEncoder,
                 allow_nan=False,
@@ -493,6 +495,7 @@ def log_metric(
     params: Optional[Dict[str, Any]] = None,
     recorded_at: Optional[str] = None,
     thresholds: Optional[Dict[str, Any]] = None,
+    passed: Optional[bool] = None,
 ):
     """Logs a unit metric.
 
@@ -518,6 +521,7 @@ def log_metric(
         params=params,
         recorded_at=recorded_at,
         thresholds=thresholds,
+        passed=passed,
     )