[SC-10275] Add Satisfactory vs. Needs Attention Flags to Ongoing Monitoring Tests

[juanmleng]

Pull Request Description

What

Added a new acceptable performance flag feature to the log_metric() function that visually indicates metric status with color-coded badges. When setting passed=True, a green "Satisfactory" badge appears on the chart, while passed=False displays a yellow "Requires Attention" badge. This provides an immediate visual indication of whether metric values meet predefined performance standards, beyond just showing the raw values relative to thresholds.

Why

Monitoring metric values against thresholds is valuable, but users often need an indication of whether a metric is acceptable or problematic according to business rules. This enhancement addresses that need by:

• Enabling users to set custom acceptance criteria beyond simple threshold comparisons
• Making it immediately obvious which metrics require attention
• Supporting compliance and building the foundation for triggering escalations via workflows when metrics breach acceptable performance threshold rules.

How to Test

1. Initialize the ValidMind library with your API credentials and enable monitoring
2. Log a metric with passed=True using the following code:

   log_metric(
       key="Test Metric",
       value=0.75,
       recorded_at=datetime.now(),
       thresholds={"medium_risk": 0.6},
       passed=True
   )

3. Log another metric with passed=False:

   log_metric(
       key="Test Metric",
       value=0.55,
       recorded_at=datetime.now(),
       thresholds={"medium_risk": 0.6},
       passed=False
   )

4. Create a Metrics Over Time block in the ValidMind Platform for your model
5. Verify that the metrics show the appropriate badges (green "Satisfactory" for passed=True, yellow "Requires Attention" for passed=False)
6. Test with a custom evaluation function:

   def custom_evaluator(value):
       return value > 0.6
   
   log_metric(
       key="Test Metric",
       value=0.65,
       recorded_at=datetime.now(),
       thresholds={"medium_risk": 0.6},
       passed=custom_evaluator(0.65)
   )

Pull Request Dependencies

https://github.com/validmind/backend/pull/1501
https://github.com/validmind/frontend/pull/1403

External Release Notes

The ValidMind Library now supports visual status indicators for ongoing monitoring metrics. When using the log_metric() function, you can specify the passed parameter to add status badges to Metrics Over Time blocks.

This new parameter accepts a boolean value:

• passed=True: Displays a green "Satisfactory" badge
• passed=False: Displays a yellow "Requires Attention" badge

This feature enables straightforward visual assessment of metric performance against defined business rules. Users can either manually set the status or programmatically determine it using custom evaluation functions that implement specific acceptance criteria.

The status indicators are especially useful for:

• Quickly identifying metrics that require attention
• Supporting compliance documentation with clear visual indicators
• Enabling more targeted alerting based on metric status

Deployment Notes

Breaking Changes

Screenshots/Videos (Frontend Only)

Checklist

• PR body describes what, why, and how to test
• Release notes written
• Deployment notes written
• Breaking changes identified
• Labels applied
• PR linked to Shortcut
• Screenshots/videos added (Frontend)
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)

Areas Needing Special Review

Additional Notes

[cachafla]

Awesome 👌

[github-actions]

PR Summary

This pull request introduces a new feature to the log_metric and alog_metric functions in the validmind library. The enhancement involves adding a passed parameter, which allows users to explicitly mark whether a specific metric value should be considered "Satisfactory" or "Requires Attention". This is achieved by:

• Adding the passed parameter to both log_metric and alog_metric functions.
• Updating the JSON payload to include the passed parameter when logging metrics.
• Modifying the Jupyter notebook log_metrics_over_time.ipynb to demonstrate the usage of the passed parameter with examples and visualizations.

The notebook now includes sections that explain how to use the passed parameter to add visual badges to metric visualizations, indicating whether the metric meets the acceptance criteria. Examples are provided to show how to use the parameter directly or through a custom function to evaluate metric performance programmatically.

Additionally, the notebook's table of contents and metadata have been updated to reflect these changes.

Test Suggestions

• Test the log_metric function with passed=True and verify that the correct badge is displayed in the visualization.
• Test the log_metric function with passed=False and verify that the correct badge is displayed in the visualization.
• Test the alog_metric function with passed=True and ensure it behaves as expected asynchronously.
• Test the alog_metric function with passed=False and ensure it behaves as expected asynchronously.
• Test the log_metric function with a custom passed_fn to ensure it correctly evaluates and applies the badge based on the function's logic.
• Verify that the notebook examples execute without errors and produce the expected visual output.