Updated documentation for Metrics Over Time

[validbeck]

Internal Notes for Reviewers

sc-8186

Work with metrics over time

Part 2 of updating the docs for metric over time to include:

• LInk out to the new "Log metrics over time" notebook
• New more detailed screenshots
• Mentions of the new threshold feature

LIVE PREVIEW

OLD	NEW

After validmind/validmind-library#301 is merged, I will pull in the latest notebook updates before merging this PR.

Missing anywidget dependency

Newest update to the ValidMind Library broke the GitHub execution workflow, but I solved it by installing a dependency explicitly:

documentation/.github/actions/demo-notebook/action.yml

Line 25 in d33b9a9

pip install anywidget

All versions of the composite action/workflows have been updated to reflect this.

[github-actions]

A PR preview is available: Preview URL

[github-actions]

PR Summary

This pull request introduces several enhancements and bug fixes to the ValidMind project, focusing on documentation and monitoring capabilities. Key changes include:

1. Action and Workflow Updates:

   • Updated GitHub Actions and workflows to improve the installation and execution of dependencies. This includes adding anywidget to the list of installed packages and removing trailing spaces in YAML files.

2. Documentation Enhancements:

   • Added new Jupyter notebooks and markdown files to the site/notebooks and site/tests directories, providing detailed guides on logging metrics over time, ongoing monitoring, and custom test creation.
   • Updated existing documentation to improve clarity and consistency, such as removing LaTeX notation references and enhancing descriptions.

3. New Features:

   • Introduced new custom tests and monitoring capabilities, including ScoreBandDiscriminationMetrics and CalibrationCurveDrift, to evaluate model performance and stability over time.
   • Added support for logging and visualizing metrics over time with customizable thresholds, enabling better tracking of model performance trends.

4. Code Quality Improvements:

   • Refactored code in Jupyter notebooks to remove unnecessary conditional checks and streamline test execution.
   • Improved code formatting and consistency across various files.

These changes aim to enhance the usability and functionality of the ValidMind platform, providing users with more robust tools for model risk management and monitoring.

Test Suggestions

• Verify the installation of all dependencies in the updated GitHub Actions workflows.
• Run the new Jupyter notebooks to ensure they execute without errors and produce the expected outputs.
• Test the new custom tests and monitoring features to confirm they work as intended and provide accurate results.
• Check the visualization of metrics over time to ensure thresholds are displayed correctly.
• Review the updated documentation for clarity and completeness.

[github-actions]

PR Summary

This pull request introduces several enhancements and bug fixes to the ValidMind project, focusing on documentation and monitoring capabilities. Key changes include:

1. Action and Workflow Updates: The GitHub Actions and workflows have been updated to improve the installation and execution of dependencies. Notably, the anywidget package has been added to the list of dependencies in the action.yml files for demo, prod, and staging notebooks.

2. Documentation Enhancements: Several markdown files and Jupyter notebooks have been added or updated to provide better guidance on using ValidMind for monitoring and logging metrics over time. This includes detailed instructions on setting up and using ValidMind for ongoing monitoring of application scorecards and logging metrics over time.

3. New Code Samples: New Jupyter notebooks have been introduced, such as application_scorecard_executive.ipynb, application_scorecard_full_suite.ipynb, and application_scorecard_with_ml.ipynb, which provide comprehensive examples of using ValidMind for model documentation and validation.

4. Custom Tests: A new custom test, ScoreBandDiscriminationMetrics, has been added to evaluate discrimination metrics across different score bands for credit risk assessment.

5. Monitoring Tests: New markdown files have been added to describe various monitoring tests, such as CalibrationCurveDrift, ClassDiscriminationDrift, and ClassImbalanceDrift, among others. These tests help in evaluating changes in model performance and data distribution over time.

6. Image Assets: New image assets have been added to support the documentation, providing visual aids for understanding the logging and monitoring processes.

Overall, these changes aim to enhance the usability and functionality of ValidMind, particularly in the areas of model monitoring and documentation.

Test Suggestions

• Verify the installation of the anywidget package in the GitHub Actions workflows.
• Run the new Jupyter notebooks to ensure they execute without errors and produce the expected outputs.
• Test the new custom test ScoreBandDiscriminationMetrics to ensure it calculates metrics correctly.
• Validate the markdown documentation for clarity and accuracy in describing the monitoring tests.
• Check the rendering of new image assets in the documentation to ensure they display correctly.

[validbeck]

@juanmleng Can you review these docs-side and let me know if they look good in conjunction with the notebook edits and approve if so? 🙏🏻

https://docs-demo.vm.validmind.ai/pr_previews/beck/sc-8186/documentation-support-threshold-lines-in/guide/monitoring/work-with-metrics-over-time.html

(n.b. Updated notebook will not show up here until I merge in the changes from validmind/validmind-library#301)

[github-actions]

A PR preview is available: Preview URL

[juanmleng]

Great job Beck! Just left a small suggestion.

[github-actions]

A PR preview is available: Preview URL

[github-actions]

A PR preview is available: Preview URL