New notebook series: ValidMind for model validation

[validbeck]

Internal Notes for Reviewers

9378

ValidMind for model validation

Brand new series combining the strategies I learned during editing Validate an application scorecard model from @MichaelIngvarRoenning and the model that you "build" in the ValidMind for model development series. This series mirrors the structure of the existing development series, and is built to be incorporated into our updated validation training path:

Model development series	Updated validation training path

🚨 THESE NOTEBOOKS BUILD ON EACH OTHER BUT ARE MEANT TO BE ABLE TO BE RUN INDEPENDENTLY. 🚨

As Jupyter Notebooks are closed environments, it means that the "Setting up" section of the notebooks need to be repetitive as the later cells rely on variable setup and outputs from the initial section of cells. I edited these down to be as streamlined as possible, but this is the same strategy we had to employ in the development series notebooks due to the limitations of notebooks.

1 — Set up ValidMind for validation

LIVE PREVIEW

Quick conceptual overview, this introduces users to ValidMind as a validator and walks them through setting up a model for validation, previewing templates/reports/etc.

2 — Start the validation process

LIVE PREVIEW

• Here the validator performs some data quality tests, same as in the development series — the only difference is they also learn to run some comparison tests.

• SOMEONE SHOULD DOUBLE CHECK:

  • Do the comparison tests make sense / are they accurate examples
  • Does the validation report section the user is directed to insert the ClassImbalance test results to make sense

3 — Develop a potential challenger model

LIVE PREVIEW

• Here is where the notebooks diverge from the developer series, we instead import the champion logistic regression model created by the development series to evaluate

• SOMEONE SHOULD DOUBLE CHECK:

  • Is a random forest classification model a reasonable challenger here?
  • Do the model validation tests make sense here, and are they accurate?
  • LOGGING A FINDING: I noticed that the MinimumAccuracy test doesn't actually pass for our finding! I don't know if that's just an inaccurate assumption, but I thought it was a neat inclusion point for being introduced to findings. SOMEONE PLEASE RUN THIS NOTEBOOK AND LET ME KNOW IF YOU GET THE SAME RESULTS / IF THIS IS A REASONABLE EXAMPLE.

4 — Finalize validation and reporting

LIVE PREVIEW

• Here we include the same custom test the development series did, just to walk the user through the process — the only difference is the custom test is run for both models instead of just the champion.

• SOMEONE SHOULD DOUBLE CHECK:

  • Does the way the custom test is run for both models accurate / still good examples?
  • The section on "Verify test runs" I just stole from Michael's notebook and modified with our datasets/models — do these tests still make sense as examples / are they accurate? 🙏🏻

ValidMind for model development

• Upon suggestion by @nrichers, I've renumbered these notebooks to be less "opaque" as not everyone is familiar with the uni-course type structure worldwide.

• Quick qualitative edits as I "validated" the model built in this series, most notably the "Train simple logistic regression model" section in notebook 2 got a quick edit as there was some repetition in code and the way the dataset split was performed was being flagged in the validation process for incompatible dataset structure: Refer to Slack conversation context

Before	After

@cachafla helped me out with this adjustment so it should pass muster but just in case.

External Release Notes

Check out our new introductory series of notebooks tailored to model validators — ValidMind for model validation:

• 1 — Set up ValidMind for validation
• 2 — Start the validation process
• 3 — Develop a potential challenger model
• 4 — Finalize validation and reporting

These new notebooks break down using ValidMind for your end-to-end model validation process based on common scenarios. Learn the basics of the ValidMind Library with these interactive notebooks designed to introduce you to basic ValidMind concepts and get you familiar with tasks such as how to run and log quality, performance, comparison, and other types of tests with ValidMind, develop potential challenger models, work with validation report tools, and more. After you've completed your learning journey with these notebooks, you'll have a sample validation report ready to go.

[nrichers]

@validbeck thank you, I think this new numbering scheme is much simpler and it's great to see these validation notebooks in the tutorials/ folder! 🚀

I will let others address your questions related to SOMEONE SHOULD DOUBLE CHECK but the format changes and the content I read through — and quite enjoyed reading through — LGTM.

[MichaelIngvarRoenning]

Personally I would merge the first and second notebook. I don't think it makes sense to have a separate notebook for the setting up the library. I think this can be merged with notebook 2.

The rest looks great!

[validbeck]

Personally I would merge the first and second notebook. I don't think it makes sense to have a separate notebook for the setting up the library. I think this can be merged with notebook 2.

Thanks Michael! We're aiming to align this with the developer notebook series (that we build on for training, which has 4 modules and one in-depth platform introductory model) so I think it makes more sense to keep the 1st notebook light, but good to know the rest looks good!

[github-actions]

PR Summary

This pull request introduces several enhancements to the ValidMind model validation notebooks. The key changes include:

1. New Model Validation Notebooks: A new series of four introductory notebooks for model validation has been added. These notebooks guide users through setting up the ValidMind Library for validation, starting the model validation process, developing potential challenger models, and finalizing validation and reporting.

2. Custom Test Implementation: The notebooks now include sections on implementing custom tests, including creating confusion matrix plots and using external test providers. This allows users to extend the default tests provided by ValidMind with their own custom tests.

3. Improved Documentation and Instructions: The notebooks have been updated with detailed instructions and explanations, including how to log test results, add findings, and assess compliance within the ValidMind Platform.

4. Sample Data and Models: The notebooks utilize a sample dataset (Bank Customer Churn Prediction) and a sample logistic regression model (lr_model_champion.pkl) to demonstrate the validation process.

5. Version Update: The version of the ValidMind Library has been updated from 2.8.17 to 2.8.18 in the pyproject.toml and __version__.py files.

Test Suggestions

• Run the new model validation notebooks to ensure they execute without errors.
• Verify that the custom test implementation correctly logs results to the ValidMind Platform.
• Check that the instructions for setting up and using the ValidMind Library are clear and accurate.
• Test the functionality of the external test provider by running tests from the my_tests directory.
• Ensure that the sample dataset and model are correctly loaded and used in the notebooks.