diff --git a/Makefile b/Makefile index dc0c017b5..078b6ac19 100644 --- a/Makefile +++ b/Makefile @@ -95,5 +95,5 @@ check: copyright format lint test verify-copyright verify-exposed-credentials en .PHONY: docs quarto-docs notebook: - @python notebooks/templates/e2e_template.py + poetry run python notebooks/templates/e2e_template.py git status | grep -v 'notebooks/templates' diff --git a/notebooks/code_samples/capital_markets/capital_markets_template.yaml b/notebooks/code_samples/capital_markets/capital_markets_template.yaml new file mode 100644 index 000000000..c891a1c43 --- /dev/null +++ b/notebooks/code_samples/capital_markets/capital_markets_template.yaml @@ -0,0 +1,223 @@ +- id: model_metadata + title: Model Metadata and Stakeholders + index_only: true + sections: + - id: metadata + title: Metadata + parent_section: model_metadata + guidelines: + - Provide detailed metadata to uniquely identify the model, ensuring + that each entry is traceable and consistent with internal records. + - Specify the platform to enable validation and governance teams to + understand technical dependencies. + - Metadata should be aligned with enterprise systems (e.g., Model Risk + Management System). + - Be specific about versioning and any platform customizations. + - id: stakeholders + title: Stakeholders + parent_section: model_metadata + guidelines: + - List all individuals responsible for the development, use, and + oversight of the model. + - Ensure clarity in responsibilities to avoid overlap and gaps in + accountability. + - Include specific roles, such as "Data Scientist," "Validation Lead," + or "Business Analyst," and describe their responsibilities relative to + the model. +- id: business_context + title: Business Context and Purpose + index_only: true + sections: + - id: business_problem + title: Business Problem and Objectives + parent_section: business_context + guidelines: + - Clearly describe the problem the model addresses and its alignment + with business goals. + - Include specific use cases, outputs, and highlight regulatory + expectations to demonstrate compliance. + - Specify compliance requirements, such as IFRS, Basel III or SR11-7, as + applicable. + - id: products_and_risks + title: Products and Risks + parent_section: business_context + guidelines: + - Detail how the model impacts products or processes and specify + associated risks (e.g., market, credit, or operational risks). + - Provide a description of business impacts and compliance with + applicable regulations. +- id: model_design + title: Model Methodology and Design + index_only: true + sections: + - id: theoretical_foundations + title: Theoretical Foundations + parent_section: model_design + guidelines: + - Explain the methodology, assumptions, and logic underlying the model. + - Clearly document how key risks (e.g., credit default or liquidity) are + addressed within the model. + - Highlight any simplifications or approximations and their implications. + - id: architecture + title: Model Architecture + parent_section: model_design + guidelines: + - Use diagrams to illustrate the model's structure and data flow. + - Include visual flowcharts detailing how inputs are transformed into + outputs. + - Highlight dependencies affecting validation or performance. + - id: selection + title: Model Selection and Justification + parent_section: model_design + guidelines: + - Document the decision-making process, including comparisons to + alternative approaches. + - Provide rationale for model selection using performance metrics like + R-squared or RMSE. + - Highlight reasons for rejecting alternatives. +- id: model_data + title: Model Data + index_only: true + sections: + - id: input_data + title: Input Data + parent_section: model_data + guidelines: + - Detail the provenance and quality of input data, including + preprocessing steps like imputation or outlier detection. + - Highlight known issues, such as stale data or incomplete time series. + - Suggest standard tests, e.g., null value checks, distribution + matching, or correlation analysis. + - id: dataset_characteristics + title: Development Dataset + parent_section: model_data + guidelines: + - Summarize dataset characteristics like size, representativeness, and + scope. + - Include validation metrics such as sampling error or coverage ratios. + - id: outputs + title: Outputs + parent_section: model_data + guidelines: + - Define outputs, their usage, and storage mechanisms. + - Highlight data formats (e.g., APIs, flat files) and ensure outputs are + validated for consistency and accuracy. +- id: model_testing + title: Model Testing + index_only: true + sections: + - id: diagnostic_testing + title: Diagnostic Testing + parent_section: model_testing + guidelines: + - Provide details of diagnostic tests performed to ensure model + performance and identify anomalies. + - Include standard diagnostic tests such as - Residual analysis to check + model predictions. - Comparison of predicted versus actual outcomes + for validity. + - Summarize findings and highlight any performance gaps. + - id: sensitivity_stress_testing + title: Sensitivity and Stress Testing + parent_section: model_testing + guidelines: + - Sensitivity Testing. Describe how changes in input variables affect + model outputs. + - Stress Testing. Document model performance under extreme conditions or + assumptions. + - Use tests to measure how minor changes in key parameters affect + results. + - Simulate scenarios like extreme economic downturns to evaluate + robustness. + - id: performance_testing + title: Performance Testing + parent_section: model_testing + guidelines: + - Provide details of performance metrics such as RMSE, AUC, or + precision/recall. + - Benchmark comparisons. Compare performance with industry standards or + alternative models. + - Include visual aids like ROC curves or confusion matrices to + illustrate performance. + - id: back_testing + title: Back-Testing + parent_section: model_testing + guidelines: + - Highlight alignment of predictions with observed outcomes through + historical analysis. + - Test model predictions against historical data outcomes. + - Document discrepancies and propose remediation steps. +- id: implementation + title: Model Implementation + index_only: true + sections: + - id: production_environment + title: Production Environment + parent_section: implementation + guidelines: + - Describe the implementation environment, such as cloud platforms like + AWS or GCP. + - Highlight integration points, such as database connectors or REST + APIs, ensuring consistency with design specifications. + - id: implementation_testing + title: Implementation Testing + parent_section: implementation + guidelines: + - Document verification steps, including parallel runs against legacy + systems and end-to-end pipeline testing. +- id: limitations_adjustments + title: Assumptions, Limitations, and Adjustments + index_only: true + sections: + - id: assumptions + title: Assumptions + parent_section: limitations_adjustments + guidelines: + - List assumptions critical to model functionality and provide + justifications. + - Include potential impact analyses if assumptions fail. + - id: limitations + title: Limitations + parent_section: limitations_adjustments + guidelines: + - Highlight known limitations and mitigation strategies. + - Discuss implications for model performance or reliability. + - id: adjustments + title: Adjustments + parent_section: limitations_adjustments + guidelines: + - Document overrides and their justification, including governance + processes. +- id: monitoring_controls_documentation + title: Model Monitoring and Controls + index_only: true + sections: + - id: ongoing + title: Ongoing Model Monitoring + parent_section: monitoring_controls_documentation + guidelines: + - Outline a monitoring plan, including performance metrics, monitoring + frequency, and escalation thresholds. + - Include drift analysis of input data distributions and stability + metrics for output consistency. + - id: governance + title: Governance + parent_section: monitoring_controls_documentation + guidelines: + - Define access controls, version management, and governance frameworks. + - Highlight periodic audits and role-based access controls. +- id: documentation_references + title: Documentation References + index_only: true + sections: + - id: supporting_documents + title: Supporting Documents + parent_section: documentation_references + guidelines: + - Provide references to related documentation, such as validation + reports. + - Maintain a change log for systematic traceability. + - id: appendices + title: Appendices + parent_section: documentation_references + guidelines: + - Use appendices for supplementary data, testing results, and glossaries. diff --git a/notebooks/code_samples/capital_markets/quickstart_option_pricing_models_quantlib.ipynb b/notebooks/code_samples/capital_markets/quickstart_option_pricing_models_quantlib.ipynb index 1269254ad..3755fbf63 100644 --- a/notebooks/code_samples/capital_markets/quickstart_option_pricing_models_quantlib.ipynb +++ b/notebooks/code_samples/capital_markets/quickstart_option_pricing_models_quantlib.ipynb @@ -262,11 +262,25 @@ "\n", "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", "\n", - "2. Under **TEMPLATE**, select `Capital markets`.\n", + "2. Under **TEMPLATE**, select `Capital Markets`.\n", "\n", "3. Click **Use Template** to apply the template." ] }, + { + "cell_type": "markdown", + "id": "41c4edca", + "metadata": {}, + "source": [ + "
Can't select this template?\n", + "

\n", + "Your organization administrators may need to add it to your template library:\n", + "
" + ] + }, { "cell_type": "markdown", "id": "2012eb82", diff --git a/notebooks/code_samples/code_explainer/model_source_code_documentation_template.yaml b/notebooks/code_samples/code_explainer/model_source_code_documentation_template.yaml new file mode 100644 index 000000000..92141e751 --- /dev/null +++ b/notebooks/code_samples/code_explainer/model_source_code_documentation_template.yaml @@ -0,0 +1,138 @@ +- id: code_overview + title: Codebase Overview + guidelines: + - Describe the overall structure of the source code repository. + - Identify main modules, folders, and scripts. + - Highlight entry points for training, inference, and evaluation. + - State the main programming languages and frameworks used. + contents: + - content_type: text + content_id: code_structure_summary +- id: model_overview + title: Model Overview + guidelines: + - Describe the overall structure of the source code repository. + - Identify main modules, folders, and scripts. + - Highlight entry points for training, inference, and evaluation. + - State the main programming languages and frameworks used. + contents: + - content_type: text + content_id: model_overview +- id: environment_setup + title: Environment and Dependencies + guidelines: + - List Python packages and system dependencies (OS, compilers, etc.). + - Reference environment files (requirements.txt, environment.yml, + Dockerfile). + - Include setup instructions using Conda, virtualenv, or containers. + contents: + - content_type: text + content_id: setup_instructions +- id: data_interface + title: Data Ingestion and Preprocessing + guidelines: + - Specify data input formats and sources. + - Document ingestion, validation, and transformation logic. + - Explain how raw data is preprocessed and features are generated. + contents: + - content_type: text + content_id: data_handling_notes +- id: model_implementation + title: Model Implementation Details + guidelines: + - Describe the core model code structure (classes, functions). + - Link code to theoretical models or equations when applicable. + - Note custom components like loss functions or feature selectors. + contents: + - content_type: text + content_id: model_code_description +- id: training_pipeline + title: Model Training Pipeline + guidelines: + - Explain the training process, optimization strategy, and hyperparameters. + - Describe logging, checkpointing, and early stopping mechanisms. + - Include references to training config files or tuning logic. + contents: + - content_type: text + content_id: training_logic_details +- id: evaluation_pipeline + title: Evaluation and Validation Code + guidelines: + - Describe how validation is implemented and metrics are calculated. + - Include plots and diagnostic tools (e.g., ROC, SHAP, confusion matrix). + - State how outputs are logged and persisted. + contents: + - content_type: text + content_id: evaluation_logic_notes +- id: inference_pipeline + title: Inference and Scoring Logic + guidelines: + - Detail how the trained model is loaded and used for predictions. + - Explain I/O formats and APIs for serving or batch scoring. + - Include any preprocessing/postprocessing logic required. + contents: + - content_type: text + content_id: inference_mechanism +- id: configuration_management + title: Configuration and Parameters + guidelines: + - Describe configuration management (files, CLI args, env vars). + - Highlight default parameters and override mechanisms. + - Reference versioning practices for config files. + contents: + - content_type: text + content_id: config_control_notes +- id: testing_and_validation + title: Unit and Integration Testing + guidelines: + - List unit and integration tests and what they cover. + - Mention testing frameworks and coverage tools used. + - Explain testing strategy for production-readiness. + contents: + - content_type: text + content_id: test_strategy_overview +- id: logging + title: Logging and Monitoring Hooks + guidelines: + - Describe logging configuration and structure. + - Highlight real-time monitoring or observability integrations. + - List key events, metrics, or alerts tracked. + contents: + - content_type: text + content_id: logging_notes +- id: version_control + title: Code and Model Versioning + guidelines: + - Describe Git usage, branching, tagging, and commit standards. + - Include model artifact versioning practices (e.g., DVC, MLflow). + - Reference any automation in CI/CD. + contents: + - content_type: text + content_id: version_tracking_description +- id: security_and_compliance + title: Security and Access Control + guidelines: + - Document access controls for source code and data. + - Include any encryption, PII handling, or compliance measures. + - Mention secure deployment practices. + contents: + - content_type: text + content_id: security_policies_notes +- id: execution_examples + title: Example Runs and Scripts + guidelines: + - Provide working script examples (e.g., `train.py`, `predict.py`). + - Include CLI usage instructions or sample notebooks. + - Link to demo datasets or test scenarios. + contents: + - content_type: text + content_id: runnable_examples +- id: known_issues_and_todos + title: Known Issues and Future Improvements + guidelines: + - List current limitations or technical debt. + - Outline proposed enhancements or refactors. + - Reference relevant tickets, GitHub issues, or roadmap items. + contents: + - content_type: text + content_id: issues_and_improvements_log diff --git a/notebooks/code_samples/code_explainer/quickstart_code_explainer_demo.ipynb b/notebooks/code_samples/code_explainer/quickstart_code_explainer_demo.ipynb index 1eb1ef747..6d77ce74b 100644 --- a/notebooks/code_samples/code_explainer/quickstart_code_explainer_demo.ipynb +++ b/notebooks/code_samples/code_explainer/quickstart_code_explainer_demo.ipynb @@ -184,8 +184,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk - CECL`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." @@ -208,6 +206,19 @@ "3. Click **Use Template** to apply the template." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
Can't select this template?\n", + "

\n", + "Your organization administrators may need to add it to your template library:\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/notebooks/code_samples/credit_risk/application_scorecard_executive.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_executive.ipynb index 2115a88b5..693144634 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_executive.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_executive.ipynb @@ -163,8 +163,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk - CECL`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/credit_risk/application_scorecard_full_suite.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_full_suite.ipynb index 83a6d276c..06c98f584 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_full_suite.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_full_suite.ipynb @@ -177,8 +177,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk - CECL`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/credit_risk/application_scorecard_with_bias.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_with_bias.ipynb index a91560831..d5d5757eb 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_with_bias.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_with_bias.ipynb @@ -178,8 +178,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk - CECL`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/credit_risk/application_scorecard_with_ml.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_with_ml.ipynb index 357d572b7..a5067f2da 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_with_ml.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_with_ml.ipynb @@ -190,8 +190,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk - CECL`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/credit_risk/document_excel_application_scorecard.ipynb b/notebooks/code_samples/credit_risk/document_excel_application_scorecard.ipynb index cc7685c9c..5d875def2 100644 --- a/notebooks/code_samples/credit_risk/document_excel_application_scorecard.ipynb +++ b/notebooks/code_samples/credit_risk/document_excel_application_scorecard.ipynb @@ -179,8 +179,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk - CECL`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb b/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb index 17385b8e1..0fdd6e0d1 100644 --- a/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb +++ b/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb @@ -170,8 +170,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `{use-case}`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/custom_tests/integrate_external_test_providers.ipynb b/notebooks/code_samples/custom_tests/integrate_external_test_providers.ipynb index ca1403fb6..62650af90 100644 --- a/notebooks/code_samples/custom_tests/integrate_external_test_providers.ipynb +++ b/notebooks/code_samples/custom_tests/integrate_external_test_providers.ipynb @@ -183,8 +183,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb b/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb index 22ab661f7..58953711e 100644 --- a/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb +++ b/notebooks/code_samples/model_validation/validate_application_scorecard.ipynb @@ -185,8 +185,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk — CECL`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down — don’t worry, we’ll adjust these permissions next for validation.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/nlp_and_llm/foundation_models_integration_demo.ipynb b/notebooks/code_samples/nlp_and_llm/foundation_models_integration_demo.ipynb index 633c99e0c..346ecb6e1 100644 --- a/notebooks/code_samples/nlp_and_llm/foundation_models_integration_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/foundation_models_integration_demo.ipynb @@ -130,8 +130,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Analytics`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/nlp_and_llm/gen_ai_rag_template.yaml b/notebooks/code_samples/nlp_and_llm/gen_ai_rag_template.yaml new file mode 100644 index 000000000..5ff6064e9 --- /dev/null +++ b/notebooks/code_samples/nlp_and_llm/gen_ai_rag_template.yaml @@ -0,0 +1,534 @@ +- id: conceptual_soundness + title: Conceptual Soundness + index_only: true + sections: + - id: model_overview + title: Model Overview + guidelines: + - Provide a clear and concise description of the model's main concept, + explaining the underlying financial theory or economic rationale. + - Justify the choice of the model, algorithm, or approach, relating it + to the financial institution's needs, objectives, and any relevant + industry standards. + - Discuss any alternative models or approaches considered during the + model development process, and explain why they were not selected. + - Describe any key assumptions made in the model and their potential + implications on the model's results and performance. + - Explain how the model's structure and design contribute to its + robustness, stability, and reliability in a financial context. + parent_section: conceptual_soundness + contents: + - content_id: model_overview + content_type: text + - id: intended_use_business_use_case + title: Intended Use and Business Use Case + index_only: true + condensed: true + parent_section: conceptual_soundness + sections: + - id: intended_use + title: Intended Use + parent_section: intended_use_business_use_case + guidelines: + - Clearly describe the specific business use case(s) for which the + model is designed, including the intended users and the financial + institution's objectives. + - Explain how the model fits into the overall business strategy and + decision-making processes of the financial institution. + - id: regulatory_requirements + title: Regulatory Requirements + parent_section: intended_use_business_use_case + guidelines: + - Detail any specific risks, regulatory requirements, or other + considerations associated with the model's intended use, and how + they have been addressed in the model development process. + - id: model_limitations + title: Model Limitations + parent_section: intended_use_business_use_case + guidelines: + - Discuss the model's intended scope, including any limitations, + boundaries, or exclusions in its applicability. + - Describe any potential external factors, such as economic or + market conditions, that could impact the model's performance, and + how they have been considered in the model development process. + - id: model_selection + title: Model Selection + guidelines: + - Provide a detailed description of the selected model, including its + algorithm, mathematical foundations, and key features that make it + suitable for the intended use case. + - Explain the rationale behind choosing the specific model, and how it + addresses the financial institution's objectives, regulatory + requirements, and risk management needs. + - Compare the selected model with alternative models or approaches that + were considered during the model development process, highlighting + their strengths and weaknesses, and explaining why the chosen model is + the most appropriate. + - Describe any model customizations or adaptations made to better align + it with the financial institution's needs, and discuss the potential + impact of these changes on the model's performance. + - Explain any potential limitations or drawbacks of the selected model + in the context of the intended use case, and how they have been + mitigated or addressed during the model development process. + parent_section: conceptual_soundness +- id: data_preparation + title: Data Preparation + index_only: true + sections: + - id: data_description + title: Data description + guidelines: + - Provide a comprehensive overview of the data sources used in the + model, including internal and external sources, and specify the data's + time period, frequency, and granularity. + - Describe the main variables and features used in the model, including + both input and output variables, and explain their relevance to the + model's purpose and intended use case. + - Detail any data transformations, preprocessing, or feature engineering + performed on the raw data to prepare it for model input, and explain + the rationale for these transformations. + - Address any data quality concerns, such as missing values, outliers, + or inconsistencies, and describe the methods used to handle these + issues. + - Discuss any potential biases, dependencies, or limitations in the data + that could impact the model's performance, and explain how they have + been considered or mitigated during the data preparation process. + contents: + - content_id: text_data_quality + content_type: text + options: + default_text: | + # Text Data Quality Metrics and Tests + - content_type: test + content_id: validmind.data_validation.Duplicates + - content_type: test + content_id: validmind.data_validation.nlp.StopWords + - content_type: test + content_id: validmind.data_validation.nlp.Punctuations + - content_type: test + content_id: validmind.data_validation.nlp.CommonWords + - content_type: test + content_id: validmind.data_validation.nlp.LanguageDetection + - content_type: test + content_id: validmind.data_validation.nlp.Toxicity + - content_type: test + content_id: validmind.data_validation.nlp.PolarityAndSubjectivity + - content_type: test + content_id: validmind.data_validation.nlp.Sentiment + parent_section: data_preparation + - id: feature_selection + title: Feature Selection and Engineering + guidelines: + - Describe the process used to select the most relevant features for the + model, including any feature selection techniques or criteria applied, + such as correlation analysis, mutual information, or forward/backward + selection. + - Explain the rationale behind including or excluding specific features, + and discuss their importance and contribution to the model's + performance and intended use case. + - Detail any feature engineering techniques applied to create new + features or transform existing ones, such as dimensionality reduction, + aggregation, or interaction terms, and explain their relevance and + purpose in the context of the model. + - Describe any data normalization or scaling techniques used to + standardize the input features, and explain the rationale for their + application in the model. + - Discuss potential multicollinearity, redundancy, or other issues among + the selected features, and describe the methods used to address these + concerns during the feature selection and engineering process. + parent_section: data_preparation + contents: + - content_id: feature_selection + content_type: text +- id: model_development + title: Model Development + index_only: true + sections: + - id: model_training + title: Model Training + guidelines: + - Describe the model training process, including the algorithm used, any + hyperparameters or settings, and the optimization techniques employed + to minimize the loss function or maximize the objective function. + - Detail the model validation and selection process, including the use + of cross-validation, holdout samples, or other techniques to assess + the model's performance and prevent overfitting. + - Provide a summary of the training results, including performance + metrics such as accuracy, precision, recall, F1 score, or other + relevant measures, depending on the model's intended use case. + - Discuss any challenges, issues, or trade-offs encountered during the + model training process, such as overfitting, underfitting, or class + imbalance, and explain how they were addressed or mitigated. + - Describe any tuning or optimization steps performed to improve the + model's performance, such as hyperparameter tuning, feature selection, + or other adjustments, and explain the rationale for these changes. + parent_section: model_development + contents: + - content_id: model_training + content_type: text + - id: prompt_validation + title: Prompt Validation + guidelines: + - Provide documentation of the prompt used for the model. Describe the + type of prompting used (e.g. few-shot, zero-shot, chain-of-thought + etc.). Explain how the prompt was constructed and how it was + validated. + parent_section: model_development + contents: + - content_type: test + content_id: validmind.prompt_validation.Bias + options: + title: Bias + - content_type: test + content_id: validmind.prompt_validation.Clarity + options: + title: Clarity + - content_type: test + content_id: validmind.prompt_validation.Conciseness + options: + title: Conciseness + - content_type: test + content_id: validmind.prompt_validation.Delimitation + options: + title: Delimitation + - content_type: test + content_id: validmind.prompt_validation.NegativeInstruction + options: + title: Negative Instruction + - content_type: test + content_id: validmind.prompt_validation.Specificity + options: + title: Specificity + - id: model_evaluation + title: Model Evaluation + parent_section: model_development + guidelines: + - Describe the process used to evaluate the model's performance on a + test or validation dataset that was not used during training, to + assess its generalizability and robustness. + - Present the key performance metrics for the model evaluation, such as + accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, or + other relevant measures, depending on the model's intended use case. + - Provide graphical representations of the model's performance, such as + confusion matrices, ROC curves, or residual plots, to help visualize + its effectiveness and identify any areas for improvement. + - Discuss the model's performance in the context of its intended use + case, and compare it to any benchmarks, industry standards, or + alternative models, as appropriate. + - Identify any limitations, weaknesses, or areas for improvement in the + model's performance, and discuss potential strategies for addressing + these concerns in future iterations or updates. + sections: + - id: embedding_model + title: Embedding Model + parent_section: model_evaluation + guidelines: + - Embedding model guidelines + contents: + - content_type: test + content_id: validmind.model_validation.embeddings.StabilityAnalysisRandomNoise + - content_type: test + content_id: validmind.model_validation.embeddings.StabilityAnalysisSynonyms + - content_type: test + content_id: validmind.model_validation.embeddings.StabilityAnalysisTranslation + - content_type: test + content_id: validmind.model_validation.embeddings.CosineSimilarityHeatmap + - content_type: test + content_id: validmind.model_validation.embeddings.CosineSimilarityDistribution + - content_type: test + content_id: validmind.model_validation.embeddings.PCAComponentsPairwisePlots + - id: retrieval_model + title: Retrieval model + parent_section: model_evaluation + guidelines: + - Retrieval model guideline + contents: + - content_type: test + content_id: validmind.model_validation.ragas.ContextRecall + - content_type: test + content_id: validmind.model_validation.ragas.ContextEntityRecall + - content_type: test + content_id: validmind.model_validation.ragas.ContextPrecision + - content_type: test + content_id: validmind.model_validation.ragas.ContextPrecisionWithoutReference + - id: rag_model + title: RAG model + parent_section: model_evaluation + guidelines: + - RAG model guideline + contents: + - content_type: test + content_id: validmind.model_validation.ragas.SemanticSimilarity + - content_type: test + content_id: validmind.model_validation.ragas.Faithfulness + - content_type: test + content_id: validmind.model_validation.ragas.ResponseRelevancy + - content_type: test + content_id: validmind.model_validation.ragas.AnswerCorrectness + - content_type: test + content_id: validmind.model_validation.TokenDisparity + - content_type: test + content_id: validmind.model_validation.BleuScore + - content_type: test + content_id: validmind.model_validation.RougeScore + - content_type: test + content_id: validmind.model_validation.MeteorScore + - content_type: test + content_id: validmind.model_validation.ragas.AspectCritic + - content_type: test + content_id: validmind.model_validation.ToxicityScore + - content_type: test + content_id: validmind.model_validation.RegardScore +- id: monitoring_governance + title: Monitoring and Governance + index_only: true + sections: + - id: monitoring_plan + title: Monitoring Plan + guidelines: + - Describe the plan for ongoing monitoring of the model's performance, + including the frequency of evaluations, the performance metrics to be + assessed, and any thresholds or triggers for action. + - Explain the process for identifying and addressing any changes in the + model's performance or the underlying data that may require model + updates, recalibration, or retraining. + - Detail the procedures for model validation and backtesting, to ensure + the model remains accurate, reliable, and compliant with regulatory + requirements and industry standards. + - Discuss the governance framework in place to oversee the model's use, + including the roles and responsibilities of various stakeholders, such + as model developers, validators, and risk managers. + - Describe the model's documentation and version control procedures, to + ensure that changes, updates, and improvements are properly tracked + and recorded. + contents: + - content_id: monitoring_plan + content_type: text + options: + default_text: > + To ensure the ongoing effectiveness of the model, it will be + monitored on a regular basis. The monitoring plan includes the + following steps: + + + 1. **Performance Metrics Monitoring:** The key performance metrics + such as Accuracy, Precision, Recall, F1 Score, and ROC-AUC will be + + tracked on a regular basis. These metrics will be computed for + both the training, validation and test datasets to identify any + signs of + + overfitting or underperformance. + + 2. **Data Drift Monitoring:** Over time, the distribution of the + input data may change, a phenomenon known as data drift. This + could + + impact the model's performance. We will monitor the distributions + of the input features and the target variable to detect any + significant + + changes that may require retraining of the model. + + 3. **Outcome Monitoring:** The actual outcomes (whether a customer + exited the bank or not) will be compared with the model's + predictions + + to assess the model's performance in a real-world setting. + + 4. **Feature Importance Monitoring:** The importance of different + features for the model's predictions will be tracked. If there are + + significant shifts in feature importance, this could indicate + changes in the underlying patterns in the data. + + 5. **Periodic Model Retraining:** Depending on the findings from + the above monitoring activities, the model may need to be + retrained + + periodically. This will be done using the most recent data to + ensure that the model stays up-to-date with the latest patterns + and trends. + + + In terms of governance, a clear process will be put in place for + managing these monitoring activities. This will include clear + roles + + and responsibilities, documentation of the monitoring results, and + a process for deciding when and how to take action based on the + + monitoring results (for example, when to retrain the model). Any + major decisions about the model (such as changes to the model or + its + + features) will be made in a transparent and accountable manner, + with appropriate documentation and sign-off. + parent_section: monitoring_governance + - id: monitoring_implementation + title: Monitoring Implementation + guidelines: + - Describe the tools, systems, or platforms used to implement the + monitoring plan, including any relevant software, data pipelines, or + reporting tools. + - Detail the process for collecting and storing the data needed for + ongoing monitoring, including any data preprocessing, cleaning, or + transformation steps required. + - Explain the procedures for analyzing the model's performance metrics + and generating monitoring reports, including any statistical tests or + visualizations used to assess the model's performance and stability. + - Discuss the escalation process and communication channels for + reporting any significant deviations in the model's performance, as + well as the decision-making process for determining appropriate + actions, such as model updates or recalibration. + - Describe any training or educational programs in place to ensure that + relevant stakeholders, such as model developers, validators, and risk + managers, are equipped to understand, interpret, and act on the + monitoring results. + parent_section: monitoring_governance + contents: + - content_id: monitoring_implementation + content_type: text + options: + default_text: > + Implementing the monitoring plan requires a systematic approach + with clear steps and procedures. Here are the steps for + implementing + + the monitoring plan: + + + 1. **Establish Baseline Metrics:** At the outset, we'll establish + baseline performance metrics for the model. These will serve as + + reference points for future comparison. + + 2. **Automate Metric Calculation:** Performance metrics such as + Accuracy, Precision, Recall, F1 Score, and ROC-AUC will be + automatically + + calculated and recorded for both the training and validation + datasets after each run of the model. + + 3. **Set Up Data Drift Monitors:** We'll set up automated + monitoring of the distributions of the input features and the + target + + variable. Any significant changes in these distributions will + trigger alerts. + + 4. **Implement Outcome Monitoring:** We'll compare the model's + predictions with the actual outcomes on a regular basis. + Discrepancies + + will be investigated to understand the root cause. + + 5. **Track Feature Importance:** We'll keep a record of feature + importance as indicated by the model. Any significant shifts in + + feature importance over time will be thoroughly reviewed. + + 6. **Establish Retraining Protocols:** Based on the findings from + the above monitoring activities, we'll establish clear protocols + + for when and how the model should be retrained. This may include + criteria for triggering a retraining, procedures for carrying out + + the retraining, and protocols for testing and validating the + retrained model before it is put back into service. + + 7. **Document and Review:** All monitoring activities and their + results will be documented in a transparent and accessible manner. + + These documents will be reviewed regularly by a designated team or + individual to ensure that any issues are promptly identified and + addressed. + + + By implementing this monitoring plan, we can ensure that the model + continues to perform well and that any issues are quickly + + identified and addressed. The ultimate aim is to ensure that the + model continues to provide accurate and reliable predictions that + + can support the bank's decision-making processes. + - id: governance_plan + title: Governance Plan + guidelines: + - Describe the overall governance framework and processes established to + ensure proper oversight and management of the model, including the + roles and responsibilities of key stakeholders such as model + developers, validators, and risk managers. + - Detail the policies and procedures for model risk management, + including model risk identification, assessment, and mitigation + strategies. + - Explain the model approval process, including any internal or external + reviews, audits, or regulatory assessments that must be completed + before the model is put into production. + - Discuss the procedures for ongoing model maintenance, updates, and + improvements, including the documentation and version control + processes to track and record changes to the model. + - Describe the contingency plans in place to manage potential model + failures or performance issues, such as fallback models, alternative + data sources, or manual processes, and explain the criteria for + activating these contingency measures. + parent_section: monitoring_governance + contents: + - content_id: governance_plan + content_type: text + options: + default_text: > + Effective governance of the model is crucial to ensure its + reliability, security, and compliance with regulatory + requirements. + + Here is the plan for model governance: + + + 1. **Roles and Responsibilities:** Clear roles and + responsibilities will be assigned for model development, + validation, deployment, + + monitoring, and retraining. This will include a model owner, who + will have overall responsibility for the model, as well as others + + responsible for specific tasks. + + 2. **Model Documentation:** Comprehensive documentation will be + maintained for the model, including details of its development, + + validation, and performance, as well as any changes made to the + model or its inputs over time. This documentation will be updated + + regularly and will be accessible to all relevant stakeholders. + + 3. **Change Control:** Any changes to the model or its inputs will + be subject to a strict change control process, including + + documentation of the proposed change, review and approval by a + designated authority, testing and validation of the change, and a + + post-implementation review. + + 4. **Security and Access Control:** Measures will be put in place + to ensure the security of the model and its data, including + + access controls, data encryption, and regular security audits. + + 5. **Regulatory Compliance:** The model will be designed and + operated in compliance with all relevant regulatory requirements, + + and its compliance will be regularly reviewed and confirmed. + + 6. **Auditability:** The model and its operations will be + auditable, with clear and accessible records that can be reviewed + by + + internal or external auditors. + + 7. **Training and Awareness:** All individuals involved in the + development, operation, and oversight of the model will receive + + appropriate training and will be kept aware of their + responsibilities, the model's performance, and any relevant + developments or issues. diff --git a/notebooks/code_samples/nlp_and_llm/hugging_face_integration_demo.ipynb b/notebooks/code_samples/nlp_and_llm/hugging_face_integration_demo.ipynb index dd7e03d05..c424355de 100644 --- a/notebooks/code_samples/nlp_and_llm/hugging_face_integration_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/hugging_face_integration_demo.ipynb @@ -132,8 +132,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Analytics`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/nlp_and_llm/hugging_face_summarization_demo.ipynb b/notebooks/code_samples/nlp_and_llm/hugging_face_summarization_demo.ipynb index 8209e65c2..14816ce33 100644 --- a/notebooks/code_samples/nlp_and_llm/hugging_face_summarization_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/hugging_face_summarization_demo.ipynb @@ -129,8 +129,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Analytics`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/nlp_and_llm/llm_summarization_demo.ipynb b/notebooks/code_samples/nlp_and_llm/llm_summarization_demo.ipynb index 159a16e79..fe6e87956 100644 --- a/notebooks/code_samples/nlp_and_llm/llm_summarization_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/llm_summarization_demo.ipynb @@ -163,8 +163,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Sales/Prospecting`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb b/notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb index 49633b3a9..eff5de7fb 100644 --- a/notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb @@ -159,8 +159,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Analytics`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/nlp_and_llm/rag_benchmark_demo.ipynb b/notebooks/code_samples/nlp_and_llm/rag_benchmark_demo.ipynb index 35969b1eb..0296db81b 100644 --- a/notebooks/code_samples/nlp_and_llm/rag_benchmark_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/rag_benchmark_demo.ipynb @@ -198,8 +198,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Analytics`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." @@ -217,11 +215,24 @@ "\n", "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", "\n", - "2. Under **TEMPLATE**, select `Gen AI RAG Template`.\n", + "2. Under **TEMPLATE**, select `Gen AI RAG`.\n", "\n", "3. Click **Use Template** to apply the template." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
Can't select this template?\n", + "

\n", + "Your organization administrators may need to add it to your template library:\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb b/notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb index 485c81ffd..ca868c641 100644 --- a/notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb @@ -172,9 +172,6 @@ "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", - "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Analytics`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." @@ -192,11 +189,24 @@ "\n", "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", "\n", - "2. Under **TEMPLATE**, select `Gen AI RAG Template`.\n", + "2. Under **TEMPLATE**, select `Gen AI RAG`.\n", "\n", "3. Click **Use Template** to apply the template." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
Can't select this template?\n", + "

\n", + "Your organization administrators may need to add it to your template library:\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/notebooks/code_samples/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb b/notebooks/code_samples/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb index 1ff6ca3e0..caa2aabac 100644 --- a/notebooks/code_samples/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb +++ b/notebooks/code_samples/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb @@ -169,8 +169,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb b/notebooks/code_samples/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb index 04f64a31e..14f14821a 100644 --- a/notebooks/code_samples/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb +++ b/notebooks/code_samples/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb @@ -167,8 +167,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/regression/quickstart_regression_full_suite.ipynb b/notebooks/code_samples/regression/quickstart_regression_full_suite.ipynb index 90992448e..91765950e 100644 --- a/notebooks/code_samples/regression/quickstart_regression_full_suite.ipynb +++ b/notebooks/code_samples/regression/quickstart_regression_full_suite.ipynb @@ -137,8 +137,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/time_series/quickstart_time_series_full_suite.ipynb b/notebooks/code_samples/time_series/quickstart_time_series_full_suite.ipynb index 27e572717..4b4865f0b 100644 --- a/notebooks/code_samples/time_series/quickstart_time_series_full_suite.ipynb +++ b/notebooks/code_samples/time_series/quickstart_time_series_full_suite.ipynb @@ -171,8 +171,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_samples/time_series/quickstart_time_series_high_code.ipynb b/notebooks/code_samples/time_series/quickstart_time_series_high_code.ipynb index 33a2e7220..55c24a46a 100644 --- a/notebooks/code_samples/time_series/quickstart_time_series_high_code.ipynb +++ b/notebooks/code_samples/time_series/quickstart_time_series_high_code.ipynb @@ -172,8 +172,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Analytics`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." @@ -191,7 +189,7 @@ "\n", "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", "\n", - "2. Under **TEMPLATE**, select `Time Series Forecasting`.\n", + "2. Under **TEMPLATE**, select `Time Series Forecasting with ML`.\n", "\n", "3. Click **Use Template** to apply the template." ] diff --git a/notebooks/code_sharing/credit_risk/assign_prediction_probabilities.ipynb b/notebooks/code_sharing/credit_risk/assign_prediction_probabilities.ipynb index cb4a7195e..246c0a30f 100644 --- a/notebooks/code_sharing/credit_risk/assign_prediction_probabilities.ipynb +++ b/notebooks/code_sharing/credit_risk/assign_prediction_probabilities.ipynb @@ -105,8 +105,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_sharing/insurance_mortality/insurance_validation_demo.ipynb b/notebooks/code_sharing/insurance_mortality/insurance_validation_demo.ipynb index 995f87013..1c481e0a9 100644 --- a/notebooks/code_sharing/insurance_mortality/insurance_validation_demo.ipynb +++ b/notebooks/code_sharing/insurance_mortality/insurance_validation_demo.ipynb @@ -114,8 +114,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb b/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb index fc885bb19..890e60d5f 100644 --- a/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb +++ b/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb @@ -167,8 +167,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk - Underwriting - Loans`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." @@ -186,7 +184,7 @@ "\n", "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", "\n", - "2. Under **TEMPLATE**, select `Time Series Forecasting`.\n", + "2. Under **TEMPLATE**, select `Time Series Forecasting with ML`.\n", "\n", "3. Click **Use Template** to apply the template." ] diff --git a/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb b/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb index 40ef8e454..85a6c2731 100644 --- a/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb +++ b/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb @@ -273,8 +273,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/code_sharing/regression/regression_unit_metrics.ipynb b/notebooks/code_sharing/regression/regression_unit_metrics.ipynb index 3af977523..3d126563f 100644 --- a/notebooks/code_sharing/regression/regression_unit_metrics.ipynb +++ b/notebooks/code_sharing/regression/regression_unit_metrics.ipynb @@ -150,8 +150,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/assign_scores_complete_tutorial.ipynb b/notebooks/how_to/assign_scores_complete_tutorial.ipynb index 998a78ace..933da518a 100644 --- a/notebooks/how_to/assign_scores_complete_tutorial.ipynb +++ b/notebooks/how_to/assign_scores_complete_tutorial.ipynb @@ -168,8 +168,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Analytics`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/configure_dataset_features.ipynb b/notebooks/how_to/configure_dataset_features.ipynb index 30d86a1ee..6af578eef 100644 --- a/notebooks/how_to/configure_dataset_features.ipynb +++ b/notebooks/how_to/configure_dataset_features.ipynb @@ -158,8 +158,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/customize_test_result_descriptions.ipynb b/notebooks/how_to/customize_test_result_descriptions.ipynb index c3aca95e1..ab36da913 100644 --- a/notebooks/how_to/customize_test_result_descriptions.ipynb +++ b/notebooks/how_to/customize_test_result_descriptions.ipynb @@ -121,8 +121,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/document_multiple_results_for_the_same_test.ipynb b/notebooks/how_to/document_multiple_results_for_the_same_test.ipynb index 74e04e1ab..59edac4cb 100644 --- a/notebooks/how_to/document_multiple_results_for_the_same_test.ipynb +++ b/notebooks/how_to/document_multiple_results_for_the_same_test.ipynb @@ -175,8 +175,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/load_datasets_predictions.ipynb b/notebooks/how_to/load_datasets_predictions.ipynb index a2dacdeea..050269cac 100644 --- a/notebooks/how_to/load_datasets_predictions.ipynb +++ b/notebooks/how_to/load_datasets_predictions.ipynb @@ -178,8 +178,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/log_metrics_over_time.ipynb b/notebooks/how_to/log_metrics_over_time.ipynb index f3bbfbdfa..058a54e19 100644 --- a/notebooks/how_to/log_metrics_over_time.ipynb +++ b/notebooks/how_to/log_metrics_over_time.ipynb @@ -173,8 +173,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Credit Risk - CECL`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/run_documentation_sections.ipynb b/notebooks/how_to/run_documentation_sections.ipynb index fb4412df9..340b6b864 100644 --- a/notebooks/how_to/run_documentation_sections.ipynb +++ b/notebooks/how_to/run_documentation_sections.ipynb @@ -168,8 +168,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/run_documentation_tests_with_config.ipynb b/notebooks/how_to/run_documentation_tests_with_config.ipynb index a96df1a5b..669f240c4 100644 --- a/notebooks/how_to/run_documentation_tests_with_config.ipynb +++ b/notebooks/how_to/run_documentation_tests_with_config.ipynb @@ -172,8 +172,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/run_tests/1_run_dataset_based_tests.ipynb b/notebooks/how_to/run_tests/1_run_dataset_based_tests.ipynb index a901849c4..d032c6c77 100644 --- a/notebooks/how_to/run_tests/1_run_dataset_based_tests.ipynb +++ b/notebooks/how_to/run_tests/1_run_dataset_based_tests.ipynb @@ -170,8 +170,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." @@ -584,7 +582,7 @@ ], "metadata": { "kernelspec": { - "display_name": "3.10.13", + "display_name": "Python 3", "language": "python", "name": "python3" }, diff --git a/notebooks/how_to/run_tests/2_run_comparison_tests.ipynb b/notebooks/how_to/run_tests/2_run_comparison_tests.ipynb index 5aa8d78db..ebd3f6236 100644 --- a/notebooks/how_to/run_tests/2_run_comparison_tests.ipynb +++ b/notebooks/how_to/run_tests/2_run_comparison_tests.ipynb @@ -174,8 +174,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/run_tests_that_require_multiple_datasets.ipynb b/notebooks/how_to/run_tests_that_require_multiple_datasets.ipynb index d292c4461..55c0857c7 100644 --- a/notebooks/how_to/run_tests_that_require_multiple_datasets.ipynb +++ b/notebooks/how_to/run_tests_that_require_multiple_datasets.ipynb @@ -170,8 +170,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/run_unit_metrics.ipynb b/notebooks/how_to/run_unit_metrics.ipynb index 235c538a9..d9ffe1175 100644 --- a/notebooks/how_to/run_unit_metrics.ipynb +++ b/notebooks/how_to/run_unit_metrics.ipynb @@ -166,8 +166,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/how_to/use_dataset_model_objects.ipynb b/notebooks/how_to/use_dataset_model_objects.ipynb index 12431963c..76faa2ffd 100644 --- a/notebooks/how_to/use_dataset_model_objects.ipynb +++ b/notebooks/how_to/use_dataset_model_objects.ipynb @@ -179,8 +179,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/quickstart/quickstart_model_documentation.ipynb b/notebooks/quickstart/quickstart_model_documentation.ipynb index e88d9a803..933a7d331 100644 --- a/notebooks/quickstart/quickstart_model_documentation.ipynb +++ b/notebooks/quickstart/quickstart_model_documentation.ipynb @@ -226,8 +226,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index ffa1bc42a..ff59a7514 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -195,8 +195,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down — don’t worry, we’ll adjust these permissions next for validation.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/templates/_install-initialize-validmind.ipynb b/notebooks/templates/_install-initialize-validmind.ipynb index b5114ce8a..73a67824d 100644 --- a/notebooks/templates/_install-initialize-validmind.ipynb +++ b/notebooks/templates/_install-initialize-validmind.ipynb @@ -52,22 +52,8 @@ "\n", "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", "\n", - "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))" - ] - }, - { - "cell_type": "markdown", - "id": "install-usecase", - "metadata": {}, - "source": [ - "            For example, to register a model for use with this notebook, select the following use case: `{use-case}`" - ] - }, - { - "cell_type": "markdown", - "id": "install-stakeholders", - "metadata": {}, - "source": [ + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/templates/e2e_template.py b/notebooks/templates/e2e_template.py index be4914c32..feaf16268 100644 --- a/notebooks/templates/e2e_template.py +++ b/notebooks/templates/e2e_template.py @@ -199,11 +199,13 @@ def open_in(filepath): def create_notebook(): """Creates a new Jupyter Notebook file by asking the user for a filename and opens it.""" - filename = input("Enter the name for the new notebook (without .ipynb extension): ").strip() - if not filename: + raw = input("Enter the name for the new notebook (without .ipynb extension): ").strip() + if not raw: print("Filename cannot be empty, file not created") return + # Normalize: replace spaces with underscores, enforce lowercase + filename = raw.replace(" ", "_").lower() if not filename.endswith(".ipynb"): filename += ".ipynb" @@ -474,41 +476,6 @@ def replace_variables( else: print_func("No value entered for {template}, skipping replacement") - # Handle {use-case} - if "{use-case}" in content or use_case_value is not None: - uc = use_case_value - if uc is None: - uc = input_func( - "Enter a value to replace {use-case} or type SKIP to exclude the template selection: " - ).strip() - else: - uc = uc.strip() - - if uc.upper() == "SKIP": - # Remove markdown cells whose id starts with 'install-usecase' - try: - nb = json.loads(content) - usecase_rules = {"install-usecase": {"markdown"}} - nb, removed = _remove_cells_by_prefix_rules(nb, usecase_rules) - if removed: - content = json.dumps(nb, ensure_ascii=False, indent=1) - print_func( - "Skipped use-case selection; removed " - f"{removed} cell(s) matching prefixes ['install-usecase']." - ) - else: - print_func("Skipped use-case selection; no matching cells found.") - except json.JSONDecodeError: - print_func( - "Skipped use-case selection, but file is not valid notebook JSON; no cells removed." - ) - elif uc: - content = content.replace("{use-case}", uc) - print_func(f"Use case: {uc}") - replacements_made = True - else: - print_func("No value entered for {use-case}, skipping replacement") - with open(filepath, "w", encoding="utf-8") as f: if not content.endswith("\n"): content += "\n" diff --git a/notebooks/tutorials/model_development/1-set_up_validmind.ipynb b/notebooks/tutorials/model_development/1-set_up_validmind.ipynb index f85b592d8..ad72e1abf 100644 --- a/notebooks/tutorials/model_development/1-set_up_validmind.ipynb +++ b/notebooks/tutorials/model_development/1-set_up_validmind.ipynb @@ -215,8 +215,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", "5. Click **Register Model** to add the model to your inventory." diff --git a/notebooks/tutorials/model_validation/1-set_up_validmind_for_validation.ipynb b/notebooks/tutorials/model_validation/1-set_up_validmind_for_validation.ipynb index b9154b80c..212e341eb 100644 --- a/notebooks/tutorials/model_validation/1-set_up_validmind_for_validation.ipynb +++ b/notebooks/tutorials/model_validation/1-set_up_validmind_for_validation.ipynb @@ -177,8 +177,6 @@ "\n", "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - " For example, to register a model for use with this notebook, select the following use case: `Marketing/Sales - Attrition/Churn Management`\n", - "\n", "4. Select your own name under the **MODEL OWNER** drop-down — don’t worry, we’ll adjust these permissions next for validation.\n", "\n", "5. Click **Register Model** to add the model to your inventory."