diff --git a/site/about/overview-model-documentation.qmd b/site/about/overview-model-documentation.qmd index 17c56cb014..443492984f 100644 --- a/site/about/overview-model-documentation.qmd +++ b/site/about/overview-model-documentation.qmd @@ -146,7 +146,7 @@ How the {{< var validmind.developer >}} works: [^1]: [Model risk management](overview-model-risk-management.qmd) -[^2]: [Supported models](/developer/supported-models.qmd) +[^2]: [Supported model frameworks](/developer/supported-model-frameworks.qmd) [^3]: [Customize document templates](/guide/templates/customize-document-templates.qmd) diff --git a/site/developer/_sidebar.yaml b/site/developer/_sidebar.yaml index 3b1680ea69..0569551bfe 100644 --- a/site/developer/_sidebar.yaml +++ b/site/developer/_sidebar.yaml @@ -10,7 +10,7 @@ website: # USING THE VARIABLE IN THE LINK TEXT MESSES UP THE MOBILE VIEW - text: "ValidMind Library" file: developer/validmind-library.qmd - - developer/supported-models.qmd + - developer/supported-model-frameworks.qmd - text: "---" - text: "Quickstart" - notebooks/quickstart/quickstart_model_documentation.ipynb diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd new file mode 100644 index 0000000000..801c6df8f8 --- /dev/null +++ b/site/developer/supported-model-frameworks.qmd @@ -0,0 +1,269 @@ +--- +# Copyright © 2023-2026 ValidMind Inc. All rights reserved. +# Refer to the LICENSE file in the root of this repository for details. +# SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial +title: "Supported model frameworks" +date: last-modified +aliases: + - /guide/supported-models.html + - /developer/model-documentation/supported-models.html + - /developer/supported-models.html +listing: + - id: next-models + type: grid + max-description-length: 250 + sort: false + fields: [title, description] + contents: + - ../model-testing/testing-overview.qmd + - ../model-testing/test-descriptions.qmd + - ../samples-jupyter-notebooks.qmd +--- + +The {{< var validmind.developer >}} supports a wide range of model frameworks for testing and documentation. This page explains which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. + +::: {.callout} +## The library is framework-agnostic + +Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. +::: + +## Framework support + +The {{< var validmind.developer >}} provides wrapper classes for common ML frameworks. Choose the wrapper that matches your model's framework: + +| Framework | Wrapper Class | Installation | +|-----------|---------------|--------------| +| scikit-learn, XGBoost, CatBoost, LightGBM, StatsModels | `SKlearnModel` | `pip install validmind` | +| PyTorch | `PyTorchModel` | `pip install validmind[pytorch]` | +| Hugging Face Transformers | `HFModel` | `pip install validmind` | +| LLM endpoints (OpenAI, Azure, etc.) | `FoundationModel` | `pip install validmind[llm]` | +| Any Python callable | `FunctionModel` | `pip install validmind` | +| R models (via rpy2) | `RModel` | `pip install validmind rpy2` | + +To install all optional dependencies: + +```bash +pip install validmind[all] +``` + +### Class hierarchy + +The following diagram shows how model wrapper classes relate to each other: + +```{mermaid} +classDiagram + VMModel <|-- SKlearnModel + VMModel <|-- PyTorchModel + VMModel <|-- HFModel + VMModel <|-- FunctionModel + VMModel <|-- RModel + SKlearnModel <|-- XGBoostModel + SKlearnModel <|-- CatBoostModel + SKlearnModel <|-- StatsModelsModel + FunctionModel <|-- FoundationModel + FunctionModel <|-- PipelineModel +``` + +## Test input requirements + +Different tests require different inputs from your model and dataset. Understanding these requirements helps you run the right tests for your use case. + +### When `predict()` is required + +Most model tests call your model's `predict()` method to generate predictions. This includes: + +- Performance metrics (accuracy, precision, recall, F1) +- Error analysis tests +- Robustness tests + +### When `predict_proba()` is needed + +Classification metrics that evaluate probability outputs require `predict_proba()`: + +- ROC-AUC score +- Precision-recall curves +- Calibration tests +- Probability distribution analysis + +If your model doesn't have `predict_proba()`, these tests will be skipped or return an error. + +### Test input flow + +```{mermaid} +flowchart LR + subgraph Inputs + Model[Model Object] + Dataset[Dataset] + Precomputed[Precomputed Predictions] + end + + Model --> predict["predict()"] + Model --> predict_proba["predict_proba()"] + Precomputed --> Dataset + + subgraph TestTypes[Test Types] + ModelTests[Model Tests] + DatasetTests[Dataset Tests] + ClassificationTests[Classification Metrics] + end + + predict --> ModelTests + predict --> DatasetTests + predict_proba --> ClassificationTests + Dataset --> DatasetTests +``` + +### Using precomputed predictions + +If you can't provide a model object (for example, if your model runs in a separate environment), you can pass precomputed predictions directly to the dataset: + +```python +vm_dataset = vm.init_dataset( + dataset=df, + target_column="target", + prediction_values=predictions, # numpy array of predictions + probability_values=probabilities # optional: for classification +) +``` + +Alternatively, if predictions are already in your dataframe: + +```python +vm_dataset = vm.init_dataset( + dataset=df, + target_column="target", + prediction_column="predicted", # column name in df + probability_column="probability" # optional: for classification +) +``` + +### Dataset-only tests + +Some tests analyze data quality and don't require a model at all: + +- Missing value analysis +- Class imbalance detection +- Feature correlation +- Outlier detection +- Data drift tests + +## Custom model wrappers + +For models that don't fit standard framework patterns, use these flexible wrappers. + +### FunctionModel + +Wrap any Python callable as a model: + +```python +from validmind.models import FunctionModel + +def my_predict(X): + # Your prediction logic here + return predictions + +vm_model = vm.init_model( + model=FunctionModel(predict_fn=my_predict), + input_id="my_model" +) +``` + +### PipelineModel + +Chain multiple models or processing steps: + +```python +from validmind.models import PipelineModel + +vm_model = vm.init_model( + model=PipelineModel(models=[preprocessor, model]), + input_id="my_pipeline" +) +``` + +## GenAI and LLM support + +The {{< var validmind.developer >}} provides specialized support for large language models and generative AI. + +### FoundationModel + +Use `FoundationModel` for LLM endpoints: + +```python +from validmind.models import FoundationModel +from validmind.prompt import Prompt + +prompt = Prompt( + template="Classify the sentiment: {text}", + variables=["text"] +) + +vm_model = vm.init_model( + model=FoundationModel( + prompt=prompt, + model="gpt-4", # or your model endpoint + ), + input_id="sentiment_classifier" +) +``` + +### LLM test suites + +Install LLM dependencies: + +```bash +pip install validmind[llm] +``` + +Available test suites for LLMs include: + +- Prompt injection detection +- Output consistency +- Hallucination detection +- Toxicity analysis +- Bias evaluation + +## RAG evaluation + +For retrieval-augmented generation (RAG) systems, the {{< var validmind.developer >}} integrates with [RAGAS](https://docs.ragas.io/) for comprehensive evaluation. + +### Dataset requirements + +RAG evaluation requires specific fields in your dataset: + +| Field | Type | Description | +|-------|------|-------------| +| `user_input` | str | The user's query | +| `response` | str | Model output | +| `retrieved_contexts` | List[str] | Retrieved context chunks | +| `reference` | str | Ground truth (required for some tests) | + +### Available RAG tests + +- **Faithfulness** — Measures how well the response is grounded in retrieved contexts +- **Context Recall** — Evaluates if relevant information was retrieved +- **Context Precision** — Measures relevance of retrieved contexts +- **Answer Relevancy** — Assesses if the response addresses the query + +## Python and dependency compatibility + +The {{< var validmind.developer >}} requires: + +- **Python:** >=3.9, <3.13 +- **Core dependencies:** pandas, numpy, scikit-learn + +Optional dependencies for specific frameworks: + +| Extra | Frameworks | +|-------|------------| +| `pytorch` | PyTorch, torchvision | +| `llm` | OpenAI, langchain, ragas | +| `xgboost` | XGBoost | +| `catboost` | CatBoost | +| `all` | All optional dependencies | + +## What's next + +:::{#next-models} +::: diff --git a/site/developer/supported-models.qmd b/site/developer/supported-models.qmd deleted file mode 100644 index 76e961af43..0000000000 --- a/site/developer/supported-models.qmd +++ /dev/null @@ -1,158 +0,0 @@ ---- -# Copyright © 2023-2026 ValidMind Inc. All rights reserved. -# Refer to the LICENSE file in the root of this repository for details. -# SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial -title: "Supported models" -date: last-modified -aliases: - - /guide/supported-models.html - - /developer/model-documentation/supported-models.html -listing: - - id: next-models - type: grid - max-description-length: 250 - sort: false - fields: [title, description] - contents: - - ../model-testing/testing-overview.qmd - - ../model-testing/test-descriptions.qmd - - ../samples-jupyter-notebooks.qmd ---- - -The {{< var validmind.developer >}} provides out-of-the-box support for testing and documentation for an array of model types and modeling packages. - -## What is a supported model? - -A _supported model_ refers to a model for which predefined testing or documentation functions exist in the {{< var validmind.developer >}}, provided that the model you are developing is documented using a supported version of our {{< var vm.developer >}}. These model types cover a very large portion of the models used in commercial and retail banking. - -::: {.callout} -## {{< var vm.product >}} does not limit our users to specific model types. - -- The {{< var validmind.developer >}} is extensible to support future model types or modeling packages to accomodate rapid developments in the AI space, including the advent of large language models (LLMs). -- You always have the flexibility to implement custom tests and integrate external test providers.[^1] -::: - - - -## Supported model types - -::: {.column-margin} -::: {.feature} -Vendor models -: {{< var vm.product >}} offers support for both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). - -::: - -::: - -### Traditional statistical models - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-30-ns} - -#### Linear regression -Models relationship between a scalar response and one or more explanatory variables. - -::: - -::: {.w-30-ns} - -#### Logistic regression -Models relationship between a scalar response and one or more explanatory variables. - -::: - -::: {.w-30-ns} -#### Time series -Analyzes data points collected or sequenced over time. - -::: - -:::: - -### Machine learning models - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns .pr2} -#### Hugging Face-compatible models -- Natural language processing (NLP) text classification — Categorizes text into predefined classes. -- Tabular classification — Assigns categories to tabular dataset entries. -- Tabular regression — Predicts continuous outcomes from tabular data. - -#### Neural networks -- Long short-term memory (LSTM) — Processes sequences of data, remembering inputs over long periods. -- Recurrent neural network (RNN) — Processes sequences by maintaining a state that reflects the history of processed elements. -- Convolutional neural network (CNN) — Primarily used for processing grid-like data such as images. - - -::: - -::: {.w-50-ns .pl2 .pr2} - -#### Tree-based models
(XGBoost / CatBoost / random forest) -- Classification — Predicts categorical outcomes using decision trees. -- Regression — Predicts continuous outcomes using decision trees. - -#### K-nearest neighbors (KNN) -- Classification — Assigns class by majority vote of the k-nearest neighbors. -- Regression — Predicts value based on the average of the k-nearest neighbors. - -#### Clustering -- K-means — Partitions _n_ observations into _k_ clusters in which each observation belongs to the cluster with the nearest mean. - - -::: - -:::: - -### Generative AI models - -#### Large language models (LLMs) -- Classification — Categorizes input into predefined classes. -- Text summarization — Generates concise summaries from longer texts. - -## Supported modeling libraries and other tools - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns} - -- **[scikit-learn](https://scikit-learn.org/stable/)** — A Python library for machine learning, offering a range of supervised and unsupervised learning algorithms. - -- **[statsmodels](https://www.statsmodels.org/stable/index.html)** — A Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests and exploring data. - -- **[PyTorch](https://pytorch.org/)** — An open-source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing. - -- **[Hugging Face Transformers](https://huggingface.co/docs/transformers/en/index)** — Provides thousands of pre-trained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, and text generation. - -- **[XGBoost](https://xgboost.readthedocs.io/en/stable/)** — An optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable, implementing machine learning algorithms under the Gradient Boosting framework. - -::: - -::: {.w-50-ns} - -- **[CatBoost](https://catboost.ai/)** — An open-source gradient boosting on decision trees library with categorical feature support out of the box, for ranking, classification, regression, and other ML tasks. - -- **[LightGBM](https://lightgbm.readthedocs.io/en/stable/)** — A fast, distributed, high-performance gradient boosting (GBDT, GBRT, GBM, or MART) framework based on decision tree algorithms, used for ranking, classification, and many other machine learning tasks. - -- **R models, via [rpy2 - R in Python](https://rpy2.github.io/)** — Facilitates the integration of R's statistical computing and graphics capabilities with Python, allowing for R models to be called from Python. - -- **Large language models (LLMs), via [OpenAI-compatible APIs](https://platform.openai.com/docs/introduction)** — Access to advanced AI models trained by OpenAI for a variety of natural language tasks, including text generation, translation, and analysis, through a compatible API interface. This support includes both the OpenAI API and the Azure OpenAI Service via API. -::: - -:::: - -## What's next - -:::{#next-models} -::: - - - - -[^1]: - - - [Implement custom tests](/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb) - - [Integrate external test providers](/notebooks/code_samples/custom_tests/integrate_external_test_providers.ipynb) \ No newline at end of file diff --git a/site/faq/faq-integrations.qmd b/site/faq/faq-integrations.qmd index b7c3b0c3bf..9f0b6b8168 100644 --- a/site/faq/faq-integrations.qmd +++ b/site/faq/faq-integrations.qmd @@ -14,7 +14,7 @@ listing: sort: false fields: [title, description] contents: - - ../developer/supported-models.qmd + - ../developer/supported-model-frameworks.qmd - ../about/overview-llm-features.qmd - ../about/deployment/deployment-options.qmd categories: ["supported libraries", "supported languages", "integrations", "images", "large language models", "explainability", "deployment options", "validmind library"] @@ -82,7 +82,7 @@ We will be implementing connector interfaces allowing extraction of relationship [^1]: [{{< var validmind.developer >}}](/developer/validmind-library.qmd) -[^2]: [Supported modeling libraries and other tools](/developer/supported-models.qmd#supported-modeling-libraries-and-other-tools) +[^2]: [Supported model frameworks](/developer/supported-model-frameworks.qmd) [^3]: [Matplotlib](https://matplotlib.org/)