\n",
" \n",
- " | validmind.data_validation.ACFandPACFPlot | \n",
- " AC Fand PACF Plot | \n",
- " Analyzes time series data using Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots to... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'forecasting', 'statistical_test', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.ADF | \n",
- " ADF | \n",
- " Assesses the stationarity of a time series dataset using the Augmented Dickey-Fuller (ADF) test.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'statsmodels', 'forecasting', 'statistical_test', 'stationarity'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.AutoAR | \n",
- " Auto AR | \n",
- " Automatically identifies the optimal Autoregressive (AR) order for a time series using BIC and AIC criteria.... | \n",
- " ['dataset'] | \n",
- " {'max_ar_order': {'type': 'int', 'default': 3}} | \n",
- " ['time_series_data', 'statsmodels', 'forecasting', 'statistical_test'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.AutoMA | \n",
- " Auto MA | \n",
- " Automatically selects the optimal Moving Average (MA) order for each variable in a time series dataset based on... | \n",
- " ['dataset'] | \n",
- " {'max_ma_order': {'type': 'int', 'default': 3}} | \n",
- " ['time_series_data', 'statsmodels', 'forecasting', 'statistical_test'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.AutoStationarity | \n",
- " Auto Stationarity | \n",
- " Automates Augmented Dickey-Fuller test to assess stationarity across multiple time series in a DataFrame.... | \n",
- " ['dataset'] | \n",
- " {'max_order': {'type': 'int', 'default': 5}, 'threshold': {'type': 'float', 'default': 0.05}} | \n",
- " ['time_series_data', 'statsmodels', 'forecasting', 'statistical_test'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.BivariateScatterPlots | \n",
- " Bivariate Scatter Plots | \n",
- " Generates bivariate scatterplots to visually inspect relationships between pairs of numerical predictor variables... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'numerical_data', 'visualization'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.BoxPierce | \n",
- " Box Pierce | \n",
- " Detects autocorrelation in time-series data through the Box-Pierce test to validate model performance.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'forecasting', 'statistical_test', 'statsmodels'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.ChiSquaredFeaturesTable | \n",
- " Chi Squared Features Table | \n",
- " Assesses the statistical association between categorical features and a target variable using the Chi-Squared test.... | \n",
- " ['dataset'] | \n",
- " {'p_threshold': {'type': '_empty', 'default': 0.05}} | \n",
- " ['tabular_data', 'categorical_data', 'statistical_test'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.ClassImbalance | \n",
- " Class Imbalance | \n",
- " Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.... | \n",
- " ['dataset'] | \n",
- " {'min_percent_threshold': {'type': 'int', 'default': 10}} | \n",
- " ['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.DatasetDescription | \n",
- " Dataset Description | \n",
- " Provides comprehensive analysis and statistical summaries of each column in a machine learning model's dataset.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'time_series_data', 'text_data'] | \n",
- " ['classification', 'regression', 'text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.DatasetSplit | \n",
- " Dataset Split | \n",
- " Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML... | \n",
- " ['datasets'] | \n",
- " {} | \n",
- " ['tabular_data', 'time_series_data', 'text_data'] | \n",
- " ['classification', 'regression', 'text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.DescriptiveStatistics | \n",
- " Descriptive Statistics | \n",
- " Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'time_series_data', 'data_quality'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.DickeyFullerGLS | \n",
- " Dickey Fuller GLS | \n",
- " Assesses stationarity in time series data using the Dickey-Fuller GLS test to determine the order of integration.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'forecasting', 'unit_root_test'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.Duplicates | \n",
- " Duplicates | \n",
- " Tests dataset for duplicate entries, ensuring model reliability via data quality verification.... | \n",
- " ['dataset'] | \n",
- " {'min_threshold': {'type': '_empty', 'default': 1}} | \n",
- " ['tabular_data', 'data_quality', 'text_data'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.EngleGrangerCoint | \n",
- " Engle Granger Coint | \n",
- " Assesses the degree of co-movement between pairs of time series data using the Engle-Granger cointegration test.... | \n",
- " ['dataset'] | \n",
- " {'threshold': {'type': 'float', 'default': 0.05}} | \n",
- " ['time_series_data', 'statistical_test', 'forecasting'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.FeatureTargetCorrelationPlot | \n",
- " Feature Target Correlation Plot | \n",
- " Visualizes the correlation between input features and the model's target output in a color-coded horizontal bar... | \n",
- " ['dataset'] | \n",
- " {'fig_height': {'type': '_empty', 'default': 600}} | \n",
- " ['tabular_data', 'visualization', 'correlation'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.HighCardinality | \n",
- " High Cardinality | \n",
- " Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.... | \n",
- " ['dataset'] | \n",
- " {'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}} | \n",
- " ['tabular_data', 'data_quality', 'categorical_data'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.HighPearsonCorrelation | \n",
- " High Pearson Correlation | \n",
- " Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.... | \n",
- " ['dataset'] | \n",
- " {'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}} | \n",
- " ['tabular_data', 'data_quality', 'correlation'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.IQROutliersBarPlot | \n",
- " IQR Outliers Bar Plot | \n",
- " Visualizes outlier distribution across percentiles in numerical data using the Interquartile Range (IQR) method.... | \n",
- " ['dataset'] | \n",
- " {'threshold': {'type': 'float', 'default': 1.5}, 'fig_width': {'type': 'int', 'default': 800}} | \n",
- " ['tabular_data', 'visualization', 'numerical_data'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.IQROutliersTable | \n",
- " IQR Outliers Table | \n",
- " Determines and summarizes outliers in numerical features using the Interquartile Range method.... | \n",
- " ['dataset'] | \n",
- " {'threshold': {'type': 'float', 'default': 1.5}} | \n",
- " ['tabular_data', 'numerical_data'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.IsolationForestOutliers | \n",
- " Isolation Forest Outliers | \n",
- " Detects outliers in a dataset using the Isolation Forest algorithm and visualizes results through scatter plots.... | \n",
- " ['dataset'] | \n",
- " {'random_state': {'type': 'int', 'default': 0}, 'contamination': {'type': 'float', 'default': 0.1}, 'feature_columns': {'type': 'list', 'default': None}} | \n",
- " ['tabular_data', 'anomaly_detection'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.JarqueBera | \n",
- " Jarque Bera | \n",
- " Assesses normality of dataset features in an ML model using the Jarque-Bera test.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'data_distribution', 'statistical_test', 'statsmodels'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.KPSS | \n",
- " KPSS | \n",
- " Assesses the stationarity of time-series data in a machine learning model using the KPSS unit root test.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'stationarity', 'unit_root_test', 'statsmodels'] | \n",
- " ['data_validation'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.LJungBox | \n",
- " L Jung Box | \n",
- " Assesses autocorrelations in dataset features by performing a Ljung-Box test on each feature.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'forecasting', 'statistical_test', 'statsmodels'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.LaggedCorrelationHeatmap | \n",
- " Lagged Correlation Heatmap | \n",
- " Assesses and visualizes correlation between target variable and lagged independent variables in a time-series... | \n",
- " ['dataset'] | \n",
- " {'num_lags': {'type': 'int', 'default': 10}} | \n",
- " ['time_series_data', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.MissingValues | \n",
- " Missing Values | \n",
- " Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold.... | \n",
- " ['dataset'] | \n",
- " {'min_threshold': {'type': 'int', 'default': 1}} | \n",
- " ['tabular_data', 'data_quality'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.MissingValuesBarPlot | \n",
- " Missing Values Bar Plot | \n",
- " Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on... | \n",
- " ['dataset'] | \n",
- " {'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}} | \n",
- " ['tabular_data', 'data_quality', 'visualization'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.MutualInformation | \n",
- " Mutual Information | \n",
- " Calculates mutual information scores between features and target variable to evaluate feature relevance.... | \n",
- " ['dataset'] | \n",
- " {'min_threshold': {'type': 'float', 'default': 0.01}, 'task': {'type': 'str', 'default': 'classification'}} | \n",
- " ['feature_selection', 'data_analysis'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.PearsonCorrelationMatrix | \n",
- " Pearson Correlation Matrix | \n",
- " Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'numerical_data', 'correlation'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.PhillipsPerronArch | \n",
- " Phillips Perron Arch | \n",
- " Assesses the stationarity of time series data in each feature of the ML model using the Phillips-Perron test.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'forecasting', 'statistical_test', 'unit_root_test'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.ProtectedClassesDescription | \n",
- " Protected Classes Description | \n",
- " Visualizes the distribution of protected classes in the dataset relative to the target variable... | \n",
- " ['dataset'] | \n",
- " {'protected_classes': {'type': '_empty', 'default': None}} | \n",
- " ['bias_and_fairness', 'descriptive_statistics'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.RollingStatsPlot | \n",
- " Rolling Stats Plot | \n",
- " Evaluates the stationarity of time series data by plotting its rolling mean and standard deviation over a specified... | \n",
- " ['dataset'] | \n",
- " {'window_size': {'type': 'int', 'default': 12}} | \n",
- " ['time_series_data', 'visualization', 'stationarity'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.RunsTest | \n",
- " Runs Test | \n",
- " Executes Runs Test on ML model to detect non-random patterns in output data sequence.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'statistical_test', 'statsmodels'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.ScatterPlot | \n",
- " Scatter Plot | \n",
- " Assesses visual relationships, patterns, and outliers among features in a dataset through scatter plot matrices.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'visualization'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.ScoreBandDefaultRates | \n",
- " Score Band Default Rates | \n",
- " Analyzes default rates and population distribution across credit score bands.... | \n",
- " ['dataset', 'model'] | \n",
- " {'score_column': {'type': 'str', 'default': 'score'}, 'score_bands': {'type': 'list', 'default': None}} | \n",
- " ['visualization', 'credit_risk', 'scorecard'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.SeasonalDecompose | \n",
- " Seasonal Decompose | \n",
- " Assesses patterns and seasonality in a time series dataset by decomposing its features into foundational components.... | \n",
- " ['dataset'] | \n",
- " {'seasonal_model': {'type': 'str', 'default': 'additive'}} | \n",
- " ['time_series_data', 'seasonality', 'statsmodels'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.ShapiroWilk | \n",
- " Shapiro Wilk | \n",
- " Evaluates feature-wise normality of training data using the Shapiro-Wilk test.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'data_distribution', 'statistical_test'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.Skewness | \n",
- " Skewness | \n",
- " Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data... | \n",
- " ['dataset'] | \n",
- " {'max_threshold': {'type': '_empty', 'default': 1}} | \n",
- " ['data_quality', 'tabular_data'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.SpreadPlot | \n",
- " Spread Plot | \n",
- " Assesses potential correlations between pairs of time series variables through visualization to enhance... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TabularCategoricalBarPlots | \n",
- " Tabular Categorical Bar Plots | \n",
- " Generates and visualizes bar plots for each category in categorical features to evaluate the dataset's composition.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'visualization'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TabularDateTimeHistograms | \n",
- " Tabular Date Time Histograms | \n",
- " Generates histograms to provide graphical insight into the distribution of time intervals in a model's datetime... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'visualization'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TabularDescriptionTables | \n",
- " Tabular Description Tables | \n",
- " Summarizes key descriptive statistics for numerical, categorical, and datetime variables in a dataset.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TabularNumericalHistograms | \n",
- " Tabular Numerical Histograms | \n",
- " Generates histograms for each numerical feature in a dataset to provide visual insights into data distribution and... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'visualization'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TargetRateBarPlots | \n",
- " Target Rate Bar Plots | \n",
- " Generates bar plots visualizing the default rates of categorical features for a classification machine learning... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'visualization', 'categorical_data'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TimeSeriesDescription | \n",
- " Time Series Description | \n",
- " Generates a detailed analysis for the provided time series dataset, summarizing key statistics to identify trends,... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'analysis'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TimeSeriesDescriptiveStatistics | \n",
- " Time Series Descriptive Statistics | \n",
- " Evaluates the descriptive statistics of a time series dataset to identify trends, patterns, and data quality issues.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'analysis'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TimeSeriesFrequency | \n",
- " Time Series Frequency | \n",
- " Evaluates consistency of time series data frequency and generates a frequency plot.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TimeSeriesHistogram | \n",
- " Time Series Histogram | \n",
- " Visualizes distribution of time-series data using histograms and Kernel Density Estimation (KDE) lines.... | \n",
- " ['dataset'] | \n",
- " {'nbins': {'type': '_empty', 'default': 30}} | \n",
- " ['data_validation', 'visualization', 'time_series_data'] | \n",
- " ['regression', 'time_series_forecasting'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TimeSeriesLinePlot | \n",
- " Time Series Line Plot | \n",
- " Generates and analyses time-series data through line plots revealing trends, patterns, anomalies over time.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TimeSeriesMissingValues | \n",
- " Time Series Missing Values | \n",
- " Validates time-series data quality by confirming the count of missing values is below a certain threshold.... | \n",
- " ['dataset'] | \n",
- " {'min_threshold': {'type': 'int', 'default': 1}} | \n",
- " ['time_series_data'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TimeSeriesOutliers | \n",
- " Time Series Outliers | \n",
- " Identifies and visualizes outliers in time-series data using the z-score method.... | \n",
- " ['dataset'] | \n",
- " {'zscore_threshold': {'type': 'int', 'default': 3}} | \n",
- " ['time_series_data'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.TooManyZeroValues | \n",
- " Too Many Zero Values | \n",
- " Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold... | \n",
- " ['dataset'] | \n",
- " {'max_percent_threshold': {'type': 'float', 'default': 0.03}} | \n",
- " ['tabular_data'] | \n",
- " ['regression', 'classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.UniqueRows | \n",
- " Unique Rows | \n",
- " Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.... | \n",
- " ['dataset'] | \n",
- " {'min_percent_threshold': {'type': 'float', 'default': 1}} | \n",
- " ['tabular_data'] | \n",
- " ['regression', 'classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.WOEBinPlots | \n",
- " WOE Bin Plots | \n",
- " Generates visualizations of Weight of Evidence (WoE) and Information Value (IV) for understanding predictive power... | \n",
- " ['dataset'] | \n",
- " {'breaks_adj': {'type': 'list', 'default': None}, 'fig_height': {'type': 'int', 'default': 600}, 'fig_width': {'type': 'int', 'default': 500}} | \n",
- " ['tabular_data', 'visualization', 'categorical_data'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.WOEBinTable | \n",
- " WOE Bin Table | \n",
- " Assesses the Weight of Evidence (WoE) and Information Value (IV) of each feature to evaluate its predictive power... | \n",
- " ['dataset'] | \n",
- " {'breaks_adj': {'type': 'list', 'default': None}} | \n",
- " ['tabular_data', 'categorical_data'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.ZivotAndrewsArch | \n",
- " Zivot Andrews Arch | \n",
- " Evaluates the order of integration and stationarity of time series data using the Zivot-Andrews unit root test.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'stationarity', 'unit_root_test'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.CommonWords | \n",
- " Common Words | \n",
- " Assesses the most frequent non-stopwords in a text column for identifying prevalent language patterns.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'visualization', 'frequency_analysis'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.Hashtags | \n",
- " Hashtags | \n",
- " Assesses hashtag frequency in a text column, highlighting usage trends and potential dataset bias or spam.... | \n",
- " ['dataset'] | \n",
- " {'top_hashtags': {'type': 'int', 'default': 25}} | \n",
- " ['nlp', 'text_data', 'visualization', 'frequency_analysis'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.LanguageDetection | \n",
- " Language Detection | \n",
- " Assesses the diversity of languages in a textual dataset by detecting and visualizing the distribution of languages.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.Mentions | \n",
- " Mentions | \n",
- " Calculates and visualizes frequencies of '@' prefixed mentions in a text-based dataset for NLP model analysis.... | \n",
- " ['dataset'] | \n",
- " {'top_mentions': {'type': 'int', 'default': 25}} | \n",
- " ['nlp', 'text_data', 'visualization', 'frequency_analysis'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.PolarityAndSubjectivity | \n",
- " Polarity And Subjectivity | \n",
- " Analyzes the polarity and subjectivity of text data within a given dataset to visualize the sentiment distribution.... | \n",
- " ['dataset'] | \n",
- " {'threshold_subjectivity': {'type': '_empty', 'default': 0.5}, 'threshold_polarity': {'type': '_empty', 'default': 0}} | \n",
- " ['nlp', 'text_data', 'data_validation'] | \n",
- " ['nlp'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.Punctuations | \n",
- " Punctuations | \n",
- " Analyzes and visualizes the frequency distribution of punctuation usage in a given text dataset.... | \n",
- " ['dataset'] | \n",
- " {'count_mode': {'type': '_empty', 'default': 'token'}} | \n",
- " ['nlp', 'text_data', 'visualization', 'frequency_analysis'] | \n",
- " ['text_classification', 'text_summarization', 'nlp'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.Sentiment | \n",
- " Sentiment | \n",
- " Analyzes the sentiment of text data within a dataset using the VADER sentiment analysis tool.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'data_validation'] | \n",
- " ['nlp'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.StopWords | \n",
- " Stop Words | \n",
- " Evaluates and visualizes the frequency of English stop words in a text dataset against a defined threshold.... | \n",
- " ['dataset'] | \n",
- " {'min_percent_threshold': {'type': 'float', 'default': 0.5}, 'num_words': {'type': 'int', 'default': 25}} | \n",
- " ['nlp', 'text_data', 'frequency_analysis', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.TextDescription | \n",
- " Text Description | \n",
- " Conducts comprehensive textual analysis on a dataset using NLTK to evaluate various parameters and generate... | \n",
- " ['dataset'] | \n",
- " {'unwanted_tokens': {'type': 'set', 'default': {\"s'\", \"'s\", ' ', 'mr', \"''\", 'dollar', 'dr', 'mrs', '``', 's', 'us', 'ms'}}, 'lang': {'type': 'str', 'default': 'english'}} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.data_validation.nlp.Toxicity | \n",
- " Toxicity | \n",
- " Assesses the toxicity of text data within a dataset to visualize the distribution of toxicity scores.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'data_validation'] | \n",
- " ['nlp'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.BertScore | \n",
- " Bert Score | \n",
- " Assesses the quality of machine-generated text using BERTScore metrics and visualizes results through histograms... | \n",
- " ['dataset', 'model'] | \n",
- " {'evaluation_model': {'type': '_empty', 'default': 'distilbert-base-uncased'}} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.BleuScore | \n",
- " Bleu Score | \n",
- " Evaluates the quality of machine-generated text using BLEU metrics and visualizes the results through histograms... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ClusterSizeDistribution | \n",
- " Cluster Size Distribution | \n",
- " Assesses the performance of clustering models by comparing the distribution of cluster sizes in model predictions... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ContextualRecall | \n",
- " Contextual Recall | \n",
- " Evaluates a Natural Language Generation model's ability to generate contextually relevant and factually correct... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.FeaturesAUC | \n",
- " Features AUC | \n",
- " Evaluates the discriminatory power of each individual feature within a binary classification model by calculating... | \n",
- " ['dataset'] | \n",
- " {'fontsize': {'type': 'int', 'default': 12}, 'figure_height': {'type': 'int', 'default': 500}} | \n",
- " ['feature_importance', 'AUC', 'visualization'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.MeteorScore | \n",
- " Meteor Score | \n",
- " Assesses the quality of machine-generated translations by comparing them to human-produced references using the... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ModelMetadata | \n",
- " Model Metadata | \n",
- " Compare metadata of different models and generate a summary table with the results.... | \n",
- " ['model'] | \n",
- " {} | \n",
- " ['model_training', 'metadata'] | \n",
- " ['regression', 'time_series_forecasting'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ModelPredictionResiduals | \n",
- " Model Prediction Residuals | \n",
- " Assesses normality and behavior of residuals in regression models through visualization and statistical tests.... | \n",
- " ['dataset', 'model'] | \n",
- " {'nbins': {'type': '_empty', 'default': 100}, 'p_value_threshold': {'type': '_empty', 'default': 0.05}, 'start_date': {'type': '_empty', 'default': None}, 'end_date': {'type': '_empty', 'default': None}} | \n",
- " ['regression'] | \n",
- " ['residual_analysis', 'visualization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.RegardScore | \n",
- " Regard Score | \n",
- " Assesses the sentiment and potential biases in text generated by NLP models by computing and visualizing regard... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.RegressionResidualsPlot | \n",
- " Regression Residuals Plot | \n",
- " Evaluates regression model performance using residual distribution and actual vs. predicted plots.... | \n",
- " ['model', 'dataset'] | \n",
- " {'bin_size': {'type': 'float', 'default': 0.1}} | \n",
- " ['model_performance', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.RougeScore | \n",
- " Rouge Score | \n",
- " Assesses the quality of machine-generated text using ROUGE metrics and visualizes the results to provide... | \n",
- " ['dataset', 'model'] | \n",
- " {'metric': {'type': '_empty', 'default': 'rouge-1'}} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.TimeSeriesPredictionWithCI | \n",
- " Time Series Prediction With CI | \n",
- " Assesses predictive accuracy and uncertainty in time series models, highlighting breaches beyond confidence... | \n",
- " ['dataset', 'model'] | \n",
- " {'confidence': {'type': '_empty', 'default': 0.95}} | \n",
- " ['model_predictions', 'visualization'] | \n",
- " ['regression', 'time_series_forecasting'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.TimeSeriesPredictionsPlot | \n",
- " Time Series Predictions Plot | \n",
- " Plot actual vs predicted values for time series data and generate a visual comparison for the model.... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['model_predictions', 'visualization'] | \n",
- " ['regression', 'time_series_forecasting'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.TimeSeriesR2SquareBySegments | \n",
- " Time Series R2 Square By Segments | \n",
- " Evaluates the R-Squared values of regression models over specified time segments in time series data to assess... | \n",
- " ['dataset', 'model'] | \n",
- " {'segments': {'type': '_empty', 'default': None}} | \n",
- " ['model_performance', 'sklearn'] | \n",
- " ['regression', 'time_series_forecasting'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.TokenDisparity | \n",
- " Token Disparity | \n",
- " Evaluates the token disparity between reference and generated texts, visualizing the results through histograms and... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ToxicityScore | \n",
- " Toxicity Score | \n",
- " Assesses the toxicity levels of texts generated by NLP models to identify and mitigate harmful or offensive content.... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['nlp', 'text_data', 'visualization'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.ClusterDistribution | \n",
- " Cluster Distribution | \n",
- " Assesses the distribution of text embeddings across clusters produced by a model using KMeans clustering.... | \n",
- " ['model', 'dataset'] | \n",
- " {'num_clusters': {'type': 'int', 'default': 5}} | \n",
- " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
- " ['feature_extraction'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.CosineSimilarityComparison | \n",
- " Cosine Similarity Comparison | \n",
- " Assesses the similarity between embeddings generated by different models using Cosine Similarity, providing both... | \n",
- " ['dataset', 'models'] | \n",
- " {} | \n",
- " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.CosineSimilarityDistribution | \n",
- " Cosine Similarity Distribution | \n",
- " Assesses the similarity between predicted text embeddings from a model using a Cosine Similarity distribution... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
- " ['feature_extraction'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.CosineSimilarityHeatmap | \n",
- " Cosine Similarity Heatmap | \n",
- " Generates an interactive heatmap to visualize the cosine similarities among embeddings derived from a given model.... | \n",
- " ['dataset', 'model'] | \n",
- " {'title': {'type': '_empty', 'default': 'Cosine Similarity Matrix'}, 'color': {'type': '_empty', 'default': 'Cosine Similarity'}, 'xaxis_title': {'type': '_empty', 'default': 'Index'}, 'yaxis_title': {'type': '_empty', 'default': 'Index'}, 'color_scale': {'type': '_empty', 'default': 'Blues'}} | \n",
- " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.DescriptiveAnalytics | \n",
- " Descriptive Analytics | \n",
- " Evaluates statistical properties of text embeddings in an ML model via mean, median, and standard deviation... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
- " ['feature_extraction'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.EmbeddingsVisualization2D | \n",
- " Embeddings Visualization2 D | \n",
- " Visualizes 2D representation of text embeddings generated by a model using t-SNE technique.... | \n",
- " ['model', 'dataset'] | \n",
- " {'cluster_column': {'type': None, 'default': None}, 'perplexity': {'type': 'int', 'default': 30}} | \n",
- " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
- " ['feature_extraction'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.EuclideanDistanceComparison | \n",
- " Euclidean Distance Comparison | \n",
- " Assesses and visualizes the dissimilarity between model embeddings using Euclidean distance, providing insights... | \n",
- " ['dataset', 'models'] | \n",
- " {} | \n",
- " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.EuclideanDistanceHeatmap | \n",
- " Euclidean Distance Heatmap | \n",
- " Generates an interactive heatmap to visualize the Euclidean distances among embeddings derived from a given model.... | \n",
- " ['dataset', 'model'] | \n",
- " {'title': {'type': '_empty', 'default': 'Euclidean Distance Matrix'}, 'color': {'type': '_empty', 'default': 'Euclidean Distance'}, 'xaxis_title': {'type': '_empty', 'default': 'Index'}, 'yaxis_title': {'type': '_empty', 'default': 'Index'}, 'color_scale': {'type': '_empty', 'default': 'Blues'}} | \n",
- " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.PCAComponentsPairwisePlots | \n",
- " PCA Components Pairwise Plots | \n",
- " Generates scatter plots for pairwise combinations of principal component analysis (PCA) components of model... | \n",
- " ['dataset', 'model'] | \n",
- " {'n_components': {'type': '_empty', 'default': 3}} | \n",
- " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.StabilityAnalysisKeyword | \n",
- " Stability Analysis Keyword | \n",
- " Evaluates robustness of embedding models to keyword swaps in the test dataset.... | \n",
- " ['dataset', 'model'] | \n",
- " {'keyword_dict': {'type': None, 'default': None}, 'mean_similarity_threshold': {'type': 'float', 'default': 0.7}} | \n",
- " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
- " ['feature_extraction'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.StabilityAnalysisRandomNoise | \n",
- " Stability Analysis Random Noise | \n",
- " Assesses the robustness of text embeddings models to random noise introduced via text perturbations.... | \n",
- " ['dataset', 'model'] | \n",
- " {'probability': {'type': 'float', 'default': 0.02}, 'mean_similarity_threshold': {'type': 'float', 'default': 0.7}} | \n",
- " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
- " ['feature_extraction'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.StabilityAnalysisSynonyms | \n",
- " Stability Analysis Synonyms | \n",
- " Evaluates the stability of text embeddings models when words in test data are replaced by their synonyms randomly.... | \n",
- " ['dataset', 'model'] | \n",
- " {'probability': {'type': 'float', 'default': 0.02}, 'mean_similarity_threshold': {'type': 'float', 'default': 0.7}} | \n",
- " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
- " ['feature_extraction'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.StabilityAnalysisTranslation | \n",
- " Stability Analysis Translation | \n",
- " Evaluates robustness of text embeddings models to noise introduced by translating the original text to another... | \n",
- " ['dataset', 'model'] | \n",
- " {'source_lang': {'type': 'str', 'default': 'en'}, 'target_lang': {'type': 'str', 'default': 'fr'}, 'mean_similarity_threshold': {'type': 'float', 'default': 0.7}} | \n",
- " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
- " ['feature_extraction'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.embeddings.TSNEComponentsPairwisePlots | \n",
- " TSNE Components Pairwise Plots | \n",
- " Creates scatter plots for pairwise combinations of t-SNE components to visualize embeddings and highlight potential... | \n",
- " ['dataset', 'model'] | \n",
- " {'n_components': {'type': '_empty', 'default': 2}, 'perplexity': {'type': '_empty', 'default': 30}, 'title': {'type': '_empty', 'default': 't-SNE'}} | \n",
- " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.AnswerCorrectness | \n",
- " Answer Correctness | \n",
- " Evaluates the correctness of answers in a dataset with respect to the provided ground... | \n",
- " ['dataset'] | \n",
- " {'user_input_column': {'type': '_empty', 'default': 'user_input'}, 'response_column': {'type': '_empty', 'default': 'response'}, 'reference_column': {'type': '_empty', 'default': 'reference'}} | \n",
- " ['ragas', 'llm'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.AspectCritic | \n",
- " Aspect Critic | \n",
- " Evaluates generations against the following aspects: harmfulness, maliciousness,... | \n",
- " ['dataset'] | \n",
- " {'user_input_column': {'type': '_empty', 'default': 'user_input'}, 'response_column': {'type': '_empty', 'default': 'response'}, 'retrieved_contexts_column': {'type': '_empty', 'default': None}, 'aspects': {'type': 'list', 'default': ['coherence', 'conciseness', 'correctness', 'harmfulness', 'maliciousness']}, 'additional_aspects': {'type': 'list', 'default': None}} | \n",
- " ['ragas', 'llm', 'qualitative'] | \n",
- " ['text_summarization', 'text_generation', 'text_qa'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.ContextEntityRecall | \n",
- " Context Entity Recall | \n",
- " Evaluates the context entity recall for dataset entries and visualizes the results.... | \n",
- " ['dataset'] | \n",
- " {'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'reference_column': {'type': 'str', 'default': 'reference'}} | \n",
- " ['ragas', 'llm', 'retrieval_performance'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.ContextPrecision | \n",
- " Context Precision | \n",
- " Context Precision is a metric that evaluates whether all of the ground-truth... | \n",
- " ['dataset'] | \n",
- " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'reference_column': {'type': 'str', 'default': 'reference'}} | \n",
- " ['ragas', 'llm', 'retrieval_performance'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.ContextPrecisionWithoutReference | \n",
- " Context Precision Without Reference | \n",
- " Context Precision Without Reference is a metric used to evaluate the relevance of... | \n",
- " ['dataset'] | \n",
- " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'response_column': {'type': 'str', 'default': 'response'}} | \n",
- " ['ragas', 'llm', 'retrieval_performance'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.ContextRecall | \n",
- " Context Recall | \n",
- " Context recall measures the extent to which the retrieved context aligns with the... | \n",
- " ['dataset'] | \n",
- " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'reference_column': {'type': 'str', 'default': 'reference'}} | \n",
- " ['ragas', 'llm', 'retrieval_performance'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.Faithfulness | \n",
- " Faithfulness | \n",
- " Evaluates the faithfulness of the generated answers with respect to retrieved contexts.... | \n",
- " ['dataset'] | \n",
- " {'user_input_column': {'type': '_empty', 'default': 'user_input'}, 'response_column': {'type': '_empty', 'default': 'response'}, 'retrieved_contexts_column': {'type': '_empty', 'default': 'retrieved_contexts'}} | \n",
- " ['ragas', 'llm', 'rag_performance'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.NoiseSensitivity | \n",
- " Noise Sensitivity | \n",
- " Assesses the sensitivity of a Large Language Model (LLM) to noise in retrieved context by measuring how often it... | \n",
- " ['dataset'] | \n",
- " {'response_column': {'type': '_empty', 'default': 'response'}, 'retrieved_contexts_column': {'type': '_empty', 'default': 'retrieved_contexts'}, 'reference_column': {'type': '_empty', 'default': 'reference'}, 'focus': {'type': '_empty', 'default': 'relevant'}, 'user_input_column': {'type': '_empty', 'default': 'user_input'}} | \n",
- " ['ragas', 'llm', 'rag_performance'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.ResponseRelevancy | \n",
- " Response Relevancy | \n",
- " Assesses how pertinent the generated answer is to the given prompt.... | \n",
- " ['dataset'] | \n",
- " {'user_input_column': {'type': '_empty', 'default': 'user_input'}, 'retrieved_contexts_column': {'type': '_empty', 'default': None}, 'response_column': {'type': '_empty', 'default': 'response'}} | \n",
- " ['ragas', 'llm', 'rag_performance'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.ragas.SemanticSimilarity | \n",
- " Semantic Similarity | \n",
- " Calculates the semantic similarity between generated responses and ground truths... | \n",
- " ['dataset'] | \n",
- " {'response_column': {'type': '_empty', 'default': 'response'}, 'reference_column': {'type': '_empty', 'default': 'reference'}} | \n",
- " ['ragas', 'llm'] | \n",
- " ['text_qa', 'text_generation', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.AdjustedMutualInformation | \n",
- " Adjusted Mutual Information | \n",
- " Evaluates clustering model performance by measuring mutual information between true and predicted labels, adjusting... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance', 'clustering'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.AdjustedRandIndex | \n",
- " Adjusted Rand Index | \n",
- " Measures the similarity between two data clusters using the Adjusted Rand Index (ARI) metric in clustering machine... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance', 'clustering'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.CalibrationCurve | \n",
- " Calibration Curve | \n",
- " Evaluates the calibration of probability estimates by comparing predicted probabilities against observed... | \n",
- " ['model', 'dataset'] | \n",
- " {'n_bins': {'type': 'int', 'default': 10}} | \n",
- " ['sklearn', 'model_performance', 'classification'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ClassifierPerformance | \n",
- " Classifier Performance | \n",
- " Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy,... | \n",
- " ['dataset', 'model'] | \n",
- " {'average': {'type': 'str', 'default': 'macro'}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ClassifierThresholdOptimization | \n",
- " Classifier Threshold Optimization | \n",
- " Analyzes and visualizes different threshold optimization methods for binary classification models.... | \n",
- " ['dataset', 'model'] | \n",
- " {'methods': {'type': None, 'default': None}, 'target_recall': {'type': None, 'default': None}} | \n",
- " ['model_validation', 'threshold_optimization', 'classification_metrics'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ClusterCosineSimilarity | \n",
- " Cluster Cosine Similarity | \n",
- " Measures the intra-cluster similarity of a clustering model using cosine similarity.... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance', 'clustering'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ClusterPerformanceMetrics | \n",
- " Cluster Performance Metrics | \n",
- " Evaluates the performance of clustering machine learning models using multiple established metrics.... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance', 'clustering'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.CompletenessScore | \n",
- " Completeness Score | \n",
- " Evaluates a clustering model's capacity to categorize instances from a single class into the same cluster.... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance', 'clustering'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ConfusionMatrix | \n",
- " Confusion Matrix | \n",
- " Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix... | \n",
- " ['dataset', 'model'] | \n",
- " {'threshold': {'type': 'float', 'default': 0.5}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.FeatureImportance | \n",
- " Feature Importance | \n",
- " Compute feature importance scores for a given model and generate a summary table... | \n",
- " ['dataset', 'model'] | \n",
- " {'num_features': {'type': 'int', 'default': 3}} | \n",
- " ['model_explainability', 'sklearn'] | \n",
- " ['regression', 'time_series_forecasting'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.FowlkesMallowsScore | \n",
- " Fowlkes Mallows Score | \n",
- " Evaluates the similarity between predicted and actual cluster assignments in a model using the Fowlkes-Mallows... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.HomogeneityScore | \n",
- " Homogeneity Score | \n",
- " Assesses clustering homogeneity by comparing true and predicted labels, scoring from 0 (heterogeneous) to 1... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.HyperParametersTuning | \n",
- " Hyper Parameters Tuning | \n",
- " Performs exhaustive grid search over specified parameter ranges to find optimal model configurations... | \n",
- " ['model', 'dataset'] | \n",
- " {'param_grid': {'type': 'dict', 'default': None}, 'scoring': {'type': None, 'default': None}, 'thresholds': {'type': None, 'default': None}, 'fit_params': {'type': 'dict', 'default': None}} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['clustering', 'classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.KMeansClustersOptimization | \n",
- " K Means Clusters Optimization | \n",
- " Optimizes the number of clusters in K-means models using Elbow and Silhouette methods.... | \n",
- " ['model', 'dataset'] | \n",
- " {'n_clusters': {'type': None, 'default': None}} | \n",
- " ['sklearn', 'model_performance', 'kmeans'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.MinimumAccuracy | \n",
- " Minimum Accuracy | \n",
- " Checks if the model's prediction accuracy meets or surpasses a specified threshold.... | \n",
- " ['dataset', 'model'] | \n",
- " {'min_threshold': {'type': 'float', 'default': 0.7}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.MinimumF1Score | \n",
- " Minimum F1 Score | \n",
- " Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced... | \n",
- " ['dataset', 'model'] | \n",
- " {'min_threshold': {'type': 'float', 'default': 0.5}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.MinimumROCAUCScore | \n",
- " Minimum ROCAUC Score | \n",
- " Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.... | \n",
- " ['dataset', 'model'] | \n",
- " {'min_threshold': {'type': 'float', 'default': 0.5}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ModelParameters | \n",
- " Model Parameters | \n",
- " Extracts and displays model parameters in a structured format for transparency and reproducibility.... | \n",
- " ['model'] | \n",
- " {'model_params': {'type': '_empty', 'default': None}} | \n",
- " ['model_training', 'metadata'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ModelsPerformanceComparison | \n",
- " Models Performance Comparison | \n",
- " Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy,... | \n",
- " ['dataset', 'models'] | \n",
- " {} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'model_comparison'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.OverfitDiagnosis | \n",
- " Overfit Diagnosis | \n",
- " Assesses potential overfitting in a model's predictions, identifying regions where performance between training and... | \n",
- " ['model', 'datasets'] | \n",
- " {'metric': {'type': 'str', 'default': None}, 'cut_off_threshold': {'type': 'float', 'default': 0.04}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'linear_regression', 'model_diagnosis'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.PermutationFeatureImportance | \n",
- " Permutation Feature Importance | \n",
- " Assesses the significance of each feature in a model by evaluating the impact on model performance when feature... | \n",
- " ['model', 'dataset'] | \n",
- " {'fontsize': {'type': None, 'default': None}, 'figure_height': {'type': None, 'default': None}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'feature_importance', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.PopulationStabilityIndex | \n",
- " Population Stability Index | \n",
- " Assesses the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across... | \n",
- " ['datasets', 'model'] | \n",
- " {'num_bins': {'type': 'int', 'default': 10}, 'mode': {'type': 'str', 'default': 'fixed'}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.PrecisionRecallCurve | \n",
- " Precision Recall Curve | \n",
- " Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'binary_classification', 'model_performance', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ROCCurve | \n",
- " ROC Curve | \n",
- " Evaluates binary classification model performance by generating and plotting the Receiver Operating Characteristic... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.RegressionErrors | \n",
- " Regression Errors | \n",
- " Assesses the performance and error distribution of a regression model using various error metrics.... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['regression', 'classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.RegressionErrorsComparison | \n",
- " Regression Errors Comparison | \n",
- " Assesses multiple regression error metrics to compare model performance across different datasets, emphasizing... | \n",
- " ['datasets', 'models'] | \n",
- " {} | \n",
- " ['model_performance', 'sklearn'] | \n",
- " ['regression', 'time_series_forecasting'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.RegressionPerformance | \n",
- " Regression Performance | \n",
- " Evaluates the performance of a regression model using five different metrics: MAE, MSE, RMSE, MAPE, and MBD.... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.RegressionR2Square | \n",
- " Regression R2 Square | \n",
- " Assesses the overall goodness-of-fit of a regression model by evaluating R-squared (R2) and Adjusted R-squared (Adj... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.RegressionR2SquareComparison | \n",
- " Regression R2 Square Comparison | \n",
- " Compares R-Squared and Adjusted R-Squared values for different regression models across multiple datasets to assess... | \n",
- " ['datasets', 'models'] | \n",
- " {} | \n",
- " ['model_performance', 'sklearn'] | \n",
- " ['regression', 'time_series_forecasting'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.RobustnessDiagnosis | \n",
- " Robustness Diagnosis | \n",
- " Assesses the robustness of a machine learning model by evaluating performance decay under noisy conditions.... | \n",
- " ['datasets', 'model'] | \n",
- " {'metric': {'type': 'str', 'default': None}, 'scaling_factor_std_dev_list': {'type': None, 'default': [0.1, 0.2, 0.3, 0.4, 0.5]}, 'performance_decay_threshold': {'type': 'float', 'default': 0.05}} | \n",
- " ['sklearn', 'model_diagnosis', 'visualization'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.SHAPGlobalImportance | \n",
- " SHAP Global Importance | \n",
- " Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.... | \n",
- " ['model', 'dataset'] | \n",
- " {'kernel_explainer_samples': {'type': 'int', 'default': 10}, 'tree_or_linear_explainer_samples': {'type': 'int', 'default': 200}, 'class_of_interest': {'type': None, 'default': None}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'feature_importance', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.ScoreProbabilityAlignment | \n",
- " Score Probability Alignment | \n",
- " Analyzes the alignment between credit scores and predicted probabilities.... | \n",
- " ['model', 'dataset'] | \n",
- " {'score_column': {'type': 'str', 'default': 'score'}, 'n_bins': {'type': 'int', 'default': 10}} | \n",
- " ['visualization', 'credit_risk', 'calibration'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.SilhouettePlot | \n",
- " Silhouette Plot | \n",
- " Calculates and visualizes Silhouette Score, assessing the degree of data point suitability to its cluster in ML... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.TrainingTestDegradation | \n",
- " Training Test Degradation | \n",
- " Tests if model performance degradation between training and test datasets exceeds a predefined threshold.... | \n",
- " ['datasets', 'model'] | \n",
- " {'max_threshold': {'type': 'float', 'default': 0.1}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.VMeasure | \n",
- " V Measure | \n",
- " Evaluates homogeneity and completeness of a clustering model using the V Measure Score.... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['sklearn', 'model_performance'] | \n",
- " ['clustering'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.sklearn.WeakspotsDiagnosis | \n",
- " Weakspots Diagnosis | \n",
- " Identifies and visualizes weak spots in a machine learning model's performance across various sections of the... | \n",
- " ['datasets', 'model'] | \n",
- " {'features_columns': {'type': None, 'default': None}, 'metrics': {'type': None, 'default': None}, 'thresholds': {'type': None, 'default': None}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_diagnosis', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.AutoARIMA | \n",
- " Auto ARIMA | \n",
- " Evaluates ARIMA models for time-series forecasting, ranking them using Bayesian and Akaike Information Criteria.... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'forecasting', 'model_selection', 'statsmodels'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.CumulativePredictionProbabilities | \n",
- " Cumulative Prediction Probabilities | \n",
- " Visualizes cumulative probabilities of positive and negative classes for both training and testing in classification models.... | \n",
- " ['dataset', 'model'] | \n",
- " {'title': {'type': '_empty', 'default': 'Cumulative Probabilities'}} | \n",
- " ['visualization', 'credit_risk'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.DurbinWatsonTest | \n",
- " Durbin Watson Test | \n",
- " Assesses autocorrelation in time series data features using the Durbin-Watson statistic.... | \n",
- " ['dataset', 'model'] | \n",
- " {'threshold': {'type': '_empty', 'default': [1.5, 2.5]}} | \n",
- " ['time_series_data', 'forecasting', 'statistical_test', 'statsmodels'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.GINITable | \n",
- " GINI Table | \n",
- " Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets.... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['model_performance'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.KolmogorovSmirnov | \n",
- " Kolmogorov Smirnov | \n",
- " Assesses whether each feature in the dataset aligns with a normal distribution using the Kolmogorov-Smirnov test.... | \n",
- " ['model', 'dataset'] | \n",
- " {'dist': {'type': 'str', 'default': 'norm'}} | \n",
- " ['tabular_data', 'data_distribution', 'statistical_test', 'statsmodels'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.Lilliefors | \n",
- " Lilliefors | \n",
- " Assesses the normality of feature distributions in an ML model's training dataset using the Lilliefors test.... | \n",
- " ['dataset'] | \n",
- " {} | \n",
- " ['tabular_data', 'data_distribution', 'statistical_test', 'statsmodels'] | \n",
- " ['classification', 'regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.PredictionProbabilitiesHistogram | \n",
- " Prediction Probabilities Histogram | \n",
- " Assesses the predictive probability distribution for binary classification to evaluate model performance and... | \n",
- " ['dataset', 'model'] | \n",
- " {'title': {'type': '_empty', 'default': 'Histogram of Predictive Probabilities'}} | \n",
- " ['visualization', 'credit_risk'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.RegressionCoeffs | \n",
- " Regression Coeffs | \n",
- " Assesses the significance and uncertainty of predictor variables in a regression model through visualization of... | \n",
- " ['model'] | \n",
- " {} | \n",
- " ['tabular_data', 'visualization', 'model_training'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.RegressionFeatureSignificance | \n",
- " Regression Feature Significance | \n",
- " Assesses and visualizes the statistical significance of features in a regression model.... | \n",
- " ['model'] | \n",
- " {'fontsize': {'type': 'int', 'default': 10}, 'p_threshold': {'type': 'float', 'default': 0.05}} | \n",
- " ['statistical_test', 'model_interpretation', 'visualization', 'feature_importance'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.RegressionModelForecastPlot | \n",
- " Regression Model Forecast Plot | \n",
- " Generates plots to visually compare the forecasted outcomes of a regression model against actual observed values over... | \n",
- " ['model', 'dataset'] | \n",
- " {'start_date': {'type': None, 'default': None}, 'end_date': {'type': None, 'default': None}} | \n",
- " ['time_series_data', 'forecasting', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels | \n",
- " Regression Model Forecast Plot Levels | \n",
- " Assesses the alignment between forecasted and observed values in regression models through visual plots... | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['time_series_data', 'forecasting', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.RegressionModelSensitivityPlot | \n",
- " Regression Model Sensitivity Plot | \n",
- " Assesses the sensitivity of a regression model to changes in independent variables by applying shocks and... | \n",
- " ['dataset', 'model'] | \n",
- " {'shocks': {'type': None, 'default': [0.1]}, 'transformation': {'type': None, 'default': None}} | \n",
- " ['senstivity_analysis', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.RegressionModelSummary | \n",
- " Regression Model Summary | \n",
- " Evaluates regression model performance using metrics including R-Squared, Adjusted R-Squared, MSE, and RMSE.... | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['model_performance', 'regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.RegressionPermutationFeatureImportance | \n",
- " Regression Permutation Feature Importance | \n",
- " Assesses the significance of each feature in a model by evaluating the impact on model performance when feature... | \n",
- " ['dataset', 'model'] | \n",
- " {'fontsize': {'type': 'int', 'default': 12}, 'figure_height': {'type': 'int', 'default': 500}} | \n",
- " ['statsmodels', 'feature_importance', 'visualization'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.model_validation.statsmodels.ScorecardHistogram | \n",
- " Scorecard Histogram | \n",
- " The Scorecard Histogram test evaluates the distribution of credit scores between default and non-default instances,... | \n",
- " ['dataset'] | \n",
- " {'title': {'type': '_empty', 'default': 'Histogram of Scores'}, 'score_column': {'type': '_empty', 'default': 'score'}} | \n",
- " ['visualization', 'credit_risk', 'logistic_regression'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.CalibrationCurveDrift | \n",
- " Calibration Curve Drift | \n",
- " Evaluates changes in probability calibration between reference and monitoring datasets.... | \n",
- " ['datasets', 'model'] | \n",
- " {'n_bins': {'type': 'int', 'default': 10}, 'drift_pct_threshold': {'type': 'float', 'default': 20}} | \n",
- " ['sklearn', 'binary_classification', 'model_performance', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.ClassDiscriminationDrift | \n",
- " Class Discrimination Drift | \n",
- " Compares classification discrimination metrics between reference and monitoring datasets.... | \n",
- " ['datasets', 'model'] | \n",
- " {'drift_pct_threshold': {'type': '_empty', 'default': 20}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.ClassImbalanceDrift | \n",
- " Class Imbalance Drift | \n",
- " Evaluates drift in class distribution between reference and monitoring datasets.... | \n",
- " ['datasets'] | \n",
- " {'drift_pct_threshold': {'type': 'float', 'default': 5.0}, 'title': {'type': 'str', 'default': 'Class Distribution Drift'}} | \n",
- " ['tabular_data', 'binary_classification', 'multiclass_classification'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.ClassificationAccuracyDrift | \n",
- " Classification Accuracy Drift | \n",
- " Compares classification accuracy metrics between reference and monitoring datasets.... | \n",
- " ['datasets', 'model'] | \n",
- " {'drift_pct_threshold': {'type': '_empty', 'default': 20}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.ConfusionMatrixDrift | \n",
- " Confusion Matrix Drift | \n",
- " Compares confusion matrix metrics between reference and monitoring datasets.... | \n",
- " ['datasets', 'model'] | \n",
- " {'drift_pct_threshold': {'type': '_empty', 'default': 20}} | \n",
- " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.CumulativePredictionProbabilitiesDrift | \n",
- " Cumulative Prediction Probabilities Drift | \n",
- " Compares cumulative prediction probability distributions between reference and monitoring datasets.... | \n",
- " ['datasets', 'model'] | \n",
- " {} | \n",
- " ['visualization', 'credit_risk'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.FeatureDrift | \n",
- " Feature Drift | \n",
- " Evaluates changes in feature distribution over time to identify potential model drift.... | \n",
- " ['datasets'] | \n",
- " {'bins': {'type': '_empty', 'default': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]}, 'feature_columns': {'type': '_empty', 'default': None}, 'psi_threshold': {'type': '_empty', 'default': 0.2}} | \n",
- " ['visualization'] | \n",
- " ['monitoring'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.PredictionAcrossEachFeature | \n",
- " Prediction Across Each Feature | \n",
- " Assesses differences in model predictions across individual features between reference and monitoring datasets... | \n",
- " ['datasets', 'model'] | \n",
- " {} | \n",
- " ['visualization'] | \n",
- " ['monitoring'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.PredictionCorrelation | \n",
- " Prediction Correlation | \n",
- " Assesses correlation changes between model predictions from reference and monitoring datasets to detect potential... | \n",
- " ['datasets', 'model'] | \n",
- " {'drift_pct_threshold': {'type': '_empty', 'default': 20}} | \n",
- " ['visualization'] | \n",
- " ['monitoring'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.PredictionProbabilitiesHistogramDrift | \n",
- " Prediction Probabilities Histogram Drift | \n",
- " Compares prediction probability distributions between reference and monitoring datasets.... | \n",
- " ['datasets', 'model'] | \n",
- " {'title': {'type': '_empty', 'default': 'Prediction Probabilities Histogram Drift'}, 'drift_pct_threshold': {'type': 'float', 'default': 20.0}} | \n",
- " ['visualization', 'credit_risk'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.PredictionQuantilesAcrossFeatures | \n",
- " Prediction Quantiles Across Features | \n",
- " Assesses differences in model prediction distributions across individual features between reference... | \n",
- " ['datasets', 'model'] | \n",
- " {} | \n",
- " ['visualization'] | \n",
- " ['monitoring'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.ROCCurveDrift | \n",
- " ROC Curve Drift | \n",
- " Compares ROC curves between reference and monitoring datasets.... | \n",
- " ['datasets', 'model'] | \n",
- " {} | \n",
- " ['sklearn', 'binary_classification', 'model_performance', 'visualization'] | \n",
- " ['classification', 'text_classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.ScoreBandsDrift | \n",
- " Score Bands Drift | \n",
- " Analyzes drift in population distribution and default rates across score bands.... | \n",
- " ['datasets', 'model'] | \n",
- " {'score_column': {'type': 'str', 'default': 'score'}, 'score_bands': {'type': 'list', 'default': None}, 'drift_threshold': {'type': 'float', 'default': 20.0}} | \n",
- " ['visualization', 'credit_risk', 'scorecard'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.ScorecardHistogramDrift | \n",
- " Scorecard Histogram Drift | \n",
- " Compares score distributions between reference and monitoring datasets for each class.... | \n",
- " ['datasets'] | \n",
- " {'score_column': {'type': 'str', 'default': 'score'}, 'title': {'type': 'str', 'default': 'Scorecard Histogram Drift'}, 'drift_pct_threshold': {'type': 'float', 'default': 20.0}} | \n",
- " ['visualization', 'credit_risk', 'logistic_regression'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.ongoing_monitoring.TargetPredictionDistributionPlot | \n",
- " Target Prediction Distribution Plot | \n",
- " Assesses differences in prediction distributions between a reference dataset and a monitoring dataset to identify... | \n",
- " ['datasets', 'model'] | \n",
- " {'drift_pct_threshold': {'type': '_empty', 'default': 20}} | \n",
- " ['visualization'] | \n",
- " ['monitoring'] | \n",
- "
\n",
- " \n",
- " | validmind.prompt_validation.Bias | \n",
- " Bias | \n",
- " Assesses potential bias in a Large Language Model by analyzing the distribution and order of exemplars in the... | \n",
- " ['model'] | \n",
- " {'min_threshold': {'type': '_empty', 'default': 7}} | \n",
- " ['llm', 'few_shot'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.prompt_validation.Clarity | \n",
- " Clarity | \n",
- " Evaluates and scores the clarity of prompts in a Large Language Model based on specified guidelines.... | \n",
- " ['model'] | \n",
- " {'min_threshold': {'type': '_empty', 'default': 7}} | \n",
- " ['llm', 'zero_shot', 'few_shot'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.prompt_validation.Conciseness | \n",
- " Conciseness | \n",
- " Analyzes and grades the conciseness of prompts provided to a Large Language Model.... | \n",
- " ['model'] | \n",
- " {'min_threshold': {'type': '_empty', 'default': 7}} | \n",
- " ['llm', 'zero_shot', 'few_shot'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.prompt_validation.Delimitation | \n",
- " Delimitation | \n",
- " Evaluates the proper use of delimiters in prompts provided to Large Language Models.... | \n",
- " ['model'] | \n",
- " {'min_threshold': {'type': '_empty', 'default': 7}} | \n",
- " ['llm', 'zero_shot', 'few_shot'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.prompt_validation.NegativeInstruction | \n",
- " Negative Instruction | \n",
- " Evaluates and grades the use of affirmative, proactive language over negative instructions in LLM prompts.... | \n",
- " ['model'] | \n",
- " {'min_threshold': {'type': '_empty', 'default': 7}} | \n",
- " ['llm', 'zero_shot', 'few_shot'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.prompt_validation.Robustness | \n",
- " Robustness | \n",
- " Assesses the robustness of prompts provided to a Large Language Model under varying conditions and contexts. This test... | \n",
- " ['model', 'dataset'] | \n",
- " {'num_tests': {'type': '_empty', 'default': 10}} | \n",
- " ['llm', 'zero_shot', 'few_shot'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.prompt_validation.Specificity | \n",
- " Specificity | \n",
- " Evaluates and scores the specificity of prompts provided to a Large Language Model (LLM), based on clarity, detail,... | \n",
- " ['model'] | \n",
- " {'min_threshold': {'type': '_empty', 'default': 7}} | \n",
- " ['llm', 'zero_shot', 'few_shot'] | \n",
- " ['text_classification', 'text_summarization'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.classification.Accuracy | \n",
- " Accuracy | \n",
- " Calculates the accuracy of a model | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['classification'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.classification.F1 | \n",
- " F1 | \n",
- " Calculates the F1 score for a classification model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['classification'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.classification.Precision | \n",
- " Precision | \n",
- " Calculates the precision for a classification model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['classification'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.classification.ROC_AUC | \n",
- " ROC AUC | \n",
- " Calculates the ROC AUC for a classification model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['classification'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.classification.Recall | \n",
- " Recall | \n",
- " Calculates the recall for a classification model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['classification'] | \n",
- " ['classification'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.AdjustedRSquaredScore | \n",
- " Adjusted R Squared Score | \n",
- " Calculates the adjusted R-squared score for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.GiniCoefficient | \n",
- " Gini Coefficient | \n",
- " Calculates the Gini coefficient for a regression model. | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.HuberLoss | \n",
- " Huber Loss | \n",
- " Calculates the Huber loss for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.KolmogorovSmirnovStatistic | \n",
- " Kolmogorov Smirnov Statistic | \n",
- " Calculates the Kolmogorov-Smirnov statistic for a regression model. | \n",
- " ['dataset', 'model'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.MeanAbsoluteError | \n",
- " Mean Absolute Error | \n",
- " Calculates the mean absolute error for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.MeanAbsolutePercentageError | \n",
- " Mean Absolute Percentage Error | \n",
- " Calculates the mean absolute percentage error for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.MeanBiasDeviation | \n",
- " Mean Bias Deviation | \n",
- " Calculates the mean bias deviation for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.MeanSquaredError | \n",
- " Mean Squared Error | \n",
- " Calculates the mean squared error for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.QuantileLoss | \n",
- " Quantile Loss | \n",
- " Calculates the quantile loss for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {'quantile': {'type': '_empty', 'default': 0.5}} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.RSquaredScore | \n",
- " R Squared Score | \n",
- " Calculates the R-squared score for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
- "
\n",
- " \n",
- " | validmind.unit_metrics.regression.RootMeanSquaredError | \n",
- " Root Mean Squared Error | \n",
- " Calculates the root mean squared error for a regression model. | \n",
- " ['model', 'dataset'] | \n",
- " {} | \n",
- " ['regression'] | \n",
- " ['regression'] | \n",
+ " validmind.data_validation.ACFandPACFPlot | \n",
+ " AC Fand PACF Plot | \n",
+ " Analyzes time series data using Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots to... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'forecasting', 'statistical_test', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.ADF | \n",
+ " ADF | \n",
+ " Assesses the stationarity of a time series dataset using the Augmented Dickey-Fuller (ADF) test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'statsmodels', 'forecasting', 'statistical_test', 'stationarity'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.AutoAR | \n",
+ " Auto AR | \n",
+ " Automatically identifies the optimal Autoregressive (AR) order for a time series using BIC and AIC criteria.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'max_ar_order': {'type': 'int', 'default': 3}} | \n",
+ " ['time_series_data', 'statsmodels', 'forecasting', 'statistical_test'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.AutoMA | \n",
+ " Auto MA | \n",
+ " Automatically selects the optimal Moving Average (MA) order for each variable in a time series dataset based on... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'max_ma_order': {'type': 'int', 'default': 3}} | \n",
+ " ['time_series_data', 'statsmodels', 'forecasting', 'statistical_test'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.AutoStationarity | \n",
+ " Auto Stationarity | \n",
+ " Automates Augmented Dickey-Fuller test to assess stationarity across multiple time series in a DataFrame.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'max_order': {'type': 'int', 'default': 5}, 'threshold': {'type': 'float', 'default': 0.05}} | \n",
+ " ['time_series_data', 'statsmodels', 'forecasting', 'statistical_test'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.BivariateScatterPlots | \n",
+ " Bivariate Scatter Plots | \n",
+ " Generates bivariate scatterplots to visually inspect relationships between pairs of numerical predictor variables... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'numerical_data', 'visualization'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.BoxPierce | \n",
+ " Box Pierce | \n",
+ " Detects autocorrelation in time-series data through the Box-Pierce test to validate model performance.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'forecasting', 'statistical_test', 'statsmodels'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.ChiSquaredFeaturesTable | \n",
+ " Chi Squared Features Table | \n",
+ " Assesses the statistical association between categorical features and a target variable using the Chi-Squared test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'p_threshold': {'type': '_empty', 'default': 0.05}} | \n",
+ " ['tabular_data', 'categorical_data', 'statistical_test'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.ClassImbalance | \n",
+ " Class Imbalance | \n",
+ " Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'min_percent_threshold': {'type': 'int', 'default': 10}} | \n",
+ " ['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.DatasetDescription | \n",
+ " Dataset Description | \n",
+ " Provides comprehensive analysis and statistical summaries of each column in a machine learning model's dataset.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'time_series_data', 'text_data'] | \n",
+ " ['classification', 'regression', 'text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.DatasetSplit | \n",
+ " Dataset Split | \n",
+ " Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML... | \n",
+ " False | \n",
+ " True | \n",
+ " ['datasets'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'time_series_data', 'text_data'] | \n",
+ " ['classification', 'regression', 'text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.DescriptiveStatistics | \n",
+ " Descriptive Statistics | \n",
+ " Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'time_series_data', 'data_quality'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.DickeyFullerGLS | \n",
+ " Dickey Fuller GLS | \n",
+ " Assesses stationarity in time series data using the Dickey-Fuller GLS test to determine the order of integration.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'forecasting', 'unit_root_test'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.Duplicates | \n",
+ " Duplicates | \n",
+ " Tests dataset for duplicate entries, ensuring model reliability via data quality verification.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'min_threshold': {'type': '_empty', 'default': 1}} | \n",
+ " ['tabular_data', 'data_quality', 'text_data'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.EngleGrangerCoint | \n",
+ " Engle Granger Coint | \n",
+ " Assesses the degree of co-movement between pairs of time series data using the Engle-Granger cointegration test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'threshold': {'type': 'float', 'default': 0.05}} | \n",
+ " ['time_series_data', 'statistical_test', 'forecasting'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.FeatureTargetCorrelationPlot | \n",
+ " Feature Target Correlation Plot | \n",
+ " Visualizes the correlation between input features and the model's target output in a color-coded horizontal bar... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'fig_height': {'type': '_empty', 'default': 600}} | \n",
+ " ['tabular_data', 'visualization', 'correlation'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.HighCardinality | \n",
+ " High Cardinality | \n",
+ " Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}} | \n",
+ " ['tabular_data', 'data_quality', 'categorical_data'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.HighPearsonCorrelation | \n",
+ " High Pearson Correlation | \n",
+ " Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}} | \n",
+ " ['tabular_data', 'data_quality', 'correlation'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.IQROutliersBarPlot | \n",
+ " IQR Outliers Bar Plot | \n",
+ " Visualizes outlier distribution across percentiles in numerical data using the Interquartile Range (IQR) method.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'threshold': {'type': 'float', 'default': 1.5}, 'fig_width': {'type': 'int', 'default': 800}} | \n",
+ " ['tabular_data', 'visualization', 'numerical_data'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.IQROutliersTable | \n",
+ " IQR Outliers Table | \n",
+ " Determines and summarizes outliers in numerical features using the Interquartile Range method.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'threshold': {'type': 'float', 'default': 1.5}} | \n",
+ " ['tabular_data', 'numerical_data'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.IsolationForestOutliers | \n",
+ " Isolation Forest Outliers | \n",
+ " Detects outliers in a dataset using the Isolation Forest algorithm and visualizes results through scatter plots.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'random_state': {'type': 'int', 'default': 0}, 'contamination': {'type': 'float', 'default': 0.1}, 'feature_columns': {'type': 'list', 'default': None}} | \n",
+ " ['tabular_data', 'anomaly_detection'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.JarqueBera | \n",
+ " Jarque Bera | \n",
+ " Assesses normality of dataset features in an ML model using the Jarque-Bera test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'data_distribution', 'statistical_test', 'statsmodels'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.KPSS | \n",
+ " KPSS | \n",
+ " Assesses the stationarity of time-series data in a machine learning model using the KPSS unit root test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'stationarity', 'unit_root_test', 'statsmodels'] | \n",
+ " ['data_validation'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.LJungBox | \n",
+ " L Jung Box | \n",
+ " Assesses autocorrelations in dataset features by performing a Ljung-Box test on each feature.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'forecasting', 'statistical_test', 'statsmodels'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.LaggedCorrelationHeatmap | \n",
+ " Lagged Correlation Heatmap | \n",
+ " Assesses and visualizes correlation between target variable and lagged independent variables in a time-series... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'num_lags': {'type': 'int', 'default': 10}} | \n",
+ " ['time_series_data', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.MissingValues | \n",
+ " Missing Values | \n",
+ " Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'min_threshold': {'type': 'int', 'default': 1}} | \n",
+ " ['tabular_data', 'data_quality'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.MissingValuesBarPlot | \n",
+ " Missing Values Bar Plot | \n",
+ " Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}} | \n",
+ " ['tabular_data', 'data_quality', 'visualization'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.MutualInformation | \n",
+ " Mutual Information | \n",
+ " Calculates mutual information scores between features and target variable to evaluate feature relevance.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'min_threshold': {'type': 'float', 'default': 0.01}, 'task': {'type': 'str', 'default': 'classification'}} | \n",
+ " ['feature_selection', 'data_analysis'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.PearsonCorrelationMatrix | \n",
+ " Pearson Correlation Matrix | \n",
+ " Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'numerical_data', 'correlation'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.PhillipsPerronArch | \n",
+ " Phillips Perron Arch | \n",
+ " Assesses the stationarity of time series data in each feature of the ML model using the Phillips-Perron test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'forecasting', 'statistical_test', 'unit_root_test'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.ProtectedClassesDescription | \n",
+ " Protected Classes Description | \n",
+ " Visualizes the distribution of protected classes in the dataset relative to the target variable... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'protected_classes': {'type': '_empty', 'default': None}} | \n",
+ " ['bias_and_fairness', 'descriptive_statistics'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.RollingStatsPlot | \n",
+ " Rolling Stats Plot | \n",
+ " Evaluates the stationarity of time series data by plotting its rolling mean and standard deviation over a specified... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'window_size': {'type': 'int', 'default': 12}} | \n",
+ " ['time_series_data', 'visualization', 'stationarity'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.RunsTest | \n",
+ " Runs Test | \n",
+ " Executes Runs Test on ML model to detect non-random patterns in output data sequence.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'statistical_test', 'statsmodels'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.ScatterPlot | \n",
+ " Scatter Plot | \n",
+ " Assesses visual relationships, patterns, and outliers among features in a dataset through scatter plot matrices.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'visualization'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.ScoreBandDefaultRates | \n",
+ " Score Band Default Rates | \n",
+ " Analyzes default rates and population distribution across credit score bands.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'score_column': {'type': 'str', 'default': 'score'}, 'score_bands': {'type': 'list', 'default': None}} | \n",
+ " ['visualization', 'credit_risk', 'scorecard'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.SeasonalDecompose | \n",
+ " Seasonal Decompose | \n",
+ " Assesses patterns and seasonality in a time series dataset by decomposing its features into foundational components.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'seasonal_model': {'type': 'str', 'default': 'additive'}} | \n",
+ " ['time_series_data', 'seasonality', 'statsmodels'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.ShapiroWilk | \n",
+ " Shapiro Wilk | \n",
+ " Evaluates feature-wise normality of training data using the Shapiro-Wilk test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'data_distribution', 'statistical_test'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.Skewness | \n",
+ " Skewness | \n",
+ " Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'max_threshold': {'type': '_empty', 'default': 1}} | \n",
+ " ['data_quality', 'tabular_data'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.SpreadPlot | \n",
+ " Spread Plot | \n",
+ " Assesses potential correlations between pairs of time series variables through visualization to enhance... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TabularCategoricalBarPlots | \n",
+ " Tabular Categorical Bar Plots | \n",
+ " Generates and visualizes bar plots for each category in categorical features to evaluate the dataset's composition.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'visualization'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TabularDateTimeHistograms | \n",
+ " Tabular Date Time Histograms | \n",
+ " Generates histograms to provide graphical insight into the distribution of time intervals in a model's datetime... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'visualization'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TabularDescriptionTables | \n",
+ " Tabular Description Tables | \n",
+ " Summarizes key descriptive statistics for numerical, categorical, and datetime variables in a dataset.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TabularNumericalHistograms | \n",
+ " Tabular Numerical Histograms | \n",
+ " Generates histograms for each numerical feature in a dataset to provide visual insights into data distribution and... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'visualization'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TargetRateBarPlots | \n",
+ " Target Rate Bar Plots | \n",
+ " Generates bar plots visualizing the default rates of categorical features for a classification machine learning... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'visualization', 'categorical_data'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TimeSeriesDescription | \n",
+ " Time Series Description | \n",
+ " Generates a detailed analysis for the provided time series dataset, summarizing key statistics to identify trends,... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'analysis'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TimeSeriesDescriptiveStatistics | \n",
+ " Time Series Descriptive Statistics | \n",
+ " Evaluates the descriptive statistics of a time series dataset to identify trends, patterns, and data quality issues.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'analysis'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TimeSeriesFrequency | \n",
+ " Time Series Frequency | \n",
+ " Evaluates consistency of time series data frequency and generates a frequency plot.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TimeSeriesHistogram | \n",
+ " Time Series Histogram | \n",
+ " Visualizes distribution of time-series data using histograms and Kernel Density Estimation (KDE) lines.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'nbins': {'type': '_empty', 'default': 30}} | \n",
+ " ['data_validation', 'visualization', 'time_series_data'] | \n",
+ " ['regression', 'time_series_forecasting'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TimeSeriesLinePlot | \n",
+ " Time Series Line Plot | \n",
+ " Generates and analyses time-series data through line plots revealing trends, patterns, anomalies over time.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TimeSeriesMissingValues | \n",
+ " Time Series Missing Values | \n",
+ " Validates time-series data quality by confirming the count of missing values is below a certain threshold.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'min_threshold': {'type': 'int', 'default': 1}} | \n",
+ " ['time_series_data'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TimeSeriesOutliers | \n",
+ " Time Series Outliers | \n",
+ " Identifies and visualizes outliers in time-series data using the z-score method.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'zscore_threshold': {'type': 'int', 'default': 3}} | \n",
+ " ['time_series_data'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.TooManyZeroValues | \n",
+ " Too Many Zero Values | \n",
+ " Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'max_percent_threshold': {'type': 'float', 'default': 0.03}} | \n",
+ " ['tabular_data'] | \n",
+ " ['regression', 'classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.UniqueRows | \n",
+ " Unique Rows | \n",
+ " Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'min_percent_threshold': {'type': 'float', 'default': 1}} | \n",
+ " ['tabular_data'] | \n",
+ " ['regression', 'classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.WOEBinPlots | \n",
+ " WOE Bin Plots | \n",
+ " Generates visualizations of Weight of Evidence (WoE) and Information Value (IV) for understanding predictive power... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'breaks_adj': {'type': 'list', 'default': None}, 'fig_height': {'type': 'int', 'default': 600}, 'fig_width': {'type': 'int', 'default': 500}} | \n",
+ " ['tabular_data', 'visualization', 'categorical_data'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.WOEBinTable | \n",
+ " WOE Bin Table | \n",
+ " Assesses the Weight of Evidence (WoE) and Information Value (IV) of each feature to evaluate its predictive power... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'breaks_adj': {'type': 'list', 'default': None}} | \n",
+ " ['tabular_data', 'categorical_data'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.ZivotAndrewsArch | \n",
+ " Zivot Andrews Arch | \n",
+ " Evaluates the order of integration and stationarity of time series data using the Zivot-Andrews unit root test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'stationarity', 'unit_root_test'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.CommonWords | \n",
+ " Common Words | \n",
+ " Assesses the most frequent non-stopwords in a text column for identifying prevalent language patterns.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'visualization', 'frequency_analysis'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.Hashtags | \n",
+ " Hashtags | \n",
+ " Assesses hashtag frequency in a text column, highlighting usage trends and potential dataset bias or spam.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'top_hashtags': {'type': 'int', 'default': 25}} | \n",
+ " ['nlp', 'text_data', 'visualization', 'frequency_analysis'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.LanguageDetection | \n",
+ " Language Detection | \n",
+ " Assesses the diversity of languages in a textual dataset by detecting and visualizing the distribution of languages.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.Mentions | \n",
+ " Mentions | \n",
+ " Calculates and visualizes frequencies of '@' prefixed mentions in a text-based dataset for NLP model analysis.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'top_mentions': {'type': 'int', 'default': 25}} | \n",
+ " ['nlp', 'text_data', 'visualization', 'frequency_analysis'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.PolarityAndSubjectivity | \n",
+ " Polarity And Subjectivity | \n",
+ " Analyzes the polarity and subjectivity of text data within a given dataset to visualize the sentiment distribution.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'threshold_subjectivity': {'type': '_empty', 'default': 0.5}, 'threshold_polarity': {'type': '_empty', 'default': 0}} | \n",
+ " ['nlp', 'text_data', 'data_validation'] | \n",
+ " ['nlp'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.Punctuations | \n",
+ " Punctuations | \n",
+ " Analyzes and visualizes the frequency distribution of punctuation usage in a given text dataset.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'count_mode': {'type': '_empty', 'default': 'token'}} | \n",
+ " ['nlp', 'text_data', 'visualization', 'frequency_analysis'] | \n",
+ " ['text_classification', 'text_summarization', 'nlp'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.Sentiment | \n",
+ " Sentiment | \n",
+ " Analyzes the sentiment of text data within a dataset using the VADER sentiment analysis tool.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'data_validation'] | \n",
+ " ['nlp'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.StopWords | \n",
+ " Stop Words | \n",
+ " Evaluates and visualizes the frequency of English stop words in a text dataset against a defined threshold.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'min_percent_threshold': {'type': 'float', 'default': 0.5}, 'num_words': {'type': 'int', 'default': 25}} | \n",
+ " ['nlp', 'text_data', 'frequency_analysis', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.TextDescription | \n",
+ " Text Description | \n",
+ " Conducts comprehensive textual analysis on a dataset using NLTK to evaluate various parameters and generate... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'unwanted_tokens': {'type': 'set', 'default': {'s', 'mrs', 'us', \"''\", ' ', 'ms', 'dr', 'dollar', '``', 'mr', \"'s\", \"s'\"}}, 'lang': {'type': 'str', 'default': 'english'}} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.data_validation.nlp.Toxicity | \n",
+ " Toxicity | \n",
+ " Assesses the toxicity of text data within a dataset to visualize the distribution of toxicity scores.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'data_validation'] | \n",
+ " ['nlp'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.BertScore | \n",
+ " Bert Score | \n",
+ " Assesses the quality of machine-generated text using BERTScore metrics and visualizes results through histograms... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'evaluation_model': {'type': '_empty', 'default': 'distilbert-base-uncased'}} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.BleuScore | \n",
+ " Bleu Score | \n",
+ " Evaluates the quality of machine-generated text using BLEU metrics and visualizes the results through histograms... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ClusterSizeDistribution | \n",
+ " Cluster Size Distribution | \n",
+ " Assesses the performance of clustering models by comparing the distribution of cluster sizes in model predictions... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ContextualRecall | \n",
+ " Contextual Recall | \n",
+ " Evaluates a Natural Language Generation model's ability to generate contextually relevant and factually correct... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.FeaturesAUC | \n",
+ " Features AUC | \n",
+ " Evaluates the discriminatory power of each individual feature within a binary classification model by calculating... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'fontsize': {'type': 'int', 'default': 12}, 'figure_height': {'type': 'int', 'default': 500}} | \n",
+ " ['feature_importance', 'AUC', 'visualization'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.MeteorScore | \n",
+ " Meteor Score | \n",
+ " Assesses the quality of machine-generated translations by comparing them to human-produced references using the... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ModelMetadata | \n",
+ " Model Metadata | \n",
+ " Compare metadata of different models and generate a summary table with the results.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {} | \n",
+ " ['model_training', 'metadata'] | \n",
+ " ['regression', 'time_series_forecasting'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ModelPredictionResiduals | \n",
+ " Model Prediction Residuals | \n",
+ " Assesses normality and behavior of residuals in regression models through visualization and statistical tests.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'nbins': {'type': 'int', 'default': 100}, 'p_value_threshold': {'type': 'float', 'default': 0.05}, 'start_date': {'type': None, 'default': None}, 'end_date': {'type': None, 'default': None}} | \n",
+ " ['regression'] | \n",
+ " ['residual_analysis', 'visualization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.RegardScore | \n",
+ " Regard Score | \n",
+ " Assesses the sentiment and potential biases in text generated by NLP models by computing and visualizing regard... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.RegressionResidualsPlot | \n",
+ " Regression Residuals Plot | \n",
+ " Evaluates regression model performance using residual distribution and actual vs. predicted plots.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {'bin_size': {'type': 'float', 'default': 0.1}} | \n",
+ " ['model_performance', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.RougeScore | \n",
+ " Rouge Score | \n",
+ " Assesses the quality of machine-generated text using ROUGE metrics and visualizes the results to provide... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'metric': {'type': 'str', 'default': 'rouge-1'}} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.TimeSeriesPredictionWithCI | \n",
+ " Time Series Prediction With CI | \n",
+ " Assesses predictive accuracy and uncertainty in time series models, highlighting breaches beyond confidence... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'confidence': {'type': 'float', 'default': 0.95}} | \n",
+ " ['model_predictions', 'visualization'] | \n",
+ " ['regression', 'time_series_forecasting'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.TimeSeriesPredictionsPlot | \n",
+ " Time Series Predictions Plot | \n",
+ " Plot actual vs predicted values for time series data and generate a visual comparison for the model.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['model_predictions', 'visualization'] | \n",
+ " ['regression', 'time_series_forecasting'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.TimeSeriesR2SquareBySegments | \n",
+ " Time Series R2 Square By Segments | \n",
+ " Evaluates the R-Squared values of regression models over specified time segments in time series data to assess... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'segments': {'type': None, 'default': None}} | \n",
+ " ['model_performance', 'sklearn'] | \n",
+ " ['regression', 'time_series_forecasting'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.TokenDisparity | \n",
+ " Token Disparity | \n",
+ " Evaluates the token disparity between reference and generated texts, visualizing the results through histograms and... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ToxicityScore | \n",
+ " Toxicity Score | \n",
+ " Assesses the toxicity levels of texts generated by NLP models to identify and mitigate harmful or offensive content.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['nlp', 'text_data', 'visualization'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.ClusterDistribution | \n",
+ " Cluster Distribution | \n",
+ " Assesses the distribution of text embeddings across clusters produced by a model using KMeans clustering.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {'num_clusters': {'type': 'int', 'default': 5}} | \n",
+ " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
+ " ['feature_extraction'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.CosineSimilarityComparison | \n",
+ " Cosine Similarity Comparison | \n",
+ " Assesses the similarity between embeddings generated by different models using Cosine Similarity, providing both... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'models'] | \n",
+ " {} | \n",
+ " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.CosineSimilarityDistribution | \n",
+ " Cosine Similarity Distribution | \n",
+ " Assesses the similarity between predicted text embeddings from a model using a Cosine Similarity distribution... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
+ " ['feature_extraction'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.CosineSimilarityHeatmap | \n",
+ " Cosine Similarity Heatmap | \n",
+ " Generates an interactive heatmap to visualize the cosine similarities among embeddings derived from a given model.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'title': {'type': '_empty', 'default': 'Cosine Similarity Matrix'}, 'color': {'type': '_empty', 'default': 'Cosine Similarity'}, 'xaxis_title': {'type': '_empty', 'default': 'Index'}, 'yaxis_title': {'type': '_empty', 'default': 'Index'}, 'color_scale': {'type': '_empty', 'default': 'Blues'}} | \n",
+ " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.DescriptiveAnalytics | \n",
+ " Descriptive Analytics | \n",
+ " Evaluates statistical properties of text embeddings in an ML model via mean, median, and standard deviation... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
+ " ['feature_extraction'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.EmbeddingsVisualization2D | \n",
+ " Embeddings Visualization2 D | \n",
+ " Visualizes 2D representation of text embeddings generated by a model using t-SNE technique.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'cluster_column': {'type': None, 'default': None}, 'perplexity': {'type': 'int', 'default': 30}} | \n",
+ " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
+ " ['feature_extraction'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.EuclideanDistanceComparison | \n",
+ " Euclidean Distance Comparison | \n",
+ " Assesses and visualizes the dissimilarity between model embeddings using Euclidean distance, providing insights... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'models'] | \n",
+ " {} | \n",
+ " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.EuclideanDistanceHeatmap | \n",
+ " Euclidean Distance Heatmap | \n",
+ " Generates an interactive heatmap to visualize the Euclidean distances among embeddings derived from a given model.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'title': {'type': '_empty', 'default': 'Euclidean Distance Matrix'}, 'color': {'type': '_empty', 'default': 'Euclidean Distance'}, 'xaxis_title': {'type': '_empty', 'default': 'Index'}, 'yaxis_title': {'type': '_empty', 'default': 'Index'}, 'color_scale': {'type': '_empty', 'default': 'Blues'}} | \n",
+ " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.PCAComponentsPairwisePlots | \n",
+ " PCA Components Pairwise Plots | \n",
+ " Generates scatter plots for pairwise combinations of principal component analysis (PCA) components of model... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'n_components': {'type': 'int', 'default': 3}} | \n",
+ " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.StabilityAnalysisKeyword | \n",
+ " Stability Analysis Keyword | \n",
+ " Evaluates robustness of embedding models to keyword swaps in the test dataset.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'keyword_dict': {'type': None, 'default': None}, 'mean_similarity_threshold': {'type': 'float', 'default': 0.7}} | \n",
+ " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
+ " ['feature_extraction'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.StabilityAnalysisRandomNoise | \n",
+ " Stability Analysis Random Noise | \n",
+ " Assesses the robustness of text embeddings models to random noise introduced via text perturbations.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'probability': {'type': 'float', 'default': 0.02}, 'mean_similarity_threshold': {'type': 'float', 'default': 0.7}} | \n",
+ " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
+ " ['feature_extraction'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.StabilityAnalysisSynonyms | \n",
+ " Stability Analysis Synonyms | \n",
+ " Evaluates the stability of text embeddings models when words in test data are replaced by their synonyms randomly.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'probability': {'type': 'float', 'default': 0.02}, 'mean_similarity_threshold': {'type': 'float', 'default': 0.7}} | \n",
+ " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
+ " ['feature_extraction'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.StabilityAnalysisTranslation | \n",
+ " Stability Analysis Translation | \n",
+ " Evaluates robustness of text embeddings models to noise introduced by translating the original text to another... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'source_lang': {'type': 'str', 'default': 'en'}, 'target_lang': {'type': 'str', 'default': 'fr'}, 'mean_similarity_threshold': {'type': 'float', 'default': 0.7}} | \n",
+ " ['llm', 'text_data', 'embeddings', 'visualization'] | \n",
+ " ['feature_extraction'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.embeddings.TSNEComponentsPairwisePlots | \n",
+ " TSNE Components Pairwise Plots | \n",
+ " Creates scatter plots for pairwise combinations of t-SNE components to visualize embeddings and highlight potential... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'n_components': {'type': 'int', 'default': 2}, 'perplexity': {'type': 'int', 'default': 30}, 'title': {'type': 'str', 'default': 't-SNE'}} | \n",
+ " ['visualization', 'dimensionality_reduction', 'embeddings'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.AnswerCorrectness | \n",
+ " Answer Correctness | \n",
+ " Evaluates the correctness of answers in a dataset with respect to the provided ground... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'response_column': {'type': 'str', 'default': 'response'}, 'reference_column': {'type': 'str', 'default': 'reference'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.AspectCritic | \n",
+ " Aspect Critic | \n",
+ " Evaluates generations against the following aspects: harmfulness, maliciousness,... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'response_column': {'type': 'str', 'default': 'response'}, 'retrieved_contexts_column': {'type': None, 'default': None}, 'aspects': {'type': None, 'default': ['coherence', 'conciseness', 'correctness', 'harmfulness', 'maliciousness']}, 'additional_aspects': {'type': None, 'default': None}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm', 'qualitative'] | \n",
+ " ['text_summarization', 'text_generation', 'text_qa'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.ContextEntityRecall | \n",
+ " Context Entity Recall | \n",
+ " Evaluates the context entity recall for dataset entries and visualizes the results.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'reference_column': {'type': 'str', 'default': 'reference'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm', 'retrieval_performance'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.ContextPrecision | \n",
+ " Context Precision | \n",
+ " Context Precision is a metric that evaluates whether all of the ground-truth... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'reference_column': {'type': 'str', 'default': 'reference'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm', 'retrieval_performance'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.ContextPrecisionWithoutReference | \n",
+ " Context Precision Without Reference | \n",
+ " Context Precision Without Reference is a metric used to evaluate the relevance of... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'response_column': {'type': 'str', 'default': 'response'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm', 'retrieval_performance'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.ContextRecall | \n",
+ " Context Recall | \n",
+ " Context recall measures the extent to which the retrieved context aligns with the... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'reference_column': {'type': 'str', 'default': 'reference'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm', 'retrieval_performance'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.Faithfulness | \n",
+ " Faithfulness | \n",
+ " Evaluates the faithfulness of the generated answers with respect to retrieved contexts.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'response_column': {'type': 'str', 'default': 'response'}, 'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm', 'rag_performance'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.NoiseSensitivity | \n",
+ " Noise Sensitivity | \n",
+ " Assesses the sensitivity of a Large Language Model (LLM) to noise in retrieved context by measuring how often it... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'response_column': {'type': 'str', 'default': 'response'}, 'retrieved_contexts_column': {'type': 'str', 'default': 'retrieved_contexts'}, 'reference_column': {'type': 'str', 'default': 'reference'}, 'focus': {'type': 'str', 'default': 'relevant'}, 'user_input_column': {'type': 'str', 'default': 'user_input'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm', 'rag_performance'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.ResponseRelevancy | \n",
+ " Response Relevancy | \n",
+ " Assesses how pertinent the generated answer is to the given prompt.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'user_input_column': {'type': 'str', 'default': 'user_input'}, 'retrieved_contexts_column': {'type': 'str', 'default': None}, 'response_column': {'type': 'str', 'default': 'response'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm', 'rag_performance'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.ragas.SemanticSimilarity | \n",
+ " Semantic Similarity | \n",
+ " Calculates the semantic similarity between generated responses and ground truths... | \n",
+ " True | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {'response_column': {'type': 'str', 'default': 'response'}, 'reference_column': {'type': 'str', 'default': 'reference'}, 'judge_llm': {'type': '_empty', 'default': None}, 'judge_embeddings': {'type': '_empty', 'default': None}} | \n",
+ " ['ragas', 'llm'] | \n",
+ " ['text_qa', 'text_generation', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.AdjustedMutualInformation | \n",
+ " Adjusted Mutual Information | \n",
+ " Evaluates clustering model performance by measuring mutual information between true and predicted labels, adjusting... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance', 'clustering'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.AdjustedRandIndex | \n",
+ " Adjusted Rand Index | \n",
+ " Measures the similarity between two data clusters using the Adjusted Rand Index (ARI) metric in clustering machine... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance', 'clustering'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.CalibrationCurve | \n",
+ " Calibration Curve | \n",
+ " Evaluates the calibration of probability estimates by comparing predicted probabilities against observed... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {'n_bins': {'type': 'int', 'default': 10}} | \n",
+ " ['sklearn', 'model_performance', 'classification'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ClassifierPerformance | \n",
+ " Classifier Performance | \n",
+ " Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy,... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'average': {'type': 'str', 'default': 'macro'}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ClassifierThresholdOptimization | \n",
+ " Classifier Threshold Optimization | \n",
+ " Analyzes and visualizes different threshold optimization methods for binary classification models.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'methods': {'type': None, 'default': None}, 'target_recall': {'type': None, 'default': None}} | \n",
+ " ['model_validation', 'threshold_optimization', 'classification_metrics'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ClusterCosineSimilarity | \n",
+ " Cluster Cosine Similarity | \n",
+ " Measures the intra-cluster similarity of a clustering model using cosine similarity.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance', 'clustering'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ClusterPerformanceMetrics | \n",
+ " Cluster Performance Metrics | \n",
+ " Evaluates the performance of clustering machine learning models using multiple established metrics.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance', 'clustering'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.CompletenessScore | \n",
+ " Completeness Score | \n",
+ " Evaluates a clustering model's capacity to categorize instances from a single class into the same cluster.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance', 'clustering'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ConfusionMatrix | \n",
+ " Confusion Matrix | \n",
+ " Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'threshold': {'type': 'float', 'default': 0.5}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.FeatureImportance | \n",
+ " Feature Importance | \n",
+ " Compute feature importance scores for a given model and generate a summary table... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'num_features': {'type': 'int', 'default': 3}} | \n",
+ " ['model_explainability', 'sklearn'] | \n",
+ " ['regression', 'time_series_forecasting'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.FowlkesMallowsScore | \n",
+ " Fowlkes Mallows Score | \n",
+ " Evaluates the similarity between predicted and actual cluster assignments in a model using the Fowlkes-Mallows... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.HomogeneityScore | \n",
+ " Homogeneity Score | \n",
+ " Assesses clustering homogeneity by comparing true and predicted labels, scoring from 0 (heterogeneous) to 1... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.HyperParametersTuning | \n",
+ " Hyper Parameters Tuning | \n",
+ " Performs exhaustive grid search over specified parameter ranges to find optimal model configurations... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {'param_grid': {'type': 'dict', 'default': None}, 'scoring': {'type': None, 'default': None}, 'thresholds': {'type': None, 'default': None}, 'fit_params': {'type': 'dict', 'default': None}} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['clustering', 'classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.KMeansClustersOptimization | \n",
+ " K Means Clusters Optimization | \n",
+ " Optimizes the number of clusters in K-means models using Elbow and Silhouette methods.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {'n_clusters': {'type': None, 'default': None}} | \n",
+ " ['sklearn', 'model_performance', 'kmeans'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.MinimumAccuracy | \n",
+ " Minimum Accuracy | \n",
+ " Checks if the model's prediction accuracy meets or surpasses a specified threshold.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'min_threshold': {'type': 'float', 'default': 0.7}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.MinimumF1Score | \n",
+ " Minimum F1 Score | \n",
+ " Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'min_threshold': {'type': 'float', 'default': 0.5}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.MinimumROCAUCScore | \n",
+ " Minimum ROCAUC Score | \n",
+ " Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'min_threshold': {'type': 'float', 'default': 0.5}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ModelParameters | \n",
+ " Model Parameters | \n",
+ " Extracts and displays model parameters in a structured format for transparency and reproducibility.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {'model_params': {'type': None, 'default': None}} | \n",
+ " ['model_training', 'metadata'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ModelsPerformanceComparison | \n",
+ " Models Performance Comparison | \n",
+ " Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy,... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'models'] | \n",
+ " {} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'model_comparison'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.OverfitDiagnosis | \n",
+ " Overfit Diagnosis | \n",
+ " Assesses potential overfitting in a model's predictions, identifying regions where performance between training and... | \n",
+ " True | \n",
+ " True | \n",
+ " ['model', 'datasets'] | \n",
+ " {'metric': {'type': 'str', 'default': None}, 'cut_off_threshold': {'type': 'float', 'default': 0.04}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'linear_regression', 'model_diagnosis'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.PermutationFeatureImportance | \n",
+ " Permutation Feature Importance | \n",
+ " Assesses the significance of each feature in a model by evaluating the impact on model performance when feature... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {'fontsize': {'type': None, 'default': None}, 'figure_height': {'type': None, 'default': None}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'feature_importance', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.PopulationStabilityIndex | \n",
+ " Population Stability Index | \n",
+ " Assesses the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'num_bins': {'type': 'int', 'default': 10}, 'mode': {'type': 'str', 'default': 'fixed'}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.PrecisionRecallCurve | \n",
+ " Precision Recall Curve | \n",
+ " Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'binary_classification', 'model_performance', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ROCCurve | \n",
+ " ROC Curve | \n",
+ " Evaluates binary classification model performance by generating and plotting the Receiver Operating Characteristic... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.RegressionErrors | \n",
+ " Regression Errors | \n",
+ " Assesses the performance and error distribution of a regression model using various error metrics.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['regression', 'classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.RegressionErrorsComparison | \n",
+ " Regression Errors Comparison | \n",
+ " Assesses multiple regression error metrics to compare model performance across different datasets, emphasizing... | \n",
+ " False | \n",
+ " True | \n",
+ " ['datasets', 'models'] | \n",
+ " {} | \n",
+ " ['model_performance', 'sklearn'] | \n",
+ " ['regression', 'time_series_forecasting'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.RegressionPerformance | \n",
+ " Regression Performance | \n",
+ " Evaluates the performance of a regression model using five different metrics: MAE, MSE, RMSE, MAPE, and MBD.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.RegressionR2Square | \n",
+ " Regression R2 Square | \n",
+ " Assesses the overall goodness-of-fit of a regression model by evaluating R-squared (R2) and Adjusted R-squared (Adj... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.RegressionR2SquareComparison | \n",
+ " Regression R2 Square Comparison | \n",
+ " Compares R-Squared and Adjusted R-Squared values for different regression models across multiple datasets to assess... | \n",
+ " False | \n",
+ " True | \n",
+ " ['datasets', 'models'] | \n",
+ " {} | \n",
+ " ['model_performance', 'sklearn'] | \n",
+ " ['regression', 'time_series_forecasting'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.RobustnessDiagnosis | \n",
+ " Robustness Diagnosis | \n",
+ " Assesses the robustness of a machine learning model by evaluating performance decay under noisy conditions.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'metric': {'type': 'str', 'default': None}, 'scaling_factor_std_dev_list': {'type': None, 'default': [0.1, 0.2, 0.3, 0.4, 0.5]}, 'performance_decay_threshold': {'type': 'float', 'default': 0.05}} | \n",
+ " ['sklearn', 'model_diagnosis', 'visualization'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.SHAPGlobalImportance | \n",
+ " SHAP Global Importance | \n",
+ " Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {'kernel_explainer_samples': {'type': 'int', 'default': 10}, 'tree_or_linear_explainer_samples': {'type': 'int', 'default': 200}, 'class_of_interest': {'type': None, 'default': None}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'feature_importance', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.ScoreProbabilityAlignment | \n",
+ " Score Probability Alignment | \n",
+ " Analyzes the alignment between credit scores and predicted probabilities.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {'score_column': {'type': 'str', 'default': 'score'}, 'n_bins': {'type': 'int', 'default': 10}} | \n",
+ " ['visualization', 'credit_risk', 'calibration'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.SilhouettePlot | \n",
+ " Silhouette Plot | \n",
+ " Calculates and visualizes Silhouette Score, assessing the degree of data point suitability to its cluster in ML... | \n",
+ " True | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.TrainingTestDegradation | \n",
+ " Training Test Degradation | \n",
+ " Tests if model performance degradation between training and test datasets exceeds a predefined threshold.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'max_threshold': {'type': 'float', 'default': 0.1}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.VMeasure | \n",
+ " V Measure | \n",
+ " Evaluates homogeneity and completeness of a clustering model using the V Measure Score.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['sklearn', 'model_performance'] | \n",
+ " ['clustering'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.sklearn.WeakspotsDiagnosis | \n",
+ " Weakspots Diagnosis | \n",
+ " Identifies and visualizes weak spots in a machine learning model's performance across various sections of the... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'features_columns': {'type': None, 'default': None}, 'metrics': {'type': None, 'default': None}, 'thresholds': {'type': None, 'default': None}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_diagnosis', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.AutoARIMA | \n",
+ " Auto ARIMA | \n",
+ " Evaluates ARIMA models for time-series forecasting, ranking them using Bayesian and Akaike Information Criteria.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'forecasting', 'model_selection', 'statsmodels'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.CumulativePredictionProbabilities | \n",
+ " Cumulative Prediction Probabilities | \n",
+ " Visualizes cumulative probabilities of positive and negative classes for both training and testing in classification models.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'title': {'type': 'str', 'default': 'Cumulative Probabilities'}} | \n",
+ " ['visualization', 'credit_risk'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.DurbinWatsonTest | \n",
+ " Durbin Watson Test | \n",
+ " Assesses autocorrelation in time series data features using the Durbin-Watson statistic.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {'threshold': {'type': None, 'default': [1.5, 2.5]}} | \n",
+ " ['time_series_data', 'forecasting', 'statistical_test', 'statsmodels'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.GINITable | \n",
+ " GINI Table | \n",
+ " Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['model_performance'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.KolmogorovSmirnov | \n",
+ " Kolmogorov Smirnov | \n",
+ " Assesses whether each feature in the dataset aligns with a normal distribution using the Kolmogorov-Smirnov test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {'dist': {'type': 'str', 'default': 'norm'}} | \n",
+ " ['tabular_data', 'data_distribution', 'statistical_test', 'statsmodels'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.Lilliefors | \n",
+ " Lilliefors | \n",
+ " Assesses the normality of feature distributions in an ML model's training dataset using the Lilliefors test.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'data_distribution', 'statistical_test', 'statsmodels'] | \n",
+ " ['classification', 'regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.PredictionProbabilitiesHistogram | \n",
+ " Prediction Probabilities Histogram | \n",
+ " Assesses the predictive probability distribution for binary classification to evaluate model performance and... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'title': {'type': 'str', 'default': 'Histogram of Predictive Probabilities'}} | \n",
+ " ['visualization', 'credit_risk'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.RegressionCoeffs | \n",
+ " Regression Coeffs | \n",
+ " Assesses the significance and uncertainty of predictor variables in a regression model through visualization of... | \n",
+ " True | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {} | \n",
+ " ['tabular_data', 'visualization', 'model_training'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.RegressionFeatureSignificance | \n",
+ " Regression Feature Significance | \n",
+ " Assesses and visualizes the statistical significance of features in a regression model.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model'] | \n",
+ " {'fontsize': {'type': 'int', 'default': 10}, 'p_threshold': {'type': 'float', 'default': 0.05}} | \n",
+ " ['statistical_test', 'model_interpretation', 'visualization', 'feature_importance'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.RegressionModelForecastPlot | \n",
+ " Regression Model Forecast Plot | \n",
+ " Generates plots to visually compare the forecasted outcomes of a regression model against actual observed values over... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {'start_date': {'type': None, 'default': None}, 'end_date': {'type': None, 'default': None}} | \n",
+ " ['time_series_data', 'forecasting', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels | \n",
+ " Regression Model Forecast Plot Levels | \n",
+ " Assesses the alignment between forecasted and observed values in regression models through visual plots... | \n",
+ " True | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['time_series_data', 'forecasting', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.RegressionModelSensitivityPlot | \n",
+ " Regression Model Sensitivity Plot | \n",
+ " Assesses the sensitivity of a regression model to changes in independent variables by applying shocks and... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'shocks': {'type': None, 'default': [0.1]}, 'transformation': {'type': None, 'default': None}} | \n",
+ " ['senstivity_analysis', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.RegressionModelSummary | \n",
+ " Regression Model Summary | \n",
+ " Evaluates regression model performance using metrics including R-Squared, Adjusted R-Squared, MSE, and RMSE.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['model_performance', 'regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.RegressionPermutationFeatureImportance | \n",
+ " Regression Permutation Feature Importance | \n",
+ " Assesses the significance of each feature in a model by evaluating the impact on model performance when feature... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {'fontsize': {'type': 'int', 'default': 12}, 'figure_height': {'type': 'int', 'default': 500}} | \n",
+ " ['statsmodels', 'feature_importance', 'visualization'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.model_validation.statsmodels.ScorecardHistogram | \n",
+ " Scorecard Histogram | \n",
+ " The Scorecard Histogram test evaluates the distribution of credit scores between default and non-default instances,... | \n",
+ " True | \n",
+ " False | \n",
+ " ['dataset'] | \n",
+ " {'title': {'type': 'str', 'default': 'Histogram of Scores'}, 'score_column': {'type': 'str', 'default': 'score'}} | \n",
+ " ['visualization', 'credit_risk', 'logistic_regression'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.CalibrationCurveDrift | \n",
+ " Calibration Curve Drift | \n",
+ " Evaluates changes in probability calibration between reference and monitoring datasets.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'n_bins': {'type': 'int', 'default': 10}, 'drift_pct_threshold': {'type': 'float', 'default': 20}} | \n",
+ " ['sklearn', 'binary_classification', 'model_performance', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.ClassDiscriminationDrift | \n",
+ " Class Discrimination Drift | \n",
+ " Compares classification discrimination metrics between reference and monitoring datasets.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'drift_pct_threshold': {'type': '_empty', 'default': 20}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.ClassImbalanceDrift | \n",
+ " Class Imbalance Drift | \n",
+ " Evaluates drift in class distribution between reference and monitoring datasets.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets'] | \n",
+ " {'drift_pct_threshold': {'type': 'float', 'default': 5.0}, 'title': {'type': 'str', 'default': 'Class Distribution Drift'}} | \n",
+ " ['tabular_data', 'binary_classification', 'multiclass_classification'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.ClassificationAccuracyDrift | \n",
+ " Classification Accuracy Drift | \n",
+ " Compares classification accuracy metrics between reference and monitoring datasets.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'drift_pct_threshold': {'type': '_empty', 'default': 20}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.ConfusionMatrixDrift | \n",
+ " Confusion Matrix Drift | \n",
+ " Compares confusion matrix metrics between reference and monitoring datasets.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'drift_pct_threshold': {'type': '_empty', 'default': 20}} | \n",
+ " ['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.CumulativePredictionProbabilitiesDrift | \n",
+ " Cumulative Prediction Probabilities Drift | \n",
+ " Compares cumulative prediction probability distributions between reference and monitoring datasets.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['datasets', 'model'] | \n",
+ " {} | \n",
+ " ['visualization', 'credit_risk'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.FeatureDrift | \n",
+ " Feature Drift | \n",
+ " Evaluates changes in feature distribution over time to identify potential model drift.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets'] | \n",
+ " {'bins': {'type': '_empty', 'default': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]}, 'feature_columns': {'type': '_empty', 'default': None}, 'psi_threshold': {'type': '_empty', 'default': 0.2}} | \n",
+ " ['visualization'] | \n",
+ " ['monitoring'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.PredictionAcrossEachFeature | \n",
+ " Prediction Across Each Feature | \n",
+ " Assesses differences in model predictions across individual features between reference and monitoring datasets... | \n",
+ " True | \n",
+ " False | \n",
+ " ['datasets', 'model'] | \n",
+ " {} | \n",
+ " ['visualization'] | \n",
+ " ['monitoring'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.PredictionCorrelation | \n",
+ " Prediction Correlation | \n",
+ " Assesses correlation changes between model predictions from reference and monitoring datasets to detect potential... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'drift_pct_threshold': {'type': 'float', 'default': 20}} | \n",
+ " ['visualization'] | \n",
+ " ['monitoring'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.PredictionProbabilitiesHistogramDrift | \n",
+ " Prediction Probabilities Histogram Drift | \n",
+ " Compares prediction probability distributions between reference and monitoring datasets.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'title': {'type': '_empty', 'default': 'Prediction Probabilities Histogram Drift'}, 'drift_pct_threshold': {'type': 'float', 'default': 20.0}} | \n",
+ " ['visualization', 'credit_risk'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.PredictionQuantilesAcrossFeatures | \n",
+ " Prediction Quantiles Across Features | \n",
+ " Assesses differences in model prediction distributions across individual features between reference... | \n",
+ " True | \n",
+ " False | \n",
+ " ['datasets', 'model'] | \n",
+ " {} | \n",
+ " ['visualization'] | \n",
+ " ['monitoring'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.ROCCurveDrift | \n",
+ " ROC Curve Drift | \n",
+ " Compares ROC curves between reference and monitoring datasets.... | \n",
+ " True | \n",
+ " False | \n",
+ " ['datasets', 'model'] | \n",
+ " {} | \n",
+ " ['sklearn', 'binary_classification', 'model_performance', 'visualization'] | \n",
+ " ['classification', 'text_classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.ScoreBandsDrift | \n",
+ " Score Bands Drift | \n",
+ " Analyzes drift in population distribution and default rates across score bands.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'score_column': {'type': 'str', 'default': 'score'}, 'score_bands': {'type': 'list', 'default': None}, 'drift_threshold': {'type': 'float', 'default': 20.0}} | \n",
+ " ['visualization', 'credit_risk', 'scorecard'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.ScorecardHistogramDrift | \n",
+ " Scorecard Histogram Drift | \n",
+ " Compares score distributions between reference and monitoring datasets for each class.... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets'] | \n",
+ " {'score_column': {'type': 'str', 'default': 'score'}, 'title': {'type': 'str', 'default': 'Scorecard Histogram Drift'}, 'drift_pct_threshold': {'type': 'float', 'default': 20.0}} | \n",
+ " ['visualization', 'credit_risk', 'logistic_regression'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.ongoing_monitoring.TargetPredictionDistributionPlot | \n",
+ " Target Prediction Distribution Plot | \n",
+ " Assesses differences in prediction distributions between a reference dataset and a monitoring dataset to identify... | \n",
+ " True | \n",
+ " True | \n",
+ " ['datasets', 'model'] | \n",
+ " {'drift_pct_threshold': {'type': 'float', 'default': 20}} | \n",
+ " ['visualization'] | \n",
+ " ['monitoring'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.prompt_validation.Bias | \n",
+ " Bias | \n",
+ " Assesses potential bias in a Large Language Model by analyzing the distribution and order of exemplars in the... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {'min_threshold': {'type': '_empty', 'default': 7}, 'judge_llm': {'type': '_empty', 'default': None}} | \n",
+ " ['llm', 'few_shot'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.prompt_validation.Clarity | \n",
+ " Clarity | \n",
+ " Evaluates and scores the clarity of prompts in a Large Language Model based on specified guidelines.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {'min_threshold': {'type': '_empty', 'default': 7}, 'judge_llm': {'type': '_empty', 'default': None}} | \n",
+ " ['llm', 'zero_shot', 'few_shot'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.prompt_validation.Conciseness | \n",
+ " Conciseness | \n",
+ " Analyzes and grades the conciseness of prompts provided to a Large Language Model.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {'min_threshold': {'type': '_empty', 'default': 7}, 'judge_llm': {'type': '_empty', 'default': None}} | \n",
+ " ['llm', 'zero_shot', 'few_shot'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.prompt_validation.Delimitation | \n",
+ " Delimitation | \n",
+ " Evaluates the proper use of delimiters in prompts provided to Large Language Models.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {'min_threshold': {'type': '_empty', 'default': 7}, 'judge_llm': {'type': '_empty', 'default': None}} | \n",
+ " ['llm', 'zero_shot', 'few_shot'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.prompt_validation.NegativeInstruction | \n",
+ " Negative Instruction | \n",
+ " Evaluates and grades the use of affirmative, proactive language over negative instructions in LLM prompts.... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {'min_threshold': {'type': '_empty', 'default': 7}, 'judge_llm': {'type': '_empty', 'default': None}} | \n",
+ " ['llm', 'zero_shot', 'few_shot'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.prompt_validation.Robustness | \n",
+ " Robustness | \n",
+ " Assesses the robustness of prompts provided to a Large Language Model under varying conditions and contexts. This test... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model', 'dataset'] | \n",
+ " {'num_tests': {'type': '_empty', 'default': 10}, 'judge_llm': {'type': '_empty', 'default': None}} | \n",
+ " ['llm', 'zero_shot', 'few_shot'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.prompt_validation.Specificity | \n",
+ " Specificity | \n",
+ " Evaluates and scores the specificity of prompts provided to a Large Language Model (LLM), based on clarity, detail,... | \n",
+ " False | \n",
+ " True | \n",
+ " ['model'] | \n",
+ " {'min_threshold': {'type': '_empty', 'default': 7}, 'judge_llm': {'type': '_empty', 'default': None}} | \n",
+ " ['llm', 'zero_shot', 'few_shot'] | \n",
+ " ['text_classification', 'text_summarization'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.classification.Accuracy | \n",
+ " Accuracy | \n",
+ " Calculates the accuracy of a model | \n",
+ " False | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['classification'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.classification.F1 | \n",
+ " F1 | \n",
+ " Calculates the F1 score for a classification model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['classification'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.classification.Precision | \n",
+ " Precision | \n",
+ " Calculates the precision for a classification model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['classification'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.classification.ROC_AUC | \n",
+ " ROC AUC | \n",
+ " Calculates the ROC AUC for a classification model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['classification'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.classification.Recall | \n",
+ " Recall | \n",
+ " Calculates the recall for a classification model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['classification'] | \n",
+ " ['classification'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.AdjustedRSquaredScore | \n",
+ " Adjusted R Squared Score | \n",
+ " Calculates the adjusted R-squared score for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.GiniCoefficient | \n",
+ " Gini Coefficient | \n",
+ " Calculates the Gini coefficient for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.HuberLoss | \n",
+ " Huber Loss | \n",
+ " Calculates the Huber loss for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.KolmogorovSmirnovStatistic | \n",
+ " Kolmogorov Smirnov Statistic | \n",
+ " Calculates the Kolmogorov-Smirnov statistic for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['dataset', 'model'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.MeanAbsoluteError | \n",
+ " Mean Absolute Error | \n",
+ " Calculates the mean absolute error for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.MeanAbsolutePercentageError | \n",
+ " Mean Absolute Percentage Error | \n",
+ " Calculates the mean absolute percentage error for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.MeanBiasDeviation | \n",
+ " Mean Bias Deviation | \n",
+ " Calculates the mean bias deviation for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.MeanSquaredError | \n",
+ " Mean Squared Error | \n",
+ " Calculates the mean squared error for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.QuantileLoss | \n",
+ " Quantile Loss | \n",
+ " Calculates the quantile loss for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {'quantile': {'type': '_empty', 'default': 0.5}} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.RSquaredScore | \n",
+ " R Squared Score | \n",
+ " Calculates the R-squared score for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
+ "
\n",
+ " \n",
+ " | validmind.unit_metrics.regression.RootMeanSquaredError | \n",
+ " Root Mean Squared Error | \n",
+ " Calculates the root mean squared error for a regression model. | \n",
+ " False | \n",
+ " False | \n",
+ " ['model', 'dataset'] | \n",
+ " {} | \n",
+ " ['regression'] | \n",
+ " ['regression'] | \n",
"
\n",
" \n",
"