Responsible AI Guidelines

AI systems can cause a variety of fairness-related harms, including harms involving people’s individual experiences with AI systems or the ways that AI systems represent the groups to which they belong. Prioritizing fairness in AI systems is a sociotechnical challenge.
Responsible AI Guidelines suggest the best way to build fairness, interpretability, privacy, and security into these systems.

1. Error Analysis:

Error analysis is a fairly very important steps in the whole ML lifecycle, this helps in identifying the behavior of model and also correct it as needed at different level.
Error could be because of many reason, let’s start with input

Input/Data: Accelerated framework, helps to identify if there is any error associated with data by checking

• Missing data
• Ensuring appropriate train and test split
• Augmenting data

Feature Selection:
Another level of error may occur because of inappropriate feature selection or data leak

• Accelerated ML aims to determine data leak by applying different framework which are common in practice (Yet to be implemented)
• It also aims to determine outliers so because of outliers the model should not get impacted (Yet to be implemented)
• Accelerated ML also aims to determine model drift which is basically change in the data distribution of train and test on production, which may result into wrong prediction (yet to be implemented)

Others:
Other area where the model could go wrong would be performance metric, basically which performance metric is good for what scenario, e.g. if we have imbalance data then simple accuracy as performance metric might not be good as this might favor the dominant class if our model is simply predicting dominant class as class label.

Also Accelerated ML, aims to compare baseline model with new model to determine if the new model is better than base model or not. (yet to be implemented)

2. Model Interpretability & Exploration

• Model Explanations:
  To explain the models and understand the value and accuracy/other metrics of their findings based on various cohorts/subgroups in data. Interpretability is also important to debug machine learning models and make informed decisions about how to improve them.
  
  
• Feature Importance:
  To improve the efficiency and effectiveness of a predictive model on the problem, we can apply Feature importance since it provides us with a basis for dimensionality reduction and feature selection. It refers to a class of techniques that assign a score to input features based on how useful they are at predicting a target variable. Feature importance scores provide insight into the dataset. The relative scores highlight which features may be the most relevant to the target, and the converse, which features are the least relevant. We are using ML Interpret and Exploration classes so that from a wide number of variables, we can pick those variables only that provide maximum variability along the prediction column of these classes. We can choose out of two flavors of feature importance implementations on need basis that is: Global (Based on whole dataset, aggregated value) and Local (Record to record basis). Local measures focus on the contribution of features for a specific prediction, whereas global measures take all predictions into account.

3. Handle Data Biasness

Data bias in machine learning is a type of error in which certain elements of a dataset are more heavily weighted and/or represented than others. A biased dataset does not accurately represent a model’s use case, resulting in skewed outcomes, low accuracy levels, and analytical errors.​ We identify subclasses that are stereotyped of being undersampled and oversample them (using SMOTE) to give equal opportunity.

We can also use Fairlearn to ​assess and mitigate model unfairness. ​
To mitigate fairness related risks we can apply optimization strategy based on thresholds (Single threshold, Demographic balance, Equal Opportunity, Equal Accuracy, Subgroup Thresholds) to change the predictions for model​ without retraining the model.