Error analysis

How it works

Error analysis systematically identifies, categorizes, and resolves prediction errors to improve model accuracy and fairness. The core principles are:

• Identify errors: Pinpoint the model's prediction errors. This involves comparing the model's predictions with the ground truth to find discrepancies. Errors can be classified into different types, such as false positives and false negatives.

• Categorize errors: Classify errors based on their characteristics. This helps you better understand the root causes, such as data imbalance, insufficient features, or model bias. This process may require domain expertise and human judgment.

• Analyze root causes: Analyze the causes of each error category. This step is critical because it directly guides model optimization. This may involve analyzing data quality, model design issues, feature engineering, or data representation problems.

• Take corrective actions: Based on the analysis, the development team can take specific measures to address the model's issues. These measures might include data cleaning, rebalancing the dataset, modifying the model architecture, introducing new features, or using different algorithms.

• Iterate and evaluate: Error analysis is not a one-time task but an iterative process. After each model modification, you must perform another error analysis to assess whether the changes were effective, improved performance, or introduced new issues.

• Document and report: To ensure transparency and interpretability, thoroughly document the error analysis process, findings, and corrective actions. This also helps team members understand the model's limitations and provides valuable feedback for other project phases.

This tutorial uses the responsible-ai-toolbox in PAI DSW to perform error analysis on a model that predicts whether an individual's annual income exceeds 50K based on census data.

Before you begin

• A DSW instance. If you do not have one, see Create a DSW instance. The recommended configuration is as follows:

  • Recommended instance type: ecs.gn6v-c8g1.2xlarge

  • Image: Python 3.9 or later is recommended. In this tutorial, the official image used is tensorflow-pytorch-develop:2.14-pytorch2.1-gpu-py311-cu118-ubuntu22.04.

  • Model: The responsible-ai-toolbox supports regression and binary classification models from the Scikit-learn framework.

• Training dataset: Use your own training dataset. To use the sample dataset for this tutorial, see Step 3: Prepare the dataset.

• Algorithm model: Use your own algorithm model. To use the sample model for this tutorial, see Step 5: Train the model.

Step 1: Go to the DSW gallery

1. Log on to the PAI console.

2. In the upper-left corner, select a region based on your needs.

3. In the left-side navigation pane, choose QuickStart > Notebook gallery. Search for "Responsible AI-Error Analysis" and click Open in DSW on the card.

4. Select a DSW instance and click Open Notebook. The "Responsible AI-Error Analysis" notebook opens.

Step 2: Import dependencies

Install the raiwidgets dependency for the responsible-ai-toolbox, used for evaluation.

!pip install raiwidgets==0.34.1

Import the Responsible AI and Scikit-learn dependencies for model training.

# Import dependencies for Responsible AI

import zipfile
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

import pandas as pd
from lightgbm import LGBMClassifier
from raiutils.dataset import fetch_dataset
import sklearn
from packaging import version
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

Step 3: Prepare the dataset

Download and extract the census dataset. The extracted folder contains the training data adult-train.csv and the test data adult-test.csv.

# Specify the name of the dataset file.
outdirname = 'responsibleai.12.28.21'
zipfilename = outdirname + '.zip'

# Download and extract the dataset.
fetch_dataset('https://publictestdatasets.blob.core.windows.net/data/' + zipfilename, zipfilename)
with zipfile.ZipFile(zipfilename, 'r') as unzip:
    unzip.extractall('.')

Step 4: Preprocess the data

1. Load the training data adult-train.csv and the test data adult-test.csv.

2. Split the training and testing data into feature variables and target variables. The target variable is the actual result that the model predicts. Feature variables are the variables in each data instance other than the target variable. In this example, the target variable is income, and the feature variables include workclass, education, and marital-status.

3. Convert the training data to the NumPy format for training.

# Load the training and test data.
train_data = pd.read_csv('adult-train.csv', skipinitialspace=True)
test_data = pd.read_csv('adult-test.csv', skipinitialspace=True)


# Define the columns for the features and the target variable.
target_feature = 'income'
categorical_features = ['workclass', 'education', 'marital-status',
                        'occupation', 'relationship', 'race', 'gender', 'native-country']

# Define a function to split the features and the target variable.
def split_label(dataset, target_feature):
    X = dataset.drop([target_feature], axis=1)
    y = dataset[[target_feature]]
    return X, y


# Split the features and the target variable.
X_train_original, y_train = split_label(train_data, target_feature)
X_test_original, y_test = split_label(test_data, target_feature)


# Convert to the NumPy format.
y_train = y_train[target_feature].to_numpy()
y_test = y_test[target_feature].to_numpy()

# Define a test sample.
test_data_sample = test_data.sample(n=500, random_state=5)

To load your own dataset in CSV format:

import pandas as pd

# Load your own dataset in CSV format.
# Use pandas to read the CSV file.
try:
    data = pd.read_csv(filename)
except:
    pass

Step 5: Train the model

Build a data training pipeline using Scikit-learn to train a binary classification model.

# Define the ohe_params parameter based on the scikit-learn version.
if version.parse(sklearn.__version__) < version.parse('1.2'):
    ohe_params = {"sparse": False}
else:
    ohe_params = {"sparse_output": False}

# Define the classification pipeline for feature transformation. The input parameter X represents the training data.
def create_classification_pipeline(X):
    pipe_cfg = {
        'num_cols': X.dtypes[X.dtypes == 'int64'].index.values.tolist(),
        'cat_cols': X.dtypes[X.dtypes == 'object'].index.values.tolist(),
    }
    num_pipe = Pipeline([
        ('num_imputer', SimpleImputer(strategy='median')),
        ('num_scaler', StandardScaler())
    ])
    cat_pipe = Pipeline([
        ('cat_imputer', SimpleImputer(strategy='constant', fill_value='?')),
        ('cat_encoder', OneHotEncoder(handle_unknown='ignore', **ohe_params))
    ])
    feat_pipe = ColumnTransformer([
        ('num_pipe', num_pipe, pipe_cfg['num_cols']),
        ('cat_pipe', cat_pipe, pipe_cfg['cat_cols'])
    ])

    pipeline = Pipeline(steps=[('preprocessor', feat_pipe),
                               ('model', LGBMClassifier(random_state=0))])

    return pipeline
    
# Create the classification model training pipeline.
pipeline = create_classification_pipeline(X_train_original)

# Train the model.
model = pipeline.fit(X_train_original, y_train)

Step 6: Add the Responsible AI component

Add the error analysis component to Responsible AI and run the computation.

# Import the RAI dashboard components.
from raiwidgets import ResponsibleAIDashboard
from responsibleai import RAIInsights

# Define the RAIInsights object.
from responsibleai.feature_metadata import FeatureMetadata
feature_metadata = FeatureMetadata(categorical_features=categorical_features, dropped_features=[])
rai_insights = RAIInsights(model, train_data, test_data_sample, target_feature, 'classification',
                           feature_metadata=feature_metadata)

# Add the error analysis component.
rai_insights.error_analysis.add()

# Perform the RAI computation.
rai_insights.compute()

Step 7: Create the Responsible AI dashboard

1. Create different cohorts by applying filters to analyze specific data segments. Examples include:

• Age is less than 65 and hours-per-week is greater than 40.

• Marital status is "Never-married" or "Divorced".

• Index is less than 20.

• Predicted Y is >50K.

• True Y is >50K.

2. Import the ResponsibleAIDashboard and use the responsible-ai-toolbox to analyze the model.

from raiutils.cohort import Cohort, CohortFilter, CohortFilterMethods
import os
from urllib.parse import urlparse

# Age is less than 65 and hours-per-week is greater than 40.
cohort_filter_age = CohortFilter(
    method=CohortFilterMethods.METHOD_LESS,
    arg=[65],
    column='age')
cohort_filter_hours_per_week = CohortFilter(
    method=CohortFilterMethods.METHOD_GREATER,
    arg=[40],
    column='hours-per-week')

user_cohort_age_and_hours_per_week = Cohort(name='Cohort Age and Hours-Per-Week')
user_cohort_age_and_hours_per_week.add_cohort_filter(cohort_filter_age)
user_cohort_age_and_hours_per_week.add_cohort_filter(cohort_filter_hours_per_week)

# Marital status is "Never-married" or "Divorced".
cohort_filter_marital_status = CohortFilter(
    method=CohortFilterMethods.METHOD_INCLUDES,
    arg=["Never-married", "Divorced"],
    column='marital-status')

user_cohort_marital_status = Cohort(name='Cohort Marital-Status')
user_cohort_marital_status.add_cohort_filter(cohort_filter_marital_status)

# Index is less than 20.
cohort_filter_index = CohortFilter(
    method=CohortFilterMethods.METHOD_LESS,
    arg=[20],
    column='Index')

user_cohort_index = Cohort(name='Cohort Index')
user_cohort_index.add_cohort_filter(cohort_filter_index)

# Predicted Y is >50K.
cohort_filter_predicted_y = CohortFilter(
    method=CohortFilterMethods.METHOD_INCLUDES,
    arg=['>50K'],
    column='Predicted Y')

user_cohort_predicted_y = Cohort(name='Cohort Predicted Y')
user_cohort_predicted_y.add_cohort_filter(cohort_filter_predicted_y)

# True Y is >50K.
cohort_filter_true_y = CohortFilter(
    method=CohortFilterMethods.METHOD_INCLUDES,
    arg=['>50K'],
    column='True Y')

user_cohort_true_y = Cohort(name='Cohort True Y')
user_cohort_true_y.add_cohort_filter(cohort_filter_true_y)

cohort_list = [user_cohort_age_and_hours_per_week,
               user_cohort_marital_status,
               user_cohort_index,
               user_cohort_predicted_y,
               user_cohort_true_y]

# Create the Responsible AI dashboard.
metric_frame_tf = ResponsibleAIDashboard(rai_insights, cohort_list=cohort_list, feature_flights="dataBalanceExperience")

# Set the URL for redirection.
metric_frame_tf.config['baseUrl'] =  'https://{}-proxy-{}.dsw-gateway-{}.data.aliyun.com'.format(
    os.environ.get('JUPYTER_NAME').replace("dsw-",""),
    urlparse(metric_frame_tf.config['baseUrl']).port,
    os.environ.get('dsw_region') )

Step 8: View the error analysis

Click the URL to open the Responsible AI dashboard.

View the error analysis: