Accuracy RCA Scan

scan_accuracy_metrics_rca is the typical RCA scan. You can find an example notebook here.

An example config file is as below:

{
    "dataset": {
        "label": "income",
        "bias_params": {
            "protected": "gender",
            "privileged": 1,
            "unprivileged": 0,
            "positive_outcome_label": 1,
            "negative_outcome_label": 0
        },
        "train_valid_test_splits": [0.0, 1.0, 0.0],
        "remove_protected_from_features": true
    },
    "scan_accuracy_metrics": {
        "thresholds": {
            "accuracy": [0.8, 1.0],
            "true_pos_rate": [0.75, 1.0],
            "true_neg_rate":  [0.7, 1.0]           
        }
    },
    "scan_accuracy_metrics_rca": {
        "thresholds": {
            "accuracy": [0.8, 1.0],
            "true_pos_rate": [0.7, 1.0]          
        },
        "metric_filter": ["accuracy", "true_pos_rate"], 
        "minimum_segment_size": 1000
    }
}

The syntax to run the scan after logging the model and dataset is the following:

snapshot.scan_accuracy_metrics_rca()

Like with all RCA scans the principle behind the scan is that it searches through different combinations of records and it finds those combinations for which the metric is outside the thresholds. As per the usual scans, you can set the thresholds for what constitutes an issue for your use case. You can also filter out the metrics you want/do not want RCA for, using, for example,:

"metric_filter": ["accuracy", "true_pos_rate"]

To make the metrics per group meaningful, it assigns a minimum number of records that constitutes a group, but you can change this by using the following syntax/parameter as per config example below:

"minimum_segment_size": 1000

The minimum segment size will impact the results. We recommend setting the minimum segment size at 2% of the sample size. However if 2% is less than a significant segment size for your sample (e.g. less than 1000), please increase it. By default the scans use 2% of your sample size.

At the moment we only have results retrieval through the IDE and by snapshot using the following syntax and then call each of the elements.

(segments_accuracy, issues_accuracy, issue_summary_accuracy)  = snapshot.scan_accuracy_metrics_rca()

The end results give business rules to the segments to help you understand the records you’re having an issue with.

We are working to add more retrieval methods.

Out of the box you can scan for the following metrics:

1. Accuracy - % correct out of total

2. True positive rate - the proportion positive outcome labels that are correctly classified out of all positive outcome labels

3. True negative rate - the proportion negative outcome labels that are correctly classified out of all negative outcome labels