Drift RCA Scan
Description of Drift RCA Scan
There are currently two kinds of RCA drift scan in the etiq library used to scan for feature and target drift respectively:
- 1.
scan_drift_metrics_rca - 2.
scan_target_drift_metrics_rca
As a quick refresher:
Feature drift takes place when the distributions of the input features changes. For instance, perhaps you built your model on a sample dataset from the winter period and it's now summer, and your model predicting what kind of dessert people are more likely to buy is not longer as accurate.
Similarly to feature drift, target drift is about distribution of the predicted feature changing from one time period to the next.
Imagine if only a part of the data drifted your overall tests might not pick up on it, but this test would. The scan it will auto-discover problematic segments on its own without the need for the user to specify segments to test.
RCA scans for concept drift are currently not implemented but it is on our roadmap and should be introduced in the near future.
For both types of scans, you have to set the parameters below:
Parameter | Description |
|---|---|
thresholds | This is dictionary indexed by measure name specifying an lower and upper threshold that indicates a drift issue |
drift_measures | This is a list of drift measures to use for the scan. If not specified then all drift measures including using defined ones are used |
ignore_lower_threshold | This boolean flag allows the lower threshold to be ignored when scanning for segments with drift issues (this is by default set to True) |
ignore_upper_threshold | This boolean flag allows the lower threshold to be ignored when scanning for segments with drift issues (this is by default set to True) |
minimum_segment_size | This allows the user to set the minimum number of samples in a segment before the segment can be considered significant. By default this is set to 2% of the number of samples |
features (additional parameter only used for scan_drift_metrics_rca) | This is a list of features to consider when scanning for feature drift. By default all features in the snapshot dataset will be scanned |
An example config file is as follows
{
"dataset": {
"label": "income",
"bias_params": {
"protected": "gender",
"privileged": 1,
"unprivileged": 0,
"positive_outcome_label": 1,
"negative_outcome_label": 0
},
"train_valid_test_splits": [0.8, 0.2, 0.0],
"cat_col": "cat_vars",
"cont_col": "cont_vars"
},
"scan_target_drift_metrics_rca": {
"thresholds": {
"psi": [0.0, 0.15]
},
"drift_measures": ["psi"],
"ignore_lower_threshold": true,
"ignore_upper_threshold": false,
"minimum_segment_size": 1000
},
"scan_drift_metrics_rca": {
"thresholds": {
"psi": [0.0, 0.15]
},
"drift_measures": ["psi"],
"ignore_lower_threshold": true,
"ignore_upper_threshold": false,
"minimum_segment_size": 1000,
"features": ["hours-per-week"]
}
}
You can use any of the drift measures already provided.
The minimum segment size will impact the results. We recommend setting the minimum segment size at 2% of the sample size. However if 2% is less than a significant segment size for your sample (e.g. less than 1000), please increase it. By default the scans use 2% of your sample size.
For more details about how RCA scans work for target and feature drift see the notebook at https://github.com/ETIQ-AI/ml-testing/tree/main/RCA/RCA%20Drift%20Metrics.
Last modified 10mo ago