RCA Type Scans

Some scans are simple tests that show whether a metric is above or below certain thresholds set by the user. Other scans are more complex and look at whether for a part of the sample dataset a metric is below or above the threshold. We call this RCA - root cause analysis. This will help you discover segments of customers or groups of records for which the model has a lower than expected accuracy or groups for which bias thresholds are not met.
Imagine if your accuracy metrics tests have picked up issues, this test finds out exactly which segment has the issue, which should help you fix it sooner.
If only a part of the data drifted or only a segment is underperforming, your overall tests might not pick up on it, but this test would. While you can pre-set segments you are interested in, if you just run the scan as is, it will discover problematic segments on its own.
This pipeline is experimental.
RCA type tests are also very useful when it comes to the drivers that impact a model’s decision. We don’t include explainability scans in our public release. However if this is something you need, email us [email protected]
You should expect a level of variability of your model for different parts of your dataset. However, if the variability is too high, with larger segments failing you might want to understand the causes of this, and address them.

Scan Types - Previous

Bias

Next - RCA

Accuracy RCA Scan

Last modified 1mo ago

Copy link