Etiq Docs
Search…
Welcome to Etiq
Intro
Etiq is an ML testing platform for data scientists and ML engineers. Use Etiq’s lightweight tools to identify ML specific issues. This will help prevent accuracy loss in production and reduce time it takes to validate a model and transition it from prototype to production-level pipelines. As early as initial built stages, test your system to prevent operational issues downstream arising from poorly functioning or misunderstood ML models.
Our concept is to provide error detection functionality that looks like unit tests and is as easy to use and as modular as unit tests, but is backed by deep pipelines.
Passed and failed tests % by model version
This public release includes tests in the following areas: Accuracy, Leakage, Drift and Bias
For other error areas, like explainability, robustness, sensitivity - get in touch as there is functionality currently outside the main API but which can be provided to you on demand. We’re constantly adding more out-of-the-box tests so stay tuned for next releases.
In addition to out-of-the-box functionality we also have multiple customization options and a lower level API available because we know you want tests that are specific to your use case.
Other tools focus primarily on helping you optimise for metrics during the experimentation phase. But metrics calculated in a pre-production environment really are just indicators of what might happen in production. With Etiq, you can use them as what they are: indicators and go beyond optimisation. As early as the experimentation phase, you can easily start testing for potential issues to ensure your model performs as well as possible in production.
When to use Etiq / Use cases
.jpg?alt=media&token=786f99f6-3115-423a-b8a7-4f7835a7e942)
Use Etiq during build, validation, productionalizing and when the model is live
We recommend using Etiq’s test functionality throughout the model build process including in your production pipeline.
Having similar tests throughout the pipeline will help ensure that the deployed model does in production what the person who designed it pre-production expects it to do.
Etiq functionality covers binary classification models trained on tabular data. Etiq’s tests and interface work best with models related somehow to people: customers, employees, accounts, etc.
Our tests are model type specific which means they can be more user friendly and go in deeper than more general versions. However, if you want to apply Etiq tests to your segmentation, regression projects, or recommendation engines, get in touch with us. If you have an ensemble model, you can apply testing functionality throughout the build, and most of the tests will be applicable at your final training step.
We started off with algorithmic bias, which we think is still the most complex and most problematic area in machine learning today, and from there we branched out to other error areas. Because of our focus and expertise in this area, our tool can be used in regulated environments such as finance, insurance.
Etiq supports models from XGBoost, LightGBM, PyTorch, TensorFlow, Keras and scikit-learn.
Why use Etiq for ML testing?
Multi-step pipelines: The problem with testing for ML is that sometimes you need a multi-step pipeline to run one test. Especially for bias tests, more complex concept drift, any explainability tests. That means you need to log the results of each of the steps, and productionalize the testing itself as you would your actual model. Etiq does that for you.
Additional benefits:
- Discrete tests that you can plug in at different steps: with ML you do have to test even once your model is in production. But your model is complex enough without some heavy testing functionality to stop it from running properly. You can add specific tests at different points in your pipeline, and take them out, run them in parallel or run them once you have the outputs of the model. Whatever the set-up, you can centralize the results in the same location and voila you have a detailed working Model monitoring tool.
- Custom, custom, custom: you can customize the thresholds of your test, and add your own custom metrics to create your own custom tests.
- Root cause analysis: some of our tests help you understand why an issue is happening much faster. For instance, your overall accuracy is looking good, but there is a segment which is really underperforming. The test can show you which segment that is (without you having to pre-set the segments).
- Distributed functionality: some of your tests will be pretty heavy going. We have functionality for you to run a step in the test in a separate distributed environment, and then bring back relevant output to a step in your existing environment
- Documented by default: when testing part of the struggle is actually documenting both the test results and the test template itself and the parameters it used. Etiq solves this problem by having config files associated with each test suite. This way you don't need to spend additional time documenting the tests you've run, you can just check the config associated with the test suite.
Coming up: if someone comes up with a different (more accurate) SHAP implementation tomorrow and you want to include it in your explainability tests instead of the old SHAP implementation, you don't want to have to change your whole existing testing suite. You should be able to take one step out (old SHAP) and replace it with another step (new SHAP).
As we're developing our low level API, you will have the freedom to build your tests in as great a detail as you want. We hope that the final product will also cater to the community who is building better bias identification, explainability and drift algorithms. Do you want to experiment with causality, adversarial debiasing or a new drift metric - we're working hard towards building a product that can support quick experimentation in this area.
Last modified 6d ago
Copy link