Welcome to Etiq

Intro

Etiq is an ML testing platform for data scientists and ML engineers. Use Etiq’s lightweight tools to identify ML specific issues. This will help prevent accuracy loss in production and reduce time it takes to validate a model and transition it from prototype to production-level pipelines. As early as initial built stages, test your system to prevent operational issues downstream arising from poorly functioning or misunderstood ML models.

Our concept is to provide error detection functionality that provides the look and feel and ease of use as unit tests, but backed by deep pipelines.

Passed and failed tests % by model version

This package includes tests in the following areas: Accuracy, Data Issues, Leakage, Drift and Bias

For other error areas, like explainability, robustness, sensitivity - get in touch as there is functionality currently outside the main API but which can be provided to you on demand. We’re constantly adding more out-of-the-box tests so stay tuned for next releases.

In addition to out-of-the-box capabilties we also have multiple customization options and a low-level API available. These provide additional functionality to ensure you can meet the needs of your specific use cases.

Whilst other tools focus primarily on helping you optimise for metrics during the experimentation phase, at Etiq we know that metrics calculated in a pre-production environment are only likely to be indicators of true performance whilst in production. With Etiq, you can use them for what they are: indicators and go beyond optimisation. As early as the experimentation phase, you can easily start testing for potential issues to ensure your model performs as well as possible in production.

When to use Etiq / Use cases

Use Etiq during build, validation, productionalizing and when the model is live

We recommend using Etiq’s test functionality throughout the model build process including in your production pipeline.

Having similar tests throughout the pipeline will help ensure that the deployed model does in production what the person who designed it pre-production expects it to do.

In production, at the moment we provide functionality for batch processing only.

Etiq supports models from XGBoost, LightGBM, PyTorch, TensorFlow, Keras and scikit-learn.

Why use Etiq for ML testing?

Multi-step pipelines: The problem with testing for ML is that sometimes you need a multi-step pipeline to run one test. Especially for bias tests, more complex concept drift, any explainability tests. That means you need to log the results of each of the steps, and productionalize the testing itself as you would your actual model. Etiq does that for you.

Additional benefits:

• Discrete tests that you can plug in at different steps: with ML you do have to test even once your model is in production. But your model is complex enough without some heavy testing functionality to stop it from running properly. You can add specific tests at different points in your pipeline, and take them out, run them in parallel or run them once you have the outputs of the model. Whatever the set-up, you can centralize the results in the same location and voila you have a detailed working Model monitoring tool.

• Custom, custom, custom: you can customize the thresholds of your test, and add your own custom metrics to create your own custom tests.

• Root cause analysis: some of our tests help you understand why an issue is happening much faster. For instance, your overall accuracy is looking good, but there is a segment which is really underperforming. The test can show you which segment that is (without you having to pre-set the segments).

• Documented by default: when testing part of the struggle is actually documenting both the test results and the test template itself and the parameters it used. Etiq solves this problem by having config files associated with each test suite. This way you don't need to spend additional time documenting the tests you've run, you can just check the config associated with the test suite.