Etiq AI - Data Science Copilot
Etiq 1.x Docs
Etiq 1.x Docs
  • Welcome to Etiq
  • Quickstart
  • Usage Plans
  • Key Concepts
    • Snapshot
    • Scan
    • Config
    • Custom Tests
  • Scan Types
    • Accuracy
    • Leakage
    • Drift
    • Bias
    • Data Issues
    • Data Fingerprinting
  • RCA
    • RCA Type Scans
    • Accuracy RCA Scan
    • Bias RCA Scan
    • Drift RCA Scan
  • Dashboard
    • Dashboard Components
    • Project Sharing
    • Data Synchronisation
  • Integrations
    • Airflow
    • Great Expectations
  • FAQ & Other
    • FAQ
    • API Documentation
    • Low Level API
Powered by GitBook
On this page
  • Overview
  • Use Cases
  • Running An Existing Expectation Suite
  • Importing Existing Results
  • Declaring Expectations In Code
  • Declaring Expectations in JSON Config
  • Integration API Documentation

Was this helpful?

  1. Integrations

Great Expectations

Great Expectations Integration with ETIQ library details

PreviousAirflowNextFAQ

Last updated 1 year ago

Was this helpful?

Feature added in Etiq 1.6

Overview

ETIQ adds integration with the OSS library. This allows you to quickly add a suite of tests for your dataset and show the results in the ETIQ dashboard.

The Great Expectations python library needs to be installed if you want to use this functionality;

pip install great_expectations

Use Cases

The library exposes Snapshot.scan_expectations() through which we can run suites or import existing results. Suites can either come from existing contexts, manually added via code or via JSON.

For the results to be shown in the ETIQ dashboard, you need to run etiq.login() before starting the scan.

Running An Existing Expectation Suite

If you've got an existing expectation suite you can pass it that suite as an argument to Snapshot.scan_expectations()

import great_expectations as ge

context = ge.get_context()

# What Suites do we have?
suite_names = context.list_expectation_suite_names()
# For example...
chosen_suite = suite_names[0]

# Now tell ETIQ to run them;
(segments, issues, aggregates) = snapshot.scan_expectations(context=context, suite_name=chosen_suite)

Importing Existing Results

If you've already run the suite, you can just pass those in. This will cause the results to be uploaded into the dashboard;

# existing_suite_checkpoint is a checkpoint from your own existing suite.
my_results = existing_suite_checkpoint.run()

# This will upload results to the dashboard if logged in.
# And return the results in the etiq style of three datasets showing segments,
# issues and aggregated issues as pandas datasets.
(segments, issues, aggregates) = snapshot.scan_expectations(results=my_results)

Declaring Expectations In Code

# Regular snapshot created in ETIQ
snapshot = project.snapshots.create(
    name="My Simple expectations",
    dataset=dataset,
    model=etiq.model.DefaultXGBoostClassifier(),
)

# This is our regular GX validator.
validator = snapshot.get_validator()

# Now we can declare our expectations as normal GX. For example;
validator.expect_column_values_to_not_be_null("age")
validator.expect_column_values_to_be_between("age", min_value=0, max_value=70)

# Finally actually run the validation by calling `scan_expectations`
segments, issues, aggregates = snapshot.scan_expectations(validator=validator)

These expectations are only run when scan_expectations is called.

Declaring Expectations in JSON Config

Much of the ETIQ library can be driven using a JSON config file. We provide a simple way to specify expectations in this text based format;

expectation_config.json
{
  "scan_expectations": {
    "json_suite": [
      {
        "expect_column_values_to_not_be_null": "age"
      },
      {
        "expect_column_values_to_be_between": {
          "column": "age",
          "min_value": 0,
          "max_value": 120,
        }
      }, 
    ]
  }
}

The Python code is then just:

# ...
from etiq import etiq_config
# ...

with etiq_config("expectation_config.json"):
    segments, issues, aggregates = snapshot.scan_expectations()

Syntax/Required Sections

  • scan_expectations - An Object.

  • scan_expectations.json_suite - A list of JSON expectations.

Expectations can be specified in two ways;

  1. If the expectation takes a single argument, then it can just be {<expectation_name>: <argument>}

  2. If the expectation takes more arguments, the argument itself should be an object where each key-value pair represents an argument and value.

The above expectation translates to:

validator.expect_column_values_to_not_be_null("age")
validator.expect_column_values_to_be_between(column="age", min_value=0, max_value=120)

Integration API Documentation

Snapshot.get_validator() -> Validator

This convenience method returns a validator object for creating expectation suites against.

Snapshot.scan_expectations(validator, context, suite_name, results)

This is our entry point to run expectations etc. Unless using the JSON method above, you must pass one or more of these arguments in:

  • validator - A GX Validator object to run.

  • context - A GX context object. This would contain an existing suite to run.

  • suite_name - String, name of suite to run from the context.

  • results - Existing results to send to the dashboard.

You can declare your expectations in code directly e.g. as part of a notebook. We supply a helper method Snapshot.get_validator() to quickly get a Great Expectations .

This will then have all the expectations available to you. The Great Expectations website holds a useful .

validator object
reference list
Great Expectations
Running An Existing Expectation Suite
Importing Existing Results
Declaring Expectations in Code
Declaring Expectations in JSON