Snapshot
Etiq works via a lightweight logging mechanic. You log your data and your model (a snapshot) and then you run a scan on it - which is the testing functionality itself. As you experiment with more and more snapshots, you keep scanning your model versions and all the test results and issues found get sent to a centralised dashboard.
A snapshot is a combination of dataset and model, especially for in pre-production testing. To start testing your system you need to log your snapshot to Etiq, and to do so you’d log the dataset and the model. For an end-to-end notebook example, go here.
#Log your dataset
dataset = etiq.BiasDatasetBuilder.dataset(data_encoded, label="<target-feature-name>")
#can also use SimpleDatasetBuilder
#Log your already trained model
model = Model(model_architecture=standard_model, model_fitted=model_fit)
# Creating a snapshot
snapshot = project.snapshots.create(name="<snapshot-name>", dataset=dataset, model=model, bias_params=etiq.BiasDatasetBuilder.bias_params())
For validation and production stages, snapshots are not produced in the course of experimentation, they are produced as a model is deployed and runs in production. But from the point of view of Etiq’s logging mechanic, they get logged the same way. Each time your model scores a new batch of data, it records a new snapshot. However the information needed for testing is slightly different in production vs. pre-production and the tests themselves are a bit different. For drift type tests or generally in production, you might not have the available model, but more importantly, you would need the dataset you’re considering for drift and the benchmark dataset you are comparing against.
This is how you’d log it to Etiq:
# Log a dataset with the comparison data
dataset_s = etiq.SimpleDatasetBuilder.dataset(data_encoded, label="<target-feature-name>")
# Log a dataset with the data from your current view
todays_dataset_s = etiq.SimpleDatasetBuilder.dataset(todays_dataset_df, label="<target-feature-name>")
# Create the snapshot
snapshot = project.snapshots.create(name="<snapshot-name>", dataset=todays_dataset_s, comparison_dataset=dataset_s, model=None)
At the moment we support uploading pandas or spark dataframes to the Etiq dataset object, but we are adding new formats all the time. The dataset you use should be already transformed in such a way that it can be inputted to a model class from any of the libraries mentioned. While Etiq contains some transformations, we recommend using your own. Especially with certain types of transformations (such as normalization) , please do NOT apply your transformation to your whole dataset prior to splitting it into train_test_valid as this will can contribute to leakage. (we will be adding scans to check for this as well in the future).
NB: Note that there might be certain scans not available for certain datasets. For example currently (as of version 1.3.8) you can only run data and feature/target drift scans on snapshots containing spark datasets.
There are currently two types of datasets. These are the SimpleDataset and the BiasDataset. The SimpleDataset is a container for the data to be used by the Etiq package. This includes training, validation and testing data along with metadata identifying categorical, continuous, id and date features.
The BiasDataset contains additional metadata identifying "bias" features.
In order to create a simple dataset from a pandas dataframe the following function:
etiq.SimpleDatasetBuilder.dataset(features, target,
label, cat_col, cont_col, id_col, date_col,
convert_date_cols, datetime_format,
train_valid_test_splits,
random_seed,
name)
can be used. The only required parameter is a dataframe containing the features (and target). If no other parameters are supplied etiq will either use defaults or make its best guess for the other parameters.
The target label (specified using the label parameter) or a separate dataframe containing the targets (specified using the target parameter) should also be specified otherwise the last column of the features dataframe will be chosen as the target by default. The other parameters are as follows:
- cat_col - A list of columns containing categorical data
- cont_col - A list of columns containing continuous data
- id_col - A list of columns containing id data (note these are not used by the model)
- date_col - A list of columns containing date information (note these are also not used by the model)
- convert_date_cols - A True/False flag that determines whether or not to convert the datetime columns from strings to the native datetime format. (This defaults to False)
- datetime_format - The format to use when converting the datetime columns.
- train_valid_test_splits - Tuple containing the training/validation/testing split proportions
- random_seed: Number used to seed the random number generator for random splits
- name - The name to use for the dataset.
Note that the non-dataframe parameters can be loaded from the dataset section of a config file. See the dataset config options.
If, however, the dataframe has already been split into training, validation and testing then the following function:
etiq.SimpleDatasetBuilder.datasets(training_features, training_target,
validation_features, validation_target,
testing_features, testing_target,
label, cat_col, cont_col, id_col, date_col,
convert_date_cols, datetime_format,
name)
can be used where
- training_features - a dataframe containing the training features
- training_target - an (optional) dataframe containing the training target
- validation_features - a dataframe containing the validation feature
- validation_target - an (optional) dataframe containing the validation target
- testing_features - a dataframe containing the testing features
- testing_target - an (optional) dataframe containing the testing target
The other parameters are identical to the previous dataset constructor.
If the etiq.spark module is installed a simple dataset can be constructed from a spark dataframe. The two constructors that can be used are as follows:
etiq.SimpleSparkDatasetBuilder.dataset(features,
label, cat_col, cont_col, id_col, date_col,
convert_date_cols, datetime_format,
train_valid_test_splits,
random_seed,
name)
where features is a spark dataframe and the other parameters are as described in the previous section. To create a simple dataset from "pre-split" spark dataframes we use
etiq.SimpleSparkDatasetBuilder.datasets(training_features,
validation_features,
testing_features,
label, cat_col, cont_col, id_col, date_col,
convert_date_cols, datetime_format,
name)
where the parameters are as described in the previous section.
NB: Both features and targets have to be defined in the same spark dataframe.
In order to create a bias dataset from a pandas dataframe the following function:
etiq.BiasDatasetBuilder.dataset(features, target,
label, cat_col, cont_col, id_col, date_col,
convert_date_cols, datetime_format,
train_valid_test_splits,
bias_params,
random_seed,
name)
can be used. where
- bias_param - a named tuple specifying the bias meta data.
The other parameters are as described for the simple dataset.
The corresponding function for creating a bias dataset where training, validation and testing dataframes (at least one needs to be specified) is
etiq.BiasDatasetBuilder.datasets(training_features, training_target,
validation_features, validation_target,
testing_features, testing_target,
label, cat_col, cont_col, id_col, date_col,
convert_date_cols, datetime_format,
bias_params,
name)
In order to create a bias dataset from a pandas dataframe the following function:
etiq.BiasSparkDatasetBuilder.dataset(features, target,
label, cat_col, cont_col, id_col, date_col,
convert_date_cols, datetime_format,
train_valid_test_splits,
bias_params,
random_seed,
name)
can be used. where
- bias_param - a named tuple specifying the bias meta data.
The other parameters are as described for the simple dataset.
The corresponding function for creating a bias dataset where training, validation and testing dataframes (at least one needs to be specified) is
etiq.BiasSparkDatasetBuilder.datasets(training_features,
validation_features,
testing_features,
label, cat_col, cont_col, id_col, date_col,
convert_date_cols, datetime_format,
bias_params,
name)
You can use any already trained model from the supported libraries: XGBoost, LightGBM, PyTorch, TensorFlow, Keras and scikit-learn. It should be compatible with any model that uses the sklearn fit/predict convention.
For example purposes, we also provide out-of-the box model architectures for some model types:
DefaultXGBoostClassifier (a wrapper around XGBoost classifier), DefaultRandomForestClassifier (a wrapper around the random forest classifier from sklearn) and DefaultLogisticRegression (a wrapper around the logistic regression classifier from sklearn). However, most use cases will use own fit model or pre-calculated model
# Load the dataset
dataset = etiq.BiasDatasetBuilder.datasets(training_features=test,
validation_features=valid)
# Load the bias parameters
bias_params = etiq.BiasDatasetBuilder.bias_params()
# Create your already trained model and log it.
model = Model(model_architecture=standard_model, model_fitted=model_fit)
# Create a Snapshot
snapshot = project.snapshots.create(name="<snapshot-name>",
dataset=dataset,
model=model,
bias_params=bias_params)
Call a wrap around model from Etiq library using the following syntax. For a notebook example, go here
#Log your dataset
dataset = etiq.BiasDatasetBuilder.dataset(data_encoded, label="<target-feature-name>")
# Load our model
from etiq.model import DefaultXGBoostClassifier
model = DefaultXGBoostClassifier()
# Creating a snapshot
snapshot = project.snapshots.create(name="<snapshot-name>", dataset=dataset, model=model, bias_params=etiq.BiasDatasetBuilder.bias_params())
Sometimes it is not possible to directly use a machine learning model in python for various reasons. However we still want to evaluate the model performance. In order to accommodate such a usecase we make available a "pre-calculation" model. This simply contains the prediction labels for the desired model we would like to evaluate on a dataset.
An example of how to use such a model is provided below.
# This bit is done by someone with access to the model
train, valid = train_test_split(data_encoded, test_size=0.2, random_state=17)
train.reset_index(inplace=True, drop=True)
valid.reset_index(inplace=True, drop=True)
# Split data then train the model
y_train = train['income'].copy() # labels we're going to train the model to predict
x_train = train.drop(columns=['income'])
y_valid = valid['income'].copy()
x_valid = valid.drop(columns=['income'])
# train a model to predict 'income'
standard_model = MyCustomModel()
model_fit = standard_model.fit(x_train, y_train)
y_train_pred = standard_model.predict(x_train)
y_valid_pred = standard_model.predict(x_valid)
model_df = data_encoded.copy()
model_df['predicted'] = standard_model.predict(data_encoded.drop(columns=['income'])
model_df.to_csv('data-with-predictions.csv', index=False)
# The precalculated csv file is then provided.
# This bit can be run by someone without access to the model
import pandas as pd
import etiq
labelled_data_with_predictions = pd.read_csv('data-with-predictions.csv')
labelled_data_without_predictions = labelled_data_with_predictions.drop(
['predicted'], axis=1)
precalc_model = etiq.model.PrecalculatedModel(labelled_data_with_prediction,
prediction_label='predicted')
dataset = etiq.SimpleDatasetBuilder.dataset(labelled_data_without_predictions,
label="income")
# Creating a snapshot
snapshot = project.snapshots.create(name="<snapshot-name>",
dataset=dataset,
model=precalc-model)
# Run model performance scans
Last modified 2mo ago