Quickstart

Requirements

Currently etiq-copilot is only avaliable on Windows and Linux, due to build issues on MacOS

Python 3.10 - 3.13

We currently support the VSCode IDE and Jupyter Notebooks with our extension

Installation

In order to use the Etiq you need to install the Python package to your local environment.

Python Package

Install the etiq-copilot python package from PyPi:

pip install etiq-copilot

Runnable examples: See the Etiq public example repository for example Python scripts that can be executed and scanned locally.

Usage

Use DebuggerCodeScanner to run a python file under Etiq's instrumentation. Pass the target source code to scan_code; Etiq executes the code, captures the runtime trace and observed objects, and returns a CodeScannerResult that can be used to inspect lineage outputs.

from pathlib import Path

from etiq_copilot.engine.implementations.scanner.code_scanner import DebuggerCodeScanner
from etiq_copilot.engine.implementations.scanner.scan_results import CodeScannerResult


def scan_file(scan_file_path: Path | str) -> CodeScannerResult:
    scan_file_path = Path(scan_file_path)
    original_code = scan_file_path.read_text(encoding="utf-8")
    scanner = DebuggerCodeScanner()
    return scanner.scan_code(code_str=original_code)

Example usage:

scan_results = scan_file("test_repo/iris_lineage_test.py")

Example Target Script

The quickstart uses this iris pipeline as the target script:

from sklearn import datasets
import pandas as pd


iris = datasets.load_iris()

iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)
iris_df["species_id"] = iris.target

species_lookup_df = pd.DataFrame(
    {
        "species_id": range(len(iris.target_names)),
        "species": iris.target_names,
    }
)

iris_with_species_df = iris_df.merge(species_lookup_df, on="species_id", how="left")

measurement_columns = iris.feature_names
clean_measurements_df = iris_with_species_df.dropna(subset=measurement_columns).copy()

# Deliberate bad intermediate for verification examples. This object is empty
# and should be visible to Etiq, while the final report continues from the
# correct cleaned dataframe.
deliberate_empty_features = clean_measurements_df[
    clean_measurements_df["species"] == "not-a-real-species"
][measurement_columns].copy()

wide_petal_df = clean_measurements_df[
    clean_measurements_df["petal length (cm)"] >= 4.0
].copy()

wide_petal_df["petal_area"] = (
    wide_petal_df["petal length (cm)"] * wide_petal_df["petal width (cm)"]
)

species_summary_df = (
    wide_petal_df.groupby("species", as_index=False)
    .agg(
        flower_count=("species_id", "count"),
        avg_petal_length=("petal length (cm)", "mean"),
        avg_petal_area=("petal_area", "mean"),
    )
    .sort_values("avg_petal_area", ascending=False)
)

final_report_df = species_summary_df.assign(
    rank=range(1, len(species_summary_df) + 1),
    source_rows=len(wide_petal_df),
)

Example Scan Output

After scanning the iris pipeline, inspect the scan result:

lineage_json = scan_results.create_full_lineage_graph(graph_format="json")

print("scan_errors:", scan_results.scan_errors)
print("dataframes:", scan_results.list_dataframes())
print("models:", scan_results.list_models())
print("agents:", scan_results.list_agents())
print("lineage_json:", lineage_json[:80] + "...")

Example output:

scan_errors: None
dataframes: ['iris_df', 'species_summary_df', 'iris_with_species_df', 'species_lookup_df', 'deliberate_empty_features', 'wide_petal_df', 'clean_measurements_df', 'final_report_df']
models: []
agents: []
lineage_json: {"objects": [{"style": "filled", "fillcolor": "#FFE18E", "shape": "circle", "fon...

The full lineage_json value contains the generated graph. A shortened excerpt looks like this:

{
  "objects": [
    {
      "label": "preds",
      "shape": "circle",
      "fillcolor": "#FFE18E"
    },
    {
      "label": "amodel.predict",
      "shape": "diamond",
      "fillcolor": "#46A0FF"
    }
  ],
  "edges": [
    {
      "tail": 5,
      "head": 2
    },
    {
      "tail": 2,
      "head": 1
    }
  ]
}

The full lineage graph can be exported with create_full_lineage_graph(graph_format="json"). Generated node IDs can differ between runs.

Example Lineage Graph