Core Concepts

Lineage Graph

The lineage graph is the derived graph showing relationships between functions and lineage objects. It is based on both static analysis and runtime tracing.

The default format is dot. Use json when another program or agent needs to consume the graph output.

Lineage Object Nodes

Lineage object nodes are the common concept for objects captured by Etiq and represented in the lineage graph. Today this includes dataset/dataframe lineage objects and model lineage objects. The lineage object concept is intentionally broader because Etiq will support more object types over time.

Dataset Lineage Objects

Dataset lineage objects are dataframe-like objects captured during the run.

Model Lineage Objects

Model lineage objects are model states captured during the run.

Agent Lineage Objects

Agent lineage objects are agent states captured during the run.

Unstructured Lineage Objects

Unstructured lineage objects are any states captured which don't fit into the above categories, such as strings or other arbitrary types.

Code Nodes And Source

Each captured state stores a source node. The node points back to the code Etiq associated with that captured object.

Use the node when you need source-level evidence, such as:

• the code snippet for a captured object

• whether the object was captured at module level or inside a function

• the scope that contains the captured object

Example captured state:

{
    "state_type": "DataframeState",
    "names": {"adf"},
    "line_no": 4,
    "value_type": "DataFrame",
    "node_type": "FunctionDef",
    "node_source": """
def add_one(adf):
    new_df = adf + 1
    return new_df
""",
    "scope_type": "FunctionDef",
    "scope_repr_first_line": "FunctionDef(",
}