Agentic Workflows

When running etiq with agentic workflows, it will automatically capture any agent objects and the lineage of all unstructured data it uses or produces. By default we support pydantic-ai, and langchain agents.

Here is a example agent workflow, ran with etiq:

from etiq_copilot.engine.implementations.scanner import DebuggerCodeScanner

src = """
import pandas as pd
from pydantic_ai import Agent
from pydantic_ai.models.function import FunctionModel, AgentInfo
from pydantic_ai import ModelMessage, TextPart, models
from pydantic_ai import ModelResponse
from random import random

models.ALLOW_MODEL_REQUESTS = False


def mock_model_call(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    return ModelResponse(parts=[TextPart(f"Hello etiq! {random()}")])


def query(agent: Agent, df: pd.DataFrame):
    df_summary = df.describe(include="all").to_string()
    prompt = (
        f"Here is a summary of the dataset:{df_summary}Please provide a brief insight."
    )
    result = agent.run_sync(prompt)
    return result.output


raw_data = {
    "date": ["2024-01-01", "2024-01-02", "2024-01-03"],
    "value": [10, 15, 25],
}
df = pd.DataFrame(raw_data)
model = FunctionModel(function=mock_model_call)

agent = Agent(
    model=model,
    system_prompt="You are a helpful data assistant.",
)

response = query(agent, df)

print(response)"""

scanner = DebuggerCodeScanner()
res = scanner.scan_code(src)
print(res.create_full_lineage_graph())

This will print out the lineage graph in the dot format, which can be used with any graphviz dot visualiser such as: https://dreampuf.github.io/GraphvizOnline/

Lineage Graph

The lineage from the above example looks like so:

We have yellow circles showing any dataframe states, green circles showing the captured agent, and purple circles showing the unstructured states. The blue diamonds represent function calls or assignments done with these states. Lastly the large blue square shows that all the states inside were captured in our defined query function.

Adding Other Agent Libraries

As mentioned, by default we will capture any pydantic-ai or langchain agents, however if these aren't your cup of tea you can register any other agent framework yourself!

To add your own agent library, you will need to edit eitq's parser config, which contains a list of all registered object types to capture in the lineage. When adding your own agent library, please be sure to only add the agent type and not any primative types, such as strings, lists, etc. Doing so will lead to capturing nearly everything in code, and producing a very large lineage graph.

When adding an agent type, etiq uses the "qualified name" rather than the traditional typing. We have included a utility function get_qname_from_object which will take any live object, and return its qualified name.

Here is an example, adding Google's ADK as a registered type:

from etiq_copilot.engine.implementations.scanner import AstParser
from etiq_copilot.engine.implementations.scanner.helpers.astroid_utils import get_qname_from_object

from google.adk.agents import Agent

my_agent = Agent(name = "my_agent")

qualified_name = get_qname_from_object(my_agent)

parser = AstParser("")

parser.register_type(qualified_name)
parser.save_config()

The registered types can be seen with parser.types_to_capture and can be removed with parser.remove_type(...) After saving your changes with parser.save_config() the changes will be reflected in the next code scan.