# Welcome to Etiq ## Etiq Core Etiq Core is a in-depth observability framework for agent, copilot or human-created code. It builds a causal graph of your code by tracing lineage objects back to the functions that produced them, so developers and agents can verify outputs, catch coding mistakes, and reduce hallucinations. * deterministic lineage built from code and execution artifacts, not LLM-generated claims * runtime execution and graph generation, going beyond static analysis * low tracing overhead * no manual instrumentation * support for data, ML/AI pipelines, agent builds, and large codebases * deployment through library functions, VS Code extension, and Jupyter Notebook extension For coding agents, Etiq provides a causal view of what actually happened when agents modify code, run tools, and generate artifacts across multiple steps. It does this without requiring the user or the agent to manually instrument the code. ### Benefits :arrow\_forward: *In-depth verification* :arrow\_forward: *Deterministic observability and auditability inside agent-created code* :arrow\_forward: *Helps instrument your tests deterministically on copilot/agent code* :arrow\_forward: *Debugging improvements and better performing agents* ### Where does Etiq sit in the observability stack? Coding-agent observability usually has three parts: what the agent intended, what the platform observed, and what the generated code actually did. Etiq focuses on the third part. ### Observability Layers | Layer | What it shows | Typical tools or records | | ----------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------ | | Orchestration and state | What the agent planned, which steps it took, and what final outputs it stored | agent state, memory, tool records, artifact stores | | Runtime telemetry | What the platform observed during execution | traces, logs, metrics, spans, HTTP calls, subprocesses, exit codes | | Code execution lineage | Which functions produced or transformed interim lineage objects | Etiq lineage graph, captured dataframes, models, agent states | Etiq answers questions the other layers usually cannot: which function produced this dataframe, model, or output; which interim lineage object was wrong; and what should be tested before trusting the final result. ### Why Etiq Adds A Separate Layer Agent orchestration can show what the agent planned and which tools it invoked. Runtime telemetry can show that code ran and which external calls happened. Etiq fills the gap inside the executed code by tracing interim lineage objects and their producer functions. That makes Etiq useful for: * granular verification of data and AI pipelines * targeted debugging when an interim step is wrong * auditability across the lineage of data, models, and generated outputs * coding agents with longer task horizons and multi-step code execution Etiq does not replace orchestration, memory, artifact storage, or OpenTelemetry. It adds causal traceability for the code the agent runs. ### Supported Stack Supported stack areas include: * Python * Spark * SQL The core workflow is: 1. Point Etiq at a python entry file. 2. Scan the code. 3. Retrieve lineage and captured states. 4. Inspect datasets, models, agents, and graph output. ### Pages * [Quickstart](/quickstart) * [Use Cases](/use-cases) * [Scan Outputs](/working-with-scan-results) * [Agent Instruction](/agent-instruction) # Quickstart ## Requirements {% hint style="danger" %} Currently etiq-copilot is only avaliable on Windows and Linux, due to build issues on MacOS {% endhint %} Python 3.10 - 3.13 We currently support the VSCode IDE and Jupyter Notebooks with [our extension](/etiq-extension) ## Installation In order to use the Etiq you need to install the Python package to your local environment. ### Python Package Install the `etiq-copilot` python package from PyPi: `pip install etiq-copilot` ### Usage Use `DebuggerCodeScanner` to run a python file under Etiq's instrumentation. Pass the target source code to `scan_code`; Etiq executes the code, captures the runtime trace and observed objects, and returns a `CodeScannerResult` that can be used to inspect lineage outputs. ```python from pathlib import Path from etiq_copilot.engine.implementations.scanner.code_scanner import DebuggerCodeScanner from etiq_copilot.engine.implementations.scanner.scan_results import CodeScannerResult def scan_file(scan_file_path: Path | str) -> CodeScannerResult: scan_file_path = Path(scan_file_path) original_code = scan_file_path.read_text(encoding="utf-8") scanner = DebuggerCodeScanner() return scanner.scan_code(code_str=original_code) ``` Example usage: ```python scan_results = scan_file("test_repo/iris_lineage_test.py") ``` ### Example Target Script The quickstart uses this iris pipeline as the target script: ```python from sklearn import datasets import sklearn.model_selection from sklearn.ensemble import RandomForestClassifier import pandas as pd empty_dataframe = pd.DataFrame(columns=["a", "b"]) iris = datasets.load_iris() iris_df = pd.DataFrame(iris.data, columns=iris.feature_names) iris_df["target"] = iris.target iris_train_df, iris_test_df = sklearn.model_selection.train_test_split( iris_df, test_size=0.2, random_state=31779, ) amodel = RandomForestClassifier(random_state=0) iris_training_features = iris_train_df[iris.feature_names].copy() iris_test_features = iris_test_df[iris.feature_names].copy() iris_target_training = iris_train_df["target"].copy() amodel.fit(iris_training_features, iris_target_training) iris_target_testing = iris_test_df["target"].copy() preds = amodel.predict(iris_test_features) ``` ### Example Scan Output After scanning the iris pipeline, inspect the scan result: ```python lineage_json = scan_results.create_full_lineage_graph(graph_format="json") print("scan_errors:", scan_results.scan_errors) print("dataframes:", scan_results.list_dataframes()) print("models:", scan_results.list_models()) print("agents:", scan_results.list_agents()) print("lineage_json:", lineage_json[:80] + "...") ``` Example output: ``` scan_errors: None dataframes: ['iris_target_testing', 'iris_df', 'iris_training_features', 'preds', 'empty_dataframe', 'iris_target_training', 'iris_train_df', 'iris_test_features', 'iris_test_df'] models: ['amodel'] agents: [] lineage_json: {"objects": [{"style": "filled", "fillcolor": "#FFE18E", "shape": "circle"... ``` The full `lineage_json` value contains the generated graph. A shortened excerpt looks like this: ```json { "objects": [ { "label": "preds", "shape": "circle", "fillcolor": "#FFE18E" }, { "label": "amodel.predict", "shape": "diamond", "fillcolor": "#46A0FF" } ], "edges": [ { "tail": 5, "head": 2 }, { "tail": 2, "head": 1 } ] } ``` The full lineage graph can be exported with `create_full_lineage_graph(graph_format="json")`. Generated node IDs can differ between runs. ### Example Lineage Graph
# Agentic Workflows When running etiq with agentic workflows, it will automatically capture any agent objects and the lineage of all unstructured data it uses or produces. By default we support `pydantic-ai`, and `langchain` agents. Here is a example agent workflow, ran with etiq: ```python from etiq_copilot.engine.implementations.scanner import DebuggerCodeScanner src = """ import pandas as pd from pydantic_ai import Agent from pydantic_ai.models.function import FunctionModel, AgentInfo from pydantic_ai import ModelMessage, TextPart, models from pydantic_ai import ModelResponse from random import random models.ALLOW_MODEL_REQUESTS = False def mock_model_call(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse: return ModelResponse(parts=[TextPart(f"Hello etiq! {random()}")]) def query(agent: Agent, df: pd.DataFrame): df_summary = df.describe(include="all").to_string() prompt = ( f"Here is a summary of the dataset:{df_summary}Please provide a brief insight." ) result = agent.run_sync(prompt) return result.output raw_data = { "date": ["2024-01-01", "2024-01-02", "2024-01-03"], "value": [10, 15, 25], } df = pd.DataFrame(raw_data) model = FunctionModel(function=mock_model_call) agent = Agent( model=model, system_prompt="You are a helpful data assistant.", ) response = query(agent, df) print(response)""" scanner = DebuggerCodeScanner() res = scanner.scan_code(src) print(res.create_full_lineage_graph()) ``` This will print out the lineage graph in the **dot format**, which can be used with any graphviz dot visualiser such as: ### Lineage Graph The lineage from the above example looks like so:
We have **yellow circles** showing any dataframe states, **green circles** showing the captured agent, and **purple circles** showing the unstructured states. The **blue diamonds** represent function calls or assignments done with these states. Lastly the large **blue square** shows that all the states inside were captured in our defined query function. ### Adding Other Agent Libraries As mentioned, by default we will capture any pydantic-ai or langchain agents, however if these aren't your cup of tea you can register any other agent framework yourself! To add your own agent library, you will need to edit eitq's parser config, which contains a list of all registered object types to capture in the lineage. When adding your own agent library, please be sure to **only** add the agent type and not any primative types, such as strings, lists, etc. Doing so will lead to capturing nearly everything in code, and producing a very large lineage graph. When adding an agent type, etiq uses the "qualified name" rather than the traditional typing. We have included a utility function `get_qname_from_object` which will take any live object, and return its qualified name. Here is an example, adding Google's ADK as a registered type: ```python from etiq_copilot.engine.implementations.scanner import AstParser from etiq_copilot.engine.implementations.scanner.helpers.astroid_utils import get_qname_from_object from google.adk.agents import Agent my_agent = Agent(name = "my_agent") qualified_name = get_qname_from_object(my_agent) parser = AstParser("") parser.register_type(qualified_name) parser.save_config() ``` The registered types can be seen with `parser.types_to_capture` and can be removed with `parser.remove_type(...)` \ After saving your changes with `parser.save_config()` the changes will be reflected in the next code scan. # Core Concepts ### Lineage Graph The lineage graph is the derived graph showing relationships between functions and lineage objects. It is based on both static analysis and runtime tracing. The default format is `dot`. Use `json` when another program or agent needs to consume the graph output. ### Lineage Object Nodes Lineage object nodes are the common concept for objects captured by Etiq and represented in the lineage graph. Today this includes dataset/dataframe lineage objects and model lineage objects. The lineage object concept is intentionally broader because Etiq will support more object types over time. #### Dataset Lineage Objects Dataset lineage objects are dataframe-like objects captured during the run. #### Model Lineage Objects Model lineage objects are model states captured during the run. #### Agent Lineage Objects Agent lineage objects are agent states captured during the run. #### Unstructured Lineage Objects Unstructured lineage objects are any states captured which don't fit into the above categories, such as strings or other arbitrary types. ### Code Nodes And Source Each captured state stores a source node. The node points back to the code Etiq associated with that captured object. Use the node when you need source-level evidence, such as: * the code snippet for a captured object * whether the object was captured at module level or inside a function * the scope that contains the captured object Example captured state: ```python { "state_type": "DataframeState", "names": {"adf"}, "line_no": 4, "value_type": "DataFrame", "node_type": "FunctionDef", "node_source": """ def add_one(adf): new_df = adf + 1 return new_df """, "scope_type": "FunctionDef", "scope_repr_first_line": "FunctionDef(", } ``` # Working with Scan Results ## Scan Outputs `CodeScannerResult` is the main object returned by `DebuggerCodeScanner().scan_code(...)`. It stores the captured states, lineage graph outputs, scan errors, and source nodes Etiq uses to build lineage. This page uses a small dataframe example throughout. ```python from etiq_copilot.engine.implementations.scanner import DebuggerCodeScanner src = """import pandas as pd df = pd.DataFrame([1]) def add_one(adf): new_df = adf + 1 return new_df df2 = add_one(df) """ scanner = DebuggerCodeScanner() result = scanner.scan_code(src) ``` ### What The Scan Captures Every captured object has a state. Etiq uses the state to build lineage, including parent/child relationships and function mappings. The state also stores the Astroid node for the captured code location. For this example, Etiq captures four dataframe states: | Captured name | Source line | Captured node | | ------------- | ----------: | ------------------------ | | `df` | 2 | `df = pd.DataFrame([1])` | | `adf` | 4 | `def add_one(adf): ...` | | `new_df` | 5 | `new_df = adf + 1` | | `df2` | 8 | `df2 = add_one(df)` | You can inspect the raw state store with `result.values`: ```python for state in result.values: print(state.names, state.line_no, state.node.as_string()) ``` You can also use the `CodeScannerResult` methods below: ### Lineage Graph Use `create_full_lineage_graph()` to generate the lineage graph. ```python lineage_graph_dot = result.create_full_lineage_graph() lineage_graph_json = result.create_full_lineage_graph(graph_format="json") ``` What you get: | Call | Output | | ------------------------------------------------------- | ----------------- | | `result.create_full_lineage_graph()` | DOT graph string | | `result.create_full_lineage_graph(graph_format="json")` | JSON graph string | For this example, both outputs are strings. The exact string length and generated node IDs can differ between runs. ### Dataset Lineage Objects Use `list_dataframes()` when you only need the names of captured dataframe lineage objects. ```python result.list_dataframes() ``` Example output: ``` ['new_df', 'adf', 'df2', 'df'] ``` Use `get_dataframes()` when you need the state objects. ```python dataframe_states = result.get_dataframes() ``` For this example, `get_dataframes()` returns four dataframe states: `df`, `adf`, `new_df`, and `df2`. ### Model Lineage Objects Use `list_models()` and `get_models()` for captured model lineage objects. ```python result.list_models() result.get_models() ``` ### Agent States Use `list_agents()` and `get_agent_states()` for captured agent states. ```python result.list_agents() result.get_agent_states() ``` ### Unstructured States Use `get_unstructured_states()` for captured states that do not fit a more specific lineage object category. ```python result.get_unstructured_states() ``` ### Paths Between States Use `get_shortest_path(parent_node, child_node)` to inspect a lineage path between two captured data states. ```python df_state = result.get_dataframes()[0] adf_state = result.get_dataframes()[1] path = result.get_shortest_path(df_state, adf_state) ``` For this example, the returned path connects the function argument state back to the original dataframe state: ``` adf -> df ``` ### Scan Errors Use `scan_errors` to inspect scan errors before relying on downstream outputs. ```python result.scan_errors ``` ### Source Nodes And Scope Each captured state stores a source node. The node points back to the code Etiq associated with that captured object. Use the node when you need to answer source-level questions: * where the captured object came from, such as line number and scope * the source snippet Etiq associated with the captured object For most lineage workflows, use the `CodeScannerResult` methods above. The node is mainly useful for source evidence and debugging. #### Module In the example, `df` is created at the top level of the script: ```python df = pd.DataFrame([1]) ``` After the scan, find the captured state for `df` and inspect its source node: ```python df_state = next(state for state in result.values if "df" in state.names) df_node = df_state.node print("name:", df_state.names) print("line:", df_state.line_no) print("source:", df_node.as_string()) print("scope:", type(df_node.scope()).__name__) ``` Output: ``` name: {'df'} line: 2 source: df = pd.DataFrame([1]) scope: Module ``` `Module` means the captured object came from the outermost script scope, not from inside a function. #### Function-local object ```python new_df_state = next( state for state in result.values if "new_df" in state.names ) new_df_node = new_df_state.node print("name:", new_df_state.names) print("source:", new_df_node.as_string()) print("scope:", type(new_df_node.scope()).__name__) ``` Example output: ``` name: {'new_df'} source: new_df = adf + 1 scope: FunctionDef ``` `new_df` is created inside `add_one`, so `scope()` returns `FunctionDef`. Use `node.as_string()` for the source snippet. Use `node.scope()` when you need to know whether the captured object came from module-level code, a function body, or another scope. ### Scanning a Codebase For a codebase, scan the entry file that starts the run. The entry file can import and call functions from other local files. Etiq executes the entry file and captures lineage objects produced along that execution path. Example project: ``` project/ pipeline.py transforms.py ``` `transforms.py` contains a helper function: ```python def add_one(adf): new_df = adf + 1 new_df2 = new_df + 10 return new_df2 ``` `pipeline.py` is the entry file: ```python import pandas as pd from transforms import add_one df = pd.DataFrame([1]) df2 = add_one(df) ``` Scan `pipeline.py`. You do not need to scan `transforms.py` separately; it is called by the entry file during execution. ```python from pathlib import Path from etiq_copilot.engine.implementations.scanner import DebuggerCodeScanner def scan_file(scan_file_path: Path | str): scan_file_path = Path(scan_file_path) source = scan_file_path.read_text(encoding="utf-8") scanner = DebuggerCodeScanner() return scanner.scan_code(code_str=source) result = scan_file("project/pipeline.py") ``` Run this from the project root so local imports such as `from transforms import add_one` resolve normally. Use the entry file for the workflow you want to observe. Any imported code that runs as part of that workflow is part of the execution path Etiq observes. Example output from scanning `pipeline.py`: ``` scan_errors: None dataframes: ['adf', 'df', 'new_df', 'new_df2', 'df2'] models: [] agents: [] states: - ['df'] line 5 node Assign source df = pd.DataFrame([1]) - ['adf'] line 1 node FunctionDef source def add_one(adf): ... - ['new_df'] line 2 node Assign source new_df = adf + 1 - ['new_df2'] line 3 node Assign source new_df2 = new_df + 10 - ['df2'] line 6 node Assign source df2 = add_one(df) ``` This shows Etiq capturing lineage objects from the entry file and from the imported function that ran during the entry file's execution. In particular, Etiq captures both `new_df` and `new_df2`, even though they are created inside `add_one` in `transforms.py`. # Agent Instruction This page gives AI agents the minimum operating instructions needed to use Etiq against a python script. ### Use This File When Use this file when an agent needs to: * scan a python script with Etiq * generate a lineage graph * inspect captured lineage objects * return a structured summary of scan outputs ### Purpose Use Etiq to scan a python file and it's run, generate lineage, and retrieve captured lineage objects such as dataframes, models, unstructured data, and agent states. ### Project Context * Product name: Etiq * Install package: `etiq-copilot` * python import namespace: `etiq_copilot` * Main scan result object: `CodeScannerResult` from `DebuggerCodeScanner().scan_code(source_code)` * Example target file: `test_repo/iris_pipeline.py` * Related docs: * Quickstart * Core Concepts * Scan Outputs * Agentic Workflows ### Setup Commands Install with `pip`: ```bash pip install etiq-copilot ``` Install with `uv`: ```bash uv pip install etiq-copilot ``` ### Files That Matter * `docs/etiq/quickstart.md`: scanner setup and minimal file scan example * `docs/etiq/core-concepts.md`: lineage graph and lineage object concepts * `docs/etiq/scan-results.md`: `CodeScannerResult` methods, captured states, source nodes, and larger-codebase entry-file scans * `test_repo/iris_pipeline.py`: example target script ### Scan Helper Use this helper to scan a target file and return a `CodeScannerResult`. ```python from pathlib import Path from etiq_copilot.engine.implementations.scanner.code_scanner import DebuggerCodeScanner from etiq_copilot.engine.implementations.scanner.scan_results import CodeScannerResult def scan_file(scan_file_path: Path | str) -> CodeScannerResult: scan_file_path = Path(scan_file_path) original_code = scan_file_path.read_text(encoding="utf-8") scanner = DebuggerCodeScanner() return scanner.scan_code(code_str=original_code) ``` ### Minimum Working Routine ```python target_file = "test_repo/iris_pipeline.py" scan_results = scan_file(target_file) result = { "target_file": target_file, "scan_errors": scan_results.scan_errors, "lineage_objects": { "datasets": scan_results.list_dataframes(), "models": scan_results.list_models(), "agents": scan_results.list_agents(), }, "source_evidence": [ { "names": sorted(state.names), "source": state.node.as_string(), "scope": type(state.node.scope()).__name__, } for state in scan_results.get_dataframes() ], "lineage_graph": { "format": "json", "value": scan_results.create_full_lineage_graph(graph_format="json"), }, } ``` ### Default Workflow 1. Set the target file path explicitly. 2. Scan the entry file with `scan_file(...)`. 3. Read `scan_results.scan_errors`. 4. If `scan_errors` is non-empty, report them before drawing conclusions. 5. Generate lineage with `create_full_lineage_graph(graph_format="json")` when another tool or agent needs to parse the graph. 6. List captured lineage objects with `list_dataframes()`, `list_models()`, and `list_agents()`. 7. Retrieve full state objects only when names or graph output are not enough. 8. Use `state.node.as_string()` and `state.node.scope()` only when source evidence is needed. ```python target_file = "test_repo/iris_pipeline.py" scan_results = scan_file(target_file) scan_errors = scan_results.scan_errors lineage_json = scan_results.create_full_lineage_graph(graph_format="json") lineage_objects = { "datasets": scan_results.list_dataframes(), "models": scan_results.list_models(), "agents": scan_results.list_agents(), } source_evidence = [ { "names": sorted(state.names), "source": state.node.as_string(), "scope": type(state.node.scope()).__name__, } for state in scan_results.get_dataframes() ] ``` ### API Commands Use these public methods and properties. | When you need to | Use | Output | | ------------------------------ | ------------------------------------------------------------- | -------------------------- | | Check scan issues | `scan_results.scan_errors` | Scan error details | | Generate parseable lineage | `scan_results.create_full_lineage_graph(graph_format="json")` | JSON graph string | | Generate visual lineage source | `scan_results.create_full_lineage_graph(graph_format="dot")` | DOT graph string | | List dataset lineage objects | `scan_results.list_dataframes()` | `list[str]` | | Retrieve dataset states | `scan_results.get_dataframes()` | Dataframe state objects | | List model lineage objects | `scan_results.list_models()` | `list[str]` | | Retrieve model states | `scan_results.get_models()` | Model state objects | | List agent states | `scan_results.list_agents()` | `list[str]` | | Retrieve agent states | `scan_results.get_agent_states()` | Agent state objects | | Retrieve other captured states | `scan_results.get_unstructured_states()` | Unstructured state objects | ### Task Recipes #### Summarize A Target Script ```python target_file = "test_repo/iris_pipeline.py" scan_results = scan_file(target_file) dataframe_states = scan_results.get_dataframes() model_states = scan_results.get_models() agent_states = scan_results.get_agent_states() unstructured_states = scan_results.get_unstructured_states() summary = { "target_file": target_file, "scan_errors": scan_results.scan_errors, "dataframes": scan_results.list_dataframes(), "models": scan_results.list_models(), "agents": scan_results.list_agents(), "counts": { "dataframes": len(dataframe_states), "models": len(model_states), "agents": len(agent_states), "unstructured": len(unstructured_states), }, "states": [ { "names": sorted(state.names), "state_type": type(state).__name__, "line_no": state.line_no, "node_type": type(state.node).__name__, "source": state.node.as_string(), "scope": type(state.node.scope()).__name__, } for state in dataframe_states ], } ``` #### Generate Lineage For Another Agent ```python lineage_json = scan_results.create_full_lineage_graph(graph_format="json") ``` Return the graph as a string and state that the graph schema should not be treated as stable until a versioned schema is published. #### Inspect Dataset State Objects ```python for state in scan_results.get_dataframes(): print(state) ``` Use state objects when the agent needs captured values, source evidence, or Etiq metadata for a lineage object. #### Inspect Source Evidence ```python for state in scan_results.get_dataframes(): print("names:", state.names) print("source:", state.node.as_string()) print("scope:", type(state.node.scope()).__name__) ``` Use this when the user asks where a lineage object came from in the code. #### Inspect Model State Objects ```python for state in scan_results.get_models(): print(state) ``` Use this when the user asks which models were created, used, or captured. ### Decision Rules * If the user asks for lineage output that another tool will parse, use `graph_format="json"`. * If the user asks for visualization-oriented output, use `graph_format="dot"`. * If `scan_errors` is non-empty, report the errors before summarizing lineage. * If names from `list_dataframes()`, `list_models()`, or `list_agents()` are sufficient, do not retrieve full state objects. * If the target workflow spans multiple files, scan the entry file that starts the run. * If source evidence or captured values are needed, retrieve state objects with the corresponding `get_*` method and inspect `state.node`. * If the task requires installing Etiq, ask for approval before running an install command. * If the task requires deleting files, changing public APIs, or modifying generated output, ask before proceeding. ### Do Not * Do not rely on private internals. * Do not assume all lineage objects are dataframes or models forever. * Do not assume graph JSON has a stable schema until that schema is published. * Do not ignore `scan_errors`. * Do not install dependencies without approval. * Do not reformat unrelated files when editing docs or examples. ### Completion Criteria Before finishing an agent task, report: * target file scanned * scan errors, or that no scan errors were reported * lineage object names found * graph format generated, if any * commands run * checks skipped and why * known limitations or follow-up Use this response shape when returning structured results: ```python result = { "target_file": target_file, "scan_errors": scan_results.scan_errors, "lineage_objects": { "datasets": scan_results.list_dataframes(), "models": scan_results.list_models(), "agents": scan_results.list_agents(), }, "source_evidence": [ { "names": sorted(state.names), "source": state.node.as_string(), "scope": type(state.node.scope()).__name__, } for state in scan_results.get_dataframes() ], "lineage_graph": { "format": "json", "value": lineage_json, }, } ``` # Use Cases ## **Testing** Agent-generated code can pass a final-output check while still containing incorrect intermediate logic. A data or analytics agent may use the wrong input, apply an incorrect filter, introduce leakage, or perform the wrong join or aggregation - and still produce a plausible result. Etiq creates testable boundaries throughout the execution by linking interim lineage objects to the functions that produced them. Tests and verification checks can therefore be applied to the relevant function-and-artifact pairs, rather than only to the final output or the complete generated script. Use Etiq to: * test intermediate datasets, models, agent states, and generated artifacts * verify the inputs and outputs of individual pipeline stages * target tests at the parts of the execution affected by an agent’s changes * compare expected and observed behaviour across repeated runs * add deterministic checks without relying on the agent to instrument or describe its own code **Benefit**: More granular testing of agent-created code, with failures tied to the execution step and state that produced them rather than reported only at the end of the workflow. Etiq’s function-and-artifact lineage is specifically intended to support test harnesses and checks on interim outputs that conventional agent traces do not expose. ## **Debugging** Logs and agent traces can show that a script ran, a tool was called, or a task failed. They do not always reveal which transformation first made the underlying result incorrect. Etiq traces outputs backwards through the functions and interim lineage objects on which they depend. Developers and agents can begin with a suspect result, inspect its producer function, and follow the relevant dependency path upstream until they identify the earliest meaningful divergence. Use Etiq to: * identify the function that produced a wrong interim result * distinguish a bad input from a function that transformed a valid input incorrectly * inspect captured inputs and outputs at the point of failure * expand into nested functions only where additional detail is required * limit reruns and repairs to the affected branch of the workflow **Benefit**: Faster and more targeted root-cause analysis. Instead of searching the entire repository, transcript, or run history, the reviewer can focus on the execution path that contributed to the failed output. The graph can point to the producer of an incorrect interim artifact, while nested-function drill-down allows inspection to start at a meaningful stage and deepen only along the suspect branch. ## **Observability and Governance** Agent orchestration records what an agent planned and which tools it invoked. Runtime telemetry records events around the execution, such as model calls, subprocesses, timings, logs, and exit codes. These layers remain important, but they do not necessarily explain what happened inside the generated code. Etiq adds a separate code-execution-lineage layer. It records the functions that ran, the lineage objects they consumed and produced, and the dependencies connecting those states to the final result. Use Etiq to: * trace a generated output back through its underlying code and data lineage * inspect the interim data, model, or agent states associated with a run * establish which function produced or changed a governed artifact * preserve deterministic evidence independently of the agent’s own explanation * connect review decisions, tests, and approvals to the execution evidence they used * provide reviewers with a navigable record of how an output was produced **Benefit**: Stronger auditability and governance for agent-created code. Teams can retain evidence of the transformations behind an output, rather than relying only on the agent’s plan, a final artifact, or platform-level telemetry. This complements orchestration and OpenTelemetry rather than replacing them: those layers describe the agent and surrounding runtime, while Etiq provides granular visibility into the code execution itself. ## **Long-Horizon Agents** Long-horizon coding agents work across extended sequences of planning, code generation, execution, review, delegation, retries, and resumption. As the task grows, its state becomes distributed across functions, artifacts, agent calls, retries, and points in time. It can no longer be reconstructed reliably from the latest message or a summary of the conversation. Etiq provides an execution-grounded state and memory layer for these workflows. It records what actually ran, which inputs and outputs were observed, what was produced, and which later results depend on earlier execution. The etiq graph can also support more reliable context selection. Instead of repeatedly passing the full conversation, complete execution history, or every available artifact to the agent, a workflow can retrieve the graph region relevant to the current decision. While semantic state records the objective, plan, constraints, decisions, and rationale, the execution state supplied by the etiq graph records the functions that ran, the inputs they consumed, the outputs they produced, and their dependencies. The etiq execution graph can help constitute a form of shared state and memory for these more complex agents. The complete history of a long-running workflow may contain more detail than the agent needs for any one decision. Etiq can help a context builder select a bounded graph region based on the current operation. Below example agent operations on the etiq graph:
OperationPurpose
PlanningRetrieve trusted upstream outputs, relevant prior evidence, and known gaps before deciding what to do next.
ReviewCompare observed execution with the task objective and acceptance criteria.
RetraceFollow a failed or suspect result backwards to the earliest meaningful divergence.
RepairChange or rerun the affected branch from the last trusted upstream boundary.
ResumeContinue from the latest reviewed state without repeating trusted work.
MemoryRetrieve comparable failures and previously reviewed repairs with their supporting evidence.
Each operation can request a different bounded view of the graph. Review may need a stage and its artifacts, while retrace may need a deeper dependency path and resume may need only the latest trusted boundary and unresolved downstream work. **Benefits:** * More reliable context selection by retrieving only the graph region relevant to the current decision * Targeted failure recovery without replaying the full conversation * Smaller rerun scope by preserving trusted upstream and unrelated work * Reduced memory drift by retaining executed state rather than repeatedly summarising it * Inspectable hand-offs between agents or workers through graph-linked artifacts * Checkpoint and resume based on reviewed execution rather than conversational position * Independent verification of what the code actually did, rather than relying on the agent’s own description The graph can therefore operate as more than an audit record. It can provide the shared evidence and bounded retrieval layer from which the agent decides what it can trust, which context it needs, and where it should continue. We will publish our example long horizon harness using the etiq graph shortly.



# Etiq Extension Etiq Extension for VSCode and Jupyter Notebook. ## What is the Etiq Extension Etiq also comes with 2 separate extensions, one for VSCode and the other for Jupyter notebook. The Etiq extension lives directly in your IDE providing your with the tools to make developing and debugging your AI and ML pipelines effortless. Our easy-to-use extension allows you to scan your data and code to build a lineage of your script, our testing recommendation engine analyses your lineage to recommend the most appropriate tests to ensure the code and pipelines you build are robust and fair, and where issues arise, our RCA agents navigate through the lineage to identify where any problems start, diagnose the root cause and provide you a fix so you are back up and running quickly. No disruption. No headaches. No wasted time wrestling with obscure bugs. Just a smarter, smoother way to build AI. ## Key Features ### Lineage The Etiq Extension provides you with full visibility of your entire ML pipeline, so you never get lost again. Analyse your scripts directly in your IDE and visualise the interplay between your data and code giving you insight into its logical flow. Lineage works with your legacy code and models, and supports you as you build new ones. With the ability to navigate through the entire lineage, zooming in and out of nodes and links, and exporting to image files you can ensure you never get lost again when writing complex piplines.

The lineage of a short script

### Testing Recommendations Analysing your lineage, Etiq Extension recommends the most appropriate tests for your script based on your data, code and the interplay between the two. You can then run the tests directly from the Etiq extension giving you full knowledge on the current status of your script and whether you could need to make changes. The tests are configurable to your own requirements, and because Etiq identifies where they're needed to the line number, you can ensure that any pipelines you build are as robust and fair as possible. ### Root Cause Agents Where tests fail, Etiq's Data Science agents are able to use all the information available to them to find the root cause of an issue and diagnose the problem. Combining the complete lineage of the script, the complete understanding of the logical flow of your data and code, with the information from the nature of the test failure, the Data Science agents are able to identify the exact line number in the code where the issue appeared or was introduced and will provide recommended actions to resolve. # Extension How To Quick install: Etiq Extension for VSCode and Jupyter Notebook. Have any questions or looking for support from the team, join our Slack User Community: [![Join the Etiq Slack User Community](/files/fqzWMGaASpF1KA9KZ75n)](https://join.slack.com/t/etiqusercommunity/shared_invite/zt-3q79hvvdz-MS1KqV9V142_2CO_AUiUhg) ## Requirements Python 3.10 - 3.13 We currently support the VSCode IDE and Jupyter Notebooks with our extension ## Installation In order to use the Etiq Extensions you need to install the Python package to your local environment, then one of the VSCode Extension or the JupyterLab Extension depending on your IDE of choice. ### Python Package Install the `etiq-copilot` python package from PyPi: `pip install etiq-copilot` ### VSCode Extension Download the VSCode extension from the VSCode Marketplace, either search for Etiq in the Extensions pane on your VSCode or via the link below: ### JupyterLab Extension Install the `jupyterlab-etiq` python package from PyPi: `pip install jupyterlab-etiq` ## Getting Started with Etiq in VSCode ⚠️ **Warning**! Please be sure that you have [selected the correct Python Interpreter ](https://code.visualstudio.com/docs/python/environments#_working-with-python-interpreters)in your VSCode, this should be the `venv` or other virtual environment where `etiq-copilot` was installed. If you see an error like the below you are that is likely to be the issue
### Starting Etiq in VSCode We recommend you start with the example scripts available on our Github here: Within the `lineage_example` directory you'll find `lineage_example.py` Open that file in your VSCode window Open the VSCode command palette. You can access the VS Code Command Palette in a number of ways. * Shift+Command+P (Mac) / Ctrl+Shift+P (Windows/Linux). Note that this command is a reserved keyboard shortcut in Firefox. * F1 * From the Application Menu, click **View > Command Palette**. Type into the search bar `etiq` and select `etiq: Show Panel`

Searching for the Etiq Panel in the Command Palette

This will then bring up the Etiq Extension to the right of your code in VSCode as below

Showing the Etiq Panel

### Exposing the Etiq Output Pane By opening the Etiq Output Pane, it allows you to see what Etiq is doing. View Output: Shift+Command+O (Mac) / Ctrl+Shift+O (Windows/Linux). Select `Etiq` from the highlighted dropdown box

Output pane from VSCode with highlighted dropdown

Once you can see `Listening on port 5556` in the window you are ready to scan your first script! ### Scanning a file - Creating your first lineage in VSCode With your script open on the left, click `Scan the current file` in the Etiq extension on the right. This will then scan your code and data and show you your Lineage in the Lineage pane.

A successfully scanned script and Lineage

## Getting Started with Etiq in JupyterLab ### Starting Etiq in JupyterLab We recommend you start with the example scripts available on our Github here: Within the `lineage_example` directory you'll find `lineage_example.ipynb` Open this in your Jupyter Notebook Server ### Scanning a file - Creating your first lineage in JupyterLab
a test notebook open
You can then open the sidebar on the right on the Lineage tab
If you now click the Scan file button within this tab, Etiq will analyse the entire notebook and display the lineage.
You can also select an individual cell, ensuring that previous cells have been executed, and display the lineage just for the data and code objects in that cell.
# Data Science Agents A quick start guide to enable and using Etiq's Data Science Agents #### 🚨Please note that currently our Agent functionality is only compatible with VSCode⚠️ Have any questions or looking for support from the team, join our Discord server: ### Setting up Etiq with an agent In order to use Etiq's Agent functionality you need to have an API key for one of the following LLM providers: * OpenAI * Anthropic * Google With Etiq installed click on the Etiq logo on the left of the primary sidebar in VSCode.
This will open up the Etiq LLM Copilot Selector. Choose from the drop down your LLM provider of choice, and then enter a valid API key for that provider in the box below.
If this has been successful you'll see the RCA tab in the Etiq Panel on the right had side change to DSA (Data Science Agent).
### Using the Data Science Agent The Data Science Agent replaces the Root Cause Analysis (RCA) functionality of the core Etiq Data Science Copilot. Clone our demo scripts repo Within the `duplicates-example` directory you'll find `duplicates_example.py` Using this script first you scan the current file
Move to the testing recommendations tab and select a relevant test to run, here we're going to choose: ```python Line 24: df_with_duplicates = df_with_duplicates.sample(frac=1, random_state=42).reset_index(drop=True) ``` And we will run the `etiq.scan_duplicate_rows` test
You'll see that the test has now failed
Click on the run DSA tab, this will change to `Waiting...` This will start the Data Science Agent which will now analyse both the data and teh code to understand what is the root of the issue, here the duplicate rows, and provide a fix. Once the agent is finished the `Waiting...` will change to Re-Run DSA
Scroll to the top and now click on the DSA tab. This will bring up the specific lineage for the affected data and code
If you scroll down the DSA result section will suggest a fix, if you click Open Full Diff it will show that change to be made on your script. Underneath that will be an expandable Code Fix Explanation
If you now make the suggested change to the script, save the script rerun the code scan, and rerun the `etiq.scan_duplicate_rows` test you'll see that it has now passed.
# Welcome to Etiq ### Intro Etiq is an ML testing platform for data scientists and ML engineers. Use Etiq’s lightweight tools to identify ML specific issues. This will help prevent accuracy loss in production and reduce time it takes to validate a model and transition it from prototype to production-level pipelines. As early as initial built stages, test your system to prevent operational issues downstream arising from poorly functioning or misunderstood ML models. Our concept is to provide error detection functionality that provides the look and feel and ease of use as unit tests, but backed by deep pipelines. ![Passed and failed tests % by model version](/files/2Z6Iallk13RUE1lvzp1p) This package includes tests in the following areas: Accuracy, Data Issues, Leakage, Drift and Bias {% hint style="info" %} For other error areas, like explainability, robustness, sensitivity - get in touch as there is functionality currently outside the main API but which can be provided to you on demand. We’re constantly adding more out-of-the-box tests so stay tuned for next releases. {% endhint %} In addition to out-of-the-box capabilties we also have multiple customization options and a low-level API available. These provide additional functionality to ensure you can meet the needs of your specific use cases. Whilst other tools focus primarily on helping you optimise for metrics during the experimentation phase, at Etiq we know that metrics calculated in a pre-production environment are only likely to be indicators of true performance whilst in production. With Etiq, you can use them for what they are: indicators and go beyond optimisation. As early as the experimentation phase, you can easily start testing for potential issues to ensure your model performs as well as possible in production. ### When to use Etiq / Use cases ![Use Etiq during build, validation, productionalizing and when the model is live](/files/S0nX0hS0CTOxoa2EFCrk) We recommend using Etiq’s test functionality throughout the model build process including in your production pipeline. Having similar tests throughout the pipeline will help ensure that the deployed model does in production what the person who designed it pre-production expects it to do. In production, at the moment we provide functionality for batch processing only. {% hint style="info" %} Etiq supports models from XGBoost, LightGBM, PyTorch, TensorFlow, Keras and scikit-learn. {% endhint %} ### Why use Etiq for ML testing? **Multi-step pipelines**: The problem with testing for ML is that sometimes you need a multi-step pipeline to run one test. Especially for bias tests, more complex concept drift, any explainability tests. That means you need to log the results of each of the steps, and productionalize the testing itself as you would your actual model. Etiq does that for you. Additional benefits: * **Discrete tests that you can plug in at different steps**: with ML you do have to test even once your model is in production. But your model is complex enough without some heavy testing functionality to stop it from running properly. You can add specific tests at different points in your pipeline, and take them out, run them in parallel or run them once you have the outputs of the model. Whatever the set-up, you can centralize the results in the same location and voila you have a detailed working **Model monitoring tool**. * **Custom, custom, custom**: you can customize the thresholds of your test, and add your own custom metrics to create your own custom tests. * **Root cause analysis**: some of our tests help you understand why an issue is happening much faster. For instance, your overall accuracy is looking good, but there is a segment which is really underperforming. The test can show you which segment that is (without you having to pre-set the segments). * **Documented by default**: when testing part of the struggle is actually documenting both the test results and the test template itself and the parameters it used. Etiq solves this problem by having config files associated with each test suite. This way you don't need to spend additional time documenting the tests you've run, you can just check the config associated with the test suite. # Quickstart ### Sign-up and install The Etiq library supports Python versions 3.8, 3.9, 3.10, 3.11 and 3.12 on Windows, Mac and Linux. {% hint style="info" %} With the release of Etiq 1.6.0 the package is now compatible with Apple Silicon processors. Due to dependencies in the package you may need to install `libomp` via `homebrew` If you haven't already, install homebrew on your computer: Then run the following in your terminal: `brew install libomp` {% endhint %} {% hint style="info" %} If you're looking to use our Great Expectations integration, please ensure you install `great-expectations <= 0.18.19` due to breaking changes being introduced with `v.1.0.0` If you have any questions please contact us at {% endhint %} To start with, go to the [dashboard site](https://dashboard.etiq.ai/), sign-up and login. If you want to deploy directly on your AWS instance, just go to our AWS Marketplace listing and deploy from [there](https://aws.amazon.com/marketplace/pp/prodview-q5opksxavexbs?sr=0-1\&ref_=beagle\&applicationId=AWSMPContessa) (using Etiq via AWS Marketplace incurs a cost however). {% hint style="warning" %} If you have purchased version 1.2 via AWS Marketplace please go to [this section](/etiq-1.x-documentation/v-1.2-aws-marketplace/1.2-functionality) of the docs. {% endhint %} #### Once you've signed-up to the dashboard, check up this interactive demo: {% embed url="" %} Click to navigate through the demo {% endembed %} #### Below are detailed instructions: To start logging tests from your notebook or other IDE to your dashboard you will need a token to associate your session with your account. To create this token, once in your account go to the Token Management window and just click on Add New Access Token. Then copy and paste into your notebook. ![Log tests to the centralized dashboard](/files/wuvVngYl6eMk54SnKUac) Download and install Etiq: ```python pip install etiq ``` {% hint style="info" %} For install considerations please go to [this section.](/etiq-1.x-documentation/faq-and-other/faq#install-best-practice) {% endhint %} Then import it in your IDE & log to the dashboard: ```python import etiq from etiq import login as etiq_login etiq_login("https://dashboard.etiq.ai/", "") ``` Exciting news :tada::tada::tada: etiq for spark is now also available. The data and drift tests you know and love applied to more data than ever before. To install & import just run the below: ```python pip install etiq-spark import etiq.spark ``` Go to an [example notebook](https://github.com/ETIQ-AI/demo) or keep reading to get an understanding of the key concepts used in the tool. Please don't leave your token lying around as if anyone finds it they can use it to retrieve information stored about your pipelines similarly to how you use a password/username authentication. {% hint style="warning" %} Data about your test results get stored on Etiq's AWS instance. However your datasets and models will not actually be stored anywhere, so you can rest assured. If your security set-up is such that you would need a deployment entirely on your cloud instance or on prem just get in touch with us - {% endhint %} ### Projects A project is a collection of snapshots. To start using the versioning and dashboard functionality, please set a project and a project name. You only have to run it once per session and all the details logged as part of data pipelines or debias pipelines will be stored. Once you go to your dashboard you will be able to see each of your projects and dig deeper into each of them. ```python #Create or open project project = etiq.projects.open(name="") #Retrieve all projects all_projects = etiq.projects.get_all_projects() print(all_projects) ``` ### Log a snapshot to etiq - key principles This step is just about logging the relevant information so you can run your tests/scans afterwards. You will need to log your model, your training and test dataset and the config file which defines key parameters, such as what's the predicted features, what are the categorical/continuous features, etc. Depending at which stage in the model build/production you are and what type of scans you are running, you will want to log differently: 1. If you are using Etiq's wrapper around model classes, then essentially you log as you train. You can input your entire dataset (in an appropriate format, e.g. already encoded, or already transformed). And in the config you can give different % to the train/validation/test split, e.g. `"train_valid_test_splits": [0.8, 0.1, 0.1]` 2. If you have already built your model, then you will need to log a hold-out sample to Etiq as your dataset, and this sample will need to be in a format appropriate for being scored by your scoring function. When you log the split in the config, you should reflect this as your hold-out sample is a validation sample: "train\_valid\_test\_splits": \[0.0, 1.0, 0.0]. You can also use this set-up for production type use cases. {% hint style="info" %} Scans like bias sources and leakage are about tests on the training dataset. For more details on how to run these scans, go to their corresponding sections: [Leakage](/etiq-1.x-documentation/scan-types/leakage) and [Bias](/etiq-1.x-documentation/scan-types/bias#bias-sources-scan) {% endhint %} ### Log a snapshot to etiq - example for already trained model First you will need to load your config file. This file contains relevant parameters which will be useful in logging the rest of the elements so make sure you log this before you create your snapshot. 
with etiq.etiq_config("./config_demo.json"):
    #load your dataset
    #log your already trained model
    #create a snapshot
    #conduct metrics scan
You can also load your config file in the way shown below. However, we prefer the "modern" way shown above because the config below is only used within the with block and doesn't persist until overridden, as in the global example above. ```python etiq.load_config(“./config_demo.json”) ``` Example configs are provided [here](https://github.com/ETIQ-AI/demo/tree/main/Demo%20account%20snapshots) and also below. For details on what to log to config check the [Config Key Concept](/etiq-1.x-documentation/key-concepts/config). For details on how to adjust the config for different scan types, check [Accuracy](/etiq-1.x-documentation/scan-types/accuracy#setting-up-accuracy-scans), [Leakage](/etiq-1.x-documentation/scan-types/leakage#setting-up-leakage-scans), [Drift](/etiq-1.x-documentation/scan-types/drift#setting-up-drift-scans), [Bias](https://docs.etiq.ai/etiq-1.x-documentation/pages/1q9I0XF3YfTCYsC9zs3Y#production-vs.-pre-production) or relevant notebooks by scan type [here](https://github.com/ETIQ-AI/demo/tree/main/Scans%20by%20type). ```json { "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.0, 1.0, 0.0], "remove_protected_from_features": false }, "scan_accuracy_metrics": { "thresholds": { "accuracy": [0.8, 1.0], "true_pos_rate": [0.6, 1.0], "true_neg_rate": [0.6, 1.0] } }, "scan_bias_metrics": { "thresholds": { "equal_opportunity": [0.0, 0.2], "demographic_parity": [0.0, 0.2], "equal_odds_tnr": [0.0, 0.2], "individual_fairness": [0.0, 0.8], "equal_odds_tpr": [0.0, 0.2] } }, "scan_leakage": { "leakage_threshold": 0.85 } } ``` {% hint style="success" %} For example notebooks and config files, just go to our [demo repository](https://github.com/ETIQ-AI/demo). {% endhint %} Next, you will log your dataset and your model. To log your dataset please log the test dataset that you used to assess your model. (There are 2 scans for which your training dataset will be needed: scan\_bias*\_*sources and scan\_leakage - for more details look at [Scan Types](/etiq-1.x-documentation/scan-types/accuracy)). {% hint style="warning" %} If your dataset is not in a format your model can score, the scan will not run! If you have a use case where you can use demographic feature in your training dataset, you have the option to leave it in using this clause in the config: "remove\_protected\_from\_features": false The default is that the demographic feature is removed in the scoring. This is because in regulated use cases you shouldn't use the demographic/protected feature to train your model on, but the scan still needs information about the demographic if you want to run bias scans. {% endhint %} ```python from etiq import Model #log your dataset dataset = etiq.BiasDatasetBuilder.dataset(test, label="") #can also use SimpleDatasetBuilder #Log your already trained model model = Model(model_architecture=standard_model, model_fitted=model_fit) ``` {% hint style="info" %} Parameter 'model\_architecture' refers to model architecture, and is optional, e.g. ``` standard_model = XGBClassifier(use_label_encoder=False, eval_metric='logloss', random_state=4) ``` Parameter 'model\_fitted' refers to model fit, however you store it, e.g.: ``` model_fit = standard_model.fit(x_train, y_train) ``` You can also specify the 'model\_fitted' parameter only. {% endhint %} And create your snapshot: ```python snapshot = project.snapshots.create(name="", dataset=dataset, model=model, bias_params=etiq.BiasDatasetBuilder.bias_params()) ``` {% hint style="info" %} For drift-type scans you will not need a model, instead you can have a dataset, e.g. this month's dataset, and a benchmark dataset that you're comparing against, e.g. last months' dataset. For more details on how to set-up drift scans, go [here](/etiq-1.x-documentation/scan-types/drift) or for example notebooks with drift go [here](https://github.com/ETIQ-AI/demo/tree/main/Scans%20by%20type/Drift). {% endhint %} ### Run scans on your snapshot Now you are ready to run scans on your snapshot: ```python snapshot.scan_accuracy_metrics() snapshot.scan_bias_metrics() ``` The above is an example using an already trained model in pre-production. For a full notebook on this go [here](https://github.com/ETIQ-AI/demo/tree/main/Example%20scans%20-%20Already%20trained%20model). If you want to use one of Etiq's pre-configured model classes see an example [here](https://github.com/ETIQ-AI/demo/tree/main/Example%20scans%20-%20Pre-configured%20model). If you want to use the scans in production, just email us . A demo integration with Airflow will be available shortly.
{% hint style="warning" %} Threshold values in the example config files are for example purposes. Different use cases will require different thresholds. As the AI regulation sector matures we will add corresponding standards and suggested thresholds, but this will never be a hard and fast rule, it will be a suggestion. What might work for one use case will not work for another. {% endhint %} You have the option to add the categorical and continuous features in your config, as per the example below. This is useful for certain types of scans which translate the findings into business rules, but you have to remember to update your config if you take out or add new features. ```json { "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.0, 1.0, 0.0], "remove_protected_from_features": false, "cat_col": ["workclass", "relationship", "occupation", "gender", "race", "native-country", "marital-status", "income", "education"], "cont_col": ["age", "educational-num", "fnlwgt", "capital-gain", "capital-loss", "hours-per-week"] }, "scan_accuracy_metrics": { "thresholds": { "accuracy": [0.8, 1.0], "true_pos_rate": [0.6, 1.0], "true_neg_rate": [0.6, 1.0] } }, "scan_bias_metrics": { "thresholds": { "equal_opportunity": [0.0, 0.2], "demographic_parity": [0.0, 0.2], "equal_odds_tnr": [0.0, 0.2], "individual_fairness": [0.0, 0.8], "equal_odds_tpr": [0.0, 0.2] } }, "scan_leakage": { "leakage_threshold": 0.85 } } ``` If you do not want to scan for bias or do not have in your dataset information about protected features, you can just not add that information to your config. An example config for a data drift use case below. For more details on this example check the [github repo](https://github.com/ETIQ-AI/ml-testing/tree/main/Scans%20by%20type/Drift) and/or the section about [Drift](/etiq-1.x-documentation/scan-types/drift) ```json { "dataset": { "label": "income", "train_valid_test_splits": [0.0, 1.0, 0.0] }, "scan_drift_metrics": { "thresholds": { "psi": [0.0, 0.15], "kolmogorov_smirnov": [0.05, 1.0] }, "drift_measures": ["kolmogorov_smirnov" , "psi"] } } ``` # Usage Plans You have a few options in terms of how to use Etiq: 1. If you are looking to understand if Etiq is the right tool for you then you can sign-up to our dashboard for free and use the library. The results of your tests will be stored on your dashboard instance in our cloud set-up. Our cloud provider is AWS and only you have access to the test results information. Note that your datasets and models will not be sent outside your environment. We recommend you use this during your due diligence process. 2. If you want to use Etiq within your own environment for test purposes but are weary of sending any information outside your environment, you can use the library only and not login to the dashboard. However by default that will give you limited functionality (you are heavily restricted on the number of features your model and datasets use). This is also free to test, but very limited. 3. If you want to purchase Etiq for use within your own environment, you have a few options: * Are you on AWS - great! You can use Etiq directly from [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-q5opksxavexbs?sr=0-1\&ref_=beagle\&applicationId=AWSMPContessa). * Are you on a different cloud provider - that's ok! Reach out to us and we'll support. 4. If you want functionality beyond the one available in the test library and dashboard, e.g. explainability, robustness or team share functionality, or if you just want a demo from us, get in touch: {% hint style="info" %} Etiq's free SaaS version is not intended for multiple users/instances to use the same account/token. A temporary notebook environment like Google Colab or using the same key across different machines may lead to unexpected results. {% endhint %} # Snapshot ### Logging a snapshot Etiq works via a lightweight logging mechanic. You log your data and your model (a snapshot) and then you run a scan on it - which is the testing functionality itself. As you experiment with more and more snapshots, you keep scanning your model versions and all the test results and issues found get sent to a centralised dashboard. A snapshot is a combination of dataset and model, especially for in pre-production testing. To start testing your system you need to log your snapshot to Etiq, and to do so you’d log the dataset and the model. For an end-to-end notebook example, go [here](https://github.com/ETIQ-AI/ml-testing/tree/main/Scans%20by%20pipeline%20stage/Example%20scans%20-%20Already%20trained%20model). {% hint style="warning" %} **Before you log a snapshot you will need to load your** [**config file**](/etiq-1.x-documentation/key-concepts/config)**. Otherwise you will get an error.** {% endhint %} ```python #Log your dataset dataset = etiq.BiasDatasetBuilder.dataset(data_encoded, label="") #can also use SimpleDatasetBuilder #Log your already trained model model = Model(model_architecture=standard_model, model_fitted=model_fit) # Creating a snapshot snapshot = project.snapshots.create(name="", dataset=dataset, model=model, bias_params=etiq.BiasDatasetBuilder.bias_params()) ``` For validation and production stages, snapshots are not produced in the course of experimentation, they are produced as a model is deployed and runs in production. But from the point of view of Etiq’s logging mechanic, they get logged the same way. Each time your model scores a new batch of data, it records a new snapshot. However the information needed for testing is slightly different in production vs. pre-production and the tests themselves are a bit different. For [drift](/etiq-1.x-documentation/scan-types/drift#what-is-drift) type tests or generally in production, you might not have the available model, but more importantly, you would need the dataset you’re considering for drift and the benchmark dataset you are comparing against. This is how you’d log it to Etiq: ```python # Log a dataset with the comparison data dataset_s = etiq.SimpleDatasetBuilder.dataset(data_encoded, label="") # Log a dataset with the data from your current view todays_dataset_s = etiq.SimpleDatasetBuilder.dataset(todays_dataset_df, label="") # Create the snapshot snapshot = project.snapshots.create(name="", dataset=todays_dataset_s, comparison_dataset=dataset_s, model=None) ``` ### Dataset At the moment we support uploading pandas or spark dataframes to the Etiq dataset object, but we are adding new formats all the time. The dataset you use should be already transformed in such a way that it can be inputted to a model class from any of the libraries mentioned. While Etiq contains some transformations, we recommend using your own. Especially with certain types of transformations (such as normalization) , please do NOT apply your transformation to your whole dataset prior to splitting it into train\_test\_valid as this will can contribute to leakage. (we will be adding scans to check for this as well in the future). **NB: Note that there might be certain scans not available for certain datasets.** There are currently two types of datasets. These are the SimpleDataset and the BiasDataset. The SimpleDataset is a container for the data to be used by the Etiq package. This includes training, validation and testing data along with metadata identifying categorical, continuous, id and date features. The BiasDataset contains additional metadata identifying "bias" features. #### Create a simple dataset from a Pandas DataFrame In order to create a simple dataset from a pandas dataframe the following function: ``` etiq.SimpleDatasetBuilder.dataset(features, target, label, cat_col, cont_col, id_col, date_col, convert_date_cols, datetime_format, train_valid_test_splits, random_seed, name) ``` can be used. The only required parameter is a dataframe containing the features (and target). If no other parameters are supplied etiq will either use defaults or make its best guess for the other parameters. The target label (specified using the ***label*** parameter) or a separate dataframe containing the targets (specified using the **target** parameter) should also be specified otherwise the last column of the features dataframe will be chosen as the target by default. The other parameters are as follows: * **cat\_col** - A list of columns containing categorical data * **cont\_col** - A list of columns containing continuous data * **id\_col** - A list of columns containing id data (note these are not used by the model) * **date\_col** - A list of columns containing date information (note these are also not used by the model) * ***convert\_date\_cols*** - A True/False flag that determines whether or not to convert the datetime columns from strings to the native datetime format. (This defaults to False) * ***datetime\_format*** - The format to use when converting the datetime columns. * **train\_valid\_test\_splits** - Tuple containing the training/validation/testing split proportions * **random\_seed**: Number used to seed the random number generator for random splits * **name** - The name to use for the dataset. Note that the non-dataframe parameters can be loaded from the dataset section of a config file. See the [dataset config options](/etiq-1.x-documentation/key-concepts/config#dataset-details-options). If, however, the dataframe has already been split into training, validation and testing then the following function: ```python etiq.SimpleDatasetBuilder.datasets(training_features, training_target, validation_features, validation_target, testing_features, testing_target, label, cat_col, cont_col, id_col, date_col, convert_date_cols, datetime_format, name) ``` can be used where * **training\_features** - a dataframe containing the training features * **training\_target** - an (optional) dataframe containing the training target * **validation\_features** - a dataframe containing the validation feature * **validation\_target** - an (optional) dataframe containing the validation target * **testing\_features** - a dataframe containing the testing features * **testing\_target** - an (optional) dataframe containing the testing target The other parameters are identical to the previous dataset constructor. #### Create a simple dataset from a Spark DataFrame If the etiq.spark module is installed a simple dataset can be constructed from a spark dataframe. The two constructors that can be used are as follows: ```python etiq.SimpleSparkDatasetBuilder.dataset(features, label, cat_col, cont_col, id_col, date_col, convert_date_cols, datetime_format, train_valid_test_splits, random_seed, name) ``` where **features** is a spark dataframe and the other parameters are as described in the previous section. To create a simple dataset from "pre-split" spark dataframes we use ``` etiq.SimpleSparkDatasetBuilder.datasets(training_features, validation_features, testing_features, label, cat_col, cont_col, id_col, date_col, convert_date_cols, datetime_format, name) ``` where the parameters are as described in the previous section. **NB: Both features and targets have to be defined in the same spark dataframe.** #### Create a Bias dataset from a Pandas DataFrame In order to create a bias dataset from a pandas dataframe the following function: ``` etiq.BiasDatasetBuilder.dataset(features, target, label, cat_col, cont_col, id_col, date_col, convert_date_cols, datetime_format, train_valid_test_splits, bias_params, random_seed, name) ``` can be used. where * bias\_param - a named tuple specifying the bias meta data. The other parameters are as described for the simple dataset. The corresponding function for creating a bias dataset where training, validation and testing dataframes (at least one needs to be specified) is ``` etiq.BiasDatasetBuilder.datasets(training_features, training_target, validation_features, validation_target, testing_features, testing_target, label, cat_col, cont_col, id_col, date_col, convert_date_cols, datetime_format, bias_params, name) ``` #### Create a Bias dataset from a Spark DataFrame In order to create a bias dataset from a pandas dataframe the following function: ``` etiq.BiasSparkDatasetBuilder.dataset(features, target, label, cat_col, cont_col, id_col, date_col, convert_date_cols, datetime_format, train_valid_test_splits, bias_params, random_seed, name) ``` can be used. where * bias\_param - a named tuple specifying the bias meta data. The other parameters are as described for the simple dataset. The corresponding function for creating a bias dataset where training, validation and testing dataframes (at least one needs to be specified) is ``` etiq.BiasSparkDatasetBuilder.datasets(training_features, validation_features, testing_features, label, cat_col, cont_col, id_col, date_col, convert_date_cols, datetime_format, bias_params, name) ``` ### Model You can use any already trained model from the supported libraries: XGBoost, LightGBM, PyTorch, TensorFlow, Keras and scikit-learn. It should be compatible with any model that uses the sklearn fit/predict convention. For example purposes, we also provide out-of-the box model architectures for some model types: `DefaultXGBoostClassifier` (a wrapper around XGBoost classifier), `DefaultRandomForestClassifier` (a wrapper around the random forest classifier from sklearn) and `DefaultLogisticRegression` (a wrapper around the logistic regression classifier from sklearn). However, most use cases will use own fit model or pre-calculated model Call a model already fitted using the following syntax. For a notebook example, go [here](https://github.com/ETIQ-AI/ml-testing/tree/main/Scans%20by%20pipeline%20stage/Example%20scans%20-%20Already%20trained%20model). 

# Load the dataset
dataset = etiq.BiasDatasetBuilder.datasets(training_features=test,
                                               validation_features=valid)
# Load the bias parameters
bias_params = etiq.BiasDatasetBuilder.bias_params()

# Create your already trained model and log it.
model = Model(model_architecture=standard_model, model_fitted=model_fit)

# Create a Snapshot
snapshot = project.snapshots.create(name="<snapshot-name>",
                                        dataset=dataset,
                                        model=model,
                                        bias_params=bias_params)
Call a wrap around model from Etiq library using the following syntax. For a notebook example, go [here](https://github.com/ETIQ-AI/ml-testing/tree/main/Scans%20by%20pipeline%20stage/Example%20scans%20-%20Pre-configured%20model) ```python #Log your dataset dataset = etiq.BiasDatasetBuilder.dataset(data_encoded, label="") # Load our model from etiq.model import DefaultXGBoostClassifier model = DefaultXGBoostClassifier() # Creating a snapshot snapshot = project.snapshots.create(name="", dataset=dataset, model=model, bias_params=etiq.BiasDatasetBuilder.bias_params()) ``` #### Pre-calculated model Sometimes it is not possible to directly use a machine learning model in python for various reasons. However we still want to evaluate the model performance. In order to accommodate such a usecase we make available a "pre-calculation" model. This simply contains the prediction labels for the desired model we would like to evaluate on a dataset. An example of how to use such a model is provided below. 
# This bit is done by someone with access to the model
train, valid = train_test_split(data_encoded, test_size=0.2, random_state=17)
train.reset_index(inplace=True, drop=True)
valid.reset_index(inplace=True, drop=True)

# Split data then train the model
y_train = train['income'].copy() # labels we're going to train the model to predict
x_train = train.drop(columns=['income'])
y_valid = valid['income'].copy() 
x_valid = valid.drop(columns=['income'])

# train a model to predict 'income'
standard_model = MyCustomModel()    
model_fit = standard_model.fit(x_train, y_train)
y_train_pred = standard_model.predict(x_train)
y_valid_pred = standard_model.predict(x_valid)

model_df = data_encoded.copy()
model_df['predicted'] = standard_model.predict(data_encoded.drop(columns=['income'])
model_df.to_csv('data-with-predictions.csv', index=False)

# The precalculated csv file is then provided.
# This bit can be run by someone without access to the model
import pandas as pd
import etiq

labelled_data_with_predictions = pd.read_csv('data-with-predictions.csv')
labelled_data_without_predictions = labelled_data_with_predictions.drop(
                                      ['predicted'], axis=1)
precalc_model = etiq.model.PrecalculatedModel(labelled_data_with_prediction,
                                               prediction_label='predicted')
dataset = etiq.SimpleDatasetBuilder.dataset(labelled_data_without_predictions, 
                                              label="income") 


# Creating a snapshot
snapshot = project.snapshots.create(name="<snapshot-name>", 
                                     dataset=dataset, 
                                     model=precalc-model)
# Run model performance scans
# Scan The scan is a test pipeline applied to a snapshot. The scan is testing whether the model/snapshot has a specific issue. The function to call a scan is a one liner: ```python snapshot.scan_() #for example snapshot.scan_bias_metrics() ``` ### Issues, Metrics, Thresholds A scan is a testing pipeline, and a lot of scans just test for one issue only, e.g. is accuracy above a certain threshold. But other scans, e.g. scan\_bias\_sources test for a lot of different issues at the same time because it is more efficient. Which is why we added **ISSUE** as a sub-element of the scan. Whether an issue is found or not is based on whether the **METRIC** associated with the issue is outside acceptable **THRESHOLDS**. You can set thresholds based on your use case, although we provide config files with suggested thresholds to get you started. You can also add custom metrics as per this section. A subset of metrics is **MEASURES**. In our convention, measures are used more to uncover causes of issues rather than high level issues on your snapshot/model , e.g. a correlation coefficient would be a measure, but the line is blurry. ### Scans Summary
Scan TypeIssueMetric/MeasureRelease
Accuracy metricsIs accuracy above or below accepted threshold? Accuracy above threshold can also be a problem

Accuracy (no of correctly labelled/total)

TPR: true positive rate

TNR: true negative rate

1.3.1
Bias metricsIs given bias metric above or below acceptable threshold?

Equal opportunity Demographic parity Equal_odds_TNR

Individual fairness

Individual fairness counterfactuals

1.3.1
Bias sourcesWhat are proxies and sampling issues that could lead to bias later on? For automatically derived business rules, use the option auto in your config fileThis scan uses a measure of correlation (4 options based on types of features: Pearson, Cramer's V, Rank-Biserial, Point-Biserial), plus differential measures between demographic groups. 1.3.5
LeakageTarget leakage: Has the target leaked into a feature you use in your model? Demographic leakage: Has a demographic feature leaked into a feature in your model?This scan uses a measure of correlation (4 options based on types of features: Pearson, Cramer's V, Rank-Biserial, Point-Biserial), rather than a metric.1.3.5
Drift MetricsFeature drift: Has the feature dataset changed from the initial/benchmark dataset? For which feature?Kolmogorov- Smirnov Jensen-Shannon Distance PSI: Population Stability Index 1.3.1
Drift MetricsTarget drift: Has the target feature distribution changed from the initial/benchmark dataset?Kolmogorov- Smirnov Jensen-Shannon Distance PSI: Population Stability Index 1.3.1
Drift MetricsConcept drift: Have the relationships between target and features changed from initial/benchmark dataset?Earth Mover's Distance Kullback-Leibler Divergence Jensen-Shannon Distance1.3.5
Accuracy Metrics RCAAre there segments in the data where the model seems to be performing considerably worse than average?

Accuracy (no of correctly labelled/total)

TPR: true positive rate

TNR: true negative rate

1.3.3
Bias Metrics RCAAre there segments in data where the model is performing worse for a demographic group than for another?

Equal opportunity Demographic parity Equal_odds_TNR

Individual fairness

1.3.3
Drift Metrics RCAFeature Drift Metrics RCA: Are there segments in the data where feature drift was observed? Target Drift Metrics RCA: Are there segments in the data where target drift was observed?Kolmogorov- Smirnov Jensen-Shannon Distance PSI: Population Stability Index 1.3.5
Data Issues Scans (Autogenerated)Are there any data issues in my dataset? (Based on comparison with a comparison dataset).

Identical Feature

Missing Feature

Unknown Feature

Missing Feature Category:

Unknown Feature Category

Feature Value Below Minimum

Feature Value Above Maximum


1.3.6
Data Issues Scans (one dataset only)Are there any data issues in my dataset? (Based on one dataset only).Order violation Missing ID Duplicate Records 1.3.7
{% hint style="success" %} Etiq also has available test suites on other areas such as explainability related issues, or drift metrics RCA which are not currently part of the public release. If you are interested in them just get in touch with us {% endhint %} ### Parameters To be able to use the scans, you will also need to login parameters about the dataset. As the scans primarily handle classification problems at this stage, the parameters are as follows: * **For all scans**: * ‘label’ - Feature you are predicting * ‘train\_valid \_test\_splits’ (if your model is already trained and you’re providing only the test dataset for scans please set the % accordingly) * Optional: ‘cat\_col’ - list of categorical features * Optional: ‘cont\_col’ - list of continuous features {% hint style="info" %} Parameters ‘cat\_col’ and ‘cont\_col’ are optional, but for the scans relying on correlations, having these parameters logged means the scan can use the right measure) {% endhint %} * For bias scans: * ‘protected’ - a demographic feature or features that you are checking for bias for (protected characteristics) - for more information please see Bias Scans section * ‘privileged’ - usually the majority class or the class not protected by legislation * ‘unprivileged’ - the minority class or the class protected by legislation * ‘positive\_outcome\_label’ - for bias type tests it’s important to know which outcome label for the predicted feature is a positive outcome for the individual (e.g. low likelihood of default on a loan, or high likelihood of performing well in a role). This allows you to set-up the test to understand if the group that needs to be ‘protected’ is more likely to be treated negatively by the model. (For more details please see the Bias tests section) * ‘negative\_outcome\_label’ - a negative outcome for the individual (e.g. high likelihood of default of a loan) {% hint style="info" %} Having appropriate and accurate labels for your features means that you’ll be able to make use of the automated segment discovery and business rules creation that come with the dashboard. {% endhint %} Metrics, thresholds and parameters are customized as part of the config file (see next Key concept) # Config Configuration file description An etiq config file is a JSON format file which allows users to set default parameters to be used when loading datasets or running scans. Note that using a config file is **entirely optional** and it is possible to run any scan without providing default parameters in the config. The parameters provided in the config file for a dataset can also be overridden by providing explicit arguments in the scan itself. A config can be loaded in globally (i.e. the config will persist throughout the session once loaded unless a different config is subsequently loaded) using the load\_config function e.g. ```python etiq.load_config("./config_demo.json") ``` assuming the config file is config\_demo.json. A context manager equivalent is also provided. This can be used like the following: ```python with etiq.etiq_config("./config_demo.json"): # Scans under this config ``` These are config options that we have built to match our existing scans and requirements. If you need a specific option you cannot find reach out to us (). In the future you'll be able to add further options yourself. ### Config Structure * A '**dataset**' section containing default parameters to be used when loading datasets. These include bias parameters if applicable. For example: ``` "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.0, 1.0, 0.0], "remove_protected_from_features": true } ``` * **Scan specific sections** corresponding to each type of scan and which include the metrics, thresholds and other options available for each of the scans (note that these are optional and do not have to provided in order to run the scans). An example for a "**scan\_accuracy\_metrics**" below: ``` "scan_accuracy_metrics": { "thresholds": { "accuracy": [0.8, 1.0], "true_pos_rate": [0.6, 1.0], "true_neg_rate": [0.6, 1.0] } positive_outcome_label: 1, negative_outcome_label: 0 } ``` This allows us to run the corresponding scan with just those default parameter i.e. ```python snapshot.scan_accuracy_metrics() ``` If for some reason we want to run an accuracy metrics scan with the positive and negative outcome labels flipped we can override the default parameters in the config file by running ```python scan_accuracy_metrics(positive_outcome_label=0, negative_outcome_label=1) ``` Depending on the type of scan you're running and stage (pre vs. in-production), there are multiple additional config options available. ### Dataset Details Options The [Scan Types](/etiq-1.x-documentation/scan-types/accuracy) section describes which dataset parameters are relevant for each scan, e.g. bias\_params. But there are some parameters that can be used across scans: * **train\_valid\_test\_splits** - * To input which features are categorical and which are continuous, you can use **cat\_col** and **cont\_col** and then add the name of the features. You also have the option to do this outside the config if easier as per the [Snapshot](/etiq-1.x-documentation/key-concepts/snapshot#dataset) section. * You also have the option **remove\_protected\_from\_features**. When you are building a model in a regulated sector, you will not be able to use a protected demographic feature directly in the model. However you will need the protected feature(s) to assess whether you have a bias issue, so you will need this to be part of your dataset. This is not the case for other use cases perhaps. Thus you have the option either to consider the protected feature(s) as part of the model, or to consider them just for the purposes of assessing whether the model has a bias issue. ```json "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.8, 0.2, 0.0], "cat_col": ["workclass", "relationship", "occupation", "gender", "race", "native-country", "marital-status", "income", "education"], "cont_col": ["age", "educational-num", "fnlwgt", "capital-gain", "capital-loss", "hours-per-week"], "remove_protected_from_features": true } ``` ### RCA Type scans * **Defining the issue based on thresholds**. When searching for issues you can add options as to which interval you want to be considered, e.g. if you only want to find issues where accuracy is under a certain threshold rather than above, you can ask give it the option "ignore\_upper\_threshold": true * **Adding a minimum segment size** - e.g. only surfacing segments which are big enough according to your use case. * **Filtering on which metrics you want calculated -** "metric\_filter": \["accuracy", "true\_pos\_rate", "true\_neg\_rate"] . This will help you only run what you need. ```json "scan_accuracy_metrics_rca": { "thresholds": { "accuracy": [0.0, 0.5], "true_pos_rate": [0.0, 0.5], "true_neg_rate": [0.0, 0.5] }, "ignore_lower_threshold": false, "ignore_upper_threshold": true, "metric_filter": ["accuracy", "true_pos_rate", "true_neg_rate"], "minimum_segment_size": 1000, }, ``` ### Drift Type Scans For drift metrics scans both the regular scan and RCA scan we have a few different options: * For both scan\_drift\_metrics and scan\_drift\_metrics\_rca, you will be able to select which metrics you want using the option "**drift\_measures**". For more info on what drift measures you can choose check out the [Drift](/etiq-1.x-documentation/scan-types/drift) section. * As with typical RCA scans for the drift RCA scan you can choose what to do with the thresholds and also can choose the minimum segment size, beyond which you will not consider the issue * **Features**: you can choose which features to restrict the drift scan on if you so wish. The config option is "features" followed by the names of the features * **Number of bins**: for target and concept drift type scans, you can choose the number of bins you want to use. We will add binning options for feature drift type scan as well in the near future. ```json "scan_drift_metrics": { "thresholds": { "psi": [0.0, 0.15], "kolmogorov_smirnov": [0.05, 1.0] }, "drift_measures": ["kolmogorov_smirnov", "psi"] }, "scan_drift_metrics_rca": { "thresholds": { "psi": [0.0, 0.15], "kolmogorov_smirnov": [0.05, 1.0] }, "drift_measures": ["psi", "kolmogorov_smirnov"], "ignore_lower_threshold": true, "ignore_upper_threshold": false, "features": null, "minimum_segment_size": 1000 }, "scan_target_drift_metrics_rca": { "thresholds": { "psi": [0.0, 0.15], "kolmogorov_smirnov": [0.05, 1.0] }, "drift_measures": ["psi", "kolmogorov_smirnov"], "ignore_lower_threshold": true, "ignore_upper_threshold": false, "features": null, "minimum_segment_size": 1000 }, "scan_concept_drift_metrics": { "thresholds": { "earth_mover_distance": [0.0, 0.2], "kl_divergence": [0.0, 0.2], "jensen_shannon_distance": [0.0, 0.2] }, "drift_measures": ["earth_mover_distance", "kl_divergence", "jensen_shannon_distance"], "number_of_bins": 10 }, ``` ### Scans with Correlation/Association Measures There are scan types which are based on different types of correlation/association measures. These include scan\_bias\_sources: ```json "scan_bias_sources": { "auto": true, "nr_groups": 20, "continuous_continuous_measure" : "pearsons", "categorical_categorical_measure": "cramersv", "categorical_continuous_measure": "rankbiserial", "binary_continuous_measure": "pointbiserial" }, ``` as well as scans related to leakage, such as scan\_target\_leakage and scan\_demographic\_leakage ```json "scan_target_leakage": { "leakage_threshold": 0.85, "minimum_segment_size": 1000, "continuous_continuous_measure" : "pearsons", "categorical_categorical_measure": "cramersv", "categorical_continuous_measure": "rankbiserial", "binary_continuous_measure": "pointbiserial", "minimum_segment_size": null }, "scan_demographic_leakage": { "leakage_threshold": 0.85, "minimum_segment_size": 1000, "continuous_continuous_measure" : "pearsons", "categorical_categorical_measure": "cramersv", "categorical_continuous_measure": "rankbiserial", "binary_continuous_measure": "pointbiserial", "minimum_segment_size": null } ``` As you can see from the examples above, you have options on what correlation type to use based on the type of features you are looking to assess for correlations: * "continuous\_continuous\_measure" : "pearsons" * "categorical\_categorical\_measure": "cramersv" * "categorical\_continuous\_measure": "rankbiserial" * "binary\_continuous\_measure": "pointbiserial" The options above are the default provided, but you can customize them. What these options stand for are fairly self-explanatory. E.g. for continuous features this option states that the type of correlation measure used will be pearsons. But remember: to fully utilize these options you should input what is a categorical, binary and continuous feature in the config or when running the scans. # Custom Tests How to set up your own metrics to run regular scans and RCA type scans You have multiple ways to customize your tests. You can choose the scan types, scan metrics and thresholds. Additionally, you can also add your own custom metrics and measures, include them in your config and then your scans will check for this metric (or measure) as well - both the regular scans and RCA type scans :tada::tada::tada:. For a notebook and config example check our [github repo](https://github.com/ETIQ-AI/ml-testing/tree/main/Scans%20by%20type/Custom). ### Custom Metrics for Accuracy and Bias Scans The decorators you can use to build your custom accuracy and bias metrics are as follows: | Decorator | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | prediction\_values |

refers to what the model scores

should be a list

| | actual\_values |

refers to the actuals

if your custom metric is for production, it will use the score as actual if it is provided and no actuals or model are available

should be a list

| | protected\_values |

refers to the demographic variable you want to check for bias

if you have multiple demographics please create a feature with the intersection

| | positive\_outcome |

directional, refers to what is considered a positive prediction or outcome

e.g. in the case of a lending model it would be a low risk score or if the customer is accepted for the loan, should be a value

| | negative\_outcome |

directional, refers to what is considered a negative prediction or outcome

e.g. in the case of a lending model it would be a high risk score or if the customer is rejected for the loan, should be a value

| | privileged\_class |

refers to the class in the demographics which is privileged - not protected by the legislation

should be a value

| | unprivileged\_class |

refers to the class in the demographics which is not privileged - and which is protected by the legislation

should be a value, in future releases we will add functionality for multiple values here

| They follow the parameters available in the [config file](/etiq-1.x-documentation/key-concepts/config). | | | | ------------------------------ | --------------------------------------------------- | | @etiq.metrics.accuracy\_metric | refers to logging your metric as an accuracy metric | | @etiq.metrics.bias\_metric | refers to logging your metric as a bias metric | | @etiq.custom\_metric | specifies that this is a custom metric | Below is an example of how to add a custom metric to the accuracy metrics scan suite: ```python @etiq.metrics.accuracy_metric @etiq.custom_metric @etiq.actual_values('actual') @etiq.prediction_values('predictions') def accuracy_custom(predictions=None, actual=None): """ Accuracy = nr of correct predictions/ nr of predictions """ apred = np.asarray(predictions) alabel = np.asarray(actual) return (apred == alabel).mean() ``` Below is an example of how to add a custom metric to the bias metrics scan suite: ```python @etiq.metrics.bias_metric @etiq.custom_metric @etiq.prediction_values('predictions') def gini_index(predictions): class_counts = Counter(predictions) num_values = len(predictions) sum_probs = 0.0 for aclass in class_counts: sum_probs += (class_counts[aclass]/num_values) ** 2 return 1.0 - sum_probs ``` Afterwards don’t forget to update your config file with the metric name, and thresholds you want, before you run your scan. ```json { "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.0, 1.0, 0.0] }, "scan_accuracy_metrics": { "thresholds": { "accuracy": [0.7, 0.9], "true_pos_rate": [0.75, 1.0], "true_neg_rate": [0.7, 1.0], "accuracy_custom": [0.9, 1.0] } }, "scan_bias_metrics": { "thresholds": { "equal_opportunity": [0.0, 0.2], "demographic_parity": [0.0, 0.2], "equal_odds_tnr": [0.0, 0.2], "equal_odds_tpr": [0.0, 0.2], "individual_fairness": [0.0, 0.2], "gini_index": [0.3, 0.4] } } } ``` ### Custom Metrics for Drift Scans You can now add custom metrics for drift scans as well. Examples in this [notebook](https://github.com/ETIQ-AI/ml-testing/tree/main/Scans%20by%20type/Drift). Below is an example of how to add a custom metric for feature or target drift scans: ```python from etiq.drift_measures import drift_measure from scipy.stats import wasserstein_distance @drift_measure def earth_mover_drift_measure(expected_dist, new_dist, number_of_bins=10, bucket_type='bins', **kwargs) -> float: def scale_range (input, min, max): input += -(np.min(input)) input *= (max - min)/np.max(input) input += min return input breakpoints = np.arange(0, number_of_bins + 1) / (number_of_bins) * 100 if bucket_type == 'bins': breakpoints = scale_range(breakpoints, np.min(expected_dist), np.max(expected_dist)) elif bucket_type == 'quantiles': breakpoints = np.stack([np.percentile(expected_dist, b) for b in breakpoints]) expected_percents = np.histogram(expected_dist, breakpoints)[0] / len(expected_dist) actual_percents = np.histogram(new_dist, breakpoints)[0] / len(new_dist) return wasserstein_distance(expected_percents, actual_percents)pyth ``` Below is an example of how to add a custom metric for a concept drift scan: ```python from etiq.drift_measures import concept_drift_measure @concept_drift_measure def total_variational_distance(expected_dist, new_dist): return sum(0.5 * abs(x-y) for (x,y) in zip(expected_dist, new_dist))pyth ``` Don't forget to add the new metrics to the config file: ```json { "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.0, 1.0, 0.0], "remove_protected_from_features": true }, "scan_drift_metrics": { "thresholds": { "psi": [0.0, 0.2], "kolmogorov_smirnov": [0.05, 1.0], "earth_mover_drift_measure": [0.0, 0.2] }, "drift_measures": ["kolmogorov_smirnov", "psi", "earth_mover_drift_measure"] }, "scan_concept_drift_metrics": { "thresholds": { "earth_mover_distance": [0.0, 0.05], "kl_divergence": [0.0, 0.2], "jensen_shannon_distance": [0.0, 0.2], "total_variational_distance": [0.0, 0.03] }, "drift_measures": ["earth_mover_distance", "total_variational_distance"] } } ``` To build your own drift type measures, consider the logic of feature/target drift vs. concept drift. Feature and target drift look at whether the distribution for a certain feature has changed. For an explanation of how this is calculated using the out-of-the-box metrics provided check out the [drift section](/etiq-1.x-documentation/scan-types/drift#metrics). When building your own feature/target drift measure, you can use the following parameters which stand for the following concepts: * first argument (e.g. expected\_dist in the example above): observations of a given feature in the baseline dataset * second argument (e.g. new\_dist in the example above): observations of a given feature in the new dataset that we're assessing for feature drift Concept drift looks at whether the relationships between input dataset and target feature have changed over time. The out-of-the-box measures for concept drift look at the change between 2 datasets when it comes to, for instance, the probability that if target has value 0 feature A has value 1. The measure looks not at just one probability value but conditional probabilities are calculated for the different potential combinations of target and feature values. Then the measure compares the distribution of all these probabilities in the 2 datasets. The custom measures follow the same logic. This means that the parameters which you use to build concept drift type measures stand for slightly different things than those you use for feature/target drift: * first argument (e.g. expected\_dist in the example above): probabilities of the target values given a feature value in the baseline dataset * second argument (e.g. new\_dist in the example above): probabilities of the target values given a feature value in the new dataset that we're assessing for concept drift Note that for continuous features and/or target, the values will have to be binned. ### Custom metrics for RCA type scans You can also use your own custom metric in an [RCA type scan](/etiq-1.x-documentation/rca/rca-type-scans). If we continue the drift measure example below, we can just run the feature and target drift metrics RCA scans on the snapshot using the config as per below: ```python snapshot = project.snapshots.create(name="Test Snapshot", dataset=dataset1, comparison_dataset=dataset2, model=None) #Scan for different drift types (segments_f, issues_f, issue_summary_f) = snapshot.scan_drift_metrics() ``` would yield the following results: ![Example results of drift RCA scan with custom metric - earth mover drift measure](/files/oWbtNYvFHQr2IegBuZAr) This means you can now use Etiq to fully customize your tests to your use case, as well as to experiment with the best metrics and measures. ### Custom Correlation/Association Measures For bias sources and leakage scans, we use correlation and association measures. We provide multiple correlation measures out of the box to be used based on the type of features you have: Pearson, Cramer's V, Rank-Biserial, Point-Biserial, for more info see [Bias](/etiq-1.x-documentation/scan-types/bias#bias-sources-scan). To add your own custom correlation or association metric, use the decorator @correlation\_measure and see an example below: ```python @correlation_measure def tschuprowsT(x: List[Any], y: List[Any]) -> float: """ Calculates Tschuprow's T between two (categorical) variables. Args: x (List[float]): List like values representing the first variable. y (List[float]): List like values representing the second variable. Returns: float: tschuprow's T for the two variables (assuming they are both categorical) """ if len(x) < 2: return np.nan df = pd.DataFrame({'x': x, 'y': y}) contingency_table = pd.crosstab(index=df['x'], columns=df['y']) if contingency_table.shape[1] < 2 or contingency_table.shape[0] < 2: return np.nan val = association(contingency_table, method='tschuprow') return val ``` You can then use this for the relevant scans - we recommend using them in the scan types exemplified above. # Accuracy ### Why run accuracy scans? Accuracy metrics are what I optimize my models on. Why should I have tests on accuracy metrics as well? 1. High accuracy can be indicative of a problem just as much as low accuracy * For instance if a plain accuracy metric is 10% higher than you've expected you might have leakage somewhere or another issue. 2. Optimizing for a metric pre-production does not equate to optimizing for that metric in production * You will be better off getting a good model off the ground, a model with no obvious issues, and which is likely to be robust, than trying to achieve a 1% higher accuracy with a potentially overfitting model, a model which is unfairly discriminating against protected demographic groups or with an model which will experience abrupt performance decay. ### Metrics Our accuracy scans so far provide 3 metrics: | Metric | Formula | | ------------------ | ------------------------------------------------------------------------------------------------------- | | Accuracy | % correct out of total | | True Positive Rate | the proportion positive outcome labels that are correctly classified out of all positive outcome labels | | True Negative Rate | the proportion negative outcome labels that are correctly classified out of all negative outcome labels | {% hint style="success" %} In addition to these 3 metrics, you can use custom metrics to add your own metrics. {% endhint %} ### Setting-up accuracy scans Depending on the type of scan, your use case, and the stage at which you are in the model building/deployment process, you will have multiple combinations of how to set-up your dataset and sampling % in your snapshot, and your scan. For a common classification use case please see suggested set-up below:
StageScanSnapshot set-up

Pre-production

(if you use an etiq wrapped model)

scan_accuracy_metrics()

You can use the whole dataset and set-up the split % based on whatever you prefer (leaving at least 10% in the validation sample).

Etiq dataset loader will split it for you when it creates the snapshot. By default the scan will be run on the validation sample.

The parameter ‘label’ refers to predicted (and because this is your training / test / validation it will also be your actuals)

Pre-production

(if you log your own already trained model)

scan_accuracy_metrics()

You should log your actual test / validation dataset (the sample you did not use to train the model) as validation/test by setting the split in the config file like this: train_valid_test_splits": [0.0, 1.0, 0.0].

By default the scan will be run on the validation sample. The "label" parameter will be the predicted feature, not the actual. You won’t have actuals by this stage of model deployment yet.

Production

(if you have actuals)

scan_accuracy_metrics() -TPR: true positive rate -TNR: true negative rate

Only once you have actuals you are able to run this scan in production.
You should log the dataset used in production as validation by setting the split in the config file like this: train_valid_test_splits": [0.0, 1.0, 0.0].

The "label" parameter will be the actuals feature once you have it and you will need to set-up your dataset in advance (e.g. via using Airflow)

Production

(if you do not have actuals)

scan_accuracy_metrics() -custom metric You might have custom metrics which you are labelling as accuracy but for which you do not need actuals. In this instance, the label will be what the model scores/predicts rather than the actuals.
### Production vs. pre-production So far we have packaged the scans with the following assumption: When in production you will have to create your datasets containing actuals - irrespective of what the model predicted this feature will refer to what actually happened in reality (e.g. has the customer defaulted on their loan, was the transaction fraudulent, etc.) At the moment you will have the same parameter in your config file: parameter ‘label’, but this feature will denote what actually happened in reality. If requested by our users, we are open to adding an actuals dataset type in the packaging in the future, and separately a ‘predicted’ feature. {% hint style="info" %} You can use whatever accuracy metric you want in the scans to monitor your model’s performance. However, if you are thinking about how the responses came about, some metrics will be more helpful than others. For instance, let’s take a case where you do not have control groups: a model where you are predicting default rates, and as a result of your model you are giving loans only to those people who present a low enough risk profile. Out of those people, looking at who kept paying their loans for the first time period would give you a reliable true positive rate, that might decrease over time if the loan period isn’t complete, but which at least is not misleading. However, trying to look at an overall accuracy rate would not make much sense, as you have not given loans to anyone for whom you predicted a low likelihood of repayment in the first place. A lot of algorithmic bias related problems stem from these issues. {% endhint %} ### Example notebooks For example notebooks, code and config files for accuracy scans please see [repo link](https://github.com/ETIQ-AI/demo/tree/main/Scans%20by%20type/Accuracy). # Leakage ### What is leakage? Leakage can be very detrimental to your model. If you accidentally include a feature which encodes the target, your model will look like it's performing really well and accuracy will be much higher than expected. However this will not hold in production and you will likely deploy the wrong model in production. There are multiple types of leakage, `etiq` can detect the following: **Target leakage.** This occurs when a feature leaked into the target. e.g. if you're trying to predict yearly income and accidentally a monthly salary feature is included in your dataset (for the same time period). While it seems hard to make this mistake, think of datasets with hundreds of features sources from different databases and repositories around a business, perhaps calculated by multiple teams. **Demographic leakage.** This occurs when a feature leaked into one of your protected demographics feature. e.g. if relationship status contains information related to a customer's gender, then using that relationship status as a feature in a predictive model is highly problematic. If you have a use case where you've identified that this is not a problem, then do not use this scan. However depending on the methodology you use in your model build this might pose other types of issues. ### Metrics A good indicator for the two types of leakage above is whether any of the features in the dataset are highly correlated, as this means that likely one has leaked into another. This criterion resembles the proxy issue in the bias scans; however the main difference is the level of the thresholds (the level is much higher for leakage). We provide multiple correlation measures to be used based on the type of features: Pearson, Cramer's V, Rank-Biserial, Point-Biserial. Remember to clarify in the config or the snapshot which features are of which type to be able to use fully the multiple measure functionality. You can customize this in the config, but the default and recommended version is below: * `"continuous_continuous_measure" : "pearsons"` * `"categorical_categorical_measure": "cramersv"` * `"categorical_continuous_measure": "rankbiserial"` * `"binary_continuous_measure": "pointbiserial"` ### Setting up leakage scans
StageScanSnapshot set-up

Pre-production

(if you use an etiq wrapped model)

scan_target_leakage() scan_demographic_leakage()

You can use the whole dataset and set-up the split % based on whatever you prefer (leaving at least 10% in the validation sample).

Etiq dataset loader will split it for you when it creates the snapshot. By default the scan will be run on the training sample.

Pre-production

(if you log an already trained model)

scan_target_leakage() scan_demographic_leakage()You should use your actual training dataset. By default the scan will be run on the training sample.
### Production vs. pre-production This scan is more appropriate for training/pre-production stages. ### Example Notebooks For example notebooks, code and config files for leakage scans please see [repo link](https://github.com/ETIQ-AI/demo/tree/main/Scans%20by%20type/Leakage). # Drift ### What is drift? Drift can impact your model in production and make it perform worse than you initially expected. There are a few different kinds of drift: 1. **Feature drift:** Feature drift takes place when the distributions of the input features changes. * For instance, perhaps you built your model on a sample dataset from the winter period and it's now summer, and your model predicting what kind of dessert people are more likely to buy is not longer as accurate. 2. **Target drift:** Similarly to feature drift, target drift is about distribution of the predicted feature changing from one time period to the next. 3. **Concept drift:** Concept drift occurs when the relationships between the features and the predicted changes over time. 4. **Prediction drift:** Prediction drift refers to those instances when something happened to the model scoring itself when running in production and the relationship * This means that somehow with the same or similar input dataset you'd get different predictions in the post-period as you did in the previous period. {% hint style="info" %} `etiq` does current not include scans related to prediction drift {% endhint %} ### Setting up drift scans To measure drift you will need a comparison or benchmark dataset. To load your comparison dataset use the following: ```python # Create a dataset with the comparison data dataset_s = etiq.SimpleDatasetBuilder.from_dataframe(data_encoded, target_feature='income').build() # Create a dataset with the data todays_dataset_s = etiq.SimpleDatasetBuilder.from_dataframe(todays_dataset_df, target_feature='income').build() ``` Then log your snapshot and scan the dataset for drift: ```python # Create the snapshot snapshot = project.snapshots.create(name="Test Snapshot", dataset=todays_dataset_s, comparison_dataset=dataset_s, model=None) #feature drift: snapshot.scan_drift_metrics() #target drift snapshot.scan_target_drift_metrics() #concept drift snapshot.scan_concept_drift_metrics() ``` {% hint style="info" %} To run a drift scan you will not need a model, but you will need at least 2 datasets. For each snapshot, you log the current period dataset and the previous period dataset. When you call the drift scan, it will assess whether any drift issues occurred. {% endhint %} ### Production vs. pre-production While most people think of drift as something that happens in production, in fact it is something you can test for as you build your model as well. If you have datasets from a different time period (a year ago, a quarter ago, seasonal), then you might want to see if the distributions of the features and labels have drifted over time (feature/target drift) or if the type of relationships between features and target have changed over time. Thus we recommend that if you have the datasets you use these scans pre-production to give you an indication of potential issues you might encounter when your model does go live. So far we have packaged the drift scans with the following assumption relevant for Target Drift and Concept Drift: When in production you will have to create your datasets containing actuals - irrespective of what the model predicted this feature will refer to what actually happened in reality, e.g. has the customer defaulted on their loan, was the transaction fraudulent, etc. At the moment you will have the same parameter in your config file: parameter ‘label’, but this feature will denote what actually happened in reality. If requested by our users, we are open to adding an actuals dataset type in the packaging in the future, and separately a ‘predicted’ feature. For drift in production, just like for the other scans that can be used in production, we will shortly release an Etiq + Airflow demo. If you can’t wait or you use a different orchestration tool, please email us . Depending on the type of scan, your use case, and the stage at which you are in the model building/deployment process, you will have multiple combinations of how to set-up your dataset and sampling %, and your scan. For a common classification use case please see suggested set-up below:
StageScanSnapshot set-up

Pre-production

(regardless of whether you are using an etiq wrapped model or your own model)

Feature drift: scan_drift_metrics() Target drift: scan_target_drift_metrics() Concept drift: scan_concept_drift_metrics()

You should test for feature, target and concept drift on your training dataset. It doesn’t matter how you log it.

Depending on your use case, it might be relevant to test for training/test sample differences using the drift scans.

The "label" parameter will be the predicted feature, which in this case is also the actuals, as for all the scans in pre-production.

Production

(if you don’t have actuals)

Feature drift: scan_drift_metrics()

You can log your dataset as either training or validation, but should probably log it as validation if you want to run other scans on your snapshot.

The "label" parameter will be the predicted feature, not the actual.

Production

(if you do have actuals)

Feature drift: scan_drift_metrics() Target drift: scan_target_drift_metrics() Concept drift: scan_concept_drift_metrics()

Once you have actuals you are able to run Target Drift and Concept Drift scans.

The "label" parameter will be the actuals feature.

{% hint style="info" %} Remember you won’t need the model for your drift scans, just for your other scans. {% endhint %} ### Metrics Drift scans are trying to determine differences in two probability distributions. Feature drift and target drift use PSI and KS: **Population Stability Index (PSI)**: PSI measures the shift of a population over time or the shift between two samples of a population. This involves binning the two distributions and then comparing the population percentages in each bin such that: ![](/files/o2tm4xn3wAu2tIRTs8It) where *i* represents the bin number, Actual is the present population distribution and Expected is the reference population distribution. PSI<0.1 indicates an insignificant change, 0.1\0.25 indicates a major change. **Kolmogorov-Smirnov Test:** This is a test to assess if two data samples (D1 and D2) belong to the same probability distribution. This measure is defined as: ![](/files/ZDqx1F9BIMaULZSRRbEg) where *P(x)* and *Q(x)* are the Cumulative Distribution Functions of 1-D datasets D1 and D2, and is the supremum (it is the subset of samples x that maximizes |*P(x)*-*Q(x)*|). The KS test identifies differences in location and shape of the cumulative distribution functions of samples D1 and D2, and it works well with numerical data. To assess whether concept drift took place, it’s not enough to see if the distributions of the input features or of the predicted feature have changed over time, instead we need to understand if the conditional probabilities changed over time. This is because concept drift looks not at changes in data but at changes in relationships between input features and the predicted feature. The measures included in our library for this are as below: {% hint style="success" %} You can definitely use these measures for data drift as well {% endhint %} **Kullback–Leibler (KL) Divergence**: This measures the difference between two probability distributions, These measures provide a straightforward metric for monitoring any significant changes in the input data or the model output. If Q and P represent respectively the distribution for the old and new data, then the KL divergence for Q and P is defined as: ![](/files/IGtKtzsr07touUhGA3LP) **Jensen-Shannon (JS) Divergence:** This is an extension of KL divergence such that it is symmetric and smoother. If Q and P represent respectively the distribution for the old and new data, then the JS divergence for Q and P is defined as: ![](/files/UFSlZIpGojCfaMn45Mrq) **Wasserstein Distance/Earth Mover Distance:** This is a distance measure between two 1D distributions. This measures the minimum amount of work needed to convert distribution P to distribution Q, where work is calculated by multiplying the amount of distribution weight to be moved by the distance moved. This measure is defined as: ![](/files/D5GM2tQblTviZ14Bhytq) where *n* is the number of bins, and *i* is the bin number. Use this to monitor the input distribution (single feature at a time) or prediction probabilities. {% hint style="info" %} All these measure how two probability distributions differ. But the PSI and Kolmogorov-Smirnov tests measure differences in empirical distributions and therefore are not as suitable for measuring concept drift. {% endhint %} **Custom Drift Measures**: It is possible to define custom drift measures to be used with etiq. This is done using the `drift_measure` decorator for a custom target or feature drift measure or `concept_drift_measure` for a custom concept drift measure. For examples please see the notebook [here](https://github.com/ETIQ-AI/demo/tree/main/Scans%20by%20type/Drift). {% hint style="warning" %} For continuous features you will want to consider pre-bucketing your features. We are adding an option so that you don't need to do this in our next release. You will also run into issues if you have fewer/more categories than in our base dataset as the distribution shifts will impact most of the metrics above. We are also adding functionality to cover this eventuality. {% endhint %} ### Example Notebooks For example notebooks, code and config files for accuracy scans please see [repo link.](https://github.com/ETIQ-AI/demo/tree/main/Scans%20by%20type/Drift) # Bias ### What is bias? In this context, bias refers to algorithmic bias. "Algorithmic bias" refers to unintended discrimination occurring as a result of an automated decision. Legislation defines a series of protected features. For example, in the UK, citizens are protected against discrimination on the basis of age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex or sexual orientation status by the Equality Act 2010. The unprivileged group within the protected feature (for example, people over 65 when age is the protected feature) tends to be discriminated against and as a result tends to be the one protected by legislation. The privileged group within the protected feature tends to not be discriminated against. If you are not tackling this issue, not only is your model potentially unethical, discriminating unintentionally and at risk from a compliance point of view, but also you are potentially leaving customer groups underserved and thus leaving money on the table. ### Bias Metrics Scan Some of the metrics commonly used in the algorithmic fairness literature that the Etiq library provides are:
MetricsDescription
Equal Opportunitymeasures the difference in true positive rate between a privileged demographic group and an unprivileged demographic group
Demographic Paritymeasures the difference between number of positive labels out of total from a privileged demographic group vs. a unprivileged demographic group)
Equal Odds TNR

measures the difference between true negative rate - privileged vs. unprivileged

The full measure in the literature looks for an optimal point where the difference in true positive rate between demographic groups as well as the difference in true negative rate between demographic groups are both minimized

Individual Fairnessmeasures whether individuals with similar features observe the same model responses
Our Bias Metrics scan uses the metrics above with certain thresholds to see if the model meets that benchmark or not. The syntax to run the scan after you’ve logged a snapshot is: `snapshot.scan_bias_metrics()` The thresholds are set by the user, but **most metrics are ideally as close to 0 as possible**, meaning that the model shouldn't really behave differently (and with detrimental outcomes) for the protected groups. The consensus in the literature (and our view) is that algorithmic bias can be mitigated but not removed entirely. {% hint style="warning" %} This is still a new area of research, and the metrics available can be misleading. For more resources please see our [research post on this topic](https://etiq.ai/research/how-fairness-metrics-can-be-misleading). {% endhint %} ### Bias Sources Scan Our Bias Sources scan identifies potential sources of bias based on a framework that includes:
SourcesDescription
Proxiesfeatures that are proxy for demographics
Sample size disparitydifference in sample sizes and size of positive/negative labels between protected demographic and the majority demographic group
Segment sizeare some customer profiles poorly represented in your sample?

Limited features /

correlation issue

features are less reliable for a certain demographic group

this is oftentimes linked with sampling but more fundamentally it could be that some groups' behaviour is less well encoded by available features

It is useful to look at these metrics globally to uncover issues across your sample. But a lot of the issues will only be visible for specific groups or specific records. The Bias Sources scan aims to identify which groups have the issues above. Bias sources scan is ran on training dataset by default as this is where the potentially harmful unfairly discriminatory pattern is learned by your model. You will not be running this scan in production. Bias metrics is ran on the validation dataset. The syntax to run the scan after you've logged the relevant config file and a snapshot is: **`snapshot.scan_bias_`sources`()`** #### You have two options of bias sources scans to run: 1. if you don't set anything in the config, the segments will be fuzzy rather than business rules. 2. if you set the *option: auto* in the config (as in the current config we are using) then the segments will be based on business rules. If you use the auto option, you will need to specify the categorical and continuous features. You can do this either from the config as in this case: ```json { "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.8, 0.1, 0.1], "remove_protected_from_features": true, "cat_col": ["workclass", "relationship", "occupation", "gender", "race", "native-country", "marital-status", "income", "education"], "cont_col": ["age", "educational-num", "fnlwgt", "capital-gain", "capital-loss", "hours-per-week"] }, "scan_bias_metrics": { "thresholds": { "equal_opportunity": [0.0, 0.2], "demographic_parity": [0.0, 0.2], "equal_odds_tnr": [0.0, 0.2], "individual_fairness": [0.0, 0.2], "equal_odds_tpr": [0.0, 0.2] } }, "scan_bias_sources": { "auto": true } } ``` Or you can run it from the notebook: ```python #Load your dataset #For bias sources you need to add some specific syntax at the moment or set-up your categorical and continuous features in the config dataset_loader = etiq.dataset(data_encoded) dl = etiq.dataset_loader.DatasetLoader(data=data_encoded, label='income', bias_params=dataset_loader.bias_params, train_valid_test_splits=[0.8, 0.1, 0.1], cat_col=cat_vars, cont_col=cont_vars, names_col = data_encoded.columns.values) from etiq.model import DefaultXGBoostClassifier # Load our model model = DefaultXGBoostClassifier() # Creating a snapshot snapshot = project.snapshots.create(name="Snapshot 2", dataset=dl.initial_dataset, model=model, bias_params=dataset_loader.bias_params) ``` We provide multiple correlation measures to be used based on the type of features: Pearson, Cramer's V, Rank-Biserial, Point-Biserial. Remember to clarify in the config or the snapshot which features are of which type to be able to use fully the multiple measure functionality. You can customize this in the config, but the default and recommended version is below: * "continuous\_continuous\_measure" : "pearsons" * "categorical\_categorical\_measure": "cramersv" * "categorical\_continuous\_measure": "rankbiserial" * "binary\_continuous\_measure": "pointbiserial" There are many additional sources of bias, which require more background or context knowledge than just observing the data or the model: * **'Tainted' examples:** the target variable is reflective of past bias * e.g. a model predicting who might make a good hire using data on who was hired in the past not on who was the objectively best candidate for the role * **Skewed sample:** the dataset is not representative of the population for which the model will be used ### Production vs. pre-production
StageScanSnapshot set-up

Pre-production

(etiq wrapped model)

Bias Sources: scan_bias_sources()

You can use the whole dataset and set-up the split % based on whatever you prefer (leaving at least % in the validation sample). Etiq dataset loader will split it for you when it creates the snapshot.

By default the scan will be run on the training sample.

The parameter ‘label’ refers to predicted (and because this is your training/test/validation it will also be your actuals)

Pre-production

(etiq wrapped model)

Bias Metrics:

scan_bias_metrics()

You can use the whole dataset and set-up the split % based on whatever you prefer. Etiq dataset loader will split it for you when it creates the snapshot.

By default the scan will be run on the validation sample.

The parameter ‘label’ refers to predicted (and because this is your training/test/validation it will also be your actuals)

Pre-production

(already trained user model)

Bias Sources: scan_bias_sources()

You should log your actual training dataset as training by setting the split in the config file like this: train_valid_test_splits": [1.0, 0.0, 0.0].

By default the scan will be run on the training sample.

You will have to run this scan separately from the bias metrics and bias accuracy scans. (We are working on changing this).

The parameter ‘label’ refers to predicted (and because this is your training/test/validation it will also be your actuals)

Pre-production

(already trained user model)

Bias Metrics:

scan_bias_metrics()

You should log your actual test/validation dataset (the sample you did not use to train the model) as validation by setting the split in the config file like this: train_valid_test_splits: [0.0, 1.0, 0.0].

By default the scan will be run on the validation sample.

The "label" parameter will be the predicted feature, not the actual. You won’t have actuals by this stage of model deployment yet.

Production

Bias Metrics:

scan_bias_metrics()

Individual_Fairness; Demographic_Parity

You should log your dataset as validation.

By default the scan will be run on the validation sample.

These metrics do not require actuals. The "label" parameter will be the predicted feature, not the actual. You won’t have actuals by this stage of model deployment yet.

Production

Bias Metrics:

scan_bias_metrics()

Equal_Opportunity;

Equal_Odds

Only once you have actuals you are able to run this scan in production.

You should log your dataset as validation by setting the split in the config file like this: train_valid_test_splits": [0.0, 1.0, 0.0].

By default the scan will be run on the validation sample.

The "label" parameter will be the actuals feature once you have it and you will need to set-up your dataset in advance (e.g. via using Airflow)

StageScanSnapshot set-up

Pre-production

(etiq wrapped model)

Bias Sources: scan_bias_sources()

You can use the whole dataset and set-up the split % based on whatever you prefer (leaving at least % in the validation sample). Etiq dataset loader will split it for you when it creates the snapshot.

By default the scan will be run on the training sample.

The parameter ‘label’ refers to predicted (and because this is your training/test/validation it will also be your actuals)

Pre-production

(etiq wrapped model)

Bias Metrics:

scan_bias_metrics()

You can use the whole dataset and set-up the split % based on whatever you prefer. Etiq dataset loader will split it for you when it creates the snapshot.

By default the scan will be run on the validation sample.

The parameter ‘label’ refers to predicted (and because this is your training/test/validation it will also be your actuals)

Pre-production

(already trained user model)

Bias Sources: scan_bias_sources()

You should log your actual training dataset as training by setting the split in the config file like this: train_valid_test_splits": [1.0, 0.0, 0.0].

By default the scan will be run on the training sample.

You will have to run this scan separately from the bias metrics and bias accuracy scans. (We are working on changing this).

The parameter ‘label’ refers to predicted (and because this is your training/test/validation it will also be your actuals)

Pre-production

(already trained user model)

Bias Metrics:

scan_bias_metrics()

You should log your actual test/validation dataset (the sample you did not use to train the model) as validation by setting the split in the config file like this: train_valid_test_splits: [0.0, 1.0, 0.0].

By default the scan will be run on the validation sample.

The "label" parameter will be the predicted feature, not the actual. You won’t have actuals by this stage of model deployment yet.

Production

Bias Metrics:

scan_bias_metrics()

Individual_Fairness; Demographic_Parity

You should log your dataset as validation.

By default the scan will be run on the validation sample.

These metrics do not require actuals. The "label" parameter will be the predicted feature, not the actual. You won’t have actuals by this stage of model deployment yet.

Production

Bias Metrics:

scan_bias_metrics()

Equal_Opportunity;

Equal_Odds

Only once you have actuals you are able to run this scan in production.

You should log your dataset as validation by setting the split in the config file like this: train_valid_test_splits": [0.0, 1.0, 0.0].

By default the scan will be run on the validation sample.

The "label" parameter will be the actuals feature once you have it and you will need to set-up your dataset in advance (e.g. via using Airflow)

### **Bias Scans Limitations** Bias is one of the most complex topics today. We started Etiq to help teams tackle this problem. We don’t believe that having a few scans in place is enough to tackle this problem. We don’t think our bias sources scans are by any means exhaustive. Additionally the metrics themselves are often misleading - we have published some research on this topic [here](https://etiq.ai/research/how-fairness-metrics-can-be-misleading). However, if via these scans, data science and engineering teams at least start considering algorithmic bias and fairness as a problem they should tackle, as important if not more important than accuracy based performance, or drift, or data issues, then we feel at least part of our mission is accomplished. If you are interested in this problem in more depth, we’d be very happy to hear from you. We have done research in the space and have additional pipelines built as part of the lower level API which we’re happy to share and run you through if you’re interested (email us ). ### **Example notebooks** For example notebooks, code and config files for accuracy scans please see [repo link](https://github.com/ETIQ-AI/demo/tree/main/Scans%20by%20type/Bias). # Data Issues ### Data Issues Description Data collection and validation forms an essential part of any machine learning pipeline. A number of issues could come up at the data collection phase and the Etiq library provides a way of detecting these. Instead of having the user define explicit rules as to what constitutes valid data the rules are automatically generated based on an exemplar dataset. **The different kinds of data issues detected are:**
Data IssuesDescriptions
Identical FeatureThis is a data issue where a feature in one dataset has values which are just identical copies of the exemplar dataset.
Missing FeatureThis is a data issue where a feature in the exemplar dataset is missing from the comparison dataset.
Unknown FeatureThis is a data issue where a feature in the comparison dataset is missing from the exemplar dataset.
Missing Feature CategoryThis is a data issue where a categorical feature has values in the exemplar dataset which are missing from the comparison dataset.
Unknown Feature CategoryThis is a data issue where a categorical feature has values in the comparison dataset which are missing from the exemplar dataset.
Feature Value Below MinimumThis is a data issue where a continuous feature has value(s) in the comparison dataset which are lower than the minimum value for that feature in the exemplar dataset.
Feature Value Above MaximumThis is a data issue where a continuous feature has a value(s) in the comparison dataset which are higher than the maximum value for that feature in the exemplar dataset.
Order ViolationThis is a data issue where for a particular record the ordering of two features is violated e.g. a start date happens later than an end date. This is only available where a snapshot has a single dataset.
Missing IDThis is a data issue for a particular record where an id feature has a missing value. This is only available where a snapshot has a single dataset.
Duplicate RecordThis is a data issue where a particular record is a duplicate of at least one other record. Note that records are identified by the tuple of all the id values unless a subset is specified. This is only available where a snapshot has a single dataset
### Data Issues Scans Just as you do for drift, you will have to create your snapshot using the dataset you are assessing for issues and a comparison dataset. The example below shows how to do a data issues scan where we have an exemplar dataset against which we want to compare another dataset. ``` snapshot = project.snapshots.create(name="Data Issues Snapshot", dataset=base_dataset, comparison_dataset=comparison_dataset, model=None) snapshot.scan_data_issues() ``` If, however, we only want to scan a single dataset the following example would be more appropriate ``` snapshot = project.snapshots.create(name="Data Issues Snapshot", dataset=base_dataset, model=None) snapshot.scan_data_issues() ``` {% hint style="info" %} There are a number of config parameters that can be set to switch the different tests on and off or only search certain features for certain issues where applicable. For a full range of config options see the example notebook for data issues. {% endhint %} ### Privacy Considerations If you use data issues scans on our SaaS version, you have the option to leave out details on the Data profile charts by using the config option below: ```python base_snapshot = project.snapshots.create(name="Base Snapshot", dataset=base_dataset, model=None, generate_data_profiles=True) ``` {% hint style="warning" %} However the distribution charts might still pick up some details you do not want to exit your organization's environment. If you need help on using or testing Etiq on-prem or on your own instance - just reach out to us directly: . Don't forget we are also on [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-q5opksxavexbs?sr=0-1\&ref_=beagle\&applicationId=AWSMPContessa) (release 1.3.4) {% endhint %} ### Example Notebooks For example notebooks, code and config files for accuracy scans please see [repo link](https://github.com/ETIQ-AI/ml-testing/tree/main/Scans%20by%20type/Data%20Issues). # Data Fingerprinting {% hint style="info" %} Added in etiq 1.6.0 {% endhint %} Data Fingerprinting enables rapid comparison of two related datasets by creating a set of metrics for each feature, testing whether they match and using this data to determine the relationship between the two. ### Use Cases: Oftentimes data scientists or analysts pick up issues with data based on higher-level aggregates, e.g. there is a discrepancy in the monthly sum for a given category of payment. With the fingerprinting feature, we have added these types of aggregate/pivot tests into our testing suite. Tests can be carried out on complete datasets, or on a subset of features. Typical uses of the fingerprinting feature are: * "Fingerprinting" a dataset: calculating the dataset’s metrics (min, max, mean, median, missing, sum, unique, std for each column - more details on these below) * Testing whether two datasets’ metrics (fingerprints) match, within a certain tolerance * At the dataset level, testing whether two datasets have the same number of rows * Creating summary objects that provide detailed results from these tests ### Metrics: By default, for each dataset,the following metrics are determined for each column of a suitable type (a subset of these metrics can also be calculated if preferred). The overall count of rows in each dataset is also evaluated. | Metric Name | Description | Per Table or Per Feature? | | ----------- | --------------------------------------- | ------------------------- | | count | How many rows are there in the dataset? | Table | | min | Minimum value | Feature | | max | Maximum value | Feature | | mean | Mean value | Feature | | median | Median value | Feature | | missing | How many rows are missing values? | Feature | | sum | Sum of values | Feature | | unique | How many unique values? | Feature | | std | Standard Deviation | Feature | **Table 1:** Metric names and descriptions. ### Data Relationships Pairs of datasets can be related in four possible ways: * `pivot` - One dataset is an aggregation of the other. * `replica` - one dataset has the same columns but different data (e.g. sales data from month to month) * `sampling` - one dataset is a row-wise sample of another * `part` - one dataset has a subset of columns from the other dataset (and all rows for those columns) The user does not need to specify how the datasets are related; the relationship will be inferred from the datasets automatically. ### Usage It is easy to run this - you will need two etiq snapshot objects representing the two different datasets. In your notebook or script you can simply; ```python # Assuming these are existing snapshots; segments, issues, issueaggregates = snapshot_2.scan_fingerprint(snapshot_1) ``` As with our other scan\_\* methods on each snapshot, we return three [Pandas dataframe](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) objects with the results in. If you are connected to the etiq dashboard in this session, the scan results will also be uploaded and shown under the snapshot. #### Segments A list of “segments” within the data. Presently just a single “all” segment. #### Issues A list of issues we’ve found with the data. An issue occurs when a feature metric does not match between tables. We show the name of the feature and the metric which did not match. You can supply arguments to the `scan_fingerprint` method to tune this output - for example if you expect a feature to vary between datasets, then you can adjust the margin so that false positives do not occur. See the API Usage section for more.

DataFrame of issues found with the join

#### Issue Aggregates This table shows an aggregate of all tests run - that is, the features and metrics tested against each plus the count of how many tests ran for each plus how many failed.
The “threshold” column shows the error margin used for that test. See API Usage below for details.

Issue Aggregates DataFrame

### API Usage #### Margin Specification Often the data will not be identical so we allow the user to specify a margin of error specified as a fraction between 0 (no error) and 1 (complete difference). A margin of 1 will never find any errors! Margin can either be set for the entire dataset with the margin keyword when we call the scan; ```python snapshot_2.scan_fingerprint(snapshot_1, margin=0.25) ``` Alternatively you may wish to specify on a per-feature level as your data may vary more in some columns more than others. This is supported with the “per\_field\_margin” keyword which consists of a dictionary whose keys are feature names and the values are the margin specified as floats. Items not specified in this dict will use the default margin; ```python snapshot_2.scan_fingerprint( snapshot_1, margin=0.25, per_field_margin={"age": 0.1, "country": 0.8}, ) ``` #### Metrics There may be times when it does not make sense to run all metrics - perhaps you are only interested in one or two, or you know that others will vary too much between datasets resulting in false positives. The “metrics” argument allows you to specify which metrics to run. The list of available metrics is listed above in the Metrics section. ```python snapshot_2.scan_fingerprint(snapshot_1, metrics=["sum", "min", "max"]) ``` # RCA Type Scans Root Cause Analysis Type Scans Some scans are simple tests that show whether a metric is above or below certain thresholds set by the user. Other scans are more complex and look more in depth at what could be causing the issue. We call this RCA - root cause analysis. These scans will help you discover segments of customers or groups of records for which the model has a lower than expected accuracy or groups for which bias thresholds are not met. You should expect a level of variability of your model for different parts of your dataset. However, if the variability is too high, with larger segments failing you might want to understand the causes of this, and address them. These scans help you find out exactly which segment has an issue, which should help you fix it sooner. Imagine if only a part of the data drifted or only a segment is underperforming, your overall tests might not pick up on it, but this test would. While you can pre-set segments you are interested in, if you just run the scan as is, it will discover problematic segments on its own. RCA type tests are also very useful when it comes to the drivers that impact a model’s decision. We don’t include explainability scans in our public release. However if this is something you need, email us . # Accuracy RCA Scan `scan_accuracy_metrics_rca` is the typical RCA scan. You can find an example notebook [here](https://github.com/ETIQ-AI/ml-testing/tree/main/RCA). An example config file is as below: ```json { "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.0, 1.0, 0.0], "remove_protected_from_features": true }, "scan_accuracy_metrics": { "thresholds": { "accuracy": [0.8, 1.0], "true_pos_rate": [0.75, 1.0], "true_neg_rate": [0.7, 1.0] } }, "scan_accuracy_metrics_rca": { "thresholds": { "accuracy": [0.8, 1.0], "true_pos_rate": [0.7, 1.0] }, "metric_filter": ["accuracy", "true_pos_rate"], "minimum_segment_size": 1000 } } ``` The syntax to run the scan after logging the model and dataset is the following: ```python snapshot.scan_accuracy_metrics_rca() ``` Like with all RCA scans the principle behind the scan is that it searches through different combinations of records and it finds those combinations for which the metric is outside the thresholds. As per the usual scans, you can set the thresholds for what constitutes an issue for your use case. You can also filter out the metrics you want/do not want RCA for, using, for example,: `"metric_filter": ["accuracy", "true_pos_rate"]` To make the metrics per group meaningful, it assigns a minimum number of records that constitutes a group, but you can change this by using the following syntax/parameter as per config example below: `"minimum_segment_size": 1000` {% hint style="warning" %} The minimum segment size will impact the results. We recommend setting the minimum segment size at 2% of the sample size. However if 2% is less than a significant segment size for your sample (e.g. less than 1000), please increase it. By default the scans use 2% of your sample size. {% endhint %} At the moment we only have results retrieval through the IDE and by snapshot using the following syntax and then call each of the elements. ```python (segments_accuracy, issues_accuracy, issue_summary_accuracy) = snapshot.scan_accuracy_metrics_rca() ``` The end results give business rules to the segments to help you understand the records you’re having an issue with. We are working to add more retrieval methods. Out of the box you can scan for the following metrics: 1. **Accuracy** - % correct out of total 2. **True positive rate** - the proportion positive outcome labels that are correctly classified out of all positive outcome labels 3. **True negative rate** - the proportion negative outcome labels that are correctly classified out of all negative outcome labels # Bias RCA Scan We have 2 RCA type tests for bias: * `scan_bias_sources` * `scan_bias_metrics_rca` `scan_bias_sources` is described in more detail in [this section](/etiq-1.x-documentation/scan-types/bias#bias-sources-scan). The auto option means that it essentially acts as an RCA scan, but there are multiple types of issues that it searches for. `scan_bias_metrics_rca` is a typical RCA scan. An example config file is as below: ```json { "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.0, 1.0, 0.0], "remove_protected_from_features": true }, "scan_bias_metrics": { "thresholds": { "equal_opportunity": [0.0, 0.2], "demographic_parity": [0.0, 0.2], "equal_odds_tnr": [0.0, 0.2], "individual_fairness": [0.0, 0.2], "equal_odds_tpr": [0.0, 0.2] } }, "scan_bias_metrics_rca": { "thresholds": { "demographic_parity": [0.0, 0.3] }, "metric_filter": ["demographic_parity"], "ignore_lower_threshold": true, "ignore_upper_threshold": false, "minimum_segment_size": 1000 } } ``` The syntax to run the scan after logging the model and dataset is the following: ```python snapshot.scan_bias_metrics_rca() ``` Like with all RCA scans the principle behind the scan is that it searches through different combinations of records and it finds those combinations for which the metric is outside the thresholds. As per the usual scans, you can set the thresholds for what constitutes an issue for your use case. You can also filter out the metrics you want/do not want RCA for, using for example: `"metric_filter": ["accuracy", "true_pos_rate"]` To make the metrics per group meaningful, it assigns a minimum number of records that constitutes a group, but you can change this by using the following syntax/parameter as per config example above: `"minimum_segment_size": 1000` You can also forgo checking for issues below lower threshold or above higher threshold if you want to using this syntax: `"ignore_lower_threshold": true` {% hint style="warning" %} The minimum segment size will impact the results. We recommend setting the minimum segment size at 2% of the sample size. However if 2% is less than a significant segment size for your sample (e.g. less than 1000), please increase it. By default the scans use 2% of your sample size. {% endhint %} At the moment we only have results retrieval through the IDE and by snapshot using the following syntax and then call each of the elements. ``` (segments_bias, issues_bias, issue_summary_bias) = snapshot.scan_bias_metrics_rca() ``` The end results give business rules to the segments to help you understand the records you’re having an issue with. We are working to add more retrieval methods. Out of the box you can scan for the following metrics: 1. **Equal Opportunity:** measures the difference in true positive rate between a privileged demographic group and an unprivileged demographic group. 2. **Demographic Parity:** measures the difference between number of positive labels out of total from a privileged demographic group vs. a unprivileged demographic group) 3. **Equal Odds TNR:** measures the difference between true negative rate - privileged vs. unprivileged. The full measure in the literature looks for an optimal point where the difference in true positive rate between demographic groups as well as the difference in true negative rate between demographic groups are both minimized. 4. **Individual Fairness:** measures whether individuals with similar features observe the same model responses # Drift RCA Scan Description of Drift RCA Scan There are currently two kinds of RCA drift scan in the etiq library used to scan for feature and target drift respectively: 1. `scan_drift_metrics_rca` 2. `scan_target_drift_metrics_rca` As a quick refresher: Feature drift takes place when the distributions of the input features changes. For instance, perhaps you built your model on a sample dataset from the winter period and it's now summer, and your model predicting what kind of dessert people are more likely to buy is not longer as accurate. Similarly to feature drift, target drift is about distribution of the predicted feature changing from one time period to the next. For more details look at the [Drift Scan Type section](/etiq-1.x-documentation/scan-types/drift). Imagine if only a part of the data drifted your overall tests might not pick up on it, but this test would. The scan it will auto-discover problematic segments on its own without the need for the user to specify segments to test. {% hint style="info" %} RCA scans for concept drift are currently not implemented but it is on our roadmap and should be introduced in the near future. {% endhint %} For both types of scans, you have to set the parameters below: | Parameter | Description | | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `thresholds` | This is dictionary indexed by measure name specifying an lower and upper threshold that indicates a drift issue | | `drift_measures` | This is a list of drift measures to use for the scan. If not specified then all drift measures including using defined ones are used | | `ignore_lower`*`_`*`threshold` | This boolean flag allows the lower threshold to be ignored when scanning for segments with drift issues (this is by default set to True) | | `ignore_upper`*`_`*`threshold` | This boolean flag allows the lower threshold to be ignored when scanning for segments with drift issues (this is by default set to True) | | `minimum_segment`*`_`*`size` | This allows the user to set the minimum number of samples in a segment before the segment can be considered significant. By default this is set to 2% of the number of samples | |

features

(additional parameter only used for scan\_drift\_metrics\_rca)

| This is a list of features to consider when scanning for feature drift. By default all features in the snapshot dataset will be scanned | An example config file is as follows ```json { "dataset": { "label": "income", "bias_params": { "protected": "gender", "privileged": 1, "unprivileged": 0, "positive_outcome_label": 1, "negative_outcome_label": 0 }, "train_valid_test_splits": [0.8, 0.2, 0.0], "cat_col": "cat_vars", "cont_col": "cont_vars" }, "scan_target_drift_metrics_rca": { "thresholds": { "psi": [0.0, 0.15] }, "drift_measures": ["psi"], "ignore_lower_threshold": true, "ignore_upper_threshold": false, "minimum_segment_size": 1000 }, "scan_drift_metrics_rca": { "thresholds": { "psi": [0.0, 0.15] }, "drift_measures": ["psi"], "ignore_lower_threshold": true, "ignore_upper_threshold": false, "minimum_segment_size": 1000, "features": ["hours-per-week"] } } ``` You can use any of the drift measures already provided. {% hint style="warning" %} The minimum segment size will impact the results. We recommend setting the minimum segment size at 2% of the sample size. However if 2% is less than a significant segment size for your sample (e.g. less than 1000), please increase it. By default the scans use 2% of your sample size. {% endhint %} For more details about how RCA scans work for target and feature drift see the notebook at[ ](https://github.com/ETIQ-AI/ml-testing/tree/main/RCA/RCA%20Drift%20Metrics). # Dashboard Components The main option of retrieving your results is via the dashboard. The dashboard has two main components: A Snapshots List view and a Scans List view. We're adding a Project Overview which will be available in the next release. ### Snapshots List ![Snapshots View](/files/2Z6Iallk13RUE1lvzp1p) This view allows you to quickly compare snapshots and see which is performing better. Also during production it helps you tell if there are any issues arising with the latest data feed. All the metrics and visuals are aggregations of the scan results: * Snapshot name and date/timestamp * % Passed - out of the scans ran on the snapshot, how many found no issues (if a scan finds no issues, then it is passed, if it finds at least one issue then it is failed) * Accuracy % - The value of the accuracy metric on the snapshot * Model Health - Red/amber/green scale. This is currently pre-set based on % of scans passed (red - 30% and below, yellow - 30% - 70%, green - 70% and above), however in future iterations we will give you the option to customise it * Accuracy, Data Leakage, Robustness, Drift, Sensitivity, Bias - are all %scans passed out of scans ran in the given area * Top issue found: what is the issue that impacts the biggest % of the sample. E.g. if an accuracy issue that impacts the whole sample was found at the same time as a bias issue that impacts one segment only, the top issue found surfaced in this view would be accuracy, but you can check for details on all the other issues in the Scans detail view * Sample Impact %: for the top issue found, what is the % of sample impacted ### Scans List ![Scans list view](/files/tnRk0VP6vFpKtavz2PqA) For each snapshot, this provides: a detailed view of the scans performed, broken down by each issue type tested as part of each scan. Fields and definitions are as per previous view. Additional fields include details on the snapshot itself, as well as the following: * Issue type scan tested for * Metric used * Threshold used by the scan * Parameters used by the scan, e.g. demographic feature * Whether the issue was found or not * No. of features an issue was found for if any * No. of segments an issue was found for if any You can also use the Compare functionality to see which tests are passing from one version of the model to the other. {% hint style="info" %} Instead of linking to the dashboard, you can also link to your usual toolkits or retrieve your results in your IDE as per available notebooks. If you want to use our API to integrate with other tools, just get in touch: {% endhint %} # Project Sharing How to use the dashboard to share projects with other users. Project sharing allows other users to view your project. It can also allow you to create new snapshots against it within the Python client. When logging in, you are presented with a list of projects;

Project List Page

You can then click the image share icon to bring up the sharing dialog. Here you can choose which user(s) to share the project with;

The Project Sharing Dialog

Selecting a user in the list will grant access. Clearing the checkbox next to them will remove their access. Click "OK" to save the changes. Note that you can't remove yourself as the owner from the project! Finally the other user will be able to view the project within their project list;

How other users see your shared project.

Users cannot delete or share further projects they do not own. # Data Synchronisation Synchronising data between the dashboard and a Python/Jupyter client Your projects, snapshots etc. can all be sent up to the dashboard for better analysis. Whether you're running a Python script or a Jupyter notebook, you can easily link your project with your dashboard. This is good if; * You use different machines that don't share storage. * You are creating results on a cloud system. * You are creating results on multiple systems. * You want to see your results centrally on the Etiq Dashboard. ### Creating Access Tokens To do this, you need to log into the dashboard and create an "Access Token". This is a secret key used to submit content to the dashboard;

Click "Manage Access Tokens" in the menu

Access Token Management Page

1. Enter a useful name for the token in the "Token Name" box. This will help identify it later on when it comes to removing it. 2. Click "Add" 3. The token is created. Copy it safely to a file somewhere. {% hint style="warning" %} You can't come back to this page to view the full token. So be sure to copy the token somewhere! {% endhint %} ### Using Access Tokens Within your Python code, we should call this directly **before** we open any projects or snapshots; ```python etiq.login("http://dashboard.example.org", "my-token-text") ``` We use the pair of "dashboard url" and "token text" to identify where and to whom the projects are sent. That is, the user which created the token will own projects created using this token. From hereon, all records created will be synchronised with the dashboard until the script finishes or `etiq.logout()` is called. {% hint style="danger" %} Don't keep access tokens in source code! {% endhint %} For security reasons, and to aid in convenience if the token changes, you should either keep the auth key in a file and read that file at runtime; ```python with open("key.txt") as f: etiq.login("https://dashboard.example.org", f.read().strip()) ``` Or as an environment variable; ```python import os # These environment variables need to be set yourself outside the script! etiq.login(os.environ["ETIQ_DASHBOARD_URL"], os.environ["ETIQ_TOKEN"]) ``` If you log in above and you're in an interactive session, you'll be notified that the login was a success; `'Connection successful. Projects and pipelines will be displayed in the dashboard. 😀'` If this token is invalid, a traceback will be raised. ### Login Scope You must login each session to continue persisting, it is only kept in memory. If you want to stop sharing but have already logged in, calling `etiq.logout()` will "forget" the token and work will only be available locally. ### Accessing Existing Projects You can also access projects that you've created previously or have been shared with you using the Python API; ```python # Either you know what it's called: project = etiq.projects.open("My Existing Project", create_if_missing=False) # Or you can pick and choose: all_projects = etiq.projects.get_all_projects() ``` Note the use of the parameter `create_if_missing` above. By default if you open a project and it's not there, we will just quietly create it for you. But if you know you want to add to an existing project, then it's best to use this so we can be sure we've got an existing project. ### Adding To Existing Shared Projects When you have an existing project you can add snapshots to that as though it were your own. This way you can collaborate on projects with team members. ### Deleting Access Tokens Simply click the "Delete" button within the token management page. Note that this makes the token invalid and will not work anymore. The `etiq.login` method will fail if we try to use it from this point onwards. # Airflow How to use Etiq with Airflow Etiq can easily be used as a library in [Airflow](https://airflow.apache.org/) DAGs. Since Etiq is available as a python module it can be used with airflow with no changes. There are two different instances when you might want to use Etiq with Airflow. ### Using Etiq within a pre-existing Airflow set-up If you already use Airflow as an orchestration tool for your pipelines, you can easily integrate Etiq in your existing DAGs as an additional few steps. This will give you on-going monitoring and testing. An example DAG using Etiq to determine dataset drift is available [here](https://github.com/ETIQ-AI/ml-testing/blob/main/integrations/airflow/dags/etiq_feature_drift.py). ![Example DAG for feature & target drift detection using Etiq](/files/p8gVMEJM6sRqcmxoCbRm) We also recommend setting up multiple tests at different points in your DAGs. The benefit is that all the test results will be centralized in your dashboard instance. This will give you a view of how your pipelines are performing at every single step. We are adding additional tagging functionality to make it easy for you to group the tests and instantly see which test failure happen at which point in your DAG. You can also use Etiq tests as triggers. For instance, you can set-up a DAG in such a way that: if a drift test fails the next step is automated model retrain. ### Separate container - Etiq + Airflow The second instance in which you can use Etiq and Airflow together is if irrespective of your orchestration or deployment set-up, you want to automate testing/monitoring using Etiq and Airflow. We provide an out-of-the-box [docker-compose script](https://github.com/ETIQ-AI/ml-testing/blob/main/integrations/airflow/docker-compose.yaml) for you with appropriate settings. Only requirements for using this container is docker-compose and setting up the environmental variables if different from defaults provided. ### Environmental variables The DAG can be used in your own Airflow environment. The following environmental variables can be defined * **AIRFLOW\_VAR\_ETIQ\_CONFIG** - The location of the config file to be used with etiq. An example etiq config file is available [here](https://github.com/ETIQ-AI/ml-testing/blob/main/integrations/airflow/config/config.json). * **AIRFLOW\_VAR\_ETIQ\_DATA** - The data directory location i.e. the location where the base and latest sub-directories are located. * **AIRFLOW\_CONN\_ETIQ\_FS** - Defines the airflow connection (etiq\_fs) to be used for the config file and datasets. * **AIRFLOW\_VAR\_ETIQ\_PROJECT** - The etiq project name to use. * **AIRFLOW\_VAR\_ETIQ\_DASHBOARD** - (Optional) The location for the etiq dashboard to log results (e.g. , or wherever your dashboard is deployed on your cloud instance) * **AIRFLOW\_VAR\_ETIQ\_TOKEN** - (Required if dashboard variable is set) The token to use to login to the dash board specified in *AIRFLOW\_VAR\_ETIQ\_DASHBOARD***.** {% hint style="info" %} Note that, currently, datasets have to be csv based. For specific database integrations or other orchestration tools reach out so us directly: {% endhint %} # Great Expectations Great Expectations Integration with ETIQ library details {% hint style="info" %} Feature added in Etiq 1.6 {% endhint %} ### Overview ETIQ adds integration with the OSS [Great Expectations](https://greatexpectations.io/) library. This allows you to quickly add a suite of tests for your dataset and show the results in the ETIQ dashboard. The Great Expectations python library needs to be installed if you want to use this functionality; `pip install great_expectations` ### Use Cases The library exposes `Snapshot.scan_expectations()` through which we can run suites or import existing results. Suites can either come from existing contexts, manually added via code or via JSON. For the results to be shown in the ETIQ dashboard, you need to run `etiq.login()` before starting the scan. * [Running An Existing Expectation Suite](#running-an-existing-expectation-suite) * [Importing Existing Results](#importing-existing-results) * [Declaring Expectations in Code](#declaring-expectations-in-code) * [Declaring Expectations in JSON](#declaring-expectations-in-json-config) ### Running An Existing Expectation Suite If you've got an existing expectation suite you can pass it that suite as an argument to `Snapshot.scan_expectations()` ```python import great_expectations as ge context = ge.get_context() # What Suites do we have? suite_names = context.list_expectation_suite_names() # For example... chosen_suite = suite_names[0] # Now tell ETIQ to run them; (segments, issues, aggregates) = snapshot.scan_expectations(context=context, suite_name=chosen_suite) ``` ### Importing Existing Results If you've already run the suite, you can just pass those in. This will cause the results to be uploaded into the dashboard; ```python # existing_suite_checkpoint is a checkpoint from your own existing suite. my_results = existing_suite_checkpoint.run() # This will upload results to the dashboard if logged in. # And return the results in the etiq style of three datasets showing segments, # issues and aggregated issues as pandas datasets. (segments, issues, aggregates) = snapshot.scan_expectations(results=my_results) ``` ### Declaring Expectations In Code You can declare your expectations in code directly e.g. as part of a notebook. We supply a helper method `Snapshot.get_validator()` to quickly get a Great Expectations [validator object](https://docs.greatexpectations.io/docs/reference/api/validator/validator/Validator_class). This will then have all the expectations available to you. The Great Expectations website holds a useful [reference list](https://greatexpectations.io/expectations). ```python # Regular snapshot created in ETIQ snapshot = project.snapshots.create( name="My Simple expectations", dataset=dataset, model=etiq.model.DefaultXGBoostClassifier(), ) # This is our regular GX validator. validator = snapshot.get_validator() # Now we can declare our expectations as normal GX. For example; validator.expect_column_values_to_not_be_null("age") validator.expect_column_values_to_be_between("age", min_value=0, max_value=70) # Finally actually run the validation by calling `scan_expectations` segments, issues, aggregates = snapshot.scan_expectations(validator=validator) ``` These expectations are only run when `scan_expectations` is called. ### Declaring Expectations in JSON Config Much of the ETIQ library can be driven using a JSON config file. We provide a simple way to specify expectations in this text based format; {% code title="expectation\_config.json" %} ```json { "scan_expectations": { "json_suite": [ { "expect_column_values_to_not_be_null": "age" }, { "expect_column_values_to_be_between": { "column": "age", "min_value": 0, "max_value": 120, } }, ] } } ``` {% endcode %} The Python code is then just: ```python # ... from etiq import etiq_config # ... with etiq_config("expectation_config.json"): segments, issues, aggregates = snapshot.scan_expectations() ``` #### Syntax/Required Sections * `scan_expectations` - An Object. * `scan_expectations.json_suite` - A list of JSON expectations. Expectations can be specified in two ways; 1. If the expectation takes a single argument, then it can just be `{: }` 2. If the expectation takes more arguments, the argument itself should be an object where each key-value pair represents an argument and value. The above expectation translates to: ```python validator.expect_column_values_to_not_be_null("age") validator.expect_column_values_to_be_between(column="age", min_value=0, max_value=120) ``` ## Integration API Documentation #### Snapshot.get\_validator() -> Validator This convenience method returns a validator object for creating expectation suites against. #### Snapshot.scan\_expectations(validator, context, suite\_name, results) This is our entry point to run expectations etc. Unless using the JSON method above, you must pass one or more of these arguments in: * `validator` - A GX Validator object to run. * `context` - A GX context object. This would contain an existing suite to run. * `suite_name` - String, name of suite to run from the `context`. * `results` - Existing results to send to the dashboard. # FAQ ## What can Etiq help you with? Test and monitor your data and AI pipelines. We cover tabular data and pretty much any modelling methodology. We actively maintain Etiq for the following libraries: XGBoost, LightGBM, PyTorch, TensorFlow, Keras and scikit-learn, however the tool can be used on pretty much any models from any libraries. ## What issue types do you cover? Data issues & leakage, performance and robustness related issues, different types of drift, bias and ethics; for more details pls. see our [scan types](/etiq-1.x-documentation/scan-types/accuracy) section. ## How do you deploy Etiq? Sign-up for the dashboard, pip install the Etiq library, import it in your python or spark based IDE and you are good to go! For a quick start guide please see [this link.](/etiq-1.x-documentation/quickstart) ## How long will it take me to get Etiq up-to-speed? Some users report a time-to-value as low as 15 minutes. Because you can use Etiq as you would a library straight into any python or spark based IDE, you can use it straight away. ## What if I have extremely large data? Can Etiq support? Etiq has a spark version which can handle billion+ rows datasets comfortably. ## What if I want to code up my own metric to use in tests and for monitoring? Sure, we have ample functionality for you to include [your own metric.](/etiq-1.x-documentation/key-concepts/custom-tests) ## What if I want someone to tell me what tests and monitors to use? Ping us an email: . We have extensive templates for step-by-step testing for: time series, financial sector classification, recommender models and many others. ## How are you different from other testing/monitoring tools out there? Privacy first & lightweight - no data/models leave your environment and you can use Etiq straight from your IDE/on your laptop In depth testing - for any performance, drift or bias metrics (whether provided out-of-the-box or custom) we provide [in depth root cause analysis](/etiq-1.x-documentation/rca/rca-type-scans) for you to understand which segments of you data are underperforming and why. ## How can I integrate with orchestration tools? [Here](/etiq-1.x-documentation/integrations/airflow) is an example of how to use Etiq + Airflow. You can use Etiq with any orchestration tools and also within environments such as Sagemaker, Databricks and pretty much any python or spark based environment. ## Will my data leave my environment? Your datasets or models will never be stored in Etiq or go out of your environment. However, if you are using our SaaS instance, **results** data only and [data profiles (if you choose to store them)](/etiq-1.x-documentation/scan-types/data-issues#privacy-considerations), will be stored on the SaaS instance. If you are assessing to see if Etiq is fit for your purposes: if you'd prefer not to send your results data to the dashboard, don't link your testing to the dashboard and results data will only be stored locally on your laptop or cloud instance. You can use Etiq locally during your session via the IDE. When you decide to purchase Etiq, if you are on AWS you can do so with a one-click purchases from [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-q5opksxavexbs?sr=0-1\&ref_=beagle\&applicationId=AWSMPContessa). Alternatively you can reach out for enterprise solutions. ## How much will Etiq cost me? Depends on the functionality you need, please see pricing [here](https://etiq.ai/pricing) or directly from [AWS marketplace](https://aws.amazon.com/marketplace/pp/prodview-q5opksxavexbs?sr=0-1\&ref_=beagle\&applicationId=AWSMPContessa). ## How do I submit an issue/comment/ask for help? Just email us: [**info@etiq.ai**](mailto:info@etiq.ai) ## **Install best practice** We advise that (for security reasons): In environments where multiple users should have segmented access to different dataset, the etiq library should be installed in a different virtual environment for each individual user. In environments where the breach of segmented access is deemed high, imported modules should be inspected for manipulation of the Etiq.ai framework. Please always respect your organisation’s security policies and, in doubt, please contact the person in charge of security within your organisation. ### Licensing terms Please see our licensing terms [here](https://drive.google.com/file/d/174JYJv7JlUwc9FQDrQ-0A8_6c_j7oqJW/view?usp=sharing). # Low Level API More complex scans, such as bias\_sources, have underneath longer pipelines with multiple steps. This section describes these pipelines' structure. These can currently be used in the IDE and customized, but they will not be displayed in the dashboard. As this API evolves we will surface methods that allow you to add these pipelines to the dashboard as well. ## DataPipeline To follow the example analysis below, download the Adult dataset from or load it in the notebook as a Pandas dataframe from the samples included in the library. A demo notebook is available at [https://github.com/ETIQ-AI/demo/blob/main/DemoAdultLibrary03.ipynb](https://github.com/ETIQ-AI/demo/blob/main/DemoAdultLibrary01.ipynb) ```bash data = load_sample('adultdata') ``` The DataPipeline object has the model we'd like to evaluate, the dataset used to train it and associated fairness metrics. Below, we define the parameters for the debiasing process using the BiasParams structure. This allows us to specify the protected category (often a demographic feature you'd like to mitigate bias for) using the `protected` parameter; specify who is in the privileged and unprivileged groups (these are set using the `privileged` and `unprivileged` parameters respectively); specify what is the positive outcome and the negative outcome in this dataset (these are set using the `positive_outcome_label` and `negative_outcome_label` parameters respectively). ```python debias_param = BiasParams(protected='gender', privileged='Male', unprivileged='Female', positive_outcome_label='>50K', negative_outcome_label='<=50K') ``` {% hint style="info" %} Even if your model does not use the specific demographic features you want to identify bias for, you should include this in the dataset. (etiq will automatically exclude it later during any model refitting). It is important to note that the protected feature is removed from the dataset for the purposes of training a model and will only be used to evaluate the model for bias. {% endhint %} Specify transforms like Dropna or EncodeLabels to make sure data are numeric and without missing values. Preferably use your own transform functions. ```bash transforms = [Dropna, EncodeLabels] ``` The DatasetLoader reads in the data, applies any transformations, splits the data into training, validation and test datasets and sets aside the test dataset to avoid data leakage in your analysis. The training and validation datasets are loaded into the Dataset class. ```bash dl = DatasetLoader(data=data, label='income', transforms=transforms, bias_params=debias_param, train_valid_test_splits=[0.8, 0.1, 0.1], names_col = data.columns.values) ``` Choose the metrics you want computed for this project. ```bash metrics_initial= [accuracy, equal_opportunity] ``` Each of these metrics measure how well our model is performing when classifying the data. For more details please see [Definitions.](broken://pages/-MhOSvRAH6Oj3oODRueX#how-is-bias-measured) Load the model you'd like to evaluate with the dataset or choose one of the classifiers that are already available. For this test release these `DefaultXGBoostClassifier` (a wrapper around XGBoost classifier), `DefaultRandomForestClassifier` (a wrapper around the random forest classifier from sklearn) and `DefaultLogisticRegression` (a wrapper around the logistic regression classifier from sklearn). ```bash clf_model = DefaultXGBoostClassifier() ``` {% hint style="success" %} You ***can*** use a pre-trained model and are not restricted to the model classes we have wrappers for. We just provided some widely-used model classes for ease of use. Models from other libraries (Etiq supports models from XGBoost, LightGBM, PyTorch, TensorFlow, Keras and scikit-learn) may be used by wrapping them in the Etiq`Model` class . We could, for example, create an LGBMClassifier model, train it and use the trained model. ```python import lightgbm as lgb lgb_model = lgb.LGBMClassifier() fitted_lgb = lgb_model.fit(X_train, y_train) clf_model = Model(model_architecture=lgb_model, model_fitted=fitted_lgb) ``` {% endhint %} Now you can create the DataPipeline. The DatasetLoader class will take the data, transform it, split it into training/validation/testing data and load it in. The DataPipeline computes your metrics of interest on the Dataset, using the model you provided. ```bash pipeline_initial = DataPipeline(dataset_loader=dl, model=clf_model, metrics=metrics_initial) pipeline_initial.run() ``` {% hint style="warning" %} Remember your dataset has as many features as you want but in this limited release library the DataPipeline will only pick up on the first 15 features {% endhint %} ## DebiasPipeline DebiasPipeline takes as inputs a data pipeline, an identify and/or repair method and metrics you want to use to evaluate your model. Identify methods are as the name suggests are intended to help you identify bias issues. Repair methods are designed to help fix or mitigate the issues identified and include implemented algorithms from the fairness literature. The current repair pipeline we provide is at the pre-processing level, i.e. changes the dataset with the objective that some of the sources of bias in it will be mitigated. Other methods at in-processing or post-processing stages will be more effective from an optimization point of view, but they might not address some of the issues in the data, which is why this is a good starting area. In our full solution we have additional pipelines. An example debiasing pipeline is given below ```bash identify_pipeline = IdentifyBiasSources(nr_groups=20, # nr of segments based on using unsupervised learning to group similar rows train_model_segment=True, group_def=['unsupervised'], fit_metrics=[accuracy, equal_opportunity]) # the DebiasPipeline aims to mitigate sources of bias by applying different types of repair algorithms # the library offers implementations of repair algorithms described in the academic fairness literature repair_pipeline = RepairResamplePipeline(steps=[ResampleUnbiasedSegmentsStep(ratio_resample=1)], random_seed=4) debias_pipeline = DebiasPipeline(data_pipeline=pipeline_initial, model=xgb, metrics=metrics_initial, identify_pipeline=identify_pipeline, repair_pipeline=repair_pipeline) debias_pipeline.run() ``` {% hint style="success" %} IdentifyBiasSources is the type of pipeline you are using. For this test release we are providing this pipeline. Similarly RepairResamplePipeline denotes what type of repair pipeline it is. As a convention anything that is a pipeline type is \Pipeline The parameters for the identify pipeline available in this release are as follows: * group\_definition = unsupervised. This is a type of pipeline method that looks for groups (i.e. segments of the dataset) that have issues that could cause bias. In our test version we have only released one option but in our full package we have multiple options. * nr\_groups - Experiment with a few different options based on how large your dataset is. This refers to how many groups/segments you think your dataset could be split into. Remember this is just one of the pipelines we provide and arguably not our most interesting one. If you want to explore using our other pipelines get in touch with us: {% endhint %} As with the data pipeline, when running the pipeline, we get the logs of how the pipeline has run: ```bash INFO:etiq_core.pipeline.DebiasPipeline36:Starting pipeline INFO:etiq_core.pipeline.DebiasPipeline36:Start Phase IdentifyPipeline844 INFO:etiq_core.pipeline.IdentifyPipeline844:Starting pipeline INFO:etiq_core.pipeline.IdentifyPipeline844:Completed pipeline INFO:etiq_core.pipeline.DebiasPipeline36:Completed Phase IdentifyPipeline844 INFO:etiq_core.pipeline.DebiasPipeline36:Start Phase RepairPipeline558 INFO:etiq_core.pipeline.RepairPipeline558:Starting pipeline INFO:etiq_core.pipeline.RepairPipeline558:Completed pipeline INFO:etiq_core.pipeline.DebiasPipeline36:Completed Phase RepairPipeline558 INFO:etiq_core.pipeline.DebiasPipeline36:Refitting model INFO:etiq_core.pipeline.DebiasPipeline36:Computed metrics for the repaired dataset INFO:etiq_core.pipeline.DebiasPipeline36:Completed pipeline ``` {% hint style="info" %} In the fairness literature, mitigation is considered to be the likely terminology as these types of issues are hard to remove entirely. Our usage of the term repair & debias refers primarily to mitigation, rather than removal. {% endhint %} ## Output methods Now that you've checked the logs and the etiq pipeline ran, to retrieve the outputs, use the following methods: #### Metrics ```python debias_pipeline.get_protected_metrics() ``` Example output: ```bash {'DataPipeline502': [{'accuracy': ('privileged', 0.84, 'unprivileged', 0.93)}, {'equal_opportunity': ('privileged', 0.6901408450704225,'unprivileged',0.55)}], 'DebiasPipeline426': [{'accuracy': ('privileged', 0.82, 'unprivileged', 0.91)}, {'equal_opportunity': ('privileged', 0.6539235412474849,'unprivileged', 0.65)}]} ``` #### Issues found by the pipeline Our library is intended for you to test your models and see if there are any issues. The pipeline surfaces potential issues, and then it's up to you whether you consider them to be issues for your specific model or not. For more details on definitions please see Definitions tab ```bash debias_pipeline.get_issues_summary() ``` Example output ![](/files/-MhUgnvMRRnYXXRiBshm) To help make sense of the segments, we also have a profiler method which gives you an idea about the rows found to have specific issues. ```bash debias_pipeline.get_profiler() ``` ![](/files/-MhUhbVnaOuMIrFVcCzQ) To understand more about the types of errors this pipeline finds, please use the following method; it will give you definitions and thresholds used. Also, please see a discussion of different bias sources at this[ link](https://etiq.ai/research/sources-of-unintended-bias-in-training-data). ``` debias_pipeline.get_thresholds() ``` #### # 1.2 Functionality If you have purchased Etiq v 1.2 via AWS marketplace - use these docs. Provides bias identification functionality and a different dashboard. Version 1.3 will be available via AWS Marketplace shortly. ## Use cases & limitations A typical use case for etiq: Let's say you are building a predictive model using tabular customer data. You have wrangled your data and tried a few model classes. Now you want to see if your model is discriminating unintentionally against certain demographic groups, e.g. based on gender, ethnicity, age, etc. and you *do* have access to the demographic label. This is where you can use the etiq library. Etiq library provides different kinds of pipelines that are intended to plug in to your existing pipelines and test them for a specific purpose. The pipelines currently available focus on identifying and mitigating unintended discrimination. Etiq pipelines provide identify methods, repair methods, metrics to evaluate outcomes including fairness metrics. For more details on the theoretical underpinnings of our methods go to Definitions. We'd like to stress that the 'fairness' literature and methodology is a very wide field, with a lot of divergent opinions. Where applicable we will refer to the framework we are using, but some of our approaches are experimental. In addition to the library the solution also includes a dashboard that presents results of the different pipelines logged by the library. This additional functionality includes the ability to retrieve results of pipelines from one session to another. {% hint style="info" %} If you want support from us, or submit any comments, feature requests, issues or bugs, please login to our [slack channel](https://etiqcore.slack.com) or email us: {% endhint %} ## Quickstart The Etiq library supports Python versions 3.6, 3.7, 3.8 and 3.9 on Windows, Mac and Linux. We do not support Mac m1 at the moment. We recommend using `pip` to install the Etiq library and its dependencies. ``` pip install etiq-core ``` From your python environment import etiq\_core ``` from etiq_core import * ``` Once you have imported the library go to the dashboard site on your AWS-hosted version, sign-up and login. To start storing metrics you ran from your notebook or other IDE to your dashboard you will need a token to associate your session with your account. To create this token, once in your account go to the Token Management window and just click on Add New Access Token. Then copy and paste into your notebook. ![](/files/zk8v5jYuWHpyBX9ep4nm) From your notebook just login to the dashboard and you're all set to go. Now as you log different pipelines and debiasing pipelines and tie them to a project you'll be able to retrieve them both via the dashboard and via your notebook across sessions. ``` etiq_login("", "") ``` {% hint style="warning" %} Data about your pipelines and debiasing pipelines get stored on Etiq's AWS instance. However your datasets and models will not actually be stored anywhere, so you can rest assured. If your security set-up is such that you would need a deployment entirely on your cloud instance or on prem just get in touch with us - {% endhint %} Please don't leave your token lying around as if anyone finds it they can use it to retrieve information stored about your pipelines. Similarly to how you use a password/username authentication. ## Projects To start using the versioning and dashboard functionality, please set a project and a project name. You only have to run it once per session and all the details logged as part of data pipelines or debias pipelines will be stored. Once you go to your dashboard you will be able to see the metrics of all your pipelines & debiasing pipelines logged split by the project name. ``` #start the project our_project = Project(name="TestAdult") ``` ## DataPipeline To follow the example analysis below, download the Adult dataset from or load it in the notebook as a Pandas dataframe from the samples included in the library. A demo notebook is available [here](https://github.com/ETIQ-AI/ml-testing/tree/main/Legacy_1.2.4) ``` data = load_sample('adultdata') ``` The DataPipeline object has the model we'd like to evaluate, the dataset used to train it and the fairness metrics that are most relevant to our project. Below, we define the parameters for the debiasing process using the BiasParams structure. This allows us to specify the protected category (often a demographic feature you'd like to mitigate bias for) using the `protected` parameter; specifiy who is in the privileged and unprivileged groups (these are set using the `privileged` and `unprivileged` parameters respectively); specify what is the positive outcome and the negative outcome in this dataset (these are set using the `positive_outcome_label` and `negative_outcome_label` parameters respectively). ``` debias_param = BiasParams(protected='gender', privileged='Male', unprivileged='Female', positive_outcome_label='>50K', negative_outcome_label='<=50K') ``` {% hint style="info" %} Even if your model does not use the specific demographic features you want to identify bias for, you should include this in the dataset. (etiq will automatically exclude it later during any model refitting). It is important to note that the protected feature is removed from the dataset for the purposes of training a model and will only be used to evaluate the model for bias. {% endhint %} Specify transforms like Dropna or EncodeLabels to make sure data are numeric and without missing values. ``` dl = DatasetLoader(data=data, label='income', transforms=transforms, bias_params=debias_param, train_valid_test_splits=[0.8, 0.1, 0.1], names_col = data.columns.values) ``` Choose the metrics you want computed for this project. ```bash metrics_initial= [accuracy, equal_opportunity] ``` Each of these metrics measure how well our model is performing when classifying the data. For example the **`accuracy`** metric returns the fraction of the training dataset which is correctly classified. The **`equal_opportunity`** metric measures the difference in true positive rate between a privileged demographic group and an unprivileged demographic group. The other available metrics used to evaluate model performance are * **`accuracy`** (proportion of outcomes correctly classified out of total outcomes) * **`true_neg_rate`** (the proportion negative outcome labels that are correctly classified out of all negative outcome labels) * **`true_pos_rate`** (the proportion positive outcome labels that are correct out of all positive outcome labels`)` * **`demographic_parity`** (the difference between number of positive labels out of total from a privileged demographic group vs. a unprivileged demographic group) * **`equal_odds_tpr & equal_odds_tnr`** (unlike with equal\_opportunity, this criteria looks at difference between true positive rate - privileged vs. unpriviledge and true negative rate - privileged vs. unprivileged, with the aim of ensuring that the difference for both metrics are minimal) {% hint style="info" %} For a discussion on how metrics behave and recommended usage, please see our [blogpost.](https://etiq.ai/research/how-fairness-metrics-can-be-misleading) {% endhint %} Load the model you'd like to evaluate with the dataset or choose one of the classifiers that are already available. For this release these are the available wrappers: `DefaultXGBoostClassifier` (a wrapper around XGBoost classifier), `DefaultRandomForestClassifier` (a wrapper around the random forest classifier from sklearn) and `DefaultLogisticRegression` (a wrapper around the logistic regression classifier from sklearn). ``` clf_model = DefaultXGBoostClassifier() ``` You ***can*** use a pre-trained model and are not restricted to the model classes we have wrappers for. We just provided some widely-used model classes for ease of use. Models from other libraries (Etiq supports models from XGBoost, LightGBM, PyTorch, TensorFlow, Keras and scikit-learn) may be used by wrapping them in the Etiq`Model` class . We could, for example, create an LGBMClassifier model, train it and use the trained model. ``` import lightgbm as lgb lgb_model = lgb.LGBMClassifier() fitted_lgb = lgb_model.fit(X_train, y_train) clf_model = Model(model_architecture=lgb_model, model_fitted=fitted_lgb) ``` Now you can create the DataPipeline. The DatasetLoader class will take the data, transform it, split it into training/validation/testing data and load it in. The DataPipeline computes your metrics of interest on the Dataset, using the model you provided. ``` pipeline_initial = DataPipeline(dataset_loader=dl, model=clf_model, metrics=metrics_initial) pipeline_initial.run() ``` ## DebiasPipeline DebiasPipeline takes as inputs a data pipeline, an identify and/or repair method and metrics you want to use to evaluate your model. Identify methods are as the name suggests are intended to help you identify bias issues. Repair methods are designed to help fix or mitigate the issues identified and include implemented algorithms from the fairness literature. The current repair pipeline we provide is at the pre-processing level, i.e. changes the dataset with the objective that some of the sources of bias in it will be mitigated. Other methods at in-processing or post-processing stages will be more effective from an optimization point of view, but they might not address some of the issues in the data, which is why this is a good starting area. In our full solution we have additional pipelines. An example debiasing pipeline is given below ``` identify_pipeline = IdentifyBiasSources(nr_groups=20, # nr of segments based on using unsupervised learning to group similar rows train_model_segment=True, group_def=['unsupervised'], fit_metrics=[accuracy, equal_opportunity]) # the DebiasPipeline aims to mitigate sources of bias by applying different types of repair algorithms # the library offers implementations of repair algorithms described in the academic fairness literature repair_pipeline = RepairResamplePipeline(steps=[ResampleUnbiasedSegmentsStep(ratio_resample=1)], random_seed=4) debias_pipeline = DebiasPipeline(data_pipeline=pipeline_initial, model=xgb, metrics=metrics_initial, identify_pipeline=identify_pipeline, repair_pipeline=repair_pipeline) debias_pipeline.run() ``` IdentifyBiasSources is the type of pipeline you are using. For this release we are providing this pipeline. Similarly RepairResamplePipeline denotes what type of repair pipeline it is. The parameters for the identify pipeline available in this release are as follows: * group\_definition = unsupervised. This is a type of pipeline method that looks for groups (i.e. segments of the dataset) that have issues that could cause bias. In our test version we have only released one option but in our full package we have multiple options. * nr\_groups - Experiment with a few different options based on how large your dataset is. This refers to how many groups/segments you think your dataset could be split into. {% hint style="info" %} Version 1.3 has more functionality across bias pipelines and other areas. {% endhint %} As with the data pipeline, when running the pipeline, we get the logs of how the pipeline has run: ``` INFO:etiq_core.pipeline.DebiasPipeline36:Starting pipeline INFO:etiq_core.pipeline.DebiasPipeline36:Start Phase IdentifyPipeline844 INFO:etiq_core.pipeline.IdentifyPipeline844:Starting pipeline INFO:etiq_core.pipeline.IdentifyPipeline844:Completed pipeline INFO:etiq_core.pipeline.DebiasPipeline36:Completed Phase IdentifyPipeline844 INFO:etiq_core.pipeline.DebiasPipeline36:Start Phase RepairPipeline558 INFO:etiq_core.pipeline.RepairPipeline558:Starting pipeline INFO:etiq_core.pipeline.RepairPipeline558:Completed pipeline INFO:etiq_core.pipeline.DebiasPipeline36:Completed Phase RepairPipeline558 INFO:etiq_core.pipeline.DebiasPipeline36:Refitting model INFO:etiq_core.pipeline.DebiasPipeline36:Computed metrics for the repaired dataset INFO:etiq_core.pipeline.DebiasPipeline36:Completed pipeline ``` {% hint style="info" %} In the fairness literature, mitigation is considered to be the likely terminology as these types of issues are hard to remove entirely. Our usage of the term repair & debias refers primarily to mitigation, rather than removal. {% endhint %} ## Output methods Now that you've checked the logs and the etiq pipeline ran, to retrieve the outputs, use the following methods: #### Metrics ```python debias_pipeline.get_protected_metrics() ``` Example output: ``` {'DataPipeline502': [{'accuracy': ('privileged', 0.84, 'unprivileged', 0.93)}, {'equal_opportunity': ('privileged', 0.6901408450704225,'unprivileged',0.55)}], 'DebiasPipeline426': [{'accuracy': ('privileged', 0.82, 'unprivileged', 0.91)}, {'equal_opportunity': ('privileged', 0.6539235412474849,'unprivileged', 0.65)}]} ``` #### Issues found by the pipeline Our library is intended for you to test your models and see if there are any issues. The pipeline surfaces potential issues, and then it's up to you whether you consider them to be issues for your specific model or not. For more details on definitions please see Definitions tab ```bash debias_pipeline.get_issues_summary() ``` Example output ![](/files/-MhUgnvMRRnYXXRiBshm) To help make sense of the segments, we also have a profiler method which gives you an idea about the rows found to have specific issues. ```bash debias_pipeline.get_profiler() ``` ![](/files/-MhUhbVnaOuMIrFVcCzQ) To understand more about the types of errors this pipeline finds, please use the following method; it will give you definitions and thresholds used. Also, please see a discussion of different bias sources at this[ link](https://etiq.ai/research/sources-of-unintended-bias-in-training-data). ``` debias_pipeline.get_thresholds() ``` {% hint style="info" %} In release 1.3 you are able to customize the definitions and thresholds. We will make this release available on AWS Marketplace shortly. {% endhint %} #### Evaluate method If you've just built a pipeline using a repair method and want to see if the issues you've identified before, use the evaluate method ```bash evaluate_debias = EvaluateDebiasPipeline(debias_pipeline=debias_pipeline, identify_pipeline=identify_pipeline) evaluate_debias.run() evaluate_debias.get_issues_summary_before_repair() evaluate_debias.get_issues_summary_after_repair() ``` ## Results Retrieval across Sessions To see all your projects and pipelines from your notebook or IDE use the methods below: ``` projects = get_all_projects() projects ``` This should give you an output like the one below: ``` [, ] ``` If you just opened a new session but want the pipelines and debiasing pipelines to be logged as part of the project you used in your previous session make sure you find the ID of the project and set your current project to the id you are looking for: ``` set_current_project(projects[1]) ``` To see what pipelines are associated with the project, use the methods below: ``` our_project.get_all_data_pipelines() #get pipelines by type, use pipeline type name, e.g. IdentifyBiasSources our_project.get_all_pipelines_by_type(IdentifyBiasSources) ``` #### # 1.2 Example Notebooks For example notebooks for 1.2 check out [this repo](https://github.com/ETIQ-AI/ml-testing/tree/main/Legacy_1.2.4) and in particular [this notebook](https://github.com/ETIQ-AI/ml-testing/blob/main/Legacy_1.2.4/DemoAdultLibrary03.ipynb). # 1.2 Definitions ## What is algorithmic bias? "Algorithmic bias" refers to unintended discrimination occurring as a result of an automated decision. The term "protected feature" refers to a specific demographic characteristic such as age or sex. Legislation defines a series of protected features. For example, in the UK, citizens are protected against discrimination on the basis of age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex or sexual orientation status by the Equality Act 2010. The unprivileged group within the protected feature (for example, people over 65 when age is the protected feature) tends to be discriminated against and as a result tends to be the one protected by legislation. The privileged group within the protected feature tends to not be discriminated against. We are also using terminology such as "debias" in the library to expedite articulation. The consensus in the literature (and our view) is that algorithmic bias can be mitigated but not removed entirely. ## How is bias measured? There is no consensus on the most appropriate way to measure bias, however depending on the framework used, there are some key metrics worth knowing. Before we get into this, a quick explanation of model building. A model uses data to make predictions. During training, a model "learns" of a way to use training data to understand which combination of features predict a positive or negative outcome (labels). Testing the model on a validation or test dataset lets the user quantify how accurate are the model predictions. There are many ways to measure bias and this is an on-going research topic. One way to measure bias is to compute fairness metrics on the predictions and ground truth that a trained model makes for a dataset. The fairness metrics attempt to encode in mathematical terms a notion of what a fair outcome should be for the model and the dataset. It is important to consider whether a particular fairness metric encapsulates the notion of a fair model for each individual project. Some of the metrics commonly used in the algorithmic fairness literature that the Etiq library provides are: * Demographic parity - is the ratio of users predicted to be positive over all the users the same for all groups in a demographic? For instance, is the proportion of women accepted for an interview the same as the proportion of men? * Equal opportunity - is the model as accurate for all demographic groups? Is the true positive rate the same for all demographics? True positives rate measures the proportion of actual positives that are correctly identified as such (e.g., the percentage of sick people who are correctly identified as having the condition). If the true positives rate is lower for a group then likely that group is experiencing bias. * Equal odds - an extension on Equal opportunity. It does not look just at true positive rates but also at false negative rates for the different demographic groups to ensure that the model performs equally well for all the different groups. * Individual fairness - a different angle on bias is to ensure that customers who display the same characteristics are treated the same. This does not yet have a clear definition. Demographic parity, equal opportunity and equal odds are described in [this paper](https://arxiv.org/abs/1610.02413), and individual fairness is described in [this paper](https://arxiv.org/abs/1104.3913). {% hint style="info" %} The fairness & algorithmic bias literature is very complex and there is no consensus on how to measure and mitigate algorithmic bias. {% endhint %} ## What are some solutions and general approaches to the algorithmic bias problem? In our understanding of the fairness literature, below are the key general areas: **Optimization**: pre-processing, in-processing and post-processing methods which attempt to optimize for both fairness metrics and accuracy. Some examples include: mapping the training data to a space independent of the specific demographic, adversarial debiasing, calibrating the model once it's built. The repair approaches can be anywhere from repairs that are very non-intrusive, e.g. resampling to those that are changing the labels and feature distribution quite heavily. **Causality:** Causality type approaches overlap with both counterfactuals and optimization ones, but are firmly rooted in the idea that a dataset can be modelled into a causal graph which can then point if belonging to a certain demographic class impacts other feature and via them impacts the outcome. ## What framework do you use to measure, identify and mitigate the extent of the problem? For the pipeline we released we are using group metrics and sources of bias approaches. The sources of bias framework used relies [on this lecture](https://mrtz.org/nips17/#/). According to this framework, there are roughly 5 areas of sources of bias (within the model build process, outside issues like team diversity, data collection, etc.). 3 of them are visible from the data and/or model: * **proxies** - features that are proxy for demographics * **sample size disparity** - sample size for the protected demographic group is quite a bit lower than for the majority class * **limited features** - features might be less reliable for a certain demographic group than for a majority class The remaining 2 sources of bias require more background or context knowledge: * **'tainted' examples** - the target variable is reflective of past bias, e.g. a model predicting who might make a good hire using data on who was hired in the past not on who was the objectively best candidate for the role * **skewed sample** - the dataset is not representative of the population for which the model will be used As expected, different bias sources can be mitigated by different repairs. The repair we focus on at the moment is only at pre-processing stage - changing the dataset in such a way as to mitigate some of the inherent bias issues it presents. {% hint style="info" %} This pipeline is experimental. {% endhint %} 1.3 includes additional pipelines and we will make them available on AWS Marketplace shortly. If you want to use them just get in touch: **** ## I'm just starting to look into the algorithmic bias topic. What are some resources to read for further reference? If you are just starting off and are a fan of academic papers, take a look [at this survey paper](https://arxiv.org/abs/2010.04053). ## How many records should I use to get an outcome from this pipeline? Our test samples so far were on minimum 20K rows. If you try it on datasets fewer than 10K please submit any issues or questions on our [slack](https://etiqcore.slack.com) channel. ## How confident can I be of the results of the pipeline? Our goal is for our pipelines to be transparent enough in terms of the outcome that it will be clear to the user how reliable the results are. We are also working on adding stability measures to our library.